Stories
Slash Boxes
Comments

SoylentNews is people

posted by Fnord666 on Thursday April 12 2018, @10:31AM   Printer-friendly
from the for-the-"cloud" dept.

Submitted via IRC for AndyTheAbsurd

IBM is launching what it calls a "skinny mainframe" for cloud computing. The system is built around IBM z14 mainframe technology, and it features a 19-inch industry standard, single-frame case design, allowing for easy placement into public cloud data centers and for private cloud deployments.

[...] With the mainframe in high demand and more relevant than ever, IBM worked closely on the design with more than 80 clients, including managed service providers, online banks, and insurance firms, to reinvent the mainframe for a whole new class of users.

The new z14 and LinuxOne offerings also bring significant increases in capacity, performance, memory, and cache across nearly all aspects of the system. A complete system redesign delivers this capacity growth in 40 percent less space, standardized to be deployed in any data center. The z14 ZR1, announced today, can be the foundation for an IBM Cloud Private solution, creating a "data center in a box" by co-locating storage, networking, and other elements in the same physical frame as the mainframe server.

The z14 ZR1 delivers 10 percent more capacity than its predecessor, the z13s, and, at 8TB, twice the memory. The system can handle more than 850 million fully encrypted transactions per day.

Source: https://venturebeat.com/2018/04/09/ibm-launches-skinny-mainframe-for-the-cloud/

Also at The Register

Technical Introduction(IBM Redbook)


Original Submission

This discussion has been archived. No new comments can be posted.
Display Options Threshold/Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
(1)
  • (Score: 2) by JoeMerchant on Thursday April 12 2018, @11:54AM (2 children)

    by JoeMerchant (3937) on Thursday April 12 2018, @11:54AM (#665860)

    How many "transactions" per day does the average human do?

    If that number is ~100, then 1000 of these could serve the whole world's (human) transaction needs.

    Next question: for every human-to-machine "transaction" how many machine-to-machine transactions are there? Have we already been marginalized?

    --
    🌻🌻 [google.com]
    • (Score: 1, Funny) by Anonymous Coward on Thursday April 12 2018, @01:15PM (1 child)

      by Anonymous Coward on Thursday April 12 2018, @01:15PM (#665895)

      So you are saying the required number of computers has increased since 1943? (grin)
      https://en.wikipedia.org/wiki/Thomas_J._Watson#Famous_attribution [wikipedia.org]

      Although Watson is well known for his alleged 1943 statement, "I think there is a world market for maybe five computers," there is scant evidence he said it. Author Kevin Maney tried to find the origin of the quote, but has been unable to locate any speeches or documents of Watson's that contain this, nor are the words present in any contemporary articles about IBM. ...

         

      • (Score: 2) by TheRaven on Friday April 13 2018, @07:50AM

        by TheRaven (270) on Friday April 13 2018, @07:50AM (#666354) Journal
        If he did say it, then it might turn out to be prophetic if you count a cloud provider as a single computer. Microsoft, Amazon, Google and IBM account for something like 90% of all hosted computing and they're increasingly moving towards custom hardware and platforms that treat at least a rack, if not an entire data centre, as a single computer.
        --
        sudo mod me up
  • (Score: 2, Interesting) by conn8d on Thursday April 12 2018, @01:42PM

    by conn8d (6887) on Thursday April 12 2018, @01:42PM (#665912)

    The Indian Business Machine (IBM) is busy trying to be competitive in their home in India. I say good luck after firing all those US engineers.

  • (Score: 4, Insightful) by Nerdfest on Thursday April 12 2018, @01:47PM (7 children)

    by Nerdfest (80) on Thursday April 12 2018, @01:47PM (#665914)

    Anyone buying into one of these things is foolish. Better performance can be had at a tenth of the price, and you actually own it and don't need to pay IBM for "MIPS" or buying special processors to avoid the cost. At least you're not stuck with z/os anymore.

    • (Score: 2, Interesting) by Anonymous Coward on Thursday April 12 2018, @01:53PM

      by Anonymous Coward on Thursday April 12 2018, @01:53PM (#665916)

      I take it you are willing to port over old mainframe software for free? A working legacy system is worth much more than your vaporware replacement.

    • (Score: 4, Interesting) by martyb on Thursday April 12 2018, @04:09PM (3 children)

      by martyb (76) Subscriber Badge on Thursday April 12 2018, @04:09PM (#665993) Journal

      Anyone buying into one of these things is foolish. Better performance can be had at a tenth of the price, and you actually own it and don't need to pay IBM for "MIPS" or buying special processors to avoid the cost. At least you're not stuck with z/os anymore.

      Disclaimer: I worked at IBM testing updates to their VM Operating System [wikipedia.org] in the 1980's; things have likely changed a bit since then.

      The thing that struck me about IBM Mainframes was not phenomenal computing resources, but the tremendous amount of data that could flow through their system.

      A device, such as a 3270 terminal, was not connected directly to the mainframe. Nope, it would be connected to a concentrator (IIRC). Back in the day, one of these could handle inputs from something like 64 different terminals. It would gather up and buffer multiple requests into a single 'chunk' of data.

      So, the concentrator was connected to the mainframe, right?

      Nope.

      You see, the mainframe was so efficient at performing transactions on the data, that it could sit there waiting for data to process if all it had was data fed in from concentrators.

      So, concentrators were not up to the task by themselves, that's why multiple concentrators would feed into a single channel. And, you guessed it, there were multiple channels feeding into the mainframe itself.

      From ancient memories, an IBM mainframe with two processors, 32MB of main memory, and 256MB of extended memory could support on the order of 1,000 terminals. Oh yeah, along with a whole array of disk and tape drives hanging off the system, too.

      To put this in perspective, this is around the same timeframe that IBM released the IBM AT [wikipedia.org] personal computer which came with PC DOS 3.0 and sported a 6 MHz Intel 80286 microprocessor. Though it supported up to 16MB of RAM, the base configuration came with just 256 KB. The introductory price was $6000.

      tl;dr: Mainframes were important in large part because of the huge volume of data that they could rapidly and reliably process.

      PS: Forgot to mention that the other big thing about mainframes was legacy support. IBM took exceptional pains to make sure that its latest processors in the 360/370/etc. line were fully backward compatible with prior versions. One's investment in legacy code would carry forward in newer hardware releases.

      PPS: I would love to hear from people with more recent experience as to how well those traditions of throughput and backward compatibility have been maintained!

      --
      Wit is intellect, dancing.
      • (Score: 2) by looorg on Thursday April 12 2018, @04:26PM (2 children)

        by looorg (578) on Thursday April 12 2018, @04:26PM (#666003)

        It's been about a decade since I worked on them but what you said still hold/held true back then. You don't buy a Mainframe to crunch numbers (it would probably be a horrible machine for cryptocurrency and such), you buy (or rent) one to crunch transactions -- obscene amounts of them. That is what it was built for and that is what it's superior at -- today and for the last half decade

        If the concern is that you are stuck in z\os that isn't much of an issue anymore with the z\vm you can launch everything from old classics to solaris (it might not be that common anymore, don't know if that is even maintained anymore -- could be some opensolaris I guess) and linux (yes there is linux for the mainframe). That said if you wanna do what the machine is built for I don't see why you would really run any of that, but as noted every single user or program can live happily in it's own little VM and chug along. But it really doesn't take away from the fact that it's a machine built for processing all while maintaining total legacy and security. I don't think I have ever actually seen one really crash, sure disks break and power fucks out sometimes etc but the machine just ticks on with the redundancy -- the only thing you notice is the little message notification that it's time to replace some faulty components.

        • (Score: 2) by Nerdfest on Thursday April 12 2018, @05:15PM (1 child)

          by Nerdfest (80) on Thursday April 12 2018, @05:15PM (#666033)

          I've seen them crash several times (mostly z26, I think?) ... but every time because of operator error, which is pretty much the same on all hardware/OS these days (non-Windows) Redundancy gets you that cheaper, and with more power. Data throughput has caught up as well. It's a very bad solution to all but an extremely small set of problems. Even the remaining ones are generally solved in other ways using OTS stuff, especially for availability of people/knowledge.

          As I've said before, the worst thing about it in my experience, is the "mainframe attitude". Nothing can change ... ever. There is an *insane* amount of process that has no value other than "because that's the way we've always done it". I think they're having trouble indoctrinating a new generation, but the money is getting temping for working on the dinosaurs.

          COBOL ... *shiver*

          • (Score: 3, Funny) by looorg on Thursday April 12 2018, @05:44PM

            by looorg (578) on Thursday April 12 2018, @05:44PM (#666059)

            I'm unsure where most of them are these days but it's probably still the backbone in banking, finance, insurance and some transaction heavy commerce.
              I can't deny the attitude or mindset, it's a lot of "this is how it has always been" which might sound bad at first but there are also positive aspects such as you don't have to jump from trend to trend like some others seem to do. Which naturally might clash with this whole skinny cloud mainframe salesrep talk.
              I'm going to wait a decade or two then brush off my ibm-suit before ever thinking about doing it again -- upside being that NOTHING will have changed.

    • (Score: 3, Interesting) by JoeMerchant on Thursday April 12 2018, @09:41PM

      by JoeMerchant (3937) on Thursday April 12 2018, @09:41PM (#666170)

      They've been pushing this "lease the capacity you use" model for a long time. It makes some sense if you let them put the hardware in your rack and you're only paying for 10% of its capability, but have that option to scale up at any time just by paying more when you need it.

      We looked at their 2006 offering (not much different from today's except 12 years less capable) - and it just didn't make sense for us because we would be saturating the device's capacity for 5 minutes, then letting it stand idle for 55 minutes, then saturating it again - so we had to pay full fare for the compute power to be on-demand like that, and they were coming in at about 2x the price of similar power in a rack of Mac Pros at the time (when you untangle lease vs buy options etc.)

      --
      🌻🌻 [google.com]
    • (Score: 2) by TheRaven on Friday April 13 2018, @07:54AM

      by TheRaven (270) on Friday April 13 2018, @07:54AM (#666356) Journal
      Most people who use cloud computing resources build systems where redundancy is engineered in at a high level. The goal is to split work across a lot of nodes and not care too much if a single node fails. Most people who run software on a mainframe write code that works and assume that the hardware never ever fails. It's much cheaper to develop software for the mainframe model, but it's much cheaper to buy compute resources for the cloud model. I can imagine quite a big market for people wanting to be able to scale up software without having to worry about handling reliability in software.
      --
      sudo mod me up
  • (Score: 4, Informative) by DannyB on Thursday April 12 2018, @03:38PM (17 children)

    by DannyB (5839) Subscriber Badge on Thursday April 12 2018, @03:38PM (#665970) Journal

    From TFA . . .

    IBM wants the aging traditional mainframe to survive in the era of digital business, and it argues that the mainframe remains the most stable, secure, and mature environment for information technology, including blockchain-based processing.

    Big internet players (Google, Facebook, Twitter, Amazon, etc) have figured out how to do big boy computery stuff in the 21st century. The hardware is standardized, and it is commodity processing units. See Open Compute [opencompute.org] for example of how well the hardware is specified. It's all well thought out. Including using all DC power, backup batteries, standard sized trays, etc.

    It was nice that IBM ported Linux to mainframes back in 2000. And it was a huge boost to open source credibility. And it was good that IBM was foolishly selected by idiot SCO as their first ill-conceived target for the extortion racket troll booth against Linux. Because IBM was able to defend this, having deep pockets. So appreciation to IBM.

    It is great that you can run 41,000 instances of Linux on a single mainframe box, and replace huge numbers of other boxes. But there is an elephant in the room. All of that risk is now concentrated in a single cabinet. And exotic, expensive, and proprietary hardware. And you pay for every tiny nickel-and-dime thing you might use. It's much easier to just own a thousand cabinets of commodity hardware, where you really own it, and don't have to constantly buy some new feature for every capability you might want to use.

    Projects like Kuberneties (and similar) allow running "Docker" (or other) container formats on gigantic clusters of machines. Kubernetes is to a Cluster of CPUs, as Linux is to a single CPU. Imagine the olden days of five years ago. You've got a Hadoop cluster. (Or some other specialized cluster setup.)
    Jim says: "Hey, Joe, can I use the Hadoop cluster today?".
    Joe: "No, sorry, I won't be done with it until mid afternoon."
    That sounds a log like in ancient times before the OS:
    Jim says: "Hey, Joe, can I use the computer today".
    Joe: "No, sorry my punch-card batch job won't be done until mid afternoon."

    With Kuberneties, you would make each Hadoop node be a Docker image. Now Jim can simply ask Kubernetes to spin up a 60 node Hadoop, and then Jim can use that. Jim doesn't have to interrupt Joe who has spun up his own 80 node Hadoop cluster. When Joe is done with his Hadoop cluster, once his mid afternoon job completes, Joe will shut down his cluster. All those docker containers stop running. Neither Joe nor Jim care about what each other are doing. Sort of how when Joe and Jim share a single Linux box, they can each make independent use of it. Similarly, Joe and Jim can independently make use of the cluster hardware, without worrying about how many physical nodes actually exist within the hardware. If the cluster only has 40 physical nodes, then Joe's 80-node cluster job runs on those 40 physical nodes -- sort of like how Linux might pretend you have more memory than you actually have. Increasing the nodes in the physical cluster is the job of the data center people and is invisible to Joe and Jim, other than a sudden performance increase.

    It's not just for Hadoop. ANY type of cluster software that can be "dockerized" can run on Kubernetes. Have a cluster of PostgreSQL? Run it on kubernetes. Someone else can spin up their own independent database cluster -- without building a physical cluster.

    Kubernetes abstracts away the cluster hardware the same way Linux abstracts away a single node's hardware.

    Now what do I need mainframes for again? Aren't mainframes mostly about batch processing? Isn't their entire history about huge batch processing. First you run the Payroll Exception Processing on the mainframe. Then you change tapes, cards, etc. And run the Payroll Repeating Entry processing. (eg, "exception" is things that vary per pay cycle like time cards, "repeating entry" is things like salary or non-varying items on every pay cycle) Then you run the big batch job for Payroll Calculation. Then the batch job that prints the Payroll Register, so someone can review or skim over all of the checks that will be written. Then run the Payroll Paycheck Printing batch job. Then run the End Pay Cycle batch job that updates all records (cards / tapes) in prep for the next time you do payroll cycle.

    Or the city utility billing system. Again a series of batch jobs.

    The new term "Big Data" that you run on, say, Hadoop, isn't that just a 21st century big batch job? Map/Reduce spread across many hardware nodes because, for example, each individual employee's payroll processing is independent of everyone else's, and thus the job is "embarrassingly parallel" and works on a cluster.

    So why do I need mainframes again, and their proprietary hardware, and pricing?

    Because mainframes are extremely reliable. And yes they are. But so are cluster computers -- especially managed by tools like kubernetes. Individual node failures are tolerated. Even expected. So trying to build this amazing never fail type reliability that mainframes have seems unnecessary. The reliability is in the statistics of how many nodes you expect not to fail today.

    Furthermore, all of what I described is so standard that I can set up my docker containers and run on Kubernetes on Google or Amazon without even owning the hardware. Effectively, spin up a virtual kubernetes cluster and run it, then throw it away.

    So really, what do mainframes offer again?

    Another thing: docker containers isolate the technology (and its entire stack). I get a docker container that does a particular thing, and I don't care if it was written in PHP, or Python, or Node.js or Java, or whatever. Docker will marshal all the resources to build the container image from the docker specification file and just run it within an isolated environment. I don't care how it works, or how rube-goldberg it is within that container -- as long as it works.

    --
    People today are educated enough to repeat what they are taught but not to question what they are taught.
    • (Score: 3, Insightful) by looorg on Thursday April 12 2018, @04:10PM (2 children)

      by looorg (578) on Thursday April 12 2018, @04:10PM (#665994)

      Seriously the Mainframe era has been pronounced dead so many times now it's not even funny anymore, reading X whitepaper on how UNIX is dead has nothing on the once that proclaim the death of the Mainframe. Yet it keeps on processing transaction after transaction like nobodies business.

      • (Score: 2) by Nerdfest on Thursday April 12 2018, @05:18PM

        by Nerdfest (80) on Thursday April 12 2018, @05:18PM (#666038)

        It's hard and expensive to extract yourself from a proprietary solution you've tied your business too.

      • (Score: 2) by DannyB on Thursday April 12 2018, @06:53PM

        by DannyB (5839) Subscriber Badge on Thursday April 12 2018, @06:53PM (#666103) Journal

        People already on mainframes will no doubt keep using them for a long time. They've got it all set up and figured out.

        Since everything about mainframes seems utterly alien to anyone in the mainstream of the computer industry, I wouldn't be surprised if the reverse were also true.

        However, mainframes are probably not used for new projects.

        I imagine mainframes will continue processing for a long time. Especially considering how reliable they are. The hardware is amazing. The things to support it. Like liquid cooling, etc. But a rack of commodity hardware has amazing capability too. The difference is one of mentality. Instead of extreme reliability in the hardware, build in the expectation and tolerance of hardware failure. As I said, the reliability is based on the statistics of how many cluster nodes will not fail today.

        --
        People today are educated enough to repeat what they are taught but not to question what they are taught.
    • (Score: 2) by bob_super on Thursday April 12 2018, @05:54PM (13 children)

      by bob_super (1357) on Thursday April 12 2018, @05:54PM (#666067)

      > The reliability is in the statistics of how many nodes you expect not to fail today.

      Have you quantified the cost of those failures, and the overhead of provisioning and servicing you need to handle them?
      Obviously, the mainframe isn't winning the overall battle today, but there are people who value reliability, and the security of not having unreliable nodes. The same way that there are people who value good tested programs over rush-out-then-update code.
      The mainframe isn't dead yet, and the billions it still generates indicate it's as not clear cut as you want to make it seem.

      • (Score: 2) by DannyB on Thursday April 12 2018, @07:05PM (12 children)

        by DannyB (5839) Subscriber Badge on Thursday April 12 2018, @07:05PM (#666108) Journal

        I cannot. But I bet Google can. I've even seen postings over the years talking about the aggregate failure rates of various components. And who would know better than those who operate large clusters of commodity hardware. (not just google)

        Isn't the cost of node failures basically the cost of fixing dead hardware, and possibly costs from a loss of performance code due to fewer compute resources available throughout the cluster. The cost of hardware is part of doing business. I suppose one should treat the performance issue the same way. If you expect XX number of compute nodes to fail today, then base your expectation of overall performance to be the entire cluster minus XX. Since there will be XX nodes out of service on any given day.

        As for reliability of the software, that is based on your software's tolerance of failed nodes. Suppose you run Hadoop. I would expect (but don't know) that it is already tolerant of cluster nodes failing. Similarly, a PostgreSQL cluster surely must be tolerant of nodes failing. The same would go for any other cluster-able software that you dockerize and run on a cluster.

        As for storage. You build fault tolerance into the storage fabric. This can even be done using commodity hardware rather than expensive solutions. There was some distributed filesystem project I was looking at months back when I was diving into learning some about kubernetes. Basically it could replicate storage across multiple nodes for redundancy. It could spread the cost of access based on the fact that the same data could be fetched from any node. And it was fault tolerant. Just how fault tolerant is dependent on how many replicas you want to specify. Also, this distributed filesystem could itself even run on kubernetes as I recall. But it would need access to the underlying persistent block storage devices.

        As for the cost of provisioning and servicing. I suppose Amazon, Google, and others have this figured out. If mainframes were indeed cheaper, I suspect they would have been the obvious logical choice.

        --
        People today are educated enough to repeat what they are taught but not to question what they are taught.
        • (Score: 2) by TheRaven on Friday April 13 2018, @07:58AM (11 children)

          by TheRaven (270) on Friday April 13 2018, @07:58AM (#666358) Journal

          I cannot. But I bet Google can. I've even seen postings over the years talking about the aggregate failure rates of various components. And who would know better than those who operate large clusters of commodity hardware. (not just google)

          Google can, and Google does. And when you're processing the volume of data that Google is, then it's much cheaper to hire bright people to design software systems that are tolerant to failures of individual nodes than it is to buy hardware that incorporates redundancy at every level and transparently performs error checking and correction and presents software with the illusion that hardware never fails. For some workloads, it's cheap to use an off-the-shelf solution like Hadoop to handle this for you.

          The interesting point for this is where you have a workload that a single machine or a small handful of machines could take care of, but where you really, really, care about reliability. You could design your software stack using distributed consensus algorithms and do extensive testing to make sure that it handled node failures. Or you could just write it the simple (and easy to validate) way and lease space on an IBM mainframe to run it. I suspect that, for a lot of smallish customers, the latter option is cheaper.

          --
          sudo mod me up
          • (Score: 2) by DannyB on Friday April 13 2018, @12:56PM (10 children)

            by DannyB (5839) Subscriber Badge on Friday April 13 2018, @12:56PM (#666438) Journal

            Google can, and Google does. And when you're processing the volume of data that Google is, then it's much cheaper to hire bright people to design software systems that are tolerant to failures of individual nodes than it is to buy hardware that incorporates redundancy at every level and transparently performs error checking and correction and presents software with the illusion that hardware never fails.

            Then Google makes this technology open source. And cooperates with others in its continued development and evolution.

            I remember reading that Google can fail over entire data centers. With technology like kubernetes, that makes sense. Containers could be executed in a different data center just the same. All it would take is that the scheduler be smart enough to prioritize the preferred cluster nodes, but be able to use lower priority clusters (eg, other data centers) when no nodes available on the preferred cluster.

            I also remember reading a google engineer saying that Google's network connectivity between data centers are better than some others connectivity within data centers.

            I wonder at what point does the reliability of this exceed the reliability of mainframes? Despite their amazing and exotic (expensive) hardware.

            --
            People today are educated enough to repeat what they are taught but not to question what they are taught.
            • (Score: 2) by TheRaven on Saturday April 14 2018, @10:11AM (9 children)

              by TheRaven (270) on Saturday April 14 2018, @10:11AM (#666856) Journal
              Google does make this technology open source, but that doesn't always help. You still need to employ developers who understand the frameworks well enough to map their problem to them. This means you need developers who understand the failure modes and latency issues of distributed consensus algorithms. Those things are complicated so those developers are not cheap.

              Reliability for algorithms that have been mapped to work on a distributed environment like a Google cloud already meets or exceeds that of mainframes, but each implementation requires a lot of careful design to map a problem to an unreliable substrate. That costs money, each time. It's worth doing for Google, because all of the jobs that they use this for are huge and the cost of building a mainframe large enough to handle them is infeasible. It's not worth doing for most people. If you have a job that can run on a handful of machines, it's a lot cheaper to run it on a slice of a mainframe than to write a version of it that works as a resilient distributed system.

              --
              sudo mod me up
              • (Score: 2) by DannyB on Saturday April 14 2018, @02:30PM (8 children)

                by DannyB (5839) Subscriber Badge on Saturday April 14 2018, @02:30PM (#666933) Journal

                I believe that all you must do is map your problem into Map and Reduce operations. The infrastructure makes sure that everything gets processed correctly.

                Suppose I have four items: A, B, C and D.
                Each item must have function f1 applied to it, to produce A1, B1, C1 and D1.

                {A1, B1, C1, D1} == Map f1 over {A,B,C,D}

                Now I need to apply function f2 over each of those . . .

                {A2, B2, C2, D2} == Map f2 over {A1,B1,C1,D1}

                Now that might be your answer. But in some cases, I want to apply an operator op1 to pairs in order to Reduce it to a single result. op1 is both associative and commutative.

                Result == (A2 op1 B2) op1 (C2 op1 D2)

                (Imagine that op1 is simple addition.)

                The infrastructure takes care of reliability for you. If applying function B2 = f2( B1 ) in the 2nd map operation were to fail, the infrastructure will re-run that function application on the same data again in order to get B2. These are pure functions with no side effects. So like, applying b2 = Sqrt( B1 ), another time on another node does not cause any problem if it fails, because it is a pure function.

                Surprisingly many batch and stream processing operations can be expressed as Map / Reduce and run on clusters. You don't have to even consider reliability, you just take it for granted. Just like when you run your single program on a desktop computer.

                Google does make extensive documentation available. This is not some obscure technology. Try googling or Youtubing 'kubernetes', 'docker', etc and you'll see that this is already becoming widely supported. The approach is scalable in a way that running it on a mainframe is not.

                --
                People today are educated enough to repeat what they are taught but not to question what they are taught.
                • (Score: 2) by TheRaven on Saturday April 14 2018, @08:29PM (7 children)

                  by TheRaven (270) on Saturday April 14 2018, @08:29PM (#667029) Journal

                  I believe that all you must do is map your problem into Map and Reduce operations. The infrastructure makes sure that everything gets processed correctly.

                  Yes, that's 'all' you must do. The fact that you can write 'all' in that way implies that you have never tried to do that with anything that is not embarrassingly parallel or which is in any way latency sensitive (typical Map-Reduce jobs run for tens of minutes or hours, most of the things that companies have high-reliability requirements for need responses in milliseconds).

                  --
                  sudo mod me up
                  • (Score: 2) by DannyB on Monday April 16 2018, @02:29PM (6 children)

                    by DannyB (5839) Subscriber Badge on Monday April 16 2018, @02:29PM (#667638) Journal

                    Map / Reduce is not the only way to use a cluster.

                    I would suggest that Twitter might be latency sensitive. In 2012, Twitter rewrote from Ruby into Java. They handle billions of tweets a day, each of which triggers various events and gets routed to multiple destinations.

                    I would point out that High Speed Trading is done in Java on Linux. Also Java and Linux are things that IBM pushes on their mainframes. But that is not what I think of as the mainframe era. As Java and Linux are very suitable for clusters of commodity hardware.

                    I'm not sure what type of application you are thinking of.

                    Airline reservation system or something similar for hotels, etc? Long ago that application was probably written for mainframes. It probably could be adapted, IF NEED BE, for clusters of commodity hardware. Maybe you're thinking an application like air traffic control systems. I don't know. But I doubt that the mainframe is the only way to do it fast, reliably or economically. And especially economically.

                    Maybe there is some application for which a mainframe is somehow the only uniquely suitable solution. I don't see it. But I'm open to being enlightened.

                    --
                    People today are educated enough to repeat what they are taught but not to question what they are taught.
                    • (Score: 2) by TheRaven on Tuesday April 17 2018, @12:34PM (5 children)

                      by TheRaven (270) on Tuesday April 17 2018, @12:34PM (#668013) Journal

                      Map / Reduce is not the only way to use a cluster.

                      Right. So now you need not only the expertise to use Map Reduce, but the expertise to use a number of different frameworks and to determine which is applicable. You think that's cheaper?

                      I would suggest that Twitter might be latency sensitive. In 2012, Twitter rewrote from Ruby into Java. They handle billions of tweets a day, each of which triggers various events and gets routed to multiple destinations.

                      Next time you talk to someone at Twitter, ask them how much engineering effort they've invested in dealing with tail latency spikes from garbage collection in their distributed Java systems. Hint: It's more than most companies want to spend on software development.

                      I would point out that High Speed Trading is done in Java on Linux.

                      No, it really isn't. Unless you have a very different definition of 'high speed' to any of the people I know in HST.

                      Also Java and Linux are things that IBM pushes on their mainframes.

                      Right, because it's cheap to write a Linux program and run it on a mainframe than to write a fault-tolerant distributed system.

                      --
                      sudo mod me up
                      • (Score: 2) by DannyB on Tuesday April 17 2018, @01:14PM (4 children)

                        by DannyB (5839) Subscriber Badge on Tuesday April 17 2018, @01:14PM (#668030) Journal

                        You've got me stumped. I can't imagine what kind of application you are getting at.

                        Surely the Googles, Amazons, Twitters, Facebooks and other major players must be missing out on something HUGE in mainframes. Those ignorant sops.

                        The development, followed by rapid growth of kubernetes, the videos, talks, articles, etc. There are a lot of people that are clearly missing out on something.

                        But you have not yet identified what.

                        These people all seem to think that on commodity hardware they get vendor neutrality, high reliability, immense scalability, and economy.

                        As for Java (or any language) with GC, yes, it has costs. But it has benefits larger than those costs. Sort of like hiring software developers is a cost. Or buying expensive servers.

                        Seriously. I am open to being enlightened what solution, including mainframe, is superior.

                        --
                        People today are educated enough to repeat what they are taught but not to question what they are taught.
                        • (Score: 2) by TheRaven on Wednesday April 18 2018, @07:44AM (3 children)

                          by TheRaven (270) on Wednesday April 18 2018, @07:44AM (#668483) Journal

                          You are completely missing the point. Google, Twitter, Facebook, and so on spend a huge amount on software development. They can afford to, because they spend even more on hardware. If Google can save 0.1% on their hardware costs, that completely pays for several full-time engineers, even on Silicon Valley salaries.

                          And, while Google does open source a lot of the core building blocks for their systems, that isn't the end. You can use those building blocks to assemble a reliable distributed system that will have the same uptime guarantees as a mainframe application, but it costs time for software engineers. It is a lot cheaper to build a non-distributed system that assumes that the underlying platform is reliable.

                          For most companies, hardware costs are small. The cost of taking their problem and mapping it to a reliable distributed system is vastly more than the total cost of hosting. Moving to hardware that costs 2-3 times as much is far cheaper than paying even one full-time software engineer. If you need a reliable transactional data store, you can install an off-the-shelf database on a mainframe and, as long as your data sets are no more than a few TBs, you get more reliability than most companies need and a trivial programming model that someone fresh out of university can work on. Alternatively, you can use one of Google's systems and hire developers that understand the CAP theorem and its practical implications. They can then write software that is vastly more complicated and needs at least twice as many people to maintain it. On the plus side, it will run on any commodity cluster and will scale to much larger data sets. On the down side, you've spent vastly more money on software engineering than you'd spend in 10 years having IBM host it on a mainframe for you.

                          The point is not that a mainframe can do things that commodity hardware can't do, it's that a mainframe can do the same thing with much cheaper software than a commodity cluster. If your software costs are a larger proportion of your total costs than hardware (which they are for a lot of companies) then mainframes are attractive.

                          From your reply, I see you didn't look at the problems that Twitter had with Java GC pauses. I'd suggest that you go and read about some of their problems before assuming that it's trivial for any company to solve them.

                          --
                          sudo mod me up
                          • (Score: 2) by DannyB on Wednesday April 18 2018, @04:18PM (2 children)

                            by DannyB (5839) Subscriber Badge on Wednesday April 18 2018, @04:18PM (#668625) Journal

                            I see now what you are arguing. I agree that a single-thread approach is always easier. The problem is that there is a limit to how much you can scale it. The first type of scaling out is to make it multi-threaded onto multiple cores that all share the same memory. The next step is to make it run on multiple cores that do NOT share the same memory model but communicate by some other means. Both of those approaches are increasingly common these days. So it's not exactly rocket science.

                            As for the Twitter - Java, I was just reading yesterday about the new Graal VM announcement. In particular Twitter has been testing this and is interested in it. I suspect that running on a mainframe is not an option for Twitter. They probably have no choice but to architect their application to be massively parallel on large clusters. If you're going to use any modern language today (Java, C#, Node.js, Python, Ruby, any JVM language such as Scala, Kotlin, Clojure, etc) then GC is simply a fact of life. Java's GC and JIT have had two decades of research by multiple interested parties. I would dare say that the JVM is the best industrial strength GC runtime platform at the moment. But Graal VM sure looks promising and runs on LLVM.

                            Twitter could write in C to run on a massively parallel cluster. The software development costs of that would be vastly higher than using a higher level language.

                            I strongly suspect that Twitter had investigated various options before settling on rewriting from Ruby to Java. Even in 2012, the warts in Java and its runtime are well known. They are probably just less of a problem than other approaches available to Twitter.

                            Or I could be wrong and Twitter (and others) just has/have no clue what they are doing on their massively parallel cluster.

                            --
                            People today are educated enough to repeat what they are taught but not to question what they are taught.
                            • (Score: 2) by TheRaven on Wednesday April 18 2018, @05:09PM (1 child)

                              by TheRaven (270) on Wednesday April 18 2018, @05:09PM (#668641) Journal

                              I see now what you are arguing. I agree that a single-thread approach is always easier. The problem is that there is a limit to how much you can scale it. The first type of scaling out is to make it multi-threaded onto multiple cores that all share the same memory. The next step is to make it run on multiple cores that do NOT share the same memory model but communicate by some other means. Both of those approaches are increasingly common these days. So it's not exactly rocket science.

                              Then the next step is to make it handle communications latency in the hundreds to thousands of milliseconds. Then the next step is to make it tolerant of a single process going away at random. Then you have something of the kind that Google builds. You're right that this isn't rocket science: it's much harder. I'm getting the impression from this thread that you've never programmed a distributed system, let alone a fault-tolerant distributed system and have absolutely no idea of the complexity involved.

                              As for the Twitter - Java, I was just reading yesterday about the new Graal VM announcement. In particular Twitter has been testing this and is interested in it.

                              Okay, so after three attempts you still haven't looked up what I told you to. Twitter runs Java in a distributed system. They need to keep average latency at a speed that prevents users getting annoyed, which means well under 100ms. Most end-user requests require going to around 100 different machines for their data. In such a system, the probability that one node will be in the middle of a GC pause is approximately one and the Gc pause time is greater than their acceptable tail latency. To give you an idea of how difficult this problem is to solve well, there have been several dozen paper proposing solutions at top conferences in the fields of programming languages, networking and OS design. And that's just one of the simpler problems that you need to solve when building a system like twitter (which is pretty tolerant of data loss: if you accidentally delete the occasional tweet, most people won't notice - it's not like money is involved).

                              I suspect that running on a mainframe is not an option for Twitter

                              I never said it was, but there are a lot of companies that are smaller than Twitter, for whom the cost of developing something like Twitter's network infrastructure is infeasibly expensive. Twitter employs over a thousand people to write their software, to solve what you keep claiming are easy problems.

                              Or I could be wrong and Twitter (and others) just has/have no clue what they are doing on their massively parallel cluster.

                              Twitter knows what they're doing. They're solving hard distributed systems problems by spending a lot of money on software engineers because, for a system of their size that doesn't have hard reliability guarantees, that's a lot cheaper than building or buying reliable hardware. Again, the fact that this is the right solution for Twitter tells you absolutely nothing about the right solution for most other companies. Your argument makes as much sense as telling FedEx customers that there's no point in shipping stuff in lorries because SpaceX can do it in a rocket.

                              --
                              sudo mod me up
                              • (Score: 2) by DannyB on Wednesday April 18 2018, @06:05PM

                                by DannyB (5839) Subscriber Badge on Wednesday April 18 2018, @06:05PM (#668665) Journal

                                I'm getting the impression from this thread that you've never programmed a distributed system, let alone a fault-tolerant distributed system and have absolutely no idea of the complexity involved.

                                If you qualify that the problems are not embarrassingly parallel, then you're right. I don't work on anything that requires interconnectedness between nodes or very low latencies. Even with embarrassingly parallel, I've only toyed with it. In a Map/Reduce situation I don't have to worry about reliability, if a failed function can be abstracted away.

                                after three attempts you still haven't looked up what I told you to.

                                Do you have a link you would like me to see? I would be interested because I do think of Twitter as a gigantic Java application. Maybe the very biggest.

                                I do not know the answer to this, but I would assume Twitter has looked at Azul's Zing and either found it unsuitable, not solving the problem, too expensive or some combination.

                                I think of Kubernetes as something that lets you deploy a Docker container to Google or Amazon. Or you can use Amazon's elastic beanstalk or Google's app engine, or other similar services. Your application is run on as many nodes as necessary. They don't talk to each other -- each is independendent. Each instance can talk to the same database -- but the database might be a replicated cluster, but you don't see that. The application must not maintain any in-server state between requests. Any state must be persisted somewhere (like the database) between requests, because another node may handle the next request. I do think of this technology as highly reliable. Maybe more reliable, and definitely more scalable than a central mainframe. Once we start talking about clustering mainframes, we're already moving in the direction that makes traditional mainfames unnecessary.

                                The problems you describe are not problems I've had to solve. Mostly I'm interested in how to scale out my employer's application much more than is likely to become necessary in the near term.

                                You keep talking about reliable hardware. But the hardware in use *is* very reliable. More reliability comes from the ability to fail over or, in some situations, re-executed the failed operation on another healthy node.

                                With that qualification, I'm not sure what you are countering. That traditional mainframes still have a future which I am not aware of? (Going back to the original.)

                                I'm talking about a technology that is currently big news, and being used by people who know what they are doing. Maybe it doesn't solve some problems you are interested in. It seems that the crux of your replies are to point out problems that it does not solve. Yet these are in widespread use for problems that they do solve.

                                Getting out of the weeds and going back to my original assertions, I don't see a large future for traditional mainframes, other than for running traditional mainframe workloads.

                                But, I'm always interested to know something I didn't.

                                --
                                People today are educated enough to repeat what they are taught but not to question what they are taught.
  • (Score: 0) by Anonymous Coward on Thursday April 12 2018, @09:00PM

    by Anonymous Coward on Thursday April 12 2018, @09:00PM (#666153)

    I hope I can buy the signature edition.

(1)