from the trying-to-remember-how-we-did-it-before dept.
Some cloud-based AI systems are returning to on-premises data centers:
Cloud computing revolutionized AI and machine learning, not because the hyperscalers invented it but because they made it affordable. Nevertheless, I and some others are seeing a shift in thinking about where to host AI/ML processing and AI/ML-coupled data. Using the public cloud providers was pretty much a no-brainer for the past few years. These days, the valuing of hosting AI/ML and the needed data on public cloud providers is being called into question. Why?
Cost of course. Many businesses have built game-changing AI/ML systems in the cloud, and when they get the cloud bills at the end of the month, they understand quickly that hosting AI/ML systems, including terabytes or petabytes of data, is pricey. Moreover, data egress and ingress costs (what you pay to send data from your cloud provider to your data center or another cloud provider) will run up that bill significantly.
[...] First, the cost of traditional compute and storage equipment has fallen a great deal in the past five years or so. If you've never used anything but cloud-based systems, let me explain. We used to go into rooms called data centers where we could physically touch our computing equipment—equipment that we had to purchase outright before we could use it. I'm only half kidding.
When it comes down to renting versus buying, many are finding that traditional approaches, including the burden of maintaining your own hardware and software, are actually much cheaper than the ever-increasing cloud bills.
[...] What's the bottom line here? Cloud computing will continue to grow. Traditional computing systems whose hardware we own and maintain, not as much. This trend won't slow down. However, some systems, especially AI/ML systems that use a large amount of data and processing and happen to be latency sensitive, won't be as cost-effective in the cloud. This could also be the case for some larger analytical applications such as data lakes and data lake houses.
Some could save half the yearly cost of hosting on a public cloud provider by repatriating the AI/ML system back on-premises. That business case is just too compelling to ignore, and many won't.
Cloud computing prices may lower to accommodate these workloads that are cost-prohibitive to run on public cloud providers. Indeed, many workloads may not be built there in the first place, which is what I suspect is happening now. It is no longer always a no-brainer to leverage cloud for AI/ML.
(Score: 4, Interesting) by Opportunist on Monday August 08 2022, @09:15AM (4 children)
Because cloud computing is a lot like a marriage: Getting in is easy, getting out is pricey.
They make it very convenient for you to move all your crap into their cloud. Where your crap does what crap does when there is "unlimited" space: It grows and festers. It branches out and encompasses this and that technology because it's just so wonderfully easy to incorporate it into your system.
Dismantling that Gordian knot to return the Frankensteinian server-and-service hodgepodge into an on-prem setup is usually so costly and error-prone that many companies would rather accept the stupefyingly high cost of the cloud service before even attempting to detach themselves from it.
In a way, cloud computing is like a drug. It starts as something that seems to solve all your problems but in the end, it becomes your biggest problem.
(Score: 4, Insightful) by darkfeline on Monday August 08 2022, @09:41AM (2 children)
Actually, there is an exit strategy, because the new on-prem workloads are not the same as the old on-prem workloads. Everything is containers now. The containers running on the cloud, they run the same way on your on-prem cloud. All of the technologies created to scale at the cloud level make managing compute workload easy, or at least standardized, and those same can be scaled down or sideways, run wherever.
This isn't your grandpa's system administration anymore Toto.
Join the SDF Public Access UNIX System today!
(Score: 2) by bradley13 on Monday August 08 2022, @09:48AM (1 child)
In theory, you're right. I'm not involved in large-scale AI, so I don't know - this is a genuine question: do you really use containers? You want bare-metal access, and virtualization of any sort has its costs.
Running on-premises has its own costs: You have to have the facilities, you have to maintain the facilities, you have to provide your own redundancy, your own backup systems, etc, etc.. Doing that right has costs, which are easy to underestimate. Those costs are all nicely hidden in those cloud-service bills.
Everyone is somebody else's weirdo.
(Score: 4, Interesting) by RamiK on Monday August 08 2022, @11:34AM
Container isolation isn't done by using hardware virtualization instructions. It's done by using the same kernel memory protections that isolate one user's process from accessing the memory of another user's process. So, there's no additional context switching between process and cpu compared to running without a container. Similarly, since the file-system is presented with a virtual name space regardless (e.g. /proc... /tmp...) so that level is also equivalent to native performance. The only exception is networking where re-routing and bridging can end up as a bottleneck for some very specific and heavy (numerous connections) network loads. But since those loads are always better served as a distributed solution anyhow, the common saying that "containers have native performance" is basically true.
What's throwing you off is that, on commercial clouds, containers are always run in a VM so the 20-25% overhead is still being paid.
compiling...
(Score: 5, Informative) by choose another one on Monday August 08 2022, @10:55AM
Not a bad analogy, but you stopped too soon. Getting out is not just expensive, it can involve lots of lawyers and the loss of most of your stuff.
Far too many people simply don't ever work out that outsourcing all their redundancy backup and business continuity to a cloud provider simply makes that provider a single point of failure.
Where is your business critical compute service and data? - at our cloud provider
Do you have redundancy, failover, georedundancy? - yes at our cloud provider
Do you have full backup strategy, offsite backup storage etc.? - yes, again our cloud provider does all that
Do you have a backup cloud provider? - huh, do we have a what?
Pick a "too big to fail" cloud provider and they will be too big to negotiate with (or you will be too small for them to care about) on price, licencing, etc. - you sign up for terms and conditions that can basically be changed at any time on their whim. and base your business case on that.
Pick a smaller cloud provider you can do deals with and get along term contract to back your business case... and you have the risk of that cloud provider failing itself.
That risk is not theoretical, I once worked for a company that had a cloud provision / datacentre / hosting side (I was not in that bit).
It went bust.
The fallout was, for a while (everyone gets their 15mins right?), legendary.
Administrators literally threatened to turn the lights off at datacentres unless they were paid loads of extra cash within hours.
Contract? SLA? - forget it, all the staff that provided that were let go without pay.
Customers lost access to not only service but data, some got it back but only after months of lawyering.
Some customers had servers (some in co-lo) that they had paid for, that the company had never paid the manufacturer for - more lawyers to figure out who even owns the hardware let alone the data
I could go on but suffice to say that "Pricey" and "messy", don't even begin to cover it.
Yep.
(Score: 2) by Revek on Monday August 08 2022, @11:48AM (1 child)
Old fashioned madness. Why own and control when you can rent.
This page was generated by a Swarm of Roaming Elephants
(Score: 4, Touché) by FatPhil on Monday August 08 2022, @12:15PM
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 5, Insightful) by datapharmer on Monday August 08 2022, @12:37PM
It's the circles of I.T. and we go back and forth about every 7-10 years and they feed off each-other. The winners are the component manufacturers.
Circle 1:
1. To insure data integrity store everything in a datacenter and run everything server side. Send result to client.
2. Determine clients have all these idle compute cycles and servers are expensive. Start offloading to clients.
3. Realize this can be done fully client side without servers at all - just distribute the load amongst all the nodes.
4. Become terrified that there is no control over any of this untrusted data and it is a garbled mess that is hard to manage. Move everything to a datacenter for backup/verification.
5. Now that everything is in a datacenter already and nice and organized why not run it server side?
Circle 2:
1. Technology is a huge capital cost. Offer a way to rent or use the resource without major upfront expense ("The IBM model")
2. These rental contracts are expensive. We'd be better off just buying the equipment outright and moving some of the data processing to the client to save costs.
3. Gimmick of the year ("cloud" or "AI" or "blockchain" "web3" or "quantum compute") can't be done with our existing technology and we don't want to pay huge upfront costs. Maybe we can rent someone else's equipment?
4. These rental contracts are expensive and the hardware is commoditized. Why not just buy the equipment outright?
5. Management of this equipment is expensive (technicians, cooling, huge rooms, support contracts, programmers, database engineers). Maybe we should outsource this under a contract to someone that specializes in it to lower this cost?
(Score: 2, Interesting) by krokodilerian on Monday August 08 2022, @03:29PM (4 children)
In short, pretty much anyone can do the math and see when it's cheaper to run your own hardware, rent it, or use whole cloud. There are some variables (do you have the people to handle infrastructure, for example), but for a lot of companies (in my experience) anything larger than 2 racks full with equipment is cheaper to run by yourself. At some sizes it becomes cheaper even to build your own datacenter.
And running your own infrastructure is not something extremely hard or arcane to learn and do, in some cases it's a lot less complex than the k8s crap I've seen people put together.
(Score: 3, Interesting) by SomeRandomGeek on Monday August 08 2022, @04:58PM (3 children)
Yup. Cloud is pretty much always cheaper. If you do the math and decided that it is cheaper to own it yourself, it is because you forgot to include something important in your calculations:
Did you account for the cost of the dedicated space with the specialized cooling and the specialized power?
Did you account for the cost of making everything reliable and redundant when you were doing the power and cooling calculation?
Did you account for the cost of the operations team that keeps the hardware running?
Did you account for the cost of money for all the equipment that you paid for upfront, but the benefit of which will only be realized over the course of years?
Did you account for the probability that the whole damned project might get canceled and you will get stuck owning a bunch of computers on which you are seeing no return at all?
Did you account for the need to switch to a different class of hardware a few years into the project, and the assets that get stranded that way?
Did you account for the lost revenue during the time that you are building out a datacenter before you can actually use it?
Did you account for the cost of building out hardware for the worst case because you were unable to predict your usage in advance?
Did you handwave any of that stuff I just mentioned away by saying: Oh, we already have all of that stuff in-house, and therefore I think it is free?
(Score: 5, Touché) by gawdonblue on Monday August 08 2022, @09:57PM
Just a few of the cloud issues have had to deal with at my place of work that spring to mind:
Did you account for the credit card expiring that's paying for that inter-business message queueing service and twelve businesses now wondering why they cannot talk to each other?
Did you account for the change in the "fixed" IP address that will occur in your business partner's cloud based server when their provider "upgrades" them?
Did you account for the growth in your data volume such that the costs go from $10k/month to $100k/month, because 1.5 times the data should be 10x the price, right?
Did you account for the fact that setting up a testing environment for these services is also expensive so discourages you from creating one to test out forced "upgrades" - even if your provider will bother to provide UAT facilities on the service in the first place?
Did you account for the fact that the provider will "upgrade" your services at their whim and break your application for over a month before they finally acknowledge they made a mistake before rolling it back?
Did you account for the fact that those forced upgrades now mean that if you can get a region to UAT the upgrades you have to devote serious resources to testing each one because you're not sure what may be changing and how it will break you?
Did you account for the fact that the provider has their own maintenance outage schedules that are not the same as your own so your 24x7 business is now scheduling around external factors, not you and your business partners?
These are just a few of the painful things from the increasingly cloudy place where I work. I am sure that there are plenty of others that I do not hear about, because the cloud-worshippers are rather embarrassed by the issues that arise from the move to other peoples servers that was supposed to make IT cheaper and easier (hint: it doesn't).
(Score: 2) by sjames on Tuesday August 09 2022, @03:21AM
Only in specific circumstances. It really depends on scale and workload.
Small scale could really be done with a couple servers and a nice UPS in the office. It's crazy the amount of money small operations blow on cloudy solutions.
At a larger scale, a rack or two in a local small data center will often be cheaper, possibly using the cloud to handle peak loads and/or DR.
It's not like you don't need skilled people to develop and maintain a cloud presence. Many of those people are already competent to handle hardware.
At huge scales, the cloud bill is so big that you might as well be making loan payments on a data center and hardware.
Balanced against your points, you forgot that nobody cares as much about your downtime as you do. If you own it, you won't be hit with an altered deal at the last minute, including constant churn on service APIs. Certainty has a value.
(Score: 1) by krokodilerian on Tuesday August 09 2022, @02:47PM
> Did you account for the cost of the dedicated space with the specialized cooling and the specialized power?
As noted, a whole DC is for larger projects, otherwise, single rack with power and cooling in a dedicated facility is rarely more than $1k/month, and all that's in there.
> Did you account for the lost revenue during the time that you are building out a datacenter before you can actually use it?
You're not building a datacenter, see first point.
> Did you account for the cost of making everything reliable and redundant when you were doing the power and cooling calculation?
That's pretty much N+K for the hardware you're using, depending on design (2 routers, 2 switches, enough RAM/CPU capacity to lose 1 or 2 compute nodes, etc).
> Did you account for the cost of the operations team that keeps the hardware running?
If you have large enough case to warrant 2 racks of hardware, you already have an operations team that should be able to handle this. As noted, this is a lot less complicated than a lot of the modern "infrastructure as code" crap, and deploying even bloated crap like openstack to manage all the hardware is not really rocket science. Sending remote hands to replace drives & failed nodes comes up to 4h/month, tops, except when expanding (in which case it's still irrelevant price-wise compared to anything else).
> Did you account for the cost of money for all the equipment that you paid for upfront, but the benefit of which will only be realized over the course of years?
> Did you account for the probability that the whole damned project might get canceled and you will get stuck owning a bunch of computers on which you are seeing no return at all?
> Did you account for the need to switch to a different class of hardware a few years into the project, and the assets that get stranded that way?
> Did you account for the cost of building out hardware for the worst case because you were unable to predict your usage in advance?
That's called business planning and calculating TCO. If you're not a startup and have a good idea what you're doing, these are pretty much trivial (and we do this for some of our customers regularly, to plan their own private cloud deployments).
I've done this on different sizes, like for 7PiB of data in 2009, and comparing the cost then of the local deployment to the amount this would cost even in backblaze, that one was cheaper per even per year, you didn't even need 5 years to recoup the hardware.
So yes, if you have no idea what could happen and if you'll be gone/dead tomorrow, then yes. But then, you might rethink if you want to actually do that at all :))
(Score: 3, Interesting) by bzipitidoo on Monday August 08 2022, @11:33PM
Right now, almost all the Internet is in private hands. Users are at the mercy of private ISPs. One of the things that makes running your own data center harder is an unaccountable ISP controlling your connection. At any moment, the ISP can cut you off. And they do sometimes. One piracy accusation, without any proof or due process, can be enough of an excuse. All the time, we hear of unfair takedowns of content on YouTube. Or they just blandly change a policy, for our protection you understand, and block a few ports such as ports 80 and 443, and play stupid, disclaiming all responsibility for your web server suddenly being inaccessible. Yes, of course there's a huge problem with bots constantly searching for vulnerabilities. And you'd better have spam filtering in place before you dare connect to the Internet, if you're going to run your own email server. But you ought to have that option.
There was that proposal about a year ago in which some private equity firm offered to buy the .org domain. They couldn't come up with any good reason why they would do a better job, and it was pretty clear their real goal was rent seeking.
The whole point of the USPS was to make sure no private organization could monopolize communication. They didn't do a damned thing to compete with Ma Bell during the age of the land line. It finally took anti-trust legal action to break Ma Bell up, in 1984. And now, over the past 25 years, it's been happening again. Rather than create and offer Internet service to all, the Post Office seems bent on reducing themselves to a mere parcel service. Even that is extremely unfair. Ordinary folks pay 5x as much to ship a physical item, compared to a heavyweight like Amazon. We really need more public Internet options. The lack there is one of the things that drives businesses into the cloud.
(Score: 2) by Lester on Tuesday August 09 2022, @05:33PM
In the cloud, you hire someone to take care of keeping the software in its last version, to take care of keeping the hardware humming, to manage backups and restores, It is accessible from anywhere not with a domestic router, and take care of firewalls and security.
Is it expensive? Well, nobody knows.
The first hint is that not everything needs to be accessible from anywhere and needs a great bandwidth. The second point is that big projects like the mentioned have budget that allows them to hire technicians, and even the project staff has a good tech level. The third is that such projects use a lot software customized and/or fine tunned by the staff that can't be done by the standard procedures of a cloud business.
I'm sure that cloud is overused and abused, it is a bubble that will burst.