SoylentNews is people
SoylentNews
Someone purchased the eight year old Cheyenne supercomputer for $480k. Failing hardware. Leaking water system. What would it be good for? Selling for parts would flood the market. Testing the parts would take forever. They also have to pay for transport from it's current location. Originally built by SGI.
https://gsaauctions.gov/auctions/preview/282996
https://www.popsci.com/technology/for-sale-government-supercomputer-heavily-used/
https://www.tomshardware.com/tech-industry/supercomputers/multi-million-dollar-cheyenne-supercomputer-auction-ends-with-480085-bid
Cheyenne Supercomputer - Water Cooling System
Components of the Cheyenne Supercomputer
Installed Configuration: SGI ICE™ XA.
E-Cells: 14 units weighing 1500 lbs. each.
E-Racks: 28 units, all water-cooled
Nodes: 4,032 dual socket units configured as quad-node blades
Processors: 8,064 units of E5-2697v4 (18-core, 2.3 GHz base frequency, Turbo up to 3.6GHz, 145W TDP)
Total Cores: 145,152
Memory: DDR4-2400 ECC single-rank, 64 GB per node, with 3 High Memory E-Cells having 128GB per node, totaling 313,344 GB
Topology: EDR Enhanced Hypercube
IB Switches: 224 units
Moving this system necessitates the engagement of a professional moving company. Please note the four (4) attached documents detailing the facility requirements and specifications will be provided. Due to their considerable weight, the racks require experienced movers equipped with proper Professional Protection Equipment (PPE) to ensure safe handling. The purchaser assumes responsibility for transferring the racks from the facility onto trucks using their equipment.
Please note that fiber optic and CAT5/6 cabling are excluded from the resale package.
The internal DAC cables within each cell, although removed, will be meticulously labeled, and packaged in boxes, facilitating potential future reinstallation.
Any ideas (serious or otherwise) of suitable uses for this hardware?
(Score: 5, Interesting) by VLM on Saturday May 11 2024, @03:31PM (4 children)
Most of what they bought is not obsolete or failing.
Sure the CPUs are not cutting edge, go buy some new MB/CPU and replace those leaky water cooling blocks while you're at it and enjoy your partially new partially old supercomputer.
Some tech advances pretty slowly. Lets say they throw literally everything into the dumpster and pay to have it hauled away but keep the 224 Infiniband switches. I'm guessing eight thousand odd CPUs and 224 switches means these are the 48 port models (must be at least 36 ports but probably more with 4-d interconnections and trunk and management lines). Nvidia will sell you a nice new MQM8700 for a mere $30K each. So if you were planning on building a supercomputer of your own and buying "a couple hundred Infiniband switches" you could buy 224 new MQM8700s for a mere $6.7M or buy 224 slightly used switches from this place for a mere $480K. So all you have to do is spend less than $6.2M on removal, shipping, and disposal of everything that's not an Infiniband switch. Consider that even scrap value of 8K Infiniband cables is worth something. I just checked ConnectZone and a 15 meter Infiniband cable will set you back $400. So 8K of those cables would cost somewhat in excess of $3M even with some bulk discounts. Sure the labor cost of safe reusable removal is not $0 but the labor is still probably less than $3M minus the $480K used price, just for the cables alone.
I think it would be hard not to profit if you're in the business of running a supercomputer, or can middleman with peeps who do. I mean, it gets to the point where this is so cheap you can buy it for spare parts.
Some years ago I looked into IB "because I could" and its barely within reach if I bought used gear but its still a bit too rich for my blood so I installed 10G ethernet and ran vSAN and later CEPH on it. Works pretty well. Figure you're gonna pay $100/port for each end of a 10G ethernet and more like $1000 for each end of an Infiniband (not too accurate, but close enough). A new 16 port netgear XS716T used to set you back $1K and some years later I could replace that with a new MS510TXM with only 10 ports for about $600, so prices are slowly dropping. Trendnet today sells a new TEG-7124 which is "pretty much" my several years old XS716T for almost exactly half the price I paid for my XS716T.
Ironically I'm old enough to remember when ethernet was new you'd spend like $500 for each end of 10M and later 100M FE. So prices have gone down both absolute and relative to inflation WRT networking gear.
(Score: 4, Insightful) by sgleysti on Saturday May 11 2024, @04:31PM (1 child)
I briefly contemplated making a small cluster. What seemed to give the best FLOPs / $ was the highest end consumer grade Ryzen CPUs. For the interconnect, I figured I could do a 5-node star topology with 4-port 10G ethernet cards and give each node a direct connection to all the others. It didn't look too expensive because I wouldn't have needed a switch and could have used direct attach cables. I would have used the 1G motherboard ethernet and a switch for management.
In the end, I figured that would have been a lot of money, time, and effort for something I didn't really need. Any modern processor these days is incredibly fast if you use a language that compiles to machine code and takes advantage of the vector units in the CPU.
(Score: 3, Interesting) by VLM on Saturday May 11 2024, @05:07PM
Yeah I wanted the experience of messing with very interesting infrastructure software and I want reliable "production" storage (using proxmox/CEPH right now), I was not motivated by FLOPS. Some are, which is fine. It's a big hobby, plenty of space.
The economics is constantly changing. The 10G copper switch market right now seems to run "meh" $70/port (fiber usually more), and PCI cards run $50 to $100 for single ports and $200 to $300 for multiports like your 4-port.
So for five nodes your 4-port mesh would cost about $1250 just in cards alone whereas using 1 port cards and a switch would cost 5 times $50 plus "about" $500 for a very small switch so figure maybe $750. Cable cost can add up, 5 cables vs 20-ish cables.
Strictly economically you'd do better with the switch design in 2024, although the economics have varied widely in the past.
I'm using dual 10G copper ports as LAGs to each node and there are load balancing issues but I "can" run up to 20G in theory.
Of course if you want the fun of running FRR or whatever routing platform on your partial mesh then you kind of have to run multi-port if the point is having fun with routing protocols. Although with a full mesh I guess you could just use static routing? I'd want to set up OSPF just for the LOLs "because I can" but not everyone enjoys routing as a hobby...
The main rule of home lab is have fun, so here I am looking at ebay search results and daydreaming while writing this. So today on ebay I can get all the used Infiniband cards I want for about $50/each and probably at least half of them actually work, and switches that might or might not work run around $100 to $200 so for somewhat under a kilobuck, between the cost of your 10G multiport cards and my 10G switch design, I could play with Infiniband at home; how much is fun and bragging rights worth? I guess I could start small with one switch and two cards and see what happens...
Another fun feature of InfiniBand IIRC is there's a wide range of semi-compatible port speeds where the slowest is 10GB and the highest is unaffordable but something like 400GB, and secondly tuning Linux to use all the bandwidth under TCP/IP is an art form which makes it "fun". At least I think it would be fun. Very few people can push 400GB/s in their basement, maybe I will be one of the first. Note that consumer SSDs crap out around "sixteen GB/s" or so because that's as fast as PCIe4 can run, this means I will have to do weird things with RAID arrays.
I've always wanted to have Infiniband in my basement so this might end up my summer project; we'll see. I'll probably do it "someday" anyway.
So many other things I could do to fill my infinite spare time. Such as run 10G fiber from my basement "datacenter" to my office and put 10G in my office so I would have very fast access to my storage...
(Score: 3, Interesting) by captain normal on Saturday May 11 2024, @07:37PM (1 child)
That's assuming someone is only after the hardware. Removing data from any type of drive is hard. The location of this from the GSA Auction site is the NCAR-WY SuperComputing Center in Cheyenne, Wy. I could see some interested parties looking to mine data from a large government supercomputer looking to buy and drag this back home to scan for data.
https://en.wikipedia.org/wiki/NCAR-Wyoming_Supercomputing_Center [wikipedia.org]
The Musk/Trump interview appears to have been hacked, but not a DDOS hack...more like A Distributed Denial of Reality.
(Score: 3, Informative) by looorg on Saturday May 11 2024, @10:35PM
Except as far as I can tell no storage was included or put up for auction. None. Beyond RAM but that should be blank by now. So as far as I can tell zero chance of an actual data leakage here.