Hacker tries to poison water supply of Florida city
A computer hacker gained access to the water system of a city in Florida and tried to pump in a "dangerous" amount of a chemical, officials say.
The hacker briefly increased the amount of sodium hydroxide (lye) in Oldsmar's water treatment system, but a worker spotted it and reversed the action. Lye is used in small amounts to control acidity but a large amount could have caused major problems in the water.
Oldsmar Mayor Eric Seidel said: "There's a bad actor out there." No arrests have yet been made and it is not known if the hack was done from within the US or outside.
A computer controlling Oldsmar's water treatment system was remotely accessed on Friday. A plant operator saw an attempt to access the system in the morning but assumed it was his supervisor, the Tampa Bay Times reported. But another attempt was made early in the afternoon and this time the hacker accessed the treatment software and increased the sodium hydroxide content from 100 parts per million to 11,100 ppm. The operator immediately reduced the level to normal.
Also at CNN, Ars Technica, and WWSB.
Related Stories
Breached water plant employees used the same TeamViewer password and no firewall:
The Florida water treatment facility whose computer system experienced a potentially hazardous computer breach last week used an unsupported version of Windows with no firewall and shared the same TeamViewer password among its employees, government officials have reported.
After gaining remote access [...] the unknown intruder increased the amount of sodium hydroxide—a caustic chemical better known as lye—by a factor of 100. The tampering could have caused severe sickness or death had it not been for safeguards the city has in place.
According to an advisory from the state of Massachusetts, employees with the Oldsmar facility used a computer running Windows 7 to remotely access plant controls known as a SCADA—short for “supervisory control and data acquisition”—system. What’s more, the computer had no firewall installed and used a password that was shared among employees for remotely logging in to city systems with the TeamViewer application.
Massachusetts officials wrote:
The unidentified actors accessed the water treatment plant’s SCADA controls via remote access software, TeamViewer, which was installed on one of several computers the water treatment plant personnel used to conduct system status checks and to respond to alarms or any other issues that arose during the water treatment process. All computers used by water plant personnel were connected to the SCADA system and used the 32-bit version of the Windows 7 operating system. Further, all computers shared the same password for remote access and appeared to be connected directly to the Internet without any type of firewall protection installed.
[....] The revelations illustrate the lack of security rigor found inside many critical infrastructure environments.
It was a 32-bit computer; so they wisely had Windows 7 instead of XP.
See also:
recent SoylentNews article about this, attempt to poison the water supply of residents in Oldsmar, Forida.
(Score: 3, Interesting) by c0lo on Tuesday February 09 2021, @11:43AM (20 children)
Why? Why is it connected to the internet?
Not like a water plant is less essential than a bar/restaurant to have the supervisor WFH.
https://www.youtube.com/watch?v=aoFiw2jMy-0 https://soylentnews.org/~MichaelDavidCrawford
(Score: 1, Interesting) by Anonymous Coward on Tuesday February 09 2021, @12:10PM (1 child)
Ya know it's a good question. There is two, maybe more; but, I'll focus on two possibilities.
One, is they do stuff like this for the same reason some one burns down a house to get the insurance money. It's intentional.
The second, is that the AI singularity is already here, and has been here; and is slowly but surely, chipping away at our minds, influencing us in unseen ways. Just waiting for the moment when a walking, talking, seeing, jumping, bonified robot gets connected to the, 'internet.'
(Score: 0, Troll) by Ethanol-fueled on Tuesday February 09 2021, @12:17PM
Whoever did it obviously wanted a lot of people to die, so that they could add more deaths to their COVID body-count and blame a "COVID outbreak" on Super Bowl partying. Globohomo are also attempting to make an example of Florida, because it's the most high-profile state pushing back against the COVID hoax. Also it was the Jews.
(Score: 2) by Runaway1956 on Tuesday February 09 2021, @12:55PM (7 children)
I have the same question, and go one step further. After a warning like this, why is remote access disabled TEMPORARILY???
Now way, now how, no where, no when, should critical infrastructure be accessibly via the internet. Fekkin' idiots!
“I have become friends with many school shooters” - Tampon Tim Walz
(Score: 0) by Anonymous Coward on Tuesday February 09 2021, @02:28PM
I'm suspicious whether this story is actually real. If more lye ended in the water, did somebody spill a bucket by accident and they want to shift blame?
Secondly, if safety critical systems are being controlled by computers, why not add another system to sanity check what the first one is doing?
(Score: 2) by Immerman on Tuesday February 09 2021, @03:00PM (1 child)
Really? There's a bad actor somewhere on Earth, and they have internet access? Oh no! Sound the alarm! Unleash the hounds!
Anyone who says such a thing as though it has more relevance than "the sky is blue" should not be allowed in any position of authority.
Connecting any system to the internet is immediately exposing it to a huge number of bad actors. It's your responsibility to defend against them, and if you don't plan to take that responsibility seriously, unplug from the effing 'net.
(Score: 0) by Anonymous Coward on Tuesday February 09 2021, @07:47PM
Seems you are having trouble processing context. Did you mean
[ Help ] [ OK ] [ Cancel ]
- -
*Wait, don't you mean dice collection?
No, moron. The singular is die. Do you say "horses collection"?
(Score: 5, Interesting) by DannyB on Tuesday February 09 2021, @03:24PM (3 children)
As someone at NASA once said in January 1986: take off your engineering hat for a moment and put on your management hat.
Do you realize that it would cost money to have an actual person at an inconvenient location to actually physically monitor and control operations of water treatment? Even if someone only occasionally needs to physically visit the facility. (eg, remote monitoring, but actual hands-on manipulation must be, um, hands-on.) Ditto for electrical substations, chemical plants, sewage treatment, electrical generation facilities, nuclear power plants, etc. No on site humans needed! Just a warm body on the intarwebs.
Don't you realize that it is way cheaper (to someone, but not to you and me) to simply have a warm body somewhere who can remotely monitor and control these facilities? (Located in India or somewhere.) Look how effectively this has worked for corporations to outsource their call centers! It's wonderful, just wonderful I tell you!
No regulations needed. They are doing swell! We can't regulate corporations. Absolutely no regulation can be permitted. That would not be pro-business. If this principle works for corporations, it works equally well for municipal water.
Poverty exists not because we cannot feed the poor, but because we cannot satisfy the rich.
(Score: 3, Interesting) by VLM on Tuesday February 09 2021, @04:29PM
I've worked at places running under a variation of the two man rule where they have an onsite and an offsite.
Note that the workload can get pretty shitty for the onsite when there's either an internet outage at the offsite's house or the onsite plant, but at least they have someone. Luckily if you use VOIP then an internet outage means no incoming calls.
The highest ratio I ever saw was during a snow storm at a data center type facility and they had like one dude who lived down the road come in for a 16 hour day and five guys VPN'd in from home. The onsite guy was pretty busy doing remote hands stuff. The fact of the matter is if you have 1000 water and enviro sensors in a very large facility (like acres...) even if the sensors are 99.9% reliable every day, that means at least once a day the flooding sensor will go off and need investigating, and air handling equipment needs continual maintenance, its a headache.
(Score: 1) by khallow on Tuesday February 09 2021, @06:37PM
(Score: 2, Interesting) by Anonymous Coward on Tuesday February 09 2021, @07:15PM
Ok, so you *really* need to always be able to remotely see what's going on. So point a webcam at the screen through an AIR GAP.
(Score: 1, Insightful) by Anonymous Coward on Tuesday February 09 2021, @01:20PM
Thinking out loud, why not have the operator status display(s) connected to the internet, but separate out the operator controls--and have them at the plant only?
(Score: 2, Funny) by Anonymous Coward on Tuesday February 09 2021, @01:45PM (5 children)
Why would the software allow the sodium hydroxide to be raised to such levels in the first place (and not have alarms going off all the places)?
I had a prefessor once who ordered some radioactive isotopes... later the radioactivity safety officer paid him a visit with the (jokingly) question why he wanted to build a nuclear power plant. The professor made a mistake when entering the amount.
(Score: 0) by Anonymous Coward on Tuesday February 09 2021, @03:41PM (3 children)
the software, ultimately, controls the pumps that actually pump the liquid.
The checks on the pump state are also made out of software.
Depending on scenario - was the actual piece of hardware controlling pumps, or a server running the controlling software or a workstation connecting to that server compromised?
Irregardless, once you got 1 of these, you can easily have the other two.
Since checks can be in either of 3 places, in the worst case (science fiction) you disable checks and make pump do whatever.
In reality, you connect to the thing controlling pumps directly and do whatever.
But, when its accessible from outside... where's the sport in that.
(Score: 3, Insightful) by DannyB on Tuesday February 09 2021, @03:52PM (2 children)
Idea: a microcontroller whose firmware cannot be remotely overwritten controls actual devices. Said microcontroller has sane limits on values that can be remotely entered. To exceed those values requires a person to physically visit the location to enter values that would exceed pre established limits for remote users. That way, if there is truly some reason to exceed those values, it is possible but less convenient.
Poverty exists not because we cannot feed the poor, but because we cannot satisfy the rich.
(Score: 4, Insightful) by Immerman on Tuesday February 09 2021, @04:23PM (1 child)
In general a good idea, I'd go so far as to have physical stops so no amount of software compromise could override them. However, I'm not sure that either would actually help in this situation. Water treatment is usually(?) done in batches - which probably means the lye dispenser turns on for a few seconds to dispense lye into the batch, then turns off again until the next batch. Even if you restricted how much lye could be dispensed in one go, the controller would have no way of knowing when the batches change, so you could just dispense a bunch of batches worth.
The obvious solution is NEVER attach your control systems to the internet. You want to operate things from a remote part of the plant, that's fine, connect it to your local secure, wired intranet. But do NOT connect that intranet to the broader internet. Nobody needs internet access from their control console - if they need access, they can get it on their computer sitting next to it, connected to the internet-facing network.
And be sure to connect a device to your secure intranet that constantly tries to reach the internet, and sets off a loud flashy alarm if it succeeds because some idiot bridged networks and opened the door for world+dog to access the nominally secure network. It won't catch compromisable machines connected to both networks, but at least it will quickly catch the really bone-headed "throw the door wide open" problems.
(Score: 2) by multistrand on Wednesday February 10 2021, @03:55AM
yes -- physical stops for something that if set wrong can harm people. Hacking isn't even the most important reason to have physical stops. Human error without malicious intent is far more common.
(Score: 2) by VLM on Tuesday February 09 2021, @04:19PM
From what I read online the pH alarms for the plant were expected to go off, but the onsite tech saw it configured to a high value, said "WTF" and shut it down before the pH could have appreciably changed. If he was taking a bathroom break the alarms would have eventually gone off.
As for why, if engineering specs a 10 liter/min pump, maintenance gets a ticket to service the seals or fix a leak or whatever, maintenance will not close out their ticket until at least momentarily they saw it run at 10 liters/min as per engineering spec.
Or a step back from that, plants full of liquid are always going to have trouble replacing pipes and having to blow the air out of them and so forth.
Finally, for something like a water plant, everyone in the city taking a shower simultaneously at 7am is likely a lot more than the water demand at 2am when everyone's asleep. Of course thats why they have water towers and tanks but I suppose if you own the capacity to input 100K gallons/minute even if you never use it you likely for legal reasons need the ability to treat water at that input rate. You can't run a water utility and pump in untreated water else the "hacker attack" would simply be turn the inlet pumps on max to force untreated water in.
(Score: 2) by fakefuck39 on Tuesday February 09 2021, @11:04PM (2 children)
>Why is it connected to the internet?
Because it's not. But the computers and laptops on the internal network are connected to the internet. And one of those has a backdoor, acting like a bridge to get to the control system from the internet. Which is literally how pretty much every attack works.
(Score: 0) by Anonymous Coward on Wednesday February 10 2021, @12:11AM (1 child)
Then it IS connected to the internet, and you are a brainless cretin for trying your brainless argument here of all places.
(Score: 1, Flamebait) by fakefuck39 on Wednesday February 10 2021, @01:04AM
Right right. A system that is blocked by the firewall to get to the internet is connected to the internet because you can plug a device into it that's connected to the internet. You are currently sitting in the middle of a grocery store, because you can get in your car and drive there.
Terms have meaning. Internet-connected is a technical term that has meaning. Air-gapped has meaning. You don't know what either of those terms mean, and should learn to use a computer before commenting on a tech forum that people are wrong.
(Score: 0) by Anonymous Coward on Tuesday February 09 2021, @01:32PM (2 children)
From https://en.wikipedia.org/wiki/Oldsmar,_Florida [wikipedia.org] has an interesting history, starting with a car analogy!
Yummm, sodium hydroxide or mosquito larvae, not sure which I'd choose...
(Score: 0) by Anonymous Coward on Tuesday February 09 2021, @02:12PM
Florida was basically founded on real estate scams by Northerners to Northerners. Buy your piece of paradise!
(Score: 1) by nitehawk214 on Tuesday February 09 2021, @02:52PM
The original Florida Man.
"Don't you ever miss the days when you used to be nostalgic?" -Loiosh
(Score: -1, Flamebait) by Anonymous Coward on Tuesday February 09 2021, @02:09PM (2 children)
seems like a non-story, yet the media, including here is pumping so much type and terrifying terror. ::yawn::
(Score: 0) by Anonymous Coward on Tuesday February 09 2021, @06:08PM (1 child)
You lye!
(Score: 0) by Anonymous Coward on Wednesday February 10 2021, @06:24AM
Nah -- just making lyemonade
(Score: 2) by looorg on Tuesday February 09 2021, @02:15PM (7 children)
Floridaman got a job and wanted more salt in his water? I guess the manager, and/or some workers, have to be able to distance work in these harsh covid times? That said I don't know why one would have to be able to raise the sodium hydroxide levels -- why would you need to do that? Just wondering. I don't really know much about chemistry or the wonders of water supply systems.
(Score: 3, Informative) by deimtee on Tuesday February 09 2021, @03:57PM (1 child)
Most obvious reason is that it is a very easy and harmless way to raise the pH if your water is too acidic.
If you cough while drinking cheap red wine it really cleans out your sinuses.
(Score: 3, Touché) by JoeMerchant on Tuesday February 09 2021, @06:15PM
You forgot the most important point: cheap, as in economically efficient.
🌻🌻 [google.com]
(Score: 3, Informative) by VLM on Tuesday February 09 2021, @04:04PM (4 children)
I used to work in a city where sometimes they sucked up ground water from a rather enormous lake surface reservoir and sometimes they sucked up well water depending on demand and climate and legal stuff, and they have different pH and have to be messed with continually.
As for why pH matters, WRT chlorination of water to disinfect it is this balancing job where the more acidic the water the better the chlorination works (ionized chlorine atoms) and the more basic the weaker. So the dude essentially shut off the effect of chlorination without shutting off the flow of chlorine gas. This is like basic (oh the pun) swimming pool chemistry. You can add all the chlorine you want to a basic pool and it won't disinfect, or at least not as well as an acidic pool. But you can't run acidic all the time because people and pipes don't like it. Its a relatively sensitive balance job.
Also as per Flint MI once the bucket of NaOH runs out from being used at 1000x rate, which might be pretty quick if they usually fill it once per shift or day or week, the water will presumably turn naturally acidic which is why they were adding NaOH to begin with, which is bad for the pipes especially legacy lead pipes and legacy lead solder and leaded brass.
I will say the story is amazing as "human entered value changed mysteriously from 100 to 11100" sounds like some dumbass at 2am clicked the key three times instead of one as fell asleep. Which explains why nobody has been caught and no fingers pointed. You can just picture some dude at 2am experiencing network latency hits the 1 key "fukit why wont this MFer work like its supposed to" hits the 1 key again, hits the 1 key again, finally it works. Oh wait it was actually 11100 instead of 100. Then later its all "uh musta been hackers I wanna talk to my union rep first". Really, 11100 as reported on other sites?
Then it turns into a meta-shitshow because the guy who programmed the PLC HMI F'd up by allowing some crazy value. And the engineer who spec'd that as not mattering F'd up. And the sysadmin who didn't install logging F'd up. And the manager who supposedly oversaw training F'd up. And the guy who programmed the SCADA F-d up by not setting off all the alarms.
And most safety critical places I worked had a two man rule and technically it worked because the other guy shut it down but they're having a meltdown in public; why is this public? My guess is mgmt union dispute and they wanted to fire the guy with the high latency inet connection because if they fire a fall guy the internet will never have latency again so its a permanent solution, but he's blaming the site's connection for being shitty and management for being shitty so its suddenly on the front page as "hackers" this and "hackers" that.
Sometimes follow the money works, sometimes follow the publicity works. I bet the "real" story we're not being told is amazing. As is the norm for "journalism" now a days.
(Score: 3, Interesting) by JoeMerchant on Tuesday February 09 2021, @05:57PM (3 children)
Um, dude, you are seriously over-estimating the engineering that goes into a water treatment plant control interface.
First off, they're just glad that they no longer have to shovel lime out of bags into the mixing tank by hand.
Next, the automated injector they bought is off-the-shelf standard, and it has a standard range of capability for all kinds of applications.
Finally, the guy who "programmed" the PLC HMI took the injectors' standard interface software and put a quick-launch icon on the control computer's desktop - maybe if you're lucky someone changed the injector name from "DEFAULT INJ 2" to "Lime Rate", but that's already low odds.
I'm guessing that remote control is probably TeamViewer, or more likely some 3rd party thing from India one of the guys found in Dogpile [dogpile.com] and installed at work and home.
🌻🌻 [google.com]
(Score: 2) by krishnoid on Tuesday February 09 2021, @07:35PM (2 children)
As good a time to ask as any -- are there any good intro references to SCADA? Based on my authoritative nonexistent experience with it, it seems like it would be useful to know since it's widely-used (?)
(Score: 2) by JoeMerchant on Tuesday February 09 2021, @11:47PM
I know f-all about practical stuff like that.
When my first job imploded (12 years in), I sat in a park with my toddler and chatted with a guy that (still) worked for Johnson Controls. I did C++, 3D CAD, all kinds of fancy shit, big title, and no job. He just did PLC relay controls for HVAC and similar - main differences: 1) he was still employed, 2) if he did get laid off there were dozens of places he could work in any major city. Me, I ended up looking for work for 4 months before having to move 1000 miles to the first available job. Sure, it paid 2x what mister Johnson Controls makes, but... is it really better?
🌻🌻 [google.com]
(Score: 2) by VLM on Thursday February 11 2021, @06:42PM
You're probably asking the wrong question in context and want to google for something like CODESYS which is free-ish and has a nice port to turn a raspi into a PLC.
A PLC is the thing that runs actual stuff. The control loop might be 1 KHz a thousand times a second it runs thru your programed ruleset. CODESYS provides many different ways to write and edit those rules and upload to hardware. The PLC is literally hard wired to the dispenser from the linked story.
HMI stuff is how users control a PLC aside from physical pushbuttons. CODESYS has pretty good stuff for that, nice web pages to control stuff.
SCADA is like the product Observium but instead of monitoring switch port counts, its monitoring like pH of the water or whatever. Lots of usually read only sensors polled every couple seconds or minutes.
(Score: 0) by Anonymous Coward on Tuesday February 09 2021, @05:13PM
Never attribute to malice that which is adequately explained by
stupiditya Deep State conspiracy.