from the does-anyone-really-know-what-time-it-is? dept.
Ars Technica reports on a vulnerability where unencrypted Network Time Protocol (NTP) traffic can be exploited by man-in-the-middle attacks to arbitrarily set the times of computers to cause general chaos and/or carry out other attacks, such as exploiting expired HTTPS certificates.
While NTP clients have features to prevent drastic time changes, such as setting the date to ten years in the past, the paper on the attacks presents various methods for bypassing these protections.
There is a pdf of the report available.
Related Stories
The New Yorker has a non-technical article, The Thorny Problem of Keeping the Internet's Time, about the Network Time Protocol (NTP) from both the software and protocol perspectives. It gives a surprisingly good summary of the background of both as well as the current situation and the issues holding back the next steps. If you have networked computers, especially servers, in any capacity then you are certainly familiar with the NTP or at least its supporting utilities. NTP was developed by David Mills, who by the late 1970s, after a *little*-bit-of-improvementer his PhD, eventually ended up at COMSAT where he started working on it for ARPANET. He still works on it despite failed eyesight.
In N.T.P., Mills built a system that allowed for endless tinkering, and he found joy in optimization. "The actual use of the time information was not of central interest," he recalled. The fledgling Internet had few clocks to synchronize. But during the nineteen-eighties the network grew quickly, and by the nineties the widespread adoption of personal computers required the Internet to incorpoa-*little*-bit-of-improvementrate millions more devices than its first designers had envisioned. Coders created versions of N.T.P. that worked on Unix and Windows machines. Others wrote "reference implementations" of N.T.P.—open-source codebases that exemplified how the protocol should be run, and which were freely available for users to adapt. Government agencies, including the National Institute of Standards and Technology (NIST) and the U.S. Naval Observatory, started distributing the time kept by their master clocks using N.T.P.
A loose community of people across the world set up their own servers to provide time through the protocol. In 2000, N.T.P. servers fielded eighteen billion time-synchronization requests from several million computers—and in the following few years, as broadband proliferated, requests to the busiest N.T.P. servers increased tenfold. The time servers had once been "well lit in the US and Europe but dark elsewhere in South America, Africa and the Pacific Rim," Mills wrote, in a 2003 paper. "Today, the Sun never sets or even gets close to the horizon on NTP." Programmers began to treat the protocol like an assumption—it seemed natural to them that synchronized time was dependably and easily available. Mills's little fief was everywhere.
NTP servers keep the world's computers' clocks in synchrony, but there has been negligible amount of money kicked upstream to the project or even to Mills. Poul-Henning Kamp (PHK) gave a talk in 2015 at FOSDEM, Ntimed, an NTPD replacement, about where he saw things heading back in 2015 and how refactoring NTPd would be neither time nor resource efficient.
Previously:
(2015) New Attacks on Network Time Protocol can Defeat HTTPS and Create Chaos
(2015) Finance, Workload Troubles for Developer of Reference NTP Implementation
(2015) OpenNTPD 5.7p1 Released
(2014) What Time Is It? Time for Multiple NTP Vulnerabilities!
(Score: 2) by inertnet on Thursday October 22 2015, @11:35PM
One way to avoid this is to connect to multiple NTP servers and only accept time differences that a majority of them report back. Another is to only allow for a limited time change per 24 hours.
(Score: 3, Informative) by chrysosphinx on Friday October 23 2015, @12:45AM
Both methods you suggest are still exploitable. The first one is just heuristic expecting mim attacks are nonlocal, the second one is even worse since it can prevent recovery from attacks.
(Score: 2) by frojack on Friday October 23 2015, @02:31AM
The first one is just heuristic expecting mim attacks are nonlocal,
This method is actually built into most ntp clients.
The client will arbitrarily toss out the Outliers and select the most reliable of the rest.
No, you are mistaken. I've always had this sig.
(Score: 2, Interesting) by xav on Friday October 23 2015, @01:06PM
I'm under the impression you didn't understand chrysosphinx's answer, pardon me if I'm wrong.
With a local MitM attack, all responses from all NTP servers can be manipulated, and they will all appear to be in agreement.
(Score: 3, Informative) by Hyperturtle on Friday October 23 2015, @01:32AM
they used to, and still do, sell NTP servers for localized network purposes (localized meaning the administration of an organizational unit).
I generally configure NTP and SNTP to sync with a local, presumably reliable, device. Clock drift can be an issue and it is best to never ever use a virtual machine as a clock source that isn't otherwise corrected by a hardware peer acting as a more reliable source.
For example, workstations may get time from a virtualized domain controller, and that virtual DC hopefully gets time from a hardware box. And not the hyperv OS that is part of the domain that was virtualized and thus is dependent on its on dependency to work and provide time. You laugh; i have fixed this numerous times. P2V everything seems to be a mental affliction.
In any event, the device acting as an NTP source has to suffer from a low amount of drift. Atomic clocks are sold for that (not inexpensive), but you can also set a time sync to an internet device that hopefully is reliable (there are terms for it, one is called stratum, that is mostly irrelevant for my summary on the topic, but I mention it so anyone looking this up can spot an important consideration in their configurations).
That way, you get time from a hopefully reliable source, and then advertise that out to your internal devices. If everything on your network is going to time.windows.com or something, there is a problem that needs correction.
Often, your ISP has a router on their network you can pull time from. Or, if you pay for a voip service, those clocks may get time from somewhere off your network, and you might be able to piggyback off of it.
Keeping as much of your time concerns localized, or at least limited to a few known sources with trusted stability, is not easy, but Windows has never made it easy. It's trivial in linux and in network hardware (even cisco and juniper) to set up NTP properly, it's the source that causes the most issues... so if you set your core switch up as the source for windows and the phones, and then have that core switch get time from the ISP or telco provider, you're probably in better shape than 90% of the businesses out there who just act surprised when the time isn't the same on everything. There are better solutions for your company, I am sure, but for free advice this will eliminate practically every risk the article discusses unless your ISP has their internal network hacked or worse.
If your core switch dies, time is going to be the least of your concerns. if the ISP goes down, your core switch will be able to handle the time problem for long enough for you to get the ISP up again. If you can pull from multiple ISP resources over the same link, that'd be better, but ultimately, having a stable time source that isn't random is a very good choice lacking the resources to do it "the right way" which involves money for hardware or money for subscriptions to encrypted time services.
Ever wonder how a school keeps all the clocks on the wall synced? services like that exist and are not horribly expensive, so if you really dont trust your network hardware to keep time, you can pursue that sort of solution.
One thing not to do is use your firewall device as the time source. Have it read time from somewhere else, and have your network go through it but not to it. Or stay local. If someone takes over the firewall you're hosed in more ways than one, plus you'll be out of time besides.. it would be best not to put it all in the same basket in case you want to read your logs later and have an idea as to when it all happened...
(Score: 3, Interesting) by frojack on Friday October 23 2015, @02:26AM
Its FAR from as complex as you seem to make it. Buy an atomic clock? Seriously? Are you launching rockets or what?
Forget your local ISP, most of them are clueless.
Sync your clock to one of the pool sources (like just about every linux distro does out of the box) and call it a day.
Don't trust pools? Hard code it to hit your State's University Clock, almost every large EDU runs a stratum 1, 2, and 3 clocks.
Most are underutilized. You can use their stratum 1 clock and nobody will care. But you should probably use their stratum 2.
http://www.washington.edu/tools/time.cgi [washington.edu]
No, you are mistaken. I've always had this sig.
(Score: 4, Informative) by MrNemesis on Friday October 23 2015, @07:11AM
Those who are either truly paranoid or in need of really accurate time can buy themselves a GPS or radio clock and plug it into their local NTP server, thus having a stratum 1 inside your data centre and obviating the need for reliance on an external network.
If you don't have true paranoia or the need for really accurate time, it's not expensive to do it because it's geeky and cool.
Turning your Raspberry Pi into a stratum 1 server [satsignal.eu]
GPS and radio inputs [pvelectronics.co.uk] for your Nixie Tube clock [pvelectronics.co.uk].
"To paraphrase Nietzsche, I have looked into the abyss and been sick in it."
(Score: 2) by Hyperturtle on Friday October 23 2015, @07:24PM
(Awesome, someone who knows what the nixie tube clock is! I wanted one to put into my pipboy 3000 burned out clock from the fallout 3 collectors clock. Those things broke within weeks. Having a real nixie tube clock in it would be awesome... but then I guess I am a geek when it comes to that.)
I entirely approve of the raspberry pi; it can do it, managed switches can do it, routers can do it, windows servers can do it, and windows servers can do it with the addition of 3rd party software so that way they can more reliably do it (that windows time service is not very good.. maybe its fixed up post 2012 but I've had to take pains to correct for drift).
I have to concede that perhaps a raspberry pi or core switch is not the most reliable of time clocks -- i really dont know how reliable the clock in a raspberry pi is, or if it is emulated or what. But if it acts as a server and pulls time from another location, it can be used as the focus point for a small network. That source may still be at risk, but it prevents everything from going out and keeps this traffic local.
I imagine the raspberry pi can do the NTPSEC, do you know if that has been ported to any of the distributions?
I get the idea that many people here do not work in large environments, and that some of my ideas can be bit overkill (or simply not applicable) -- but for a place with network hardware that is unmanaged and such, your solution is a great choice.
I expect MS to push the existing "IOT" windows 10 for raspberry pi into a windows 2016 nano server for raspberry pi. If they do, I'd hope to use that to create a type of headless read only DC for small offices, but that's just me speculating. It could be the cost is prohibitive compared to $50, but we'll see. Many places, I expect would adopt this type of solution than have to learn linux, cool not withstanding.
Anyway this type of solution is great for the small business/medium business and for home use to experiment with, so if anyone wants to ignore my other thoughts, go with this one to keep traffic mostly local and limit the exposure of regular NTP traffic. I recommended a similar choice for network switches and such, but it now occurs to me that this may be easier for many to do, and cheaper besides!
(Score: 2) by Hyperturtle on Friday October 23 2015, @06:18PM
Right, I avoided using the public pools because those are what were cited as being vulnerable. Someone between you and them can alter your NTP because it isn't local, set back your clock, and then present you an invalid expired fake certficate that suddenly is more valid than it was before, since the time is now showing it's a valid cert and there was no way to protect the NTP exchange in the pools.
Almost anything over the internet is the problem, because NTP is generally not encrypted. If you said to use an IPSEC tunnel to a place that had time, then sure. But I do not agree with your suggestion because it is the dependency on such that has perpetuated the ability to cause problem cited.
Keeping it local is why I suggested it.
There are atomic clocks that are heinously expensive and over kill, and there clocks that use GPS to sync with satellites and have a 10 megabit network connection for not that much money... you just have to not have 10 floors of metal blocking that signal.
atomic clocks for network time also are available for a few thousand dollars and less.
I do realize that many places with a few servers--do not have a few thousand for an atomic clock. Heck I dont have one, but it would be cool to get a nixie tube one :)
Part of my disconnect is that I do not have a good grasp on the workplace people here have, and such costs may be pure fantasy for them, as they were for you. I don't want to discount such ideas under the assumption everything has to be done on the cheap, but I also provided cheap ways to do it. I am guessing you don't have a phone system managed by one vendor, a data network by another, and windows by other people.
Many places have desk phones that display a different time than the desk computers, which also is different from the the cell phone in your pocket.
When there is an issue, checking logs on network hardware to compare who came in when and on what -- oh look the network is 12 minutes off and the servers are 5 minutes off and the phones are 2 minutes off from the servers and wait what was the central NTP time source? it went through the compromised firewall?
These are concerns I have to deal with, so I try to contain the time locally, while using a single host or two to pull time from trusted resources and then set those devices as ntp servers.
There is no way I would let everything on the network pull time from some outside place, even if it is convenient. It may be low priority to correct, but I ultimately want control of the time done locally, and the actual time servers themselves -- I have options and is not too hard to set up, either with a dedicated device or reliance on the ISP router.
If you have an MPLS cloud, often those telco routers are great for this because they often themselves are synced to an atomic clock at the ISP or paid for to access by the ISP, and you can get the benefit of a high stratum time source by proxy of using the telco router. Note I say telco, not internet. Having internet access doesn't mean using the internet to get that access to NTP -- you likely can access telco specific routers within their network prior to passing their network edge to reach the internet, or in MPLS, just in their local cloud anyway and never come close to the internet.
Anyway, there are many options. It can be made harder than it has to be, yes, but I wanted to avoid the really easy solution because the problem was that solution. It wasn't that there is no time, its that the time itself is insecure and using public resources to get time is to be trusted as a convenience at best without further administrative correction.
(Score: 2) by ledow on Friday October 23 2015, @07:24AM
They still do.
I work in a school and we've just bought a system from a manufacturer that has an NTP clock on it, and uses a licensed frequency to sync all your clock and bell systems across the site as well. It's used in stock exchanges, etc. to sync everything together, and it doesn't come cheap.
Although NTP isn't extremely secure, it works the same way as Bitcoin - if you have enough good servers on your list, you will be fine and cancel out the dodgy ones. Using things like the NTP pool is a way to both greatly enhance the number of peers you see, but also to introduce untrusted peers into the mix, and it appears to do quite well thanks. If you don't trust it, there's no reason you can't run a local NTP server - most networks need one if for no other reason that to get the phones in sync with the computers, etc. Just choose a good peer and then have your clients sync from your server and the Internet too. If either local or remote get out of sync, whatever is in the minority will be rejected.
The most dangerous thing for this would be, for example, time.microsoft.com - an NTP server used and relied on by millions but a single point of failure. I never got why Windows doesn't let you use multiple peers without having to do it manually in the registry. It's just ridiculous.
However, on most networks, being even 5 minutes out of sync will set off alarm bells. You won't be able to authenticate against your domain, on Windows, and Kerberos tickets will start failing on other systems. So it's not going to happen without you noticing. Sure, it's not a SPECIFIC defence against it, but your clocks getting out of sync isn't going to go unnoticed for very long.
(Score: 2, Informative) by Snospar on Friday October 23 2015, @07:25AM
Interesting post, but this made me chuckle:
Ever wonder how a school keeps all the clocks on the wall synced?
All the schools I've been in had clocks that were synched manually, by a janitor on a ladder, twice a year when entering and leaving "daylight saving time". The rest of the year drift could be measured in minutes (if you were lucky) and some clocks were so inaccessible they were an hour out for half the year!
Huge thanks to all the Soylent volunteers without whom this community (and this post) would not be possible.
(Score: 2, Informative) by mystik on Friday October 23 2015, @11:50AM
There's also a time sync service from the utility, at least in the USA.
https://en.wikipedia.org/wiki/Utility_frequency#Long-term_stability_and_clock_synchronization [wikipedia.org]
In my elementary school, when I was a young'un a few times a year, the wall clocks in every classroom would start audibly buzzing. the hands would advance until it reached 12:00, and then all at once, stop buzzing, and resume telling time normally. I'm pretty sure that was a mechanism of this function, the clocks were all resetting due to maintain consistency with the classroom bells.
Why aren't you encrypting your mail?
(Score: 0) by Anonymous Coward on Friday October 23 2015, @09:13AM
Why use your own atomic clock, if the time of the official one is broadcasted anyway? [wikipedia.org]
Sure, in principle someone could interfere with that signal, but to do that, he would have to send with such a strong sender that he would be found in no time. Especially given that senders in that frequency range are not exactly known for their smallness.
(Score: 2) by darkfeline on Friday October 23 2015, @02:58AM
If NTP is still unencrypted, and someone MITMs all of your connections, multiple NTP servers won't do any good.
Really, the solution to a lot of problems is to use encryption and trust signatures, it might be the next on the list, right after "make backups".
Join the SDF Public Access UNIX System today!
(Score: 1, Funny) by Anonymous Coward on Thursday October 22 2015, @11:51PM
Time is relative, especially at tequila bars.
(Score: 2) by Gaaark on Friday October 23 2015, @01:35AM
"Heck ya! It's 5 o'drunk somewhere, amiright?"
--- Please remind me if I haven't been civil to you: I'm channeling MDC. I have always been here. ---Gaaark 2.0 --
(Score: 3, Informative) by frojack on Friday October 23 2015, @02:12AM
While NTP clients have features to prevent drastic time changes, such as setting the date to ten years in the past,
Try 16 minutes. Even a few hours off is enough for most NTP clients to intentionally not sync.
Quoting the ntp site: [ntp.org]
-g
Normally, ntpd exits if the offset exceeds the sanity limit, which is 1000 s by default. If the sanity limit is set to zero, no sanity checking is performed and any offset is acceptable. This option overrides the limit and allows the time to be set to any value without restriction; however, this can happen only once. After that, ntpd will exit if the limit is exceeded. This option can be used with the -q option.
No, you are mistaken. I've always had this sig.
(Score: 5, Interesting) by PhilSalkie on Friday October 23 2015, @02:47AM
Fortunately, some quite reliable folks have been working on issues like this for a while now:
http://www.ntpsec.org/ [ntpsec.org]
Eric said recently that they're about two weeks from a beta release - if these issues aren't already addressed, I'm sure they will be soon.
Personally, my desktop connects to a stratum 0 time source - NMEA GPS receiver, RS-232 interface board with PPS (which I designed - want one?), and GPSD.
Typically I see jitter values below 10 microseconds, and don't have to worry about somebody changing my clock over the interwebs.
(Score: 2) by Lagg on Friday October 23 2015, @01:33PM
I'd totally ask you for prices if I hadn't gotten rid of my last serial port boards. Have been interested in self-sufficient time sources for a while now
http://lagg.me [lagg.me] 🗿