from the if-only-there-were-a-device-in-my-home-that-kept-time dept.
A mysterious issue is affecting the default Windows NTP server (time.windows.com), according to multiple complaints coming from Reddit and Twitter users, screwing up everyone's computer clocks.
Based on reports, the time.windows.com NTP server is sending Windows users the incorrect time, sometimes off by seconds, but in other cases, off even by hours. The issue was spotted today, April 3, early in the morning, and is ongoing for at least 10 hours.
The impact was felt immediately by servers that rely on the Windows NTP service to schedule and execute tasks. Unhappy admins found their servers launching routines early or too late, botching scripts and crashing their applications.
[...] UPDATE: A Microsoft representative acknowledged the issue. "We investigated and quickly resolved the issue our time service experienced," the Microsoft spokesperson wrote in an email. Tests carried out by Bleeping Computer confirmed the Windows NTP serrvice is up and running at the time of this update.
Source: Bleeping Computer
(Score: 3, Interesting) by driverless on Wednesday April 05 2017, @11:35PM (6 children)
That's why I use my own NTP server for time, even if you're using a non-spyware version of Windows there are still various services that phone home at different times, of which W32Time is the most obvious one. So I've got a GPS-disciplined FE-5680 rubidium frequency standard (you can get them on eBay for around USD100) running as my in-house NTP server. Admittedly that's a bit overkill, but you can hang a GPS dongle off a Pi and get more or less the same thing.
Assuming you don't just want to use pool.ntp.org for your time. I've got enough stuff running in-house that doesn't otherwise need to go online that I prefer to run my own NTP.
(Score: 2) by chromas on Thursday April 06 2017, @01:02AM
On Linux, I used to point crony to time.windows.com. Now I use systemd-timesyncd. How terrible am I?
(Score: 2) by LoRdTAW on Thursday April 06 2017, @01:45PM (4 children)
Curious, What is your NTP hardware setup? Are you running a dedicated NTP server or something custom?
(Score: 2) by driverless on Thursday April 06 2017, @02:00PM (3 children)
It's mostly off-the-shelf, but somewhat custom, apart from the 5680 I'm using one of these [tindie.com], and the rest is just a generic Arm embedded controller for which anything similar will do, e.g. a Pi. It requires a bit of hacking around to get it going, but it's vastly cheaper than e.g. a Symmetricom.
(Score: 2) by LoRdTAW on Thursday April 06 2017, @02:42PM (2 children)
Ha! I was googling around after I looked at your post and found the exact same board. Then I figure I'd inquire about your setup. I have also been looking at getting a 5680 or 5650 and build a similar setup just for fun. And as a side note, my company does a lot of hermetic sealing for Frequency.
(Score: 2) by driverless on Thursday April 06 2017, @02:59PM (1 child)
If you've got access to Silicon Chip magazine, you can find an article on DIY'ing it here [siliconchip.com.au]. I went for the Tindie option because it was pre-built and tested so I wouldn't have to worry about things not working properly. Since the 5680s are recycled from things like old cellphone towers they can be of variable quality and remaining operational life, and sometimes a bit finicky to get going. The painful thing is that if it doesn't work fairly quickly you can end up sinking an awful lot of time into it trying to figure out what's wrong.
(Score: 2) by LoRdTAW on Thursday April 06 2017, @03:12PM
Thank you for the information.
(Score: 2) by SomeGuy on Wednesday April 05 2017, @11:41PM
That's what happens when everyone does it the same way. One fackup and suddenly all the clocks in the world are blinking "12:00".
(Score: 2, Touché) by Anonymous Coward on Wednesday April 05 2017, @11:43PM
Unhappy admins found their servers launching routines early or too late, botching scripts and crashing their applications.
More like "exposing race conditions in scripts and hidden assumptions in their applications". See how helpful Microsoft is being?
(Score: 1, Interesting) by Anonymous Coward on Thursday April 06 2017, @12:03AM (1 child)
For example, affected people probably weren't able to visit any OCSP-stapling website, because the stapling is only valid for a fairly narrow window of time. And they might well have thought the problem was with the website.
(Score: 0) by Anonymous Coward on Thursday April 06 2017, @05:36AM
Tor refuses to work when the clock is badly wrong. "Our clock is %s behind the time published in the consensus network status document (%s GMT). Tor needs an accurate clock to work correctly. Please check your time and date settings!"
(Score: 3, Touché) by Rosco P. Coltrane on Thursday April 06 2017, @12:20AM (2 children)
I bet their telemetry servers don't either.
(Score: 2) by MostCynical on Thursday April 06 2017, @12:50AM
More likely they have cache servers (your own WinPC, for the telemetry), which will wait and send when the main server is back up.
Now we know why PCs never know how long a download is going to take - sometimes seconds are a lot further apart than others...
"I guess once you start doubting, there's no end to it." -Batou, Ghost in the Shell: Stand Alone Complex
(Score: 2) by NotSanguine on Thursday April 06 2017, @12:51AM
I bet their telemetry servers don't either.
Because reasons. [xkcd.com]
No, no, you're not thinking; you're just being logical. --Niels Bohr
(Score: 0) by Anonymous Coward on Thursday April 06 2017, @01:15AM
You apply the patch, it will try to convert all your network into Win10 spyware.
(Score: 5, Funny) by goody on Thursday April 06 2017, @01:17AM (1 child)
Microsoft discovered the time problem today at 2 PM Pacific and had it fixed by 3 PM yesterday.
(Score: 4, Funny) by driverless on Thursday April 06 2017, @02:02PM
Microsoft discovered the time problem today at 2 PM Pacific and had it fixed by 3 PM yesterday.
The fix was reported to be available in 5 minutes... 30 seconds... 17 hours... 2 minutes... 8 hours... 10 seconds... 2 days... 15 minutes... ... ...
(Score: 2) by inertnet on Thursday April 06 2017, @01:30AM
with the Pentium FDIV bug: "Current time is 03:29.9999973251:21.99989960954"
(Score: 2) by deimios on Thursday April 06 2017, @04:42AM (2 children)
Windows time servers actually send time?! That's the first time I've heard about it. I've been installing NetTime pointed to real NTP servers since forever because I could never get windows to do it.
(Score: 0) by Anonymous Coward on Thursday April 06 2017, @07:30AM (1 child)
Not only that, but Windows understands the NTP protocol, so you can just go into control panel -> date and time and change it to use pool.ntp.org. No 3rd party software needed.
I'm more surprised that time.windows.com is still operational, because last week after we had to replace the BIOS battery in an old server after the power went out, we realized that server thought it was 2002, and trying to get it to connect to time.windows.com failed (and had done so for the week or so before we realized the problem). Googling the failure only gave us other people who also could not connect to time.windows.com, and replacing it with pool.ntp.org solved the problem.
(Score: 2) by tangomargarine on Thursday April 06 2017, @03:47PM
Not only that, but Windows understands the NTP protocol, so you can just go into control panel -> date and time and change it to use pool.ntp.org. No 3rd party software needed.
What version of Windows are you using? My Windows 10 computer at work is off by ~3 minutes and I can't find what you're talking about.
"Is that really true?" "I just spent the last hour telling you to think for yourself! Didn't you hear anything I said?"
(Score: 2) by ledow on Thursday April 06 2017, @07:48AM (1 child)
Anyone using just one timeserver is an idiot anyway. If one skews like this, it will just be ignored. If you ONLY have one, of course it messes everything up. And I echo the sentiment others have expressed, that time.windows.com is more often down than up (I assume some kind of rate-limiting or similar, but it's nowhere near reliable).
You should be sticking in at least pool.ntp.org but more usefully 0..pool.ntp.org, 1..pool.ntp.org and 2..pool.ntp.org
Amazingly, Windows only show you one in the interface (which is just hilarious) but you can push as many as you like via GPO / the registry. Most Linux distros use multiple ones of either NTP pool or their own time servers.
That said, if you care about time in any significant fashion (e.g. the 5-minute Kerberos login window on Windows networks), then you should be using a mixture of in-house and multiple external servers. Otherwise when your Internet goes down for a day, your clients all start to slide out of sync and eventually will refuse to log on to the domain.
It's not hard to set up an internal NTP server, and it takes basically zero resources. Also, are you really also manually time-syncing your phones, PCs, etc. systems?
Hell, I run public NTP pool servers, it takes literally a minute or so to set up on any Internet-connected Linux machine.
(Score: 0) by Anonymous Coward on Thursday April 06 2017, @01:29PM
Is time.microsoft.com really just one time server, or does it operate like pool.ntp.org?
(Score: 3, Insightful) by Justin Case on Thursday April 06 2017, @02:41PM
We investigated and quickly resolved the issue
The MS attitude has always been "we put out the fire as soon as we knew about it!" as if that is OK. What more could anyone have expected us to do? Hey man we're trying really really hard here! Don't judge us by the outcome!
Never "hey maybe we should stop building our houses out of straw?"
(Score: 3, Insightful) by digitalaudiorock on Thursday April 06 2017, @02:45PM (2 children)
Why should anyone be surprised, coming from a company that decided that BIOS times should be stored in local time? WTF is that all about? I've used only Linux for ages with no dual boot, so I never run into that one, but it makes my skip crawl just thinking about it.
(Score: 2) by digitalaudiorock on Thursday April 06 2017, @02:47PM
typo: skip = skin
(Score: 2) by Justin Case on Thursday April 06 2017, @02:53PM
The whole problem is that MS designed (I use the word very loosely) its software for Personal Computers, and then idiots, with a lot of encouragement from MS of course, thought this junk might be just fine for Business Computers.
Their view was user-centric. "This is the time where I am." not "This is the time."
Also "these are my files on my disk" so no need for a permissions system, not at first.
When networks appeared "this is my network" so of course everything trusts everything else.
Hell they even have the abomination of naming a file system object "My Documents"!
And on and on...
In case any of you young'uns think I'm criticizing them for a lack of foresight, let it be clear that Unix had all of the above solved correctly a decade before Bill Gates got the idea he was a qualified programmer. So, you know, he could have just learned from the professionals, but noooooooo.