curl is a text-based utility and library for transferring data identified by their URLs. It is now year-2038 safe even on 32-bit systems. Daniel Stenberg, the orginal hacker of curl, has overseen a year-2038 fix for 32-bit systems. Without specific modifications, 32-bit systems cannot handle dates beyond 03:14:07 UTC on 19 January 2038. After that date, the time counter flips over and starts over again at zero, which would be the beginning of the UNIX epoch known as 00:00:00 UTC on 1 January 1970. Given the pervasiveness of 32-bit embedded systems and their long service lives, this is a serious problem and good (essential) to have fixed decades in advance. The OpenBSD project was the first major software project to take steps to avoid potential disaster from 32-bit time and awareness has since started to spread to other key software project such as curl.
Related Stories
March 20th, 2018, Daniel Stenberg notes twenty years of his flexible, multi-protocol, text-based utility, curl. It is a very common client-side file transfer utility. The associated development libraries, libcurl are a couple of years younger.
curl itself and components from libcurl are found nearly everywhere these days. Due to such widespread use, it is hard to be precise with usage numbers, but conservative estimates suggest billions of people every day are using it, though mostly under the hood several layers down inside devices they own. It is the Internet transfer utility of choice for thousands of software applications. It is found in cars, television sets, routers, printers, audio equipment, mobile phones, tablets, settop boxes, and media players for starters.
A detailed, free-of-charge, ebook, Everything curl, covers basically everything there is to know about curl, libcurl, and the associated project.
Earlier on SN:
Daniel Stenberg, author of the ubiquitous URL fetcher cURL and the libcurl multiprotocol file transfer library, and recipient of the 2017 Polheim Prize, has been blocked again from attending a US-based conference. Daniel has written a post in his blog about his two-year odyssey through the byzantine US bureaucracy to try to get permission to attend a work-related conference in California. He has been in the US nine times previously but despite pages of paperwork, hundreds of dollars in fees, and personal visits to the embassy, no dice. As a result the conference will have to move outside the US and probably Canada too if it wants to stay open to the world's top talent.
Earlier on SN:
US Visa Applications May Soon Require Five Years of Social Media Info (2018)
Reducing Year 2038 Problems in curl (2018)
cURL turns Seventeen Today (2015)
Steven Bellovin, Professor of Computer Science at Columbia University writes briefly with a concrete example of how the Y2038 threat works.
[...] just as with Y2K, the problems don't start when the magic date hits; rather, they start when a computer first encounters dates after the rollover point, and that can be a lot earlier. In fact, I just had such an experience.
A colleague sent me a file from his Windows machine; looking at the contents, I saw this.
$ unzip -l zipfile.zip
Archive: zipfile.zip
Length Date Time Name
——— ——— ———
2411339 01-01-2103 00:00 Anatomy...
——— ———Look at that date: it's in the next century! (No, I don't know how that happened.) But when I looked at it after extracting on my [MacOS] computer, the date was well in the past:
$ ls -l Anatomy...
-rw-r-r-@ 1 smb staff 2411339 Nov 24 1966 Anatomy...Huh?
After a quick bit of coding, I found that the on-disk modification time of the extracted file was 4,197,067,200 seconds since the Epoch. That's larger than the limit! But it's worse than that. I translated the number to hexadecimal (base 16), which computer programmers use as an easy way to display the binary values that computers use internally. It came to
FA2A29C0. (Since base 16 needs six more digits than our customary base 10, we use the letters A–F to represent them.) The first "F", in binary, is1111. And the first of those bits is the so-called sign bit, the bit that tells whether or not the number is negative. The value ofFA2A29C0, if treated as a signed, 32-bit number, is -97,900,096, or about 3.1 years before the Epoch. Yup, that corresponds exactly to the November 24, 1966 date my system displayed. (Why should +4,197,067,200 come out to -97,900,096? As I indicated, that's moderately technical, but if you want to learn the gory details, the magic search phrase is "2's complement".)
While attention is paid to desktops and servers, they are easy to replace and have very short lifetimes. In contrast embedded systems should be receiving special action already since they are seldomly or never updated and have lifespans measured in decades.
Previously:
Reducing Year 2038 Problems in curl (2018)
The Time Is... 1500000000 (2017)
curl developer Daniel Stenberg has gone through his project's security problems and calculated that 51 out of curl's 98 security vulnerabilities have been C mistakes. The total number of bugs in the database is about 6.6k, meaning that not quite 1.5% have been security flaws.
Let me also already now say that if you check out the curl security section, you will find very detailed descriptions of all vulnerabilities. Using those, you can draw your own conclusions and also easily write your own blog posts on this topic!
This post is not meant as a discussion around how we can rewrite C code into other languages to avoid these problems. This is an introspection of the C related vulnerabilities in curl. curl will not be rewritten but will continue to support backends written in other languages.
It seems hard to draw hard or definite conclusions based on the CVEs and C mistakes in curl's history due to the relatively small amounts to analyze. I'm not convinced this is data enough to actually spot real trends, but might be mostly random coincidences.
After the stats and methodology, he goes into more detail about the nature of the 51 bugs and the areas in the program (and library) where they occur. In general, the problems sort out into buffer overreads, buffer overflows, use after frees, double frees, and NULL mistakes.
Previously:
(2020) curl up 2020 and Other Conferences Go Online Only
(2019) Google to Reimplement Curl in Libcrurl
(2018) Daniel Stenberg, Author of cURL and libcurl, Denied US Visit Again
(2018) Twenty Years of cURL on March 20, 2018
(2018) Reducing Year 2038 Problems in curl
(2017) Eric Raymond: "The long goodbye to C"
(Score: 5, Informative) by FatPhil on Friday February 02 2018, @09:32AM (3 children)
Are you sure it doesn't just become massively negative, and start to represent some date/time in 1901?
Great minds discuss ideas; average minds discuss events; small minds discuss people; the smallest discuss themselves
(Score: 2) by canopic jug on Friday February 02 2018, @10:01AM (2 children)
Apparently it depends on the system or package whether the 32-bit value is signed or unsigned. So there is apparently no general solution. But you'll probably want to check with someone knowledgeable or find some other authoritative answer.
Money is not free speech. Elections should not be auctions.
(Score: 0) by Anonymous Coward on Friday February 02 2018, @12:00PM (1 child)
But the overflow only occurs if the value is signed.
Also, if you are comparing the values yourself, and assuming it's an application that doesn't need dates before 1970, the fix should be as simple as casting to unsigned before comparing.
Of course that will cause another overflow problem by 2106. Assuming that there still will be 32-bit systems around at that time.
(Score: 0) by Anonymous Coward on Friday February 02 2018, @12:29PM
All things considered it also depends on what the application uses the date for. Most calls to time functions are to set short term timers (unless the rollover occurs exactly during the lifetime of that timer nothing happens), print a date to screen/a log etc. A rollover would not have any effect on these other than displaying the wrong date in the latter two cases. These kind of things are more an issue if you rely on earlier dates from a database/external timestamp to do tasks.
(Score: 4, Interesting) by TheRaven on Friday February 02 2018, @11:40AM (7 children)
Some other systems have a later epoch. I'm not sure what they do at the OS level, but Cocoa exposes a time with a 2001 epoch, so has a 2068 problem not a 2038 one (and they're likely to have killed off 32-bit software long before 2038 anyway). I think Windows has one somewhere in the middle. POSIX requires a 1970 epoch, but as long as gmtime and localtime work, most code will work unmodified if you simply move the epoch date, unless they're writing it out to files (several filesystems use different epoch dates, because there's no need to support files created in the past, so don't use a signed number: even with a 32-bit number and a 1970 epoch, that's equivalent to a 2038 signed epoch).
Specifying the epoch time was a big mistake for POSIX.
sudo mod me up
(Score: 3, Funny) by KritonK on Friday February 02 2018, @01:22PM (2 children)
Oddly enough, all my files were created in the past!
I think you meant that there's no need to support files created before the epoch.
(Score: 2, Funny) by Anonymous Coward on Friday February 02 2018, @02:54PM (1 child)
OMG! so are mine!
do i have to patch my system?
(Score: 4, Funny) by Immerman on Friday February 02 2018, @04:12PM
You could, unfortunately TachyonOS is the only viable alternative, and it only supports files created in the future.
(Score: 2) by darkfeline on Friday February 02 2018, @11:23PM (3 children)
That's like saying, don't hard code the value of pi, what if it changes?
The Unix Epoch is CONSTANT, otherwise timestamps fail to be, you know, unique.
Join the SDF Public Access UNIX System today!
(Score: 2) by TheRaven on Monday February 05 2018, @11:34AM (2 children)
Who cares? The epoc is an arbitrary point in time that lets you use a counter to identify other points in time. Times stored in a filesystem may not use the UNIX epoc, they may use an arbitrary point, as long as they also store that epoc time. Anything sent on the wire is defined by the protocol, not by your OS notion of time. If you're embedding UNIX epoc-relative numbers in your data files then you're either using 64-bit values in which case when you do the local-system-epoc-to-UNIX-epoc conversion you can perform the extension, or you're using 32-bit formats because you don't expect your files to be around after 2038 and if they are then you're screwed anyway.
sudo mod me up
(Score: 2) by darkfeline on Monday February 05 2018, @06:02PM (1 child)
And how are you going to store the epoch time, standardize an epoch epoch time, and store the epoch time as the number of seconds since the epoch epoch? Congratulations, you've just invented a roundabout way of using 64-bit time.
Join the SDF Public Access UNIX System today!
(Score: 2) by TheRaven on Tuesday February 06 2018, @10:40AM
sudo mod me up
(Score: 1, Touché) by Anonymous Coward on Friday February 02 2018, @03:50PM (1 child)
on the contrary my friend, say 5 minutes before the zero hour the funding will be at an astronomical level
(Score: 2) by DeathMonkey on Friday February 02 2018, @07:24PM
This will be fixed by Germans or Indians.
The US no longer has the will to take the advice of scientists and take preventative action.
(Score: 1) by flippynet on Friday February 02 2018, @05:51PM (2 children)
Linux has had this fixed on x86_64 for years (this is on redhat 6.6!!):
so in this environment it works for a long long time:
web browsers don't seem to have this problem either.
intel 32 bit platforms are almost completely nonexistent anymore. when 2005 called and asked for my netbook back, i told him to come and get it but he never showed up, the bastard.
but the problem still exists on ARM32 (3.10.18 on an iMX6):
even so, i have to wonder............ will we still be using curl in 2038? we'll probably all have our implants by then.
(Score: 2) by bob_super on Friday February 02 2018, @06:13PM
> intel 32 bit platforms are almost completely nonexistent anymore.
True, only a few hundred million embedded systems ... almost nothing !
Gladly, most will indeed be obsolete by 2038.
But how many nuclear subs will not be?
(Score: 2) by driverless on Monday February 05 2018, @08:12AM
That's only for Linux though. Under Windows it works for an __int64 time.
(Score: 2) by HiThere on Friday February 02 2018, @06:50PM (7 children)
I have a suspicion that the work is unnecessary, as most things will be changed within two decades.
OTOH, it's something that *ought* to be done properly anyway. The question is "What does properly mean?" and that's extremely use-case dependent. For some purposes using even an extra byte is a mistake...but those cases won't be running POSIX anyway. But you clearly shouldn't use a date for a short timer. And signed numbers in times are generally a bad idea...but not always.
The suggestion about depending on gmtime and friends function is wrong, because one often needs time expressed as a number. Sometime an integer is required, but often a float is better. I'd say that the time should be in seconds past the epoch as a double, and this should give sufficient resolution (often spuriously exact) on most systems. Unfortunately, this means the further you get from the epoch, the less exact the time is.
There probably *isn't* a universally good answer. For my purposes the floating number of seconds (not even double) since the epoch would suffice quite well, but others have other needs. My best guess at the answer is to have the epoch be defined by a global variable signed 32 bit integer as the number of years since, oh, January 1st, 1970, and time defined as a system default integer number of microseconds since the epoch. That should handle all the common cases. and have a second time defined as a 64 bit float number of seconds since the epoch as defined by the system. (I'd specify a 128 bit float, but 64 bits is more accurate than the current clocks already.) This would give you minimal overhead, and allow the epoch to be easily updated in case of need.
Javascript is what you use to allow unknown third parties to run software you have no idea about on your computer.
(Score: 2) by AthanasiusKircher on Friday February 02 2018, @07:39PM (1 child)
While "most things" is likely true, lots of basic embedded code in OSes and such will likely still be around.
No offense intended, but this was precisely the attitude that created the Y2K "bug," which ended up costing hundreds of billions of dollars as old programmers expert in dead computer languages were pulled out of retirement to fix things... things that no one expected to still be around a couple decades after the code was first written.
(Note: there are those who claim a lot of Y2K prep was unnecessary and that the potential critical failures were overestimated. One can probably argue that there was a bit of overkill beyond major critical systems. But clearly there still was a lot of minor old code that was still in use and would have needed patching eventually... even if it wouldn't have resulted in a catastrophic failure on January 1st, 2000.)
(Score: 2) by HiThere on Saturday February 03 2018, @02:00AM
The thing is we've been (slowly) working on the problem ever since before the 64 bit chips became common. Places where it is critical should already have been patched. And for places where it's not critical....up until last year I still had a MSWindows95 system running. It was annoying that I couldn't set the time, but it sure wasn't important.
And since it's still two decades off, there shouldn't be any need for more than normal updates to software. (Of course you can say that some applications won't update their software, but they won't be helped by this kind of thing anyway.)
I remember that people started talking about how to address the 2038 problem back in the 1990's. So anything critical that people are willing to fix should already be fixed, and for the rest we're going to be forced to depend on obsolescence. Just make sure that any new software or updates to old software handles it properly. And that should be basic routine, not anything special.
I.O.W. if there's a new update to curl, and *that* is the main feature, then I don't see the reason for it. It should be built into the next standard update, the next standard release, whatever.
Javascript is what you use to allow unknown third parties to run software you have no idea about on your computer.
(Score: 3, Insightful) by darkfeline on Friday February 02 2018, @11:35PM (1 child)
The "proper" fix is to use a 64 bit signed int, which gives you more resolution than you can even measure on a quartz clocked CPU (nanoseconds or picoseconds) and more bits than needed to represent the entire lifetime of the universe, from Big Bang to heat death.
Join the SDF Public Access UNIX System today!
(Score: 2) by HiThere on Saturday February 03 2018, @01:43AM
But for some systems that's using an extra 32 bits every time you invoke the time. Which is why I used two standards. The 64 bit int of nanoseconds would be a reasonable replacement for the 64 bit float I suggested as the second form, and you can argue quite reasonably that it shouldn't have a flexible epoch, but there also needs to be a more compact form.
Actually, there need to be several standard forms, but they could all be based off of one of the other two bases. Sometimes what you need is a 16 bit time with a step size of 30 seconds. Usually memory isn't so tight that it really matters, but sometimes it does, and it's good to have well defined standards to drop into place. Float times have the advantage that different precisions of clock can each give all the precision they have available. etc. Of course you can do that with an int, too, but integers carry a false implication of total precision.
Javascript is what you use to allow unknown third parties to run software you have no idea about on your computer.
(Score: 2) by TheRaven on Monday February 05 2018, @11:43AM (2 children)
I presume that's addressed to me, but it implies that you didn't actually read my post. You use gmtime when you want a time that is relative to something specific. You use an epoc-relative time when you want some numbers that you can compare, but if you actually care when the epoc zero time was then you are doing it wrong.
Sounds unlikely. A 32-bit floating point value has a 24-bit mantissa. If you're only counting seconds, then that gives you a range of 194 days. If you count milliseconds, it gives you a range of 4 days.
OpenStep uses a double for NSTimeValue, which gives you around 285 million years at second precision, 285 thousand years at milisecond precision, 285 years at microsecond precision, and about 100 hours at nanosecond precision.
If you want two epochs, why not have a 64-bit one defined at a fixed point and a 32-bit one defined as system boot time (UNIX actually does define a since system boot timer already)? For most uses, the 32-bit one is fine. It will handle uptimes of a few decades (unlikely, even for VMs)
sudo mod me up
(Score: 2) by HiThere on Monday February 05 2018, @06:11PM (1 child)
OK, that's less coverage than I had supposed. But it would still work for my current purposes.
System boot time would work for some purposes, but not for any that had to be persistent. I don't really want two epochs, but for different purposes I want an accurate timer that can be relied on to be approximately identical (nearly monotonic) across machines, and I also want a compact time. It would be really nice if you could combine them into one time, but the requirements conflict. The long time epoch is so that different machines in different places will agree on what time it is, and the compact time epoch is so that you can refer to times in the "recent" past and future without having overflows. Boot time can make it difficult to refer to things that happened last year...or even two hours ago.
Javascript is what you use to allow unknown third parties to run software you have no idea about on your computer.
(Score: 2) by TheRaven on Tuesday February 06 2018, @10:35AM
sudo mod me up