March 20th, 2018, Daniel Stenberg notes twenty years of his flexible, multi-protocol, text-based utility, curl. It is a very common client-side file transfer utility. The associated development libraries, libcurl are a couple of years younger.
curl itself and components from libcurl are found nearly everywhere these days. Due to such widespread use, it is hard to be precise with usage numbers, but conservative estimates suggest billions of people every day are using it, though mostly under the hood several layers down inside devices they own. It is the Internet transfer utility of choice for thousands of software applications. It is found in cars, television sets, routers, printers, audio equipment, mobile phones, tablets, settop boxes, and media players for starters.
A detailed, free-of-charge, ebook, Everything curl, covers basically everything there is to know about curl, libcurl, and the associated project.
Earlier on SN:
Related Stories
Daniel Stenberg lets the world know that cURL, the little command line utility that lets you download stuff off the internet via HTTP along with a bunch of other protocols, has turned 17 today (March 20). Considering that it is also available to all of us for use in our programs as a nifty little library called 'libcurl', and that PHP, the most common web development language depends on libcurl for handling HTTP requests, we can be happy that cURL exists. I personally cannot count the number of times it has saved me and the machines I administer.
curl is a text-based utility and library for transferring data identified by their URLs. It is now year-2038 safe even on 32-bit systems. Daniel Stenberg, the orginal hacker of curl, has overseen a year-2038 fix for 32-bit systems. Without specific modifications, 32-bit systems cannot handle dates beyond 03:14:07 UTC on 19 January 2038. After that date, the time counter flips over and starts over again at zero, which would be the beginning of the UNIX epoch known as 00:00:00 UTC on 1 January 1970. Given the pervasiveness of 32-bit embedded systems and their long service lives, this is a serious problem and good (essential) to have fixed decades in advance. The OpenBSD project was the first major software project to take steps to avoid potential disaster from 32-bit time and awareness has since started to spread to other key software project such as curl.
curl developer Daniel Stenberg has gone through his project's security problems and calculated that 51 out of curl's 98 security vulnerabilities have been C mistakes. The total number of bugs in the database is about 6.6k, meaning that not quite 1.5% have been security flaws.
Let me also already now say that if you check out the curl security section, you will find very detailed descriptions of all vulnerabilities. Using those, you can draw your own conclusions and also easily write your own blog posts on this topic!
This post is not meant as a discussion around how we can rewrite C code into other languages to avoid these problems. This is an introspection of the C related vulnerabilities in curl. curl will not be rewritten but will continue to support backends written in other languages.
It seems hard to draw hard or definite conclusions based on the CVEs and C mistakes in curl's history due to the relatively small amounts to analyze. I'm not convinced this is data enough to actually spot real trends, but might be mostly random coincidences.
After the stats and methodology, he goes into more detail about the nature of the 51 bugs and the areas in the program (and library) where they occur. In general, the problems sort out into buffer overreads, buffer overflows, use after frees, double frees, and NULL mistakes.
Previously:
(2020) curl up 2020 and Other Conferences Go Online Only
(2019) Google to Reimplement Curl in Libcrurl
(2018) Daniel Stenberg, Author of cURL and libcurl, Denied US Visit Again
(2018) Twenty Years of cURL on March 20, 2018
(2018) Reducing Year 2038 Problems in curl
(2017) Eric Raymond: "The long goodbye to C"
(Score: 2, Interesting) by Ingar on Wednesday March 21 2018, @10:23PM (7 children)
Congratulations to the cURL team.
No thanks to me though, I usually use wget. Are there any compelling reasons to switch?
(Score: 1, Informative) by Anonymous Coward on Wednesday March 21 2018, @10:28PM
Curl is somewhat more useful in shell scripts, and I've had issues with wget complaining about certificates on https stuff that curl handled without any problems, but they're interchangeable for most stuff.
(Score: 2) by cocaine overdose on Wednesday March 21 2018, @10:28PM (2 children)
(Score: 0) by Anonymous Coward on Thursday March 22 2018, @03:23AM (1 child)
I don't know if it is still true, but libcurl's ABI/API are constantly changing, leading to breakage in any binary apps you have that use them if your operating system isn't smart enough to keep old copies of the libraries around.
Furthermore there have been a LOT of CVEs on libcurl over the years and as a result binary applications can end up negligently insecure as a result of the inability to update the library for security fixes without having to recompile many apps linking to libcurl.
But that is just my 2 cents as a 20 year vetern of linux, wget, curl and company.
(Score: 2) by cocaine overdose on Thursday March 22 2018, @03:29AM
(Score: 4, Interesting) by mmh on Wednesday March 21 2018, @10:37PM
Anything that is hard in wget, is easy and cURL, the opposite is true too. The two tools compliment each other very well and if you do any type web-scraping/web-automation knowing how to use both tools well is a huge time saver.
(Score: 3, Informative) by Marand on Thursday March 22 2018, @02:19AM
Curl makes more sense in scripts usually, otherwise no. They're similar tools with slightly different end use-cases, like apt vs. apt-get. That said, I've mostly abandoned wget in favour of Aria2 [github.io] for manual downloading. I switched years ago because it had support for some features that were (at the time) missing from wget, like metalinks and magnet links; I'm sure by now wget has probably mostly caught up, but I haven't had a compelling reason to switch back. That said, I still don't see any options in wget's man page for parallel downloads, which is something you can do with the --max-concurrent-downloads flag in Aria2. (Or use '-j', which should be easy to remember if you use make.)
(Score: 2) by iWantToKeepAnon on Friday March 23 2018, @06:25PM
"Happy families are all alike; every unhappy family is unhappy in its own way." -- Anna Karenina by Leo Tolstoy
(Score: -1, Informative) by Anonymous Coward on Wednesday March 21 2018, @10:41PM (7 children)
https://www.howtogeek.com/340688/whats-coming-in-windows-10s-redstone-4-update-available-march-2018/ [howtogeek.com]
(Score: 2) by takyon on Wednesday March 21 2018, @10:53PM (2 children)
[SIG] 10/28/2017: Soylent Upgrade v14 [soylentnews.org]
(Score: -1, Troll) by Anonymous Coward on Thursday March 22 2018, @12:48AM (1 child)
Tar is very much more useful than curl.
Curl is distributed in a tar file.
Any fool can hack together an HTTP client.
HTTP is trivial.
Curl is fucking unnecessary.
Fuck Curl.
(Score: 3, Informative) by Anonymous Coward on Thursday March 22 2018, @02:13AM
curl is not just for http , it talk DICT, FILE, FTP, FTPS, Gopher, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, Telnet and TFTP too
(Score: 5, Informative) by canopic jug on Thursday March 22 2018, @05:17AM (3 children)
curl and tar coming to Windows 10 soon
Wrong. Only the names "curl", "wget", and "tar" are coming to the Vista series. What M$ has under those names is broken and the parts that work provide almost no functionality [github.com]. Although a trademark infringment case would be more appropriate, the author of curl has only submitted a Github pull request asking M$ to take the name 'curl' down [github.com] because they are misusing it.
Notice that there was no resolution of the problem. M$ responded not only with a bureacratic feint, it is now trying to shit in the RFC pool.
This is typical of M$ claiming to have something they don't in order to tick boxes on managers' checklists and prevent loss of Windows sales. At the same time, it is designed to give a very bad impression of the original open source tools to those same managers when the things under the name. Same shit from M$, just a different day.
Money is not free speech. Elections should not be auctions.
(Score: 1, Informative) by Anonymous Coward on Thursday March 22 2018, @09:50AM (1 child)
I'm well aware of the curl and wget aliases; there was a SoylentNews story about them in August of 2016: https://soylentnews.org/article.pl?sid=16/08/23/0317252 [soylentnews.org]
Notice that it says nothing about tar. Your Github links are both dated August 2016; nothing was said there about tar. What's to be called the Spring Creators Update was made available for testing in December of 2017 and is expected to be generally available next month.
> M$ responded not only with a bureacratic feint, it is now trying to shit in the RFC pool.
You're citing a discussion on Github that began and ended in August of 2016.
Your misinformation is out of date.
(Score: 3, Informative) by canopic jug on Thursday March 22 2018, @05:36PM
Go to the github links shown above today and notice that there has been no action from M$ part to fix the problem even now in 2018. Press refresh on your browser [imgur.com] and notice that the information still stands: no action has been taken from M$ to resolve the problem.
It is your tactics which are out of date.
It is a long-standing tactic for M$ shills to point to old but still relevant problems and then assert that they are no longer true due to the passage of time.
Money is not free speech. Elections should not be auctions.
(Score: 1, Troll) by Anonymous Coward on Thursday March 22 2018, @09:55AM
This is what Microsoft say on their corporate blog [microsoft.com]:
According to the quote above, the curl and wget are distinct from the aliases that existed in PowerShell, which you falsely said "are coming to the Vista series" (those were there from 2016). And the above also says that the tar command didn't ship with Windows before.
But thanks for misinforming us.