Russ Cox, who developed the dependency/package management system for Go, writes about the problems with software dependencies. A choice excerpt:
Dependency managers now exist for essentially every programming language. [...] The arrival of this kind of fine-grained, widespread software reuse is one of the most consequential shifts in software development over the past two decades. And if we’re not more careful, it will lead to serious problems.
A package, for this discussion, is code you download from the internet. Adding a package as a dependency outsources the work of developing that code [...] to someone else on the internet, someone you often don’t know. By using that code, you are exposing your own program to all the failures and flaws in the dependency. Your program’s execution now literally depends on code downloaded from this stranger on the internet. Presented this way, it sounds incredibly unsafe. Why would anyone do this?
(Score: 4, Informative) by DannyB on Thursday January 24 2019, @10:39PM (12 children)
They say, the first step is to admit you have a problem.
Wean yourself off of so many dependencies.
Make sure dependencies you use are safe, and don't let them change under your nose without express permission. It would help if automated build tools would locally cache hashes of those versioned dependencies, and even if the same item and version must be re-fetched, the new download of Foobar v.1.23 could be re-verified that it is exactly the same as what it was before.
If a new version 1.24 of Foobar comes along, the developer should have to agree to bring this into the project build.
When Java programmers talk about Dependency Injection, they don't mean caffeine.
The server will be down for replacement of vacuum tubes, belts, worn parts and lubrication of gears and bearings.
(Score: 2) by AssCork on Thursday January 24 2019, @11:19PM (2 children)
The solution to this last part would be "unit tests", wherein a developer spends 250% of their time coding 'tests', then 50% of their time coding actual code to pass the tests.
Source: The internet (because no psychopath would actually do this)
Just popped-out of a tight spot. Came out mostly clean, too.
(Score: 2) by DannyB on Friday January 25 2019, @04:06PM (1 child)
You Knit tests may not uncover malware hidden in one of your dependencies. As long as all the unit tests are passed you won't notice anything.
The server will be down for replacement of vacuum tubes, belts, worn parts and lubrication of gears and bearings.
(Score: 2) by AssCork on Monday February 04 2019, @09:47PM
Build tests take care of that before signing and staging.
And again during Release.
Just popped-out of a tight spot. Came out mostly clean, too.
(Score: 2) by c0lo on Thursday January 24 2019, @11:32PM (7 children)
Because reinventing the wheel is so... pleasurable. I can spend my entire life re-implementing loggers and HTTP client-end libs and what not.
The main software one has in mind can wait for as long as necessary, "delayed gratification" is a virtue. That software one would like to see created? That's a nothing in the bigger picture - one can wait for around 60+ years to become financial independent enough to escape the rat race.
Ah! If rewriting utility libraries doesn't excite you enough, making sure a new version of such a dependency does patch the older bugs and doesn't reintroduce other surely will surely bring you to orgasm.
https://www.youtube.com/@ProfSteveKeen https://soylentnews.org/~MichaelDavidCrawford
(Score: 1, Funny) by Anonymous Coward on Friday January 25 2019, @04:15AM (2 children)
Why would it take you a lifetime to write a logger and an http client? Are you using an iPhone with autocorrect to code?
(Score: 3, Touché) by c0lo on Friday January 25 2019, @04:35AM (1 child)
Those two, won't. "What not" on the other side... I mean, look, try writing whatnots yourself and you'll see.
I don't get this one. Maybe because I never used an iPhone.
https://www.youtube.com/@ProfSteveKeen https://soylentnews.org/~MichaelDavidCrawford
(Score: 2) by kazzie on Friday January 25 2019, @08:12AM
It's a reference to https://soylentnews.org/article.pl?sid=19/01/24/214255 [soylentnews.org]
(Score: 0) by Anonymous Coward on Friday January 25 2019, @04:39AM
I think there is a balance here. There is a difference between using the standard HTTP client-end libs and something like left-pad or is-odd. For goodness sakes, there are packages on npm that are literally one line negations of other packages.
(Score: 2) by Immerman on Friday January 25 2019, @08:13PM (2 children)
I've got to disagree about accepting dependency updates un-verified. That's a really big source of potential new bugs that can sometimes be a problem even with professional packages, much less those developed and updated by "someone on the internet". Especially since all you need to do is run your unit tests to verify that everything still works correctly. You do have comprehensive a unit test suite, right? Okay fine, I admit it, neither do I. And I'd bet good money that neither do many/most of the authors of all those dependencies that you're trusting not to break.
(Score: 0) by Anonymous Coward on Friday January 25 2019, @10:37PM (1 child)
What is one to do if a dependency 5-steps removed from the software one develops uses a vulnerable OpenSSL version? Vuln that one learns long after the software one has been released?
(Score: 2) by Immerman on Saturday January 26 2019, @01:01AM
Then it behooves you to update your library, doesn't it? The point is not to not update your libraries, it's to not update them automatically without first verifying that the update doesn't break anything.
I've thought that the ideal way to handle external library updates is for the executable to specify the latest "certified-good" version of the libraries it uses, even supply a copy in it's internal program space. And then let users decide whether they want to use that version, or the latest independently updated version, based on default or program-specific library settings. Probably you'd usually default to "use latest", but having the option to switch to the "certified good" or "last working" version at the press of a button would be very valuable. Ideally the OS would even keep track of library updates so that you could flag a program as buggy and have it automatically revert (for that program) any libraries that were updated since the last non-buggy use.
(Score: 2, Funny) by Anonymous Coward on Friday January 25 2019, @05:02AM
My name is Anonymous Coward and I have too many software dependencies.
Thanks Bill! I feel much better now.
(Score: 1, Informative) by Anonymous Coward on Thursday January 24 2019, @10:49PM (2 children)
1 - Laziness
2 - lack of ability
3 - unrealistic deadlines
(Score: 2) by arslan on Thursday January 24 2019, @11:08PM
To be fair we've been trying to dumb down programming even since it started. Where there's something good to be had, we've always endeavor to do more, quicker and cheaper.
(Score: 1) by NPC-131072 on Thursday January 24 2019, @11:20PM
No idea [github.com]
(Score: 0, Funny) by Anonymous Coward on Thursday January 24 2019, @10:50PM (1 child)
The obvious reason is that they are a millennial. As one of the most ludicrous generations ever to exist (safe zones, full-stack tattoos, pink hair, near-infinite gender choices, etc), linking in a program from a complete stranger is par for the course for them.
(Score: 0) by Anonymous Coward on Sunday January 27 2019, @03:48AM
You neglected to mention ginormous hair-buns. In my neck of the woods, the bitches wear their hair in huge buns, smack on the top of their heads. They look like they're growing new heads on top of their existing ones.
(Score: 4, Insightful) by Anonymous Coward on Thursday January 24 2019, @10:54PM (16 children)
Most software requires an operating system, which is a whole heap of code written by people you don't know, and which you have no real idea what most of it does.
Why would anyone do this?
(Score: 2, Informative) by Anonymous Coward on Thursday January 24 2019, @11:10PM (7 children)
Don't forget the BIOS. It's insecure turtles all the way down, son.
(Score: 2) by fyngyrz on Thursday January 24 2019, @11:15PM (6 children)
Don't forget the microcode in the CPU, FPU and GPU, either.
--
Bread is like the sun. It rises in
the yeast, and sets in the waist.
(Score: 1, Interesting) by Anonymous Coward on Thursday January 24 2019, @11:33PM (5 children)
Intel provides that code for a very nice price though - [$0]. One of the reasons they are better than AMD, along with cpu durability,
(Score: 5, Insightful) by The Mighty Buzzard on Thursday January 24 2019, @11:55PM
They do not provide source to the management engine though, which kind of makes the microcode irrelevant.
My rights don't end where your fear begins.
(Score: 2) by fyngyrz on Friday January 25 2019, @02:00AM (3 children)
As TMB pointed out — not all of it, they don't. So really, you have no idea what might be going on under that heat sink.
--
When I dunk my cookies, I think of you.
I hold them under until the bubbles stop.
(Score: 0) by Anonymous Coward on Friday January 25 2019, @04:45AM (2 children)
If I remove the heatsink while it is running to check the cpu intel throttles, but AMD burns up: https://www.youtube.com/watch?v=Xf0VuRG7MN4 [youtube.com]
This is why I say intel is more durable.
(Score: 2) by Immerman on Friday January 25 2019, @08:15PM
If you remove the heatsink while using your computer, you've got bigger issues than the quality of your CPU...
(Score: 0) by Anonymous Coward on Saturday January 26 2019, @12:05AM
That video is almost 14 years old.
(Score: 2) by MostCynical on Thursday January 24 2019, @11:15PM (3 children)
If systemd continues to grow, it will be a complete OS soon enough.
"I guess once you start doubting, there's no end to it." -Batou, Ghost in the Shell: Stand Alone Complex
(Score: 4, Funny) by fadrian on Friday January 25 2019, @12:32AM (1 child)
Don't worry - by that time, emacs should have absorbed enough AI to fight it.
That is all.
(Score: 0) by Anonymous Coward on Friday January 25 2019, @04:20AM
systemd should implement the emacs doctor psychotherapist
(Score: 3, Insightful) by DannyB on Friday January 25 2019, @04:47PM
I'm not worried about systemd growth. Nor about the size of Emacs. Neither of these have grown anything like the
bloatgrowth and new features of Java. A sprawling bytecode VM managed runtime with GC that grows and is almost an entire OS unto itself.The server will be down for replacement of vacuum tubes, belts, worn parts and lubrication of gears and bearings.
(Score: 1, Touché) by Anonymous Coward on Thursday January 24 2019, @11:18PM
Most of us, if we had to code the OS first, ( then libraries, then then ) we never would get to the application..
But at least having the code available means you *could* review it for yourself.
(Score: 5, Funny) by Anonymous Coward on Friday January 25 2019, @02:13AM (2 children)
True, and that's why you should avoid Linux which is written mostly at night by hackers. If you want an OS you can trust then you need professionally-written software from a trustworthy vendor like Microsoft.
(Score: -1, Offtopic) by Anonymous Coward on Friday January 25 2019, @03:01AM (1 child)
Going for the +5, Funny, are we? Unfortunately, I always post as AC so I can't give any mod points. But if I did, this one would surely with the internet for today.
(Score: 2) by DannyB on Friday January 25 2019, @04:49PM
Hey, the +5 Troll is much more coveted than the +5 Funny.
I have gotten plenty of +5 Funny around here. But only one +5 Troll on SN, and one, a long time ago on the green site.
The server will be down for replacement of vacuum tubes, belts, worn parts and lubrication of gears and bearings.
(Score: 4, Interesting) by JoeMerchant on Thursday January 24 2019, @11:29PM (8 children)
I can think of one very good reason we're using one particular library: "standards compliance." We need to import/export images in "DICOM standard format." Except: DICOM is a messy, evolving thing. Better for us to rely on a DICOM package maintained by strangers who live and breathe DICOM on a daily basis and keep up with the evolving standard through them, instead of us adding two or three full time developers to do the same thing in-house.
Can these strangers screw up? Of course they can, which is why we test our code before release. If they ever abandon the library, then we might have to port over to a better maintained package. If there aren't any maintained packages out there, perhaps the standard isn't so valuable after all, or perhaps it has matured and we can rely on the last stable package well into the future.
Do we go grabbing every package in sight, willy nilly, just to add bullet points to our spec sheet? No, _we_ don't do that, but if you do, you might be heading for the type of problems the author is FUD mongering about.
🌻🌻🌻 [google.com]
(Score: 3, Insightful) by The Mighty Buzzard on Friday January 25 2019, @12:32AM
The author's not wrong for simple things but has lost his bloody mind if he thinks you should never use a library. There's not a thing in the world wrong with writing your own interface to someone's web API and only including what needs to be there but it'd be dogfuckingly stupid recreate all the code necessary to send a bloody TLS request instead of using libraries.
My rights don't end where your fear begins.
(Score: 2) by sjames on Friday January 25 2019, @01:47AM (3 children)
Well considered use of a library for a specific purpose makes plenty of sense. The problem is too many projects importing dozens of libraries from all over, sometimes just to use a function or two, or even a few constants. And of course, those imports do their own imports.
(Score: 2) by JoeMerchant on Friday January 25 2019, @02:39AM (2 children)
My most evil dependency is Qt - my code is f-ing married to it, and there's not a viable alternative that does what it does. Of course, Qt is quite well supported and the quality has been generally good. If it ever goes downhill I guess I'll just use the old reliable versions instead of updating to the latest all the time like I do now.
🌻🌻🌻 [google.com]
(Score: 2) by sjames on Saturday January 26 2019, @02:58AM (1 child)
GUI libraries are a real issue. On the one hand, it's a big dependency ball and the dependency runs deep since it dictates the entire flow of execution. On the other, it's the only sane way to have a GUI at all. I certainly wouldn't expect you to reinvent that wheel and speak raw X protocol from your code.
(Score: 2) by JoeMerchant on Saturday January 26 2019, @10:38PM
Then X becomes the dependency - yeah, it's the only sane way...
Someone should submit the story about Google and Oracle and the API copyright cases currently flipping over in appeals court - that's a mess waiting to explode.
🌻🌻🌻 [google.com]
(Score: 3, Informative) by darkfeline on Friday January 25 2019, @09:57AM (2 children)
I suppose I am partially at fault for choosing that particular excerpt. As we all know, no one reads TFA.
If you actually read the article, you will find that Russ doesn't argue against adding dependencies, but rather that we should properly vet them and that perhaps we now need tools to help us validate and update our dependencies rather than just download and distribute them.
> Do we go grabbing every package in sight, willy nilly, just to add bullet points to our spec sheet? No, _we_ don't do that
May I introduce you to the modern NPM/Node.JS programmer, where we have such lovely things as the is-odd package, a package with a million weekly downloads that implements a single function that returns true if a number is odd. Of course, there's the corresponding is-even package, which is implemented as !isOdd.
Join the SDF Public Access UNIX System today!
(Score: 2) by JoeMerchant on Friday January 25 2019, @01:53PM (1 child)
Not true, I read TFA in at least 1% of cases that I comment on ;-)
🌻🌻🌻 [google.com]
(Score: 0) by Anonymous Coward on Friday January 25 2019, @11:54PM
Anathema!
(Score: 4, Insightful) by DrkShadow on Thursday January 24 2019, @11:43PM (2 children)
One big point of shared libraries is that you're not reimplementing code that has already been implemented -- that you're not tasked with implementing it, testing it, fixing it, adding features to it, and so on. Other people do these things, and you sit back and benefit. The argument is that it's much less likely that there is an error in a widely reused package than there is an error in your custom, reinvented wheel of a feature.
Are there going to be _more_ errors in external libraries? Would there be more errors than if you wrote _ALL_ the included functionality yourself? Will you be able to fix all errors as they occur?
Would you trust software where every implementation of SSL was coded by each developer, or would you trust software that uses the current version of OpenSSL? How do you feel about your SSH server? Should each distro include their own, custom-written SSH server?
This article is FUD. In my experience, things _now_ are a _great_deal_ more stable and secure and usable than they were 20 years ago.
(Score: -1, Troll) by Anonymous Coward on Friday January 25 2019, @05:56AM
FTFY. We are NOT in your fantasy world of rainbow-pooping unicorns and infallible upstream devs; we who write and maintain real code, have to deal with the ugly reality as it is.
(Score: 2) by darkfeline on Friday January 25 2019, @09:50AM
I suppose I am partially at fault for choosing that particular excerpt. As we all know, no one reads TFA.
If you actually read the article, you will find that Russ doesn't argue against adding dependencies, but rather that we should properly vet them and that perhaps we now need tools to help us validate and update our dependencies rather than just download and distribute them.
Join the SDF Public Access UNIX System today!
(Score: 2, Interesting) by Anonymous Coward on Friday January 25 2019, @05:31AM (2 children)
I remember when programmers wrote their own functions, their own routines, their own libraries. A library was just a convenient mechanism for managing a group of related functions. There were external libraries but they rarely changed and were well documented and eventually formed the core of standards like POSIX.
Libraries were shared by developers working on the same project but responsibility for the code being used resided with the developers - and their managers, who were older, greybeard engineers, not wanna-be MBAs, by the way.
Everything being used was stored in source code repositories under strict version control. I've supported more different version control systems than you have fingers and toes, by the way. Really. GitHub is just the latest in a long, long line. This, too, shall pass.
Everything was stored locally and backed up locally. We had highly available clusters, based on NFS, but everyone in the building was smart enough, back then, to know that the 'cloud' drawn on the whiteboard during meetings was imaginary.
Machines were not accessible via the Internet but we did have modems, which were quite adequate because everything we were doing with source code editing and compilation did not involve a graphic user interface, anyway - although many of the projects being worked on, did, particularly back when X was big.
This is what happens when you let developers do operations - so-called "DevOps" - really, an excuse for hanging two jobs onto one person, with one person's pay. Suckers!
Real sysadmins understand dependencies. They ruthlessly eliminate offsite and remote dependencies, secure the source code to those external elements they DO depend upon, and lock everything down so it doesn't change.
Today's corporate leaders and managers think that because they have installed Windows and maybe Linux once or twice in college, they understand systems administration, and are qualified to make critical engineering decisions related to the future of their company, IE, choosing to base the entire future of their company on AWS.
You can't fix stupid. Against stupidity, the Gods, themselves, struggle, in vain. Etc. Talking to these folks is like talking to a brick wall. It's better use of one's time to make some popcorn and watch the slow-motion train wreck in action. You can't stop them. Don't waste your time trying. They don't want to listen.
Those whom do not study the lessons of history are doomed to repeat them. Have fun.
A minor tirade: as a systems administrator, I learned UNIX. Then, I learned how to rewrite sendmail configurations. Then, I learned how to automate, with shellscripts, and crontab entries. Then I learned another release of UNIX. Then I learned another vendor's UNIX. Then I learned how to compile kernels. Then I learned how to manage routing tables. Then I learned SNMP. Etc, etc, etc.
Each one of these skills is extremely complex and extremely valuable, all on its own - even today. The proof is in how many poseurs there are out there pretending to possess or understand these skills.
This was all decades ago, back in the 1980s, in just the first few years of my career, back when they called us "systems managers", before "information technology" existed, back when it was still MIS.
Since then I have learned and forgotten more UNIX lore than I care to think about. I'm a walking encyclopedia.
Then came Linux. Do you know how many Linux releases there are out there? When I was at Linuxcare, I was responsible for a library of ~150 different Linux releases. And that was ~2000!
About the same time, Cisco came out. Then other router companies. Then firewalls. Lots of firewalls. And network monitors. And N-tier architectures, from every Internet vendor in existence. All those releases! All those manuals! All those commands!
Do you know how many Linux and FreeBSD kernels I have customized and compiled, from source, in the past twenty years?
And now these cheap fucks want me to learn AWS, and Python, and Google Cloud, and Ruby, and Chef ... write code all day ... AND be on call, all night ... AND manage MongoDB, along with the four or five other databases I know ... and ... and ... and ... and ... the list just keeps growing.
Fuck these assholes. I have no use for AWS, personally. If I have to learn it on my own time, I quit.
I think someone needs to make a statistical analysis of how many people there are in the world ... how FEW of them are interested in information technology AND actually qualified, intellectually speaking, to contribute meaningfully to the industry (as opposed to profiting from the glitz of IT and the power to deny services) ... and what the resulting odds are of a given employer locating a candidate who has all that crap on their resume AND is willing to put up with being lied to and manipulated and employed on a temporary basis, over, and over, and over, and over AND is qualified to manage their crap one-of-a-kind infrastructure built by a rapid succession of college students and contractors.
I can actually make better money, per hour, and enjoy life much more, working outdoors, cutting and selling firewood to my neighbors, out in the country.
~childo
(Score: 0) by Anonymous Coward on Friday January 25 2019, @08:08AM
The people you describe are not stupid, they are in the business of surviving and living.
For that purpose they make a forest of complexity around themselves.
Meetings, coding, politics, bureaucracy is complex, because more people have reson to exist in that complexity.
In a complex situation nobody is worthless =)
By the way, in school it was said to me that the age of modern human begun whith the keeping of cattle.
Cattle come in many forms, as example.
You might think, as example, that the sweeds care for the mentally ill cuz they are nice.
Actually, they are not, they are ceeping the mentally ill as political cattle.
As long as they exist, and are the lowest there is, and they have sufficient living standards, nobody can come to the "somewhat" higher level little-people and take what they have with a good motivation. You have to go for the lowest first.
And so, it is good to keep a good stock of the lowest breeds =)
-zug
(Score: 0) by Anonymous Coward on Friday January 25 2019, @09:42AM
Best of luck with the lumberjack gig.
(Score: 0) by Anonymous Coward on Friday January 25 2019, @07:20AM (1 child)
IMHO, there's a middle ground. You'd be nuts to implement your own crypto routines and image libraries in C. OTOH, a few years ago a bunch of web sites failed because of a JavaScript one-line library that was being linked by a bunch of people.
Good development is somewhere in the middle. The one time I found points of conflict was where somebody wanted me to try some new build system that required Python. My makefiles worked fine. I rejected that. I'm sure somebody loves the Python based build system for some reason... they started with it, they should stay with it. Don't force that person to switch to makefiles. Don't force me to use your build system... unless it's part of some massive integration where everybody has to do it the same way. Then you just have to suck it up sometimes, and sometimes the idiots who throw the kitchen sink into dependencies win, and then stuff that used to be meaningful gets pared down to one line, then somebody figures we could live without it and... ooops.
(Score: 0) by Anonymous Coward on Friday January 25 2019, @07:51AM
There is always middle ground, but there is no room for that on the internet. This is where we RAAAGE baby!
(Score: 0) by Anonymous Coward on Friday January 25 2019, @11:03AM
Talk about running code downloaded from strangers. 99% of people do it all day every day. And 99% of them do get fucked.
(Score: 3, Insightful) by rigrig on Friday January 25 2019, @12:43PM (1 child)
Sure, that is a direct quote from TFA. And then the author answers it in the next paragraph
After which he gets to the point
So yes, you can (and should) use dependencies instead of reinventing all wheels yourself, but pay attention to what you are including.
And there it looks like the Node.js ecosystem has a problem, take for example escape-string-regexp [npmjs.com]:
This is a trivial 8-line piece of code [github.com], with a 10:1 meta-data to code ratio...
Given the choice, simply copy-pasting this seems a lot saner than adding it as a dependency.... But a whole bunch of "real" packages do depend on it, so you don't get that choice if you want to include any of those.
Which means you end up with a ton of dependencies, so it's pretty much impossible to vet them all, and you end up with problems like this [lwn.net] or this [lwn.net].
No one remembers the singer.
(Score: 0) by Anonymous Coward on Friday January 25 2019, @01:59PM
True. But main problem is there is no review of the code changes in these dependencies.
Copy/pasting is a problem in itself.
(Score: 2) by DannyB on Friday January 25 2019, @08:36PM (2 children)
It is because modern software has so many features.
My experience as a Java developer . . .
Need a report engine . . . pull in a library, with 15 dependencies.
Need to export PDFs . . . pull in a library with more dependencies.
Need to read and write Microsoft XLSX and DOCX formats . . . more dependencies.
Need to read and write XML?
Need to read and write JSON?
Need to convert JSON to and from Java Beans?
Need to manipulate PDF documents?
Need to manipulate JSON, XML, etc?
Need an ORM system such as Hibernate ?
Need a database upgraded management tool like Liquibase.org ?
And the list goes on.
And we wonder why so many dependencies.
The first step is to admit you have a problem.
At least I manually control which dependencies go into the project, and exactly when they get any version upgrades.
The server will be down for replacement of vacuum tubes, belts, worn parts and lubrication of gears and bearings.
(Score: 0) by Anonymous Coward on Friday January 25 2019, @09:44PM (1 child)
As a Java developer, who is already leary of the number of dependencies used for many projects, I started using nodejs, and was utterly horrified. I was looking at something like 6,000 dependencies from 5,000 different developers for a single project. I'm horrified if I end up with over 100 jars in Java.
If I have ~100 dependencies, I can look up each dependency and determine if I actually need it or not, and even audit it to some degree. With 6,000 dependencies...well, I suppose I have to accept that there is nothing I can do.
(Score: 2) by DannyB on Saturday January 26 2019, @04:09AM
I can relate to that. I have less, I think quite a bit less, than 100 dependencies. I also am able to control which versions of which ones get in. I have the possibility to audit them if I wish. In some cases I have looked at the source code for various reasons, if nothing else, merely to study.
I would think 100 jars in java would be an astonishing number of dependencies. I would expect a few dozen to be "normal" in a program that has a LOT of sophisticated features.
The server will be down for replacement of vacuum tubes, belts, worn parts and lubrication of gears and bearings.
(Score: 0) by Anonymous Coward on Monday January 28 2019, @07:05PM
i think rust's dependency manager, Cargo locks the versions of dependencies and you have to explicitly approve new version installs.