106 lines
4.8 KiB
Markdown
106 lines
4.8 KiB
Markdown
---
|
|
title: Dependency Hell
|
|
date: 2014-11-20
|
|
---
|
|
|
|
Dependency Hell
|
|
===============
|
|
|
|
A lot of the problem that I have run into when doing development with
|
|
nearly any stack I have used is dependency management. This relatively
|
|
simple-looking problem just becomes such an evil, evil thing to tackle.
|
|
There are several schools of thought to this. The first is that
|
|
dependencies need to be frozen the second you ever see them and are only
|
|
upgraded once in a blue moon when upstream introduces a feature you need
|
|
or has a CVE released. The second is to have competent maintainers
|
|
upstream that follow things like semantic versioning.
|
|
|
|
### Ruby
|
|
|
|
Let's take a look at how the Ruby community solves this problem.
|
|
|
|
One job I had made us need to install **five** versions of the Ruby
|
|
interpreter in order to be compatible with all the different projects
|
|
they wrote. To manage the five versions of the Ruby interpreter, they
|
|
suggested using a widely known tool called
|
|
[rbenv](https://github.com/sstephenson/rbenv).
|
|
|
|
This isn't actually the full list of rubies that job required. I have
|
|
decided not to reveal that out of interest of privacy as well as the
|
|
fact that even Gentoo did not ship a version of gcc old enough to build
|
|
the oldest ruby.
|
|
|
|
After all this, of course, all the dependencies are locked using the gem
|
|
tool and another helper called bundler. It's just a mess.
|
|
|
|
There are also language design features of ruby that really do not help
|
|
with this all that just make simple things like "will this code run or
|
|
not" be determined at runtime. To be fair, Python is the same way, as is
|
|
nearly every other scripting language. In the case of Lua this is
|
|
*beyond vital* because of the fact that Lua is designed to be embedded
|
|
into pretty much anything, with arbitrary globals being set willy-nilly.
|
|
Consequently this is why you can't make an autocomplete for lua without
|
|
executing the code in its preferred environment (unless you really just
|
|
guess based on the requires and other files present in the directory).
|
|
|
|
### Python
|
|
|
|
The Python community has largely copied the ruby pattern for this, but
|
|
they advocate creating local, project-specific prefixes with all of the
|
|
packages/eggs you installed and a list of them instead of compiling an
|
|
entire Python interpreter per project. With the Python 2-\>3 change a
|
|
lot of things did break. This is okay. There was a major version bump.
|
|
Of course compiled modules would need to be redone after a change like
|
|
that. I think the way that Python handles Unicode in version 3 is ideal
|
|
and should be an example for other languages.
|
|
|
|
Virtualenv and pip is not as bad as using bundler and gem for Ruby.
|
|
Virtualenv very clearly makes changes to your environment variables that
|
|
are easy to compare and inspect. This is in contrast to the ruby tools
|
|
that encourage global modifications of your shell and supercede the
|
|
packaged versions of the language interpreter.
|
|
|
|
The sad part is that I see [this pattern of senseless locking of
|
|
versions continuing
|
|
elsewhere](https://github.com/tools/godep) instead of proper
|
|
maintenance of libraries and projects.
|
|
|
|
### Insanity
|
|
|
|
To make matters worse, people suggest you actually embed all the source
|
|
code for every dependency inside the repository. Meaning your commit
|
|
graphs and code line counts are skewed based on the contents of your
|
|
upstream packages instead of just the code you wrote. Admittedly,
|
|
locking dependencies like this does mean that fantastic language level
|
|
tools such as [go
|
|
get](https://golang.org/cmd/go/#hdr-Download_and_install_packages_and_dependencies)
|
|
work again, but overall it is just not worth the pain
|
|
of having to manually merge in patches from upstream (but if you do
|
|
think it is worth the pain contact me, I'm open for contract work)
|
|
making sure to change the file paths to match your changes.
|
|
|
|
### The Solution
|
|
|
|
I believe the solution to all this and something that needs to be a
|
|
wider community effort for users of all programming languages is the use
|
|
of a technique called [semantic
|
|
versioning](http://semver.org/). In
|
|
some lanaguages like Go where the [import paths are based on repository
|
|
paths](https://golang.org/doc/code.html#PackagePaths), this may mean that
|
|
a new major version has a different repository. This is okay. Backward
|
|
compatability is good. After you make a stable (1.0 or whathaveyou)
|
|
release, nothing should be ever taken away or changed in the public API.
|
|
If there needs to be a change in how something in the public API works,
|
|
you must keep backwards compatabilty. As soon as you take away or modify
|
|
something in the public API, you have just made a significant enough
|
|
change worthy of a major release.
|
|
|
|
We need to make semver a de-facto standard in the community instead of
|
|
freezing things and making security patches hard to distribute.
|
|
|
|
Also, use the standard library more. It's there for a reason. It doesn't
|
|
change much so the maintainers are assumed to be sane if you trust the
|
|
stability of the language.
|
|
|
|
This is just my \$0.02.
|