How to Solve the “Dependency Hell” with Python

If you are programming, you know the pain of installing 3rd-party libraries or updating them. Especially if you are working on different platforms. After manual installation of a library, often you find out that a dependency (yet another library used by the one you want to install) need to be installed as well.

Dependency hell

There is another dimension to this problem – different platforms have different package systems. To name a few: Mac OSX uses MacPorts or brew, Ubuntu uses apt and CentOS uses yum. If you are like me, then you might prefer to use Mac OSX at home and one of the Linux distributions on a computational cluster. All those systems/distributions have their own libraries with different versions, compiler versions, versions of dependencies, etc.

Even if you get libraries you need, they might have different versions on different platforms. Because distributions ship their own versions. There is an official term called dependency hell.

Dependency hell is a colloquial term for the frustration of some software users who have installed software packages which have dependencies on specific versions of other software packages. Michael Jang (2006). Linux annoyances for geeks. O’Reilly Media. ISBN 9780596552244. Retrieved 2012-02-16.

 

How nice it would be to have it done automatically? No more configure scripts! Huge time saver!

The idea is to have the same set of libraries across all (Unix-like) systems. Ideally, you want to compile your software on different platforms and you want to have the same version of the required libraries across different platforms.

What libraries are we talking about? If you are programming, then there is good chance you want to install CMake, gcc, Qt, you name it.

Solution

We can break down the installation of a library into 6 actions for each platform that you are using:

  1. Install dependencies (that have to be taken into account),
  2. download package on each platform,
  3. patch (if necessary, apply small compatibility changes),
  4. configure,
  5. compile the library,
  6. install.

There are two ways to implement the solution:

  1. Most obvious approach: do every action manually.
  2. Smart aproach: automate actions with a magic Python script. One program – “dependency hell” problem solved.

Magic Python script(s)

To make use of the smart approach, meaning to automate the installation process, we suggest to write a Python script. Actually, two Python scripts: one will contain the description of the classes (we will discuss it below) and the second one will apply these classes to each library you need to install.

Python script #1

Let’s start with the first Python script that contains a base class called ExternalLibrary().

The most important function in this class is the function Run() which executed the steps 1-6 from the previous section. Note that this function is recursive, because in the step 1 “Installing dependencies” also calls the function Run() for each dependency.

The code snippet below shows the function Run() and the steps 1-6 to install a library.

Let’s us have a look closer at the step number 4: configure. Configuration is done by build systems. Build systems create a make file to compile the library self and its dependencies. Different libraries use different build systems. To name a few: Qt and gcc use gnuconf, CMake and MySQL use CMake on Linux platforms. To make a distinction between different build systems, we have created two derived classes: ExternalLibraryCMake() and ExternalLibraryGNUConf(). The only difference between these two is the Configure() function.

 

Python script #2

The second script is a list of the libraries you need with specific parameters. For example, let’s consider installing CMake 3.4.3. Of course, we have to know

  • where to download the library,
  • setup the source directory,
  • configuration parameters,
  • dependencies.
Prior to configure/compile a library (step 4 and 5) , the environment must be setup in such a way that the library knows where to find the libraries that it depends on. To do that, the Python program creates a simple bash script that sets variables such as LD_LIBRARY_PATH or PATH. Here are step-by-step instructions how to do it.

 

Then the Python script will look like this:

 

As you can see, there are some checks to avoid doing the same operation twice. I.e, the package has already been downloaded or extracted.

A recursive function can easily produce the dependency graph. This graph is a visual check that all dependencies are correctly set-up:

 

part of the Dependency graph for our in-house code

How to handle updates?

A package update can easily be added by adding another small class that depends on the earlier version of the package. Only a couple of functions need to be changed. Once a more recent version become available, for example to install Qt 5.7.0, we have added a simple derived class ExternalLibraryQt570(ExternalLibraryQt560) based on previous library version Qt 5.6.0. The code will look like this:

Parallel compilation

A side effect of using a Python script for automatic installation of 3rd-party libraries is parallel compilation. At the runtime, your Python script can check how many processors are available on your computer. Then your Python script automatically launches a parallel compilation and utilises all CPU power you’ve got.

Summary

Let us summarise the advantages of using the Python scripts to automate the installation of the 3rd-party libraries and solve dependency hell:

  • no manual repetitive work: ones setup in the Python script, it will run automatically on different platforms;
  • updates are easily added;
  • all knowledge about configuration system for each library is in Python script and not lost;
  • launch compilation in parallel automatically utilising multi-core CPUs.

To quote Elsa from the Disney movie Frozen: “Recursivity never bothered me anyway” 🙂

How do you update 3rd-party libraries?

Let us know in the comment box below if you would like to have the code in GitHub.


Get EZNumeric’s future articles in your inbox:

0 comments on “How to Solve the “Dependency Hell” with PythonAdd yours →

Leave a Reply

Your email address will not be published. Required fields are marked *