author-image

John Mc Quillan

Python Environments With Pyenv And Virtualenv

We’ll assume that you have at least a basic knowledge of how to install python.

When creating this post I used Python 3.10 as my system version of python and pip version 21.2. The guidance is for macOS. If using Windows or Unix, you will need to make some adjustments to the installation of system software packages.

Why you might need virtual environments

When you first use Python, you will probably use a globally installed version with globally installed libraries. You might have installed it yourself or your computer might come with it pre-installed. MacOS has been shipping with a very old version (2.7) of Python for a long time. If you installed the developer tools for Mac, it will have installed Python version 3.X (accessed as python3). This will work well enough if you are working on a single project or multiple projects with very similar setups.

Problems arise when you work on multiple dissimilar projects. You have one project that requires a particular library and that library in turns requires another library. Consider the case shown below. Program A and program B each require a dependency that in turn requires dependency C. But, they each need a different version of program C and can’t control the version as it’s dependency comes via an intermediate package.

graph TD;
    PA[Program A]-->DA[Dependency A V1];
    PB[Program B]-->DB[Dependency B V1];
    DA-->DC1[Dependency C V1];
    DB-->DC2[Dependency C V2];

This is the problem that virtual environments allow us to solve. With a single global environment, we can only have one version of dependency C, which means we cannot simultaneously satisfy the needs of program A and program B. If we can create separate virtual environments for program A and program B, we will be able to have different versions of program C in each virtual environment.

Python version management

Python version management is a related issue. Sometimes you will have a program or library that doesn’t work with a particular version of Python. As an example, I recently tried to install the latest version of the rasterio library to discover that it wouldn’t work with the most recent version of Python (3.10 at time of writing) but would work with the previous version. This is a very common scenario as library maintainers struggle to update, test and release their code quickly after a python upgrade.

Python version management allows us to address this problem. Some virtual environment tools address python versioning and some view it as a separate issue to be solved by another tool.

Virtual environment and version management tools

Fortunately, there are some solutions available. Sadly, there isn’t a single universal solution and there are pluses and minuses to the different options. Some of the more popular options are:-

  • virtualenv - works with python version 2 but venv is preferred with version 3
  • venv - venv became part of python from version 3.3 and is the officially anointed successor to virtualenv
  • anaconda / conda - anaconda offers virtual environments but is primarily focused on use of python for data science.
    An important point to note is that anaconda has its own package manger, conda with its own repository that is distinct from the main pypi repository
  • pyenv / pyenv-virtualenv - pyenv is a version manager and it has a plugin, pyenv-virtualenv that manages both pyenv and venv

In this post, we describe the use of the pyenv / pyenv-virtualenv combination.

Installation

macOS Command Line Tools

If you have been using python, you may have already had reason to install the macOS command line tools. If not, we would recommend that you do that first. There are two main options. Most developers installxCode, which is Apple’s IDE for developing macOS and iOS applications. When you install xCode, it will prompt you to install the command line tools. However, you can install the command line tools without xCode, by executing this command in a terminal window.

xcode-select --install

Tool Chain

First, let’s install pyenv and pyenv-virtualenv. There are several ways to install these tools, including building from source. But for most macOS users, Homebrew is probably the most convenient option. To install with Homebrew, execute

brew upgrade
brew install pyenv
brew install pyenv-virtualenv

Now we need to make some changes to our environment to pick up some of the paths used by pyenv. On macOS, the environment is usually managed in .bash_profile in the user’s home directory. So, go ahead and edit ~/.bashrc and add the following lines:

# pyenv setup
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
export PATH="$PYENV_ROOT/shims:${PATH}"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

For this change to take effect, you’ll need to open a new shell or source ~/.bash_profile.

Setting up Virtual Environments

Now that we have the tools installed, let’s set up some virtual environments. Let’s first look to see what virtual environments are currently available to us. The command and it’s output will look something like this:-

pyenv versions
* system (set by /Users/guest/.pyenv/version)

So far the only environment we have is the default system environment.

So let’s add some environments, but first let’s see what versions of python are available. Executing the following command will list the versions of python that are available remotely.

pyenv install -l

The list is quite long and you will most likely only be interested in the subset that start with a numeric version number. You can filter the list as follows.

pyenv install -l | grep -E ^[[:blank:]]*[0-9]

We’ll install version 3.10.2 which is the latest stable release at time of writing.

pyenv install 3.10.2

And, now if we list versions, we should see the newly installed version in the list.

pyenv versions
* system (set by /Users/guest/.pyenv/version)
  3.10.2

Now that we have a python version installed under the control of pyenv, let’s set up a virtual environment that uses it. To that we’ll run the following command

pyenv virtualenv 3.10.2 myproj_3_10_2

If you list the available virtual environments, you should see the newly created one.

pyeenv virtualenvs
  3.10.2/envs/myproj_3_10_2 (created from /Users/guest/.pyenv/versions/3.10.2)

To use the virtual environment, you need to activate it. When a virtual environment is activated, the command prompt should change to reflect the fact.

pyenv activate myproj_3_10_2

pyenv deactivate does what you would expect.

When the environment is active, running python will run the version associated with that environment and packages that you install with pip are installed into that environment only. They won’t be available or visible in the global python space or in other environments.

We gave the example earlier of being unable to install rasterio with version 3.10 of python. By the time tou read this, that issue may be resolved. Nevertheless, consider what we do in that case. We would simply install a python 3.9, using pyenv and create a virtual environment into which we install rasterio.

pyenv install -l | grep -E ^[[:blank:]]*[0-9]
pyenv install 3.9.10
pyenv virtualenv 3.9.10 rasterproj_3_9_10
pyenv activate rasterproj_3_9_10
pyenv -m pip install rasterio

Summary

Sometimes, you will need to create python programs on your machine that have incompatible dependency graphs. Sometimes, your dependencies will only work with specific versions of python. These problems can be addressed by virtual environments. In the post, we showed one option for managing virtual environments and multiple python versions, using pyenv and its pyenv-virtualenv plugin. You might want to explore some of the other available options yourself.

If you’d like to keep up with new posts, please consider following us.

SO WHAT DO YOU THINK ?

We love to talk about ideas, innovation, technology and just about anything else. Really, we love to talk.

Contact fathom
a