We’ll assume that you have at least a basic knowledge of how to
install python.
When creating this post I used Python 3.10 as my system version of python and pip version 21.2. The guidance is for macOS. If using Windows or Unix, you will need to make some adjustments to the installation of system software packages.
Why you might need virtual environments
When you first use Python, you will probably use a globally installed version with
globally installed libraries. You might have installed it yourself or your computer
might come with it pre-installed. MacOS has been shipping with a very old version (2.7) of
Python for a long time. If you installed the developer tools for Mac, it will have
installed Python version 3.X (accessed as python3). This will work well enough if you
are working on a single project or multiple projects with very similar setups.
Problems arise when you work on multiple dissimilar projects. You have one project that requires a particular library and that library in turns requires another library. Consider the case shown below. Program A and program B each require a dependency that in turn requires dependency C. But, they each need a different version of program C and can’t control the version as it’s dependency comes via an intermediate package.
graph TD;
PA[Program A]-->DA[Dependency A V1];
PB[Program B]-->DB[Dependency B V1];
DA-->DC1[Dependency C V1];
DB-->DC2[Dependency C V2];
This is the problem that virtual environments allow us to solve. With a single global environment, we can only have one version of dependency C, which means we cannot simultaneously satisfy the needs of program A and program B. If we can create separate virtual environments for program A and program B, we will be able to have different versions of program C in each virtual environment.
Python version management
Python version management is a related issue. Sometimes you will have a program or library
that doesn’t work with a particular version of Python. As an example, I recently tried
to install the latest version of the rasterio library to discover that it wouldn’t work
with the most recent version of Python (3.10 at time of writing) but would work with the
previous version. This is a very common scenario as library maintainers struggle to update,
test and release their code quickly after a python upgrade.
Python version management allows us to address this problem. Some virtual environment tools address python versioning and some view it as a separate issue to be solved by another tool.
Virtual environment and version management tools
Fortunately, there are some solutions available. Sadly, there isn’t a single universal solution and there are pluses and minuses to the different options. Some of the more popular options are:-
- virtualenv - works with python version 2 but venv is preferred with version 3
- venv - venv became part of python from version 3.3 and is the officially anointed successor to virtualenv
- anaconda / conda - anaconda offers virtual environments but is primarily focused on use of python for data science.
An important point to note is that anaconda has its own package manger, conda with its own repository that is distinct from the main pypi repository - pyenv / pyenv-virtualenv - pyenv is a version manager and it has a plugin, pyenv-virtualenv that manages both pyenv and venv
In this post, we describe the use of the pyenv / pyenv-virtualenv combination.
Installation
macOS Command Line Tools
If you have been using python, you may have already had reason to install the macOS
command line tools. If not, we would recommend that you do that first.
There are two main options. Most developers installxCode,
which is Apple’s IDE for developing macOS and iOS applications. When you install xCode,
it will prompt you to install the command line tools. However, you can install the
command line tools without xCode, by executing this command in a terminal window.
xcode-select --install
Tool Chain
First, let’s install pyenv and pyenv-virtualenv. There are several ways to install these tools, including building from source. But for most macOS users, Homebrew is probably the most convenient option. To install with Homebrew, execute
brew upgrade
brew install pyenv
brew install pyenv-virtualenv
Now we need to make some changes to our environment to pick up some of the paths used by pyenv.
On macOS, the environment is usually managed in .bash_profile in the user’s home directory.
So, go ahead and edit ~/.bashrc and add the following lines:
# pyenv setup
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
export PATH="$PYENV_ROOT/shims:${PATH}"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
For this change to take effect, you’ll need to open a new shell or source ~/.bash_profile.
Setting up Virtual Environments
Now that we have the tools installed, let’s set up some virtual environments. Let’s first look to see what virtual environments are currently available to us. The command and it’s output will look something like this:-
pyenv versions
* system (set by /Users/guest/.pyenv/version)
So far the only environment we have is the default system environment.
So let’s add some environments, but first let’s see what versions of python are available. Executing the following command will list the versions of python that are available remotely.
pyenv install -l
The list is quite long and you will most likely only be interested in the subset that starts with a numeric version number. You can filter the list as follows.
pyenv install -l | grep -E '^[[:blank:]]*[0-9]'
We’ll install version 3.10.2 which is the latest stable release at time of writing.
pyenv install 3.10.2
And, now if we list versions, we should see the newly installed version in the list.
pyenv versions
* system (set by /Users/guest/.pyenv/version)
3.10.2
Now that we have a python version installed under the control of pyenv, let’s set up a virtual environment that uses it. To that we’ll run the following command
pyenv virtualenv 3.10.2 myproj_3_10_2
If you list the available virtual environments, you should see the newly created one.
pyeenv virtualenvs
3.10.2/envs/myproj_3_10_2 (created from /Users/guest/.pyenv/versions/3.10.2)
To use the virtual environment, you need to activate it. When a virtual environment is activated, the command prompt should change to reflect the fact.
pyenv activate myproj_3_10_2
pyenv deactivate does what you would expect.
When the environment is active, running python will run the version associated with
that environment and packages that you install with pip are installed into that
environment only. They won’t be available or visible in the global python space or
in other environments.
We gave the example earlier of being unable to install rasterio with version 3.10 of python. By the time you read this, that issue may be resolved. Nevertheless, consider what we do in that case. We would simply install a python 3.9, using pyenv and create a virtual environment into which we install rasterio.
pyenv install -l | grep -E '^[[:blank:]]*[0-9]'
pyenv install 3.9.10
pyenv virtualenv 3.9.10 rasterproj_3_9_10
pyenv activate rasterproj_3_9_10
pyenv -m pip install rasterio
Summary
Sometimes, you will need to create python programs on your machine that have incompatible dependency graphs. Sometimes, your dependencies will only work with specific versions of python. These problems can be addressed by virtual environments. In the post, we showed one option for managing virtual environments and multiple python versions, using pyenv and its pyenv-virtualenv plugin. You might want to explore some of the other available options yourself.
If you’d like to keep up with new posts, please consider following us.