Prefix-Based Environments and Why Activation Is Only Path Surgery

An environment is a claim that a particular set of installed packages is the one a program will see. Every mechanism that enforces this claim works by controlling a search path, and every such mechanism has the same weakness: the path is recorded somewhere as a string, and strings do not survive relocation.

Python’s virtual environment is the clearest case because it does almost nothing beyond path manipulation, and because the consequences of that are visible in ordinary use.

What the venv actually creates

A virtual environment is a directory containing a pyvenv.cfg file, a bin directory (or Scripts on Windows), and a lib tree with a site-packages subdirectory. The interpreter is not copied by default on POSIX systems; bin/python is a symbolic link to the base interpreter. On Windows, where symlinks require elevated privileges or developer mode, the launcher executables are copied instead.

The pyvenv.cfg file records the base interpreter’s location, its version, and whether the environment includes the system site-packages. This file is what makes the directory an environment rather than a folder of packages: at startup, the interpreter looks for pyvenv.cfg beside the running executable and in the directory above it, and if it finds one it sets its prefix to that directory and computes the standard library path relative to the base installation recorded inside.

The effect on sys.path is the whole point. Entries are searched in order, and the order is: the script’s directory or the current directory, then any PYTHONPATH entries, then the standard library, then the environment’s site-packages, then the base installation’s site-packages if system packages were included. Because the environment’s directory precedes the base’s, an installed package in the environment wins over an identically named one in the base. Because both come after the script directory and PYTHONPATH, a file in either of those wins over both. sys.prefix reports the environment; sys.base_prefix reports the base installation; code that compares the two can detect that it is running inside an environment.

That comparison is the only reliable runtime test for whether an environment is in effect, and it is worth checking where the site directory sits in the path rather than only comparing prefixes, because a PYTHONPATH inherited from the shell can put a directory ahead of site-packages.

import sys, sysconfig

def describe():
    site = sysconfig.get_paths()["purelib"]
    ahead = sys.path[: sys.path.index(site)] if site in sys.path else None
    return {
        "venv": sys.prefix != sys.base_prefix,
        "prefix": sys.prefix,
        "site_packages": site,
        "ahead_of_site": ahead,
    }

ahead_of_site is the field that matters in practice. It lists the entries that win over the environment’s packages. A PYTHONPATH entry appears there, and so does the script’s own directory, which the interpreter inserts as the first entry when running a file. The result is an interpreter whose sys.prefix names the environment while its imports come from somewhere else entirely.

Nothing about this is a security boundary. The interpreter is the same binary, running as the same user, with the same access to the filesystem. An environment partitions the import namespace, and that is the extent of it.

Activation is a shell script, not a mode of the interpreter

activate scripts set an environment variable naming the environment and prepend the environment’s bin directory to PATH. That is all. The interpreter does not read the variable during normal operation; it discovers its environment from the location of its own executable, which is why running bin/python directly works without activation.

The practical consequences follow from that detail. Two environments can be active in different shells, and each shell’s python resolves to a different interpreter because PATH differs. A process launched from a shell inherits the PATH at launch and keeps it, so a long-running server started before an environment change continues to use the old one. And a program that invokes python by name without inheriting the shell’s PATH, such as a cron job or a system service, gets the system interpreter regardless of what the developer activated.

This is the most common source of “it works in my shell” divergence, and no amount of environment discipline fixes it, because the mechanism is advisory. The fix is to name the interpreter by absolute path in whatever launches the process.

Entry-point scripts embed the absolute path

Installing a package with a console entry point writes a small script into the environment’s bin directory. On POSIX, that script’s first line is a shebang naming an interpreter, and the interpreter named is the one that performed the installation, written as an absolute path.

That choice makes the script independent of PATH, which is the right decision for a script that must run the right interpreter. It also makes the environment non-relocatable: moving or renaming the directory leaves every console script pointing at a path that no longer exists, and executing one fails with an error about a missing interpreter rather than anything about the package.

Windows has no shebang. The launcher is a small executable that reads the script’s embedded configuration, and it likewise records the interpreter path, so the same relocation problem applies. Copying an environment between machines, or building one inside a container at one path and mounting it at another, breaks both variants identically.

The standard workaround is to invoke the module rather than the script: bin/python -m package.module uses the interpreter at that path and lets it find the module in its own site-packages. This is why documentation for many tools recommends the -m form for anything scripted.

Where a shebang is unavoidable, the portable form is #!/usr/bin/env python3, which defers interpreter selection to PATH. That trades relocatability for correctness: the script will find a Python 3, not necessarily the intended one. The kernel also imposes a length limit on the line, and long absolute paths in deep directory trees can exceed it, producing an execution failure with no useful diagnostic. The kernel parses #! lines from a fixed-size buffer holding the first bytes of the file, so an interpreter path and its single optional argument must fit inside that buffer or be truncated; the size is a compile-time constant that has changed between kernel versions, so the safe rule is that a shebang should stay short.

Relocation, sharing, and the copy-versus-link trade

Copying the interpreter rather than linking it makes an environment self-contained and larger, and it makes upgrades of the base interpreter invisible to the environment. Linking makes the environment small and keeps it tied to the base, so a base interpreter upgrade changes the environment’s behaviour without any package change. Neither is universally correct; the choice depends on whether the environment is a reproducible artifact or a working directory.

Sharing one environment between projects is possible and defeats the purpose. The isolation is per-directory, and installing two projects’ dependencies into one environment means the second install can upgrade a package the first requires. The conflict appears as an import error in one project after working in the other, with no record of which install caused it. Environments are cheap to create and the cost of creating one per project is small compared with the cost of debugging a shared one.

Environments also do not capture non-Python dependencies. A package that links against a system library finds whatever version the operating system provides, and an environment that works on one machine can fail on another with a different library version. This is the boundary where environment isolation stops being sufficient and OS-level isolation begins.

Containers and store paths as stronger isolation

A container image isolates at the filesystem and process level: the packages are installed into an image with its own root filesystem, and the interpreter, shared libraries, and data files are all pinned by the image’s content. A container digest identifies the whole environment, so two hosts running the same image run the same bytes, subject to the kernel they share.

The cost is size and rebuild latency. An image carrying a language runtime and a dependency tree is large, and every dependency change rebuilds the layer that contains it. The benefit is that the environment boundary now includes system libraries, which is the part a virtual environment cannot control.

Nix-style store paths take a different route to the same goal. Every package is built into a path derived from a hash of its inputs, and those paths are recorded in the executables and libraries that depend on them, so relocation is not attempted: the absolute paths are content-derived and therefore identical on any machine with the same inputs. Two versions of a library coexist because their store paths differ, and a program’s reference to one is baked in at build time.

That design removes the relocation problem by removing relocation. It introduces a different cost: the store is large, paths are long and unreadable, and every dependency must be expressed in the build language rather than installed imperatively. The trade is between an environment that is a mutable directory of installed things and one that is a set of immutable, content-addressed closures.

For most projects the practical arrangement is layered: a container or a pinned base image for the system, an environment inside it for the language’s packages, and a lockfile for the package set. Each layer answers a question the layer below cannot, and each layer’s guarantee is only as strong as its weakest recorded input. The record-keeping problem is the one described in Lockfiles as Reproducibility Contracts, and the artifact selection inside each layer depends on the compatibility tags covered in Platform Tags and Binary Compatibility in Distributed Artifacts.

The residual failure mode is worth naming: an environment that is perfectly isolated still runs with the invoking user’s permissions and the host’s kernel. Isolation of the import namespace is not isolation of the process, and conflating the two produces systems that are carefully pinned and still trivially able to read anything the user can read.