Have you ever tried to use someone else’s repo and run into issues with package installation and breaking package versions? Or tried to work on a really old repository and been foiled while trying to set it up?
Projects have a standards problem - we need to realize that all the work we are doing exist in the framework of a project. By defining the elements of a project we can identify the parts that need to be made transparent and the tools (renv/venv) for making that happen.
What is a project?
- Code, data files, config files, images/assets
- Defined / reproducible environment
- Defined language version
- Defined package versions and requirements
What it’s not:
- Your editor
- The actual packages / repositories
- System dependencies
Reproducible package environments for Python - At a glance
Step 1: Use scanned Python packages
Pip can be already set up for you by your admins (with sudo pip config set --global global.index-url https://pkg.current.posit.team/blocked-python/latest/simple) whenever you use Workbench to use packages from package manager, no configuration needed. Check with:
pip config list
# pip from stackoverflow article: https://stackoverflow.com/questions/51874974/how-can-i-get-pip-to-list-the-repositories-its-using
pip download --no-cache-dir "foo<0" 2>&1 | grep Looking
# uv: https://docs.astral.sh/uv/guides/integration/alternative-indexes/
cat pyproject.tomlAs a user, configure pip to use your repository of choice with:
pip config set global.index-url https://pkg.current.posit.team/blocked-python/latest/simple
pip config set --global global.trusted-host pkg.current.posit.teamAlternatively, for a specific project add these to the top of the requirements.txt file
--index-url https://pkg.current.posit.team/blocked-python/latest/simple
--trusted-host pkg.current.posit.teamStep 2: Use environment tracking
# Activate your virtual environment:
python -m venv .venv
. .venv/bin/activate# Take a snapshot of the environment:
pip freeze > requirements.txtStep 3: Easy collaboration
# Send a colleague the link to your project on git, they'll restore your environment with:
pip install -r requirements.txtPython projects setup
Setup the venv environment:
python -m venv .venv
. .venv/bin/activate
# .venv\Scripts\activate # WindowsUpgrade pip and then install needed packages:
pip install --upgrade pip
python -m pip install --upgrade pip wheel setuptools rsconnect-python
pip install -r requirements.txtRun the application:
shiny run --reload app.pyLeave a virtual environment with:
deactivateJupyter kernels, Quarto, and VS Code
# Create a venv and install packages
python -m venv .venv
. .venv/bin/activate
source .venv/bin/activate
pip install -r requirements.txt
pip install --upgrade pip
# Register as kernel
pip install ipykernel jupyter
python3 -m ipykernel install --name "new_venv" --user
# Check for missing dependencies
python -m pip check
# Automatically uprade all packages from requirements.txt
pip install pip-upgrader
pip-upgrade requirements.txt
pip install -r requirements.txt
# Freeze your requirements
pip freeze > requirements.txt
Requires jupyter extensionFor quarto and vs code:
If you create a virtual environment with venv in the env/ directory as described above, Visual Studio Code should automatically discover that environment when you load a workspace from the environment’s parent directory. Source: https://quarto.org/docs/projects/virtual-environments.html#vs-code
python3 -m venv env
source env/bin/activate
python3 -m pip freeze > requirements.txt
python3 -m pip install -r requirements.txtDeploy
rsconnect-python CLI
rsconnect deploy shiny .Git-backed
Update the code, and then run:
rsconnect write-manifest shiny --overwrite .Commit the new manifest.json file to the git repo along with the code.
Project updates
Create the requirements file:
python -m pip freeze > requirements.txtrsconnect write-manifest shiny .If you are running into deploy issues where there are breaking packages you can edit the requirements file explicitly:
# requirements.txt generated by rsconnect-python on 2022-09-21 14:59:58.865441
streamlit==1.11.0To use a Package Manager repository with a specific project defined by a requirements.txt file, add -i [repositoryURL] to the top of your file, for example:
-i https://packagemanager.posit.co/pypi/latest/simple
pandas
scipyHow to configure a pypi repository globally (using pip.conf): https://docs.posit.co/resources/install-python/#optional-configure-a-pypi-repository
Troubleshooting
Issues with Python not being on path
Set it manually to an installed Python version with:
alias python="/opt/python/3.9.17/bin/python"Set it in your .bashrc on mac or linux so that it is set for your profile every time you log in (typically this is located in the root directory of your home folder):
# add this to your .bashrc
export PATH=/opt/python/3.11.9/bin:$PATHCheck for the available python versions (if typically installed):
ls -ld /opt/python/*