Lisa Anders - questionable.quarto - Python projects set up and maintenance

Have you ever tried to use someone else’s repo and run into issues with package installation and breaking package versions? Or tried to work on a really old repository and been foiled while trying to set it up?

Projects have a standards problem - we need to realize that all the work we are doing exist in the framework of a project. By defining the elements of a project we can identify the parts that need to be made transparent and the tools (renv/venv) for making that happen.

What is a project?

Code, data files, config files, images/assets
Defined / reproducible environment
Defined language version
Defined package versions and requirements

What it’s not:

Your editor
The actual packages / repositories
System dependencies

Reproducible package environments for Python - At a glance

Step 1: Use scanned Python packages

Pip can be already set up for you by your admins (with sudo pip config set --global global.index-url https://pkg.current.posit.team/blocked-python/latest/simple) whenever you use Workbench to use packages from package manager, no configuration needed. Check with:

pip config list

# pip from stackoverflow article: https://stackoverflow.com/questions/51874974/how-can-i-get-pip-to-list-the-repositories-its-using
pip download --no-cache-dir "foo<0" 2>&1 | grep Looking

# uv: https://docs.astral.sh/uv/guides/integration/alternative-indexes/
cat pyproject.toml

As a user, configure pip to use your repository of choice with:

pip config set global.index-url https://pkg.current.posit.team/blocked-python/latest/simple
pip config set --global global.trusted-host pkg.current.posit.team

Alternatively, for a specific project add these to the top of the requirements.txt file

--index-url https://pkg.current.posit.team/blocked-python/latest/simple
--trusted-host pkg.current.posit.team

Step 2: Use environment tracking

# Activate your virtual environment: 
python -m venv .venv
. .venv/bin/activate

# Take a snapshot of the environment: 
pip freeze > requirements.txt

Step 3: Easy collaboration

# Send a colleague the link to your project on git, they'll restore your environment with:
pip install -r requirements.txt

Python projects setup

Setup the venv environment:

python -m venv .venv
. .venv/bin/activate
# .venv\Scripts\activate # Windows

Upgrade pip and then install needed packages:

pip install --upgrade pip
python -m pip install --upgrade pip wheel setuptools rsconnect-python
pip install -r requirements.txt

Run the application:

shiny run --reload app.py

Leave a virtual environment with:

deactivate

Jupyter kernels, Quarto, and VS Code

# Create a venv and install packages
python -m venv .venv
. .venv/bin/activate
source .venv/bin/activate
pip install -r requirements.txt
pip install --upgrade pip

# Register as kernel
pip install ipykernel jupyter
python3 -m ipykernel install --name "new_venv" --user

# Check for missing dependencies
python -m pip check

# Automatically uprade all packages from requirements.txt 
pip install pip-upgrader
pip-upgrade requirements.txt
pip install -r requirements.txt

# Freeze your requirements
pip freeze > requirements.txt

Requires jupyter extension

For quarto and vs code:

If you create a virtual environment with venv in the env/ directory as described above, Visual Studio Code should automatically discover that environment when you load a workspace from the environment’s parent directory. Source: https://quarto.org/docs/projects/virtual-environments.html#vs-code

python3 -m venv env
source env/bin/activate
python3 -m pip freeze > requirements.txt
python3 -m pip install -r requirements.txt

Deploy

rsconnect-python CLI

rsconnect deploy shiny .

Git-backed

Update the code, and then run:

rsconnect write-manifest shiny --overwrite .

Commit the new manifest.json file to the git repo along with the code.

Project updates

Create the requirements file:

python -m pip freeze > requirements.txt

rsconnect write-manifest shiny .

If you are running into deploy issues where there are breaking packages you can edit the requirements file explicitly:

# requirements.txt generated by rsconnect-python on 2022-09-21 14:59:58.865441
streamlit==1.11.0

To use a Package Manager repository with a specific project defined by a requirements.txt file, add -i [repositoryURL] to the top of your file, for example:

-i https://packagemanager.posit.co/pypi/latest/simple
pandas
scipy

How to configure a pypi repository globally (using pip.conf): https://docs.posit.co/resources/install-python/#optional-configure-a-pypi-repository

Troubleshooting

Issues with Python not being on path

Set it manually to an installed Python version with:

alias python="/opt/python/3.9.17/bin/python"

Set it in your .bashrc on mac or linux so that it is set for your profile every time you log in (typically this is located in the root directory of your home folder):

# add this to your .bashrc
export PATH=/opt/python/3.11.9/bin:$PATH

Check for the available python versions (if typically installed):

ls -ld /opt/python/*

Resources

Posit Connect User Guide: Shiny for Python