Reproduceable Workflows

RStudio/Posit Solutions Engineering (Lisa Anders)

RStudio, PBC

Outline

• The data science workflow
• New tools and tricks: Development, Sharing, and Production
• Environment Management Strategies: Using renv and public package manager
• Case Study

Press the S key (or use the Navigation Menu) to show the presentation speaker view (multiplex).

Data Science Workflow

Develop -> Share -> Productionize

“Illustration from Hadley Wickham’s talk”The Joy of Functional Programming (for Data Science).” by Allison Horst

Development: - Writing effective code - Connecting to data sources and systems

Sharing: - Sharing outputs and securing access - Delivering actionable insights

Production: - Code reproduceability - Managing packages and reproducing environments - Collaborating with team members - Implementing operational workflows and scaling

Story segue. Manufacturing monitors project. Transferring code, hosting the output, challenges with numbers being different from other reports.

New tools and tricks: Development

tidymodels - The tidymodels framework is a collection of packages for modeling and machine learning using tidyverse principles.

vetiver - Vetiver provides fluent tooling to version, deploy, and monitor a trained model.

pins - The pins package publishes data, models, and other R objects, making it easy to share them across projects and with your colleagues.

plumber APIs - Plumber allows you to create a web API by merely decorating your existing R source code with roxygen2-like comments.

tidymodels https://www.tidymodels.org/, vetiver https://vetiver.rstudio.com/, pins https://pins.rstudio.com/, plumber https://www.rplumber.io/

Development:

• Writing effective code
• Connecting to data sources and systems

New tools and tricks: Sharing

quarto - A new open-source scientific and technical publishing system that works with R, Python, Julia, Javascript, and many other languages.

shinyuieditor - A visual tool for building the UI portion of a Shiny application that generates clean and human-readable code.

flexdashboard - Flexible and easy to specify row and column-based layouts. Components are intelligently re-sized to fill the browser and adapted for display on mobile devices.

rstudio/posit connect - Hosting for analytics, dashboards, API’s, pinned datasets while working in an enterprise environment.

quarto https://quarto.org/, shinyuieditor https://rstudio.github.io/shinyuieditor/, flexdashboard https://pkgs.rstudio.com/flexdashboard/, Connect https://www.rstudio.com/products/connect/

Sharing:

• Sharing outputs and securing access
• Delivering actionable insights

New tools and tricks: Productionize

Version control - GitHub, Inc. is an Internet hosting service for software development and version control using Git. Other options include Bitbucket, Gitlab, and Azure DevOps.

renv - renv helps manage library paths (and other project-specific state) to help isolate your project’s dependencies.

rstudio/posit package manager - In addition to providing standard mirrors of CRAN, Bioconductor, and PyPI, you can track changes over time or freeze packages to specific versions, to help ensure reproducibility and ease collaboration.

git https://git-scm.com/, renv https://rstudio.github.io/renv, Package Manager https://rstudio.com/products/package-manager/ and Public Package Manager https://packagemanager.rstudio.com/

Production:

• Code reproduceability
• Managing packages and reproducing environments
• Collaborating with team members
• Implementing operational workflows and scaling

Environment Management Strategies

Great data science work should be reproducible and collaborative.

Environments Strategy Maps https://solutions.rstudio.com/environments/reproduce/

Being able to repeat experiments is the foundation of all science. Reproducing work is also critical for business applications: scheduled reporting, team collaboration, project validation.

Environment Management Strategies: - Snapshot and Restore - The snapshot and restore strategy fits when package access is open and users are responsible for reproducibility. - Shared Baseline - The shared baseline strategy fits when administrators or R champions are responsible for creating an environment where less experienced users can easily share and re-run work. - Validated - The validated strategy is similar to the shared baseline strategy. The main difference is the validated strategy targets teams wishing to restrict access to a particular set of packages and teams wishing to approve or audit changes to the package environment.

Example: Using renv, git, and Public Package Manager

Step 1: Use pre-compiled packages

• Go to Public Package Manager

• Click on Get Started -> Setup -> Distribution and select your OS -> Select Latest or Freeze and follow the instructions below the calendar.

• For example:

options(repos = c(REPO_NAME = "https://packagemanager.rstudio.com/all/latest"))

Congratulations! You are now using hosted binary packages, no more struggling against c headers! to a particular set of packages and teams wishing to approve or audit changes to the package environment.

https://support.rstudio.com/hc/en-us/articles/360046703913-FAQ-for-RStudio-Public-Package-Manager

Package Manager supports users for accessing binaries of their packages. UI for searching through packages. Get information about it, crucially any system pre-requisites. If you need to grab an old version Package Manager hosts the old versions. Useful working with renv because if you need to restore a project that is older then likely it will need that older version. RStudio has a sync server where we take changes from CRAN and build the binaries. When you go to install a package we have binaries created. Fast, doesn’t need to be compiled from source. Useful for large packages like arrow (anything with system dependencies) to install quickly since it won’t need to build and prevent build failures.

We evaluate CRAN, Bioconductor, and PyPI each business day (Monday through Friday) and publish new snapshots when updates are available. Then, RStudio Public Package Manager syncs these snapshots daily. For example, suppose a CRAN package gets updated on Saturday. We will publish a new snapshot to the RStudio Package Service sometime on Monday, usually Monday afternoon. Then, the CRAN sync will pick up the change on Tuesday at 12:00 am. Occasionally, if there are very important updates, we generate an extra snapshot to make the updates available sooner. The timing of each snapshot varies based on the number of changes and the number of dependencies involved.

Example: Using renv, git, and Public Package Manager

Step 2: Use environment tracking

options(repos = c(REPO_NAME = "https://packagemanager.rstudio.com/all/latest")) # We've already done this

## Set up a new version controlled R project and install renv
install.packages("renv")
library(renv)

## Initialize your project with renv
renv::init()

## After creating an R script and loading a couple libraries take a snapshot of the project 
renv::snapshot():

## Repeat a couple times, changing the packages being called so that we see something interesting when we run:
renv::history()

## Optionally, revert back to an earlier snapshot:
renv::revert()

renv https://rstudio.github.io/renv, Package Manager https://rstudio.com/products/package-manager/ and Public Package Manager https://packagemanager.rstudio.com/

Example: Using renv, git, and Public Package Manager

Step 3: Easy collaboration

options(repos = c(REPO_NAME = "https://packagemanager.rstudio.com/all/latest")) # We've already done this

## Send a colleague the link to your project on git, they'll restore your environment with:
renv::restore()

renv https://rstudio.github.io/renv, Package Manager https://rstudio.com/products/package-manager/ and Public Package Manager https://packagemanager.rstudio.com/

Congratulations! You now have a project that has all dependencies explicitly tracked. You can come back to this project weeks, years later and will be able to create exactly the environment it needs in order to successfully run with one command, renv::restore().

Case Study

Bike Predict https://solutions.rstudio.com/example/bike_predict/

https://solutions.rstudio.com/example/bike_predict/

• Case Study
  • Bike predict
  • Workbench, Connect, Package Manager
  • Git version control and CI/CD
  • renv package management
  • Vetiver
  • Modularizing the code: API, Pins, Apps

Where to go from here

What They Forgot to Teach You About R : https://rstats.wtf/

Happy Git with R : https://happygitwithr.com/

Get started with renv in the RStudio IDE: https://docs.posit.co/ide/user/ide/guide/environments/r/renv.html

Vetiver

• Vetiver blog announcement post, https://www.rstudio.com/blog/announce-vetiver/
• Vetiver tidyverse announcement post, https://www.tidyverse.org/blog/2022/06/announce-vetiver/

Using Public Package Manager : https://support.rstudio.com/hc/en-us/articles/360046703913-FAQ-for-RStudio-Public-Package-Manager

Interested in our Enterprise Products? Click here