Murahashi [Matt] Kenichi Posted on May 30 nbwipers: Setup and Troubleshooting # datascience # git # rust # tooling What is nbwipers? nbwipers is a CLI tool that strips outputs and metadata from Jupyter notebooks before git commit. Written in Rust - faster than nbstripout Supports git clean filter Works with .ipynb files Why use it? Jupyter notebooks store cell outputs inside the .ipynb file (JSON). This causes problems: Noisy diffs - output changes pollute every commit Repo size - images and large outputs bloat the repo Security - sensitive data can leak in outputs (API keys, query results) The solution: strip outputs automatically on git add via a clean filter. Why not nbstripout? nbstripout is written in Python. It is slow - git status , git diff , and git add all became noticeably slow on this repo because nbstripout was invoked for every .ipynb file. The main cause is Python startup time. With 100+ notebooks, nbstripout can take 40+ seconds where a Rust-based tool takes ~1 second. Faster alternatives: Tool Language Notes nbstripout-fast Rust Up to 200x faster; no git filter install support nbwipers Rust Inspired by nbstripout-fast; adds git filter + pyproject.toml config nbwipers is essentially nbstripout-fast with better git integration. Switching to nbwipers fixed the slowness. Setup 1. Install felixgwilliams/nbwipers is now in the aqua registry as of v4.517.0 . Using aqua , add to aqua.yaml : packages : - name : felixgwilliams/nbwipers@v0.6.2 Enter fullscreen mode Exit fullscreen mode Then run: aqua install Enter fullscreen mode Exit fullscreen mode 2. Configure git filter Run once per repo (writes to .git/config ): git config filter.nbwipers.clean "nbwipers clean -" git config filter.nbwipers.smudge cat git config filter.nbwipers.required true Enter fullscreen mode Exit fullscreen mode Or edit .git/config directly: [filter "nbwipers"] clean = nbwipers clean - smudge = cat required = true Enter fullscreen mode Exit fullscreen mode required = true makes the co
LIVE