Create a template for your Jupyter Notebooks
Quick starts for your data science investigations
In a nicely written article, Set Your Jupyter Notebook up Right with this Extension [1], William Koerhsen describes how to use Jupyter notebook extensions to improve productivity by creating a notebook template. This template serves as the starting point for each of your data science journeys. We have all found ourselves typing the same boilerplate code at the start of a new investigation.
The essence of [1] is that you can put a Javascript file into a particular folder used by Jupyter’s nbextensions system, and it will be called by Jupyter each time a new notebook is created. In his article, he creates a new notebook with some basic, common Python boilerplate, and also enables a hook that forces you to rename the notebook from the default “Untitled.ipynb”. He has a companion article, Jupyter Notebook Extensions [2] that shows how to install Jupyter notebook extensions.
There is another interesting article at DrivenData.org, Cookiecutter Data Science [3] that details a helpful organization for Data Science projects.
In my particular case, the notebook template that I use expands on both [1] and [3]. It adds in some code that I know I should include but am often too lazy to add in to a one-off notebook (I’m talking about you, unittest!) Too many times I have started a project that I thought I would only use once, but ended up using multiple times, or expanded the project beyond what I initially envisioned. To avoid an unruly mess, it helps me to have some common structure that I can fill in as needed.
Although all the hard work of figuring out notebook extensions and reverse engineering Jupyter startup code has been done by W Koehrsen, manually editing the Javascript file to add each cell in the template is somewhat tedious. For any template notebook more than a few lines, there is a good chance of introducing a syntax error into the javascript file.
This article (and the code in the corresponding github repository) simplifies this process by (semi-)automatically generating the main.js file from a template Jupyter notebook. It is only semi-automatic as you must manually save your template as a .py file before running the code in notebook-template-generator.ipynb with Jupyter.
Prerequisites
If you don’t yet have Jupyter Extensions, check out this article: [2] or just run the following code in a command prompt:
pip install jupyter_contrib_nbextensions && jupyter contrib nbextensions install --user
and then start a new notebook server and navigate to the extensions tab). Having worked in the security space for a long time, I don't run as an administrator on my machine, so adding '--user' is necessary, at least in my case.
You will need to grab a copy of the “setup” folder from the GitHub repository referenced in [1], which is here. You can find the path where the setup directory needs to go by running the notebook-template-generator notebook; if the path does not exist it will show the path where it expects it. On my Mac using virtualenv, it ends up here:
~/development/Python/Virtualenvs/py37/lib/python3.7/site-packages/jupyter_contrib_nbextensions/nbextensions/setup
This is in my home directory because I used the "--user" option when I installed jupyter nbextensions.
The GitHub repository for this article with notebook-template.ipynb and notebook-setup-generator.ipynb is here.
Running
Open both notebook-template.ipynb and notebook-setup-generator.ipynb with Jupyter. Edit notebook-template.ipynb to contain whatever you would like in a basic Jupyter notebook. Under the File menu, choose “Download as…” and pick “Python (.py)”. On macOS, this will be saved as notebook-template.py.html in the ~/Downloads directory (change get_notebook_template_path() if you want to put it somewhere else).
Next, go to notebook-setup-generator.ipynb and select Run All from the Cell menu. If all goes well, it will ask you if you would like to overwrite the existing setup/main.js file. The next time you create a new Jupyter notebook, it will be populated with a fresh copy of the cells from your version of notebook-template.ipynb.
Additional Notes
I have tested this on macOS 10.14.3 with Python 3.7.2 in a virtual environment. The parser in generate_setup_javascript() is very basic, so it’s possible that complicated notebook-template.ipynb files will not be parsed correctly.
In Koehrsen’s original Javascript function promptName, it checks to see if the new notebook is called “Untitled”; if so, it prompts you to rename it. I have that commented out in the js_postamble string in notebook-setup-generator.ipynb as it is painful when debugging your template, but feel free to re-enable it when you are happy with your template.
References
[1] W. Koehrsen, Set Your Jupyter Notebook up Right with this Extension, (2019), https://towardsdatascience.com/set-your-jupyter-notebook-up-right-with-this-extension-24921838a332
[2] W. Koehrsen, Jupyter Notebook Extensions, (2018), https://towardsdatascience.com/jupyter-notebook-extensions-517fa69d2231
[3] DrivenData.org, Cookiecutter Data Science, http://drivendata.github.io/cookiecutter-data-science/
[4] W. Koehrsen, How to Automatically Import Your Favorite Libraries into IPython or a Jupyter Notebook , https://towardsdatascience.com/how-to-automatically-import-your-favorite-libraries-into-ipython-or-a-jupyter-notebook-9c69d89aa343
[5] Chris Moffitt, Building a Repeatable Data Analysis Process with Jupyter Notebooks (2018), https://pbpython.com/notebook-process.html
