AboutBlogMediaTags

ModuleNotFoundError in Jupyter: A Reproduced Kernel Failure Matrix

Runcell Team,

ModuleNotFoundError in Jupyter: A Reproduced Kernel Failure Matrix

You activate your virtual environment. You run python -c "import pandas" and it works. You start Jupyter, run the same import in a cell, and get:

ModuleNotFoundError: No module named 'pandas'

The usual advice is “activate your venv before starting Jupyter” or “install ipykernel”. Both are sometimes right and neither explains anything, which is why this error keeps coming back.

This page is a failure matrix. Four virtual environments were built from scratch and nine notebook runs were executed against them — eight that started a kernel and one that could not — with each cell’s real output pasted below. The scripts that produce every line are at the bottom, so you can re-run the whole thing.

The short version: for the kernel you are almost certainly using, PATH does not decide which Python your notebook runs. Two other things do, and which one applies depends on one string in a JSON file you have probably never opened.

The decision table

If you only want the fix, find your symptom here and jump to the section.

SymptomCauseFix
Import fails in the notebook, works in the terminalThe kernel is running a different interpreter than your terminal. Which of the two causes below applies is not something the kernel’s name in the menu can tell you — run the audit script or the two-line diagnostic cell first—
…and the audit says argv[0] is a bare name, rewritten to the server’s sys.executableThe spec ipykernel ships has a bare argv[0], so the kernel follows whichever interpreter runs jupyter — section 2Install Jupyter into the project env (fix B) or register the env as its own kernel (fix A)
…and the audit says argv[0] is an absolute pathA kernelspec someone registered points at a different env than you thinkFix A against the right env, then select it
!pip install x says “successfully installed” and the next cell still fails!pip runs the shell’s pip, not the kernel’s — section 4Use %pip install x
”Kernel died” / FileNotFoundError on startThe kernelspec points at a venv that has been deleted or moved — section 6jupyter kernelspec remove NAME, then re-register
Switching the PATH or re-activating the venv changes nothingExpected. PATH is not the variable — section 3See the two fixes in section 5

The lab

Four virtual environments, all on the same interpreter so that the Python version is never the confounder. Only package placement differs.

EnvpandasipykerneljupyterlabRole
launcher—yesyesthe env that happens to have Jupyter
project-ayesyes—registered as a kernel
project-byes——the env you activated
project-cyesyesyesthe env that fixes it

Verbatim from run-matrix.sh:

==================================================================
0. VERSIONS
==================================================================
python (launcher) 3.12.7
jupyterlab        4.6.3
nbconvert         7.17.1
jupyter_client    8.10.0
platform          macOS-26.5.2-arm64-arm-64bit
ipykernel         7.3.0

==================================================================
1. THE FOUR ENVIRONMENTS
==================================================================
env        python   pandas     ipykernel   jupyterlab
launcher   3.12.7   -          7.3.0       4.6.3
project-a  3.12.7   3.0.5      7.3.0       -
project-b  3.12.7   3.0.5      -           -
project-c  3.12.7   3.0.5      7.3.0       4.6.3

In a terminal, project-b imports pandas without complaint:
  project-b/bin/python -> 3.0.5

Kernels Jupyter can see:
Available kernels:
  project-a    $LAB/jupyter-data/kernels/project-a
  python3      $LAB/launcher/share/jupyter/kernels/python3

project-b is the situation almost everyone is in: the env you work in has your packages, has no Jupyter of its own, and was never registered as a kernel.

Paths in the pasted output have the lab directory rewritten to $LAB by the script’s own norm filter, so the lines stay readable. Nothing else in any pasted block is edited.

The two kinds of kernelspec

A kernel is not a Python environment. A kernel is a JSON file that says how to start a process. There are two ways that file comes into existence, and they behave differently.

(a) The default spec that ships inside the ipykernel wheel. You get it for free the moment ipykernel is installed anywhere; you never created it.

{
 "argv": [
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3 (ipykernel)",
 "language": "python",
 "metadata": {
  "debugger": true,
  "supported_encryption": "curve"
 },
 "kernel_protocol_version": "5.5"
}

(b) A spec written by python -m ipykernel install.

{
 "argv": [
  "$LAB/project-a/bin/python",
  "-Xfrozen_modules=off",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python (project-a)",
 "language": "python",
 "metadata": {
  "debugger": true,
  "supported_encryption": "curve"
 },
 "kernel_protocol_version": "5.5"
}

argv[0] is a bare name in (a) and an absolute path in (b). That is the whole difference, and it is the reason the same error has two different fixes.

A bare python is not resolved through PATH. jupyter_client intercepts three exact names before the process is ever launched:

jupyter_client resolves the bare name -- manager.py, format_kernel_cmd():
391:        if cmd and cmd[0] in {
392-            "python",
393-            "python%i" % sys.version_info[0],
394-            "python%i.%i" % sys.version_info[:2],
395-        }:
396-            # executable is 'python' or 'python3', use sys.executable.
397-            # These will typically be the same,
398-            # but if the current process is in an env
399-            # and has been launched by abspath without
400-            # activating the env, python on PATH may not be sys.executable,
401-            # but it should be.
402-            cmd[0] = sys.executable
403-

sys.executable there is the interpreter of whichever process is starting the kernel. In this lab that process is jupyter nbconvert --execute; when you click Run in JupyterLab it is the jupyter lab server process. Either way it is the process that ran the jupyter command, and it is not the shell’s PATH. So:

With the default python3 kernelspec, your notebook runs on the same interpreter your jupyter command runs on. Not the one you activated. Not the first one on PATH.

What was measured and what is inferred. The five runs below were executed with nbconvert, so the nbconvert case is measured. The JupyterLab case is the same KernelManager.format_kernel_cmd() code path — that is one function in jupyter_client, which both go through — but it was not executed here.

Note the exact membership test: only python, python3 and python3.12 get rewritten — that last one is built from the starting process’s own version. A kernelspec that says python3.9 while that process runs 3.12 matches none of them, is left alone, and then really is resolved through PATH by the OS. That case exists, but it is not the one you get by default: the shipped spec says plain python in both ipykernel versions checked here (6.29.5 and 7.3.0). The audit script below reports which of the three situations each of your kernels is in.

This is not new behaviour and not a quirk of one release. The bare-name argv[0] is shipped by ipykernel 6.29.5 and 7.3.0 alike, and the rewrite is present in jupyter_client 8.0.3 and 8.10.0 — the two versions available on the test machine. It is just invisible, because nothing in the UI tells you which interpreter is behind “Python 3 (ipykernel)”.

The matrix

Five notebook executions. Every one runs this notebook:

import sys
print("sys.executable:", sys.executable)
print("sys.prefix    :", sys.prefix)
import pandas as pd
print("pandas", pd.__version__)

The only things that change between runs are the kernel name, the directory at the head of PATH, and which jupyter binary is invoked.

==================================================================
3. THE MATRIX
==================================================================
--- A  kernel=python3  PATH head=$LAB/launcher/bin  jupyter=$LAB/launcher/bin/jupyter
sys.executable: $LAB/launcher/bin/python3.12
sys.prefix    : $LAB/launcher
ModuleNotFoundError: No module named 'pandas'

--- B  kernel=python3  PATH head=$LAB/project-b/bin  jupyter=$LAB/launcher/bin/jupyter
sys.executable: $LAB/launcher/bin/python3.12
sys.prefix    : $LAB/launcher
ModuleNotFoundError: No module named 'pandas'

--- C  kernel=project-a  PATH head=$LAB/launcher/bin  jupyter=$LAB/launcher/bin/jupyter
sys.executable: $LAB/project-a/bin/python
sys.prefix    : $LAB/project-a
pandas 3.0.5

--- D  kernel=project-a  PATH head=$LAB/project-b/bin  jupyter=$LAB/launcher/bin/jupyter
sys.executable: $LAB/project-a/bin/python
sys.prefix    : $LAB/project-a
pandas 3.0.5

--- E  kernel=python3  PATH head=$LAB/launcher/bin  jupyter=$LAB/project-c/bin/jupyter
sys.executable: $LAB/project-c/bin/python3.12
sys.prefix    : $LAB/project-c
pandas 3.0.5

Read it as three pairs:

ComparisonWhat changedResult
A vs BPATH head moved to an env that does have pandasIdentical failure. PATH is not the variable.
C vs DSame PATH change, on a registered kernelIdentical success. PATH is not the variable here either.
A vs ESame kernelspec name, same PATH — only the jupyter binary differsFailure becomes success. The server’s interpreter is the variable.

A vs E is the one worth memorising. Nothing about the notebook changed, nothing about PATH changed, and the kernel is called python3 in both cases. The only difference is which environment the jupyter command came from — and that flipped the outcome.

To be precise about what “the same kernel” means here: these are two different kernelspec files that happen to share the name python3, because each env’s Jupyter finds the spec shipped inside its own sys.prefix. Both files contain the identical bare "python", so both go through the rewrite — and both land on their own server’s interpreter. That is the point: the name in the kernel menu tells you nothing.

This is why “activate your venv first” works sometimes: activating puts the venv’s bin first on PATH, so if the venv contains its own jupyter, you launch a different server and get a different kernel interpreter. If it does not contain one — which is the project-b case, and the common one — activation changes nothing at all. Case B is that non-fix, executed.

What !pip and %pip actually run

The other half of the confusion: you hit the error, run !pip install pandas in a cell, watch pip report success, re-run the import, and get the same error.

! runs a shell command. The shell’s pip is whatever PATH says — which, per the section above, has nothing to do with the kernel’s interpreter.

==================================================================
4. WHAT !pip AND %pip ACTUALLY RUN
==================================================================
--- F1  kernel=project-a  PATH head=$LAB/project-b/bin  jupyter=$LAB/launcher/bin/jupyter
kernel sys.executable : $LAB/project-a/bin/python
!pip  resolves to     : $LAB/project-b/bin/pip
%pip  resolves to     : $LAB/project-a/bin/python -m pip

--- F2  kernel=project-a  PATH head=$LAB/launcher/bin  jupyter=$LAB/launcher/bin/jupyter
kernel sys.executable : $LAB/project-a/bin/python
!pip  resolves to     : $LAB/launcher/bin/pip
%pip  resolves to     : $LAB/project-a/bin/python -m pip

Same kernel in both runs. The !pip line is measured — the cell calls shutil.which("pip"), which is exactly what the shell would resolve — and it points at two different environments, neither of which is the kernel’s.

The %pip line is not a measurement; it is what the magic is defined to do. IPython 9.17.1, IPython/core/magics/packaging.py, lines 93–105:

    @line_magic
    def pip(self, line):
        """Run the pip package manager within the current kernel.

        Usage:
          %pip install [pkgs]
        """
        python = sys.executable
        if sys.platform == "win32":
            python = '"' + python + '"'
        else:
            python = shlex.quote(python)

        self.shell.system(" ".join([python, "-m", "pip", line]))

sys.executable inside the kernel process is that kernel’s own interpreter, so %pip always installs into the interpreter the cell is running on. (That is not the same as “always what you wanted” — if you are on the wrong kernel, %pip faithfully installs into the wrong kernel. It removes one variable, not two.)

Prefer %pip over !pip. The two lines look almost identical, and only %pip is tied to the kernel by construction. !pip is not always wrong — it is right whenever the shell’s pip happens to belong to the kernel’s environment — but you have to know that, and the whole point of this page is that you usually do not. If you want the shell form anyway, write it as !"{sys.executable}" -m pip install x — with the quotes. Without them an interpreter path containing a space (~/My Project/.venv/bin/python) is split into two shell arguments and the command fails. That is exactly why the implementation above runs the path through shlex.quote().

%conda exists for the same reason, though it works differently: the same source file resolves the conda executable from CONDA_EXE or from the kernel’s own sys.prefix, and raises if the kernel is not in a conda environment at all. This lab contains no conda environment, so that path was read in the source but not executed here.

The two fixes

Both of these make the interpreter explicit. Pick based on how many environments you juggle.

Fix A: register the env as its own kernel

Best when you switch between several projects in one Jupyter server.

# from inside the environment that has your packages
python -m pip install ipykernel
python -m ipykernel install --user --name project-a --display-name "Python (project-a)"

Then switch the notebook to it, which is the step that actually changes anything. Registering adds a kernel to the menu; it does not move your open notebook onto it, and a notebook left on “Python 3 (ipykernel)” fails exactly as before. In JupyterLab that is Kernel → Change Kernel → Python (project-a), then restart. From the command line it is explicit:

jupyter nbconvert --to notebook --execute \
    --ExecutePreprocessor.kernel_name=project-a analysis.ipynb

That flag is how cases C and D in the matrix above select this kernel.

Registration writes an absolute argv[0], which is why C and D succeed regardless of PATH. The cost is that the path is frozen: move or delete the venv and the kernel breaks permanently — see section 6.

Fix B: put Jupyter in the project env

Best when one project equals one environment. No kernelspec to manage and nothing to go stale.

python -m pip install jupyterlab
python -m jupyterlab          # or: python -m jupyter lab

Invoking it as python -m rather than as the bare jupyter command removes the last ambiguity: the server is guaranteed to run on the interpreter you just named, so the default python3 kernelspec resolves to that same interpreter. Case E is this fix, executed — note that it succeeded with launcher/bin still first on PATH.

This does not rescue a notebook that is pinned to a named kernel. A .ipynb stores the kernel it was saved with in its own metadata, and that wins. Case E works because the notebook asks for python3, the default. Here is the same notebook with its metadata set to project-a, run by project-c’s Jupyter with no kernel flag at all:

==================================================================
5. A NOTEBOOK PINNED TO A KERNEL IGNORES WHICH JUPYTER YOU LAUNCH
==================================================================
pinned.ipynb asks for kernel 'project-a' in its own metadata.
Launched from project-c's jupyter, with NO --kernel_name flag:
sys.executable: $LAB/project-a/bin/python
sys.prefix    : $LAB/project-a
pandas 3.0.5

It ran project-a, not project-c. Fix B changes what new notebooks get by default; an existing notebook keeps asking for whatever it was saved with. Switch it back to the default kernel, or use fix A and point it at the right one. Then confirm with the diagnostic cell — not with the kernel’s display name.

A kernelspec that points at an env that is gone

Registered kernels rot. The path is frozen at registration time, so deleting or renaming the venv leaves a kernel that still appears in the menu and cannot start.

==================================================================
6. A KERNELSPEC THAT POINTS AT AN ENV THAT IS GONE
==================================================================
ghost kernel argv[0] -> $LAB/project-deleted/bin/python
--- executing a notebook on it:
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/opt/homebrew/Cellar/python@3.12/3.12.7_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '$LAB/project-deleted/bin/python'

In JupyterLab the same condition is usually reported as the kernel failing to start or dying immediately; the FileNotFoundError above is what nbconvert --execute surfaces on the command line, where the traceback is not swallowed. Clean it up with:

jupyter kernelspec list
jupyter kernelspec remove ghost

How often this happens is not something one machine can tell you, but it is easy to accumulate without noticing: running the audit script below against a second, unrelated Jupyter installation on the same machine found 2 of its 8 registered kernels pointing at interpreters that no longer exist, both left behind by deleted temporary virtual environments. Unlike every other output on this page, that run is not reproduced here — its output is a list of someone’s project directories. One anecdote, offered as a reason to run the script rather than as a rate.

The audit script

A matrix on someone else’s machine does not tell you what is wrong on yours. This script does: for every Python kernel Jupyter can see, it resolves argv[0] the same way jupyter_client does, checks that the interpreter still exists, and reports whether that interpreter can find the packages you name. Kernels it cannot check — non-Python languages, and specs whose argv[0] is a wrapper script or a template rather than a plain interpreter — are listed and labelled, not silently skipped.

Run it with the same Python that runs your Jupyter server — if you are not sure which that is, python -m jupyter --version from the env you suspect will tell you whether that env has Jupyter at all.

python audit-kernels.py pandas

Two limitations worth knowing before you trust its output:

#!/usr/bin/env python3
"""Report, for every kernel Jupyter can see, which interpreter it will actually start."""
import os
import re
import shutil
import subprocess
import sys
 
from jupyter_client.kernelspec import KernelSpecManager
 
# find_spec() locates a module without executing it -- except that for a dotted name it
# imports the parent package, which can raise anything at all and can print to stdout.
# So: catch BaseException (deliberate -- this is a read-only diagnostic, and one odd
# argument must never cost you the results for every other kernel), and tag the two
# result lines so a parent package's chatter cannot be mistaken for a result.
PROBE = (
    "import sys, importlib.util as u\n"
    "def has(m):\n"
    "    try:\n"
    "        return 'yes' if u.find_spec(m) else 'NO'\n"
    "    except BaseException:\n"
    "        return 'NO'\n"
    "env = '%s %s' % (sys.version.split()[0], sys.prefix)\n"
    "pkgs = ' '.join('%s=%s' % (m, has(m)) for m in sys.argv[1:])\n"
    "sys.stdout.write('\\n@@ENV %s\\n@@PKG %s\\n' % (env, pkgs))\n"
)
 
 
def _tagged(stdout, tag):
    """Pull one tagged line out of the probe's stdout, ignoring anything else printed."""
    for line in stdout.splitlines():
        if line.startswith(tag + " "):
            return line[len(tag) + 1:]
    return None
 
 
def resolve(argv):
    """Mirror jupyter_client.manager.KernelManager.format_kernel_cmd."""
    exe = argv[0]
    aliases = {"python", "python%i" % sys.version_info[0],
               "python%i.%i" % sys.version_info[:2]}
    if exe in aliases:
        return sys.executable, "rewritten to the server's sys.executable"
    return exe, "taken from the kernelspec"
 
 
_PYTHON_EXE = re.compile(r"^python(w)?([23](\.\d+)?)?$")
 
 
def _looks_like_python(exe):
    """Reject wrappers and launchers we cannot probe by running `exe -c ...`.
 
    Matches python, pythonw, python3, python3.12 and the .exe forms, and nothing else —
    `python3-wrapper` and `python-launcher` are deliberately rejected.
    """
    base = os.path.basename(exe).lower()
    if base.endswith(".exe"):          # Windows kernelspecs point at python.exe
        base = base[:-4]
    return bool(_PYTHON_EXE.match(base))
 
 
def main(packages):
    print("Jupyter server interpreter: %s\n" % sys.executable)
    for name, entry in sorted(KernelSpecManager().get_all_specs().items()):
        spec = entry["spec"]
        argv = spec.get("argv", [])
        print("kernel %r  (%s)" % (name, spec.get("display_name", "?")))
        print("  resource dir : %s" % entry["resource_dir"])
        print("  argv[0]      : %s" % (argv[0] if argv else "<empty>"))
        if not argv:
            print("  -> broken kernelspec: empty argv\n")
            continue
        if spec.get("language") != "python":
            print("  -> not a Python kernel (language=%r); not checked\n"
                  % spec.get("language"))
            continue
        if any("{" in a for a in argv[:1]) or not _looks_like_python(argv[0]):
            print("  -> argv[0] is not a plain interpreter path (wrapper or template);"
                  " not checked\n")
            continue
        exe, how = resolve(argv)
        print("  interpreter  : %s  (%s)" % (exe, how))
        if not os.path.isabs(exe):
            found = shutil.which(exe)
            if found is None:
                print("  -> %s is not on PATH here; this kernel cannot start\n" % exe)
                continue
            print("  note         : resolved through PATH to %s;"
                  " a different PATH at launch gives a different answer" % found)
            exe = found
        if not os.path.exists(exe):
            print("  -> MISSING: this kernel cannot start\n")
            continue
        try:
            out = subprocess.run([exe, "-c", PROBE, *packages],
                                 capture_output=True, text=True, timeout=60)
        except subprocess.TimeoutExpired:
            print("  -> probe timed out after 60s; this interpreter is not answering\n")
            continue
        except OSError as err:
            print("  -> could not run it: %s\n" % err)
            continue
        env = _tagged(out.stdout, "@@ENV")
        if out.returncode != 0 or env is None:
            last = out.stderr.strip().splitlines()[-1:] or ["no output"]
            print("  -> probe failed: %s\n" % last[0])
            continue
        version, prefix = env.split(maxsplit=1)
        print("  version      : %s" % version)
        print("  sys.prefix   : %s" % prefix)
        if packages:
            print("  packages     : %s" % (_tagged(out.stdout, "@@PKG") or ""))
        print()
 
if __name__ == "__main__":
    main(sys.argv[1:])

Against the lab, it prints exactly the three states this page is about — one dead kernel, one correct kernel, and one kernel silently pointing at the server’s own environment:

Jupyter server interpreter: $LAB/launcher/bin/python

kernel 'ghost'  (Python (ghost))
  resource dir : $LAB/jupyter-data/kernels/ghost
  argv[0]      : $LAB/project-deleted/bin/python
  interpreter  : $LAB/project-deleted/bin/python  (taken from the kernelspec)
  -> MISSING: this kernel cannot start

kernel 'project-a'  (Python (project-a))
  resource dir : $LAB/jupyter-data/kernels/project-a
  argv[0]      : $LAB/project-a/bin/python
  interpreter  : $LAB/project-a/bin/python  (taken from the kernelspec)
  version      : 3.12.7
  sys.prefix   : $LAB/project-a
  packages     : pandas=yes

kernel 'python3'  (Python 3 (ipykernel))
  resource dir : $LAB/launcher/share/jupyter/kernels/python3
  argv[0]      : python
  interpreter  : $LAB/launcher/bin/python  (rewritten to the server's sys.executable)
  version      : 3.12.7
  sys.prefix   : $LAB/launcher
  packages     : pandas=NO

The last block is the bug, stated plainly: the kernel called “Python 3” is the launcher environment, and launcher does not have pandas.

If you would rather not install anything, the two-line version of this check, pasted into a cell, is enough to tell you which environment you are in:

import sys
print(sys.executable)
print(sys.prefix)

Compare that against which python in the terminal where the import worked. If they differ, you have found your problem and the rest of this page is the fix.

When the environment is the thing you are trying to avoid

Two situations where the right move is not to fix the kernel at all:

Limits of this page

Reproducing this

Two scripts, both below in full — they are the same files this page’s output came from, not a paraphrase. setup.sh builds the four environments with pinned versions; run-matrix.sh prints sections 0 through 6 of this page, in that order.

export LAB="$HOME/kernel-lab" PY=python3.12
./setup.sh
./run-matrix.sh

setup.sh:

#!/usr/bin/env bash
# Builds the four environments the matrix runs against. Idempotent: delete $LAB and
# re-run to rebuild from scratch.
#
#   launcher/   jupyterlab + nbconvert, NO pandas   <- the env that "has Jupyter"
#   project-a/  pandas + ipykernel, registered as a kernel
#   project-b/  pandas only, never registered       <- the env you activated
#   project-c/  pandas + jupyterlab + nbconvert     <- the env that fixes it
set -euo pipefail
LAB="${LAB:?set LAB to the lab directory}"
PY="${PY:-python3.12}"
mkdir -p "$LAB"
cd "$LAB"
 
for e in launcher project-a project-b project-c; do
  [ -d "$e" ] || "$PY" -m venv "$e"
done
 
# jupyter_client, ipykernel and ipython are pinned in every env that runs a kernel: the
# article quotes jupyter_client's source, ipykernel's shipped kernelspec and IPython's
# %pip implementation, so a floating version in any of them would change what the article
# is about. project-a is included because F1/F2 execute on its kernel.
PINS_JUPYTER='jupyterlab==4.6.3 nbconvert==7.17.1 jupyter_client==8.10.0 ipykernel==7.3.0 ipython==9.17.1'
 
./launcher/bin/pip  install --quiet --disable-pip-version-check $PINS_JUPYTER
./project-a/bin/pip install --quiet --disable-pip-version-check "pandas==3.0.5" "ipykernel==7.3.0" "jupyter_client==8.10.0" "ipython==9.17.1"
./project-b/bin/pip install --quiet --disable-pip-version-check "pandas==3.0.5"
./project-c/bin/pip install --quiet --disable-pip-version-check "pandas==3.0.5" $PINS_JUPYTER
 
export JUPYTER_DATA_DIR="$LAB/jupyter-data"
export JUPYTER_CONFIG_DIR="$LAB/jupyter-config"
rm -rf "$JUPYTER_DATA_DIR/kernels"
./project-a/bin/python -m ipykernel install --user --name project-a \
    --display-name "Python (project-a)" >/dev/null
 
cat > nb.ipynb <<'NB'
{"cells":[
 {"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],
  "source":["import sys\n","print(\"sys.executable:\", sys.executable)\n","print(\"sys.prefix    :\", sys.prefix)"]},
 {"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],
  "source":["import pandas as pd\n","print(\"pandas\", pd.__version__)"]}],
 "metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},
 "language_info":{"name":"python"}},"nbformat":4,"nbformat_minor":5}
NB
 
cat > pip-nb.ipynb <<'NB'
{"cells":[
 {"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],
  "source":["import sys, shutil\n","print(\"kernel sys.executable :\", sys.executable)\n","print(\"!pip  resolves to     :\", shutil.which(\"pip\"))\n","print(\"%pip  resolves to     :\", sys.executable + \" -m pip\")"]}],
 "metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},
 "language_info":{"name":"python"}},"nbformat":4,"nbformat_minor":5}
NB
# Same notebook, but with its kernelspec metadata pinned to the registered kernel.
LAB="$LAB" "$LAB/launcher/bin/python" - <<'PYNB'
import json, os
lab = os.environ["LAB"]
nb = json.load(open(os.path.join(lab, "nb.ipynb")))
nb["metadata"]["kernelspec"] = {"display_name": "Python (project-a)",
                                "language": "python", "name": "project-a"}
json.dump(nb, open(os.path.join(lab, "pinned.ipynb"), "w"))
PYNB
 
echo "lab ready in $LAB"

run-matrix.sh:

#!/usr/bin/env bash
# Reproduces the Jupyter "ModuleNotFoundError in the notebook, works in the terminal"
# matrix. Every block the article quotes AS LAB OUTPUT comes from this script. The
# article also quotes things this script does not produce -- the notebook source, the
# excerpts from jupyter_client and IPython, the repair commands -- and says where those
# come from inline.
#
# Two things the article states are NOT produced here, and it says so inline:
#   - the audit script's own output      -> audit-kernels.py, committed as audit-output.txt
#   - "2 of 8 kernels dead" on a second machine -> not committed (private paths)
#
# Run setup.sh first.
#
# Printed paths have the lab root rewritten to $LAB so the output stays legible.
# Nothing else in the output is edited.
set -uo pipefail
LAB="${LAB:-$(cd "$(dirname "$0")" && pwd)}"
cd "$LAB"
export JUPYTER_DATA_DIR="$LAB/jupyter-data"
export JUPYTER_CONFIG_DIR="$LAB/jupyter-config"
 
norm() { sed -e "s#/private$LAB#\$LAB#g" -e "s#$LAB#\$LAB#g"; }
 
show_outputs() {
  "$LAB/launcher/bin/python" - "$1" <<'PY'
import json, sys
nb = json.load(open(sys.argv[1]))
for c in nb["cells"]:
    for o in c.get("outputs", []):
        if o.get("output_type") == "stream":
            print("".join(o["text"]).rstrip())
        elif o.get("output_type") == "error":
            print(f'{o["ename"]}: {o["evalue"]}')
PY
}
 
# Deletes the output first and refuses to print anything if nbconvert did not write a
# fresh one. Without this a failed rerun would silently show the previous run's result,
# which is the one way this script could tell a comfortable lie.
run_nb() {   # run_nb <label> <notebook> <kernel> <dir-first-on-PATH> <jupyter-binary>
  local out="$LAB/out-$1.ipynb"
  echo "--- $1  kernel=$3  PATH head=$(echo "$4" | norm)  jupyter=$(echo "$5" | norm)"
  rm -f "$out"
  PATH="$4:$PATH" "$5" nbconvert --to notebook --execute --allow-errors \
      --ExecutePreprocessor.kernel_name="$3" --output "out-$1.ipynb" "$2" >/dev/null 2>&1
  local rc=$?
  if [ ! -f "$out" ]; then
    echo "  !! nbconvert wrote no output (exit $rc). This case did not run."
    echo
    return 1
  fi
  show_outputs "$out" | norm
  echo
}
 
echo "=================================================================="
echo "0. VERSIONS"
echo "=================================================================="
"$LAB/launcher/bin/python" - <<'PY'
import sys, platform, jupyter_client, nbconvert, jupyterlab
print("python (launcher)", sys.version.split()[0])
print("jupyterlab       ", jupyterlab.__version__)
print("nbconvert        ", nbconvert.__version__)
print("jupyter_client   ", jupyter_client.__version__)
print("platform         ", platform.platform())
PY
"$LAB/project-a/bin/python" -c 'import ipykernel; print("ipykernel        ", ipykernel.__version__)'
echo
 
echo "=================================================================="
echo "1. THE FOUR ENVIRONMENTS"
echo "=================================================================="
printf '%-10s %-8s %-10s %-11s %s\n' env python pandas ipykernel jupyterlab
for e in launcher project-a project-b project-c; do
  printf '%-10s %-8s %-10s %-11s %s\n' "$e" \
    "$("$LAB/$e/bin/python" -c 'import sys;print(".".join(map(str,sys.version_info[:3])))')" \
    "$("$LAB/$e/bin/python" -c 'import pandas;print(pandas.__version__)' 2>/dev/null || echo -)" \
    "$("$LAB/$e/bin/python" -c 'import ipykernel;print(ipykernel.__version__)' 2>/dev/null || echo -)" \
    "$("$LAB/$e/bin/python" -c 'import jupyterlab;print(jupyterlab.__version__)' 2>/dev/null || echo -)"
done
echo
echo "In a terminal, project-b imports pandas without complaint:"
"$LAB/project-b/bin/python" -c 'import pandas; print("  project-b/bin/python ->", pandas.__version__)'
echo
echo "Kernels Jupyter can see:"
"$LAB/launcher/bin/jupyter" kernelspec list 2>&1 | norm
echo
 
echo "=================================================================="
echo "2. THE TWO KINDS OF KERNELSPEC"
echo "=================================================================="
echo "(a) the default spec that ships inside the ipykernel wheel:"
norm < "$LAB/launcher/share/jupyter/kernels/python3/kernel.json"
echo
echo "(b) a spec written by 'python -m ipykernel install':"
norm < "$LAB/jupyter-data/kernels/project-a/kernel.json"
echo
echo "argv[0] is a bare name in (a), an absolute path in (b)."
echo "jupyter_client resolves the bare name -- manager.py, format_kernel_cmd():"
grep -n -A 12 'if cmd and cmd\[0\] in {' \
  "$LAB/launcher/lib/python3.12/site-packages/jupyter_client/manager.py" | sed -n '1,13p'
echo
 
echo "=================================================================="
echo "3. THE MATRIX"
echo "=================================================================="
run_nb A nb.ipynb python3   "$LAB/launcher/bin"  "$LAB/launcher/bin/jupyter"
run_nb B nb.ipynb python3   "$LAB/project-b/bin" "$LAB/launcher/bin/jupyter"
run_nb C nb.ipynb project-a "$LAB/launcher/bin"  "$LAB/launcher/bin/jupyter"
run_nb D nb.ipynb project-a "$LAB/project-b/bin" "$LAB/launcher/bin/jupyter"
run_nb E nb.ipynb python3   "$LAB/launcher/bin"  "$LAB/project-c/bin/jupyter"
 
echo "=================================================================="
echo "4. WHAT !pip AND %pip ACTUALLY RUN"
echo "=================================================================="
run_nb F1 pip-nb.ipynb project-a "$LAB/project-b/bin" "$LAB/launcher/bin/jupyter"
run_nb F2 pip-nb.ipynb project-a "$LAB/launcher/bin"  "$LAB/launcher/bin/jupyter"
 
echo "=================================================================="
echo "5. A NOTEBOOK PINNED TO A KERNEL IGNORES WHICH JUPYTER YOU LAUNCH"
echo "=================================================================="
echo "pinned.ipynb asks for kernel 'project-a' in its own metadata."
echo "Launched from project-c's jupyter, with NO --kernel_name flag:"
rm -f "$LAB/out-H.ipynb"
PATH="$LAB/project-c/bin:$PATH" "$LAB/project-c/bin/jupyter" nbconvert --to notebook \
    --execute --allow-errors --output out-H.ipynb pinned.ipynb >/dev/null 2>&1
if [ -f "$LAB/out-H.ipynb" ]; then
  show_outputs "$LAB/out-H.ipynb" | norm
else
  echo "  !! nbconvert wrote no output. This case did not run."
fi
echo
 
echo "=================================================================="
echo "6. A KERNELSPEC THAT POINTS AT AN ENV THAT IS GONE"
echo "=================================================================="
"$LAB/project-a/bin/python" -m ipykernel install --user --name ghost \
    --display-name "Python (ghost)" >/dev/null 2>&1
"$LAB/launcher/bin/python" - <<'PY' | norm
import json, os
p = os.path.join(os.environ["JUPYTER_DATA_DIR"], "kernels", "ghost", "kernel.json")
k = json.load(open(p))
k["argv"][0] = k["argv"][0].replace("/project-a/", "/project-deleted/")
json.dump(k, open(p, "w"), indent=1)
print("ghost kernel argv[0] ->", k["argv"][0])
PY
echo "--- executing a notebook on it:"
rm -f "$LAB/ghost-out.ipynb"
"$LAB/launcher/bin/jupyter" nbconvert --to notebook --execute --allow-errors \
    --ExecutePreprocessor.kernel_name=ghost --output ghost-out.ipynb nb.ipynb 2>&1 \
    | tail -4 | norm
 
# Section 6 is SUPPOSED to fail -- that failure is the result being demonstrated. Under
# `set -o pipefail` its status would become the script's, so a fully successful run would
# exit nonzero and break `./run-matrix.sh && diff ...`. Exit 0 deliberately; the per-case
# guards above are what report a case that did not run.
exit 0

The versions are pinned because this page quotes those packages’ source and shipped files: pandas==3.0.5, ipykernel==7.3.0, jupyter_client==8.10.0, ipython==9.17.1, jupyterlab==4.6.3, nbconvert==7.17.1. A floating version in any env that runs a kernel would change what the page is about.

Later releases will move the line numbers pasted from manager.py and packaging.py. The rewrite rule itself is present in both jupyter_client versions checked here (8.0.3 and 8.10.0), so it is not a property of one release — but that is two data points, not a guarantee about future ones.

© Runcell.RSS