Measure the surface-code threshold with a matched prior and a dense grid

.github/workflows/notebooks.yml

@@ -5,13 +5,12 @@ name: notebooks
 # nothing ran these.
 #
 # MDOPT_NB_FAST=1 shrinks the workloads: at the committed full-scale settings
-# the suite totals roughly 10 hours (quantum_surface ~4.9h, classical_ldpc
-# ~2.5h, quantum_three_qubit ~1.6h, maxbonddim ~1.2h -- see
-# docs/source/notebook_pipeline.rst), past the 6h GitHub caps a job at and far
-# too dear to spend per pull request. The notebooks read that variable
-# themselves. What is being tested is that they execute, not the numbers they
-# produce -- the outputs are discarded, and the ones the documentation shows
-# come from a full-scale run committed by hand.
+# the suite totals roughly 34 hours (quantum_surface ~27h, maxbonddim ~3.2h,
+# classical_ldpc ~2.8h -- see docs/source/notebook_pipeline.rst), past the 6h
+# GitHub caps a job at and far too dear to spend per pull request. The
+# notebooks read that variable themselves. What is being tested is that they
+# execute, not the numbers they produce -- the outputs are discarded, and the
+# ones the documentation shows come from a full-scale run committed by hand.
 
 on:
   push:

docs/source/notebook_pipeline.rst

@@ -39,36 +39,48 @@ Regenerating the results
 
 .. code-block:: bash
 
-   python scripts/run_notebooks.py --inplace                    # all of them
+   python scripts/run_notebooks.py --inplace --timeout 172800   # all of them
    python scripts/run_notebooks.py --inplace examples/decoding/shor.ipynb
 
 ``--inplace`` writes the executed outputs back into the notebook. Commit the
 notebook afterwards; the next push to ``main`` opens the sync PR, and the docs
 follow once it is merged.
 
-Expect this to take hours. Measured on an M-series laptop, one notebook at a
-time (``classical_ldpc`` is the one estimate here -- it has not yet completed
-a full run):
+``--timeout`` is the limit per cell, in seconds. Its default of 1800 (or
+``MDOPT_NB_TIMEOUT``, if set) suits the fast CI workloads, but at full scale
+the longest cells below run for hours -- the surface-code threshold sweep
+alone took 26.6 h -- so a full regeneration needs a limit well above that;
+the last one allowed two days.
 
-===========================  =================
+Expect this to take hours: the last full run took about 34 hours. Measured on
+an M-series laptop (10 cores: 4 performance, 6 efficiency), one notebook at a
+time, with one BLAS thread per process:
+
+===========================  ==============================================================
 notebook                     full run
-===========================  =================
+===========================  ==============================================================
 ``main_component``           4 s
-``ground_state``             11 s
-``mps-rand-circ``            26 s
-``dephasing_dmrg_debug_bb``  39 s
-``shor``                     39 s
-``quantum_five_qubit``       43 s
-``maxbonddim``               1.2 h
-``quantum_three_qubit``      1.6 h
-``quantum_surface``          4.9 h
-``classical_ldpc``           ~2.5 h (estimate)
-===========================  =================
-
-The shot counts in the four expensive notebooks were chosen to land in that
-range. They were once far larger -- ``quantum_surface`` alone needed 33 hours,
-which nobody was going to run -- so the results here are deliberately noisier
-than a cluster campaign would give. The thesis figures come from the scripts in
+``quantum_five_qubit``       7 s
+``ground_state``             9 s
+``shor``                     10 s
+``mps-rand-circ``            14 s
+``dephasing_dmrg_debug_bb``  1.5 min
+``quantum_three_qubit``      4.6 min
+``classical_ldpc``           2.8 h (the truncation sweep 2.2 h, the BP comparison 36 min)
+``maxbonddim``               3.2 h
+``quantum_surface``          27.4 h (the threshold sweep alone 26.6 h, at chi_max = 128)
+===========================  ==============================================================
+
+``classical_ldpc`` and ``quantum_surface`` end with a cell that regenerates
+its data through a pool of worker processes (all but two cores by default;
+``MDOPT_NB_PROCESSES`` overrides it). Their times above are with eight
+workers; note that concurrent workers on this laptop run about three times
+slower per shot than a single process, because they share the efficiency
+cores, so a lone-process estimate understates them badly. The shot counts in
+the expensive notebooks were chosen to keep them at this scale -- the surface
+threshold sweep once needed 33 hours in the natural qubit order, which nobody
+was going to run -- so the results here are deliberately noisier than a
+cluster campaign would give. The thesis figures come from the scripts in
 ``mdopt/examples/decoding/plotting/`` and their cluster datasets, not from these
 notebooks.