Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 10 additions & 16 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,25 +16,19 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
python-version: ["3.10", "3.11", "3.12", "3.13"]

steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
- uses: actions/checkout@v5
- name: Install uv and set the Python version
uses: astral-sh/setup-uv@v6
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install flake8
python -m pip install .[test]
- name: Lint with flake8

- name: Lint with ruff
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
#flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Test with pytest
uvx ruff check

- name: Run tests
run: |
pytest -v
uv run -v pytest -v
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -128,3 +128,5 @@ dmypy.json

# Pyre type checker
.pyre/

uv.lock
20 changes: 10 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
# parallel-numpy-rng
[![tests](https://github.com/lgarrison/parallel-numpy-rng/actions/workflows/test.yml/badge.svg)](https://github.com/lgarrison/parallel-numpy-rng/actions/workflows/test.yml) [![PyPI](https://img.shields.io/pypi/v/parallel-numpy-rng)](https://pypi.org/project/parallel-numpy-rng/)

A multi-threaded random number generator backed by Numpy RNG, with parallelism provided by Numba.
A multi-threaded random number generator backed by NumPy RNG, with parallelism provided by Numba.

## Overview
Uses the "fast-forward" capability of the [PCG family of RNG](https://www.pcg-random.org),
as exposed by the [new-style Numpy RNG API](https://numpy.org/doc/stable/reference/random/index.html),
to generate random numbers in a multi-threaded manner. The key
is that you get the same random numbers regardless of how many threads were used.
This package uses the "fast-forward" capability of the [PCG family of RNG](https://www.pcg-random.org),
as exposed by the [new-style NumPy RNG API](https://numpy.org/doc/stable/reference/random/index.html),
to generate arrays of random numbers in a multi-threaded manner. The result depends only on the random
number seed, not the number of threads.

Only a two types of random numbers are supported right now: uniform and normal. More
could be added if there is demand, although some kinds, like bounded random ints, are
hard to parallelize in the approach used here.

The uniform randoms are the same as Numpy produces for a given seed, although the
The uniform randoms are the same as NumPy produces for a given seed, although the
random normals are not.

## Example + Performance
Expand All @@ -34,7 +34,7 @@ numpy_rng = np.random.default_rng(seed)
%timeit parallel_rng.standard_normal(size=10**8, dtype=np.float32, nthread=128) # 43.5 ms
```

Note that the `parallel_rng` is slower than Numpy when using a single thread, because the parallel implementation requires a slower algorithm in some cases (this is especially noticeable for float64 and normals)
Note that the `parallel_rng` is slower than NumPy when using a single thread, because the parallel implementation requires a slower algorithm in some cases (this is especially noticeable for float64 and normals)

## Installation
The code works and is [reasonably well tested](./test_parallel_numpy_rng.py), so it's probably ready for use. It can be installed from PyPI:
Expand All @@ -56,8 +56,8 @@ to the RNG, thus advancing its internal state that many times. Therefore, we wou
the second thread's RNG in a state that is already advanced *N/2* times, but without actually making
*N/2* calls to get there.

This is known as fast-forwarding, or jump-ahead. [Numpy's new RNG API](https://numpy.org/doc/stable/reference/random/index.html)
(as of Numpy 1.17) uses the PCG RNG that supports this feature, and Numpy exposes an [`advance()`
This is known as fast-forwarding, or jump-ahead. [NumPy's new RNG API](https://numpy.org/doc/stable/reference/random/index.html)
(as of NumPy 1.17) uses the PCG RNG that supports this feature, and NumPy exposes an [`advance()`
method](https://numpy.org/doc/stable/reference/random/bit_generators/generated/numpy.random.PCG64.advance.html#numpy.random.PCG64.advance)
which implements it. The new API also exposes CFFI bindings to get PCG random values,
so we can implement the core loop, including parallelism, in a Numba-compiled function
Expand All @@ -67,5 +67,5 @@ An interesting consequence of using fast-forwarding is that each random value mu
with a known number of calls to the underlying RNG, so that we know how many steps to advance
the RNG state by. This means we can't use rejection sampling, which makes a variable number of
calls. Fortunately, inverse-transform sampling can usually substitute, or more specific methods
like Box-Muller. These can be slower than rejection sampling (or whatever Numpy uses) with one
like Box-Muller. These can be slower than rejection sampling (or whatever NumPy uses) with one
thread, but even just two threads more than makes up for the difference.
39 changes: 37 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,12 +1,47 @@
[build-system]
requires = [
"setuptools>=42",
"wheel",
"setuptools_scm>=6.2",
]

build-backend = "setuptools.build_meta"

[project]
name = "parallel-numpy-rng"
authors = [
{name = "Lehman Garrison"},
]
description = "Parallel random number generation that produces the same result, regardless of the number of threads"
readme = "README.md"
requires-python = ">=3.10"
classifiers = [
"Programming Language :: Python :: 3",
]
dependencies = [
"numpy>=2",
"numba",
]
dynamic = ["version"]
license = "Apache-2.0"
license-files = ["LICENSE"]

[project.optional-dependencies]
test = ["pytest"]

[dependency-groups]
dev = ["parallel-numpy-rng[test]"]

[project.urls]
Homepage = "https://github.com/lgarrison/parallel-numpy-rng/"

[tool.setuptools]
package-dir = {"" = "src"}

[tool.setuptools.packages.find]
where = ["src"]

[tool.setuptools_scm]
write_to = "src/parallel_numpy_rng/version.py"
write_to_template = '__version__ = "{version}"'

[tool.ruff.format]
quote-style = "single"
24 changes: 0 additions & 24 deletions setup.cfg

This file was deleted.

4 changes: 2 additions & 2 deletions src/parallel_numpy_rng/__init__.py
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
from .version import __version__
from .parallel_numpy_rng import *
from .version import __version__ as __version__
from .parallel_numpy_rng import * # noqa: F403
Loading