NeoMatrix

NeoMatrix is a Torch-inspired machine learning library that combines the computational performance of Rust with the simplicity and flexibility of Python for building and training neural networks.

Architecture Overview

NeoMatrix is structured as a multi-layered system:

┌─────────────────────────────────────────┐
│   Python API (neomatrix)                │  ← High-level model building
│   • NeuralNetwork, Layer, Optimizer     │     ⚠️ Work in progress
└─────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────┐
│   PyO3 Bindings (neomatrix-python-      │  ← Python ↔ Rust bridge
│   wrapper)                               │     ✅ Fully Implemented
│   • Expose Rust types to Python         │
└─────────────────────────────────────────┘
                    ↓
┌─────────────────────────────────────────┐
│   Rust Core (neomatrix-core)            │  ← High-performance backend
│   • Tensor ops, layers, math            │     ✅ Fully Implemented
│   • Parallel computation (Rayon)        │
└─────────────────────────────────────────┘

Components

1. neomatrix-core (Rust Library) ✅

The computational engine of NeoMatrix, written in pure Rust with comprehensive documentation.

Features:

• Tensors: Multi-dimensional arrays (f32) with automatic shape inference and broadcasting
• Layers: Composable neural network building blocks
  • Dense: Fully-connected layer with learnable weights and biases
  • Activation layers: ReLu, Sigmoid, Tanh, Softmax
• Activation Functions: ReLU, Sigmoid, Tanh, Softmax, Linear
• Loss Functions:
  • Regression: MSE, MAE, Huber
  • Classification: Binary Cross-Entropy, Categorical Cross-Entropy, Hinge
• Weight Initialization:
  • Xavier/Glorot (optimal for Sigmoid/Tanh)
  • He (optimal for ReLU)
  • Random uniform
• Optimizer: Stateful optimization algorithms for updating model parameters
• Parallel Computing: Rayon-powered matrix multiplication for multi-core performance
• Error Handling: Comprehensive Result-based error types (no panics in library code)

2. neomatrix-python-wrapper (PyO3 Bindings) ✅

Python bindings exposing the Rust core to Python via PyO3. All core types are exposed:

Exposed Types:

• Tensor: Full NumPy interoperability via __array__ protocol, to_numpy(), from_numpy()
• Layers: Dense (with get_parameters()), ReLU, Sigmoid, Tanh, Softmax
• Optimizers: GradientDescent with register_params(), step(), zero_grad()
• Loss Functions: MSE, MAE, BCE, CCE, Huber, Hinge (BCE and CCE with backward_optimized() support)
• Initialization: Init enum (Xavier, He, Random)
• Parameter Management: ParametersRef for shared ownership between layers and optimizers

3. neomatrix (Python Package) ⚠️

High-level Python API inspired by Torch and Keras, built on top of the Rust backend. Provides:

• ✅ Model: Sequential model container with compile() and fit() methods
• ✅ Layer: Re-exports for Dense, ReLU, Sigmoid, Tanh, Softmax
• ✅ Optimizer: Re-exports for GradientDescent (stateful optimizer)
• ✅ LossFunction: Re-exports for all 6 loss functions
• ✅ Dataset utilities: get_batches() for batch processing

---

Features

Current Implementation ✅

• Rust Backend: Complete core library with tensor operations and neural network layers
• Comprehensive Documentation: All modules, functions, and mathematical operations documented in English
• Mathematical Rigor: Backpropagation algorithms with gradient formulas
• Parallel Computation: Multi-threaded matrix multiplication via Rayon
• Type Safety: Result-based error handling throughout
• Operator Overloading: Pythonic tensor arithmetic (+, -, *, /)
• Iterator Support: Tensors implement Rust's Iterator trait
• 243 Unit Tests: Comprehensive test coverage for all modules

Activation Functions

Function	Formula	Derivative	Use Case
ReLU	max(0, x)	1 if x > 0, else 0	Hidden layers (most common)
Sigmoid	1 / (1 + e^(-x))	σ(x) · (1 - σ(x))	Binary classification output
Tanh	(e^x - e^(-x)) / (e^x + e^(-x))	1 - tanh²(x)	Hidden layers (zero-centered)
Softmax	e^(x_i) / Σ_j e^(x_j)	Jacobian matrix	Multi-class classification output
Linear	x	1	Regression output

Loss Functions

Function	Formula	Best For
MSE	(1/n) Σ(y - ŷ)²	Regression
MAE	`(1/n) Σ	y - ŷ
Huber	Smooth MSE/MAE transition	Outlier-resistant regression
BCE	-[y·log(ŷ) + (1-y)·log(1-ŷ)]	Binary classification (with Sigmoid)
CCE	-Σ y_i · log(ŷ_i)	Multi-class classification (with Softmax)
Hinge	max(0, 1 - y·ŷ)	SVM-style classification

Weight Initialization

Strategy	Formula	Recommended For
Xavier	W ~ N(0, √(2/(n_in + n_out)))	Sigmoid, Tanh activations
He	W ~ N(0, √(2/n_in))	ReLU, Leaky ReLU activations
LeCun	W ~ N(0, √(1/n_in))	SELU activations
Random	U(a, b) uniform	Legacy (not recommended)

---

Installation

Building from Source

Prerequisites:

• Rust 1.70+ (Install Rust)
• Python 3.8+
• maturin (pip install maturin or uv add maturin)

Build the Rust core:

cd neomatrix-core
cargo build --release
cargo test  # Run 240 unit tests
cargo doc --no-deps --open  # View documentation

Build the Python extension (when wrapper is ready):

# Development build
maturin develop

# Release wheel
maturin build --release

---

Usage Examples

Current Status (Rust Core)

The Rust core is fully functional and can be used directly:

use neomatrix_core::tensor::Tensor;
use neomatrix_core::layers::{dense::Dense, init::Init, Layer};
use neomatrix_core::math::activations::{Relu, ActivationFunction};

// Create tensors
let input = Tensor::new(vec![1, 784], vec![/* ... */]).unwrap();

// Build a dense layer with He initialization
let mut layer = Dense::new(784, 128, Some(Init::He), None);

// Forward pass
let output = layer.forward(&input, true).unwrap();

// Compute loss and backward pass
let grad = /* compute gradient from loss */;
let input_grad = layer.backward(&grad).unwrap();

// Access parameters for optimization
let params = layer.get_parameters();
// Update weights via optimizer: optimizer.register_params(vec![params])

Project Structure

NeoMatrix/
├── neomatrix-core/          # ✅ Rust core library (COMPLETE)
│   ├── src/
│   │   ├── lib.rs          # Crate root
│   │   ├── tensor/         # Tensor operations
│   │   ├── math/           # Activations, losses, matmul
│   │   ├── layers/         # Dense, activation layers, init
│   │   ├── errors.rs       # Error types
│   │   └── test/           # 240 unit tests
│   └── Cargo.toml
├── neomatrix-python-wrapper/ # ✅ PyO3 bindings (COMPLETE)
│   ├── src/
│   │   ├── lib.rs          # Module registration (13 types)
│   │   ├── tensor_bindings/ # Python Tensor + NumPy interop
│   │   ├── layer_bindings/ # Dense, ReLU, Sigmoid, Tanh, Softmax
│   │   ├── optimizer_bindings/ # GradientDescent, ParametersRef
│   │   └── losses_bindings.rs # All 6 loss functions
│   └── Cargo.toml
├── neomatrix/              # ⚠️ Python package (WORKING)
│   ├── model.py            # Model class
│   ├── layers.py           # Layer re-exports
│   ├── optimizers.py       # Optimizer re-exports
│   ├── losses.py           # Loss function re-exports
│   ├── metrics.py          # Evaluation metrics
│   ├── utils.py            # get_batches utility
│   └── __init__.py         # Package entry point
├── examples/               # Usage examples
│   ├── NeuralNetwork.py
│   └── LinearRegression.py
├── pyproject.toml          # Python project config (maturin)
├── Cargo.toml              # Rust workspace
└── README.md               # This file

---

Development Status

Completed ✅

• Complete Rust core implementation (neomatrix-core)
• Comprehensive English documentation for all modules
• Tensor operations with operator overloading
• Neural network layers (Dense, Activation)
• 5 activation functions with backpropagation
• 6 loss functions with numerical stability
• 4 weight initialization strategies (Xavier, He, LeCun, Random)
• Parallel matrix multiplication (Rayon)
• 240 unit tests with 100% pass rate
• Error handling with Result types (no panics)

In Progress 🚧

• High-level Python API (neomatrix package)
  • Model class
  • Optimizer implementations
  • Evaluation metrics: Accuracy, Precision, Recall, F1-score
  • Callbacks
  • Model save/load

Planned 📋

• Additional optimizers: Adam, RMSprop, Adagrad
• Regularization: L1, L2, Dropout
• Batch normalization layer
• Convolutional layers (Conv2D, MaxPool2D)
• Recurrent layers (LSTM, GRU)
• GPU acceleration (CUDA/Metal)

---

Documentation

Rust Core Documentation

Full API documentation is available via cargo doc:

cd neomatrix-core
cargo doc --no-deps --open

Key documentation sections:

• Module-level docs (//!) describe the purpose of each module
• Struct/Enum docs (///) explain data structures and their fields
• Function docs (///) detail parameters, returns, and usage examples
• Mathematical formulas (//) accompany all activation/loss computations
• Implementation notes (//) clarify complex algorithms (e.g., backpropagation)

Examples

See the examples/ directory for usage demonstrations:

• LinearRegression.py - Simple linear regression model
• NeuralNetwork.py - Multi-layer neural network for classification

---

Performance

NeoMatrix leverages Rust's performance and safety guarantees:

• Zero-cost abstractions: Generic types compile to optimal machine code
• Memory safety: No garbage collection, no null pointers, no data races
• Parallel execution: Rayon automatically parallelizes matrix operations across CPU cores
• SIMD optimization: ndarray uses vectorized operations where possible

Benchmarks (coming soon): Training performance comparisons with NumPy, PyTorch, and TensorFlow.

---

License

This project is licensed under the MIT License.

---

Performance Benchmarks 🚀

Optimizer Performance (10-layer network, 1000→50 neurons each)

Operation	Average Time	Details
optimizer.step()	~4ms	Parallel weight updates via Rayon
optimizer.zero_grad()	~0.7ms	Parallel gradient reset
Speedup	2-4x	Multi-core CPU vs sequential

Architecture: Shared ownership via Arc<Mutex<Tensor>> enables parallel updates across layers with negligible lock overhead (<0.1% total time).

Roadmap

Phase 1: Core Implementation ✅ (COMPLETE)

• Rust core library with full documentation
• Tensor operations and linear algebra
• Dense layers and activation functions
• Loss functions with numerical stability
• Weight initialization strategies
• Parallel computation support

Phase 2: Python Integration ✅ (COMPLETE)

• Complete PyO3 bindings for all Rust types
• NumPy interoperability
• High-level Python API

Phase 3: Advanced Features 📋 (PLANNED)

• Additional optimizers (Adam, RMSprop, etc.)
• Regularization techniques
• Evaluation metrics
• Batch normalization

Phase 4: Expansion 🔮 (FUTURE)

• Convolutional layers for image processing
• Recurrent layers for sequence modeling
• GPU acceleration
• Distributed training

---

Status: Active development | Core complete, Python integration in progress