Minimal Mastery: Efficient Discriminative and Generative Modelling
This project implements a highly parameter-efficient classifier for AddNIST and a constrained generative model for CIFAR-100, each trained under strict limits on total parameters and optimisation steps. Additional methodology and visualisations have been incorporated from the accompanying project paper .
The AddNIST task involves identifying the digits embedded in each colour channel of the image and predicting their summed value (minus one), resulting in a 20-class classification problem. Under the constraint of <100k parameters and <10k optimisation steps, the model uses a compact ResNet-inspired CNN with depthwise separable convolutions and independent channel-wise feature extraction.
To illustrate the architecture, the following high-level model diagram demonstrates how features are extracted independently from each channel before being combined:
Residual blocks play a central role in enabling deeper architectures to train reliably by improving gradient flow, allowing the model to benefit from increased depth without suffering from vanishing gradients.
The classifier contains 98,228 parameters and achieved:
- 98.8% training accuracy
- 98.8% testing accuracy
Despite the tight constraints, it surpasses significantly larger models such as AlexNet, ResNet-18, and NAS-based approaches.
The training and validation performance is illustrated in the accuracy curve:
The close alignment of the curves indicates strong generalisation and minimal overfitting.
The generative component implements an SNGAN designed to stay under 1M total parameters and 50k optimisation steps. It uses spectral normalization for discriminator stability and a ResNet-style architecture for both generator and discriminator.
The adversarial setup is shown in the diagram below, illustrating how real and generated images are contrasted during training:
The final model contains 992,050 parameters and achieves an FID score of 43.50 on CIFAR-100.
A batch of non–cherry-picked generated samples is shown here, demonstrating sharp textures and good diversity:
To explore the smoothness of the learned latent space, the following visualisation shows interpolations between latent vectors:
Finally, to confirm that the model is not memorising the training set, each generated image is paired with its nearest neighbour (via LPIPS distance):
These results show that despite strict constraints, the GAN learns a structured latent space and produces recognisably coherent visual patterns.
- The classifier meets the parameter and step constraints while achieving state-of-the-art performance on AddNIST for this parameter budget.
- The GAN achieves FID 43.50 with under 1M parameters and demonstrates good sample diversity and smooth latent behaviour.