NLP From Scratch

A collection of NLP implementations built from first principles in Python — no high-level classifier libraries, no sklearn.fit, no loss functions imported from PyTorch. The point is to understand how the algorithms actually work.

Projects

1. Statistical Spellchecker — notebooks/spellchecker.ipynb

Corrects spelling errors by ranking candidate words using a noisy-channel model: argmax_c P(c) · P(error | c).

Technique

• Candidate generation: Damerau-Levenshtein edit distance up to 2 (insertions, deletions, substitutions, transpositions)
• Language model: Unigram (word frequency with Laplace smoothing) → upgraded to Bigram (previous-word context) for sentence-level correction
• Error model: Uniform weights by edit distance → improved weighted model that distinguishes transpositions (most common), keyboard-adjacent substitutions (QWERTY proximity map), and random edits, following the Kernighan et al. (1990) approach
• Training corpus: NLTK Brown Corpus (~1M words, 15 genres)

Results

Setting	Metric
Single-word accuracy (Birkbeck corpus, n=500)	29%
Sentence-level accuracy (bigram model)	68%
Sentence-level F1-score	56%

---

2. Sentiment Classifier — notebooks/sentiment-classifier.ipynb

Binary sentiment classifier (positive / negative) on IMDB movie reviews, implemented entirely in NumPy. No sklearn, PyTorch, or TensorFlow classifier, loss, or gradient functions used.

Technique

• Features: 3 hand-crafted features from the NRC Emotion Lexicon:
  • x1: normalized positive-word density
  • x2: normalized negative-word density
  • x3: log(review token count)
  • Extended to 5 features with joy density (x4) and anger+disgust density (x5)
• Model: Binary logistic regression — sigmoid activation, binary cross-entropy loss, manually derived gradients
• Training: Mini-batch gradient descent (batch size 256), 6 learning rates × 20 epochs; best LR selected by minimum validation loss on a held-out 20% split
• Dataset: 25K train / 25K test IMDB reviews (HuggingFace datasets)

Results (3-feature model, LR=0.1)

Metric	Score
Test Accuracy	66.4%
Precision	63.2%
Recall	78.4%
F1-score	0.70

---

Setup & Run

Install dependencies

pip install -r requirements.txt

Fetch required data

Most data is fetched automatically at notebook runtime:

Data	How to get it
NLTK Brown Corpus	nltk.download('brown') — called in cell 1 of spellchecker.ipynb
Birkbeck spelling error corpus	Fetched via requests from the Oxford Text Archive URL in the notebook
IMDB reviews	load_dataset("stanfordnlp/imdb") — called in sentiment-classifier.ipynb
NRC Emotion Lexicon	wget call in sentiment-classifier.ipynb
Test-Dataset-For-Sentences.csv	Not included. Required for sentence-level evaluation in Section 5 of spellchecker.ipynb. See the note in that section.

Launch notebooks

jupyter notebook notebooks/

Run each notebook top-to-bottom via Kernel → Restart & Run All.

Note on spellchecker.ipynb sentence evaluation: Section 5 requires Test-Dataset-For-Sentences.csv placed in the same directory as the notebook. That section will error without it; all other sections run fine.

---

Repo Structure

.
├── README.md
├── requirements.txt
├── .gitignore
└── notebooks/
    ├── spellchecker.ipynb          # Noisy-channel spellchecker
    └── sentiment-classifier.ipynb  # Logistic regression sentiment classifier

---

Tech

• Language: Python 3
• Core libraries: NumPy, NLTK, HuggingFace datasets
• Visualization: Matplotlib
• Evaluation only (no training): scikit-learn (confusion_matrix, ConfusionMatrixDisplay)

---

Author

Jake Wang — sole author of all implementations.