Financial LLM LoRA Benchmark

A privacy-conscious, reproducible toolkit for parameter-efficient fine-tuning and evaluation of causal language models on financial forecasting text. It separates training from benchmarking, masks prompts out of the supervised loss, defaults to deterministic decoding, and stores human-readable JSON rather than unsafe pickle files.

The default dataset identifier is FinGPT's Dow 30 forecaster dataset, which currently exposes prompt, answer, period, label, and symbol fields. Dataset contents are downloaded at runtime and are never redistributed here.

Why this version is reproducible

• The same tokenizer is used for each model's training records and model inputs.
• Prompt tokens receive label -100, so the optimization loss covers the answer rather than asking the model to memorize the prompt.
• Evaluation decodes only newly generated tokens, avoiding brittle string splitting on prompt markers.
• Greedy decoding is the default; stochastic sampling requires an explicit --sample flag and keeps a recorded seed.
• Results use JSON/JSONL and include every prediction, reference, latency, and run configuration.
• Metrics distinguish generative overlap (case-folded, non-stemmed ROUGE-1/ROUGE-L), strict exact match, and label-aware up/down direction accuracy with coverage.
• W&B reporting and interactive credential prompts are absent by default.

Installation

Python 3.10 or newer is required. Core metrics and tests are lightweight:

python -m venv .venv
source .venv/bin/activate
python -m pip install -e .
python -m unittest discover -s tests -v

Training and model evaluation need the optional ML stack:

python -m pip install -e '.[ml]'

Four-bit NF4 runs require a hardware/backend combination supported by bitsandbytes. Consult the current Transformers bitsandbytes guide before allocating a GPU runtime.

Train a LoRA adapter

financial-llm-train \
  --model meta-llama/Llama-3.2-1B-Instruct \
  --output-dir outputs/llama-1b \
  --load-in-4bit

The command follows Hugging Face's PEFT pattern: configure LoRA, wrap the base model, train only adapter parameters, and save the adapter separately. The default target modules are q_proj,k_proj,v_proj,o_proj; verify those names against a different architecture before training.

Some base models are gated. Authenticate through the Hugging Face CLI or an environment secret managed by your runtime. Never paste a token into a notebook or commit it to this repository.

Evaluate a base model or adapter

Input is JSONL with prompt and reference; id and directional label are optional. A two-record fictional schema example is included at examples/test_sample.jsonl.

# Base model
financial-llm-evaluate \
  --input your_test_set.jsonl \
  --model meta-llama/Llama-3.2-1B-Instruct \
  --output-dir results/llama-base \
  --load-in-4bit

# Fine-tuned adapter on the same base model and records
financial-llm-evaluate \
  --input your_test_set.jsonl \
  --model meta-llama/Llama-3.2-1B-Instruct \
  --adapter outputs/llama-1b/adapter \
  --output-dir results/llama-adapter \
  --load-in-4bit

Each run writes predictions.jsonl and summary.json. Compare runs only when the input records, decoding mode, token limits, device class, and software environment match. Latency comparisons across different GPU types are not meaningful.

Evaluation cautions

ROUGE measures lexical overlap, not financial correctness or investment value. Direction accuracy is only reported when parsable reference labels are supplied, and coverage shows how often a prediction actually stated an up/down direction. A sound study should additionally inspect hallucinations, evidence attribution, calibration, temporal leakage, robustness by company/period, and multiple random seeds for sampled decoding.

This code does not download prices, execute trades, or provide investment advice. Model outputs can be false, stale, or harmful if treated as recommendations.

Repository boundary

This is a clean reconstruction of the author's financial-LLM experiment. Raw Colab outputs, login traces, course materials, third-party comparison scripts, generated checkpoints, model adapters, dataset rows, and the original assignment report are intentionally excluded. See NOTICE.

License

The code is released under the MIT License. Models and datasets retain their own licenses and terms; review them before use.