This project goal is to get an LLM to follow instructions from this data via prompting (zero-shot, few-shot) or by fine-tuning a small model and evaluate how well it does. The dataset has structure and quirks that you are expected to discover and address; your design choices (prompt format, few-shot selection, evaluation) should reflect that.
| Model | Type | Parameters | Environment |
|---|---|---|---|
| DeepSeek-R1 | API (cloud) | ~67B | Cloud API |
| TinyLlama + LoRA | Local fine-tuned (instruction-tuned base) | 1.1B | Local |
| OPT-1.3B + LoRA | Local fine-tuned (base model) | 1.3B | Local |
- Source: Databricks Dolly 15K — a human-generated instruction-following dataset
- Format: JSONL with fields:
instruction,context,response,category - Categories: 8 task types — brainstorming, classification, closed_qa, creative_writing, general_qa, information_extraction, open_qa, summarization
- Splits: Train / Test
- Removed duplicates and empty responses
- Standardized field names and formatting
- Split into train/val/test sets
- Saved as clean JSONL files
- Approach: Zero-shot and few-shot prompting via cloud API
- No fine-tuning — relies entirely on pre-trained knowledge
- Evaluation: Compared zero-shot vs few-shot to measure in-context learning
- Base model: TinyLlama-1.1B-Chat (already instruction-tuned)
- Fine-tuning: LoRA (Low-Rank Adaptation) on Dolly training set
- Key insight: Starts from an instruction-tuned checkpoint — LoRA refines an already capable model
- Base model: facebook/opt-1.3b (base language model, NOT instruction-tuned)
- Fine-tuning: LoRA on Dolly training set
- Key insight: Starts from a raw pre-trained model — tests whether LoRA alone can teach instruction-following from scratch
- Stratified sampling: equal examples per category
- Up to 5 examples per category for fair comparison
- Ensures no single category dominates the aggregate metrics
| Metric | What it measures |
|---|---|
| BLEU | N-gram precision — exact word overlap with reference |
| ROUGE-1 | Unigram recall — word-level coverage |
| ROUGE-2 | Bigram recall — phrase-level coverage |
| ROUGE-L | Longest common subsequence — structural similarity |
| BERTScore F1 | Semantic similarity using contextual embeddings (roberta-large) |
Each model is evaluated both overall and per-category to identify strengths and weaknesses across different task types.
-
DeepSeek dominates all metrics — As expected, a ~67B parameter cloud model significantly outperforms 1B-class local models.
-
TinyLlama + LoRA achieves strong results considering its small size.
-
BERTScore is more forgiving than BLEU — Semantic similarity (BERTScore) scores are consistently higher than lexical overlap (BLEU), suggesting models capture meaning even when exact wording differs.
-
Few-shot prompting provides no benefit for already instruction-tuned models on diverse tasks.
| Metric | DeepSeek Zero-Shot | DeepSeek Few-Shot | TinyLlama + LoRA | OPT-1.3B + LoRA |
|---|---|---|---|---|
| BLEU | High | High | Low–Medium | Low |
| ROUGE-L | High | High | Low–Medium | Low |
| BERTScore | High | High | Medium | Medium–Low |
| Metric | Value |
|---|---|
| BLEU | 0.0174 |
| ROUGE-1 | 0.2597 |
| ROUGE-2 | 0.0877 |
| ROUGE-L | 0.1873 |
LLM-as-Judge Metric:
To avoid self-preference bias, we use Groq's Llama 3.3 70B as an independent judge, rather than letting DeepSeek evaluate its own outputs. This prevents model bias and ensures a fair comparison. The LLM judge rates each response from 1 (completely wrong) to 5 (matches or exceeds the reference), providing a human-like assessment of instruction-following quality.
LLM Judge Results:
- Zero-Shot LLM Judge: mean=4.44 (n=80)
- Few-Shot LLM Judge: mean=4.35 (n=80)
Per-Category LLM Judge Breakdown:
| Category | Count | Zero-Shot Judge | Few-Shot Judge |
|---|---|---|---|
| brainstorming | 10 | 4.3 | 4.2 |
| classification | 10 | 5.0 | 5.0 |
| closed_qa | 10 | 4.6 | 4.5 |
| creative_writing | 10 | 4.0 | 4.1 |
| general_qa | 10 | 4.6 | 4.2 |
| information_extraction | 10 | 4.2 | 4.0 |
| open_qa | 10 | 4.3 | 4.2 |
| summarization | 10 |
These results confirm that zero-shot prompting is at least as effective as few-shot for DeepSeek, even when judged by an external LLM.
The independent judge confirms the same pattern found by automatic metrics: zero-shot performs comparably or better than few-shot for DeepSeek. This convergence across four evaluation methods (BLEU, ROUGE, BERTScore, and an independent LLM judge) strengthens the conclusion that few-shot prompting provides no benefit for already instruction-tuned models on diverse tasks.
Temperature effect: Comparing temperature=0.3 vs temperature=0.0 shows that lowering temperature to fully deterministic decoding has minimal impact on aggregate metrics. This confirms that DeepSeek's instruction-following quality is robust to small temperature changes, and the zero-shot vs few-shot finding holds regardless of the sampling strategy.
Problem: HuggingFace Hub download failed repeatedly:
Fine-tune larger models (e.g., LLaMA-3-8B, Mistral-7B) using QLoRA (4-bit quantization + LoRA). This enables training 7–8B parameter models on consumer GPUs (16GB VRAM) while approaching cloud-model quality.
Systematically compare LoRA on instruction-tuned bases (e.g., LLaMA-3-8B-Instruct) vs raw bases (e.g., LLaMA-3-8B) across multiple model families to quantify the instruction-tuning advantage.
Apply Direct Preference Optimization (DPO) or RLHF after LoRA fine-tuning to improve response quality beyond supervised fine-tuning — especially for creative and open-ended categories where current models underperform.
Add human evaluation scores alongside automated metrics. BLEU/ROUGE often undervalue correct but differently-worded responses. Human ratings on fluency, relevance, and completeness would provide a more reliable comparison.
Extend evaluation to multi-turn conversations and Retrieval-Augmented Generation (RAG) pipelines, where smaller fine-tuned models combined with retrieval may close the gap with large cloud models.
Explore model compression (GPTQ, AWQ quantization) and inference optimization (vLLM, GGUF) for deploying fine-tuned models in production with minimal latency and hardware requirements.
- https://www.promptingguide.ai/
- https://huggingface.co/learn/llm-course/en/
- https://www.youtube.com/watch?v=uikZs6y0qgI
- https://arxiv.org/abs/2106.09685
- https://huggingface.co/docs/peft/main/en/conceptual_guides/lora
- https://medium.com/@pur4v/understanding-llm-evaluation-metrics-bleu-rouge-exact-match-and-bertscore-716487e40bdd
- https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0
- https://www.deepseek.com/en/