Data Pipeline V0

[kris927b]

Data Pipeline V0

This is the initial version of our DFM pipeline.
It includes:

• Language filter (Danish)
• Heuristic filters (Gopher, C4, Fineweb)
• Sentence Deduplication
• MinHash Deduplication

This is the first step towards training a commercially viable Danish Language Model.

---

This discussion was created from the release Data Pipeline V0.