chore(deps): Bump datasets from 2.21.0 to 5.0.0 in /services/knowledge-service

[dependabot]

Bumps datasets from 2.21.0 to 5.0.0.

Release notes

Sourced from datasets's releases.

5.0.0

Datasets Features

Agent traces

• Parse Agent traces messages for SFT using teich by @​lhoestq in huggingface/datasets#8232

  • Agent traces from claude_code/pi/codex and others can now be loaded with load_dataset
  • Using the teich library (new optional dependency), traces are parsed to messages to enable training on traces using e.g. trl
  • Load the data:

  >>> from datasets import load_dataset
  >>> ds = load_dataset("lhoestq/agent-traces-example", split="train")
  >>> ds[0]["messages"]
  [{'role': 'user', 'content': 'Download a random dataset from Hugging Face, use DuckDB to inspect it, and come back with a short report about it. Be concise and include: dataset name, what files/format you found, row count or rough size if you can determine it,...'
   ...]

  • Train on agent traces:

  trl sft --dataset-name lhoestq/agent-traces-example ...

  • find all the Agent traces datasets on HF here: https://huggingface.co/datasets?format=format:agent-traces&sort=trending

Next-level shuffling in streaming mode

• Use multiple input shards for shuffle buffer by @​lhoestq in huggingface/datasets#8194

  ds = load_dataset(..., streaming=True)
  ds = ds.shuffle(seed=42)
  # or configure local buffer shuffling manually, default is:
  ds = ds.shuffle(seed=42, buffer_size=1000, max_buffer_input_shards=10)

  before👎:

  after✨:

  toy example comparison

  from datasets import IterableDataset
  ds = IterableDataset.from_dict({"i": range(123_456_789)}, num_shards=1024)
  ds = ds.shuffle(seed=42)
  print("Cold start ids:")

... (truncated)

Commits

• 68ac1a9 Release: 5.0.0 (#8239)
• cfe4492 Support composed splits in streaming datasets (#8220)
• fd67320 Keep None as a real null in Json() columns instead of the string "null" (#8231)
• 10cdc81 Fix iterable skip over full Arrow blocks (#8236)
• b7c064d Parse agent traces messages for SFT using teich (#8232)
• 31e92f1 fix: embed_external_files=True for mesh support (#8224)
• d168d5f feat: add TsFile (Apache IoTDB) packaged builder with per-device wide format ...
• 992f3cf fix(map): fix progress bar exceeding total when load_from_cache_file=False (#...
• 8474a91 Fix single lance file form pylance 7.0 (#8225)
• d4284e9 feat: add 3D mesh support and MeshFolder builder (#8055)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

[letuhao]

🔒 Held — not declining.

datasets is used only by services/knowledge-service/tests/quality/anchor_runner.py (eval-only, not in CI/build). Its DocRED anchor calls load_dataset(..., trust_remote_code=True), removed in datasets 4 (script-loading dropped) — already broken on the installed 4.x regardless of this pin. A version bump neither fixes nor regresses it.

Revisit when: the anchor_runner DocRED source is reworked to a non-script dataset — bump then.

[dependabot]

Superseded by #209.