Background
GameInstances is a plain Python collection that loads hierarchical instances.json files (structure: experiments → game_instances) and exposes them as a flat list of (experiment, game_instance) tuples for iteration and lookup by game_id.
Why HF Dataset was considered
- Already using
load_dataset("colab-potsdam/playpen-data", ...) for split filtering in the RL/serve context
- HF Dataset would provide filtering, caching, streaming, and Hub publishing for free
- Two dict-typed columns (
experiment, game_instance) are sufficient and schema-uniform across existing clembench games (verified by inspection)
- Would align local instance loading with the existing HF-based split loading pattern
Why we deferred
- HF
Dataset iteration yields dicts, not (experiment, game_instance) tuples — all existing loops in runners and reset strategies would need updating
- Wrapping HF Dataset behind a tuple-iteration interface would hide the dataset API, negating the main benefits
- The current requirements (load from
GameSpec-derived path, iterate once for eval, cycle or lookup by game_id for RL) are fully met by a plain Python collection
- The path to instances is derived from
GameSpec.game_path, which is clemcore-specific context a generic HF loading script cannot know about — so load_dataset() integration would be awkward anyway
When to reconsider
- Streaming large instance sets that don't fit in memory
- Publishing game instances to the HF Hub as a proper dataset (alongside
colab-potsdam/playpen-data)
- Needing rich filtering/querying on instance metadata across games
- Training pipelines that expect a standard HF
Dataset interface directly
Potential migration path
- Introduce module-level
flatten(instances_dict) -> Dataset and load_game_instances(game_spec) -> Dataset factory functions
- Update runner loops and reset strategies to use dict-row access (
row["experiment"], row["game_instance"]) instead of tuple unpacking
- Expose
Dataset directly from GameInstances (or replace it entirely) rather than wrapping it
Background
GameInstancesis a plain Python collection that loads hierarchicalinstances.jsonfiles (structure:experiments → game_instances) and exposes them as a flat list of(experiment, game_instance)tuples for iteration and lookup bygame_id.Why HF Dataset was considered
load_dataset("colab-potsdam/playpen-data", ...)for split filtering in the RL/serve contextexperiment,game_instance) are sufficient and schema-uniform across existing clembench games (verified by inspection)Why we deferred
Datasetiteration yields dicts, not(experiment, game_instance)tuples — all existing loops in runners and reset strategies would need updatingGameSpec-derived path, iterate once for eval, cycle or lookup bygame_idfor RL) are fully met by a plain Python collectionGameSpec.game_path, which is clemcore-specific context a generic HF loading script cannot know about — soload_dataset()integration would be awkward anywayWhen to reconsider
colab-potsdam/playpen-data)Datasetinterface directlyPotential migration path
flatten(instances_dict) -> Datasetandload_game_instances(game_spec) -> Datasetfactory functionsrow["experiment"],row["game_instance"]) instead of tuple unpackingDatasetdirectly fromGameInstances(or replace it entirely) rather than wrapping it