Skip to content

[Proto] Add a Fit step and FitCohort - #320

Open
jrapin wants to merge 12 commits into
mainfrom
fit-cohort
Open

jrapin wants to merge 12 commits into
mainfrom
fit-cohort

Conversation

@jrapin

@jrapin jrapin commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Fit: one artifact over a cohort of items

Fit derives an artifact from many items, then transforms each item with it — normalization stats, a PCA basis, a trained model.

class Normalize(steps.Fit):

    def _fit(self, values):    # the cohort
        stacked = np.concatenate(list(values))
        return stacked.mean(0), stacked.std(0)

    def _run(self, value):     # one item
        mean, std = self.fitted
        return (value - mean) / std

norm = Normalize(infra={"backend": "Cached", "folder": cache})
norm.run_many(steps.FitCohort(train))  # fits on these, then transforms them
norm.run_many(test)                    # same artifact, novel items
  • Only a FitCohort is fitted on; anything else transforms with the artifact already fitted.
  • The cohort's fingerprint — or its name, Normalize(cohort="train") — scopes the artifact and every downstream cache. A name is recoverable from the config alone; another cohort needs another config (clone()).
  • The fit runs before any backend splits the batch, cached under infra. The upstream is read twice, so give an expensive one its own infra.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 1, 2026
@jrapin jrapin mentioned this pull request Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant