Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,11 +112,11 @@ func Build(ctx context.Context, cfg *Config) (*App, error) {
return nil, err
}

// 2. Artifact registry — one loader is the single source of truth
// 2. Artifacts — one loader is the single source of truth. LoadSet reads
// configs/manifest_v1.json, manifest_v2.json, …
artifactLoader, _ := local.New(local.Config{Root: "configs"})
registry := artifact.NewRegistry(artifactLoader)
manifest, _ := artifact.LoadManifest(ctx, artifactLoader) // reads configs/manifest.json
artifact.RegisterFromConfig(registry, manifest)
artifacts, _ := artifact.LoadSet(ctx, artifactLoader)
registry := artifacts.CurrentRegistry()

// 3. Payment
paymentRepo := NewPaymentRepo(db)
Expand Down
60 changes: 38 additions & 22 deletions artifact/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,11 @@ Return `artifact.ErrNotFound` when the resource is absent; return other errors o

### `Registry`

Holds registered loaders and a manifest table `(id, kind, version) → (loaderType, path)`. Created once at startup and shared read-only at runtime.
Resolves `(id, kind, version) → path` for **one manifest** and fetches through the single loader. Shared read-only at runtime.

### `Set`

Holds every published manifest of one source, one `Registry` per manifest version, and names the current one. A flow records `set.Current()` when it starts and resolves every artifact through `set.Registry(ctx, version)` until it ends, so a deploy never changes the templates of a running flow. See [Versioned manifests](#versioned-manifests).

---

Expand Down Expand Up @@ -83,51 +87,61 @@ artifact.Latest[*EmailTemplate](ctx, reg, "welcome")
### 2. Wire up at startup

```go
// One loader is the single source of truth for this registry.
// One loader is the single source of truth for every manifest and artifact.
loader, err := local.New(local.Config{Root: "/etc/configs"})
reg := artifact.NewRegistry(loader)

// Register artifact rows directly in code...
reg.RegisterArtifact("welcome_email", "email_template", "v2", "emails/welcome.v2.json")
// Load manifest_v1.json, manifest_v2.json, … and pick the current version.
set, err := artifact.LoadSet(ctx, loader) // highest version
set, err := artifact.LoadSet(ctx, loader, artifact.WithPinnedVersion(4)) // or a pinned one
```

A bare `Registry` can still be built by hand, for tests:

// ...or load them from the manifest, fetched through the same loader.
cfg, err := artifact.LoadManifest(ctx, loader)
artifact.RegisterFromConfig(reg, cfg)
```go
reg := artifact.NewRegistry(loader)
reg.RegisterArtifact("welcome_email", "email_template", "", "emails/welcome.json")
```

The loader is required — `NewRegistry(nil)` panics. `RegisterFromConfig` validates that each row has an `id`, `kind`, and `path`, so misconfiguration is caught at startup rather than at first access.
The loader is required — `NewRegistry(nil)` panics and `LoadSet(ctx, nil)` fails. `RegisterFromConfig` rejects a row without an `id`, `kind`, or `path`, and two rows with the same `id`, `kind` and `version`.

### 3. Fetch

```go
// Fetch the newest registered version
// Inside a flow: the version the flow recorded when it started.
reg, err := set.Registry(ctx, flow.ManifestVersion)
tmpl, err := artifact.Latest[EmailTemplate](ctx, reg, "welcome_email")

// Fetch a specific pinned version (e.g. to resume a long-running workflow
// that must stay on the version it started with)
tmpl, err := artifact.Get[EmailTemplate](ctx, reg, "welcome_email", "v2")
// Outside any flow, for an artifact requested by explicit version (static data):
list, err := artifact.GetExplicit[StaticList](ctx, set, "ports", "2")
```

Both calls: fetch bytes via the loader → call `Parse` → return typed value.
All calls: fetch bytes via the loader → call `Parse` → return typed value. Within one manifest an artifact without an explicit version has a single row, so `Latest` returns it.

---

## Manifest file
## Versioned manifests

The manifest is a single JSON file named `manifest.json`, resolved at the loader's
root (`artifact.ManifestFilename`). `LoadManifest` fetches it through the loader,
so the manifest lives alongside the artifacts it catalogs — one source for both.
There is no per-row `loader`: the registry has exactly one.
Manifests are `manifest_v1.json`, `manifest_v2.json`, … at the loader's root (`artifact.ManifestPath(v)`), alongside the artifacts they catalog. There is no per-row `loader`: the source has exactly one.

```json
{
"artifacts": [
{ "id": "welcome_email", "kind": "email_template", "version": "v2", "path": "emails/welcome.v2.json" },
{ "id": "import_clearance", "kind": "workflow", "version": "v10", "path": "workflows/import_clearance.v10.json" }
{ "id": "welcome_email", "kind": "email_template", "path": "emails/welcome_v2.json" },
{ "id": "import_clearance", "kind": "workflow", "path": "workflows/import_clearance.json" },
{ "id": "ports", "kind": "static_data", "version": "2", "path": "static/ports_v2.json" }
]
}
```

The rules that make a manifest version a stable snapshot:

- **Published files never change.** A change to an artifact is a new file (`welcome_v2.json`), and a new manifest, with the next version number, points the same ID at it. Every older manifest keeps resolving what it resolved when it was published.
- **Artifacts refer to each other by ID.** A task template names its workflow and render config by ID, and the manifest decides which file each ID is. Changing a render config needs a new manifest, not a new task template.
- **Versions are contiguous from 1.** `LoadSet` probes upward until the loader reports `ErrNotFound`, so a gap hides every later manifest. Any other load error fails `LoadSet`; for S3 this means the reader needs `s3:ListBucket`, or a missing key is reported as access denied.
- **An explicit `version` keeps its path.** A row with a `version` is fetched by that version outside any flow (`GetExplicit`), so every manifest that lists it must point at the same file. `LoadSet` rejects one that does not.
- **The current version** is the highest one, unless `WithPinnedVersion` names an older one for a rollback. With a pin, `LoadSet` reads only manifests up to it, so a broken newer manifest cannot block the rollback.
- **Within a manifest, the unversioned row wins.** `Latest` returns the row without a version when there is one, even if the manifest also lists explicit versions of the same ID.

---

## Built-in loaders
Expand All @@ -138,7 +152,7 @@ There is no per-row `loader`: the registry has exactly one.
| `artifact/loaders/s3` | `s3.New(ctx, s3.Config{...})` | Reads objects from an S3 bucket; builds the client from `Region`/`Endpoint`/credentials (S3-compatible stores supported); `Prefix` scopes an in-bucket sub-folder for multi-deployment buckets |
| `artifact/loaders/github` | `github.New(github.Config{...})` | Reads files from a GitHub repo at a pinned `Ref` over the Contents API (`net/http`); `BasePath` scopes a sub-directory for multi-deployment repos. Set `UseRawHost` to fetch public repos via `raw.githubusercontent.com` (no token, no REST rate limit) |

Both wrap their underlying "not found" errors as `artifact.ErrNotFound` so callers can use `errors.Is`.
All wrap their underlying "not found" errors as `artifact.ErrNotFound` so callers can use `errors.Is`.

### Selecting a loader from config

Expand Down Expand Up @@ -190,6 +204,8 @@ m := testutil.MemLoader{
reg := artifact.NewRegistry(m)
```

Put `manifest_v1.json` (and later versions) in a `MemLoader` to test against a `Set`.

---

## Bridging to domain types
Expand Down
6 changes: 1 addition & 5 deletions artifact/adapter/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,15 +43,11 @@ To make a type `T` (from package `dom`) loadable via the artifact registry, crea
```
5. **Expose Fetch Helpers**:
```go
// reg is one manifest version's registry: set.Registry(ctx, version).
func Load(ctx context.Context, reg *artifact.Registry, id string) (dom.T, error) {
w, err := artifact.Latest[loadable](ctx, reg, id)
return w.T, err
}

func LoadVersion(ctx context.Context, reg *artifact.Registry, id, version string) (dom.T, error) {
w, err := artifact.Get[loadable](ctx, reg, id, version)
return w.T, err
}
```

By following this pattern:
Expand Down
15 changes: 4 additions & 11 deletions artifact/adapter/workflowdef/workflowdef.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ const Kind artifact.Kind = "workflow"

// loadable is the adapter: a local type (so we may legally define methods on it)
// that embeds the pure DSL type and satisfies artifact.Artifact + artifact.Parser.
// Unexported — callers use Load/LoadVersion and never see it.
// Unexported — callers use Load and never see it.
type loadable struct {
engine.WorkflowDefinition
}
Expand All @@ -34,17 +34,10 @@ func (w *loadable) Parse(raw []byte) error {
return nil
}

// Load returns the newest version of a workflow definition. Callers get a plain
// engine.WorkflowDefinition back — they never see the adapter or know "artifact"
// was involved in fetching it.
// Load returns a workflow definition from reg, which is one manifest version's registry. Callers
// get a plain engine.WorkflowDefinition back — they never see the adapter or know "artifact" was
// involved in fetching it.
func Load(ctx context.Context, reg *artifact.Registry, id string) (engine.WorkflowDefinition, error) {
w, err := artifact.Latest[loadable](ctx, reg, id)
return w.WorkflowDefinition, err
}

// LoadVersion returns a specific pinned version (e.g. to resume a running
// instance on the version it started with).
func LoadVersion(ctx context.Context, reg *artifact.Registry, id, version string) (engine.WorkflowDefinition, error) {
w, err := artifact.Get[loadable](ctx, reg, id, version)
return w.WorkflowDefinition, err
}
21 changes: 0 additions & 21 deletions artifact/adapter/workflowdef/workflowdef_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,27 +36,6 @@ func TestWorkflowDefAdapter(t *testing.T) {
}
})

t.Run("LoadVersion returns pinned version", func(t *testing.T) {
m := testutil.MemLoader{
"wf_v3.json": []byte(`{"id": "import_clearance", "name": "Clearance V3", "version": 3}`),
"wf_v4.json": []byte(`{"id": "import_clearance", "name": "Clearance V4", "version": 4}`),
}
reg := artifact.NewRegistry(m)
reg.RegisterArtifact("import_clearance", "workflow", "v3", "wf_v3.json")
reg.RegisterArtifact("import_clearance", "workflow", "v4", "wf_v4.json")

def, err := workflowdef.LoadVersion(context.Background(), reg, "import_clearance", "v3")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if def.Version != 3 {
t.Errorf("expected Version 3, got %d", def.Version)
}
if def.Name != "Clearance V3" {
t.Errorf("expected Name 'Clearance V3', got %q", def.Name)
}
})

t.Run("Load returns latest version when multiple exist", func(t *testing.T) {
m := testutil.MemLoader{
"wf_v1.json": []byte(`{"id": "import_clearance", "name": "Clearance V1", "version": 1}`),
Expand Down
63 changes: 41 additions & 22 deletions artifact/manifest.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,55 +7,74 @@ import (
"context"
"encoding/json"
"fmt"
"log/slog"
)

// ManifestFilename is the fixed name of the manifest, resolved at the loader's
// root. The manifest lives alongside the artifacts it catalogs, so a single
// loader resolves both; consumers do not choose its location. Point the loader's
// base (Root/Bucket/Repo) at wherever the manifest and artifacts reside.
const ManifestFilename = "manifest.json"
// ManifestPath returns the path of manifest version v, resolved at the loader's root:
// manifest_v1.json, manifest_v2.json, and so on. Manifests live alongside the artifacts they
// catalog, so one loader resolves both; consumers do not choose their location. Point the
// loader's base (Root/Bucket/Repo) at the directory that holds them.
//
// A manifest is immutable once published. A change to any artifact is a new file, listed by a
// new manifest with the next version number; see Set.
func ManifestPath(v int) string {
return fmt.Sprintf("manifest_v%d.json", v)
}

type ManifestConfig struct {
Artifacts []ManifestRow `json:"artifacts"`
}

// ManifestRow names the file one artifact resolves to in a manifest.
//
// Version is optional. A row without one is the artifact for that manifest: the manifest
// itself is the version. A row with one is an artifact requested by explicit version (static
// data a form fetches by ID and version, for example); such a row must point at the same path
// in every manifest that lists it, so the version alone identifies the content.
type ManifestRow struct {
ID string `json:"id"`
Kind Kind `json:"kind"`
Version string `json:"version"` // "" allowed for unversioned
Version string `json:"version"`
Path string `json:"path"`
}

// RegisterFromConfig applies every row via RegisterArtifact. Rows are validated
// so misconfiguration is caught at startup rather than at first access: id, kind,
// and path are all required (version may be "" for unversioned artifacts). There
// is no per-row loader — the registry has a single loader, so a row only names
// what to fetch, never how.
// RegisterFromConfig applies every row via RegisterArtifact. Rows are validated so
// misconfiguration is caught at startup rather than at first access: id, kind, and path are all
// required, and no two rows may share an id, kind and version. There is no per-row loader — the
// registry has a single loader, so a row only names what to fetch, never how.
func RegisterFromConfig(r *Registry, cfg ManifestConfig) error {
seen := make(map[Key]int, len(cfg.Artifacts))
for i, row := range cfg.Artifacts {
if row.ID == "" || row.Kind == "" || row.Path == "" {
return fmt.Errorf("manifest row %d (%q/%q): id, kind, and path are all required", i, row.ID, row.Kind)
}
k := Key{ID: row.ID, Kind: row.Kind, Version: row.Version}
if first, dup := seen[k]; dup {
return fmt.Errorf("manifest rows %d and %d both list %s/%s version %q", first, i, row.ID, row.Kind, row.Version)
}
seen[k] = i
}
for _, row := range cfg.Artifacts {
r.RegisterArtifact(row.ID, row.Kind, row.Version, row.Path)
}
slog.Info("artifact manifest registered", "count", len(cfg.Artifacts))
return nil
}

// LoadManifest fetches and unmarshals the manifest from the artifact source
// itself, through the same loader that fetches the artifacts — so the catalog
// and its artifacts share one origin. The manifest always lives at
// ManifestFilename ("manifest.json") relative to the loader's root; its location
// is a fixed convention, not a parameter.
func LoadManifest(ctx context.Context, l Loader) (ManifestConfig, error) {
// LoadManifest fetches and unmarshals manifest version v from the artifact source itself,
// through the same loader that fetches the artifacts, so the catalog and its artifacts share
// one origin. A missing manifest returns an error matching ErrNotFound.
func LoadManifest(ctx context.Context, l Loader, v int) (ManifestConfig, error) {
var cfg ManifestConfig
data, err := l.Load(ctx, ManifestFilename)
data, err := l.Load(ctx, ManifestPath(v))
if err != nil {
return cfg, fmt.Errorf("load manifest %q: %w", ManifestFilename, err)
return cfg, fmt.Errorf("load manifest %q: %w", ManifestPath(v), err)
}
return parseManifest(v, data)
}

func parseManifest(v int, data []byte) (ManifestConfig, error) {
var cfg ManifestConfig
if err := json.Unmarshal(data, &cfg); err != nil {
return cfg, fmt.Errorf("unmarshal manifest %q: %w", ManifestFilename, err)
return ManifestConfig{}, fmt.Errorf("unmarshal manifest %q: %w", ManifestPath(v), err)
}
return cfg, nil
}
10 changes: 7 additions & 3 deletions artifact/registry.go
Original file line number Diff line number Diff line change
Expand Up @@ -109,9 +109,10 @@ func Get[T Artifact](ctx context.Context, r *Registry, id, version string) (T, e
return loadAndParse[T](ctx, r, e, id, kind, version)
}

// Latest fetches and parses the newest version of an artifact, as type T. For an
// unversioned artifact (a single "" entry) it simply returns that entry, so it is
// also the natural "give me the current one" call for templates and schemas.
// Latest fetches and parses an artifact as type T: the row without an explicit version if
// the registry has one, otherwise the highest explicit version. In a manifest the row without
// a version is the artifact for that manifest, so Latest is the "give me this manifest's one"
// call for templates and schemas, even when the same manifest also lists explicit versions.
func Latest[T Artifact](ctx context.Context, r *Registry, id string) (T, error) {
var zero T
kind := kindOf[T]()
Expand All @@ -120,6 +121,9 @@ func Latest[T Artifact](ctx context.Context, r *Registry, id string) (T, error)
if !ok || len(versions) == 0 {
return zero, fmt.Errorf("%w: %s/%s", ErrNotFound, id, kind)
}
if e, ok := versions[""]; ok {
return loadAndParse[T](ctx, r, e, id, kind, "")
}

best, first := "", true
for v := range versions {
Expand Down
35 changes: 32 additions & 3 deletions artifact/registry_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -268,19 +268,48 @@ func TestManifest(t *testing.T) {
t.Errorf("expected required-fields error, got %v", err)
}
})

t.Run("RegisterFromConfig rejects two rows for one id, kind and version", func(t *testing.T) {
reg := artifact.NewRegistry(testutil.MemLoader{"a.json": []byte(`{"subject":"A"}`)})
cfg := artifact.ManifestConfig{Artifacts: []artifact.ManifestRow{
{ID: "form", Kind: "email", Path: "a.json"},
{ID: "form", Kind: "email", Path: "b.json"},
}}
err := artifact.RegisterFromConfig(reg, cfg)
if err == nil || !strings.Contains(err.Error(), "rows 0 and 1") {
t.Fatalf("expected a duplicate-row error naming both rows, got %v", err)
}
// Nothing is registered from a rejected manifest.
if _, err := artifact.Latest[fakeEmail](context.Background(), reg, "form"); !errors.Is(err, artifact.ErrNotFound) {
t.Errorf("expected ErrNotFound after a rejected manifest, got %v", err)
}
})

t.Run("RegisterFromConfig accepts one id under two kinds or two versions", func(t *testing.T) {
reg := artifact.NewRegistry(testutil.MemLoader{})
cfg := artifact.ManifestConfig{Artifacts: []artifact.ManifestRow{
{ID: "form", Kind: "email", Path: "a.json"},
{ID: "form", Kind: "custom_ruleset", Path: "b.json"},
{ID: "list", Kind: "email", Version: "1", Path: "c.json"},
{ID: "list", Kind: "email", Version: "2", Path: "d.json"},
}}
if err := artifact.RegisterFromConfig(reg, cfg); err != nil {
t.Fatalf("expected no error, got %v", err)
}
})
}

func TestLoadManifest(t *testing.T) {
t.Run("Load manifest from the source root", func(t *testing.T) {
m := testutil.MemLoader{
artifact.ManifestFilename: []byte(`{
artifact.ManifestPath(1): []byte(`{
"artifacts": [
{ "id": "test_id", "kind": "email", "version": "v1", "path": "path/to/email.json" }
]
}`),
}

cfg, err := artifact.LoadManifest(context.Background(), m)
cfg, err := artifact.LoadManifest(context.Background(), m, 1)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
Expand All @@ -290,7 +319,7 @@ func TestLoadManifest(t *testing.T) {
})

t.Run("Missing manifest surfaces the loader error", func(t *testing.T) {
cfg, err := artifact.LoadManifest(context.Background(), testutil.MemLoader{})
cfg, err := artifact.LoadManifest(context.Background(), testutil.MemLoader{}, 1)
if !errors.Is(err, artifact.ErrNotFound) {
t.Errorf("expected ErrNotFound, got %v", err)
}
Expand Down
Loading
Loading