Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 1 addition & 14 deletions .github/workflows/sync-dfe-engine-types.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,7 @@
# File: .github/workflows/sync-dfe-engine-types.yml
# Purpose: Sync the OpenAPI specs to the dfe-ui types package
#
# Creates a PR in: https://github.com/hyperi-io/dfe-ui
# Target paths. The two specs are COPIED; both type files are then
# regenerated by `yarn generate` from the copied specs, so neither is listed as an
# input and both change in the PR:
# packages/dfe-engine-types/specs/openapi.json
# packages/dfe-engine-types/specs/openapi.e2e.json
# packages/dfe-engine-types/types/index.d.ts (generated)
# packages/dfe-engine-types/types/e2e.d.ts (generated)
#
# Auth: a GitHub App installation token (vars.GH_APP_CLIENT_ID + secrets.GH_APP_PRIVATE_KEY),
# scoped to the dfe-ui repo with contents + pull-requests write, which covers push,
# PR open/close and deleting superseded sync branches (--delete-branch on close)
# Optional var: DFE_UI_REPO - repo as owner/name (default: hyperi-io/dfe-ui)
# Note: Git clone URLs require owner/name; numeric repo IDs are not supported.
# How it runs and what it needs: docs/control-plane/type-safety-sync.md.

name: Sync dfe-engine-types

Expand Down
18 changes: 3 additions & 15 deletions .gitleaks.toml
Original file line number Diff line number Diff line change
@@ -1,38 +1,26 @@
# Project-specific gitleaks allowlist
# Extends the CI base configuration (ci/.gitleaks.toml)
#
# The CI runs gitleaks to detect secrets in your code. This file lets you
# exclude false positives specific to your project.
#
# Documentation: https://github.com/gitleaks/gitleaks#configuration
# Project false positives for gitleaks: https://github.com/gitleaks/gitleaks#configuration

[extend]
useDefault = true

[allowlist]
description = "Project-specific false positive allowlist"

# Paths to exclude from scanning (regex patterns)
paths = [
# Test files often contain fake tokens/secrets
'''tests/.*\.py''',
'''tests/fixtures/.*''',
# Example configuration files
'''.env.example''',
]

# Regex patterns for known false positives
regexes = [
'''test-token''',
'''example\.com''',
'''llamatron''',
# DFE API key format used in documentation examples
# The API key in the docs' examples.
'''dfe_ak_live_8a3f''',
# Makefile e2e ClickHouse wipe: curl -u takes shell variable references, never a
# literal credential. Both forms stay -- gitleaks scans the whole history.
# The Makefile's curl -u takes shell variables, not a credential. History holds both forms.
'''DFE_ENGINE_CLICKHOUSE_USERNAME:-default\}:\$\$\{DFE_ENGINE_CLICKHOUSE_PASSWORD''',
'''\$\$ch_user:\$\$ch_pass''',
]

# Stopwords - findings containing these are likely false positives
stopwords = ["example", "sample", "test", "mock", "fake", "dummy", "placeholder"]
10 changes: 3 additions & 7 deletions .hyperi-ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,11 @@ release:
library: "2.14.15"

test:
min_coverage: 70 # Current: 75%. Target: 80%. Ratchet up as coverage improves.
min_coverage: 70
native_tools: [helm]
python:
parallel: auto
# Peak RSS of one process in a full core run was 1206 MiB (/usr/bin/time -v, 11 workers).
# Just over one worker's peak RSS in a full core run.
worker_mem_mb: 1280
full:
# A release waits on a green full-tier run.
Expand All @@ -51,11 +51,7 @@ test:

quality:
python:
# semgrep: warn matches hyperi-ci's own default (quality/semgrep.py:36), so
# this is a deliberate pin rather than a downgrade. Its 38 findings are
# Flask and Django rules firing on a FastAPI codebase -- the SQL-injection
# hit is api/v1/queries.py:232, a raw-query endpoint behind
# require_action(query_execute), where taking a caller's SQL is the point.
# hyperi-ci's default. Its findings are Flask and Django rules on a FastAPI app, and the raw-query endpoint takes SQL on purpose.
semgrep: warn
# Above hyperi-ci's warn default: these three are at zero, so a new finding fails the PR.
vulture: blocking
Expand Down
81 changes: 31 additions & 50 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ versioning and changelog generation. All commits must follow this format:
[optional footer(s)]
```

### Types
The types and the release each one cuts:

| Type | Description | Version Bump |
|------|-------------|--------------|
Expand All @@ -33,8 +33,6 @@ versioning and changelog generation. All commits must follow this format:
| `chore` | Other changes that don't modify src or test files | None |
| `revert` | Reverts a previous commit | Varies |

### Breaking Changes

For breaking changes that require a major version bump, add `!` after the type
or include `BREAKING CHANGE:` in the footer:

Expand All @@ -44,7 +42,7 @@ feat!: remove deprecated API endpoints
BREAKING CHANGE: The /v1/users endpoint has been removed. Use /v2/users instead.
```

### Examples
Examples:

```text
feat(auth): add OAuth2 support for Google login
Expand All @@ -58,21 +56,10 @@ refactor(core)!: restructure module exports
BREAKING CHANGE: Named exports are now used instead of default exports.
```

### Scope

Scope is optional but recommended. Use it to indicate the area of the codebase
affected (e.g., `api`, `auth`, `core`, `cli`, `docs`).

## Semantic Versioning

This project follows [Semantic Versioning 2.0.0](https://semver.org/):

- **MAJOR** (X.0.0): Breaking changes that require users to modify their code
- **MINOR** (0.X.0): New features that are backwards-compatible
- **PATCH** (0.0.X): Bug fixes and minor improvements

Versions are automatically determined by semantic-release based on commit
messages. Do not manually update version numbers.
semantic-release works out the version from the commit types, following [Semantic Versioning 2.0.0](https://semver.org/). Never edit a version number by hand.

## License for Contributions

Expand All @@ -86,17 +73,16 @@ anniversary of its release.
## Running locally

To run the engine as a plain local process (a `uv` virtualenv, no containers or
Kubernetes) -- including the no-dependencies auth spike and how it wires to
dfe-hyperdx / dfe-ui / a local dfe-deploy clone -- see [docs/local-dev.md](docs/local-dev.md).
Kubernetes), with or without ClickHouse, and wire it to dfe-hyperdx, dfe-ui and a
local dfe-deploy clone, see [docs/local-dev.md](docs/local-dev.md).

## How to Contribute

1. **Fork the repository** and create your branch from `main`
2. **Make your changes** following the commit message format above
3. **Test your changes** to ensure they work as expected
4. **Submit a pull request** with a clear description of what you've done
Fork the repository and branch from `main`. Commit in the format above, test the
change, and open a pull request that says what it does.
A question that is not a bug goes in an issue.

### Pull Request Checklist
Before you open it, check:

- [ ] Commits follow the conventional commit format
- [ ] Tests pass (if applicable)
Expand All @@ -109,7 +95,7 @@ Open every pull request against `main`. `.github/workflows/ci.yml` fires `pull_r
If you stack anyway, two things will bite:

- Retarget every child to `main` BEFORE merging its parent, then push the child so the `pull_request` run fires. Merging a parent with `--delete-branch` deletes the child's base and GitHub closes the child. The two states deadlock: you cannot reopen a PR whose base branch is gone, and you cannot retarget a closed PR. Recovery is to recreate the base ref at any commit, reopen, retarget to `main`, then delete the ref again.
- A squash merge rewrites the history the child sits on, so replay only the child's own commits: `git rebase --onto main <old-base-head> <child-branch>`.
- A squash merge rewrites the history the child sits on, so replay the child's commits and nothing else: `git rebase --onto main <old-base-head> <child-branch>`.

## Testing your branch in a full stack

Expand All @@ -119,13 +105,13 @@ see running, gets a full stack: on one host with
[dfe-docker](https://github.com/hyperi-io/dfe-docker), or on a Kubernetes cluster
with [dfe-infra](https://github.com/hyperi-io/dfe-infra/blob/main/docs/deployment/README.md).
Both take the image tag or branch you nominate for each app. Where that stack
runs is your own concern and is never committed here.
runs is your concern and is never committed here.

## Run the checks locally before you push

HyperI projects gate every push through CI (lint, format, tests, secret scan,
dependency audit, build). Run the SAME checks locally first so your change lands
green instead of bouncing:
pass CI the first time:

```bash
hyperi-ci check
Expand All @@ -138,21 +124,17 @@ the same suite CI runs -- and reports what to fix. Run it before every push and
pull request. It is the single best way to give your change the best chance of
surviving CI.

## CI/CD Workflow

When your pull request is merged to `main`:

1. **semantic-release** analyses commit messages since the last release
2. Determines the next version number based on commit types
3. Generates/updates the CHANGELOG
4. Creates a new GitHub release with release notes
5. Publishes the package (if applicable)

This happens automatically - no manual intervention required.
## Releases

## Questions
A merge to `main` runs CI and releases nothing. A release runs when the merge
commit has a `Release: true` trailer, or when a maintainer dispatches the CI
workflow (`.github/workflows/ci.yml`) with `from-head=true` to release HEAD or
`tag=<existing tag>` to publish that tag again. hyperi-ci then:

If you have questions about contributing, please open an issue or contact us.
1. Works out the next version from the commit types since the last release.
2. Tags it and writes the GitHub release with its notes.
3. Publishes the container image and the Helm chart to GHCR. No wheel goes to
PyPI (`.hyperi-ci.yaml`, `release.destinations`).

## Context

Expand All @@ -175,7 +157,6 @@ System design and why it is built this way:
| `src/dfe_engine/schema/` | ClickHouse DDL planning and apply |
| `src/dfe_engine/source/` | Source definitions. Receiver and loader routing is COMPILED from these |
| `src/dfe_engine/gitops/` | every mutation is a git commit, and the deploy repo is the authority |
| `docs/superpowers/` | specs and plans, including designs that are NOT built yet |

### Commands that prove a change

Expand All @@ -189,28 +170,28 @@ hyperi-ci check --quick # what CI will say, faster
`ruff format` is available but the repo has not adopted it, and there is no `black`.

**`uv run pytest` does not run everything.** The default `addopts` deselects the
`integration`, `live` and `upstream` markers, so a green local run has not
`integration` and `live` markers, so a passing local run has not
touched a backing service. Pass `-m integration` to include them.

Green locally is not green in CI. **A CI run is not atomic against hyperi-ci**:
A local pass is not a CI pass. **A CI run is not atomic against hyperi-ci**:
reusable workflows resolve `@main` per JOB, so a push-event run reports success
with every real job SKIPPED. Read the `pull_request` run, per job, and check the
with every check job SKIPPED. Read the `pull_request` run, per job, and check the
durations -- a test job that "passed" in 0s did not run.

### What tends to bite

| Don't | Do | Why |
|---|---|---|
| Hardcode an app name in engine code | Add it to `dfe-infra/apps.yaml` | What an app IS is DATA. Adding one is a manifest edit plus a chart, never an engine release. |
| Treat this as an internal tool | Code to the injected config and interface | It is a PRODUCT other organisations deploy. Env-specifics -- cluster refs, cred paths, our fleet -- belong in private config repos, never here. |
| Treat this as an internal tool | Code to the injected config and interface | It is a PRODUCT other organisations deploy. Env-specifics -- cluster refs, cred paths, a deployment's hosts -- belong in that deployment's private config repo, never here. |
| Hand-edit compiled routing | Change the Source | Receiver and loader routing is derived, so the API reports a hand edit as DRIFT and re-syncs over it. |
| Mock a backing service | testcontainers, a real one | No mocks as proof. `pass` and `TODO` are not functionality. |
| Mock a backing service | testcontainers, a running one | No mocks as proof. `pass` and `TODO` are not functionality. |
| `from typing import List, Dict, Optional` | `list[str]`, `str \| None` | Built-in generics, and `requires-python = ">=3.14"`. |
| `datetime.utcnow()` | `datetime.now(UTC)` | Deprecated since 3.12. |
| stdlib `logging` | `from scalo.logger import logger` | House rule, so every service emits one structured JSON format. |
| Raw `httpx` with hand-rolled retry | `scalo.http.HttpClient` | House rule. Importing httpx for its exception and response TYPES is correct and expected. |
| `yaml.safe_load` | `dfe_engine.yaml_utils` | ruamel, YAML 1.2. YAML 1.1 reads `no` as boolean false. |
| `except Exception:` then a silent fallback | Catch the specific error, log with context, re-raise | 12 sites already return `False`/`None`/`[]` with no trace, out of 251 `except Exception:` sites in `src/`. An operator cannot tell failure from empty. |
| `except Exception:` then a silent fallback | Catch the specific error, log with context, re-raise | A bare fallback to `False`, `None` or `[]` leaves no trace, and an operator cannot tell failure from empty. |
| Assume a config key does something | Grep for its consumer | Config that parses and is never read is a recurring defect here. |
| Trust `gh issue list` | `gh issue view <n>` | It caps at 30 without `--limit`, so absence is not evidence of closure. |

Expand All @@ -234,9 +215,9 @@ moves.

| repo | how it interacts |
|---|---|
| dfe-infra | Deploys this engine, and owns `apps.yaml` -- a byte copy is vendored here at `appmgmt/apps.yaml` as the last resort in the resolution order. Lockstep: a catalogue entry the engine does not carry is not reflected by its management API. |
| dfe-infra | Deploys this engine, and owns `apps.yaml` -- a byte copy sits here at `src/dfe_engine/appmgmt/apps.yaml`, read when `DFE_ENGINE_APP_CATALOGUE_FILE` names no other file. Lockstep: a catalogue entry the engine does not carry is not reflected by its management API. |
| dfe-schemas | Ships the schema and DDL trees as a PACKAGE, not a submodule. Read out of the installed wheel unless `DFE_ENGINE_SCHEMAS_DIR` points elsewhere. |
| scalo-py | The shared Python library, plus a contract guard: this repo overrides scalo's runtime base image with its own literal, and its own test fails when the committed Dockerfile stops matching. |
| scalo-py | The shared Python library, plus a contract guard: this repo sets scalo's runtime base image as a literal, and a test here fails when the committed Dockerfile stops matching. |
| the Rust apps | Each emits its config schema (`<app> config-schema`), and the deployment mounts it at `DFE_ENGINE_APP_CONTRACT_DIR`. `/api/v1/services` takes every default and type from it, so nothing of an app's config is restated here. |
| logreducer | A declared dependency (`logreducer[enhanced]`). The sampler's `smart` mode, its default, and `anomaly` mode run it, imported lazily so the scikit-learn stack loads only when one of them does. |

Expand All @@ -245,10 +226,10 @@ moves.
| repo | how it interacts |
|---|---|
| dfe-infra | Pins this engine's image in TWO places: the dfe-engine chart and the hyperdx chart, which runs the engine image as its dashboards init container. Both move together. There is no schema-apply workload, because the engine applies schema at its own startup. |
| dfe-ui | Vendors this repo's OpenAPI spec and generates its scope types from an engine module. A workflow here re-vendors it by PR on a push to main. |
| dfe-ui | Copies this repo's OpenAPI specs into its `dfe-engine-types` package and generates its types from them. `.github/workflows/sync-dfe-engine-types.yml` opens that PR on dfe-ui when a push to main changes a spec. |

**The consequence, and it is the reason this section exists:** dfe-engine is the
ONLY controller of ClickHouse objects and Kafka topics. A change to schema,
source compilation or topic naming is a SUITE-WIDE move, not a local one --
every Rust service reads configuration this engine writes, and infra and docker
wait on it. A one-line change here can land in five other repositories.
wait on it.
Loading
Loading