Skip to content

docs: add Django ORM ingestion scaffold#4

Open
charles-turner-bot wants to merge 2 commits into
ACCESS-NRI:mainfrom
charles-turner-bot:feat/django-orm-scaffold
Open

docs: add Django ORM ingestion scaffold#4
charles-turner-bot wants to merge 2 commits into
ACCESS-NRI:mainfrom
charles-turner-bot:feat/django-orm-scaffold

Conversation

@charles-turner-bot

Copy link
Copy Markdown
Contributor

Summary

Adds a minimal Django scaffold for a possible HTTP + ORM ingestion path.

This does not replace the current direct-to-Postgres flow. It just captures the shape of a Django middle layer in one file so it can be copied into a Django app later.

Why

Charles asked whether we could put a Django service in the middle so telemetry could be POSTed to an endpoint, filtered/validated through ORM-backed models, and then written to the database.

Short answer: yes, that architecture is perfectly viable here.

What changed

  • adds django_scaffold.py with:
    • ORM models matching conda_env_sessions, conda_env_packages, and conda_env_ingest_runs
    • package normalization logic aligned with the current ingestion script
    • a batched ingest_payloads() helper using bulk_create(..., ignore_conflicts=True)
    • an example ingest_telemetry_view() POST endpoint
  • README now documents the Django middle-layer option and intended flow

Scope / non-goals

  • no Django dependency is added to the current stack
  • no running Django service is introduced in Compose
  • no attempt is made to migrate the existing SQL views to Django models in this PR
  • this is scaffold material, not a full framework conversion

Closes #3.

Validation

  • python3 -m py_compile django_scaffold.py

@charles-turner-1

charles-turner-1 commented May 20, 2026

Copy link
Copy Markdown
Collaborator

@jo-basevi I think this is roughly the workflow you had in mind? I had my bot translate the straight to postgres worklow into something that we could post to tracking services and have handled there.

Lemme know if this looks dumb as rocks to you/ isn't what you had in mind! (I mostly got it to write this to understand the idea better)

@jo-basevi

Copy link
Copy Markdown
Collaborator

Yeah I've only had a very quick read through and I think it matches up with what I was thinking - e.g. some django managed tables, a view for handling ingestion, and sending post requests to the API.

@charles-turner-1

Copy link
Copy Markdown
Collaborator

Cool - I'll start turning this into something more workable then!

@charles-turner-1

Copy link
Copy Markdown
Collaborator

@jo-basevi was just about to get to this, will it make most sense to branch off main in tracking services? My head is swimming with those merge problems 😆

@jo-basevi

Copy link
Copy Markdown
Collaborator

Yeah, can branch off main for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Explore a Django middle layer for telemetry ingestion

3 participants