Skip to content

Multi-org Student Data Files Updated with more synthetic data - #1116

Open
thelmick-unicon wants to merge 13 commits into
LIF-Initiative:mainfrom
thelmick-unicon:main
Open

Multi-org Student Data Files Updated with more synthetic data#1116
thelmick-unicon wants to merge 13 commits into
LIF-Initiative:mainfrom
thelmick-unicon:main

Conversation

@thelmick-unicon

Copy link
Copy Markdown
Contributor
Description of Change

Added additional synthetic data to the multi-org student data files.

Type of Change
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality
    to not work as expected)
  • Documentation update
  • Infrastructure/deployment change
  • Performance improvement
  • Code refactoring
  • Data Improvement
Project Area(s) Affected
  • bases/
  • components/
  • projects/
  • orchestrators/
  • frontends/
  • deployments/
  • cloudformation/ or sam/ templates
  • reference_data/
  • scripts/
  • test/ or e2e/
  • Database schema (migrations)
  • API endpoints
  • Documentation (docs/, READMEs, ARCHITECTURE.md, CLAUDE.md)

@bjagg bjagg left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overview

Reviewed as a data change rather than by reading 26k diff lines — validated the files themselves and diffed the semantics against main.

The good news first: all 12 persona files are valid JSON, cross-org name and SCHOOL_ASSIGNED_NUMBER consistency is preserved, no PascalCase-for-scalar violations are introduced, and nothing here looks like real PII. Matt's org3 record also gains the INSTITUTION_ASSIGNED_NUMBER sha256 that matches his org1 record, which is a genuine improvement for cross-org identity resolution.

Findings below are ordered by how much they'd change the merge decision.


1. Four org3 identifiers are removed, and the description doesn't mention it

All four org3 personas lose their Summit Valley University student ID:

Persona on main in this PR
Alan SCHOOL_ASSIGNED_NUMBER + SVU-20250001 SCHOOL_ASSIGNED_NUMBER only
Jenna + SVU-20250002 removed
Matt + SVU-20250005 removed
Renee + SVU-20250004 removed

The PR description says only "Added additional synthetic data," so this reads as an unintended side effect rather than a decision. It may well be deliberate — org3 is labelled "Regional University" in scripts/fix_sample_data_schema.py while the identifier says "Summit Valley University," so there's a real inconsistency someone might have been cleaning up. Either way it should be stated.

What it leaves stale. Three places still expect those identifiers, and none is touched by this PR:

  • scripts/generate_sample_users.py:1311 still emits "identifierType": "Summit Valley University student ID", with "university_id_prefix": "SVU-202500" at line 46 — so regenerating the sample data reintroduces exactly what this PR removes
  • projects/mongodb/sample_data/README.md:51 still describes Org3 as "made up Summit Valley University"
  • scripts/README_sample_users.md likewise

Tests should still passintegration_tests/test_05_cross_org.py:46 defaults to SCHOOL_ASSIGNED_NUMBER, which is preserved. But org3 loses its only org-specific identifier, which is precisely the input the identity-mapper work (#654, #388) will need to demonstrate cross-org resolution.

Either restore the identifiers, or update the generator and both READMEs so the data and its documentation agree.


2. Reference-key casing is now inconsistent within files

The dataset carries both spellings of the same keys:

PascalCase count camelCase count
OfferedByRefOrganization 109 offeredByRefOrganization 74
AssertedByRefOrganization 69 assertedByRefOrganization 38
demonstratesRefCompetency 22
awardedByRefOrganization 7

All four lowercase forms are new in this PR, and they're mixed within the same file rather than split by org — org1/Renee has 24 lowercase vs 16 PascalCase, org3/Jenna 23 vs 15.

CLAUDE.md and docs/specs/data-model-rules.md specify PascalCase for entity/object/array keys. The practical consequence: any consumer keying on the PascalCase name silently misses the lowercase half of the data.


3. refRefPosition is malformed

6 occurrences (4 in org1/Matt, 2 in org2/Alan), alongside 11 correct RefPosition.

Under the documented {relationship}Ref{Entity} convention — parse_reference_key rpartitions on the last "Ref"refRefPosition parses to relationship "ref" + entity Position, i.e. a relationship literally named "ref". The correct key is RefPosition.

Worth noting this PR removes the identical pre-existing defect refRefCourse while introducing this one, which suggests a copy-paste of the same bad pattern rather than a deliberate choice. It's also a live example of why #1026 wants to replace heuristic Ref-infix sniffing with an explicit marker.


4. Two unrelated changes are bundled in

  • scripts/fix_sample_data_schema.py — the Single-Org / single-org mapping changes from "Demo University" to "State University". That's a semantic change to a shared script, unmentioned in the description (commit f5982388, "Updated Demo University to State University for consistancy"). Worth confirming nothing else keys on "Demo University" — and there's a typo in the commit message.
  • .gitignore — adds .agents/. Harmless, but unrelated to a data PR.

5. Branch and commit hygiene

The head branch is the fork's main, so this PR carries four Merge remote-tracking branch 'upstream/main' commits interleaved with the six real ones, and it can't be rebased cleanly (currently BEHIND).

Separately, the commit subjects ("Expanded data for single org students", "Update gitignore") don't follow the Issue #XXX: description pattern commitlint.config.mjs enforces — and neither does the PR title, which becomes the squash subject given this repo's squash_merge_commit_title = PR_TITLE. Worth retitling to Issue #XXX: … before squashing if there's a tracking issue.


6. Minor

One new value carries trailing whitespace (a description ending in \n). I checked the whole diff — it's the only new instance, so this is genuinely trivial.


Verdict

Approve with changes. #1 is the one I'd resolve before merge — not because anything breaks, but because silently dropping identifiers that a generator script still produces and two READMEs still document will confuse whoever next regenerates this data.

#2 and #3 are cheap mechanical fixes and worth doing while the data is already being touched, since this is the fixture other work gets validated against.

thelmick-unicon and others added 2 commits August 10, 2026 17:39
…feedback

- Restore org3 Summit Valley student IDs (SVU-...) dropped in the first build
- Normalize camelCase reference keys to PascalCase (dataset convention)
- Fix malformed refRefPosition -> RefPosition (+ 1-elem list -> object reshape)
- Trim trailing/leading whitespace on string values

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@thelmick-unicon

Copy link
Copy Markdown
Contributor Author

@bjagg Thanks for the review! Updated based on the comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

2 participants