Refactor: Contribution-Aware Evaluation

[ArthurCRodrigues]

Overview

This issue tracks all changes needed to allow the autograder to evaluate the delta introduced by a set of files (e.g. a contribution) rather than treating every submission as a plain, context-free snapshot. Test functions gain access to which lines changed and which files are in scope for this evaluation — without the pipeline becoming domain-specific or aware of git, GitHub, or any external system.

All changes are additive and backward-compatible. Every existing caller continues to work unchanged.

---

Motivation

The autograder currently receives a flat Dict[str, SubmissionFile] and evaluates every file with equal weight. There is no way to tell the pipeline that only certain files are the subject of the evaluation, or that certain lines within those files are the ones that changed. This makes it impossible to:

• Scope structural analysis (AST parsing) to the files that actually matter
• Write TestFunctions that reason about what is new versus what was already there
• Carry arbitrary per-file context (e.g. a raw diff hunk) into test functions without polluting the core model with domain-specific vocabulary

---

Generality constraint

The autograder's core entities must not encode git, GitHub, or gamification concepts. The changes described here follow a strict rule: the pipeline knows about files and scope, not about commits, repositories, or diffs. Domain-specific context belongs in open-ended metadata fields and is the caller's responsibility to populate.

This constraint drives the key design decision in Step 1 below — replacing the originally proposed ContributionContext (which carried repo_full_name, base_commit, head_commit, commit_messages) with the leaner EvaluationScope.

---

Metadata refactor prerequisite

Before or alongside Step 1, audit and standardise metadata usage across the codebase:

• TestResultNode.metadata — already exists, typed as Dict[str, Any], used for execution metadata. ✓
• SubjectResultNode.metadata, CategoryResultNode.metadata, RootResultNode.metadata, ResultTree.metadata — exist but are never written to by any current code path. These are dormant extension points.
• Submission.submission_metadata (DB column) / SubmissionCreate.metadata (API schema) — already exists as an open-ended JSON column. The CloudExporter already uses it correctly, storing GitHub Actions environment variables (repository, commit_sha, run_id, actor, ref) here. See the section below for the exact role this field plays.
• SubmissionFile — has no metadata field today. Step 1 adds one.

No structural changes are needed to result tree metadata fields. The only alignment needed is a clear documented rule: git and domain context goes into submission_metadata / SubmissionFile.metadata, never into new typed fields on core pipeline models.

---

How metadata, EvaluationScope, and changed_lines work together

This section clarifies the role of each field, since they serve distinct purposes that are easy to conflate.

The separation of concerns

Field	Owned by	Read by pipeline	Purpose
evaluation_scope.scoped_files	Caller	Yes — StructuralAnalysisStep	Tells the pipeline which files to focus analysis on
SubmissionFile.changed_lines	Caller	Yes — StructuralAnalysisStep, test functions	Which line numbers inside a file were added or modified
SubmissionFile.metadata	Caller	No — passed through to test functions only	Per-file domain context (patch, change_status, etc.)
submission_metadata (top-level)	Caller	Never	Passive audit trail — commit SHA, repo name, run ID

submission_metadata is a receipt, not an input. It records provenance for humans and external systems. The pipeline never reads it.

SubmissionFile.metadata is an active passthrough. The pipeline carries it to TestFunction.execute() via **kwargs. Test functions that want it can read it; those that don't ignore it.

evaluation_scope and changed_lines are typed pipeline inputs. They are explicitly read by pipeline steps.

Concrete example

A gamification product calls the GitHub compare API for abc123..def456 and gets:

service/payment.py — modified — +45 -12 lines — changed lines 88-132
tests/test_payment.py — added — +5 lines

The product builds this submission request:

{
  "external_assignment_id": "group-7-repo",
  "external_user_id": "alice",
  "username": "alice",
  "evaluation_scope": {
    "scoped_files": ["service/payment.py", "tests/test_payment.py"]
  },
  "files": [
    {
      "filename": "service/payment.py",
      "content": "...full file content at HEAD...",
      "changed_lines": [88, 89, 90, 91, 92, 93, 132],
      "file_metadata": {
        "patch": "@@ -88,6 +88,51 @@ class PaymentService: ...",
        "change_status": "modified"
      }
    },
    {
      "filename": "tests/test_payment.py",
      "content": "...full file content at HEAD...",
      "changed_lines": [1, 2, 3, 4, 5],
      "file_metadata": {
        "change_status": "added"
      }
    }
  ],
  "metadata": {
    "commit_sha": "def456",
    "base_commit": "abc123",
    "repo_full_name": "org/repo"
  }
}

What each part does:

• evaluation_scope.scoped_files — StructuralAnalysisStep restricts AST parsing to these two files only. The rest of the repository (if sent) is ignored for structural analysis.
• changed_lines — a test function checking documentation coverage uses this to evaluate only the functions introduced by the contribution, not pre-existing code in the file.
• file_metadata — carries the raw patch and change_status. A test function that wants to reason about the diff hunk reads file.metadata.get("patch"). A test that only needs changed_lines ignores file_metadata entirely.
• Top-level metadata — commit_sha, base_commit, repo_full_name are stored in submission_metadata and never read by the pipeline. They allow the product to correlate a score record back to a specific commit, but the autograder does not act on them.

What a test function sees

A test that restricts its analysis to newly written code:

def execute(self, files, sandbox, *args, **kwargs):
    for file in files:
        changed = file.changed_lines or set()
        # parse AST, filter nodes to those intersecting changed_lines
        # evaluate only what was written in this contribution

A test that also uses the raw diff:

def execute(self, files, sandbox, *args, **kwargs):
    for file in files:
        patch = (file.metadata or {}).get("patch")
        change_status = (file.metadata or {}).get("change_status")
        if change_status == "added":
            # file is entirely new — evaluate the whole thing
        elif patch:
            # reason about what specifically changed using the diff hunk

The pipeline itself never inspects file.metadata. It is opaque to the autograder core and owned entirely by the caller and the test functions that choose to use it.

---

Step 1 — Extend SubmissionFile and add EvaluationScope

File: autograder/models/dataclass/submission.py

SubmissionFile changes

Add two optional fields:

Field	Type	Rationale
changed_lines	Optional[Set[int]]	1-indexed line numbers added or modified. Used directly by StructuralAnalysisStep to scope AST findings. Typed because the pipeline reads it.
metadata	Optional[Dict[str, Any]]	Open-ended per-file context. Callers may store patch, change_status, or anything else here. The pipeline never reads this — it is passed through to TestFunction.execute() via **kwargs for test functions that choose to use it.

Add is_contribution_aware convenience property returning True when changed_lines is not None.

New dataclass: EvaluationScope

@dataclass
class EvaluationScope:
    """
    Defines which files are the primary subject of this evaluation.
    When present, pipeline analysis steps (e.g. StructuralAnalysisStep)
    restrict their work to these files. When absent, all submission files
    are treated equally — preserving existing behaviour.
    """
    scoped_files: List[str]  # filenames that are the focus of evaluation

Add evaluation_scope: Optional[EvaluationScope] = None to Submission.

What is intentionally excluded from the core model: repo_full_name, base_commit, head_commit, commit_messages, DiffStats. These are git/domain concepts. Callers that need to carry this information should populate submission_metadata (submission-level) or SubmissionFile.metadata (file-level). The autograder never reads them.

---

Step 2 — Thread EvaluationScope through PipelineExecution

File: autograder/models/pipeline_execution.py

Add a convenience property:

@property
def evaluation_scope(self) -> Optional["EvaluationScope"]:
    return self.submission.evaluation_scope

Add EvaluationScope to the TYPE_CHECKING import block. No other changes.

---

Step 3 — Pass EvaluationScope into SubmissionGrader

File: autograder/services/grader/grader_service.py

Add evaluation_scope: Optional[EvaluationScope] = None to grade_from_tree() and forward it to SubmissionGrader.

File: autograder/services/grader/criteria_grader.py

Add evaluation_scope to SubmissionGrader.__init__ and store as self.evaluation_scope. Pass it as a kwarg in the test.test_function.execute(...) call alongside structural_analysis. Test functions that do not use it ignore it via **kwargs — no existing test function is affected.

---

Step 4 — Pull EvaluationScope from PipelineExecution in GradeStep

File: autograder/steps/grade_step.py

Add evaluation_scope=pipeline_exec.evaluation_scope to the grade_from_tree() call. One line addition.

---

Step 5 — Make StructuralAnalysisStep scope-aware

File: autograder/models/dataclass/structural_analysis_result.py

Add changed_lines: Dict[str, Set[int]] (default empty dict) to StructuralAnalysisResult. This carries per-file changed line sets forward so structural TestFunctions can scope their AST findings to new code only.

File: autograder/steps/structural_analysis_step.py

• When evaluation_scope is present on the submission, restrict parsing to files listed in evaluation_scope.scoped_files.
• After parsing each file, copy sub_file.changed_lines (if set) into changed_lines_map and include it in StructuralAnalysisResult.
• When evaluation_scope is absent, behaviour is identical to today — all submission files are parsed.

---

Step 6 — Update the web API layer

File: web/schemas/submission.py

Extend SubmissionFileData:

• Add changed_lines: Optional[List[int]] = None
• Add file_metadata: Optional[Dict[str, Any]] = None — maps to SubmissionFile.metadata. Named file_metadata at the API boundary to avoid collision with the top-level metadata field on SubmissionCreate.

Add EvaluationScopeData Pydantic model:

class EvaluationScopeData(BaseModel):
    scoped_files: List[str]

Add evaluation_scope: Optional[EvaluationScopeData] = None to SubmissionCreate.

File: web/api/v1/submissions.py

When building submission_files_dict, persist changed_lines and file_metadata from each SubmissionFileData. Pass evaluation_scope.model_dump() (when present) into GradingRequest.

File: web/service/grading_service.py

Add evaluation_scope: Optional[dict] = None to GradingRequest. In _run_pipeline, hydrate SubmissionFile with changed_lines (as Set[int]) and metadata from the stored dict. Reconstruct EvaluationScope from the raw dict before building AutograderSubmission.

---

Acceptance criteria

• SubmissionFile carries changed_lines and metadata as optional fields with None defaults
• EvaluationScope exists in autograder/models/dataclass/submission.py with a single scoped_files: List[str] field
• Submission carries evaluation_scope: Optional[EvaluationScope] = None
• PipelineExecution exposes evaluation_scope as a convenience property
• StructuralAnalysisStep only parses files in scoped_files when evaluation_scope is set; falls back to all files otherwise
• StructuralAnalysisResult carries changed_lines: Dict[str, Set[int]]
• SubmissionGrader passes evaluation_scope and per-file metadata to TestFunction.execute() via kwargs
• SubmissionCreate schema accepts evaluation_scope and per-file changed_lines / file_metadata
• All existing tests pass without modification
• No git, GitHub, or domain-specific field names appear in any file under autograder/
• Team alignment documented: git/domain context belongs in submission_metadata / SubmissionFile.metadata

---

Files touched

autograder/models/dataclass/submission.py                  Step 1
autograder/models/pipeline_execution.py                    Step 2
autograder/services/grader/grader_service.py               Step 3
autograder/services/grader/criteria_grader.py              Step 3
autograder/steps/grade_step.py                             Step 4
autograder/models/dataclass/structural_analysis_result.py  Step 5
autograder/steps/structural_analysis_step.py               Step 5
web/schemas/submission.py                                  Step 6
web/api/v1/submissions.py                                  Step 6
web/service/grading_service.py                             Step 6

---

Related

• ROADMAP.md § 1 — Refactor: Contribution-Aware Evaluation
• ContributionContext rejected in favour of EvaluationScope to preserve generality — see architectural review notes in session history