feat: add contribution-aware evaluation scope and file metadata

[matheusmra]

Summary

This pull request introduces contribution-aware evaluation context without coupling the core grading engine to Git, GitHub, repositories, commits, pull requests, or any other platform-specific concept.

The implementation adds two typed inputs that the pipeline is allowed to understand:

1. EvaluationScope, which identifies the submitted files that are the primary subject of the current evaluation.
2. SubmissionFile.changed_lines, which identifies the one-indexed lines that were added or modified inside a file.

It also adds SubmissionFile.metadata as an opaque per-file passthrough channel. The pipeline transports this metadata to test functions but never interprets it. This allows callers to attach information such as a raw patch, a change status, provider data, review annotations, or other repository-derived context without expanding the core model every time a new integration needs additional data.

All new fields are optional and default to None. Existing callers that submit ordinary context-free snapshots continue to use the same behavior.

Stack and base branch

This is intentionally a stacked pull request.

• Base branch: 357-feature-score-vector
• Parent pull request: feat: introduce score_vector output mapping and post-pipeline baseline comparison #360
• Head branch: 355-refactor-contribution-aware-evaluation

The local branch was created from the head of #360. Targeting 357-feature-score-vector keeps this review limited to the contribution-aware evaluation work instead of including the six commits and approximately 1,400 lines already under review in the parent pull request.

After #360 is merged, this pull request can be retargeted or rebased onto main without changing the intended feature diff.

Problem

Before this change, a submission was only a filename-to-file mapping. Every file was treated as an equally relevant snapshot, and the pipeline had no typed way to answer either of the following questions:

• Which submitted files are the actual subject of this evaluation?
• Which lines inside a submitted file were introduced or modified by the evaluated change?

This limitation prevented callers from efficiently evaluating a delta while still sending complete file contents. It also encouraged platform integrations to consider adding repository-specific concepts directly to the core engine, which would violate the architectural boundary between the general-purpose autograder and its adapters.

The implementation resolves that tension by separating typed pipeline inputs from opaque integration context.

Architectural principles

The core understands evaluation scope, not source-control systems

EvaluationScope contains only a list of filenames. It does not contain repository names, commit identifiers, branch names, pull request numbers, commit messages, compare references, or provider-specific data.

This keeps the autograder/ package reusable for any caller that needs delta-aware evaluation, including callers that derive scope from an uploaded archive, an editor session, a content management system, a code review platform, or another source.

Typed fields are reserved for data that pipeline steps actively read

The pipeline actively reads:

• EvaluationScope.scoped_files
• SubmissionFile.changed_lines

These values therefore have explicit types and documented behavior.

Domain context remains opaque

SubmissionFile.metadata is typed only as Optional[Dict[str, Any]]. The pipeline does not inspect its keys or values.

Test functions can access the same metadata through:

• SubmissionFile.metadata on each object in the files argument.
• The filename-keyed file_metadata mapping passed through execute(**kwargs).

Submission-level API metadata remains a passive audit and correlation record stored as submission_metadata. It is not converted into a typed core pipeline field and is not read during grading.

Detailed implementation

Core submission models

SubmissionFile now supports:

• changed_lines: Optional[Set[int]]
• metadata: Optional[Dict[str, Any]]
• is_contribution_aware, which reports whether changed-line information was explicitly supplied.

An explicitly empty changed-line set is different from missing information. The convenience property therefore checks changed_lines is not None instead of relying on truthiness.

The new EvaluationScope dataclass contains:

scoped_files: List[str]

Submission accepts an optional evaluation_scope with a None default.

Pipeline execution

PipelineExecution.evaluation_scope exposes the scope from the active submission. This keeps access consistent for pipeline steps while preserving a single source of truth on Submission.

Structural analysis

StructuralAnalysisStep now:

1. Reads the optional evaluation scope.
2. Converts scoped_files to a set for efficient membership checks.
3. Skips files outside the scope when a scope is present.
4. Preserves the existing code-file heuristic.
5. Parses all eligible files when no scope is provided.
6. Copies caller-supplied changed-line sets into the structural analysis result.

An explicit empty scope results in no files being parsed. An absent scope retains the previous all-files behavior.

StructuralAnalysisResult now includes:

changed_lines: Dict[str, Set[int]]

The field uses an empty-dictionary factory and was added after the existing defaulted fields to preserve compatibility for callers that may construct the dataclass positionally.

Grading and test-function context

GradeStep forwards PipelineExecution.evaluation_scope to GraderService.

GraderService forwards it into SubmissionGrader.

For every test node, SubmissionGrader:

1. Resolves the test's file target using the existing behavior.
2. Builds a filename-keyed metadata mapping from only the resolved target files.
3. Calls TestFunction.execute() with both evaluation_scope and file_metadata.

The runtime pipeline values take precedence over same-named criteria parameters, consistent with the existing handling of injected runtime context.

Filtering file_metadata by file_target prevents unrelated file context from leaking into tests that are intentionally scoped to a subset of the submission.

Web API schema

Each submitted file can now include:

{
  "filename": "service/payment.py",
  "content": "full file content",
  "changed_lines": [88, 89, 90],
  "file_metadata": {
    "change_status": "modified",
    "patch": "@@ -88,6 +88,12 @@"
  }
}

Submission requests can also include:

{
  "evaluation_scope": {
    "scoped_files": [
      "service/payment.py",
      "tests/test_payment.py"
    ]
  }
}

file_metadata is deliberately named differently from top-level metadata at the API boundary:

• file_metadata becomes SubmissionFile.metadata and is available to tests.
• Top-level metadata becomes database submission_metadata and remains passive provenance.

Web persistence and hydration

The submission endpoint stores changed_lines and file_metadata with each file record and serializes the Pydantic evaluation scope to a plain dictionary for the background grading request.

The grading service reconstructs:

• changed_lines lists as Set[int]
• file_metadata as SubmissionFile.metadata
• the raw scope dictionary as an EvaluationScope dataclass

Missing fields remain None, preserving compatibility with submissions created before this feature.

End-to-end data flow

The complete flow is:

1. A caller determines which files and lines belong to the evaluated change.
2. The caller sends full file content together with optional changed lines and opaque metadata.
3. The web adapter stores the JSON-compatible representation.
4. The background grading service reconstructs typed core objects.
5. PipelineExecution exposes the evaluation scope.
6. StructuralAnalysisStep parses only scoped, eligible source files.
7. StructuralAnalysisResult carries parsed roots and changed-line sets.
8. GradeStep forwards scope and structural context.
9. SubmissionGrader resolves each test's file target.
10. The test function receives targeted files, scope, structural analysis, and targeted file metadata.

At no point does the core engine need to know how the caller calculated the scope or metadata.

Backward compatibility

This change is additive.

• SubmissionFile.changed_lines defaults to None.
• SubmissionFile.metadata defaults to None.
• Submission.evaluation_scope defaults to None.
• GraderService.grade_from_tree() accepts evaluation_scope=None.
• SubmissionGrader accepts evaluation_scope=None.
• StructuralAnalysisResult.changed_lines defaults to an empty dictionary.
• API requests without the new fields continue to validate.
• Stored file records without the new keys hydrate with None.
• Structural analysis still parses all eligible files when no scope is provided.
• Existing test functions continue to ignore injected context through **kwargs.

No scoring rules, result-tree behavior, sandbox behavior, template registration, or test-function contracts were removed or changed incompatibly.

Test coverage

The test suite covers the feature at each architectural boundary.

Model coverage

• Optional defaults for changed lines and metadata.
• is_contribution_aware behavior for missing and explicitly empty changed-line sets.
• Optional submission evaluation scope.

Pipeline coverage

• PipelineExecution.evaluation_scope.
• GradeStep forwarding the same scope instance into the grader service.
• Structural parsing restricted to scoped files.
• Existing all-files parsing when scope is absent.
• Changed-line propagation into StructuralAnalysisResult.

Grader coverage

• Scope forwarding into TestFunction.execute().
• Filename-keyed metadata forwarding.
• Metadata filtering based on file_target.
• Preservation of the original SubmissionFile objects.

API and service coverage

• Pydantic acceptance of evaluation scope, changed lines, and file metadata.
• Persistence of the new file fields in the submission record.
• Serialization of the scope into GradingRequest.
• Reconstruction of changed-line sets, file metadata, and EvaluationScope.

Rich repository metadata regression coverage

tests/unit/test_file_metadata_passthrough.py exercises a realistic, deeply nested repository metadata payload containing:

• provider information
• change status
• raw patch text
• blob identifiers and URLs
• addition, deletion, and change statistics
• labels and requested reviewers
• boolean review state
• line annotations
• explicit null values

The test verifies that:

• hydration does not mutate the stored representation
• nested data remains unchanged
• the same metadata object is retained as opaque context
• changed lines are converted to a set
• the evaluation scope is reconstructed
• only target-file metadata reaches the test function
• metadata from an unrelated file does not leak through file_metadata

Documentation

The documentation now explains:

• the new API request fields
• the difference between typed scope, changed lines, file metadata, and submission metadata
• metadata ownership and pipeline behavior
• structural analysis scoping
• changed-line propagation
• backward-compatible behavior when scope is omitted
• the rule that platform-specific context belongs in open-ended metadata instead of typed core fields

Validation

The final committed state was validated with:

pytest -q tests/unit tests/web
644 passed

python3 scripts/validate_docs.py
All documentation checks passed

Pylint completed successfully for all changed Python files. The dedicated rich metadata regression test received a 10.00/10 pylint score. The broader changed-file lint run retained only the repository's existing advisory complexity and test-docstring warnings and exited successfully.

git diff --check also completed without whitespace errors.

Reviewer guidance

A useful review order is:

1. autograder/models/dataclass/submission.py for the public core model.
2. autograder/steps/structural_analysis_step.py for scope behavior.
3. autograder/services/grader/criteria_grader.py for test-function context.
4. web/schemas/submission.py and web/service/grading_service.py for the API round trip.
5. tests/unit/test_file_metadata_passthrough.py for the strongest end-to-end metadata invariant.
6. The documentation updates for the intended ownership rules.

Out of scope

This pull request does not:

• call a repository provider API
• calculate changed lines from a patch
• introduce repository, commit, branch, pull request, or diff models into the core package
• interpret any key inside SubmissionFile.metadata
• automatically filter every existing structural test by changed lines
• change result-tree scoring
• alter sandbox provisioning or execution

Those behaviors can be implemented by adapters or individual test functions using the general context introduced here.

Closes #355

[ArthurCRodrigues]

@matheusmra let's add a contribution related test to static analysis template in order to better visualize if these changes are working fine. I'll create an issue soon.

Let's keep the PR opened until then

[matheusmra]

@matheusmra let's add a contribution related test to static analysis template in order to better visualize if these changes are working fine. I'll create an issue soon.

Let's keep the PR opened until then

Alright 👍