[Hypothesis] Add commit_review template for check-in scoring

[marialmeida1]

If we extend the Autograder with a new commit_review template, we can support contribution-oriented check-in evaluation through a
configurable and auditable scoring pipeline, instead of relying on a single AI judgment over commit payloads.

The goal is not to treat a check-in like a traditional code submission that must compile and run. Instead, the goal is to evaluate
a contribution artifact composed of commits, patches, file stats, and repository metadata.

Context

There is a recurring evaluation need around selected Git commits being submitted as a check-in, where the system must generate a
score and a short evaluation summary.

A purely LLM-centered approach is simple to bootstrap, but it becomes harder to calibrate, explain, and evolve across different
environments such as:

• educational collaboration
• peer review between individuals
• internal engineering contribution tracking

The Autograder already has the right architectural primitives for a stronger approach:

• configurable criteria trees
• pipeline-based execution
• reusable templates
• structured result trees and feedback
• optional AI stages

This suggests a new template specialized for contribution review rather than assignment execution.

Proposal

Introduce a new template named commit_review.

This template would accept structured contribution artifacts such as:

• commits.json
• aggregated_diff.patch
• changed_files.json
• checkin_context.json
• optionally repo_context.json

The template would evaluate a check-in using a criteria tree focused on measurable contribution signals instead of semantic or
narrative interpretation.

Criteria Tree

The criteria tree should be the core of this design.

Instead of producing a single opaque score, the template should break the evaluation into explicit scoring dimensions. This makes
the result easier to explain, easier to calibrate, and easier to evolve later.

For V1, the tree should be quantitative only and organized around four main subjects inside base:

• Activity Volume
• Change Distribution
• Validation Evidence
• Technical Risk

Possible structure:

• Base
  • Activity Volume
    • minimum total line changes
    • reasonable total line changes
    • minimum commit count
  • Change Distribution
    • maximum changed files
    • maximum directories touched
    • maximum single file change
  • Validation Evidence
    • minimum test file count when source changes
    • minimum test change ratio when source changes
    • documentation evidence ratio
  • Technical Risk
    • maximum generated change ratio
    • maximum binary change ratio
    • critical path change ratio

The tree may also include:

• Bonus
  • strong test coverage ratio
  • balanced code and test distribution
• Penalty
  • sensitive files detected
  • generated artifact noise
  • overscattered change penalty

This keeps the evaluation deterministic and decomposable while still allowing different score profiles in the future.

V1 Scope

The first version should be intentionally non-AI-first.

It should score only quantitative and structurally observable signals, such as:

• total changed lines
• number of commits
• number of changed files
• number of touched directories
• largest single file change
• ratio of test changes
• ratio of generated artifacts
• presence of binary files
• presence of sensitive files
• critical path change ratio

The first implementation should work as follows:

1. Ingest the structured contribution payload.
2. Run lightweight preflight validation on required files and payload format.
3. Extract normalized metrics from commits, files, paths, and patch stats.
4. Apply deterministic test functions through the criteria tree.
5. Produce a final score, result tree, and non-AI feedback derived from metric outcomes.

This gives us a deterministic, cheap, and explainable baseline before introducing subjective scoring.

Expected value (optional)

This direction would let us:

• move away from a single opaque AI evaluation step
• score contributions using explicit and configurable rubrics
• provide more transparent feedback to users
• support different evaluation profiles later for education, collaboration, or enterprise contexts
• add AI later as an optional layer instead of a hard dependency

It also aligns well with the existing Autograder design, where scoring is driven by criteria_config, pipeline steps, and reusable
templates.

Open questions (optional)

This proposal still needs validation around a few points:

• what exact payload shape should be sent to the Autograder
• how much of contribution scoring can be meaningfully modeled with pure structural metrics
• which metrics are genuinely useful versus redundant
• how thresholds and weights should be calibrated against real check-in data
• whether project-specific context should be introduced only later or already in early versions

Validation plan (optional)

• Run the V1 template against a sample of real check-ins
• Compare the resulting scores with human judgment
• Review whether each criteria-tree branch contributes useful signal
• Adjust thresholds and weights where the scoring is clearly misaligned

References (optional)

No response

[ArthurCRodrigues]

Plan proposal (template-focused; criteria weights and payload generation belong to grading config/submission, not the template):

1. Add CommitReviewTemplate (no sandbox) under autograder/template_library/commit_review.py (or a new commit_review/ package) with a clear test registry. This template only defines test functions and their parameters.
2. Implement a shared payload/metrics helper used by all tests. It should parse submission files like commits.json, changed_files.json, aggregated_diff.patch, checkin_context.json, and optional repo_context.json, then compute normalized metrics (commit_count, total_line_changes, changed_files, directories_touched, largest_file_change, test_change_ratio, generated/binary/sensitive ratios, critical_path_ratio, etc.). No AI; deterministic only.
3. Provide test functions that map to metric checks (min/max/ratio/boolean), each returning 0/100 with metric-aware reports. Add Pydantic config_schema for threshold validation so Build Tree fails early on bad configs.
4. Register the template in TemplateLibraryService._TEMPLATE_REGISTRY and export it from autograder/template_library/__init__.py. Add docs docs/template-library/commit_review.md and link from docs/core/template-library.md. Include a sample criteria_config tree in docs, but keep it explicitly as grading-config guidance.
5. Use existing grading config and submission contracts for payload requirements: enforce required files via setup_config.required_files in the grading config and keep pipeline steps unchanged. Add unit tests for the metrics parser and key test functions.

This keeps the template scope clean (test functions + docs) while the criteria tree weights/structure and payload composition remain in grading configuration and submission integration.