Feature: Data Science Template (data_science)

[ArthurCRodrigues]

Summary

A new built-in template for the autograder that supports grading data science assignments — exercises where students read datasets, perform transformations, train models, and produce predictions or analytical outputs. The template leverages the existing asset injection feature to deliver datasets into the sandbox at /tmp/, and introduces test functions designed to validate data manipulation results, model outputs, and statistical metrics.

---

Motivation

The current template library covers:

• input_output — Command-line programs with stdin/stdout
• web_dev — HTML/CSS/JS file validation
• api_testing — HTTP endpoint validation
• static_analysis — Code quality and algorithm verification

None of these are designed for the unique requirements of data science assignments, which involve:

1. Large dataset files that students must read (CSV, JSON, Parquet) — students should NOT include these in their submission.
2. Non-deterministic outputs — ML model metrics that require tolerance-based comparison (e.g., accuracy ≥ 0.85) rather than exact string matching.
3. Structured output validation — DataFrames, series, or JSON objects that need schema/shape/value checking.
4. Multi-step pipelines — Load data → preprocess → train → predict → export — where intermediate results matter.

The asset injection feature already solves the dataset delivery problem (S3 → /tmp/ via Base64 inject), but the existing expect_output test function can only do exact string comparison. Data science assignments need richer validation primitives.

---

Design Philosophy

1. Leverage existing infrastructure — No new pipeline steps or asset injection changes required. The template uses requires_sandbox = True and relies on assets already being available at /tmp/ when tests execute.
2. Composable tests — Each test function validates one aspect (output value, file artifact, metric threshold). Complex assignments combine multiple tests in the criteria tree.
3. Tolerance-first — All numeric comparisons support configurable tolerances by default.
4. Framework-agnostic — Tests validate outputs (stdout, files), not the student's choice of library. A student can use pandas, polars, or raw Python.

---

Test Functions

Test Function	Purpose
expect_csv_output	Validates generated CSV files (columns, shape, values with tolerance)
expect_metric	Extracts metrics from stdout via regex and validates against thresholds
expect_json_output	Validates generated JSON structure and values
expect_stdout_value	Extracts and compares individual values from stdout with tolerance
expect_model_artifact	Validates that a trained model file was produced

Each test function has its own detailed sub-issue linked below.

---

Sandbox Image Requirement

The current Python sandbox (Dockerfile.python) is a minimal Alpine image without scientific packages. Data science assignments require a new sandbox image variant:

Dockerfile.python-ds

FROM python:3.11-slim

RUN groupadd -r sandbox && useradd -r -g sandbox sandbox

RUN pip install --no-cache-dir \
    numpy==1.26.4 \
    pandas==2.2.1 \
    scikit-learn==1.4.1 \
    matplotlib==3.8.3 \
    seaborn==0.13.2

RUN rm -f /usr/bin/wget /usr/bin/curl
RUN find / -perm /6000 -type f -exec chmod a-s {} \; || true

WORKDIR /app
RUN chown sandbox:sandbox /app
VOLUME ["/app"]
USER sandbox
CMD ["python3"]

Integration with Language Pool: The existing Language enum and pool system would need a new variant (e.g., Language.PYTHON_DS or a tag-based approach) to select this image for data science assignments.

---

Architecture Integration

What Changes

Component	Change Type	Description
autograder/template_library/data_science.py	New file	Template class + test functions
autograder/template_library/__init__.py	Modified	Export DataScienceTemplate
autograder/translations/	Modified	Add translation keys for test messages
sandbox_manager/images/Dockerfile.python-ds	New file	Python data science sandbox image
sandbox_manager/models/sandbox_models.py	Modified	Add PYTHON_DS language variant
docs/template-library/data_science.md	New file	Template documentation

What Does NOT Change

Component	Reason
Pipeline steps	Template tests execute within the existing GRADE step
Asset injection	Already supports /tmp/ file delivery — no changes needed
PreFlightStep	Already handles assets + setup_commands
SetupConfig model	Already supports the assets field
CriteriaTree / ResultTree	Standard test function results slot directly into the tree
Sandbox lifecycle	Standard sandbox acquisition and release

---

Configuration Example

{
  "external_assignment_id": "tp2-restaurantes",
  "template_name": "data_science",
  "languages": ["python"],
  "setup_config": {
    "assets": [
      {
        "source": "datasets/tp2/RESTAURANTES.CSV",
        "target": "/tmp/RESTAURANTES.CSV",
        "read_only": true
      }
    ],
    "python": {
      "required_files": ["analysis.py"],
      "setup_commands": ["pip install -r requirements.txt --quiet 2>/dev/null || true"]
    }
  },
  "criteria_config": {
    "base": {
      "weight": 100,
      "subjects": [
        {
          "subject_name": "Data Loading",
          "weight": 20,
          "tests": [
            {
              "name": "expect_stdout_value",
              "parameters": {
                "program_command": "python3 analysis.py stats",
                "extraction_pattern": "total_rows:\\s*(?P<value>\\d+)",
                "expected_value": 1500,
                "tolerance": 0
              },
              "weight": 100
            }
          ]
        },
        {
          "subject_name": "Preprocessing",
          "weight": 30,
          "tests": [
            {
              "name": "expect_csv_output",
              "parameters": {
                "program_command": "python3 analysis.py preprocess",
                "artifact_path": "cleaned.csv",
                "expected_columns": ["name", "cuisine", "rating", "price_range", "city"],
                "expected_shape": [1420, 5]
              },
              "weight": 100
            }
          ]
        },
        {
          "subject_name": "Model Training",
          "weight": 30,
          "tests": [
            {
              "name": "expect_model_artifact",
              "parameters": {
                "program_command": "python3 analysis.py train",
                "artifact_path": "model.pkl",
                "min_size_bytes": 2048
              },
              "weight": 30
            },
            {
              "name": "expect_metric",
              "parameters": {
                "program_command": "python3 analysis.py train",
                "metric_pattern": "accuracy:\\s*(?P<value>[\\d.]+)",
                "condition": ">=",
                "threshold": 0.75
              },
              "weight": 70
            }
          ]
        },
        {
          "subject_name": "Predictions",
          "weight": 20,
          "tests": [
            {
              "name": "expect_csv_output",
              "parameters": {
                "program_command": "python3 analysis.py predict",
                "artifact_path": "predictions.csv",
                "expected_columns": ["id", "predicted_rating"],
                "expected_shape": [300, 2],
                "tolerance": 0.5
              },
              "weight": 100
            }
          ]
        }
      ]
    }
  }
}

---

Implementation Phases

Phase 1: Core Template (MVP)

1. Create DataScienceTemplate class implementing the Template ABC
2. Implement expect_csv_output test function
3. Implement expect_metric test function
4. Implement expect_stdout_value test function
5. Register template in template_library/__init__.py
6. Add translation keys (pt-BR and en)
7. Write unit tests for each test function
8. Write template documentation

Phase 2: Extended Tests + Sandbox Image

9. Implement expect_json_output test function
10. Implement expect_model_artifact test function
11. Create Dockerfile.python-ds with scientific packages
12. Integrate data science image into the language pool system
13. Write integration tests with real sandbox execution

Phase 3: Documentation & Examples

14. Create example grading configuration in examples/
15. Add data science section to the main README template table
16. Upload sample datasets to S3 for demo purposes

---

Open Questions

1. Image selection strategy — Should we add a Language.PYTHON_DS enum variant, or use a tag/label system that allows flexible image selection per assignment?
2. Partial scoring for expect_csv_output — Proportional (80% values correct = 80 points) or binary (all-or-nothing)?
3. Setup commands for pip install — Allow any packages (network-isolated sandbox), whitelist specific packages, or pre-install everything?
4. Timeout handling — Default running_timeout: 120s may not be enough for model training. Per-assignment timeout in setup_config?
5. Reusing expect_file_artifact — Should new tests inherit from the existing ExpectFileArtifactTest in input_output, or be independent?

---

Risks & Mitigations

Risk	Mitigation
Large datasets exceeding Base64 injection limits (>~10MB)	Document max asset size. Future: streaming injection or volume mounts.
Student scripts consuming excessive memory/CPU	Existing container resource limits apply. Document recommended limits.
Non-reproducible model metrics (randomness)	Recommend random_state in instructions. Use generous tolerances.
Alpine vs. slim image compatibility	Use python:3.11-slim (Debian-based) for the DS image.
Network isolation preventing pip install	Pre-install common packages in image.

---

Sub-Issues

• Data Science Template: implement expect_csv_output test function #339 — expect_csv_output test function
• Data Science Template: implement expect_metric test function #340 — expect_metric test function
• Data Science Template: implement expect_json_output test function #341 — expect_json_output test function
• Data Science Template: implement expect_stdout_value test function #342 — expect_stdout_value test function
• Data Science Template: implement expect_model_artifact test function #343 — expect_model_artifact test function