diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 0000000..1844d5f --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,9 @@ +# Default ownership +* @BLCCoreStudio + +# Evaluation methodology and release-critical surfaces +/.github/ @BLCCoreStudio +/pyproject.toml @BLCCoreStudio +/src/ @BLCCoreStudio +/tests/ @BLCCoreStudio +/docs/ @BLCCoreStudio diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 0000000..b6cd74e --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,64 @@ +name: Bug report +description: Report a reproducible defect in TurkishEvalKit +title: "[Bug]: " +labels: + - bug +body: + - type: markdown + attributes: + value: Thanks for taking the time to provide a reproducible report. + - type: input + id: version + attributes: + label: TurkishEvalKit version or commit + placeholder: v0.x.y or commit SHA + validations: + required: true + - type: input + id: python + attributes: + label: Python version + placeholder: Python 3.x + validations: + required: true + - type: input + id: os + attributes: + label: Operating system + placeholder: Ubuntu 24.04, Windows 11, macOS 15, etc. + validations: + required: true + - type: textarea + id: reproduction + attributes: + label: Minimal reproduction + description: Include the smallest code, command, and input needed to reproduce the problem. + render: shell + validations: + required: true + - type: textarea + id: expected + attributes: + label: Expected behavior + validations: + required: true + - type: textarea + id: actual + attributes: + label: Actual behavior + validations: + required: true + - type: textarea + id: logs + attributes: + label: Traceback or logs + render: shell + - type: checkboxes + id: checks + attributes: + label: Pre-submission checks + options: + - label: I searched existing open and closed issues for duplicates. + required: true + - label: I removed secrets, private data, and credentials from this report. + required: true diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 0000000..5aef7e4 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,37 @@ +name: Feature request +description: Propose a new evaluation capability or improvement +title: "[Feature]: " +labels: + - enhancement +body: + - type: textarea + id: problem + attributes: + label: Evaluation problem + description: Describe the concrete evaluation or research problem before proposing an implementation. + validations: + required: true + - type: textarea + id: proposal + attributes: + label: Proposed capability + description: Explain the API, workflow, metric, report, or behavior you would add. + validations: + required: true + - type: textarea + id: evidence + attributes: + label: Evidence or use cases + description: Provide examples, papers, datasets, benchmarks, or real workflows that motivate the change when available. + - type: textarea + id: compatibility + attributes: + label: Compatibility and methodology impact + description: Could this change outputs, historical comparability, stored artifacts, or public APIs? + - type: checkboxes + id: checks + attributes: + label: Pre-submission checks + options: + - label: I searched existing issues and documentation for an equivalent capability. + required: true diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..1be00d9 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,36 @@ +## Summary + +Describe what this pull request changes and why. + +## Type of change + +- [ ] Bug fix +- [ ] Feature +- [ ] Evaluation or methodology change +- [ ] Documentation +- [ ] Refactor / maintenance +- [ ] CI / tooling + +## Validation + +List the commands, tests, datasets, or examples used to validate the change. + +```text +# Example +pytest +``` + +## Evaluation impact + +- [ ] This change can alter evaluation outputs, scores, rankings, calibration, or historical comparability. +- [ ] This change modifies a public API or serialized artifact. +- [ ] This change is backward compatible. + +If any evaluation result can change, explain why the new behavior is more correct and include before/after evidence where practical. + +## Checklist + +- [ ] Tests cover the changed behavior. +- [ ] Documentation is updated when user-visible behavior changes. +- [ ] No secrets, private datasets, credentials, or generated local artifacts are included. +- [ ] I reviewed the diff for unintended methodology changes. diff --git a/.github/release.yml b/.github/release.yml new file mode 100644 index 0000000..c595e70 --- /dev/null +++ b/.github/release.yml @@ -0,0 +1,33 @@ +changelog: + exclude: + labels: + - ignore-for-release + authors: + - dependabot + categories: + - title: Evaluation and methodology + labels: + - methodology + - evaluation + - title: Features + labels: + - enhancement + - feature + - title: Fixes + labels: + - bug + - fix + - title: Security + labels: + - security + - title: Documentation + labels: + - documentation + - title: Maintenance + labels: + - dependencies + - maintenance + - ci + - title: Other changes + labels: + - '*' diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 0000000..1150239 --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,36 @@ +name: CodeQL + +on: + push: + branches: [main] + pull_request: + branches: [main] + schedule: + - cron: '41 3 * * 2' + workflow_dispatch: + +permissions: + contents: read + security-events: write + +concurrency: + group: codeql-${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + analyze: + name: Analyze Python + runs-on: ubuntu-24.04 + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Initialize CodeQL + uses: github/codeql-action/init@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4 + with: + languages: python + - name: Analyze + uses: github/codeql-action/analyze@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4 + with: + category: /language:python diff --git a/.github/workflows/dependabot-auto-merge.yml b/.github/workflows/dependabot-auto-merge.yml new file mode 100644 index 0000000..b6a0557 --- /dev/null +++ b/.github/workflows/dependabot-auto-merge.yml @@ -0,0 +1,26 @@ +name: Dependabot Auto Merge + +on: + pull_request: + types: [opened, synchronize, reopened] + +permissions: + contents: write + pull-requests: write + +jobs: + dependabot: + if: github.event.pull_request.user.login == 'dependabot[bot]' && github.event.pull_request.base.ref == 'main' && github.repository_owner == 'BLCCoreStudio' + runs-on: ubuntu-24.04 + steps: + - name: Fetch Dependabot metadata + id: metadata + uses: dependabot/fetch-metadata@25dd0e34f4fe68f24cc83900b1fe3fe149efef98 # v3 + with: + github-token: ${{ github.token }} + - name: Enable auto-merge for patch updates + if: steps.metadata.outputs.update-type == 'version-update:semver-patch' + env: + GH_TOKEN: ${{ github.token }} + PR_URL: ${{ github.event.pull_request.html_url }} + run: gh pr merge --auto --squash "$PR_URL" diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..9168686 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,23 @@ +# Code of Conduct + +TurkishEvalKit is a professional, evidence-driven open-source project. + +## Expected behavior + +- Discuss code, datasets, metrics, and evaluation methodology respectfully and precisely. +- Separate reproducible findings from preferences or speculation. +- Critique implementations and research decisions rather than people. +- Disclose important assumptions, conflicts, and limitations when they affect a contribution. +- Respect maintainer decisions on project scope, compatibility, methodology, and release policy. + +## Unacceptable behavior + +Harassment, threats, discriminatory attacks, personal insults, deliberate disruption, spam, falsified evidence, or publishing another person's private information are not acceptable. + +## Enforcement + +The maintainer may edit, hide, lock, or remove content and may restrict participation when necessary. Severe or repeated violations may result in a permanent ban from project spaces. + +## Reporting + +Use an appropriate private GitHub channel for conduct concerns that should not be discussed publicly. Security vulnerabilities must follow `SECURITY.md`. diff --git a/SUPPORT.md b/SUPPORT.md new file mode 100644 index 0000000..354b128 --- /dev/null +++ b/SUPPORT.md @@ -0,0 +1,23 @@ +# Support + +TurkishEvalKit is maintained as a research-oriented open-source evaluation toolkit. Reports should contain enough evidence to distinguish implementation defects from disagreements about evaluation methodology. + +## Bugs + +Use the bug report template. Include the package version or commit, Python version, operating system, a minimal reproducible example, relevant input data where it can be shared, expected output, actual output, and the complete traceback or logs. + +## Evaluation or methodology concerns + +Describe the metric, dataset, rubric, evaluator, or pipeline involved and provide a minimal example that demonstrates the concern. Methodology changes require stronger evidence than ordinary implementation fixes because they can change historical comparability. + +## Feature requests + +Describe the evaluation problem first, then the proposed capability. Requests that duplicate existing primitives may be closed in favor of composition or documentation improvements. + +## Security + +Do not publish suspected vulnerabilities in public issues. Follow `SECURITY.md`. + +## Maintenance priority + +Priority is given to correctness, reproducibility, security, regressions, data integrity, compatibility, and changes that improve the toolkit for a broad set of evaluation workflows.