Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Default ownership
* @BLCCoreStudio

# Evaluation methodology and release-critical surfaces
/.github/ @BLCCoreStudio
/pyproject.toml @BLCCoreStudio
/src/ @BLCCoreStudio
/tests/ @BLCCoreStudio
/docs/ @BLCCoreStudio
64 changes: 64 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
name: Bug report
description: Report a reproducible defect in TurkishEvalKit
title: "[Bug]: "
labels:
- bug
body:
- type: markdown
attributes:
value: Thanks for taking the time to provide a reproducible report.
- type: input
id: version
attributes:
label: TurkishEvalKit version or commit
placeholder: v0.x.y or commit SHA
validations:
required: true
- type: input
id: python
attributes:
label: Python version
placeholder: Python 3.x
validations:
required: true
- type: input
id: os
attributes:
label: Operating system
placeholder: Ubuntu 24.04, Windows 11, macOS 15, etc.
validations:
required: true
- type: textarea
id: reproduction
attributes:
label: Minimal reproduction
description: Include the smallest code, command, and input needed to reproduce the problem.
render: shell
validations:
required: true
- type: textarea
id: expected
attributes:
label: Expected behavior
validations:
required: true
- type: textarea
id: actual
attributes:
label: Actual behavior
validations:
required: true
- type: textarea
id: logs
attributes:
label: Traceback or logs
render: shell
- type: checkboxes
id: checks
attributes:
label: Pre-submission checks
options:
- label: I searched existing open and closed issues for duplicates.
required: true
- label: I removed secrets, private data, and credentials from this report.
required: true
37 changes: 37 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: Feature request
description: Propose a new evaluation capability or improvement
title: "[Feature]: "
labels:
- enhancement
body:
- type: textarea
id: problem
attributes:
label: Evaluation problem
description: Describe the concrete evaluation or research problem before proposing an implementation.
validations:
required: true
- type: textarea
id: proposal
attributes:
label: Proposed capability
description: Explain the API, workflow, metric, report, or behavior you would add.
validations:
required: true
- type: textarea
id: evidence
attributes:
label: Evidence or use cases
description: Provide examples, papers, datasets, benchmarks, or real workflows that motivate the change when available.
- type: textarea
id: compatibility
attributes:
label: Compatibility and methodology impact
description: Could this change outputs, historical comparability, stored artifacts, or public APIs?
- type: checkboxes
id: checks
attributes:
label: Pre-submission checks
options:
- label: I searched existing issues and documentation for an equivalent capability.
required: true
36 changes: 36 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
## Summary

Describe what this pull request changes and why.

## Type of change

- [ ] Bug fix
- [ ] Feature
- [ ] Evaluation or methodology change
- [ ] Documentation
- [ ] Refactor / maintenance
- [ ] CI / tooling

## Validation

List the commands, tests, datasets, or examples used to validate the change.

```text
# Example
pytest
```

## Evaluation impact

- [ ] This change can alter evaluation outputs, scores, rankings, calibration, or historical comparability.
- [ ] This change modifies a public API or serialized artifact.
- [ ] This change is backward compatible.

If any evaluation result can change, explain why the new behavior is more correct and include before/after evidence where practical.

## Checklist

- [ ] Tests cover the changed behavior.
- [ ] Documentation is updated when user-visible behavior changes.
- [ ] No secrets, private datasets, credentials, or generated local artifacts are included.
- [ ] I reviewed the diff for unintended methodology changes.
33 changes: 33 additions & 0 deletions .github/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
changelog:
exclude:
labels:
- ignore-for-release
authors:
- dependabot
categories:
- title: Evaluation and methodology
labels:
- methodology
- evaluation
- title: Features
labels:
- enhancement
- feature
- title: Fixes
labels:
- bug
- fix
- title: Security
labels:
- security
- title: Documentation
labels:
- documentation
- title: Maintenance
labels:
- dependencies
- maintenance
- ci
- title: Other changes
labels:
- '*'
36 changes: 36 additions & 0 deletions .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: CodeQL

on:
push:
branches: [main]
pull_request:
branches: [main]
schedule:
- cron: '41 3 * * 2'
workflow_dispatch:

permissions:
contents: read
security-events: write

concurrency:
group: codeql-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
analyze:
name: Analyze Python
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Initialize CodeQL
uses: github/codeql-action/init@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
with:
languages: python
- name: Analyze
uses: github/codeql-action/analyze@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4
with:
category: /language:python
26 changes: 26 additions & 0 deletions .github/workflows/dependabot-auto-merge.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: Dependabot Auto Merge

on:
pull_request:
types: [opened, synchronize, reopened]

permissions:
contents: write
pull-requests: write

jobs:
dependabot:
if: github.event.pull_request.user.login == 'dependabot[bot]' && github.event.pull_request.base.ref == 'main' && github.repository_owner == 'BLCCoreStudio'
runs-on: ubuntu-24.04
steps:
- name: Fetch Dependabot metadata
id: metadata
uses: dependabot/fetch-metadata@25dd0e34f4fe68f24cc83900b1fe3fe149efef98 # v3
with:
github-token: ${{ github.token }}
- name: Enable auto-merge for patch updates
if: steps.metadata.outputs.update-type == 'version-update:semver-patch'
env:
GH_TOKEN: ${{ github.token }}
PR_URL: ${{ github.event.pull_request.html_url }}
run: gh pr merge --auto --squash "$PR_URL"
23 changes: 23 additions & 0 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Code of Conduct

TurkishEvalKit is a professional, evidence-driven open-source project.

## Expected behavior

- Discuss code, datasets, metrics, and evaluation methodology respectfully and precisely.
- Separate reproducible findings from preferences or speculation.
- Critique implementations and research decisions rather than people.
- Disclose important assumptions, conflicts, and limitations when they affect a contribution.
- Respect maintainer decisions on project scope, compatibility, methodology, and release policy.

## Unacceptable behavior

Harassment, threats, discriminatory attacks, personal insults, deliberate disruption, spam, falsified evidence, or publishing another person's private information are not acceptable.

## Enforcement

The maintainer may edit, hide, lock, or remove content and may restrict participation when necessary. Severe or repeated violations may result in a permanent ban from project spaces.

## Reporting

Use an appropriate private GitHub channel for conduct concerns that should not be discussed publicly. Security vulnerabilities must follow `SECURITY.md`.
23 changes: 23 additions & 0 deletions SUPPORT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Support

TurkishEvalKit is maintained as a research-oriented open-source evaluation toolkit. Reports should contain enough evidence to distinguish implementation defects from disagreements about evaluation methodology.

## Bugs

Use the bug report template. Include the package version or commit, Python version, operating system, a minimal reproducible example, relevant input data where it can be shared, expected output, actual output, and the complete traceback or logs.

## Evaluation or methodology concerns

Describe the metric, dataset, rubric, evaluator, or pipeline involved and provide a minimal example that demonstrates the concern. Methodology changes require stronger evidence than ordinary implementation fixes because they can change historical comparability.

## Feature requests

Describe the evaluation problem first, then the proposed capability. Requests that duplicate existing primitives may be closed in favor of composition or documentation improvements.

## Security

Do not publish suspected vulnerabilities in public issues. Follow `SECURITY.md`.

## Maintenance priority

Priority is given to correctness, reproducibility, security, regressions, data integrity, compatibility, and changes that improve the toolkit for a broad set of evaluation workflows.