feat: Added comprehensive unit testing and github action to run tests on new pull requests

[dklawren]

No description provided.

[Copilot]

Pull request overview

Adds a comprehensive pytest-based test suite and GitHub Actions workflow to automatically run linting, unit tests, and integration tests for the GitHub ETL pipeline.

Changes:

• Added a large pytest test suite covering main.py functions (extraction, transformation, loading, orchestration).
• Added pytest/coverage configuration and a testing guide documenting local + CI workflows.
• Added a GitHub Actions workflow to run linting/tests on pull requests; expanded dependencies to include lint/format tools.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
test_main.py	New comprehensive unit/integration test suite for main.py.
requirements.txt	Adds dev tooling dependencies (black/flake8/mypy/isort) alongside existing test deps.
pytest.ini	Configures pytest discovery, verbosity, and coverage reporting.
TESTING.md	Documents how to run tests/linting locally and in CI, plus docker-based integration testing.
.github/workflows/tests.yml	Adds CI workflow for linting, pytest runs (unit + all), coverage artifacts, and docker-compose integration job.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

main.py:536

• GITHUB_REPOS is split on commas but entries aren’t stripped/validated. Values like "owner/repo, owner/repo" (note the space) or a trailing comma will produce repo strings with leading whitespace or empty entries, which will break API URLs. Consider stripping whitespace and filtering out empty repo names before iterating.

    github_repos = []
    github_repos_str = os.getenv("GITHUB_REPOS")
    if github_repos_str:
        github_repos = github_repos_str.split(",")
    else:
        raise SystemExit(
            "Environment variable GITHUB_REPOS is required (format: 'owner/repo,owner/repo')"
        )

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cgsheeh]

I haven't looked over the tests in full yet, but wanted to leave comments about the higher-level items I noticed off the bat.

[cgsheeh]

There is a lot of content in this PR, it might take a few iterations to review it fully.

We could land some of this code faster if we split it into smaller chunks. For example, we could add linting support and tests in one PR, and deal with the tests themselves in another.

[dklawren]

There is a lot of content in this PR, it might take a few iterations to review it fully.

We could land some of this code faster if we split it into smaller chunks. For example, we could add linting support and tests in one PR, and deal with the tests themselves in another.

Good idea and sorry for so much in one PR. I create #9 just now which only includes the github action and formatting parts this PR and we can land it first and then add the unit tests in a separate PR.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

integration-test job YAML is invalid: the list items under steps: are not indented, so GitHub Actions will treat them as siblings of steps (workflow parse error). Indent the - uses: / - name: entries under steps: by two spaces.

Suggested change

	- uses: actions/checkout@v4
	- name: Run integration test with docker compose
	run: |
	docker compose up --build --abort-on-container-exit --exit-code-from github-etl
	- name: Cleanup
	if: always()
	run: docker compose down -v
	- uses: actions/checkout@v4
	- name: Run integration test with docker compose
	run: |
	docker compose up --build --abort-on-container-exit --exit-code-from github-etl
	- name: Cleanup
	if: always()
	run: docker compose down -v

[Copilot]

This section refers to a test/ directory and paths like pytest test/test_extract_pull_requests.py, but the repository’s tests live under tests/ (and pytest is configured with testpaths = ["tests"]). Update the docs and example commands/paths to tests/... so they work as written.

[Copilot]

This file won’t be collected by pytest because pyproject.toml sets testpaths = ["tests"], so top-level test_formatting.py is ignored. Either move/delete it, or adjust pytest discovery; also consider de-duplicating with tests/test_formatting.py to avoid maintaining two different formatting-test implementations.

Suggested change

	def test_black():
	cmd = ("black", "--diff", "main.py")
	output = subprocess.check_output(cmd)
	assert not output, "The python code does not adhere to the project style."
	def test_ruff():
	def check_black():
	cmd = ("black", "--diff", "main.py")
	output = subprocess.check_output(cmd)
	assert not output, "The python code does not adhere to the project style."
	def check_ruff():