Skip to content

Add CI smoke/integration tests and fake provider for testability #170

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
michaelerobertsjr wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Add CI smoke/integration tests and fake provider for testability #170

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d6fbe00
Add CI smoke/integration tests and fake provider for testability
Mar 8, 2026
0d6e456
Harden fake provider usage and complete team flows on dead branches
Mar 8, 2026
e723838
Update .github/workflows/ci.yml to add timeout
michaelerobertsjr Mar 8, 2026
33ce4e5
Retry processor startup on port conflicts in test helpers
Mar 8, 2026
f23868c
Harden CI permissions and verify tinyclaw install resolution
Mar 8, 2026
a4c19e6
Restructure CI smoke flow and harden queue/SSE handling
Mar 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
name: CI

on:
pull_request:
push:
branches: [main]

jobs:
integration:
runs-on: ubuntu-latest
timeout-minutes: 15

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'

- name: Install dependencies
run: PUPPETEER_SKIP_DOWNLOAD=true npm ci --include=dev

- name: Run integration tests
run: npm run test:integration

smoke:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, macos-latest]

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'

- name: Install dependencies
run: PUPPETEER_SKIP_DOWNLOAD=true npm ci --include=dev

- name: Run smoke tests
run: npm run test:smoke
81 changes: 69 additions & 12 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,12 @@ on:
workflow_dispatch: # Allow manual trigger

permissions:
contents: write
contents: read

jobs:
build-and-release:
permissions:
contents: read
runs-on: ubuntu-latest

steps:
Expand All @@ -23,7 +25,7 @@ jobs:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '18'
node-version: '20'
cache: 'npm'

- name: Verify tag matches package version
Expand Down Expand Up @@ -63,6 +65,7 @@ jobs:
run: npm prune --omit=dev

- name: Create bundle tarball
id: bundle
run: |
VERSION="${{ steps.get_version.outputs.version }}"
BUNDLE_NAME="tinyclaw-bundle.tar.gz"
Expand All @@ -80,6 +83,7 @@ jobs:
cp -r lib/ "$BUNDLE_DIR/"
cp -r docs/ "$BUNDLE_DIR/"
cp -r .agents/ "$BUNDLE_DIR/"
cp -r tests/ "$BUNDLE_DIR/"
Copy link
Copy Markdown

@greptile-apps greptile-apps bot Mar 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test files included in the production release tarball

tests/ is being copied into the bundle, so every end-user who downloads the official GitHub release tarball also receives the full test harness (integration helpers, fake providers, SSE fixtures, etc.).

More critically, by the time this bundle step runs, npm prune --omit=dev has already stripped tsx from node_modules/. Any user who extracts the release tarball and tries to run npm run test:smoke or npm run test:integration directly will get an error because tsx is absent from the production node_modules.

The cleanest fix is to keep the release tarball test-free and instead upload a dedicated test-only artifact for the verify-smoke job:

# In build-and-release — create a separate test artifact, don't include tests/ in the main bundle
- name: Upload test sources artifact
  uses: actions/upload-artifact@v4
  with:
    name: tinyclaw-tests
    path: |
      tests/
      package.json
      package-lock.json
      tsconfig.json

# Remove cp -r tests/ from the bundle tarball step

Then in verify-smoke, download both tinyclaw-bundle and tinyclaw-tests, overlay the tests directory on the extracted bundle, run npm ci --include=dev, and execute the smoke tests. This keeps the release artifact clean while the CI pipeline still has everything it needs.


cp tinyclaw.sh "$BUNDLE_DIR/"

Expand Down Expand Up @@ -114,10 +118,72 @@ jobs:
echo "bundle_name=$BUNDLE_NAME" >> $GITHUB_OUTPUT
echo "bundle_size=$BUNDLE_SIZE" >> $GITHUB_OUTPUT

- name: Upload bundle artifact
uses: actions/upload-artifact@v4
with:
name: tinyclaw-bundle
path: tinyclaw-bundle.tar.gz
retention-days: 7

verify-smoke:
needs: build-and-release
permissions:
contents: read
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, macos-latest]

steps:
- name: Download bundle artifact
uses: actions/download-artifact@v4
with:
name: tinyclaw-bundle

- name: Extract bundle
shell: bash
run: |
EXTRACT_DIR="$(mktemp -d)"
tar -xzf tinyclaw-bundle.tar.gz -C "$EXTRACT_DIR"
echo "BUNDLE_DIR=$EXTRACT_DIR/tinyclaw" >> "$GITHUB_ENV"

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
Comment on lines +150 to +153
Copy link
Copy Markdown

@greptile-apps greptile-apps bot Mar 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing cache: 'npm' in verify-smoke Node.js setup

Every other job in both workflows (build-and-release, integration, smoke) passes cache: 'npm' to actions/setup-node. Without it, verify-smoke downloads and installs the full npm dependency tree from the network on every run, adding unnecessary latency to every release verification.

Suggested change
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'


- name: Install dependencies
working-directory: ${{ env.BUNDLE_DIR }}
run: |
PUPPETEER_SKIP_DOWNLOAD=true npm ci --include=dev

- name: Run smoke tests
working-directory: ${{ env.BUNDLE_DIR }}
run: npm run test:smoke
Comment on lines +128 to +162
Copy link
Copy Markdown

@greptile-apps greptile-apps bot Mar 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

verify-smoke job missing timeout-minutes

The verify-smoke job spawns real queue processors, runs HTTP polling loops, and waits on SSE streams — the same class of operations that prompted the timeout-minutes: 15 that was added to the integration job in ci.yml. Without an explicit cap, a hung smoke test on Ubuntu or macOS will hold the runner for the full GitHub Actions default of 6 hours before failing, blocking any tag release.

Suggested change
verify-smoke:
needs: build-and-release
permissions:
contents: read
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, macos-latest]
steps:
- name: Download bundle artifact
uses: actions/download-artifact@v4
with:
name: tinyclaw-bundle
- name: Extract bundle
shell: bash
run: |
EXTRACT_DIR="$(mktemp -d)"
tar -xzf tinyclaw-bundle.tar.gz -C "$EXTRACT_DIR"
echo "BUNDLE_DIR=$EXTRACT_DIR/tinyclaw" >> "$GITHUB_ENV"
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
working-directory: ${{ env.BUNDLE_DIR }}
run: |
PUPPETEER_SKIP_DOWNLOAD=true npm ci --include=dev
- name: Run smoke tests
working-directory: ${{ env.BUNDLE_DIR }}
run: npm run test:smoke
verify-smoke:
needs: build-and-release
timeout-minutes: 20
permissions:
contents: read
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, macos-latest]


publish-release:
if: github.ref_type == 'tag'
needs: verify-smoke
permissions:
contents: write
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0
fetch-tags: true

- name: Download bundle artifact
uses: actions/download-artifact@v4
with:
name: tinyclaw-bundle

- name: Extract release notes from tag
id: release_notes
run: |
VERSION="${{ steps.get_version.outputs.version }}"
VERSION="${GITHUB_REF#refs/tags/}"
# Use for-each-ref to reliably get the annotated tag message (not the commit message)
TAG_MESSAGE=$(git for-each-ref --format='%(contents)' "refs/tags/$VERSION")
if [ -n "$TAG_MESSAGE" ]; then
Expand All @@ -132,7 +198,6 @@ jobs:

- name: Create GitHub Release
uses: softprops/action-gh-release@v1
if: github.ref_type == 'tag'
with:
body_path: release_notes.md
files: |
Expand All @@ -141,11 +206,3 @@ jobs:
prerelease: false
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Upload artifact (for non-tag builds)
if: github.ref_type != 'tag'
uses: actions/upload-artifact@v4
with:
name: tinyclaw-bundle
path: tinyclaw-bundle.tar.gz
retention-days: 7
Loading