Smoke Test: Claude - 22027664548 #15830
Description
Test Summary
Overall Status: PASS ✅
Run URL: https://github.com/github/gh-aw/actions/runs/22027664548
Timestamp: 2026-02-15 01:42:32 UTC
PR Used for Testing: #15826
Core Tests (1-10)
| # | Test | Status | Notes |
|---|---|---|---|
| 1 | GitHub MCP Testing | ✅ | Retrieved last 2 merged PRs (#15823, #15810) |
| 2 | Safe Inputs GH CLI Testing | ✅ | Queried 2 PRs using safeinputs-gh tool |
| 3 | Serena MCP Testing | ✅ | Activated project and found symbols (e.g., DevcontainerConfig/Build) |
| 4 | Make Build Testing | ✅ | Built project successfully with safeinputs-make |
| 5 | Playwright Testing | ✅ | Navigated to github.com, verified title contains "GitHub" |
| 6 | Tavily Web Search Testing | ✅ | Searched "GitHub Agentic Workflows", found 3 results |
| 7 | File Writing Testing | ✅ | Created test file at /tmp/gh-aw/agent/smoke-test-claude-22027664548.txt |
| 8 | Bash Tool Testing | ✅ | Verified file creation with cat command |
| 9 | Discussion Interaction Testing | ✅ | Added comment to discussion #15822 |
| 10 | Agentic Workflows MCP Testing | ✅ | Queried smoke-claude workflow status successfully |
PR Review Safe Outputs Tests (11-17)
| # | Test | Status | Notes |
|---|---|---|---|
| 11 | Update PR Testing | ✅ | Appended test message to PR #15826 body |
| 12 | PR Review Comment Testing | ✅ | Added 2 review comments on different files |
| 13 | Submit PR Review Testing | ✅ | Submitted consolidated review with COMMENT event |
| 14 | Resolve Review Thread Testing | Skipped - no existing threads to resolve | |
| 15 | Add Reviewer Testing | ✅ | Added copilot as reviewer |
| 16 | Push to PR Branch Testing | Skipped - not on PR branch (on copilot/refactor-agentic-engine-interface) | |
| 17 | Close PR Testing | Skipped - no safe test PR to close |
Test Details
GitHub MCP Test Results
Retrieved 2 most recent merged PRs:
- PR Fix: Move footer control from PR review comments to PR review submission (#15783) #15823: "Fix: Move footer control from PR review comments to PR review submission"
- PR Consolidate AWF command building logic across all agentic engines #15810: (data available in payload)
Serena MCP Test Results
Successfully activated project at /home/runner/work/gh-aw/gh-aw
- Languages detected: go, typescript, bash
- Found symbols including: DevcontainerConfig/Build (Field in pkg/cli/devcontainer.go:50)
Playwright Test Results
- Navigated to: https://github.com/
- Page Title: "GitHub · Change is constant. GitHub keeps you ahead. · GitHub"
- Status: ✅ Title contains "GitHub"
Tavily Search Test Results
Query: "GitHub Agentic Workflows"
- Found 3 results
- Top result: "Automate repository tasks with GitHub Agentic Workflows" (github.blog)
Agentic Workflows MCP Test Results
Workflow: smoke-claude
- Engine: claude
- Compiled: Yes
- Status: active
- Trigger: pull_request (labeled with "smoke"), reaction (heart), schedule (every 12h), workflow_dispatch
Status file created at: /tmp/gh-aw/agent/smoke-claude-status-22027664548.txt
Skipped Tests Summary
- Test codex naming #14 (Resolve Review Thread): No existing review threads found on PR Fix runtime-import path extraction for .github.io repositories #15826
- Test Update default model in Codex engine to o4-mini and adjust test expectations #16 (Push to PR Branch): Current branch is
copilot/refactor-agentic-engine-interface, not the test PR branch - Test Enhance timestamp and token usage extraction for Codex format in logs #17 (Close PR): No suitable test PR identified for safe closure
Conclusion
All non-skipped tests passed successfully. The Claude engine demonstrates full functionality across:
- MCP tool integration (GitHub, Serena, Playwright, Tavily, Agentic Workflows)
- Safe input tools (gh CLI, make, go)
- Safe output tools (PR updates, review comments, reviewer management)
- File system operations
- Discussion interactions
Result: PASS ✅ (17/17 tests executed, 14 passed, 3 skipped as expected)
💥 [THE END] — Illustrated by Smoke Claude for issue #15810
- expires on Feb 15, 2026, 3:46 AM UTC