Skip to content

As a developer, I want deterministic dynamic checks in test-setup.sh so that CI can verify knowledge search without LLM dependencies #252

@kiyotis

Description

@kiyotis

Situation

After setup, users need to verify that nabledge knowledge search works by executing actual /n6 commands (for CC and GHC). Currently, developers must manually test each version × tool combination.

We've confirmed both CC and GHC CLI can execute /n6 commands:

  • CC: claude -p '/n6 "question"' --model haiku
  • GHC: copilot -p "$PROMPT" --model claude-haiku-4.5 --allow-tool Bash --autopilot

Pain

  • Developers must manually verify setup works for each version × tool combination (12 total)
  • No automated feedback if setup is incomplete or knowledge files are missing
  • Users cannot easily verify installation without trial-and-error

Benefit

  • Users can run test-setup.sh and automatically verify /n6 works for all versions
  • Early detection of setup issues before users encounter them in real workflows
  • Clear feedback: [OK] if /n6 returns expected content, [FAIL] if not

Success Criteria

  • test-setup.sh includes /n6 execution tests for all 12 combinations (v6/v5/v1.4/v1.3/v1.2/all × cc/ghc)
  • Each test executes /n6 with a sample question via CC CLI or GHC CLI
  • Test validates that /n6 returns a response (not error/timeout)
  • Tests run with user's existing ANTHROPIC_API_KEY / COPILOT_GITHUB_TOKEN (if available)
  • Results clearly show [OK] or [FAIL] for each combination
  • Users can run tests after setup to verify their installation works

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions