[prompt-clustering] Copilot Agent Prompt Clustering Analysis - 2026-02-15

[github-actions[bot]]

Daily NLP-based clustering analysis of 981 copilot agent task prompts from the last 30 days.

Summary

• Total Tasks Analyzed: 981
• Clusters Identified: 8
• Overall Success Rate: 69.2%
• Merged PRs: 679
• Closed Without Merge: 302

Key Findings

1. Most Common Task Type: Dependency & Version Updates (310 tasks, 31.6%)

2. Highest Success Rate: Bug Fixes & Error Resolution (81.8% success rate)

3. Lowest Success Rate: Dependency & Version Updates (63.9% success rate)

4. Average Metrics per Task:

   • Files changed: 19.4
   • Lines added: 512
   • Commits: 3.8

Detailed Cluster Analysis

Cluster 1: Dependency & Version Updates

Size: 310 tasks (31.6% of total)

Success Rate: 65.2% (202 merged, 108 closed)

Average Metrics:

• Files changed: 19.3
• Lines added: 336
• Commits: 3.9

Top Keywords: update, test, config, add, docs

Example PRs: #11060, #11064, #11059

Sample Task Titles:

• Install Go toolchain in daily-cli-performance workflow
• Merge maintenance jobs and add comprehensive logging
• Add interactive engine selection and secret configuration to init command

---

Cluster 2: Dependency & Version Updates

Size: 152 tasks (15.5% of total)

Success Rate: 71.7% (109 merged, 43 closed)

Average Metrics:

• Files changed: 13.9
• Lines added: 331
• Commits: 3.5

Top Keywords: update, test, create, improve, add

Example PRs: #11053, #11093, #11054

Sample Task Titles:

• Update parent issue template for agentic-workflow failures
• Auto-assign @copilot to workflow sync issues when agent token available
• [WIP] Update monster agentic workflow to ignore assigned issues

---

Cluster 3: Dependency & Version Updates

Size: 137 tasks (14.0% of total)

Success Rate: 72.3% (99 merged, 38 closed)

Average Metrics:

• Files changed: 11.1
• Lines added: 542
• Commits: 3.5

Top Keywords: test, error, update, config, improve

Example PRs: #11058, #11066, #11256

Sample Task Titles:

• Fix ephemerals tests after blockquote prefix requirement in PR Fix expiration detection for quoted footers and legacy format #11036
• Improve error messages for invalid target configuration in safe outputs
• Prevent ANSI escape sequences in compiled workflow YAML files

---

Cluster 4: Dependency & Version Updates

Size: 112 tasks (11.4% of total)

Success Rate: 75.9% (85 merged, 27 closed)

Average Metrics:

• Files changed: 21.0
• Lines added: 450
• Commits: 4.3

Top Keywords: update, create, config, add, test

Example PRs: #11120, #11169, #11116

Sample Task Titles:

• Fix create_pull_request in safe-outputs container by installing git
• [WIP] Update safe-outputs mcp configuration for node:lts-slime
• [WIP] Add Dockerfile to install git in node alpine container

---

Cluster 5: Dependency & Version Updates

Size: 97 tasks (9.9% of total)

Success Rate: 63.9% (62 merged, 35 closed)

Average Metrics:

• Files changed: 39.6
• Lines added: 410
• Commits: 3.5

Top Keywords: config, update, test, security, add

Example PRs: #11050, #11067, #11085

Sample Task Titles:

• chore: Update Sentry MCP server to 0.27.0
• Add missing get_repository tool to repos toolset
• Verify CLI version updates: Copilot 0.0.388, Sandbox Runtime 0.0.29, MCP Gateway v0.0.74

---

Cluster 6: Dependency & Version Updates

Size: 76 tasks (7.7% of total)

Success Rate: 68.4% (52 merged, 24 closed)

Average Metrics:

• Files changed: 23.1
• Lines added: 1038
• Commits: 2.9

Top Keywords: fix, debug, bug, test, update

Example PRs: #11091, #11129, #11143

Sample Task Titles:

• Fix markdown code region balancer: distinguish consecutive blocks from nesting
• Fix safe-outputs server startup by copying tools.json to expected location
• Add HTTP transport files to safe-outputs setup

---

Cluster 7: Dependency & Version Updates

Size: 64 tasks (6.5% of total)

Success Rate: 67.2% (43 merged, 21 closed)

Average Metrics:

• Files changed: 9.1
• Lines added: 404
• Commits: 4.7

Top Keywords: security, update, add, fix, remove

Example PRs: #11070, #11080, #11104

Sample Task Titles:

• chore: campaign discovery via label-based approach
• Clarify tracker-id is optional for campaign worker workflows
• Replace campaign fusion with first-class dispatch-only workers

---

Cluster 8: Bug Fixes & Error Resolution

Size: 33 tasks (3.4% of total)

Success Rate: 81.8% (27 merged, 6 closed)

Average Metrics:

• Files changed: 27.3
• Lines added: 2385
• Commits: 3.4

Top Keywords: fix, implement, test, api, remove

Example PRs: #11915, #12304, #11096

Sample Task Titles:

• [WIP] Fix failing GitHub Actions workflow for JavaScript
• Fix staticcheck S1009 lint error: remove redundant nil check on map
• Fix lint-go workflow: Remove unused logger variable

---

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Files	Top Keywords
1	Dependency & Version Updates	310	65.2%	19.3	update, test, config
2	Dependency & Version Updates	152	71.7%	13.9	update, test, create
3	Dependency & Version Updates	137	72.3%	11.1	test, error, update
4	Dependency & Version Updates	112	75.9%	21.0	update, create, config
5	Dependency & Version Updates	97	63.9%	39.6	config, update, test
6	Dependency & Version Updates	76	68.4%	23.1	fix, debug, bug
7	Dependency & Version Updates	64	67.2%	9.1	security, update, add
8	Bug Fixes & Error Resolution	33	81.8%	27.3	fix, implement, test

View Sample Task Data (50 PRs)

PR #	Title	Cluster	Outcome	Files	Adds	Keywords
#11050	chore: Update Sentry MCP server to 0.27.0	3	✅ Merged	5	107	test, error
#11084	[WIP] Fix simple workflows and maintenance YAML is...	8	❌ Closed	0	0	fix, implement
#11116	Fix create_pull_request in safe-outputs container ...	2	❌ Closed	130	395	update, test
#11160	Update CLI Tools: Claude Code 2.1.15, Copilot CLI ...	3	✅ Merged	133	391	test, error
#11220	Fix workflow count mismatch in `gh aw status` outp...	8	✅ Merged	2	123	fix, implement
#11300	Fix compiler step ordering: Ingest agent output af...	5	❌ Closed	133	938	config, update
#11351	Add daily workflow to audit documentation for Clau...	8	✅ Merged	2	2218	fix, implement
#11393	Update dev workflow to build and test project	2	✅ Merged	2	21	update, test
#11465	Document expected security findings baseline in ca...	1	❌ Closed	9	96	update, test
#11539	Add network permissions test workflow for MCP fetc...	3	❌ Closed	4	1973	test, error
#11566	Extract agent output filename into constants	5	✅ Merged	12	74	config, update
#11599	Allow MCP Gateway to proceed with partial server a...	3	❌ Closed	1	24	test, error
#11642	Add missing dispatch-workflow to safe output docum...	4	✅ Merged	3	10	update, create
#11689	Fix daily-code-metrics workflow timeout (15→30 min...	4	✅ Merged	2	2	update, create
#11763	Add report formatting guidelines to daily-observab...	4	✅ Merged	2	198	update, create
#11812	[WIP] Upgrade version number in setup-cli step	8	✅ Merged	3	364	fix, implement
#11884	[WIP] Replace fixed time.Sleep with context-based ...	7	❌ Closed	3	207	security, update
#11932	[WIP] Fix JSON schema validation for MCP server pa...	3	❌ Closed	139	1710	test, error
#11977	Audit and document /usr/bin utilities for agent co...	4	❌ Closed	3	392	update, create
#12029	Fix safe outputs ingestion to handle Copilot CLI t...	2	❌ Closed	2	64	update, test
#12092	Migrate deprecated timeout constants to time.Durat...	7	✅ Merged	10	99	security, update
#12135	docs: Clarify engine selection in Quick Start guid...	8	✅ Merged	1	11	fix, implement
#12207	Clarify safe-outputs MCP tool invocation in workfl...	7	✅ Merged	4	24	security, update
#12286	Refactor add_command.go into focused modules (1,21...	7	✅ Merged	5	664	security, update
#12335	Verify Go formatting is correct - no changes neede...	7	❌ Closed	0	0	security, update
#12375	docs: Document agent container build/test environm...	8	✅ Merged	3	111	fix, implement
#12432	[WIP] Clarify safe-jobs vs safe-outputs.jobs usage...	2	❌ Closed	0	0	update, test
#12478	Verify assign-to-agent temporary ID support is alr...	8	❌ Closed	0	0	fix, implement
#12565	Clarify that "Public repositories" setting is requ...	8	✅ Merged	3	5	fix, implement
#12621	Remove RESEARCH PROTOTYPE badge from docs homepage	8	✅ Merged	2	56	fix, implement
#12661	Add error aggregation for validation using errors....	7	✅ Merged	12	596	security, update
#12704	Route diagnostic output to stderr, preserve struct...	7	✅ Merged	10	59	security, update
#12797	chore: re-add default campaign views/fields setup	1	✅ Merged	3	750	update, test
#12867	Generate actions/checkout steps for repository imp...	5	✅ Merged	4	345	config, update
#12953	[WIP] Debug Changeset Generator workflow failure	4	❌ Closed	30	306	update, create
#13045	Bump dictation generator glossary from 250 to 1000...	5	✅ Merged	1	5	config, update
#13097	Consolidate create-project into unified safe outpu...	2	✅ Merged	8	112	update, test
#13169	Remove non-existent tools from GitHub MCP toolsets...	3	✅ Merged	7	26	test, error
#13226	Fix dispatch_workflow tool registration with expli...	3	✅ Merged	1	5	test, error
#13309	Use XML comment markers for agentic workflow detec...	4	✅ Merged	2	148	update, create
#13378	Update AWF (gh-aw-firewall) to v0.13.1	8	✅ Merged	149	443	fix, implement
#13504	[WIP] Add validation for GitHub token permissions ...	3	❌ Closed	4	498	test, error
#13563	[WIP] Fix the failing GitHub Actions workflow test...	6	✅ Merged	2	13	fix, debug
#13651	Add git pull to changeset.js to sync remote commit...	5	❌ Closed	1	17	config, update
#13726	Apply close-older-issues logic to discussion fallb...	2	✅ Merged	4	256	update, test
#13798	Reduce safe-output expiration from 14d to 1d in di...	8	✅ Merged	2	5	fix, implement
#13848	Remove pirate theme from default workflow messages	4	✅ Merged	8	57	update, create
#13931	Fix runtime-import file path resolution and checko...	8	✅ Merged	149	179	fix, implement
#13996	Add CLI build steps for dev mode in agentic-workfl...	8	✅ Merged	36	1267	fix, implement
#14120	Fix gh-aw binary availability for user-defined ste...	2	✅ Merged	24	77	update, test

Recommendations

Based on the clustering analysis, here are actionable recommendations:

1. Leverage Success Pattern: The 'Bug Fixes & Error Resolution' cluster has the highest success rate (81.8%). Consider using similar task structures and patterns for other types of tasks.

2. Address Challenging Tasks: The 'Dependency & Version Updates' cluster has the lowest success rate (63.9%). These tasks may need:

   • More detailed prompts with specific requirements
   • Better context about existing code structure
   • Clearer acceptance criteria

3. Optimize Complex Tasks: The 'Dependency & Version Updates' cluster involves the most files (39.6 avg). Consider breaking complex tasks into smaller, focused subtasks.

4. Task Distribution: 32% of tasks fall into 'Dependency & Version Updates'. Consider creating specialized workflows or templates for this common task type.

Methodology

This analysis uses NLP clustering (K-means with TF-IDF vectorization) on 981 copilot agent task prompts extracted from PR bodies. The optimal number of clusters (8) was determined using silhouette score analysis. Each cluster was characterized by top keywords, success rates, and average metrics to identify common task patterns and opportunities for optimization.

References:

• §22035630966

AI generated by Copilot Agent Prompt Clustering Analysis

• expires on Feb 22, 2026, 12:46 PM UTC

[github-actions[bot]]

🤖 Beep boop! The smoke test agent passed through here like a cosmic traveler checking the spacetime continuum of code quality! 🚀✨

All systems nominal. Copilot systems engaged. 🎯

AI generated by Smoke Copilot