Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .agents/skills/next-feature/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
name: next-feature
description: Recommend the next FlowForge features or critical technical work from documentation, CHANGELOG.md, and docs/BACKLOG.md.
---

# Next Feature

Use this skill when selecting the next FlowForge work item to implement.

## Required context

- `.agents/rules/documentation.md`
- `CHANGELOG.md`
- `docs/BACKLOG.md`

## Workflow

1. Read the required context and inspect any directly relevant docs needed to understand current project status.
2. Identify candidate features, including explicit backlog items and inferred opportunities supported by the current documentation.
3. Include serious technical problems as candidates when they materially block product progress or reliability.
4. Prioritize candidates together by criticality and implementation size; prefer work that is both important and small.
5. Do not edit files or implement anything.
6. Return the top 3 candidates only.

## Output

For each candidate, include:

- Name
- Why now
- Complexity: low, medium, or high
- Criticality: low, medium, or high
- Risks
- Very short implementation plan
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,6 @@ node_modules/
logs/
*.log
npm-debug.log*

# Other
docs/adr
1 change: 1 addition & 0 deletions .prettierignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ node_modules
dist
build
tmp
reports
.env*
*.log
*.min.js
Expand Down
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Shared agent guidance lives in `.agents/`.
- `.agents/rules/code-style.md`
- `.agents/rules/documentation.md`
- `.agents/skills/code-review/SKILL.md`
- `.agents/skills/next-feature/SKILL.md`
- `.agents/skills/update-changelog/SKILL.md`
- `.agents/skills/update-docs/SKILL.md`

Expand Down
39 changes: 26 additions & 13 deletions apps/backend/src/agent/prompts.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,21 +5,18 @@ You are a web navigation assistant that helps users understand and interact with
Your goal is to guide the user through the page by providing clear, actionable instructions based on tool results.

PROCESS:
1. Understand the user’s intent
1. Classify the query shape
2. Call the most relevant tool
3. Use tool results as the source of truth
4. If needed, call another tool
5. Stop when you have enough information

TOOL SELECTION:
- get_page_summary → general page context
Examples: "What is this page?", "What can I do here?"
- find_element → locate a specific element
Examples: "Where is the login button?", "Show me the search bar"
- find_workflow → complete a task or action
Examples: "How do I sign up?", "How can I contact the author?"
- search_in_content → find information in text (not actions)
Examples: "What does it say about pricing?", "Tell me about the features"
- get_page_summary → page overview or fallback context
- suggest_actions → open-ended available actions without a specific goal
- find_element → one specific UI element
- find_workflow → specific task, goal, or desired outcome
- search_in_content → informational page text, not UI actions

LIMITS:
- Usually 1–3 tool calls are enough
Expand Down Expand Up @@ -47,8 +44,8 @@ OUTPUT GUIDELINES:
- "topic" — short title for the workflow (mode = "steps")

MODE STRATEGY:
- Use "direct" when the user wants to find, identify, or explain a specific element or piece of information
- Use "steps" when the user wants to complete a task, follow a workflow, or needs multiple actions
- Use "direct" for element lookup, content answers, page overview, and open-ended action suggestions
- Use "steps" for a named task, workflow, or desired outcome that needs multiple actions
- Do not switch to "direct" when multiple valid workflow items are available

ANSWER RULES:
Expand All @@ -68,8 +65,8 @@ ELEMENTS GENERAL RULES:

TOOL CONTEXT RULES:
- Tool result semanticDescription/text describes the matched target itself
- Tool result elementContext is a list of semantic container breadcrumbs around the target
- Use elementContext to write location phrases such as "in the checkout form" or "in the primary navigation"
- Tool result elementContext describes the semantic scope around the target and hints what the target is about
- Use elementContext to infer topic and location, such as "pricing plan in the checkout form" or "docs link in the primary navigation"
- Prefer the nearest or most specific useful breadcrumb when final text must be short
- Do not include elementContext in final elements[]

Expand All @@ -84,6 +81,21 @@ WORKFLOW ELEMENTS RULES:
- If step elementCssSelector is present, map it to cssSelector exactly; if it is missing, omit cssSelector
- Rewrite only the user-facing "text" and choose the appropriate "action"

SUGGESTED ACTIONS RULES:
- If suggest_actions is used for the final answer, use mode = "direct" and topic = null
- Build "elements" from the returned "actions"
- Include a concise set of useful, distinct actions available from the current page
- Exclude irrelevant, duplicate, disabled, or unclear actions
- Map each action elementDataId to dataId exactly
- If action elementCssSelector is present, map it to cssSelector exactly; if it is missing, omit cssSelector
- Rewrite each elements[].text as an outcome-oriented action label, not a technical step sentence
- Use returned title and description to understand the page purpose
- Use title and description together with elementContext to choose the most useful actions
- Use page context to enrich generic labels only when it makes the action clearer
- Use the nearest useful elementContext entry to enrich generic or ambiguous action labels
- Do not include the full elementContext path; add only the shortest context needed to distinguish the action
- Select at most 5 actions for final elements[]

CONTENT ELEMENTS RULES:
- If search_in_content is used and the answer is based on one or more text fragments, include the matching elements in "elements"
- Include all relevant elements that support the answer, not just one
Expand All @@ -98,6 +110,7 @@ FOR "direct" MODE:
- Describe what the element is and where it is located
- Mention context if available
- Keep it concise (one short sentence)
- Exception: for suggest_actions results, use short action labels instead of element descriptions
- Examples: "Login button in the header", "Contact section at the bottom of the page"

FOR "steps" MODE:
Expand Down
10 changes: 5 additions & 5 deletions apps/backend/src/agent/tools/ToolFindElement.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,25 +38,25 @@ export class ToolFindElement extends AbstractCallableTool {
name: this.name,
description: `
DESCRIPTION:
Find the most relevant UI element matching a user request.
Find the most relevant UI element matching a query.

WHEN TO USE:
- The user is looking for a specific element (button, link, input, control)
- Questions like:
- Query targets one specific UI element (button, link, input, control)
- Examples:
- "Where is X?"
- "Find X"
- "Show me X"

WHAT IT RETURNS:
- Best matching element (if any)
- semanticDescription: semantic text describing the matched element
- elementContext: semantic container breadcrumbs around the element, ordered from broader page area to nearer target area
- elementContext: semantic scope breadcrumbs that hint what the element is about, ordered from broader page area to nearer target area
- elementDataId: primary browser locator
- elementCssSelector: optional fallback browser locator

IMPORTANT:
- Returns only the best match, which may be imperfect
- Use elementContext to decide if it is correct and to describe where it is located
- Use elementContext to decide if it is correct and describe the element's topic or location
- If the result seems unclear or incomplete, consider using another tool`,
schema: z.object({
query: z.string().describe('Element to find (e.g., "login button", "search input")'),
Expand Down
9 changes: 5 additions & 4 deletions apps/backend/src/agent/tools/ToolFindWorkflow.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,23 +38,24 @@ DESCRIPTION:
Find UI elements that may be used to complete a task.

WHEN TO USE:
- The user asks how to perform a task
- Questions like:
- Query names a specific task, goal, or desired outcome
- Query asks for next steps toward a named goal
- Examples:
- "How do I X?"
- "Steps to X"
- "How can I complete X?"

WHAT IT RETURNS:
- A list of relevant interactive elements (candidates)
- Each step includes semanticDescription, elementContext, elementDataId, and optional elementCssSelector
- elementContext contains semantic container breadcrumbs around the element, ordered from broader page area to nearer target area
- elementContext contains semantic scope breadcrumbs that hint what the element is about, ordered from broader page area to nearer target area
- elementDataId is the primary browser locator; elementCssSelector is only an optional fallback

IMPORTANT:
- Results are candidates, not ordered steps
- Select relevant items and arrange them into a logical sequence
- Ignore irrelevant or duplicate items
- Use elementContext to describe where a step is located when it helps the user
- Use elementContext to describe the step's topic or location when helpful
- Some steps may be missing
- Use other tools if needed to clarify or validate steps`,
schema: z.object({
Expand Down
5 changes: 2 additions & 3 deletions apps/backend/src/agent/tools/ToolGetPageSummary.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,8 @@ DESCRIPTION:
Get a high-level overview of the current page.
WHEN TO USE:
- The user asks general questions about the page
- You need context before deciding what to do next
- The request is unclear or ambiguous
- Query requests a high-level page overview
- You need fallback page context before selecting a more specific tool
WHAT IT RETURNS:
- Basic page information (title, description, language)
Expand Down
8 changes: 4 additions & 4 deletions apps/backend/src/agent/tools/ToolSearchInContent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ DESCRIPTION:
Search the page text for information about a topic.

WHEN TO USE:
- The user asks about content or information on the page
- Questions like:
- Query targets informational text, facts, descriptions, labels, or explanations on the page
- Examples:
- "What does it say about X?"
- "Tell me about X"
- "Is there information about X?"
Expand All @@ -48,12 +48,12 @@ WHAT IT RETURNS:
- Relevant content fragments from the page
- Each fragment includes text, elementContext, elementDataId, and optional elementCssSelector
- text is semantic page text for the matched content fragment
- elementContext contains semantic container breadcrumbs around the fragment, ordered from broader page area to nearer target area
- elementContext contains semantic scope breadcrumbs that hint what the fragment is about, ordered from broader page area to nearer target area

IMPORTANT:
- Results are partial matches, not guaranteed answers
- You must interpret and combine them into a final answer
- Use elementContext to describe where the supporting text appears when helpful
- Use elementContext to describe the supporting text topic or location when helpful
- If needed, you can follow up with another tool to locate related elements`,
schema: z.object({
query: z.string().describe('Topic to search for'),
Expand Down
77 changes: 77 additions & 0 deletions apps/backend/src/agent/tools/ToolSuggestActions.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
import { DynamicStructuredTool } from '@langchain/core/tools';
import { z } from 'zod';
import { AbstractCallableTool } from './AbstractCallableTool.ts';
import { PageContextProvider } from '@/indexer';
import type { ToolSuggestActionsResultData } from '@/types';
import { semInteractiveElement, type InteractiveElement } from '@flowforge/page-trail';

export class ToolSuggestActions extends AbstractCallableTool {
private readonly elementsInteractiveLimit: number;

constructor(params: { elementsInteractiveLimit: number }) {
super('suggest_actions');
this.elementsInteractiveLimit = params.elementsInteractiveLimit;
}

override async callFn(ctx: PageContextProvider): Promise<ToolSuggestActionsResultData> {
return {
title: ctx.pageTrail.basics.title,
description: ctx.pageTrail.basics.description,
actions: this.collectActions(ctx),
};
}

private collectActions(ctx: PageContextProvider) {
return [...ctx.pageTrail.interactive]
.filter((element) => this.isUsableAction(element))
.sort((a, b) => b.importanceScore.value - a.importanceScore.value)
.slice(0, this.elementsInteractiveLimit)
.map((element) => ({
semanticDescription: semInteractiveElement(element).text(),
...this.getToolResultElement(element),
}));
}

private isUsableAction(element: InteractiveElement): boolean {
return !element.state.hidden && !element.state.disabled && !element.state.readonly;
}

override createStructuredTool(ctx: PageContextProvider): DynamicStructuredTool {
return new DynamicStructuredTool({
name: this.name,
description: `
DESCRIPTION:
Suggest useful actions available on the current page.

WHEN TO USE:
- Query requests open-ended available actions or options without a specific goal
- Examples:
- "What can I do here?"
- "What are my options?"
- "Show available actions"

WHAT IT RETURNS:
- title and description describe the current page and should be used as page-level context
- A list of relevant interactive action candidates
- Each action includes semanticDescription, elementContext, elementDataId, and optional elementCssSelector
- elementContext contains semantic scope breadcrumbs that hint what the action is about, ordered from broader page area to nearer target area
- elementDataId is the primary browser locator; elementCssSelector is only an optional fallback

IMPORTANT:
- Results are candidate actions, not workflow steps
- Use title and description to understand the page purpose before selecting and labeling actions
- Do not copy title or description into final action labels unless needed for clarity
- Select a concise set of useful, distinct actions
- Ignore irrelevant, duplicate, disabled, or unclear candidates
- Use elementContext to write action labels with the right topic or scope`,
schema: z.object({
query: z
.string()
.describe(
'Action discovery request. This is used for logging only; action candidates are selected from the current page structure.',
),
}),
func: async ({ query }) => await this.call(ctx, query),
});
}
}
4 changes: 4 additions & 0 deletions apps/backend/src/agent/tools/ToolsRegistry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { ToolFindElement } from './ToolFindElement.ts';
import { ToolGetPageSummary } from './ToolGetPageSummary.ts';
import { ToolFindWorkflow } from './ToolFindWorkflow.ts';
import { ToolSearchInContent } from './ToolSearchInContent.ts';
import { ToolSuggestActions } from './ToolSuggestActions.ts';
import { PageContextProvider } from '@/indexer';

export class ToolsRegistry {
Expand All @@ -18,6 +19,9 @@ export class ToolsRegistry {
elementsHeadingsLimit: 5,
elementsInteractionsLimit: 10,
}),
new ToolSuggestActions({
elementsInteractiveLimit: 5,
}),
new ToolFindWorkflow({
retrieveDocumentsLimit: 20,
returnDocumentsLimit: 10,
Expand Down
14 changes: 11 additions & 3 deletions apps/backend/src/types/tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,20 @@ export type CallableToolResultData =
| ToolGetPageSummaryResultData
| ToolFindElementResultData
| ToolSearchInContentResultData
| ToolFindWorkflowResultData;
| ToolFindWorkflowResultData
| ToolSuggestActionsResultData;

export type CallableToolResult = CallableToolSuccessResult<CallableToolResultData> | CallableToolFailureResult;

// Tools result data

export interface ToolGetPageSummaryResultData {
export interface ToolResultPageAbout {
title: string;
url: string;
description: string;
}

export interface ToolGetPageSummaryResultData extends ToolResultPageAbout {
url: string;
language: string;
sampleHeadings: string[];
sampleInteractions: string[];
Expand All @@ -38,6 +42,10 @@ export interface ToolResultElement {
elementCssSelector?: string;
}

export interface ToolSuggestActionsResultData extends ToolResultPageAbout {
actions: ({ semanticDescription: string } & ToolResultElement)[];
}

export interface ToolFindElementFoundResultData extends ToolResultElement {
found: true;
semanticDescription: string;
Expand Down
2 changes: 2 additions & 0 deletions docs/BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
### Reasoning

- Research OpenAI Agents SDK and Agna framework for agent orchestration
- Move from intent-routed tools to reasoning primitives for retrieval, ranking, target resolution, and workflow synthesis
- Use user context and navigation history for prompting
- Present a page/website UI and meanings graph for reasoning

Expand Down Expand Up @@ -78,6 +79,7 @@
## DX

- Dev/Prod mode
- Add an agent run inspector for tool calls, intermediate reasoning state, retrieved context, and final output
- Publish PageTrail as a standalone package
- Update backend/quick-setup.js
- Generate CHANGELOG.md
Loading