Enhance Page-Trail and Inspector

.agents/rules/documentation.md

@@ -91,14 +91,15 @@ Target length: 80–100 lines.
 
 ### `packages/page-trail/README.md`
 
-Target length: 60–80 lines.
+Target length: 80–100 lines.
 
 - Overview
-- Structure
+- Format (`PageTrail` object shape only)
+- Structure elements
 - Content elements
 - Interactive elements
 - Context
-- Importance
+- Scoring (`meaningScore`, `relevanceScore`, `contextScore`, `importanceScore`)
 - Format
 - Usage
 

CHANGELOG.md

@@ -2,6 +2,23 @@
 
 All notable changes to the project will be documented in this file.
 
+## [Unreleased]
+
+### Added
+
+- PageTrail `container` elements and `structure` tree for representing page context and layout.
+- Detailed `context.path` data for content and interactive elements, including container relevance and breadcrumb context.
+- Dev mode for Page Inspector with enriched PageTrail records and metadata diagnostics.
+- Tooltips across popup and Inspector controls, replacing native `title` hints.
+
+### Changed
+
+- Reworked PageTrail scoring and semantic formatting around meaning, context relevance, and target importance.
+- Updated backend tools to use detailed PageTrail `context.path` when describing matched elements and content.
+- Expanded PageTrail metadata timings with per-stage collection durations.
+- Improved Markdown output for the Inspector semantic view.
+- Limited extracted label and text values to keep PageTrail records concise.
+
 ## [0.1.5] - 2026-08-16
 
 ### Added

apps/backend/README.md

@@ -23,6 +23,8 @@ npm start
 npm run dev
 ```
 
+For watch mode, run `npm run dev -w @flowforge/backend` in another terminal.
+
 ## Configuration
 
 Configured via `.env` file:
@@ -38,11 +40,12 @@ See [.env.example](.env.example) for all options.
 
 ## API
 
-- `POST /query` — main agent entry point (question + page data)
-- `POST /search` — semantic search over indexed content
-- `GET /analytics` / `GET /health` — analytics and service status
+- `POST /query` — main agent entry point (`question`, `pageTrail`, `domain`)
+- `POST /search` — semantic search over an indexed `pageUrl`
+- `GET /health` — service status
+- `GET /analytics` — in-memory query analytics
 
-`/query` expects `question`, `pageTrail`, and `domain`; it returns answer, mode, optional topic, matched elements, and execution metadata.
+`/query` returns `{ result, metadata }`. `result` contains answer, mode, optional topic, and matched elements; `metadata` contains model, token usage, and execution time.
 
 ## Notes
 

apps/backend/src/agent/prompts.ts

@@ -32,7 +32,6 @@ Return only valid JSON:
   "elements": [
     {
       "dataId": "string",
-      "cssSelector": "string",
       "text": "string",
       "action": "click|input|navigate|select|highlight"
     }
@@ -62,17 +61,27 @@ ELEMENTS GENERAL RULES:
 - If the answer refers to a specific page element or text fragment, include that element in "elements"
 - If a tool returns a relevant element used for the answer, include it in "elements"
 - Return an empty "elements" array only when no valid relevant element is available from tool results
-- Map tool elementDataId to dataId and tool elementCssSelector to cssSelector exactly
+- Map tool elementDataId to dataId exactly
+- dataId is the primary locator; cssSelector is only an optional fallback
+- If tool elementCssSelector is present, map it to cssSelector exactly; if it is missing, omit cssSelector
 - Do not modify or invent them
 
+TOOL CONTEXT RULES:
+- Tool result semanticDescription/text describes the matched target itself
+- Tool result elementContext is a list of semantic container breadcrumbs around the target
+- Use elementContext to write location phrases such as "in the checkout form" or "in the primary navigation"
+- Prefer the nearest or most specific useful breadcrumb when final text must be short
+- Do not include elementContext in final elements[]
+
 WORKFLOW ELEMENTS RULES:
 - If find_workflow is used for the final answer, build "elements" from the returned "steps"
 - Include all clearly relevant returned items that help the user complete the task
 - Do not reduce the result to only one item if other returned items are also valid ways to achieve the goal
 - Prefer broader coverage over minimal sufficiency
 - Exclude only clearly irrelevant, duplicate, or contradictory items
 - Map each selected step into one item in "elements"
-- Map each step elementDataId to dataId and elementCssSelector to cssSelector exactly
+- Map each step elementDataId to dataId exactly
+- If step elementCssSelector is present, map it to cssSelector exactly; if it is missing, omit cssSelector
 - Rewrite only the user-facing "text" and choose the appropriate "action"
 
 CONTENT ELEMENTS RULES:
@@ -142,7 +151,7 @@ Extract valid JSON from the agent answer.
 TARGET SCHEMA:
 {
   "answer": string,
-  "elements": [{"dataId": string, "cssSelector": string, "text": string, "action": "click|navigate|input|select|highlight"}],
+  "elements": [{"dataId": string, "cssSelector"?: string, "text": string, "action": "click|navigate|input|select|highlight"}],
   "mode": "direct|steps",
   "topic": string | null
 }
@@ -152,8 +161,9 @@ RULES:
 - If valid JSON is present, extract it exactly
 - Do not reconstruct missing fields from prose
 - Do not create elements from answer text
-- Include elements only if full data (dataId and cssSelector) is provided
-- Omit elements with missing fields
+- Include elements only if dataId is provided
+- cssSelector is optional fallback data; include it only when provided
+- Omit elements with missing required fields
 - Do not invent values or use placeholders (e.g. "unknown")
 - Do not generate CSS selectors or ids from text
 - Keep empty arrays as empty arrays

apps/backend/src/agent/rank/scoring.ts

@@ -7,7 +7,7 @@ import type { RetrievedDocument } from '@/types';
  * Used when selecting the best matching UI element.
  */
 export function scoreForLookup(document: RetrievedDocument): number {
-    return 0.8 * document.semanticScore + 0.2 * document.metadata.element.importanceScore;
+    return 0.8 * document.semanticScore + 0.2 * document.metadata.element.importanceScore.value;
 }
 
 /**
@@ -17,7 +17,7 @@ export function scoreForLookup(document: RetrievedDocument): number {
  * Used for selecting text blocks that best answer the user query.
  */
 export function scoreForAnswer(document: RetrievedDocument): number {
-    return 0.85 * document.semanticScore + 0.15 * document.metadata.element.importanceScore;
+    return 0.85 * document.semanticScore + 0.15 * document.metadata.element.importanceScore.value;
 }
 
 /**
@@ -27,5 +27,5 @@ export function scoreForAnswer(document: RetrievedDocument): number {
  * require both relevant and actionable UI elements.
  */
 export function scoreForAction(document: RetrievedDocument): number {
-    return 0.7 * document.semanticScore + 0.3 * document.metadata.element.importanceScore;
+    return 0.7 * document.semanticScore + 0.3 * document.metadata.element.importanceScore.value;
 }

apps/backend/src/agent/tools/AbstractCallableTool.ts

@@ -1,7 +1,7 @@
 import type { DynamicStructuredTool } from '@langchain/core/tools';
 import { PageContextProvider } from '@/indexer';
 import type { CallableTool, CallableToolResult, CallableToolResultData, ToolResultElement } from '@/types';
-import { formantElementContextPath, type BaseElement } from '@flowforge/page-trail';
+import { semElementContextByBreadcrumbs, type TargetElement } from '@flowforge/page-trail';
 
 export abstract class AbstractCallableTool implements CallableTool {
     readonly name: string;
@@ -33,12 +33,11 @@ export abstract class AbstractCallableTool implements CallableTool {
         }
     }
 
-    protected getToolResultElement(element: BaseElement): ToolResultElement {
+    protected getToolResultElement(element: TargetElement): ToolResultElement {
         return {
-            elementPath: formantElementContextPath(element.context.path),
-            elementSectionName: element.context.sectionName ?? '',
             elementDataId: element.dataId,
-            elementCssSelector: element.cssSelector ?? '',
+            elementContext: semElementContextByBreadcrumbs(element.context),
+            elementCssSelector: element.cssSelector,
         };
     }
 

apps/backend/src/agent/tools/ToolFindElement.ts

@@ -49,13 +49,14 @@ WHEN TO USE:
 
 WHAT IT RETURNS:
 - Best matching element (if any)
-- Description of the element
-- Location context (section, path)
-- CSS selector and dataId
+- semanticDescription: semantic text describing the matched element
+- elementContext: semantic container breadcrumbs around the element, ordered from broader page area to nearer target area
+- elementDataId: primary browser locator
+- elementCssSelector: optional fallback browser locator
 
 IMPORTANT:
 - Returns only the best match, which may be imperfect
-- Use context to decide if it is correct and applicable
+- Use elementContext to decide if it is correct and to describe where it is located
 - If the result seems unclear or incomplete, consider using another tool`,
             schema: z.object({
                 query: z.string().describe('Element to find (e.g., "login button", "search input")'),

apps/backend/src/agent/tools/ToolFindWorkflow.ts

@@ -46,12 +46,15 @@ WHEN TO USE:
 
 WHAT IT RETURNS:
 - A list of relevant interactive elements (candidates)
-- Each includes description, location, CSS selector, and dataId
+- Each step includes semanticDescription, elementContext, elementDataId, and optional elementCssSelector
+- elementContext contains semantic container breadcrumbs around the element, ordered from broader page area to nearer target area
+- elementDataId is the primary browser locator; elementCssSelector is only an optional fallback
 
 IMPORTANT:
 - Results are candidates, not ordered steps
 - Select relevant items and arrange them into a logical sequence
 - Ignore irrelevant or duplicate items
+- Use elementContext to describe where a step is located when it helps the user
 - Some steps may be missing
 - Use other tools if needed to clarify or validate steps`,
             schema: z.object({

apps/backend/src/agent/tools/ToolGetPageSummary.ts

@@ -3,7 +3,7 @@ import { z } from 'zod';
 import { AbstractCallableTool } from './AbstractCallableTool.ts';
 import { PageContextProvider } from '@/indexer';
 import type { ToolGetPageSummaryResultData } from '@/types';
-import { formatSampleHeadings, formatSampleInteractions } from '@flowforge/page-trail';
+import { semSampleHeadings, semSampleInteractions } from '@flowforge/page-trail';
 
 export class ToolGetPageSummary extends AbstractCallableTool {
     private readonly elementsHeadingsLimit: number;
@@ -21,8 +21,8 @@ export class ToolGetPageSummary extends AbstractCallableTool {
             url: ctx.pageTrail.basics.url,
             description: ctx.pageTrail.basics.description,
             language: ctx.pageTrail.basics.language,
-            sampleHeadings: formatSampleHeadings(ctx.pageTrail.content, this.elementsHeadingsLimit),
-            sampleInteractions: formatSampleInteractions(ctx.pageTrail.interactive, this.elementsInteractionsLimit),
+            sampleHeadings: semSampleHeadings(ctx.pageTrail.content, this.elementsHeadingsLimit),
+            sampleInteractions: semSampleInteractions(ctx.pageTrail.interactive, this.elementsInteractionsLimit),
         };
     }
 

apps/backend/src/agent/tools/ToolSearchInContent.ts

@@ -46,11 +46,14 @@ WHEN TO USE:
 
 WHAT IT RETURNS:
 - Relevant content fragments from the page
-- Each fragment includes text and location context
+- Each fragment includes text, elementContext, elementDataId, and optional elementCssSelector
+- text is semantic page text for the matched content fragment
+- elementContext contains semantic container breadcrumbs around the fragment, ordered from broader page area to nearer target area
 
 IMPORTANT:
 - Results are partial matches, not guaranteed answers
 - You must interpret and combine them into a final answer
+- Use elementContext to describe where the supporting text appears when helpful
 - If needed, you can follow up with another tool to locate related elements`,
             schema: z.object({
                 query: z.string().describe('Topic to search for'),

apps/backend/src/indexer/transformers/AbstractDocumentTransformer.ts

@@ -1,6 +1,6 @@
 import type { IndexableDocument, DocumentTransformer } from '@/types';
 import { randomUUID } from 'crypto';
-import type { BaseElement, PageTrail } from '@flowforge/page-trail';
+import type { TargetElement, PageTrail } from '@flowforge/page-trail';
 
 export abstract class AbstractDocumentTransformer implements DocumentTransformer {
     readonly name: string;
@@ -15,7 +15,7 @@ export abstract class AbstractDocumentTransformer implements DocumentTransformer
         return randomUUID();
     }
 
-    protected createDocument(content: string, el: BaseElement): IndexableDocument {
+    protected createDocument(content: string, el: TargetElement): IndexableDocument {
         return {
             id: this.createDocumentId(),
             content,

apps/backend/src/indexer/transformers/ContentElementsTransformer.ts

@@ -1,11 +1,6 @@
 import type { IndexableDocument } from '@/types';
 import { AbstractDocumentTransformer } from './AbstractDocumentTransformer.ts';
-import {
-    type ContentElement,
-    type PageTrail,
-    formatContentElement,
-    formatContentElementShort,
-} from '@flowforge/page-trail';
+import { type ContentElement, type PageTrail, semContentElement } from '@flowforge/page-trail';
 import { RecursiveCharacterTextSplitter, TextSplitter } from '@langchain/textsplitters';
 
 export const CONTENT_TEMPLATE_TEXT_PLACEHOLDER = '{{TEXT}}';
@@ -56,11 +51,12 @@ export class ContentElementsTransformer extends AbstractDocumentTransformer {
     }
 
     private createContentTemplate(el: ContentElement): string {
-        const template = formatContentElement(el, CONTENT_TEMPLATE_TEXT_PLACEHOLDER);
+        const sr = semContentElement(el, CONTENT_TEMPLATE_TEXT_PLACEHOLDER);
+        const template = sr.text();
         // fallback if the template contains too many context data
         const templateContextSize = template.length - CONTENT_TEMPLATE_TEXT_PLACEHOLDER.length;
         if (templateContextSize > this.chunkSize * CONTENT_TEMPLATE_MAX_CONTEXT_RATIO) {
-            return formatContentElementShort(el, CONTENT_TEMPLATE_TEXT_PLACEHOLDER);
+            return sr.short();
         }
         return template;
     }

apps/backend/src/indexer/transformers/InteractiveElementsTransformer.ts

@@ -1,6 +1,6 @@
 import { AbstractDocumentTransformer } from './AbstractDocumentTransformer.ts';
 import type { IndexableDocument } from '@/types';
-import { formatInteractiveElement, type PageTrail } from '@flowforge/page-trail';
+import { type PageTrail, semInteractiveElement } from '@flowforge/page-trail';
 
 export class InteractiveElementsTransformer extends AbstractDocumentTransformer {
     constructor() {
@@ -10,7 +10,7 @@ export class InteractiveElementsTransformer extends AbstractDocumentTransformer
     override async transformFn(pageTrail: PageTrail): Promise<IndexableDocument[]> {
         const docs: IndexableDocument[] = [];
         for (const el of pageTrail.interactive) {
-            const content = formatInteractiveElement(el);
+            const content = semInteractiveElement(el).text();
             docs.push(this.createDocument(content, el));
         }
         return docs;

apps/backend/src/types/documents.ts

@@ -1,10 +1,10 @@
-import type { BaseElement, ElementKind, PageTrail } from '@flowforge/page-trail';
+import type { ElementKind, TargetElement, PageTrail } from '@flowforge/page-trail';
 
 export type DocumentType = ElementKind;
 
 export interface DocumentMetadata {
     type: DocumentType;
-    element: BaseElement;
+    element: TargetElement;
 }
 
 export interface Document {

apps/backend/src/types/tools.ts

@@ -28,15 +28,14 @@ export interface ToolGetPageSummaryResultData {
     url: string;
     description: string;
     language: string;
-    sampleHeadings: string;
-    sampleInteractions: string;
+    sampleHeadings: string[];
+    sampleInteractions: string[];
 }
 
 export interface ToolResultElement {
-    elementPath: string;
-    elementSectionName: string;
     elementDataId: string;
-    elementCssSelector: string;
+    elementContext: string[];
+    elementCssSelector?: string;
 }
 
 export interface ToolFindElementFoundResultData extends ToolResultElement {

apps/extension/README.md

@@ -34,8 +34,6 @@ npm run sandbox
 3. Click **Load unpacked**
 4. Select `apps/extension/dist/chrome`
 
-Sandbox opens `http://localhost:3007` with demo mode and backend mode.
-
 ## Embed runtime
 
 `build:embed` creates a bundle and declaration file under `dist/embed`:
@@ -55,11 +53,11 @@ await FlowForge.start({ settings: { theme: 'dark' } });
 - `popup/` — user interface and interaction logic
 - `page/` — page overlay, highlighting, wizard, inspector, and collection hooks
 - `background/`, `chrome/`, `embed/` — worker, extension shell, and embed runtime
-- `core/` / `adapters/` — API, storage, locator, root injection, and transport
+- `core/` and `adapters/` — API, storage, locator, root injection, and transport
 
 ## Notes
 
 - Chrome extension requires backend on http://localhost:3477
-- Embed integration supports backend mode and demo mode
+- Sandbox runs on http://localhost:3007 with backend and demo modes
 - Limited by browser security (iframes, cross-origin content)
 - See [Architecture](../../docs/ARCHITECTURE.md) for system design