Knowledge Graph: structured dataset ingestion and handling

[Weegy]

Problem

The Knowledge Graph currently handles document and chat ingestion well, but there is no defined path for structured datasets (tables, CSVs, database exports, and similar). Users who want to bring in structured data have to shoehorn it through the same pipeline built for unstructured content.

Proposed solution

Define and implement a dataset ingestion path into the Knowledge Graph that respects the shape of structured data (columns, types, relations) instead of treating it as plain text. Scope needs to cover at minimum: import, schema handling, and query/retrieval against ingested datasets.

[Weegy]

Triage

Priority: P1 · Category: feature · Effort: L (CSV-first v1; XL if XLSX/DB exports are in scope) · Status: needs design — storage model and privacy-on-import need a maintainer decision, the rest of the path is clear

Verified against main@1e5b24d: none of this exists yet on the ingestion side. The only structured-data ingestion touchpoint today is attachmentExtract.ts (issue #268), which routes text/csv through the plain-text path and hard-caps extraction at 20,000 chars (MAX_TEXT_CHARS) — confirming the issue's premise. (The output side already exists separately: harness-plugin-office generates XLSX/DOCX via create_xlsx/create_docx.) The existing ingestEntities/ingestFacts API is built for integration entities (Odoo, Confluence) and its docs explicitly warn to keep extras small because node properties are GIN-indexed, so exploding dataset rows into graph nodes is the wrong shape.

Implementation notes

Affected code:

• middleware/packages/harness-orchestrator/src/attachmentExtract.ts — current CSV path; branch CSV here to the dataset importer instead of the 20k-char plain-text cap
• middleware/packages/plugin-api/src/knowledgeGraph.ts — KnowledgeGraph interface (ingestTurn, ingestEntities, ingestFacts); add the dataset surface here so both backends implement it
• middleware/packages/harness-knowledge-graph-neon/src/ — neonKnowledgeGraph.ts + migrator.ts; migrations currently end at 0028_token_usage.sql, so the dataset tables land as 0029_*
• middleware/packages/harness-knowledge-graph-inmemory/ — needs parity or an explicit unsupported error for the new surface
• middleware/packages/harness-orchestrator/src/tools/readAttachmentTool.ts — pattern to follow for a new query_dataset native tool (spec + handler, wired via buildOrchestrator.ts / toolDispatchService.ts)
• middleware/src/routes/packages.ts (multipart upload pattern) and middleware/src/routes/memory.ts (auth/ACL pattern, /api/v1/memory) — templates for a new middleware/src/routes/datasets.ts
• web-ui/app/admin/ — new datasets page, following the existing kg-lifecycle / memory-backend pages

Suggested approach:

1. Storage (decision needed, recommendation below): relational sidecar in the same Neon pool (graphPool seam) — migration 0029: datasets (tenant/user scoping, name, source-file ref, inferred column schema as JSONB) + dataset_rows (dataset_id, row_index, data JSONB). One Dataset graph node per dataset (via the generic PluginEntity mapping, system='dataset') so recall/citations can link to it — rows never become graph nodes. Raw file blob goes to the existing Tigris store (precedent: 0013_teams_attachments.sql).
2. Import: POST /api/v1/datasets multipart route (pattern: packages.ts upload + memory.ts ACL). CSV parsing + column-type inference needs a new dependency (csv-parse or similar — the middleware currently has no CSV parser). Second entry point: the chat-attachment auto-ingest branch in attachmentExtract.ts.
3. Query/retrieval: query_dataset native tool — list datasets + schemas, then filter/aggregate over dataset_rows via parameterized JSONB queries or a constrained query DSL (never raw SQL from the LLM). Results must page/aggregate, never dump full tables into the turn context. Embed dataset name + column descriptions via harness-embeddings so datasets surface in recall.
4. Admin UI: upload / schema view / delete page under web-ui/app/admin/.

Risks / dependencies:

• Migration numbering: 0029 is next free in harness-knowledge-graph-neon/src/migrations; the in-flight MCP epic PR feat(mcp): MCP as a first-class capability across Orchestrator/Sub-Agent/Skill/Plugin + Control Center (epic #459) #464 also adds migrations (separate set under middleware/migrations/) — re-verify numbering at merge time.
• Privacy: imported rows bypass the turn-based privacy-guard pipeline entirely; whether privacy-detector plugins run on import (and at what cost for large files) is an open decision.
• Both KG backends implement the KnowledgeGraph interface — the in-memory one must not silently no-op.
• Scope creep: XLSX reading could reuse exceljs (already a dependency of harness-plugin-office), but sheet/type handling and DB-export ingestion add real complexity; recommend CSV-first with follow-up issues.

Acceptance criteria:

• CSV upload via REST and via chat attachment creates a dataset with inferred column types
• query_dataset answers a filter + aggregate question over an ingested dataset without exceeding the turn token budget
• CSVs are no longer silently truncated at 20,000 chars in attachmentExtract.ts
• Datasets are tenant/user-scoped (ACL model consistent with /api/v1/memory) and deletable via admin UI
• In-memory KG backend behavior for datasets is defined (parity or explicit error), covered by tests

---

Automated triage against main@1e5b24d; posted after adversarial fact-check review.

[Weegy]

Triage — status delta

Priority: P1 · Category: feature · Effort: L · Status: needs design (unchanged)

Re-verified against 327ddf8. The prior triage plan above still holds in full — nothing implementing this has shipped:

• No dataset ingestion surface exists: no query_dataset tool under middleware/packages/harness-orchestrator/src/tools/ (the attachment reader there is readAttachmentTool.ts, the template the prior comment named), no dataset_rows table, no datasets.ts route. The datasetStore.ts in harness-plugin-privacy-guard/src/v4/ is unrelated — it is Privacy Shield v4's turn-scoped, in-memory registry of interned tool-result rows, not structured dataset ingestion.
• attachmentExtract.ts still routes text/csv (line 60) through the plain-text path capped at MAX_TEXT_CHARS = 20_000 (line 17), confirming the issue's premise.
• KnowledgeGraph (plugin-api/src/knowledgeGraph.ts) still exposes no dataset ingestion surface — its ingest methods remain turn/entity/fact-shaped (ingestTurn / ingestEntities / ingestFacts, alongside ingestRun / ingestPlan).

What changed since the prior comment: PR #464 (MCP epic) — the in-flight dependency flagged for migration re-verification — is now merged and is HEAD (327ddf8). Re-verification result:

• Neon KG migrations are unchanged, still ending at 0028_token_usage.sql, so 0029 remains the next free number for the proposed datasets / dataset_rows tables in harness-knowledge-graph-neon/src/migrations/.
• feat(mcp): MCP as a first-class capability across Orchestrator/Sub-Agent/Skill/Plugin + Control Center (epic #459) #464's migrations are a separate set under middleware/migrations/ (now ending 0019_mcp_server_config.sql) — no collision with the neon KG set.

The migration-numbering risk from the prior comment is therefore resolved. The open items remain the storage-model and privacy-on-import decisions, which still need a maintainer call — status stays needs-design. The acceptance criteria in the prior comment stand as-is.

---

Automated triage against 327ddf8; posted after adversarial fact-check review.

[Weegy]

Scope reduction — everything except the admin UI has shipped.

Delivered by PR #533 (aec980da), verified against origin/main:

• API surface: middleware/packages/plugin-api/src/knowledgeGraph.ts:706-740 — ingestDataset, listDatasets, getDataset, queryDatasetRows, deleteDataset.
• Storage: middleware/packages/harness-knowledge-graph-neon/src/migrations/0029_datasets.sql, with in-memory parity in inMemoryKnowledgeGraph.ts.
• Routes: middleware/src/routes/datasets.ts, mounted at middleware/src/index.ts:2627.
• Agent access: middleware/packages/harness-orchestrator/src/tools/queryDatasetTool.ts.
• CSV no longer goes through the plain-text path and is no longer 20k-truncated: attachmentExtract.ts:139-148.

Remaining acceptance criterion: "deletable via admin UI". web-ui/app/admin/datasets/ does not exist on main. That work is fully tracked by #532 (PR #598 — CI green, currently CHANGES_REQUESTED with 4 must-fix items).

This issue is therefore reduced to a single open criterion, which #532 owns. Close this once #532 lands; no separate work is needed here.