Skip to content

Report which documentation files were excluded or lost during ingestion #143

Description

@TheRealBecks

A successful documentation build does not currently explain which source documents reached the package, which were intentionally excluded, or which were lost to read/parse failures. That makes missing documentation difficult to distinguish from an unsuccessful search.

There is already a useful skippedFiles count; this proposal extends it into actionable diagnostics rather than replacing it.

At upstream commit 149f112:

  • findMarkdownFiles() applies filename, directory, language and ignore filters. Directory read failures are caught without a diagnostic.
  • readLocalDocsFiles() deduplicates content and catches file-read failures without returning their paths or reasons.
  • buildPackage() counts split/parse failures, but does not expose the affected paths or failure reasons. A file producing no sections is also different from a parser throwing.
  • fix(context): only skip repo-meta filenames at the scan root #125 demonstrates the practical consequence: valid documentation named security.md can disappear while the build succeeds. That PR fixes a filtering rule; this issue makes similar omissions diagnosable throughout the pipeline.

Proposed behavior:

Return structured ingestion diagnostics alongside the existing build result. Distinguish intentional exclusions, duplicate content, unreadable files/directories, split/parse failures, files producing no sections, and successfully indexed documents.

The CLI should print a concise summary and offer detailed or machine-readable output with relative paths and reasons. Keep diagnostics in the result rather than requiring library code to log directly. Avoid flooding the report with unrelated source-code files when scanning a repository.

Acceptance criteria:

  • A small fixture with exclusions, duplicate documents, a read failure, a parse failure, and an empty document produces distinct, inspectable outcomes.
  • Git, ZIP, and HTML-index sources can report outcomes through the same vocabulary.
  • The summary distinguishes discovered documentation, selected files, indexed documents, and indexed sections.
  • Representative Markdown, HTML, and RST fixtures assert that expected headings and examples survive ingestion.
  • Unexpected loss can fail an explicit strict validation mode; intentional exclusions remain distinguishable from failures.

Related: #125 and #137. The objective is extraction visibility and fidelity; retrieval ranking can be evaluated separately.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions