Skip to content

feat: enhance flattener with source file downloading and retry logic - #3

Merged
igorrius merged 1 commit into
mainfrom
files
Dec 18, 2025
Merged

igorrius merged 1 commit into
mainfrom
files

Conversation

@igorrius

Copy link
Copy Markdown
Owner

This pull request introduces major improvements to the flatten-go-doc tool by refactoring its architecture for modularity, adding robust retry logic, and enabling automatic downloading of Go source files referenced in documentation. It also updates the README with new features and NotebookLM usage tips, and expands test coverage to validate source file handling. The changes are organized into new modular components (Scraper, Converter, Source Handler) for better maintainability and extensibility.

Major new features and architectural improvements:

  • Added automatic identification and downloading of linked Go source files in documentation, appending their contents to the output with code block formatting. This provides full library context in the generated Markdown. [1] [2] [3]
  • Introduced robust retry logic with exponential backoff for both scraping and source file downloads, reducing failures due to transient network issues. [1] [2] [3]
  • Refactored the codebase into modular components: Scraper (handles scraping and retries), ContentConverter (handles HTML-to-Markdown conversion), and Source Handler (handles source file detection and download). This improves maintainability and extensibility. [1] [2] [3] [4] [5]

Documentation and usability:

  • Updated README.md to document new features, including source file downloading, retry logic, and the new modular architecture. Added a section on using the tool with Google NotebookLM for enhanced AI-powered code understanding. [1] [2]

Testing enhancements:

  • Expanded tests to verify that source files are correctly detected, downloaded, and included in the output with proper formatting. Adjusted test assertions for the new behavior. [1] [2] [3] [4] [5]

Internal structure and configuration:

  • Moved configuration structs and default values to a new models.go file, adding a MaxRetries field for controlling retry behavior.
  • Cleaned up and reorganized code in flattener.go to use the new modular components and to simplify the scraping logic. [1] [2] [3]

@igorrius igorrius self-assigned this Dec 18, 2025
@igorrius igorrius added documentation Improvements or additions to documentation enhancement New feature or request labels Dec 18, 2025
@igorrius
igorrius merged commit a27af86 into main Dec 18, 2025
1 check passed
@igorrius
igorrius deleted the files branch December 18, 2025 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant