docs: rewrite README to match current scope — HTTP fetch + crawl, no browser

Author: aksOps

README.md

@@ -4,7 +4,7 @@ Fetch web pages as clean markdown for AI coding agents.
 
 [![CI](https://github.com/RandomCodeSpace/rawdoc/actions/workflows/ci.yml/badge.svg)](https://github.com/RandomCodeSpace/rawdoc/actions/workflows/ci.yml)
 
-Single Go binary. No runtime downloads, no external services, no AI, no search.
+Single Go binary. Fetches HTML, strips noise, outputs markdown. Supports single-page fetch and multi-page crawling by depth.
 
 ---
 
@@ -14,20 +14,30 @@ Single Go binary. No runtime downloads, no external services, no AI, no search.
 go install github.com/RandomCodeSpace/rawdoc@latest
 ```
 
-Single binary, no AV flags, no runtime deps.
+---
+
+## What It Does
+
+1. **Fetches** HTML via plain HTTP with browser-like headers
+2. **Strips** noise — scripts, styles, navbars, footers, ads, cookie banners, hidden elements
+3. **Extracts** main content using site-specific selectors or readability scoring
+4. **Converts** to clean markdown (headings, code blocks, tables, lists)
+5. **Crawls** linked pages when given a depth > 0
+
+Works on server-rendered sites. JS-only SPAs (React, Next.js) are not supported.
 
 ---
 
-## Quick Start
+## Usage
 
 ```bash
-# Single page
+# Single page → stdout
 rawdoc https://kubernetes.io/docs/concepts/workloads/pods/
 
-# Just code blocks
+# Just the code blocks
 rawdoc https://www.baeldung.com/spring-kafka --code-only
 
-# JSON output
+# JSON output with metadata
 rawdoc https://pkg.go.dev/fmt -f json
 
 # YAML output
@@ -36,123 +46,105 @@ rawdoc https://pkg.go.dev/fmt -f yaml
 # Save to file
 rawdoc https://example.com -o docs.md
 
-# Crawl docs to directory
+# Crawl docs to a directory (depth=2, max 50 pages)
 rawdoc https://kubernetes.io/docs/concepts/workloads/ -d 2 -o ~/docs/k8s/
 
-# Verbose — see tier decisions and token stats
+# Verbose — see fetch decisions and token stats
 rawdoc https://www.baeldung.com/spring-kafka -v
 ```
 
----
-
-## How It Works
+### Verbose Output
 
-Plain HTTP with full browser headers. Works for most documentation sites. JS-rendered pages (React, Next.js) are not supported — they require a real browser to execute JavaScript.
+```
+[tier1] https://pkg.go.dev/fmt → fetching
+[stats] input: 139.2KB (35634 tokens) → output: 43.5KB (11135 tokens) | 69% saved
+[output] wrote json to docs.json
+```
 
-Processing pipeline: **Fetch → Strip noise → Extract content → Convert to Markdown**
+All verbose output goes to stderr. stdout stays clean for piping.
 
 ---
 
-## CLI Reference
-
-```
-rawdoc [flags] <url>
-```
+## Flags
 
 ### Output
 
 | Flag | Default | Description |
 |------|---------|-------------|
-| `-o, --output PATH` | stdout | File or directory |
-| `-f, --format string` | `markdown` | `markdown\|text\|json\|yaml` |
+| `-o, --output` | stdout | File or directory |
+| `-f, --format` | `markdown` | `markdown` `text` `json` `yaml` |
 | `--code-only` | — | Extract only code blocks |
 | `--no-links` | — | Strip link URLs, keep text only |
 
 ### Crawling
 
 | Flag | Default | Description |
 |------|---------|-------------|
-| `-d, --depth int` | `0` | Crawl depth, 0 = single page |
-| `-c, --concurrency int` | `5` | Parallel fetches |
-| `--max-pages int` | `50` | Page limit |
-| `--delay duration` | `1s` | Delay between requests |
-| `--include string` | — | URL path glob to include |
-| `--exclude string` | — | URL path glob to exclude |
+| `-d, --depth` | `0` | Crawl depth (0 = single page) |
+| `-c, --concurrency` | `5` | Parallel fetches |
+| `--max-pages` | `50` | Page limit |
+| `--delay` | `1s` | Delay between requests |
+| `--include` | — | URL path glob to include |
+| `--exclude` | — | URL path glob to exclude |
 | `--sitemap` | — | Parse sitemap.xml for URL discovery |
 
 ### HTTP
 
 | Flag | Default | Description |
 |------|---------|-------------|
-| `--timeout duration` | `15s` | Request timeout |
-| `--max-time duration` | `10m` | Total runtime ceiling |
-| `--max-retries int` | `3` | Per-URL retries |
+| `--timeout` | `15s` | Per-request timeout |
+| `--max-time` | `10m` | Total runtime ceiling |
+| `--max-retries` | `3` | Per-URL retries with exponential backoff |
 | `--header K=V` | — | Extra header (repeatable) |
 
 ### Info
 
 | Flag | Description |
 |------|-------------|
-| `-v, --verbose` | Log fetch/tier decisions and token stats to stderr |
-| `-q, --quiet` | Suppress all stderr output |
+| `-v, --verbose` | Fetch log and token stats to stderr |
+| `-q, --quiet` | Suppress all stderr |
 | `--version` | Print version |
 
 ---
 
-## Output Formats
-
-| Format | Description |
-|--------|-------------|
-| `markdown` | Clean markdown with headings, lists, code blocks (default) |
-| `text` | Plain text, no markup |
-| `json` | Structured JSON with metadata (url, title, content, stats) |
-| `yaml` | Same as JSON but YAML-encoded |
-| `--code-only` | Extracts only fenced code blocks from the page |
-
----
-
-## Verbose Mode & Token Stats
-
-```
-[tier1] https://pkg.go.dev/fmt → fetching
-[stats] input: 139.2KB (35634 tokens) → output: 43.5KB (11135 tokens) | 69% saved
-[output] wrote json to docs.json
-```
-
-All verbose output goes to stderr, keeping stdout clean for piping.
-
----
-
 ## Crawl Mode
 
-Set `-d` to a depth greater than 0 to crawl linked pages under the same origin path.
-
 ```bash
-rawdoc https://kubernetes.io/docs/concepts/workloads/ -d 2 -o ~/docs/k8s/
+rawdoc https://kubernetes.io/docs/concepts/workloads/ -d 2 --max-pages 50 -o ~/docs/k8s/
 ```
 
-Output directory structure mirrors the URL path:
+Writes one `.md` file per page plus an `index.md`:
 
 ```
 ~/docs/k8s/
 ├── index.md
-├── pods/
-│   └── index.md
-├── deployments/
-│   └── index.md
-└── replicasets/
-    └── index.md
+├── workloads.md
+├── workloads-pods.md
+├── workloads-controllers-deployment.md
+└── ...
 ```
 
-Use `--sitemap` to seed the crawl from `sitemap.xml` instead of link-following.
+Stays on the same domain. Respects `--include`/`--exclude` globs and `--max-pages` limit.
+
+---
+
+## Output Formats
+
+| Format | Description |
+|--------|-------------|
+| `markdown` | Headings, code blocks, tables, lists (default) |
+| `text` | Plain text, no markup |
+| `json` | Structured: url, title, content, code_blocks, fetch_tier, token count |
+| `yaml` | Same fields as JSON |
+| `--code-only` | Only fenced code blocks from the page |
 
 ---
 
 ## Site-Specific Selectors
 
-rawdoc ships with content-extraction rules for popular doc platforms and sites, so boilerplate (navbars, footers, ads) is stripped automatically:
+Built-in content selectors for: Baeldung, Docusaurus, GitBook, ReadTheDocs, MkDocs, Spring.io, GitHub, MDN, Go pkg.dev, StackOverflow, Medium, Dev.to, Confluence, Notion.
 
-Baeldung, Docusaurus, GitBook, ReadTheDocs, MkDocs, Hugo, Spring.io, GitHub, MDN, Go pkg.dev, StackOverflow, Medium, Dev.to, Confluence, Notion
+Falls back to readability scoring when no selector matches.
 
 ---
 
@@ -161,25 +153,17 @@ Baeldung, Docusaurus, GitBook, ReadTheDocs, MkDocs, Hugo, Spring.io, GitHub, MDN
 ### Claude Code (`CLAUDE.md`)
 
 ```markdown
-## Fetching Documentation
-
-Use `rawdoc` to fetch external docs as markdown before answering questions about them:
-
-```bash
-rawdoc <url>                  # pipe to stdin or save with -o
-rawdoc <url> -f json          # structured output with metadata
-rawdoc <url> --code-only      # grab only code examples
-```
+## Tools
+rawdoc <url>              — fetch docs as markdown
+rawdoc <url> --code-only  — code blocks only
+rawdoc <url> -f json      — structured output
+rawdoc <url> -d 2 -o dir/ — crawl to local directory
 ```
 
-### Any agent via shell
+### Any Agent
 
 ```bash
-# Pipe directly into your agent
-rawdoc https://pkg.go.dev/net/http | your-agent-cli
-
-# Save first, reference later
-rawdoc https://docs.example.com/api -o /tmp/api-docs.md
+result=$(rawdoc https://docs.example.com/api)
 ```
 
 ---
@@ -189,12 +173,12 @@ rawdoc https://docs.example.com/api -o /tmp/api-docs.md
 | Code | Meaning |
 |------|---------|
 | `0` | Success |
-| `1` | Fetch failure (network error, all tiers exhausted) |
+| `1` | Fetch failure |
 | `2` | Usage error (bad flags, invalid URL) |
 
 ---
 
-## Building from Source
+## Building
 
 ```bash
 git clone https://github.com/RandomCodeSpace/rawdoc.git
@@ -205,21 +189,17 @@ go build -o rawdoc .
 Cross-compile:
 
 ```bash
-# Linux (amd64)
-GOOS=linux GOARCH=amd64 go build -o rawdoc-linux-amd64 .
-
-# Windows (amd64)
-GOOS=windows GOARCH=amd64 go build -o rawdoc-windows-amd64.exe .
-
-# macOS (Apple Silicon)
-GOOS=darwin GOARCH=arm64 go build -o rawdoc-darwin-arm64 .
+GOOS=linux   GOARCH=amd64 go build -o rawdoc-linux-amd64 .
+GOOS=windows GOARCH=amd64 go build -o rawdoc.exe .
+GOOS=darwin  GOARCH=arm64 go build -o rawdoc-darwin-arm64 .
 ```
 
----
+**Requires:** Go 1.24+
 
-## Requirements
+---
 
-| Requirement | Notes |
-|-------------|-------|
-| Go 1.24+ | Required to build from source |
+## Limitations
 
+- **JS-rendered pages** (React SPAs, Next.js CSR, Angular) return empty content — rawdoc uses plain HTTP, not a browser
+- **CAPTCHA/login-gated pages** — returns whatever the public page shows
+- **Single IP** — not designed for large-scale scraping or proxy rotation