Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ version = "0.1.8"
edition = "2024"
# Edition 2024 needs 1.85; zip and calamine both raise it to 1.88.
rust-version = "1.88"
description = "Convert documents (doc, docx, odt, rtf, epub, pdf, presentations, spreadsheets, csv) to GitHub-Flavored Markdown"
description = "Convert documents (doc, docx, odt, rtf, epub, html, pdf, presentations, spreadsheets, csv) to GitHub-Flavored Markdown"
license = "MIT"
repository = "https://github.com/firecrawl/anydoc"
readme = "README.md"
Expand Down
9 changes: 5 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![skills.sh](https://skills.sh/b/firecrawl/anydoc)](https://skills.sh/firecrawl/anydoc)

Fast Rust library that converts documents (Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV, and PDF) into clean GitHub-Flavored Markdown. Includes bindings for [Node.js](node/README.md), [Python](python/README.md), and the [browser](wasm/README.md) (WebAssembly).
Fast Rust library that converts documents (Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, HTML, CSV, and PDF) into clean GitHub-Flavored Markdown. Includes bindings for [Node.js](node/README.md), [Python](python/README.md), and the [browser](wasm/README.md) (WebAssembly).

Built by [Firecrawl](https://firecrawl.dev) to turn any office document into LLM-ready Markdown in single-digit milliseconds, with one consistent output no matter which format goes in. It powers [Firecrawl Parse](https://firecrawl.dev/parse), so if you'd rather not run it yourself, the hosted API gives you the same conversion plus our OCR models for the scanned pages anydoc can't read on its own.

Expand Down Expand Up @@ -130,7 +130,7 @@ let document = anydoc::to_document(&bytes, None)?;
- **One output for every format.** Each format parses into a shared document model and renders through a single Markdown serializer, so escaping, tables, heading anchors, and footnotes behave identically whether the input was a `.doc` from 2003 or a `.pptx` from yesterday.
- **Full document structure.** Headings with anchors, bold/italic/strikethrough, inline code and code blocks, links and internal cross-references, bulleted/numbered/nested/task lists with the source's own numbering, tables with merged cells and header rows, block quotes, footnotes and endnotes, and speaker notes.
- **Embedded assets.** Images and embedded objects render as their alt text in the Markdown, and the raw bytes stay available on the document model, tagged with their media type. Images with an external URL become ordinary Markdown images.
- **Content-based format detection.** The format is read from the bytes themselves (PDF header, RTF open group, OLE stream names, ZIP package mimetype), so mislabeled files still convert correctly.
- **Content-based format detection.** The format is read from the bytes themselves (PDF header, RTF open group, OLE stream names, ZIP package mimetype, or an HTML doctype/root element), so mislabeled files still convert correctly.
- **Fast.** Pure Rust, no ML models, no external services. Median conversion time is under 5ms per document.
- **Bindings that stay out of the way.** Node.js conversion runs on the libuv thread pool and never blocks the event loop; Python releases the GIL so other threads keep running. TypeScript types and Python stubs ship with the packages.
- **PDF support built in.** Text-based PDFs convert locally through [pdf-inspector](https://github.com/firecrawl/pdf-inspector), no OCR service required.
Expand All @@ -146,6 +146,7 @@ let document = anydoc::to_document(&bytes, None)?;
| OpenDocument | `.odt`, `.ods`, `.odp` |
| Rich Text Format | `.rtf` |
| EPUB | `.epub` |
| HTML | `.html`, `.htm` |
| CSV | `.csv` |
| PDF | `.pdf` |

Expand Down Expand Up @@ -189,7 +190,7 @@ Speed is one warm conversion per document on a Ryzen 9 9950X3D (Windows 11, 64 G

## Format detection

The format is read from the file content, using the marker its specification designates: the PDF header, the RTF open group, OLE stream names, the ZIP package mimetype and content types. CSV has no such marker, so the extension or an explicit format names it instead.
The format is read from the file content, using the marker its specification designates: the PDF header, the RTF open group, OLE stream names, the ZIP package mimetype and content types, or an HTML doctype/root element. CSV has no such marker, so the extension or an explicit format names it instead. Standalone HTML files can therefore be recognized even when an exporter gives them a `.doc` extension; MHTML is not yet supported.

```rust
Format::from_bytes(&bytes); // Some(Format::Docx), or None when nothing matches
Expand Down Expand Up @@ -234,7 +235,7 @@ document bytes
├─► format detection → content markers, not the extension
├─► format parser → one per format (doc, docx, ppt, pptx, xls,
│ xlsx, odt/ods/odp, rtf, epub, csv)
│ xlsx, odt/ods/odp, rtf, epub, html, csv)
│ │
│ └─► Document → shared model: blocks, inlines, tables,
│ footnotes, assets
Expand Down
5 changes: 3 additions & 2 deletions node/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
[![npm](https://img.shields.io/npm/v/@firecrawl/anydoc.svg)](https://www.npmjs.com/package/@firecrawl/anydoc)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/firecrawl/anydoc/blob/main/LICENSE)

Convert Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV, and PDF files into clean GitHub-Flavored Markdown. Node.js bindings for the [anydoc](https://github.com/firecrawl/anydoc) Rust crate, built by [Firecrawl](https://firecrawl.dev). Also available as a hosted API through [Firecrawl Parse](https://firecrawl.dev/parse), which adds our OCR models for the scanned pages anydoc can't read on its own.
Convert Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, HTML, CSV, and PDF files into clean GitHub-Flavored Markdown. Node.js bindings for the [anydoc](https://github.com/firecrawl/anydoc) Rust crate, built by [Firecrawl](https://firecrawl.dev). Also available as a hosted API through [Firecrawl Parse](https://firecrawl.dev/parse), which adds our OCR models for the scanned pages anydoc can't read on its own.

Every format parses into one shared document model and renders through a single Markdown serializer, so headings, tables, lists, and footnotes come out the same no matter which format goes in. Conversion runs on the libuv thread pool and never blocks the event loop. TypeScript types ship with the package.

Expand All @@ -21,6 +21,7 @@ npm install @firecrawl/anydoc
| OpenDocument | `.odt`, `.ods`, `.odp` |
| Rich Text Format | `.rtf` |
| EPUB | `.epub` |
| HTML | `.html`, `.htm` |
| CSV | `.csv` |
| PDF | `.pdf` |

Expand Down Expand Up @@ -84,7 +85,7 @@ try {

## Format detection

The format is read from the file content, using the marker its specification designates: the PDF header, the RTF open group, OLE stream names, the ZIP package mimetype and content types. CSV has no such marker, so detection returns `null` for it and the extension, or an explicit format, names it instead.
The format is read from the file content, using the marker its specification designates: the PDF header, the RTF open group, OLE stream names, the ZIP package mimetype and content types, or an HTML doctype/root element. CSV has no such marker, so detection returns `null` for it and the extension, or an explicit format, names it instead. Standalone HTML is supported; MHTML is not yet supported.

```js
formatFromBytes(bytes); // 'docx', or null when nothing matches
Expand Down
2 changes: 1 addition & 1 deletion node/cli.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

const { readFile, writeFile } = require('node:fs/promises')

const FORMATS = 'doc, docx, odt, pdf, ppt, pptx, rtf, epub, xlsx, ods, odp, csv'
const FORMATS = 'doc, docx, odt, pdf, ppt, pptx, rtf, epub, html, xlsx, ods, odp, csv'

const HELP = `anydoc: convert documents to GitHub-Flavored Markdown

Expand Down
6 changes: 4 additions & 2 deletions node/index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ export declare const enum Format {
pptx = 'pptx',
rtf = 'rtf',
epub = 'epub',
html = 'html',
xlsx = 'xlsx',
ods = 'ods',
odp = 'odp',
Expand All @@ -119,8 +120,9 @@ export declare const enum Format {
/**
* Detect the format from the content itself: the signature and identity each
* container specification designates (PDF header, RTF open group, OLE stream
* names, ZIP package mimetype/content types). Plain-text formats (CSV) carry
* no signature and return `null`; so does anything unrecognized.
* names, ZIP package mimetype/content types, or an HTML doctype/root element).
* Plain-text formats (CSV) carry no signature and return `null`; so does
* anything unrecognized.
*/
export declare function formatFromBytes(bytes: Uint8Array): Format | null

Expand Down
2 changes: 1 addition & 1 deletion node/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@firecrawl/anydoc",
"version": "0.1.8",
"description": "Convert documents (doc, docx, odt, rtf, epub, pdf, presentations, spreadsheets, csv) to GitHub-Flavored Markdown",
"description": "Convert documents (doc, docx, odt, rtf, epub, html, pdf, presentations, spreadsheets, csv) to GitHub-Flavored Markdown",
"license": "MIT",
"homepage": "https://github.com/firecrawl/anydoc#readme",
"repository": {
Expand Down
8 changes: 6 additions & 2 deletions node/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ pub enum Format {
pptx,
rtf,
epub,
html,
xlsx,
ods,
odp,
Expand All @@ -42,6 +43,7 @@ impl From<Format> for anydoc::Format {
Format::pptx => anydoc::Format::Pptx,
Format::rtf => anydoc::Format::Rtf,
Format::epub => anydoc::Format::Epub,
Format::html => anydoc::Format::Html,
Format::xlsx => anydoc::Format::Excel,
Format::ods => anydoc::Format::Ods,
Format::odp => anydoc::Format::Odp,
Expand All @@ -61,6 +63,7 @@ impl From<anydoc::Format> for Format {
anydoc::Format::Pptx => Format::pptx,
anydoc::Format::Rtf => Format::rtf,
anydoc::Format::Epub => Format::epub,
anydoc::Format::Html => Format::html,
anydoc::Format::Excel => Format::xlsx,
anydoc::Format::Ods => Format::ods,
anydoc::Format::Odp => Format::odp,
Expand All @@ -71,8 +74,9 @@ impl From<anydoc::Format> for Format {

/// Detect the format from the content itself: the signature and identity each
/// container specification designates (PDF header, RTF open group, OLE stream
/// names, ZIP package mimetype/content types). Plain-text formats (CSV) carry
/// no signature and return `null`; so does anything unrecognized.
/// names, ZIP package mimetype/content types, or an HTML doctype/root element).
/// Plain-text formats (CSV) carry no signature and return `null`; so does
/// anything unrecognized.
#[napi]
pub fn format_from_bytes(bytes: Uint8Array) -> Option<Format> {
anydoc::Format::from_bytes(&bytes).map(Format::from)
Expand Down
4 changes: 4 additions & 0 deletions node/test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ const OUTLINE = fixture('docx/handmade-outline.docx')
const RICH = fixture('docx/handmade-rich.docx')
const CSV = fixture('csv/sheet.csv')
const ENCRYPTED = fixture('malformed/encrypted--errors.odt')
const HTML = Buffer.from('<!doctype html><html><body><h1>HTML input</h1></body></html>')

test('toMarkdown detects the format from the file content', async () => {
const markdown = await toMarkdown(OUTLINE)
Expand All @@ -40,6 +41,7 @@ test('toMarkdownBytes detects the format when none is named', async () => {
// CSV carries no signature, so it has to be named.
await assert.rejects(toMarkdownBytes(await readFile(CSV)), /unrecognized file content/)
assert.match(await toMarkdownBytes(await readFile(CSV), 'csv'), /\| --- \|/)
assert.match(await toMarkdownBytes(HTML), /^# HTML input/m)
})

test('toDocument exposes the document model', async () => {
Expand All @@ -61,10 +63,12 @@ test('toDocument carries embedded assets as buffers', async () => {

test('format detection reads content, extension, and path', async () => {
assert.equal(formatFromBytes(await readFile(RICH)), 'docx')
assert.equal(formatFromBytes(HTML), 'html')
// CSV carries no signature: only the extension names it.
assert.equal(formatFromBytes(await readFile(CSV)), null)
assert.equal(formatFromExtension('.pptm'), 'pptx')
assert.equal(formatFromExtension('xls'), 'xlsx')
assert.equal(formatFromExtension('htm'), 'html')
assert.equal(formatFromPath('/tmp/report.odt'), 'odt')
assert.equal(formatFromPath('/tmp/report.unknown'), null)
})
Expand Down
5 changes: 3 additions & 2 deletions python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
[![PyPI](https://img.shields.io/pypi/v/firecrawl-anydoc.svg)](https://pypi.org/project/firecrawl-anydoc/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/firecrawl/anydoc/blob/main/LICENSE)

Convert Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV, and PDF files into clean GitHub-Flavored Markdown. Python bindings for the [anydoc](https://github.com/firecrawl/anydoc) Rust crate, built by [Firecrawl](https://firecrawl.dev). Also available as a hosted API through [Firecrawl Parse](https://firecrawl.dev/parse), which adds our OCR models for the scanned pages anydoc can't read on its own.
Convert Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, HTML, CSV, and PDF files into clean GitHub-Flavored Markdown. Python bindings for the [anydoc](https://github.com/firecrawl/anydoc) Rust crate, built by [Firecrawl](https://firecrawl.dev). Also available as a hosted API through [Firecrawl Parse](https://firecrawl.dev/parse), which adds our OCR models for the scanned pages anydoc can't read on its own.

Every format parses into one shared document model and renders through a single Markdown serializer, so headings, tables, lists, and footnotes come out the same no matter which format goes in. Conversion releases the GIL, so other threads keep running. Type stubs ship with the package.

Expand All @@ -23,6 +23,7 @@ The package installs as `firecrawl-anydoc` and imports as `anydoc`.
| OpenDocument | `.odt`, `.ods`, `.odp` |
| Rich Text Format | `.rtf` |
| EPUB | `.epub` |
| HTML | `.html`, `.htm` |
| CSV | `.csv` |
| PDF | `.pdf` |

Expand Down Expand Up @@ -70,7 +71,7 @@ The five conversion failures subclass `anydoc.ConvertError`, so catching that ha

## Format detection

The format is read from the file content, using the marker its specification designates: the PDF header, the RTF open group, OLE stream names, the ZIP package mimetype and content types. CSV has no such marker, so detection returns `None` for it and the extension, or an explicit format, names it instead.
The format is read from the file content, using the marker its specification designates: the PDF header, the RTF open group, OLE stream names, the ZIP package mimetype and content types, or an HTML doctype/root element. CSV has no such marker, so detection returns `None` for it and the extension, or an explicit format, names it instead. Standalone HTML is supported; MHTML is not yet supported.

```python
anydoc.format_from_bytes(data) # 'docx', or None when nothing matches
Expand Down
2 changes: 1 addition & 1 deletion python/anydoc/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
)

Format = Literal[
"doc", "docx", "odt", "pdf", "ppt", "pptx", "rtf", "epub", "xlsx", "ods", "odp", "csv"
"doc", "docx", "odt", "pdf", "ppt", "pptx", "rtf", "epub", "html", "xlsx", "ods", "odp", "csv"
]
"""Input format, named after the extension that identifies it. Container
variants that share a parser (`.docm`, `.xlsm`, `.ppsx`, ...) map onto these
Expand Down
8 changes: 4 additions & 4 deletions python/anydoc/_anydoc.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import os
from typing import Literal, final

Format = Literal[
"doc", "docx", "odt", "pdf", "ppt", "pptx", "rtf", "epub", "xlsx", "ods", "odp", "csv"
"doc", "docx", "odt", "pdf", "ppt", "pptx", "rtf", "epub", "html", "xlsx", "ods", "odp", "csv"
]

class ConvertError(Exception):
Expand Down Expand Up @@ -43,9 +43,9 @@ class MissingPartError(ConvertError):
def format_from_bytes(data: bytes | bytearray) -> Format | None:
"""Detect the format from the content itself: the signature and identity
each container specification designates (PDF header, RTF open group, OLE
stream names, ZIP package mimetype/content types). Plain-text formats
(CSV) carry no signature and return `None`; so does anything
unrecognized."""
stream names, ZIP package mimetype/content types, or an HTML doctype/root
element). Plain-text formats (CSV) carry no signature and return `None`; so
does anything unrecognized."""

def format_from_extension(extension: str) -> Format | None:
"""The format an extension names, with or without a leading dot."""
Expand Down
2 changes: 1 addition & 1 deletion python/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ build-backend = "maturin"
# PyPI is held by an unrelated package) and imports as `anydoc`.
[project]
name = "firecrawl-anydoc"
description = "Convert documents (doc, docx, odt, rtf, epub, pdf, presentations, spreadsheets, csv) to GitHub-Flavored Markdown"
description = "Convert documents (doc, docx, odt, rtf, epub, html, pdf, presentations, spreadsheets, csv) to GitHub-Flavored Markdown"
readme = "README.md"
license = "MIT"
requires-python = ">=3.10"
Expand Down
8 changes: 5 additions & 3 deletions python/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ create_exception!(
/// Format names, as the extension that identifies each format. Container
/// variants that share a parser (`.docm`, `.xlsm`, `.ppsx`, ...) map onto
/// these via `format_from_bytes` or `format_from_extension`.
const FORMATS: [(&str, anydoc::Format); 12] = [
const FORMATS: [(&str, anydoc::Format); 13] = [
("doc", anydoc::Format::Doc),
("docx", anydoc::Format::Docx),
("odt", anydoc::Format::Odt),
Expand All @@ -68,6 +68,7 @@ const FORMATS: [(&str, anydoc::Format); 12] = [
("pptx", anydoc::Format::Pptx),
("rtf", anydoc::Format::Rtf),
("epub", anydoc::Format::Epub),
("html", anydoc::Format::Html),
("xlsx", anydoc::Format::Excel),
("ods", anydoc::Format::Ods),
("odp", anydoc::Format::Odp),
Expand Down Expand Up @@ -127,8 +128,9 @@ fn convert_error(py: Python<'_>, error: anydoc::ConvertError) -> PyErr {

/// Detect the format from the content itself: the signature and identity each
/// container specification designates (PDF header, RTF open group, OLE stream
/// names, ZIP package mimetype/content types). Plain-text formats (CSV) carry
/// no signature and return `None`; so does anything unrecognized.
/// names, ZIP package mimetype/content types, or an HTML doctype/root element).
/// Plain-text formats (CSV) carry no signature and return `None`; so does
/// anything unrecognized.
#[pyfunction]
fn format_from_bytes(data: Vec<u8>) -> Option<&'static str> {
anydoc::Format::from_bytes(&data).map(format_name)
Expand Down
Loading