Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion docs/biome.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,15 @@
},
"files": {
"ignoreUnknown": true,
"includes": ["**", "!node_modules", "!.next", "!dist", "!build", "!.source"]
"includes": [
"**",
"!node_modules",
"!.next",
"!dist",
"!build",
"!.source",
"!src/app/global.css"
]
},
"formatter": {
"enabled": true,
Expand Down
77 changes: 77 additions & 0 deletions docs/content/docs/api-reference/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
---
title: API Reference
description: Every public class, method, and CLI command in taskito.
---

import { Cards, Card } from "fumadocs-ui/components/card";
import {
Compass,
Inbox,
ListTodo,
CheckCircle2,
Variable,
GitFork,
Workflow,
TestTube,
Terminal,
} from "lucide-react";

Reference docs for everything taskito exposes. Each page lists the full
signature, parameters, return values, and a short example.

<Cards>
<Card
icon={<Compass />}
title="Overview"
href="/docs/api-reference/overview"
description="The shape of the public API — what's stable, what's experimental."
/>
<Card
icon={<Inbox />}
title="Queue"
href="/docs/api-reference/queue"
description="Construct queues, enqueue jobs, manage workers, inspect state."
/>
<Card
icon={<ListTodo />}
title="Task"
href="/docs/api-reference/task"
description="The @queue.task decorator — options, methods, async variants."
/>
<Card
icon={<CheckCircle2 />}
title="Result"
href="/docs/api-reference/result"
description="JobResult, polling, awaiting, timeouts, sync and async APIs."
/>
<Card
icon={<Variable />}
title="Context"
href="/docs/api-reference/context"
description="current_job — progress, logging, cancellation, timeouts inside a task."
/>
<Card
icon={<GitFork />}
title="Canvas"
href="/docs/api-reference/canvas"
description="chain, group, chord — the Celery-compatible composition primitives."
/>
<Card
icon={<Workflow />}
title="Workflows"
href="/docs/api-reference/workflows"
description="The DAG builder — fan-out, fan-in, gates, conditions, sub-workflows."
/>
<Card
icon={<TestTube />}
title="Testing"
href="/docs/api-reference/testing"
description="test_mode, MockResource, fixtures for synchronous in-process verification."
/>
<Card
icon={<Terminal />}
title="CLI"
href="/docs/api-reference/cli"
description="taskito worker, dashboard, info — every flag and subcommand."
/>
</Cards>
70 changes: 70 additions & 0 deletions docs/content/docs/architecture/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
---
title: Architecture
description: How taskito works under the hood — Rust core, scheduler, storage, worker pool.
---

import { Cards, Card } from "fumadocs-ui/components/card";
import {
Compass,
GitBranch,
Layers,
CalendarClock,
Database,
Boxes,
ShieldAlert,
FileCode2,
} from "lucide-react";

The internals, explained from the outside in. Start with **Overview** for the
high-level model, then drill into the layer that interests you.

<Cards>
<Card
icon={<Compass />}
title="Overview"
href="/docs/architecture/overview"
description="The big picture — Python ↔ PyO3 ↔ Rust core, end to end."
/>
<Card
icon={<GitBranch />}
title="Job lifecycle"
href="/docs/architecture/job-lifecycle"
description="Every state a job moves through, from enqueue to result or dead letter."
/>
<Card
icon={<Layers />}
title="Worker pool"
href="/docs/architecture/worker-pool"
description="OS-thread pool, prefork children, async executor — how each model dispatches work."
/>
<Card
icon={<CalendarClock />}
title="Scheduler"
href="/docs/architecture/scheduler"
description="The Tokio polling loop — how jobs are picked, claimed, and routed to workers."
/>
<Card
icon={<Database />}
title="Storage"
href="/docs/architecture/storage"
description="The Storage trait, Diesel for SQLite/Postgres, Redis backend, table schemas."
/>
<Card
icon={<Boxes />}
title="Resources"
href="/docs/architecture/resources"
description="The 3-layer DI pipeline — argument interception, worker injection, proxy reconstruction."
/>
<Card
icon={<ShieldAlert />}
title="Failure model"
href="/docs/architecture/failure-model"
description="Retries, dead letters, circuit breakers, cancellation — what happens when things go wrong."
/>
<Card
icon={<FileCode2 />}
title="Serialization"
href="/docs/architecture/serialization"
description="Cloudpickle, JSON, MsgPack — how arguments and results cross the worker boundary."
/>
</Cards>
48 changes: 47 additions & 1 deletion docs/content/docs/architecture/scheduler.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: "The Tokio-based poll loop that dequeues, dispatches, retries, and

The scheduler runs in a dedicated Tokio single-threaded async runtime:

```
```text
loop {
sleep(50ms) or shutdown signal

Expand All @@ -23,10 +23,56 @@ loop {
}
```

> **Why those numbers?** 50 ms is fast enough that dequeue latency is imperceptible
> but slow enough to leave the CPU 99% idle when the queue is empty. The periodic
> intervals (~60, ~100, ~1200) are deliberately coprime so the four maintenance
> tasks rarely fire on the same tick — keeping each individual iteration cheap.

## Polling cycle

<Mermaid
chart={`flowchart LR
Tick(["loop tick"]) --> Sleep["sleep 50ms"]
Sleep --> Dispatch["try_dispatch()"]
Dispatch --> Periodic{"iteration<br/>multiple?"}
Periodic -->|"every ~60"| CP["check_periodic()"]
Periodic -->|"every ~100"| RS["reap_stale()"]
Periodic -->|"every ~1200"| AC["auto_cleanup()"]
Periodic -->|"otherwise"| Next["next tick"]
CP --> Next
RS --> Next
AC --> Next
Next --> Tick`}
/>

## Dispatch flow

1. `dequeue_from()` — atomically `SELECT` + `UPDATE` (pending → running) within a transaction.
2. Check rate limit — if over limit, reschedule 1s in the future.
3. Send job to worker pool via `tokio::sync::mpsc` channel.
4. Worker executes task, sends result back.
5. `handle_result()` — mark complete, schedule retry, or move to DLQ.

<Mermaid
chart={`sequenceDiagram
autonumber
participant S as Scheduler
participant DB as Storage
participant RL as RateLimiter
participant WP as WorkerPool
participant W as Worker

S->>DB: dequeue_from()
DB-->>S: pending job
S->>RL: check(task)
alt over limit
RL-->>S: limited
S->>DB: reschedule +1s
else under limit
RL-->>S: ok
S->>WP: mpsc.send(job)
WP->>W: execute()
W-->>S: result
S->>DB: handle_result()
end`}
/>
Loading
Loading