infinity-fetch 🚀

Configurable recursive fetch for paginated APIs. Works in Node.js and browsers.

Automatically re-invokes a fetch function across pages until a stop condition is met — accumulating all results into a single array. Zero dependencies.

---

How it works

                    ┌─────────────────────────────────────────────────────┐
                    │                   infinityFetch                     │
                    └────────────────────────┬────────────────────────────┘
                                             │
                         initialParams = { start: 0, limit: 100 }
                                             │
                                             ▼
                    ┌────────────────────────────────────────┐
                    │          fetcher(params)               │
                    │   api.project('x').repo('y').commits() │
                    └────────────────┬───────────────────────┘
                                     │
                    ┌────────────────▼───────────────────────┐
                    │             Response                    │
                    │  { values, isLastPage, nextPageStart,  │
                    │    size, limit, start }                 │
                    └──────┬─────────────────────┬───────────┘
                           │                     │
                    isLastPage?                  no
                           │                     │
                          yes            getNextParams()
                           │          { start: nextPageStart }
                           │                     │
                           │                     ▼
                           │        ┌────────────────────────┐
                           │        │   fetcher(nextParams)  │  ← repeats
                           │        └────────────────────────┘
                           │
                           ▼
              ┌──────────────────────────────────────┐
              │   return { items[], pages: number }  │
              │   items  → all pages accumulated     │
              │   pages  → total iterations done     │
              └──────────────────────────────────────┘

---

Installation

npm install infinity-fetch

---

Usage

pagedFetch — for Bitbucket-style paginated APIs

If your API returns { values, isLastPage, nextPageStart, size, limit, start }, use the built-in helper:

import { pagedFetch } from 'infinity-fetch';

const { items, pages } = await pagedFetch({
  fetcher: (params) => api.project('my-project').repo('my-repo').commits(params),
  limit: 100, // items per page, defaults to 100
});

console.log(`${items.length} commits fetched across ${pages} pages`);

With loading state, progress tracking, and a delay between pages:

const { items, pages } = await pagedFetch({
  fetcher: (params) => api.project('my-project').repo('my-repo').commits(params),
  limit: 100,
  maxPages: 20,
  delay: 200, // wait 200ms between each page fetch
  onStart: () => setLoading(true),
  onEnd:   () => setLoading(false),
  onPage: (pageItems, _response, pageIndex) => {
    console.log(`Page ${pageIndex + 1}: ${pageItems.length} commits`);
  },
});

---

infinityFetch — generic, fully configurable

Use this when your API has a different response shape:

import { infinityFetch } from 'infinity-fetch';

const { items, pages } = await infinityFetch({
  // The function that performs a single page request
  fetcher: (params) => github.issues.list(params),

  // Parameters for the very first request
  initialParams: { page: 1, per_page: 50 },

  // When to stop — return true on the last page
  isLastPage: (response) => response.data.length < 50,

  // How to compute params for the next page
  getNextParams: (_response, currentParams) => ({
    ...currentParams,
    page: currentParams.page + 1,
  }),

  // Which field holds the items
  getItems: (response) => response.data,

  // Optional: safety cap on number of pages
  maxPages: 100,

  // Optional: milliseconds to wait between each page fetch
  delay: 200,

  // Optional: retry failed page fetches
  retry: {
    maxRetries: 3,
    delay: (attempt) => attempt * 500,
    retryWhen: (error) => {
      return error instanceof Response && error.status >= 500;
    },
  },

  // Optional: called once before the first fetch
  onStart: () => setLoading(true),

  // Optional: called once after all pages are done — receives the final result
  onEnd: ({ items, pages }) => {
    setLoading(false);
    console.log(`Done: ${items.length} items across ${pages} pages`);
  },

  // Optional: called after each individual page
  onPage: (pageItems, _response, pageIndex) => {
    console.log(`Page ${pageIndex + 1}: ${pageItems.length} items`);
  },
});

console.log(`${items.length} issues fetched across ${pages} pages`);

---

API Reference

pagedFetch<TItem>(config)

Option	Type	Default	Description
fetcher	(params: PagedParams) => Promise<PagedResponse<TItem>>	required	Function that fetches one page
limit	number	100	Items per page
maxPages	number	Infinity	Maximum pages to fetch (safety limit)
delay	number	—	Milliseconds to wait between each page fetch
retry	InfinityFetchRetryConfig	—	Retry failed page fetches
onStart	() => void	—	Called once before the first fetch
onEnd	(result: InfinityFetchResult<TItem>) => void	—	Called once after all pages are done
onPage	(items, response, pageIndex) => void	—	Called after each individual page

PagedParams

{ start: number; limit: number }

PagedResponse<TItem> — expected response shape:

{
  values: TItem[];
  isLastPage: boolean;
  nextPageStart?: number;
  size: number;
  limit: number;
  start: number;
}

Returns: Promise<InfinityFetchResult<TItem>>

---

infinityFetch<TResponse, TParams, TItem>(config)

Option	Type	Default	Description
fetcher	(params: TParams) => Promise<TResponse>	required	Function that fetches one page
initialParams	TParams	required	Parameters for the first request
isLastPage	(response: TResponse) => boolean	required	Returns true to stop iteration
getNextParams	(response: TResponse, currentParams: TParams) => TParams	required	Computes params for the next page
getItems	(response: TResponse) => TItem[]	required	Extracts items from a response
maxPages	number	Infinity	Maximum pages to fetch (safety limit)
delay	number	—	Milliseconds to wait between each page fetch
retry	InfinityFetchRetryConfig	—	Retry failed page fetches
onStart	() => void	—	Called once before the first fetch
onEnd	(result: InfinityFetchResult<TItem>) => void	—	Called once after all pages are done
onPage	(items, response, pageIndex) => void	—	Called after each individual page

Returns: Promise<InfinityFetchResult<TItem>>

type InfinityFetchResult<TItem> = {
  items: TItem[];  // all items collected across every page
  pages: number;   // total number of pages fetched
};

type InfinityFetchRetryConfig = {
  maxRetries?: number;
  delay?: number | ((attempt: number, error: unknown) => number);
  retryWhen?: (error: unknown, attempt: number) => boolean | Promise<boolean>;
};

---

Compatibility

Environment	Support
Node.js 18+	✅
Node.js 20+	✅
Modern browsers	✅
Deno / Bun	✅
ESM	✅
TypeScript	✅ (types included)

---

Contributing

Commits must follow the Conventional Commits spec — this drives automatic versioning and changelog generation via semantic-release.

Commit prefix	Triggers
fix:	Patch release (0.0.x)
feat:	Minor release (0.x.0)
feat!: / BREAKING CHANGE:	Major release (x.0.0)
chore:, docs:, test:	No release

git commit -m "feat: add onPage callback to pagedFetch"
git commit -m "fix: handle missing nextPageStart gracefully"
git commit -m "feat!: rename items field to data"

Setting up repository secrets

For the release pipeline to work, add these secrets in Settings → Secrets and variables → Actions:

Secret	Description
NPM_TOKEN	npm automation token (npm token create --type automation)

GITHUB_TOKEN is provided automatically by GitHub Actions — no setup needed.

Enabling GitHub Pages

Go to Settings → Pages and set the source to GitHub Actions.

---

Changelog

See CHANGELOG.md for the full release history.

---

License

MIT