A rigorous Amazon product scraper for Python — with honest block detection, drift instrumentation, and a documented compliance posture.
from amznscrape import scrape
product = await scrape("B08N5WRWNW", marketplace="us")
print(product.title.value, product.price.value)Most of the ones on GitHub are dead, are vendor funnels for a paid API, or are single-file wrappers that break quietly. This one is built around six things that a survey of the field found nobody doing — each verified against live pages, not inferred from tutorials.
Amazon runs three structurally different anti-bot stacks at once, and all three answer with a 2xx status code:
| Stack | Status | How you identify it |
|---|---|---|
| AWS WAF challenge | 202 | x-amzn-waf-action response header, empty body |
| Akamai Bot Manager | 200 | bm-verify / /_sec/verify in body |
| Amazon captcha page | 200 | /errors/validateCaptcha form |
Which stack answers is not stable per domain — the same storefront blocked and then served results within twenty minutes.
Every surveyed scraper branches on resp.status_code and will record a
challenge page as product data. amznscrape classifies the response by
signature and header, then asserts positively that the expected content marker
is present. Absence of a block is not evidence of success.
It also knows that a 404 is a real 404 — a catalogue fact, not an IP problem. Retrying a nonexistent ASIN against fresh proxies just burns reputation.
Fetching amazon.co.jp from a US IP returns prices in USD, not yen. Amazon
localises currency to the detected shopper locale, independently of the TLD.
Every scraper we surveyed maps TLD → currency. That silently and irreversibly
corrupts multi-marketplace price data. Here, Price carries the currency as
parsed from the page, and every record gets a consistency assertion against what
the marketplace would normally serve.
There is no JSON-LD on Amazon product pages — the count is zero, despite what most tutorials say. What does exist, and what this library reads:
<script type="a-state">state-hydration blobsdimensionValuesDisplayData— the authoritative ASIN → variant mapcolorImages— the full-resolution image manifest- the all-offers AJAX endpoint — 141 KB and stable, versus a 2.3 MB product page, and the only reliable source of seller identity and buy-box ownership
Everyone else regexes .a-price-whole — which contains a nested
.a-price-decimal, so a naive .text yields "150." instead of "150". The
correct node is .a-offscreen.
Amazon's own A/B bucket assignments are sitting in the HTML (a-wlab-states).
amznscrape logs them with every parse, so "the price selector stopped working
for 30% of pages" becomes an answerable question.
Every field also records how it was extracted — embedded JSON, semantic selector, presentational selector, or regex. When a field's provenance shifts downward, the page structure changed while the output still looks fine. That is the early warning, and a silent 30% null rate is the real failure mode — not exceptions.
Amazon even tells you its own verdict on your request: the
social-proofing-page-state blob contains "isRobot": false. That flips
before blocking starts. We read and record it.
Individual review history moved behind a login wall in November 2024;
/product-reviews/ now redirects anonymous clients to sign-in. Popular scrapers
that predate this are silently broken.
This library scopes to public review aggregates — rating, count, and the star histogram. Scraping the rest would require authenticated sessions, which is precisely the fact pattern that cost hiQ Labs $500,000.
Selector chains live in YAML, not in Python. When Amazon changes a container, you edit a config file — you do not wait for us to cut a release.
Position-based selectors are rejected by design. An absolute XPath silently points at the wrong element when a sibling is inserted above the target — no exception, no warning, wrong price. Anchors must be semantic.
Please read this before using the library. It is short and it is not boilerplate — some of it is a design constraint, not a disclaimer.
amznscrape does not and will not include:
- CAPTCHA solving of any kind, or an enabled-by-default integration with a solving service
- Proof-of-work computation for JavaScript challenges
- Any technique for detecting and evading rate limits
Pull requests adding these will be declined.
This is a deliberate line. Litigation filed in October 2025 pleads DMCA §1201 anti-circumvention over "bypassing rate limits and anti-bot systems" — deliberately sidestepping whether the data was public and reframing the question as whether a technological measure was defeated.
What the library does instead: it identifies a challenge, backs off, rotates, and reports. Driving a real browser that executes the page's own JavaScript is ordinary browsing. Reimplementing a challenge algorithm is not, and we don't.
- Rate limiting is on by default (~1 request/second per IP, jittered). This is a good-citizenship choice, not a technical necessity.
robots.txtis respected by default, fetched per marketplace at runtime. The files genuinely differ between storefronts, so hard-coding one set of rules would be wrong. Opting out requires an explicit flag.- No personal data in default output. Reviewer names and profile URLs are excluded. Under GDPR these are personal data, and the EDPB applies a strict necessity test to scraping.
Amazon's Conditions of Use are specific rather than generic — they name exactly what a product scraper collects:
This license does not include […] any collection and use of any product listings, descriptions, or prices; […] or any use of data mining, robots, or similar data gathering and extraction tools.
There is also an AI-training exclusion added in May 2025.
Whether such terms bind a client that never logs in is genuinely contested. In Meta v. Bright Data (2024) a court held that Meta's terms governed logged-in use and that logged-off scraping of public data fell outside them. In hiQ v. LinkedIn the scraper won the CFAA question and still lost the case on contract grounds — with a $500,000 judgment and an order to destroy all derived data. Don't overstate the protection in either direction.
In the EU, additionally consider the sui generis database right (extracting a substantial part of a protected database infringes independently of copyright), and note that CJEU C-30/14 held that where a database is protected by neither copyright nor the database right, the Directive does not apply — so nothing limits the owner's contractual restrictions.
If an API meets your need, prefer it:
- Creators API — the replacement for Product Advertising API 5.0, which is
deprecated (legacy calls now return
403 AccessDeniedException). Most guides still point at the dead one. - SP-API — for data about your own seller account.
amznscrape is in no way affiliated with, authorised, maintained, sponsored or
endorsed by Amazon. This is an independent, unofficial project. Use at your own
risk.
None of the above is legal advice. You are responsible for your own use, including compliance with applicable law and with the terms of any site you access.
pip install amznscrape
# optional browser tier (Apache-2.0; see "Licensing of dependencies" below)
pip install "amznscrape[browser]"Requires Python 3.11 or newer.
# one product
amznscrape product B08N5WRWNW --marketplace us
# many, from a file of ASINs or URLs, resumable
amznscrape batch asins.txt --marketplace de --out products.jsonl
# search results
amznscrape search "mechanical keyboard" --marketplace uk --pages 3
# check what robots.txt allows for a marketplace, without scraping anything
amznscrape robots --marketplace jpimport asyncio
from amznscrape import Scraper
async def main():
async with Scraper(marketplace="de", rate_limit=1.0) as scraper:
product = await scraper.product("B08N5WRWNW")
print(product.title.value)
print(product.price.value.amount, product.price.value.currency)
# how each field was obtained -- the drift signal
print(product.provenance_map())
# Amazon's own verdict on the request, before blocking starts
print(product.diagnostics.is_robot)
asyncio.run(main())Batch scraping returns ScrapeOutcome objects rather than raising, so one
blocked page does not abort a run of ten thousand, and the failure reason is
recorded alongside the successes.
Title · brand · price · list price · currency · rating · rating count · star histogram · availability · seller name and ID · full-resolution image set · variant ASINs with their dimensions · Best Sellers Rank (all three container layouts) · feature bullets · specifications · categories · buy-box and all-offers data.
Plus per-page diagnostics: weblab A/B buckets, Amazon's isRobot verdict,
detected delivery location, and which *_feature_div containers were present.
24 marketplaces. The primary key is (asin, marketplace), never a bare
ASIN — the same ASIN resolves to different products, or to nothing, across
storefronts.
Data is fetched by the cheapest tier that can serve it, and the tier is chosen by what renders the field, not by retrying after a failure:
- Internal JSON endpoints — measured at the same latency as rendering the full page, while returning complete structured data
- HTTP + embedded structured data — for server-rendered content, an HTTP response is byte-for-byte identical to what a browser produces
- HTTP + selector chains
- Browser — only for fields known to be client-injected
Failure-based downgrade does not work here, and this is worth understanding: an HTTP client fetching a JS-rendered page gets HTTP 200 and a normal-looking shell, not an error. So the escalation trigger is a field-level content assertion, never a status code.
The distribution of which tier served each request is itself the health metric.
git clone https://github.com/thunderbit-operations/amazon-product-scraper-python
cd amazon-product-scraper-python
pip install -e ".[dev]"
pytestThe test suite runs entirely offline. Parser tests run against frozen real
HTML plus a local fixture server that reproduces the awkward parts on purpose —
the nested-decimal price trap, JS object literals that json.loads rejects, all
three BSR container variants, the currency/locale mismatch, and all three block
page types. Contributors need no proxy to work on the parser.
Tests that touch the real site are marked live and excluded by default:
pytest -m live # opt in explicitlydocs/ARCHITECTURE.md— design and rationaledocs/AMAZON-RECON.md— measured site intelligence, with the evidence grade marked on every claimdocs/EVIDENCE.md— why the tooling choices are what they are, traced to reproducible measurements
Where a design decision came from a measurement, the docs say so and cite it. Where there is no evidence, they say that too — the anti-bot layer in particular is marked as unverified rather than dressed up.
This project is Apache-2.0. One dependency choice is worth flagging: the optional
browser tier uses Patchright (Apache-2.0). Two popular alternatives,
nodriver and zendriver, are AGPL-3.0 — whose network-use clause would
propagate to anyone hosting this scraper as a service. They are deliberately not
offered.
Bug reports with a failing fixture are the most useful contribution: if a selector broke, the frozen HTML that breaks it is worth more than a description.
See the circumvention policy above for the one category of contribution that will be declined.
Built and maintained by the team behind Thunderbit, an
AI web scraper. The site intelligence in docs/AMAZON-RECON.md
— the three concurrent anti-bot stacks that all return 2xx, the selector grading,
the marketplace-scoped ASIN behaviour — came out of that work, which is why it is
dated and graded rather than asserted.
The library is standalone. Nothing in it calls a Thunderbit service, and there is no account, API key or quota anywhere in the codebase — you can read every request it makes. If you would rather not maintain selectors as Amazon changes them, Thunderbit is the hosted answer to the same problem; if you would, that is exactly what this repo is for.
Apache-2.0. See LICENSE.