Suggestions: TLS fingerprint spoofing, HTTP/2/3 support, and account unlocking

[gabearro]

Hi @vladkens first off, thanks for maintaining this library. It’s a super useful tool and I think it could be made even more resilient against Twitter’s anti-bot protections with a few enhancements

Suggested Improvements

1. TLS Fingerprint Spoofing via httpx-curl-cffi

   • Twitter (relies on Cloudflare) relies heavily on TLS fingerprint checks (JA3, SNI patterns, etc.) for bot detection
   • httpx-curl-cffi (https://github.com/vgavro/httpx-curl-cffi) provides a drop-in transport for httpx with cURL-level TLS fingerprint spoofing
   • Replacing the default httpx transport with this would make requests look much closer to a real browser session

2. HTTP/2 and HTTP/3 Support

   • Twitter endpoints seem to default to HTTP/2 (and in some cases HTTP/3)
   • Currently twscrape requests appear to be forced through HTTP/1.1, which is a detectable
   • Upgrading to use h2/h3 via the transport layer (e.g. with httpx-curl-cffi or hyper/h3 libraries) would improve stealth and performance

3. Account Unlocking Flow

   • Many accounts fail to provide a ct0 cookie simply because they’re in a "locked" state
   • With Cloudflare checks bypassed via httpx-curl-cffi, it should be possible to implement the same flow browsers do to “unlock” accounts (completing the initial challenge)

[TheCrazyLex]

I think these measures would help greatly, really cool. @vladkens

[BonifacioCalindoro]

Indeed really cool features!

I wish this was discussed through an IRC instead of public issues so the twitter devs don't know our next move 😈

[hponka]

@vladkens See this if it helps: zedeus/nitter@662ae90

[vladkens]

Optional curl-cffi support is now available in #308 — happy to get feedback from anyone who wants to try it.

pip install twscrape[curl]

When curl-cffi is installed it is picked up automatically (browser-level TLS fingerprinting, helps with Cloudflare). You can also force a specific backend via TWS_HTTP_BACKEND=curl or TWS_HTTP_BACKEND=httpx.

One breaking change: raw API methods (e.g. search_raw) now return twscrape.Response instead of httpx.Response. The interface is the same (.status_code, .json(), .text, .headers), but code that does isinstance(rep, httpx.Response) or imports httpx types from the response will break.

[gabearro]

Goat @vladkens I just accidentally closed this when I meant to comment :D but good stuff. Might be good to reopen if it makes sense, if you plan on addressing the other parts of the OG post