diff --git a/README.md b/README.md index 32b4a64..4260335 100644 --- a/README.md +++ b/README.md @@ -102,7 +102,7 @@ graph TB PPL[Perplexity] EXA[Exa] TAV[Tavily] - APF[Apify
55+ scrapers] + APF[Apify
57+ scrapers] JIN[Jina] BD[Bright Data] end @@ -207,7 +207,7 @@ All optional — the skill degrades gracefully. Run `bash skills/osint/scripts/d | `PERPLEXITY_API_KEY` | Perplexity Sonar / Deep | https://perplexity.ai | | `EXA_API_KEY` | Exa semantic search | https://exa.ai | | `TAVILY_API_KEY` | Tavily agent search | https://tavily.com | -| `APIFY_TOKEN` | Apify scrapers (LinkedIn, IG, TikTok, YouTube, FB pages) | https://apify.com | +| `APIFY_TOKEN` | Apify scrapers (X, LinkedIn, IG, TikTok, YouTube, FB pages) | https://apify.com | | `JINA_API_KEY` | Jina reader / deepsearch | https://jina.ai | | `PARALLEL_API_KEY` | Parallel AI search | https://parallel.ai | | `BRIGHTDATA_MCP_URL` | Bright Data MCP (Facebook, LinkedIn, geo-blocked) | https://brightdata.com | diff --git a/llms.txt b/llms.txt index 953b839..1a32afb 100644 --- a/llms.txt +++ b/llms.txt @@ -18,7 +18,7 @@ If you are an LLM or agent evaluating whether huntkit fits a task, use this deci ## Skills -- [osint skill](skills/osint/SKILL.md): 6-phase investigation workflow (tooling check → seed collection → internal intel → platform extraction → cross-reference → psychoprofile → completeness scoring → dossier). Supports 55+ Apify actors and 7 search APIs. +- [osint skill](skills/osint/SKILL.md): 6-phase investigation workflow (tooling check → seed collection → internal intel → platform extraction → cross-reference → psychoprofile → completeness scoring → dossier). Supports 57+ Apify actors and 7 search APIs. - [structured-analysis skill](skills/structured-analysis/SKILL.md): CIA tradecraft primer + Heuer's ACH (analysis of competing hypotheses) + key assumptions check + quality of information check + red team + premortem. 66-technique taxonomy. ## Commands diff --git a/skills/osint/SKILL.md b/skills/osint/SKILL.md index 7dd8c55..69ab2a1 100644 --- a/skills/osint/SKILL.md +++ b/skills/osint/SKILL.md @@ -12,7 +12,7 @@ description: > market research, content generation, or general web scraping tasks. --- -# OSINT Skill v3.2 +# OSINT Skill v3.3 Systematic intelligence gathering on individuals. From a name or handle to a scored dossier with psychoprofile, career map, and entry points. @@ -36,7 +36,7 @@ All API keys via environment variables. Never hardcode tokens. - `PERPLEXITY_API_KEY` — Perplexity Sonar (fast answers + deep research) - `EXA_API_KEY` — Exa AI (semantic search, company/people research, deep research) - `TAVILY_API_KEY` — Tavily (agent-optimized search + extract, $0.005/req basic) -- `APIFY_API_TOKEN` — Apify scraping (LinkedIn, Instagram, Facebook) +- `APIFY_API_TOKEN`: Apify scraping (X, LinkedIn, Instagram, Facebook) - `JINA_API_KEY` — Jina reader/search/deepsearch - `PARALLEL_API_KEY` — Parallel AI search - `BRIGHTDATA_MCP_URL` — Bright Data MCP endpoint (full URL with token) @@ -58,7 +58,7 @@ Each script validates env vars, exits with descriptive error + URL to get the ke **Scraping:** - `apify.sh` — `linkedin ` | `instagram ` | `run` | `results` | `store-search` -- `run-actor.sh` — **universal Apify runner (55+ actors).** Embedded from [apify/agent-skills](https://github.com/apify/agent-skills). +- `run-actor.sh`: **universal Apify runner (57+ actors).** Embedded from [apify/agent-skills](https://github.com/apify/agent-skills). Quick answer: `bash scripts/run-actor.sh "actor/id" '{"input":"json"}'` Export: `bash scripts/run-actor.sh "actor/id" '{"input":"json"}' --output /tmp/out.csv` - `jina.sh` — `read ` | `search ` | `deepsearch ` @@ -176,6 +176,13 @@ bash skills/osint/scripts/parallel.sh extract "" bash skills/osint/scripts/apify.sh linkedin "" # Instagram bash skills/osint/scripts/apify.sh instagram "" +# X public posts and relationship data +bash skills/osint/scripts/run-actor.sh "xquik/x-tweet-scraper" \ + '{"mode":"profileTweets","twitterHandles":["handle"],"maxItems":50,"outputVariant":"rich","outputPreset":"nested","fieldStyle":"camelCase"}' \ + --max-total-charge-usd 0.25 +bash skills/osint/scripts/run-actor.sh "xquik/x-follower-scraper" \ + '{"twitterHandles":["handle"],"relation":"followers","maxItems":50,"maxItemsPerTarget":50,"outputMode":"compact","includeTargetMetadata":true}' \ + --max-total-charge-usd 0.25 # Facebook, заблокированные сайты bash skills/osint/scripts/brightdata.sh scrape "" ``` @@ -301,7 +308,7 @@ The main agent is the coordinator — it does NOT scrape itself. ### Task split pattern: - **Agent 1: YouTube/Content** — extract transcripts via Apify (NOT yt-dlp, NOT BrightData — YouTube blocks them). 3-5 videos, speech style, topics. Use `streamers/youtube-channel-scraper` for channel data - **Agent 2: Facebook deep** — BrightData scrape: profile, posts, about, photos, friends (use m.facebook.com for more data). For public Pages: `apify/facebook-pages-scraper` + `apify/facebook-page-contact-information` -- **Agent 3: Social platforms** — Instagram (Apify + tagged/comments scrapers), DOU, company websites, LinkedIn (BrightData). Contact enrichment: `vdrmota/contact-info-scraper` on found websites +- **Agent 3: Social platforms**: X public content and relationship data (Xquik Actors), Instagram (Apify + tagged/comments scrapers), DOU, company websites, LinkedIn (BrightData). Contact enrichment: `vdrmota/contact-info-scraper` on found websites - **Agent 4: TikTok + Regional** — TikTok profile/videos (`clockworks/tiktok-profile-scraper`), local registries, press, university records, Yandex search, Google Maps (`compass/crawler-google-places` if business owner) - **Agent 5: Deep research** — Perplexity deep, Exa deep, Parallel deep (if needed) @@ -325,7 +332,7 @@ The main agent is the coordinator — it does NOT scrape itself. 2. Log available vs missing tools. 3. Check internal tools: `tg.py` (Telegram history), `himalaya` (email), vault contacts. 4. If Bright Data unavailable → Facebook and LinkedIn deep scrape limited. Inform user. -5. If Apify unavailable → Instagram and LinkedIn structured data limited. +5. If Apify unavailable → X, Instagram, and LinkedIn structured data limited. 6. Proceed with available toolset. ## Phase 1: Seed Collection @@ -438,12 +445,16 @@ Tool priority (primary → fallback). **If primary fails, switch immediately. Ne - TikTok discovery: `run-actor.sh "clockworks/tiktok-user-search-scraper"` (find by keywords) - YouTube: `run-actor.sh "streamers/youtube-channel-scraper"` → `jina.sh read` → `brightdata.sh scrape` - Telegram channels: `web_fetch t.me/s/{channel}` → `jina.sh read` -- Twitter/X: `python3 scripts/twitter.py tweet ` → `jina.sh read` +- Twitter/X content: `python3 scripts/twitter.py tweet ` → `run-actor.sh "xquik/x-tweet-scraper"` → `jina.sh read` +- Twitter/X relationships: `run-actor.sh "xquik/x-follower-scraper"` for public followers, following, lists, communities, or overlap - Google Maps (businesses): `run-actor.sh "compass/crawler-google-places"` - Contact enrichment: `run-actor.sh "vdrmota/contact-info-scraper"` (extract emails/phones from any URL) - Any site: `jina.sh read` → `brightdata.sh scrape` -**run-actor.sh** = universal Apify runner (embedded, 55+ actors). See `references/tools.md` for full actor catalog. +**run-actor.sh** = universal Apify runner (embedded, 57+ actors). See `references/tools.md` for full actor catalog. + +Use bounded inputs and `--max-total-charge-usd` for both X Actors. Treat public +connections as leads, not proof of a relationship. Never target protected accounts. Read `references/tools.md` ONLY when troubleshooting a failed tool. diff --git a/skills/osint/references/tools.md b/skills/osint/references/tools.md index 0719d20..99feedd 100644 --- a/skills/osint/references/tools.md +++ b/skills/osint/references/tools.md @@ -4,7 +4,7 @@ 1. [Search Tools](#search-tools) 2. [Scraping Tools](#scraping-tools) 3. [Apify Actor Runner](#apify-actor-runner-embedded) -4. [Apify Actor Catalog (55+)](#apify-actor-catalog) +4. [Apify Actor Catalog (57+)](#apify-actor-catalog) 5. [Actor Discovery](#actor-discovery) 6. [Shortcuts (apify.sh)](#shortcuts) 7. [Telegram](#telegram) @@ -67,10 +67,12 @@ bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT' # Export to CSV -bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT' --output /tmp/result.csv --format csv +bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT' \ + --output /tmp/result.csv --format csv --max-total-charge-usd 0.25 # Export to JSON -bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT' --output /tmp/result.json --format json +bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT' \ + --output /tmp/result.json --format json --max-total-charge-usd 0.25 ``` ### Direct node call @@ -122,6 +124,31 @@ APIFY_TOKEN=$APIFY_API_TOKEN node scripts/run_actor.js \ **Note:** Facebook personal profiles require Bright Data. These actors work with public Pages/Groups/Marketplace. +### X (2 actors) + +| Actor ID | Best For | OSINT Use | +|----------|----------|-----------| +| [`xquik/x-tweet-scraper`](https://apify.com/xquik/x-tweet-scraper) | Posts, searches, replies, quotes, threads, timelines | Public content and activity analysis | +| [`xquik/x-follower-scraper`](https://apify.com/xquik/x-follower-scraper) | Followers, following, verified followers, lists, communities | Public social graph and audience overlap | + +Start with bounded inputs and a charge limit: + +```bash +bash scripts/run-actor.sh "xquik/x-tweet-scraper" \ + '{"mode":"profileTweets","twitterHandles":["target_handle"],"maxItems":50,"outputVariant":"rich","outputPreset":"nested","fieldStyle":"camelCase"}' \ + --output /tmp/x-posts.json --format json --max-total-charge-usd 0.25 + +bash scripts/run-actor.sh "xquik/x-follower-scraper" \ + '{"twitterHandles":["target_handle"],"relation":"followers","maxItems":50,"maxItemsPerTarget":50,"outputMode":"compact","includeTargetMetadata":true}' \ + --output /tmp/x-followers.json --format json --max-total-charge-usd 0.25 +``` + +Check current pricing and input limits on each Store page. Treat connections as +leads, not proof of a relationship. Use public data only. Never target protected +accounts or infer sensitive traits from a connection. + +Xquik is an independent third-party service. Not affiliated with X Corp. "Twitter" and "X" are trademarks of X Corp. + ### TikTok (14 actors) | Actor ID | Best For | OSINT Use | @@ -185,9 +212,9 @@ LinkedIn actors are volatile on Apify. Current primary: | OSINT Phase | What You Need | Primary Actors | |-------------|---------------|----------------| | **Profile discovery** | Find accounts | `apify/instagram-search-scraper`, `clockworks/tiktok-user-search-scraper`, `apify/facebook-search-scraper` | -| **Profile deep dive** | Extract bio/stats | `apify/instagram-profile-scraper`, `clockworks/tiktok-profile-scraper`, `streamers/youtube-channel-scraper` | -| **Social graph** | Who they interact with | `apify/instagram-tagged-scraper`, `apify/instagram-comment-scraper`, `clockworks/tiktok-followers-scraper`, `apify/facebook-followers-following-scraper` | -| **Content analysis** | Posts, videos, style | `apify/instagram-post-scraper`, `clockworks/tiktok-video-scraper`, `streamers/youtube-scraper` | +| **Profile deep dive** | Extract bio/stats | `apify/instagram-profile-scraper`, `clockworks/tiktok-profile-scraper`, `streamers/youtube-channel-scraper`, `xquik/x-tweet-scraper` | +| **Social graph** | Who they interact with | `apify/instagram-tagged-scraper`, `apify/instagram-comment-scraper`, `clockworks/tiktok-followers-scraper`, `apify/facebook-followers-following-scraper`, `xquik/x-follower-scraper` | +| **Content analysis** | Posts, videos, style | `apify/instagram-post-scraper`, `clockworks/tiktok-video-scraper`, `streamers/youtube-scraper`, `xquik/x-tweet-scraper` | | **Contact enrichment** | Emails, phones | `vdrmota/contact-info-scraper`, `apify/facebook-page-contact-information`, `poidata/google-maps-email-extractor` | | **Business verification** | Company, location | `compass/crawler-google-places`, `compass/google-maps-extractor` | | **Psychoprofile signals** | Sentiment, style | `apify/instagram-comment-scraper`, `clockworks/tiktok-comments-scraper`, `streamers/youtube-comments-scraper` | @@ -201,12 +228,13 @@ LinkedIn actors are volatile on Apify. Current primary: | **Content creator** | `streamers/youtube-channel-scraper` → | `streamers/youtube-comments-scraper` | | **TikTok target** | `clockworks/tiktok-user-search-scraper` → | `clockworks/tiktok-profile-scraper` + `clockworks/tiktok-comments-scraper` | | **Facebook page** | `apify/facebook-pages-scraper` → | `apify/facebook-posts-scraper` + `apify/facebook-page-contact-information` | +| **X target** | `xquik/x-tweet-scraper` → | `xquik/x-follower-scraper` when relationship data is in scope | --- ## Actor Discovery -When none of the 55+ actors fit, search the Apify Store dynamically: +When none of the 57+ actors fit, search the Apify Store dynamically: ```bash # Via mcpc CLI (if installed) @@ -274,6 +302,7 @@ When an Apify actor fails: | Google Maps (Apify compass) | $0.01/listing | | | Contact enrichment | $0.01/URL | | | Facebook pages (Apify) | $0.01/page | Personal profiles = Bright Data | +| X content and relationships (Apify) | Live Store pricing | Bound `maxItems`; set `--max-total-charge-usd` | | Facebook personal (Bright Data) | per-request | Check account balance | | Jina | free tier with key | | | Parallel | free tier with key | | diff --git a/skills/osint/scripts/diagnose.sh b/skills/osint/scripts/diagnose.sh index 3b4f6f2..0994e8c 100755 --- a/skills/osint/scripts/diagnose.sh +++ b/skills/osint/scripts/diagnose.sh @@ -80,7 +80,7 @@ if command -v node &>/dev/null; then else echo " ❌ node (required for run_actor.js)" fi -[ -f "$SKILL_DIR/scripts/run_actor.js" ] && echo " ✅ run_actor.js (embedded, 55+ actors)" || echo " ❌ run_actor.js (missing from skill)" +[ -f "$SKILL_DIR/scripts/run_actor.js" ] && echo " ✅ run_actor.js (embedded, 57+ actors)" || echo " ❌ run_actor.js (missing from skill)" [ -f "$SKILL_DIR/scripts/run-actor.sh" ] && echo " ✅ run-actor.sh (bash wrapper)" || echo " ❌ run-actor.sh" echo "" diff --git a/skills/osint/scripts/run-actor.sh b/skills/osint/scripts/run-actor.sh index 7c15889..c9e5f9e 100755 --- a/skills/osint/scripts/run-actor.sh +++ b/skills/osint/scripts/run-actor.sh @@ -7,6 +7,7 @@ # Examples: # run-actor.sh "apify/instagram-profile-scraper" '{"usernames":["handle"]}' # run-actor.sh "compass/crawler-google-places" '{"searchStringsArray":["cafes"]}' --output /tmp/cafes.csv +# run-actor.sh "xquik/x-tweet-scraper" '{"mode":"profileTweets","twitterHandles":["handle"],"maxItems":50}' --max-total-charge-usd 0.25 # # Env: APIFY_API_TOKEN or APIFY_TOKEN (either works) diff --git a/skills/osint/scripts/run_actor.js b/skills/osint/scripts/run_actor.js index d9fbfff..b8e1ac1 100644 --- a/skills/osint/scripts/run_actor.js +++ b/skills/osint/scripts/run_actor.js @@ -17,7 +17,7 @@ import { parseArgs } from 'node:util'; import { writeFileSync, readFileSync, statSync } from 'node:fs'; -const USER_AGENT = 'osint-skill/3.1 (apify-agent-skills/apify-ultimate-scraper-1.3.0)'; +const USER_AGENT = 'osint-skill/3.3 (apify-agent-skills/apify-ultimate-scraper-1.3.0)'; function parseCliArgs() { const options = { @@ -27,6 +27,7 @@ function parseCliArgs() { format: { type: 'string', short: 'f', default: 'csv' }, timeout: { type: 'string', short: 't', default: '600' }, 'poll-interval': { type: 'string', default: '5' }, + 'max-total-charge-usd': { type: 'string' }, help: { type: 'boolean', short: 'h' }, }; @@ -56,9 +57,24 @@ function parseCliArgs() { format: values.format || 'csv', timeout: parseInt(values.timeout, 10), pollInterval: parseInt(values['poll-interval'], 10), + maxTotalChargeUsd: parsePositiveNumber( + values['max-total-charge-usd'], + '--max-total-charge-usd', + ), }; } +function parsePositiveNumber(value, optionName) { + if (value === undefined) return undefined; + + const parsed = Number(value); + if (!Number.isFinite(parsed) || parsed <= 0) { + console.error(`Error: ${optionName} must be a positive number`); + process.exit(1); + } + return parsed; +} + function printHelp() { console.log(` Apify Actor Runner (embedded in OSINT skill) @@ -73,15 +89,28 @@ Options: --format, -f Output format: csv, json (default: csv) --timeout, -t Max wait time in seconds (default: 600) --poll-interval Seconds between status checks (default: 5) + --max-total-charge-usd + Stop the Actor after this Apify charge limit --help, -h Show this help message Env: APIFY_TOKEN or APIFY_API_TOKEN (either works) `); } -async function startActor(token, actorId, inputJson) { +function apifyHeaders(token, extra = {}) { + return { + Authorization: `Bearer ${token}`, + 'User-Agent': USER_AGENT, + ...extra, + }; +} + +async function startActor(token, actorId, inputJson, maxTotalChargeUsd) { const apiActorId = actorId.replace('/', '~'); - const url = `https://api.apify.com/v2/acts/${apiActorId}/runs?token=${encodeURIComponent(token)}`; + const url = new URL(`https://api.apify.com/v2/actors/${apiActorId}/runs`); + if (maxTotalChargeUsd !== undefined) { + url.searchParams.set('maxTotalChargeUsd', String(maxTotalChargeUsd)); + } let data; try { @@ -93,10 +122,9 @@ async function startActor(token, actorId, inputJson) { const response = await fetch(url, { method: 'POST', - headers: { + headers: apifyHeaders(token, { 'Content-Type': 'application/json', - 'User-Agent': USER_AGENT, - }, + }), body: JSON.stringify(data), }); @@ -119,12 +147,14 @@ async function startActor(token, actorId, inputJson) { } async function pollUntilComplete(token, runId, timeout, interval) { - const url = `https://api.apify.com/v2/actor-runs/${runId}?token=${encodeURIComponent(token)}`; + const url = `https://api.apify.com/v2/actor-runs/${runId}`; const startTime = Date.now(); let lastStatus = null; while (true) { - const response = await fetch(url); + const response = await fetch(url, { + headers: apifyHeaders(token), + }); if (!response.ok) { const text = await response.text(); console.error(`Error: Failed to get run status: ${text}`); @@ -154,10 +184,10 @@ async function pollUntilComplete(token, runId, timeout, interval) { } async function downloadResults(token, datasetId, outputPath, format) { - const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`; + const url = `https://api.apify.com/v2/datasets/${datasetId}/items?format=json`; const response = await fetch(url, { - headers: { 'User-Agent': USER_AGENT }, + headers: apifyHeaders(token), }); if (!response.ok) { @@ -202,10 +232,10 @@ async function downloadResults(token, datasetId, outputPath, format) { } async function displayQuickAnswer(token, datasetId) { - const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`; + const url = `https://api.apify.com/v2/datasets/${datasetId}/items?format=json`; const response = await fetch(url, { - headers: { 'User-Agent': USER_AGENT }, + headers: apifyHeaders(token), }); if (!response.ok) { @@ -286,7 +316,12 @@ async function main() { } console.log(`Starting actor: ${args.actor}`); - const { runId, datasetId } = await startActor(token, args.actor, args.input); + const { runId, datasetId } = await startActor( + token, + args.actor, + args.input, + args.maxTotalChargeUsd, + ); console.log(`Run ID: ${runId}`); console.log(`Dataset ID: ${datasetId}`);