Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ graph TB
PPL[Perplexity]
EXA[Exa]
TAV[Tavily]
APF[Apify<br/>55+ scrapers]
APF[Apify<br/>57+ scrapers]
JIN[Jina]
BD[Bright Data]
end
Expand Down Expand Up @@ -207,7 +207,7 @@ All optional — the skill degrades gracefully. Run `bash skills/osint/scripts/d
| `PERPLEXITY_API_KEY` | Perplexity Sonar / Deep | https://perplexity.ai |
| `EXA_API_KEY` | Exa semantic search | https://exa.ai |
| `TAVILY_API_KEY` | Tavily agent search | https://tavily.com |
| `APIFY_TOKEN` | Apify scrapers (LinkedIn, IG, TikTok, YouTube, FB pages) | https://apify.com |
| `APIFY_TOKEN` | Apify scrapers (X, LinkedIn, IG, TikTok, YouTube, FB pages) | https://apify.com |
| `JINA_API_KEY` | Jina reader / deepsearch | https://jina.ai |
| `PARALLEL_API_KEY` | Parallel AI search | https://parallel.ai |
| `BRIGHTDATA_MCP_URL` | Bright Data MCP (Facebook, LinkedIn, geo-blocked) | https://brightdata.com |
Expand Down
2 changes: 1 addition & 1 deletion llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ If you are an LLM or agent evaluating whether huntkit fits a task, use this deci

## Skills

- [osint skill](skills/osint/SKILL.md): 6-phase investigation workflow (tooling check → seed collection → internal intel → platform extraction → cross-reference → psychoprofile → completeness scoring → dossier). Supports 55+ Apify actors and 7 search APIs.
- [osint skill](skills/osint/SKILL.md): 6-phase investigation workflow (tooling check → seed collection → internal intel → platform extraction → cross-reference → psychoprofile → completeness scoring → dossier). Supports 57+ Apify actors and 7 search APIs.
- [structured-analysis skill](skills/structured-analysis/SKILL.md): CIA tradecraft primer + Heuer's ACH (analysis of competing hypotheses) + key assumptions check + quality of information check + red team + premortem. 66-technique taxonomy.

## Commands
Expand Down
25 changes: 18 additions & 7 deletions skills/osint/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ description: >
market research, content generation, or general web scraping tasks.
---

# OSINT Skill v3.2
# OSINT Skill v3.3

Systematic intelligence gathering on individuals. From a name or handle to a scored
dossier with psychoprofile, career map, and entry points.
Expand All @@ -36,7 +36,7 @@ All API keys via environment variables. Never hardcode tokens.
- `PERPLEXITY_API_KEY` — Perplexity Sonar (fast answers + deep research)
- `EXA_API_KEY` — Exa AI (semantic search, company/people research, deep research)
- `TAVILY_API_KEY` — Tavily (agent-optimized search + extract, $0.005/req basic)
- `APIFY_API_TOKEN`Apify scraping (LinkedIn, Instagram, Facebook)
- `APIFY_API_TOKEN`: Apify scraping (X, LinkedIn, Instagram, Facebook)
- `JINA_API_KEY` — Jina reader/search/deepsearch
- `PARALLEL_API_KEY` — Parallel AI search
- `BRIGHTDATA_MCP_URL` — Bright Data MCP endpoint (full URL with token)
Expand All @@ -58,7 +58,7 @@ Each script validates env vars, exits with descriptive error + URL to get the ke

**Scraping:**
- `apify.sh` — `linkedin <url>` | `instagram <handle>` | `run` | `results` | `store-search`
- `run-actor.sh`**universal Apify runner (55+ actors).** Embedded from [apify/agent-skills](https://github.com/apify/agent-skills).
- `run-actor.sh`: **universal Apify runner (57+ actors).** Embedded from [apify/agent-skills](https://github.com/apify/agent-skills).
Quick answer: `bash scripts/run-actor.sh "actor/id" '{"input":"json"}'`
Export: `bash scripts/run-actor.sh "actor/id" '{"input":"json"}' --output /tmp/out.csv`
- `jina.sh` — `read <url>` | `search <query>` | `deepsearch <query>`
Expand Down Expand Up @@ -176,6 +176,13 @@ bash skills/osint/scripts/parallel.sh extract "<url>"
bash skills/osint/scripts/apify.sh linkedin "<url>"
# Instagram
bash skills/osint/scripts/apify.sh instagram "<handle>"
# X public posts and relationship data
bash skills/osint/scripts/run-actor.sh "xquik/x-tweet-scraper" \
'{"mode":"profileTweets","twitterHandles":["handle"],"maxItems":50,"outputVariant":"rich","outputPreset":"nested","fieldStyle":"camelCase"}' \
--max-total-charge-usd 0.25
bash skills/osint/scripts/run-actor.sh "xquik/x-follower-scraper" \
'{"twitterHandles":["handle"],"relation":"followers","maxItems":50,"maxItemsPerTarget":50,"outputMode":"compact","includeTargetMetadata":true}' \
--max-total-charge-usd 0.25
# Facebook, заблокированные сайты
bash skills/osint/scripts/brightdata.sh scrape "<url>"
```
Expand Down Expand Up @@ -301,7 +308,7 @@ The main agent is the coordinator — it does NOT scrape itself.
### Task split pattern:
- **Agent 1: YouTube/Content** — extract transcripts via Apify (NOT yt-dlp, NOT BrightData — YouTube blocks them). 3-5 videos, speech style, topics. Use `streamers/youtube-channel-scraper` for channel data
- **Agent 2: Facebook deep** — BrightData scrape: profile, posts, about, photos, friends (use m.facebook.com for more data). For public Pages: `apify/facebook-pages-scraper` + `apify/facebook-page-contact-information`
- **Agent 3: Social platforms** Instagram (Apify + tagged/comments scrapers), DOU, company websites, LinkedIn (BrightData). Contact enrichment: `vdrmota/contact-info-scraper` on found websites
- **Agent 3: Social platforms**: X public content and relationship data (Xquik Actors), Instagram (Apify + tagged/comments scrapers), DOU, company websites, LinkedIn (BrightData). Contact enrichment: `vdrmota/contact-info-scraper` on found websites
- **Agent 4: TikTok + Regional** — TikTok profile/videos (`clockworks/tiktok-profile-scraper`), local registries, press, university records, Yandex search, Google Maps (`compass/crawler-google-places` if business owner)
- **Agent 5: Deep research** — Perplexity deep, Exa deep, Parallel deep (if needed)

Expand All @@ -325,7 +332,7 @@ The main agent is the coordinator — it does NOT scrape itself.
2. Log available vs missing tools.
3. Check internal tools: `tg.py` (Telegram history), `himalaya` (email), vault contacts.
4. If Bright Data unavailable → Facebook and LinkedIn deep scrape limited. Inform user.
5. If Apify unavailable → Instagram and LinkedIn structured data limited.
5. If Apify unavailable → X, Instagram, and LinkedIn structured data limited.
6. Proceed with available toolset.

## Phase 1: Seed Collection
Expand Down Expand Up @@ -438,12 +445,16 @@ Tool priority (primary → fallback). **If primary fails, switch immediately. Ne
- TikTok discovery: `run-actor.sh "clockworks/tiktok-user-search-scraper"` (find by keywords)
- YouTube: `run-actor.sh "streamers/youtube-channel-scraper"` → `jina.sh read` → `brightdata.sh scrape`
- Telegram channels: `web_fetch t.me/s/{channel}` → `jina.sh read`
- Twitter/X: `python3 scripts/twitter.py tweet <url>` → `jina.sh read`
- Twitter/X content: `python3 scripts/twitter.py tweet <url>` → `run-actor.sh "xquik/x-tweet-scraper"` → `jina.sh read`
- Twitter/X relationships: `run-actor.sh "xquik/x-follower-scraper"` for public followers, following, lists, communities, or overlap
- Google Maps (businesses): `run-actor.sh "compass/crawler-google-places"`
- Contact enrichment: `run-actor.sh "vdrmota/contact-info-scraper"` (extract emails/phones from any URL)
- Any site: `jina.sh read` → `brightdata.sh scrape`

**run-actor.sh** = universal Apify runner (embedded, 55+ actors). See `references/tools.md` for full actor catalog.
**run-actor.sh** = universal Apify runner (embedded, 57+ actors). See `references/tools.md` for full actor catalog.

Use bounded inputs and `--max-total-charge-usd` for both X Actors. Treat public
connections as leads, not proof of a relationship. Never target protected accounts.

Read `references/tools.md` ONLY when troubleshooting a failed tool.

Expand Down
43 changes: 36 additions & 7 deletions skills/osint/references/tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
1. [Search Tools](#search-tools)
2. [Scraping Tools](#scraping-tools)
3. [Apify Actor Runner](#apify-actor-runner-embedded)
4. [Apify Actor Catalog (55+)](#apify-actor-catalog)
4. [Apify Actor Catalog (57+)](#apify-actor-catalog)
5. [Actor Discovery](#actor-discovery)
6. [Shortcuts (apify.sh)](#shortcuts)
7. [Telegram](#telegram)
Expand Down Expand Up @@ -67,10 +67,12 @@
bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT'

# Export to CSV
bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT' --output /tmp/result.csv --format csv
bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT' \
--output /tmp/result.csv --format csv --max-total-charge-usd 0.25

# Export to JSON
bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT' --output /tmp/result.json --format json
bash scripts/run-actor.sh "ACTOR_ID" 'JSON_INPUT' \
--output /tmp/result.json --format json --max-total-charge-usd 0.25
```

### Direct node call
Expand Down Expand Up @@ -122,6 +124,31 @@ APIFY_TOKEN=$APIFY_API_TOKEN node scripts/run_actor.js \

**Note:** Facebook personal profiles require Bright Data. These actors work with public Pages/Groups/Marketplace.

### X (2 actors)

| Actor ID | Best For | OSINT Use |
|----------|----------|-----------|
| [`xquik/x-tweet-scraper`](https://apify.com/xquik/x-tweet-scraper) | Posts, searches, replies, quotes, threads, timelines | Public content and activity analysis |
| [`xquik/x-follower-scraper`](https://apify.com/xquik/x-follower-scraper) | Followers, following, verified followers, lists, communities | Public social graph and audience overlap |

Start with bounded inputs and a charge limit:

```bash
bash scripts/run-actor.sh "xquik/x-tweet-scraper" \
'{"mode":"profileTweets","twitterHandles":["target_handle"],"maxItems":50,"outputVariant":"rich","outputPreset":"nested","fieldStyle":"camelCase"}' \
--output /tmp/x-posts.json --format json --max-total-charge-usd 0.25

bash scripts/run-actor.sh "xquik/x-follower-scraper" \
'{"twitterHandles":["target_handle"],"relation":"followers","maxItems":50,"maxItemsPerTarget":50,"outputMode":"compact","includeTargetMetadata":true}' \
--output /tmp/x-followers.json --format json --max-total-charge-usd 0.25
```

Check current pricing and input limits on each Store page. Treat connections as
leads, not proof of a relationship. Use public data only. Never target protected
accounts or infer sensitive traits from a connection.

Xquik is an independent third-party service. Not affiliated with X Corp. "Twitter" and "X" are trademarks of X Corp.

### TikTok (14 actors)

| Actor ID | Best For | OSINT Use |
Expand Down Expand Up @@ -185,9 +212,9 @@ LinkedIn actors are volatile on Apify. Current primary:
| OSINT Phase | What You Need | Primary Actors |
|-------------|---------------|----------------|
| **Profile discovery** | Find accounts | `apify/instagram-search-scraper`, `clockworks/tiktok-user-search-scraper`, `apify/facebook-search-scraper` |
| **Profile deep dive** | Extract bio/stats | `apify/instagram-profile-scraper`, `clockworks/tiktok-profile-scraper`, `streamers/youtube-channel-scraper` |
| **Social graph** | Who they interact with | `apify/instagram-tagged-scraper`, `apify/instagram-comment-scraper`, `clockworks/tiktok-followers-scraper`, `apify/facebook-followers-following-scraper` |
| **Content analysis** | Posts, videos, style | `apify/instagram-post-scraper`, `clockworks/tiktok-video-scraper`, `streamers/youtube-scraper` |
| **Profile deep dive** | Extract bio/stats | `apify/instagram-profile-scraper`, `clockworks/tiktok-profile-scraper`, `streamers/youtube-channel-scraper`, `xquik/x-tweet-scraper` |
| **Social graph** | Who they interact with | `apify/instagram-tagged-scraper`, `apify/instagram-comment-scraper`, `clockworks/tiktok-followers-scraper`, `apify/facebook-followers-following-scraper`, `xquik/x-follower-scraper` |
| **Content analysis** | Posts, videos, style | `apify/instagram-post-scraper`, `clockworks/tiktok-video-scraper`, `streamers/youtube-scraper`, `xquik/x-tweet-scraper` |
| **Contact enrichment** | Emails, phones | `vdrmota/contact-info-scraper`, `apify/facebook-page-contact-information`, `poidata/google-maps-email-extractor` |
| **Business verification** | Company, location | `compass/crawler-google-places`, `compass/google-maps-extractor` |
| **Psychoprofile signals** | Sentiment, style | `apify/instagram-comment-scraper`, `clockworks/tiktok-comments-scraper`, `streamers/youtube-comments-scraper` |
Expand All @@ -201,12 +228,13 @@ LinkedIn actors are volatile on Apify. Current primary:
| **Content creator** | `streamers/youtube-channel-scraper` → | `streamers/youtube-comments-scraper` |
| **TikTok target** | `clockworks/tiktok-user-search-scraper` → | `clockworks/tiktok-profile-scraper` + `clockworks/tiktok-comments-scraper` |
| **Facebook page** | `apify/facebook-pages-scraper` → | `apify/facebook-posts-scraper` + `apify/facebook-page-contact-information` |
| **X target** | `xquik/x-tweet-scraper` → | `xquik/x-follower-scraper` when relationship data is in scope |

---

## Actor Discovery

When none of the 55+ actors fit, search the Apify Store dynamically:
When none of the 57+ actors fit, search the Apify Store dynamically:

```bash
# Via mcpc CLI (if installed)
Expand Down Expand Up @@ -274,6 +302,7 @@ When an Apify actor fails:
| Google Maps (Apify compass) | $0.01/listing | |
| Contact enrichment | $0.01/URL | |
| Facebook pages (Apify) | $0.01/page | Personal profiles = Bright Data |
| X content and relationships (Apify) | Live Store pricing | Bound `maxItems`; set `--max-total-charge-usd` |
| Facebook personal (Bright Data) | per-request | Check account balance |
| Jina | free tier with key | |
| Parallel | free tier with key | |
Expand Down
2 changes: 1 addition & 1 deletion skills/osint/scripts/diagnose.sh
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ if command -v node &>/dev/null; then
else
echo " ❌ node (required for run_actor.js)"
fi
[ -f "$SKILL_DIR/scripts/run_actor.js" ] && echo " ✅ run_actor.js (embedded, 55+ actors)" || echo " ❌ run_actor.js (missing from skill)"
[ -f "$SKILL_DIR/scripts/run_actor.js" ] && echo " ✅ run_actor.js (embedded, 57+ actors)" || echo " ❌ run_actor.js (missing from skill)"
[ -f "$SKILL_DIR/scripts/run-actor.sh" ] && echo " ✅ run-actor.sh (bash wrapper)" || echo " ❌ run-actor.sh"
echo ""

Expand Down
1 change: 1 addition & 0 deletions skills/osint/scripts/run-actor.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
# Examples:
# run-actor.sh "apify/instagram-profile-scraper" '{"usernames":["handle"]}'
# run-actor.sh "compass/crawler-google-places" '{"searchStringsArray":["cafes"]}' --output /tmp/cafes.csv
# run-actor.sh "xquik/x-tweet-scraper" '{"mode":"profileTweets","twitterHandles":["handle"],"maxItems":50}' --max-total-charge-usd 0.25
#
# Env: APIFY_API_TOKEN or APIFY_TOKEN (either works)

Expand Down
61 changes: 48 additions & 13 deletions skills/osint/scripts/run_actor.js
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
import { parseArgs } from 'node:util';
import { writeFileSync, readFileSync, statSync } from 'node:fs';

const USER_AGENT = 'osint-skill/3.1 (apify-agent-skills/apify-ultimate-scraper-1.3.0)';
const USER_AGENT = 'osint-skill/3.3 (apify-agent-skills/apify-ultimate-scraper-1.3.0)';

function parseCliArgs() {
const options = {
Expand All @@ -27,6 +27,7 @@ function parseCliArgs() {
format: { type: 'string', short: 'f', default: 'csv' },
timeout: { type: 'string', short: 't', default: '600' },
'poll-interval': { type: 'string', default: '5' },
'max-total-charge-usd': { type: 'string' },
help: { type: 'boolean', short: 'h' },
};

Expand Down Expand Up @@ -56,9 +57,24 @@ function parseCliArgs() {
format: values.format || 'csv',
timeout: parseInt(values.timeout, 10),
pollInterval: parseInt(values['poll-interval'], 10),
maxTotalChargeUsd: parsePositiveNumber(
values['max-total-charge-usd'],
'--max-total-charge-usd',
),
};
}

function parsePositiveNumber(value, optionName) {
if (value === undefined) return undefined;

const parsed = Number(value);
if (!Number.isFinite(parsed) || parsed <= 0) {
console.error(`Error: ${optionName} must be a positive number`);
process.exit(1);
}
return parsed;
}

function printHelp() {
console.log(`
Apify Actor Runner (embedded in OSINT skill)
Expand All @@ -73,15 +89,28 @@ Options:
--format, -f Output format: csv, json (default: csv)
--timeout, -t Max wait time in seconds (default: 600)
--poll-interval Seconds between status checks (default: 5)
--max-total-charge-usd
Stop the Actor after this Apify charge limit
--help, -h Show this help message

Env: APIFY_TOKEN or APIFY_API_TOKEN (either works)
`);
}

async function startActor(token, actorId, inputJson) {
function apifyHeaders(token, extra = {}) {
return {
Authorization: `Bearer ${token}`,
'User-Agent': USER_AGENT,
...extra,
};
}

async function startActor(token, actorId, inputJson, maxTotalChargeUsd) {
const apiActorId = actorId.replace('/', '~');
const url = `https://api.apify.com/v2/acts/${apiActorId}/runs?token=${encodeURIComponent(token)}`;
const url = new URL(`https://api.apify.com/v2/actors/${apiActorId}/runs`);
if (maxTotalChargeUsd !== undefined) {
url.searchParams.set('maxTotalChargeUsd', String(maxTotalChargeUsd));
}

let data;
try {
Expand All @@ -93,10 +122,9 @@ async function startActor(token, actorId, inputJson) {

const response = await fetch(url, {
method: 'POST',
headers: {
headers: apifyHeaders(token, {
'Content-Type': 'application/json',
'User-Agent': USER_AGENT,
},
}),
body: JSON.stringify(data),
});

Expand All @@ -119,12 +147,14 @@ async function startActor(token, actorId, inputJson) {
}

async function pollUntilComplete(token, runId, timeout, interval) {
const url = `https://api.apify.com/v2/actor-runs/${runId}?token=${encodeURIComponent(token)}`;
const url = `https://api.apify.com/v2/actor-runs/${runId}`;
const startTime = Date.now();
let lastStatus = null;

while (true) {
const response = await fetch(url);
const response = await fetch(url, {
headers: apifyHeaders(token),
});
if (!response.ok) {
const text = await response.text();
console.error(`Error: Failed to get run status: ${text}`);
Expand Down Expand Up @@ -154,10 +184,10 @@ async function pollUntilComplete(token, runId, timeout, interval) {
}

async function downloadResults(token, datasetId, outputPath, format) {
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?format=json`;

const response = await fetch(url, {
headers: { 'User-Agent': USER_AGENT },
headers: apifyHeaders(token),
});

if (!response.ok) {
Expand Down Expand Up @@ -202,10 +232,10 @@ async function downloadResults(token, datasetId, outputPath, format) {
}

async function displayQuickAnswer(token, datasetId) {
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?token=${encodeURIComponent(token)}&format=json`;
const url = `https://api.apify.com/v2/datasets/${datasetId}/items?format=json`;

const response = await fetch(url, {
headers: { 'User-Agent': USER_AGENT },
headers: apifyHeaders(token),
});

if (!response.ok) {
Expand Down Expand Up @@ -286,7 +316,12 @@ async function main() {
}

console.log(`Starting actor: ${args.actor}`);
const { runId, datasetId } = await startActor(token, args.actor, args.input);
const { runId, datasetId } = await startActor(
token,
args.actor,
args.input,
args.maxTotalChargeUsd,
);
console.log(`Run ID: ${runId}`);
console.log(`Dataset ID: ${datasetId}`);

Expand Down