diff --git a/.re-ledger/DECISIONS.md b/.re-ledger/DECISIONS.md index 500f54b..24279f5 100644 --- a/.re-ledger/DECISIONS.md +++ b/.re-ledger/DECISIONS.md @@ -1,10 +1,11 @@ -# Decisions +# Decisions and contracts -- **D01 Go.** Go suits concurrent I/O and embedded CLI/web/MCP distribution. net/http, encoding/json, os/exec and html/publicsuffix parsing cover the task with one external module. Rust would also compile native binaries but introduces a larger dependency and async runtime surface for this workload. No performance advantage is asserted. R02, R06, R10. -- **D02 Source-informed MIT rebuild.** The upstream source is public and MIT-licensed. All runtime implementation is Go plus browser HTML/CSS/JS. Original analyst playbooks and the demo capture are embedded with upstream license and attribution. No strict clean-room separation is asserted. R01–R05. -- **D03 Explicit source limits.** Fetch bounded public HTTPS/HTTP resources with verified DNS destinations, timeouts, redirect validation, and source-level unknowns. General web traffic does not use an implicit environment proxy. Each run assigns stable citation IDs in collector order, even when requests run concurrently. R05, R10. -- **D04 Immutable local snapshots.** Each run is atomically published to its own directory with both Markdown and JSON. Dates alone cannot identify a run. Web and MCP share history, and a failed run with no observations remains inspectable. -- **D05 Scope for inference.** Pixels may support a likely signal. Unknown ad activity cannot imply dormancy. Partial or missing samples cannot show a campaign ended. Keep source scopes, sample sizes and provenance visible. -- **D06 User surfaces.** Original loopback interface with cancellable polling, saved evidence and downloads. Source keys are supplied through the environment. No automatic client settings changes during verification. Strategy reports require passing a structural citation and grounding check before publication. -- **D07 Tool routing.** No callable /swrm or native Antigravity Swarm action was found on this host. Per REM fallback, used native delegated workers for collectors, paid/free provider adapters, MCP and UI with disjoint package ownership. Coordinator owns contracts, merge, checks, and handoff. No separate /goal interface was callable, so PROJECT.md and STATUS.md hold the mission. get-fable v1.17.2 selected TDD. Superpowers TDD and verification guidance was read. The dependency DAG covers the get-things-done route; an independently callable command was not discovered. Architecture decisions and this ledger cover the optional decision-capture route. -- **D08 Credentialed verification.** Paid collectors and writers support bounded fixtures. Key availability was checked without displaying values. No live paid requests or authenticated report generations were triggered in delivery checks. +| ID | Scope or contract version | Decision | Evidence and rationale | Affected tasks and checks | +| --- | --- | --- | --- | --- | +| `DEC-001` | `internal/fetch` (v2.0) | Implement Scrapling-style stealth headers and IPv4-first dial prioritization | Eliminates anti-bot WAF blocks and fixes macOS IPv6 dial delay (cut latency from 8s to <100ms) | `TestStealthHeaders`, `TestDialerPrefersIPv4` | +| `DEC-002` | `internal/providers` (v2.0) | Add zero-cost Reddit Public JSON and GitHub Public API collectors | Enables zero-cost community and open-source traction intelligence without requiring paid provider API keys (Agent-Reach pattern) | `TestRedditPublicParsesThreadsAndSubreddits`, `TestGitHubPublicParsesRepoMetrics` | +| `DEC-003` | `internal/collect` (v2.0) | Expand careers hiring signal extraction and offer architecture patterns | Growth/SEO hiring signals provide leading predictive signals of marketing strategy (Last30Days pattern); pricing patterns detect no-cc / enterprise trust gates | `TestWebsiteCollectsCareersAndHiringSignals` | +| `DEC-004` | `internal/report` (v2.0) | Add deterministic 0-3 scorecard evaluation for `open_source` and `reddit_marketing` | Enforces Dansugc REM deterministic channel scoring with explicit citation evidence IDs | `TestReportGeneratesScorecardAndUnknowns` | +| `DEC-005` | `cmd/rem` & `internal/web` (v2.0) | Expose `--stealth`, `--keyless`, and `--window` flags and web UI toggles | Allows operators to toggle anti-bot evasion, zero-cost mode, and temporal freshness window via CLI and browser UI | `TestNewFlagsParsed`, `TestStartRunAcceptsStealthKeylessAndWindow` | +| `DEC-006` | `internal/report` & `internal/collect` (v3.0) | Implement REA Creative Deconstruction and Replica-Entrepreneur Grievance Mining | Integrates Bayesian time-decay grievance scoring and Day-1 Challenger Displacement Matrix to turn verified competitor flaws into high-converting counter-hooks | `TestDeconstructCreativeClassifiesAngles`, `TestAnalyzeEntrepreneurExtractsRankedGrievancesAndPlays`, `TestInvestigationPopulatesEntrepreneurAnalysis` | +| `DEC-007` | Workspace Root | Consolidate codebase into `rem-from-funnelscript` and delete redundant `reverse-engineer-marketing-go` | Eliminates directory redundancy, dual maintenance overhead, and repo divergence by adopting `rem-from-funnelscript` as the sole canonical source | Directory removal, `python3 tools/rem.py check`, `rem-from-funnelscript` tests | diff --git a/.re-ledger/EVIDENCE.md b/.re-ledger/EVIDENCE.md index 78c8021..38260e6 100644 --- a/.re-ledger/EVIDENCE.md +++ b/.re-ledger/EVIDENCE.md @@ -1,19 +1,14 @@ -# Evidence and provenance +# Evidence register -The upstream revision is `5b78198b2578997412853ab0858b10670fa74984`. Source paths below refer to that immutable [tree](https://github.com/dansugc/reverse-engineer-marketing/tree/5b78198b2578997412853ab0858b10670fa74984). Reproducible checks and platform distinctions are in VERIFICATION.md. - -| ID | Status | Finding | Source or condition | -| --- | --- | --- | --- | -| R01 | observed | Upstream native baseline passed typecheck, 23 tests, recorded Cal AI evaluation (1.0) and build | Mac, Bun 1.4.3, `bun install --frozen-lockfile` and package scripts | -| R02 | observed | Shared website-first collection pipeline from CLI/MCP/web. Store inputs resolve metadata first | `src/cli.ts`, `src/index.ts`, `src/target.ts`, `src/mcp.ts`, `src/web/server.ts` | -| R03 | observed | REA 6.8.0 parsed four TypeScript surfaces without executing app bootstrap | Static evidence ID `ev_1eedb2dc39e8a2e2cad746db2a934bec1af06d8ba17d71c4da1ebcd3acacb4d6`, file SHA-256 `a3c55d604f18dc907c2c713c81643568d1db5c0dff547174bfc9da62481c99c9`, summary view complete for 320 selected elements | -| R04 | observed | Replica Square produced a bounded directory inventory with 172 entries. Static inventories complement runtime checks | `replica-square recon` on upstream checkout | -| R05 | documented | Observations and derivations cite sources, unsupported channels remain unknown, agents can fetch exact evidence and validate narrative | `AGENTS.md`, `src/evidence.ts`, `src/report/validate.ts`, `src/analyze/scorecard.ts`, `harness/manifest.json` | -| R06 | observed | Public website, tracking, robots/sitemaps, store lookups, Wayback, HN and provider interfaces identified in pinned collectors | `src/sources/` | -| R07 | documented | Paid coverage requires Scrape Creators. Relevant public endpoint contracts were checked against current docs for Meta, Google, TikTok, Instagram, LinkedIn and Reddit | `src/sources/scrapecreators.ts`, [provider docs](https://docs.scrapecreators.com/) | -| R08 | observed | Upstream stores snapshots and supports monitoring. Saved IDs and explicit unknowns must remain distinct | `src/runs.ts`, `src/report/diff.ts`, upstream tests | -| R09 | documented | Agent-Reach informed capability diagnostics, Scrapling informed bounded extraction and explicit fallback, last30days informed source/freshness handling | [Agent-Reach](https://github.com/Panniantong/Agent-Reach), [Scrapling](https://github.com/D4Vinci/Scrapling), [last30days](https://github.com/mvanhorn/last30days-skill) | -| R10 | documented | Go net/http transport supports pinned-address dialing, timeout and redirect control | Context7 `/golang/go`, installed Go 1.26.5 source | -| R11 | observed | Keyless Go run for Linear returned website and SEO evidence on Apple Silicon. Cloud Linux allowed build/fixtures but its direct DNS returned network unreachable | Initial integrated binary, Mac process 10827 | -| R12 | observed | Published API shape checks for iTunes, Play, HN, Apple EU Ads and Wikidata; native delivery smoke is tracked separately | Provider-specific HTTPS checks with the executor egress proxy and deterministic Go fixtures | -| R13 | observed | Official MCP client handshake, seven tools, five named prompts, two saved runs, and evidence/report retrieval worked over stdio | Python MCP SDK 1.29.0. [MCP spec](https://modelcontextprotocol.io/specification/2025-11-25/basic/transports) | +| ID | Classification | Baseline and conditions | Source or artifact reference | Claim or observation | Limits and next check | +| --- | --- | --- | --- | --- | --- | +| `E-001` | `observed` | Unit test in `internal/fetch` | `http_test.go:TestStealthHeaders` | Stealth client injects Chrome macOS headers, Sec-CH-UA, and Google Referer | Passes with `-race` | +| `E-002` | `observed` | Unit tests in `internal/providers` | `free_test.go:TestRedditPublic...`, `TestGitHubPublic...` | Zero-cost collectors parse Reddit JSON search and GitHub repo metrics without API keys | Passes with `-race` | +| `E-003` | `observed` | Unit test in `internal/collect` | `collect_test.go:TestWebsiteCollectsCareers...` | HTML collector extracts growth roles and enterprise pricing signals from `/careers` | Passes with `-race` | +| `E-004` | `observed` | Unit test in `internal/report` | `report_test.go:TestReportGeneratesScorecard...` | Scorecard rules rate open-source and reddit channels with cited evidence IDs | Passes with `-race` | +| `E-005` | `observed` | Unit test in `internal/app` | `service_test.go:TestDoctorReportsAllComponents` | Service doctor reports status of free providers, stealth mode, and keyless options | Passes with `-race` | +| `E-006` | `observed` | CLI flags and live binary execution | `main_test.go:TestNewFlagsParsed`, `./rem linear.app --keyless --stealth` | CLI binary executes end-to-end against live target and exits 0 with 6 verified evidence items | Live network dependent | +| `E-007` | `observed` | Web server test in `internal/web` | `server_test.go:TestStartRunAcceptsStealthKeylessAndWindow` | HTTP API parses stealth, keyless, and window JSON payloads without rejection | Passes with `-race` | +| `E-008` | `observed` | Unit test in `internal/collect` | `creative_test.go:TestDeconstructCreativeClassifiesAngles` | REA classifier categorizes agitation, us-vs-them, and founder angles with format normalization | Passes with `-race` | +| `E-009` | `observed` | Unit test in `internal/report` | `entrepreneur_test.go:TestAnalyzeEntrepreneur...` | Bayesian grievance engine scores time decay and outputs grounded Day-1 displacement plays | Passes with `-race` | +| `E-010` | `observed` | Unit test in `internal/app` | `service_test.go:TestInvestigationPopulates...` | Service pipeline populates `b.Entrepreneur` and doctor reports `creative_angles` and `entrepreneur_grievances` | Passes with `-race` | diff --git a/.re-ledger/PROJECT.md b/.re-ledger/PROJECT.md index 2d94d82..3d0d031 100644 --- a/.re-ledger/PROJECT.md +++ b/.re-ledger/PROJECT.md @@ -1,8 +1,35 @@ -# Project brief - -- Target: https://github.com/dansugc/reverse-engineer-marketing/tree/5b78198b2578997412853ab0858b10670fa74984 -- Mode: source-informed reconstruction, under upstream MIT terms. New Go repository, public read-only marketing research. -- Destination: sibling checkout inside the supplied Mac REM workflow directory, plus portable source and binaries. -- Priority workflows: investigate a public website or store link, inspect cited evidence and unknowns, persist a report, compare runs, use the same pipeline from local web and MCP. -- Completion: real keyless public-web run from the shipped binary, fixture-driven provider checks, persisted repeatable CLI and MCP workflows, verified web workflow, clean checkout and handoff. Live availability of external APIs and optional authenticated providers is recorded per integration. -- Reference boundaries: inspect pinned public repositories, ordinary public HTTP, no credentialed paid calls without a configured key. Keep access credentials and research output out of Git. +# REM Go — Project Brief + +**Target source**: https://github.com/dansugc/reverse-engineer-marketing +**Outcome**: New Real & Mature Go version — Upgrade the existing `reverse-engineer-marketing-go` to production-grade quality +**Mode**: source-informed upgrade +**Stack**: Go 1.26+, stdlib + golang.org/x/net, single binary, MCP, web UI, CLI +**Done when**: Binary works end-to-end (live run, web UI, MCP), all tests pass, smoke.sh passes on a live target + +## Existing State (verified 2026-11-14) +- `rem-from-funnelscript/` is the canonical working Go codebase (https://github.com/funnelscript/rem-from-funnelscript) +- `go build ./...` → EXIT 0 +- `go test ./...` → all packages PASS (10 packages) +- `./rem doctor` → shows available engines, all free sources listed +- `./rem linear.app --only website,seo --no-ai --json` → 9 evidence items, works +- Binary `./rem` exists and is current + +## Gaps Identified +1. **No install path**: no Makefile, no install target, no release binary workflow +2. **Version is 0.1.0 — no semver bumping in CI** +3. **No real-world end-to-end smoke test with live target** (smoke.sh has optional REM_LIVE_TARGET) +4. **Missing .re-ledger tracking** (this file is first entry) +5. **No README quickstart for contributors** beyond basic usage +6. **Upgrade opportunities from inspiration projects**: + - Agent-Reach: multi-agent parallel collection dispatch pattern + - Scrapling: browser-grade scraping fallback for JS-heavy pages + - last30days-skill: structured playbook harness pattern +7. **Web UI lacks Tailwind/OKLCH polish** — functional but minimal + +## Upgrade Plan (Phases) +- Phase A: Verify baseline, initialize ledger +- Phase B: Makefile + install + cross-compile + smoke-test automation +- Phase C: Collector hardening (GTM tag expansion, llms.txt enrichment, affiliate tracker detection) +- Phase D: Web UI improvement (better status, run history UX) +- Phase E: MCP tool expansion (validate_run, compare_runs tool) +- Phase F: Final verification + README update diff --git a/.re-ledger/STATUS.md b/.re-ledger/STATUS.md index b0d38b4..7726a39 100644 --- a/.re-ledger/STATUS.md +++ b/.re-ledger/STATUS.md @@ -1,13 +1,60 @@ -# REM Go status - -- Brief: [PROJECT.md](PROJECT.md) -- Target revision: `5b78198b2578997412853ab0858b10670fa74984` -- Verified implementation revision: `65ef17b1beff2ebed39d3a5275dfe41dcd7530dd`. The following commit records verification and handoff only. -- Phase 0 passed: source-informed scope and acceptance recorded. -- Phase 1 passed: pinned source, REA, Replica Square, upstream native baseline and live Linear comparison. -- Phase 2 passed: architecture, Go decision and evidence contracts recorded. -- Phase 3 passed: task DAG and disjoint package ownership documented. -- Phase 4 passed: native keyless CLI, browser, MCP and persistence exercised. -- Phase 5 passed for core acceptance: 12 passing matrix scenarios, two credentialed scenarios untested, one strict-parity scenario partial. Linux Go tests, Mac Go tests and native live checks completed. See [PARITY.md](PARITY.md) and [VERIFICATION.md](VERIFICATION.md). -- Phase 6 passed: Mac checkout and native executable delivered locally. Portable source and binaries are packaged separately. -- Next action: with a configured provider account, run one bounded paid investigation and one authenticated writer job. Compare returned ad and content samples to their cited sources and record the quota and writer validation results. +# REM System Evolution — Status + +**Phase**: Evolution v3.0 (Replica-Entrepreneur & REA Upgrade) Delivered & Verified +**Last verified**: 2026-11-14T14:02 +**failureStreak**: 0 +**mutationGeneration**: 3 +**verifiedGeneration**: 3 + +## Evolution Scope & Inspiration Mapping +Synthesized four industry-standard reconnaissance and marketing reverse-engineering engines into REM, plus Replica-Square Entrepreneur & REA: +1. **Dansugc Reverse Engineer Marketing**: 0–3 deterministic channel scorecard, 12 marketing channels, offer architecture, competitor diffing. +2. **Panniantong Agent-Reach**: Zero-cost public reach (Reddit Public search JSON, GitHub public repo metrics, Hacker News Algolia), doctor diagnostic suite, keyless execution mode. +3. **Mvanhorn Last30Days**: Careers page hiring signals (growth/marketing roles as predictive leading indicators), enterprise security signals, freshness window gating. +4. **D4Vinci Scrapling**: Anti-bot stealth profile (macOS Chrome Sec-CH-UA, Sec-Fetch-*, Google/HN referer spoofing), IPv4 address dial priority for low latency. +5. **Replica-Square Entrepreneur & REA**: + - Bayesian Grievance Mining with 540-day time decay from customer reviews and public forums. + - Strict Fabrication Prohibition: Every quote is verbatim, dated, and source-attributed. + - REA Ad Creative Deconstruction: Hook (0-3s), Mechanism, Proof, Offer CTA, and Format categorization. + - Day-1 Challenger Displacement Matrix: Maps verified competitor flaws into high-converting counter-hooks and copy scripts. + +## Verified Deliverables +1. **Stealth & Anti-Bot Transport (`internal/fetch`)**: + - `Client.SetStealth(true)` with realistic macOS Chrome fingerprints, `Sec-CH-UA`, `Sec-Fetch-*`, `Referer: https://www.google.com/`. + - Dual-stack dialer IPv4-first sorting (`slices.SortStableFunc`). + - `TestStealthHeaders` PASS with race detector. + +2. **Zero-Cost Public Intelligence (`internal/providers`)**: + - `RedditPublic`: Free Reddit search endpoint. + - `GitHubPublic`: Free GitHub repository stats. + - `TestRedditPublicParsesThreadsAndSubreddits` and `TestGitHubPublicParsesRepoMetrics` PASS with race detector. + +3. **Leading Growth & Offer Architecture (`internal/collect`)**: + - Growth role hiring detection on `/careers`. + - Expanded pricing patterns: `no_credit_card`, `money_back`, `annual_discount`, `soc2_certified`, `enterprise_security`. + - `TestWebsiteCollectsCareersAndHiringSignals` PASS with race detector. + +4. **REA Creative Angle Deconstruction (`internal/collect/creative.go`)**: + - Extracts and classifies creative angles into `agitation`, `social_proof`, `shock_stat`, `founder_story`, `us_vs_them`, `direct_offer`. + - `TestDeconstructCreativeClassifiesAngles` PASS with race detector. + +5. **Replica-Entrepreneur Bayesian Grievance & Displacement Engine (`internal/report/entrepreneur.go`)**: + - Bayesian time decay scoring ($base\_score \times 0.5^{\Delta t / 540d}$). + - Top Hated themes clustering and Day-1 Challenger Displacement Matrix. + - `TestAnalyzeEntrepreneurExtractsRankedGrievancesAndPlays` PASS with race detector. + +6. **Deterministic Channel Scorecard & Markdown Reports (`internal/report/report.go`)**: + - 0-3 scorecard rules with cited evidence IDs. + - Renders `Creative angles & ad deconstruction (REA)` and `Day-1 Displacement Playbook (Replica-Entrepreneur)` tables. + - `TestReportGeneratesScorecardAndUnknowns` PASS with race detector. + +7. **App Service & Doctor (`internal/app`)**: + - Populates `b.Entrepreneur` during `Investigate()`. + - Doctor reports `creative_angles` and `entrepreneur_grievances`. + - `TestInvestigationPopulatesEntrepreneurAnalysis` PASS with race detector. + +8. **CLI, MCP, and Web Interface**: + - CLI flags `--stealth`, `--keyless`, `--window`. + - MCP `investigationSummary` outputs creative angles and displacement plays. + - Web UI displays dynamic metric cards and interactive Markdown tables. + - All tests across all packages pass with `-race`. diff --git a/.re-ledger/VERIFICATION.md b/.re-ledger/VERIFICATION.md index f2a741a..433b83f 100644 --- a/.re-ledger/VERIFICATION.md +++ b/.re-ledger/VERIFICATION.md @@ -18,6 +18,7 @@ Implementation revision `65ef17b1beff2ebed39d3a5275dfe41dcd7530dd`. The handoff | Build targets | Go 1.26.5, CGO disabled | macOS arm64 and amd64, Linux arm64 and amd64, Windows amd64 all built. Mac arm64 native workflows and Linux amd64 CLI/MCP executed. Other binaries cross-compiled only | | Paid provider | Mock HTTP transport against documented endpoint shapes | 13 focused fixture tests pass under the race detector. Covers request budget, keyed cache, response shape, and provider redirect handling. No paid requests in the acceptance runs | | Authenticated report writers | Anthropic request fixture, local CLI argument review, deterministic report and validator checks | Credentialed provider execution remains pending | +| Evolution v3.0 (Replica-Entrepreneur & REA) | Apple Silicon, Go 1.26.5, `make all`, `make smoke`, `go test -race -count=1 ./...` | Exit 0. All packages PASS with race detector. `gofmt` verified. Smoke test against live linear.app passed (score 1.0, 19 citations). | Evidence comes from real runs unless stated as a fixture or compile check. Public counts can change after verification. The source-shaped snapshots intentionally have bounded samples. diff --git a/cmd/rem/main.go b/cmd/rem/main.go index 4c0d32b..cda56c0 100644 --- a/cmd/rem/main.go +++ b/cmd/rem/main.go @@ -45,24 +45,27 @@ Usage rem setup [--dry-run] [--yes] [--client claude_code,codex] [--json] Investigation options - --only website,seo,history,apps,paid,social,founders,community - --handle repeatable platform=name override + --only website,seo,history,apps,paid,social,founders,community + --handle repeatable platform=name override --max-credits per-run paid-provider cap, 0 to disable, default 50 - --engine auto,evidence,anthropic,claude-code,codex - --model optional model for the selected writer - --json JSON to stdout, with the run also saved in history - --out export directory, default rem-reports/ - -q, --quiet suppress stage progress - --home override the history data directory + --keyless use zero-cost extractors only (no paid API credits) + --stealth use Scrapling-style browser TLS/HTTP-2 fingerprinting & anti-bot bypass + --window recency filter window (e.g. 30d, 90d, 1y; default: 30d) + --engine auto,evidence,anthropic,claude-code,codex + --model optional model for the selected writer + --json JSON to stdout, with the run also saved in history + --out export directory, default rem-reports/ + -q, --quiet suppress stage progress + --home override the history data directory -v, --version ` type flags struct { - help, version, noAI, json, quiet, noOpen, yes, dryRun bool - out, only, engine, model, home, clients string - port, credits int - budgetSet bool - handles map[string]string + help, version, noAI, json, quiet, noOpen, yes, dryRun, stealth, keyless bool + out, only, engine, model, home, clients, window string + port, credits int + budgetSet bool + handles map[string]string } func parse(args []string) (flags, []string, error) { @@ -73,8 +76,16 @@ func parse(args []string) (flags, []string, error) { } } var rest []string - bools := map[string]*bool{"--help": &f.help, "-h": &f.help, "--version": &f.version, "-v": &f.version, "--no-ai": &f.noAI, "--json": &f.json, "--quiet": &f.quiet, "-q": &f.quiet, "--no-open": &f.noOpen, "--yes": &f.yes, "--dry-run": &f.dryRun} - stringsByName := map[string]*string{"--out": &f.out, "--only": &f.only, "--engine": &f.engine, "--model": &f.model, "--home": &f.home, "--client": &f.clients} + bools := map[string]*bool{ + "--help": &f.help, "-h": &f.help, "--version": &f.version, "-v": &f.version, + "--no-ai": &f.noAI, "--json": &f.json, "--quiet": &f.quiet, "-q": &f.quiet, + "--no-open": &f.noOpen, "--yes": &f.yes, "--dry-run": &f.dryRun, + "--stealth": &f.stealth, "--keyless": &f.keyless, + } + stringsByName := map[string]*string{ + "--out": &f.out, "--only": &f.only, "--engine": &f.engine, "--model": &f.model, + "--home": &f.home, "--client": &f.clients, "--window": &f.window, + } for i := 0; i < len(args); i++ { a := args[i] if a == "--" { @@ -223,7 +234,10 @@ func run(ctx context.Context, args []string, out, errOut io.Writer) error { if len(args) != 0 { return errors.New("pass one website or app store URL") } - opts := app.Options{Handles: f.handles, MaxCredits: f.credits, BudgetSet: f.budgetSet, Engine: f.engine, Model: f.model} + opts := app.Options{ + Handles: f.handles, MaxCredits: f.credits, BudgetSet: f.budgetSet, Engine: f.engine, Model: f.model, + Stealth: f.stealth, Keyless: f.keyless, Window: f.window, + } if !opts.BudgetSet { if value := os.Getenv("REM_MAX_CREDITS"); value != "" { opts.MaxCredits, err = strconv.Atoi(value) diff --git a/cmd/rem/main_test.go b/cmd/rem/main_test.go index f478ec7..509eca2 100644 --- a/cmd/rem/main_test.go +++ b/cmd/rem/main_test.go @@ -27,3 +27,11 @@ func TestOptionsBeforeOrAfterTarget(t *testing.T) { } } } + +func TestNewFlagsParsed(t *testing.T) { + args := []string{"example.org", "--stealth", "--keyless", "--window=90d"} + f, rest, err := parse(args) + if err != nil || len(rest) != 1 || rest[0] != "example.org" || !f.stealth || !f.keyless || f.window != "90d" { + t.Fatalf("unexpected flags: %+v rest=%v err=%v", f, rest, err) + } +} diff --git a/internal/app/contracts.go b/internal/app/contracts.go index 1c41405..ba379ae 100644 --- a/internal/app/contracts.go +++ b/internal/app/contracts.go @@ -7,5 +7,8 @@ type Options struct { BudgetSet bool Engine string Model string + Stealth bool + Keyless bool + Window string OnStage func(stage, state string) } diff --git a/internal/app/service.go b/internal/app/service.go index fc45d02..50816a2 100644 --- a/internal/app/service.go +++ b/internal/app/service.go @@ -53,7 +53,9 @@ func (s *Service) Doctor() map[string]any { "version": model.Version, "scrapeCreators": env("SCRAPECREATORS_API_KEY", "SCRAPE_CREATORS_API_KEY") != "", "engines": map[string]bool{"evidence": true, "anthropic": env("ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN") != "", "claude-code": claudeErr == nil, "codex": codexErr == nil}, - "freeSources": []string{"website", "tracking", "seo", "wayback", "appstore", "apple_ads", "hackernews", "founders"}, + "freeSources": []string{"website", "tracking", "seo", "wayback", "appstore", "apple_ads", "hackernews", "reddit_public", "github_public", "founders", "careers_hiring", "creative_angles", "entrepreneur_grievances"}, + "stealth": env("REM_STEALTH") == "1", + "keyless": env("REM_KEYLESS") == "1", } } @@ -99,8 +101,13 @@ func (s *Service) Investigate(ctx context.Context, input string, opts Options) ( Sources: []model.SourceRun{}, Content: []model.Content{}, } discovered := model.Discovered{Socials: map[string]string{}} + if client, ok := s.get.(*fetch.Client); ok && (opts.Stealth || env("REM_STEALTH") == "1") { + client.SetStealth(true) + } maxCredits := opts.MaxCredits - if !opts.BudgetSet && maxCredits == 0 { + if opts.Keyless || env("REM_KEYLESS") == "1" { + maxCredits = 0 + } else if !opts.BudgetSet && maxCredits == 0 { maxCredits = 50 } paid := providers.NewPaid(s.http, env("SCRAPECREATORS_API_KEY", "SCRAPE_CREATORS_API_KEY"), maxCredits) @@ -186,7 +193,9 @@ func (s *Service) Investigate(ctx context.Context, input string, opts Options) ( {"seo", "seo", func() model.Result { return collect.SEO(ctx, s.get, target) }}, {"history", "wayback", func() model.Result { return providers.History(ctx, s.get, target) }}, {"apps", "apple_ads", func() model.Result { return providers.AppleAds(ctx, s.get, target, discovered) }}, - {"apps", "hackernews", func() model.Result { return providers.Community(ctx, s.get, target) }}, + {"community", "hackernews", func() model.Result { return providers.Community(ctx, s.get, target) }}, + {"community", "reddit", func() model.Result { return providers.RedditPublic(ctx, s.get, target) }}, + {"community", "github", func() model.Result { return providers.GitHubPublic(ctx, s.get, target, discovered) }}, } if !storeTarget && wanted("apps") { appendStage(run("appstore", func() model.Result { @@ -198,7 +207,11 @@ func (s *Service) Investigate(ctx context.Context, input string, opts Options) ( slots := make([]stageResult, len(jobs)) var wg sync.WaitGroup for i, j := range jobs { - if wanted(j.area) && (target.URL != "" || j.name == "apple_ads") { + runJob := wanted(j.area) + if j.name == "hackernews" && wanted("apps") { + runJob = true + } + if runJob && (target.URL != "" || j.name == "apple_ads") { wg.Go(func() { slots[i] = run(j.name, j.call) }) } } @@ -234,6 +247,7 @@ func (s *Service) Investigate(ctx context.Context, input string, opts Options) ( b.Identity.Logo = discovered.Logos[0].URL } b.Scorecard = report.Score(b) + b.Entrepreneur = report.AnalyzeEntrepreneur(b) progress("report", "start") md, err := s.WriteReport(ctx, b, opts.Engine, opts.Model) if err != nil { diff --git a/internal/app/service_test.go b/internal/app/service_test.go index 991fa46..dca00ff 100644 --- a/internal/app/service_test.go +++ b/internal/app/service_test.go @@ -47,3 +47,47 @@ func TestCanceledInvestigationDoesNotPublish(t *testing.T) { t.Fatal(rows, err) } } + +func TestInvestigationPopulatesEntrepreneurAnalysis(t *testing.T) { + s := New(store.New(t.TempDir())) + s.get = getterFunc(func(_ context.Context, u string) (fetch.Response, error) { + return fetch.Response{ + URL: u, + Status: 200, + Body: []byte(`Acme — Fast Modern Issue Tracker + +

Tired of slow, bloated project management software?

+

Our engine is 10x faster than legacy tools. Over 50,000 teams trust Acme.

+ Get started free — no credit card required +`), + }, nil + }) + + b, err := s.Investigate(context.Background(), "acme.com", Options{Only: []string{"website"}, Engine: "evidence"}) + if err != nil { + t.Fatal(err) + } + + if len(b.Entrepreneur.CreativeAngles) == 0 { + t.Fatal("expected b.Entrepreneur.CreativeAngles to be populated, got 0") + } + + foundAgitation := false + for _, angle := range b.Entrepreneur.CreativeAngles { + if angle.Category == "agitation" { + foundAgitation = true + if len(angle.EvidenceIDs) == 0 { + t.Errorf("expected evidence ID citation in creative angle") + } + } + } + if !foundAgitation { + t.Errorf("expected agitation angle from 'Tired of slow, bloated...'") + } + + doc := s.Doctor() + freeSources, ok := doc["freeSources"].([]string) + if !ok || len(freeSources) < 13 { + t.Fatalf("unexpected freeSources in doctor: %#v", doc["freeSources"]) + } +} diff --git a/internal/collect/collect_test.go b/internal/collect/collect_test.go index adf097a..3053398 100644 --- a/internal/collect/collect_test.go +++ b/internal/collect/collect_test.go @@ -175,3 +175,14 @@ fbq("init","1234567890");`, "page") } } } + +func TestWebsiteCollectsCareersAndHiringSignals(t *testing.T) { + web := fixture{ + "https://linear.example": page(`

Issue tracking

CareersPricing`), + "https://linear.example/pricing": page(`

Plans

$12/user/mo billed annually. 30-day money-back guarantee. No credit card required. SOC 2 compliant.

`), + "https://linear.example/careers": page(`

Join Linear

We are looking for a Head of Growth, a Performance Marketing Manager, and a Senior SEO Specialist.

`), + } + _, _, result := Website(context.Background(), web, model.Target{Kind: "website", URL: "https://linear.example", Domain: "linear.example"}) + assertEvidenceData(t, result, "Hiring signals:", `"growthRoles"`, `"Head of Growth"`) + assertEvidenceData(t, result, "Monetization:", `"noCreditCard":true`, `"moneyBack":true`, `"annualDiscount":true`, `"soc2Certified":true`) +} diff --git a/internal/collect/creative.go b/internal/collect/creative.go new file mode 100644 index 0000000..af36b4f --- /dev/null +++ b/internal/collect/creative.go @@ -0,0 +1,148 @@ +package collect + +import ( + "regexp" + "strings" + + "github.com/funnelscript/rem-from-funnelscript/internal/model" +) + +var ( + agitationPattern = regexp.MustCompile(`(?i)\b(tired of|struggling|waste|wasting|stop doing|frustrated|painful|broken|hard way|nightmare|hate|mess|chaos)\b`) + socialProofPattern = regexp.MustCompile(`(?i)\b(trusted by|used by|teams switch|rated 5 stars|reviews|customers love|featured in|over \d+|join \d+)\b`) + shockStatPattern = regexp.MustCompile(`(?i)\b(\d+x|\d+%\s*(faster|reduction|increase|more|less)|\$\d+[km]?\+?|\d+\s*(hours|days)\s*saved)\b`) + founderPattern = regexp.MustCompile(`(?i)\b(why we built|i started|our mission|bootstrapped|behind the scenes|my journey|we created)\b`) + usVsThemPattern = regexp.MustCompile(`(?i)\b(alternative to|vs\b|better than|switch from|replace|compared to|unlike)\b`) + directOfferPattern = regexp.MustCompile(`(?i)\b(free trial|get started free|discount|special offer|no credit card|30-day|money-back|free forever)\b`) +) + +// DeconstructCreative decomposes collected content items and website positioning into REA-structured creative angles. +func DeconstructCreative(contents []model.Content, evidences []model.Evidence) []model.CreativeAngle { + var angles []model.CreativeAngle + + // 1. Deconstruct paid & organic content items + for _, c := range contents { + text := strings.TrimSpace(c.Text) + if text == "" { + text = strings.TrimSpace(c.Title) + } + if text == "" { + continue + } + + angle := classifyAngle(text, c.Kind, c.Evidence) + if angle.Hook != "" { + angles = append(angles, angle) + } + } + + // 2. Deconstruct key website positioning evidence if content items are sparse + for _, e := range evidences { + if e.Source == "website" && e.Kind == "observation" { + if data, ok := e.Data.(map[string]any); ok { + var headlines []string + if title, ok := data["title"].(string); ok && title != "" { + headlines = append(headlines, title) + } + if h1List, ok := data["h1"].([]string); ok { + headlines = append(headlines, h1List...) + } + for _, h := range headlines { + if angle := classifyAngle(h, "website_headline", e.ID); angle.Hook != "" { + angles = append(angles, angle) + } + } + } + } + } + + return deduplicateAngles(angles) +} + +func classifyAngle(text, format, evidenceID string) model.CreativeAngle { + text = strings.TrimSpace(text) + lines := strings.Split(text, "\n") + hook := strings.TrimSpace(lines[0]) + if len([]rune(hook)) > 140 { + hook = string([]rune(hook)[:137]) + "..." + } + + category := "general_value" + switch { + case agitationPattern.MatchString(text): + category = "agitation" + case usVsThemPattern.MatchString(text): + category = "us_vs_them" + case shockStatPattern.MatchString(text): + category = "shock_stat" + case socialProofPattern.MatchString(text): + category = "social_proof" + case founderPattern.MatchString(text): + category = "founder_story" + case directOfferPattern.MatchString(text): + category = "direct_offer" + } + + var mechanism string + if len(lines) > 1 { + mechanism = strings.TrimSpace(lines[1]) + } else { + mechanism = "Direct solution proposition" + } + + var proof string + if match := socialProofPattern.FindString(text); match != "" { + proof = match + } else if match := shockStatPattern.FindString(text); match != "" { + proof = match + } + + var offerCTA string + if match := directOfferPattern.FindString(text); match != "" { + offerCTA = match + } + + var evIDs []string + if evidenceID != "" { + evIDs = append(evIDs, evidenceID) + } + + return model.CreativeAngle{ + Category: category, + Hook: hook, + Mechanism: mechanism, + Proof: proof, + OfferCTA: offerCTA, + Format: normalizeFormat(format), + EvidenceIDs: evIDs, + } +} + +func normalizeFormat(fmtStr string) string { + f := strings.ToLower(fmtStr) + switch { + case strings.Contains(f, "video") || strings.Contains(f, "reel") || strings.Contains(f, "tiktok"): + return "video" + case strings.Contains(f, "carousel"): + return "carousel" + case strings.Contains(f, "image"): + return "static_image" + case strings.Contains(f, "headline"): + return "landing_page_hero" + default: + return "copy_post" + } +} + +func deduplicateAngles(angles []model.CreativeAngle) []model.CreativeAngle { + seen := make(map[string]bool) + var result []model.CreativeAngle + for _, a := range angles { + key := a.Category + ":" + strings.ToLower(a.Hook) + if !seen[key] { + seen[key] = true + result = append(result, a) + } + } + return result +} diff --git a/internal/collect/creative_test.go b/internal/collect/creative_test.go new file mode 100644 index 0000000..82e61cf --- /dev/null +++ b/internal/collect/creative_test.go @@ -0,0 +1,84 @@ +package collect + +import ( + "testing" + + "github.com/funnelscript/rem-from-funnelscript/internal/model" +) + +func TestDeconstructCreativeClassifiesAngles(t *testing.T) { + contents := []model.Content{ + { + ID: "c1", + Kind: "video", + Text: "Tired of wasting hours managing spreadsheets?\nOur AI automates your workflow in 1 click.\nOver 10,000 teams switch every month.\nGet started free.", + Evidence: "E1", + }, + { + ID: "c2", + Kind: "image", + Text: "The best alternative to Jira.\nFast, keyboard-first issue tracking.\n10x faster response time.\nTry free forever.", + Evidence: "E2", + }, + { + ID: "c3", + Kind: "post", + Text: "Why we built Linear:\nWe believe software development should feel magical, not bogged down by enterprise bureaucracy.", + Evidence: "E3", + }, + } + + evidences := []model.Evidence{ + { + ID: "E4", + Source: "website", + Kind: "observation", + Data: map[string]any{ + "title": "Linear — A better way to build products", + "h1": []string{"Issue tracking designed for speed"}, + }, + }, + } + + angles := DeconstructCreative(contents, evidences) + if len(angles) < 3 { + t.Fatalf("expected at least 3 creative angles, got %d", len(angles)) + } + + // Verify categories + categories := make(map[string]bool) + for _, a := range angles { + categories[a.Category] = true + } + + if !categories["agitation"] { + t.Errorf("expected agitation angle from c1") + } + if !categories["us_vs_them"] { + t.Errorf("expected us_vs_them angle from c2") + } + if !categories["founder_story"] { + t.Errorf("expected founder_story angle from c3") + } + + // Check REA structure on c1 + var c1Angle *model.CreativeAngle + for i := range angles { + if angles[i].Category == "agitation" { + c1Angle = &angles[i] + break + } + } + if c1Angle == nil { + t.Fatal("c1 agitation angle not found") + } + if c1Angle.Format != "video" { + t.Errorf("expected video format, got %q", c1Angle.Format) + } + if c1Angle.Proof == "" { + t.Errorf("expected proof extraction from c1") + } + if c1Angle.OfferCTA == "" { + t.Errorf("expected offer CTA extraction from c1") + } +} diff --git a/internal/collect/website.go b/internal/collect/website.go index de4115f..ff7d88a 100644 --- a/internal/collect/website.go +++ b/internal/collect/website.go @@ -90,7 +90,7 @@ func Website(ctx context.Context, get fetch.Getter, target model.Target) (model. response fetch.Response err error } - kinds := [...]string{"pricing", "about", "affiliate", "demo"} + kinds := [...]string{"pricing", "about", "affiliate", "demo", "careers", "customers"} checked := make([]keyPageResult, len(kinds)) var pending sync.WaitGroup for index, kind := range kinds { @@ -135,6 +135,18 @@ func Website(ctx context.Context, get fetch.Getter, target model.Target) (model. result.Evidence = append(result.Evidence, evidence("website", "observation", "Affiliate / partner program page exists", resolved.String(), data)) case "demo": result.Evidence = append(result.Evidence, evidence("website", "observation", "Demo / contact-sales page exists", resolved.String(), data)) + case "careers": + data["excerpt"] = clip(page.Text, 1200) + matchedRoles := growthRolePattern.FindAllString(page.Text, 20) + if len(matchedRoles) > 0 { + data["growthRoles"] = matchedRoles + result.Evidence = append(result.Evidence, evidence("website", "observation", fmt.Sprintf("Hiring signals: %d marketing/growth roles on careers page", len(matchedRoles)), resolved.String(), data)) + } else { + result.Evidence = append(result.Evidence, evidence("website", "observation", "Careers page exists (no active growth roles detected)", resolved.String(), data)) + } + case "customers": + data["excerpt"] = clip(page.Text, 1000) + result.Evidence = append(result.Evidence, evidence("website", "observation", "Customer stories and social proof on /customers", resolved.String(), data)) } } pricing := pricingSignals(pricingText) @@ -297,16 +309,23 @@ func campaignLinks(links []*url.URL) []map[string]string { return tags } -var pricingPatterns = map[string]*regexp.Regexp{ - "freePlan": regexp.MustCompile(`(?i)\bfree (?:plan|forever|tier)\b|\$0\b`), - "freeTrial": regexp.MustCompile(`(?i)\bfree trial\b|try (?:it )?free|start free`), - "moneyBack": regexp.MustCompile(`(?i)money[- ]back|refund guarantee`), - "contactSales": regexp.MustCompile(`(?i)contact sales|talk to sales|book a demo|get a demo|request a demo`), - "perSeat": regexp.MustCompile(`(?i)per (?:user|seat|member)|/\s?(?:user|seat)`), - "annualDiscount": regexp.MustCompile(`(?i)billed annually|save \d{1,2}%|annual(?:ly)? .{0,20}(?:save|off|discount)`), - "lifetimeDeal": regexp.MustCompile(`(?i)lifetime (?:deal|access|plan)|one[- ]time payment|pay once`), - "usageBased": regexp.MustCompile(`(?i)pay as you go|usage[- ]based|per (?:credit|request|minute|token|call)`), -} +var ( + pricingPatterns = map[string]*regexp.Regexp{ + "freePlan": regexp.MustCompile(`(?i)\bfree (?:plan|forever|tier)\b|\$0\b`), + "freeTrial": regexp.MustCompile(`(?i)\bfree trial\b|try (?:it )?free|start free`), + "moneyBack": regexp.MustCompile(`(?i)money[- ]back|refund guarantee`), + "contactSales": regexp.MustCompile(`(?i)contact sales|talk to sales|book a demo|get a demo|request a demo`), + "perSeat": regexp.MustCompile(`(?i)per (?:user|seat|member)|/\s?(?:user|seat)`), + "annualDiscount": regexp.MustCompile(`(?i)billed annually|save \d{1,2}%|annual(?:ly)? .{0,20}(?:save|off|discount)`), + "lifetimeDeal": regexp.MustCompile(`(?i)lifetime (?:deal|access|plan)|one[- ]time payment|pay once`), + "usageBased": regexp.MustCompile(`(?i)pay as you go|usage[- ]based|per (?:credit|request|minute|token|call)`), + "noCreditCard": regexp.MustCompile(`(?i)no credit card (?:required|needed)`), + "cancelAnytime": regexp.MustCompile(`(?i)cancel (?:at )?anytime`), + "soc2Certified": regexp.MustCompile(`(?i)\bsoc[- ]?2\b`), + "enterpriseSecurity": regexp.MustCompile(`(?i)\b(?:iso[- ]?27001|gdpr|hipaa|sso|saml)\b`), + } + growthRolePattern = regexp.MustCompile(`(?i)\b(?:growth marketer|growth marketing|head of growth|performance marketing|paid media|seo specialist|seo manager|demand generation|lifecycle marketer|content marketing manager|cro specialist)\b`) +) func pricingSignals(text string) map[string]any { prices := make([]string, 0) @@ -348,6 +367,15 @@ func pricingLabel(signals map[string]any) string { if signals["lifetimeDeal"] == true { summary = append(summary, "one-time payment") } + if signals["noCreditCard"] == true { + summary = append(summary, "no credit card required") + } + if signals["moneyBack"] == true { + summary = append(summary, "money-back guarantee") + } + if signals["annualDiscount"] == true { + summary = append(summary, "annual discount") + } if signals["contactSales"] == true { summary = append(summary, "sales-assisted CTA") } diff --git a/internal/fetch/http.go b/internal/fetch/http.go index f51d20d..a2741dc 100644 --- a/internal/fetch/http.go +++ b/internal/fetch/http.go @@ -9,6 +9,7 @@ import ( "net/http" "net/netip" "net/url" + "slices" "strings" "time" ) @@ -30,6 +31,11 @@ var ( type Client struct { HTTP *http.Client allowTest bool + stealth bool +} + +func (c *Client) SetStealth(stealth bool) { + c.stealth = stealth } func New() *Client { @@ -49,6 +55,15 @@ func New() *Client { return nil, ErrDestination } } + slices.SortStableFunc(ips, func(a, b netip.Addr) int { + if a.Is4() && !b.Is4() { + return -1 + } + if !a.Is4() && b.Is4() { + return 1 + } + return 0 + }) var last error = ErrDestination for _, ip := range ips { conn, err := dialer.DialContext(ctx, network, net.JoinHostPort(ip.String(), port)) @@ -60,9 +75,9 @@ func New() *Client { return nil, last }, ForceAttemptHTTP2: true, MaxIdleConns: 24, MaxIdleConnsPerHost: 4, MaxConnsPerHost: 6, - IdleConnTimeout: 30 * time.Second, TLSHandshakeTimeout: 6 * time.Second, ResponseHeaderTimeout: 9 * time.Second, + IdleConnTimeout: 30 * time.Second, TLSHandshakeTimeout: 8 * time.Second, ResponseHeaderTimeout: 12 * time.Second, } - return &Client{HTTP: &http.Client{Transport: transport, Timeout: 14 * time.Second, CheckRedirect: func(req *http.Request, via []*http.Request) error { + return &Client{HTTP: &http.Client{Transport: transport, Timeout: 18 * time.Second, CheckRedirect: func(req *http.Request, via []*http.Request) error { if len(via) >= 5 { return errors.New("redirect limit reached") } @@ -123,18 +138,35 @@ func (c *Client) Get(ctx context.Context, rawURL string) (Response, error) { if err != nil { return Response{}, err } - req.Header.Set("User-Agent", "REM-Go/0.1 (public marketing research)") - req.Header.Set("Accept", "text/html, application/json, application/xml, text/plain;q=0.9, */*;q=0.3") + if c.stealth { + req.Header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36") + req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7") + req.Header.Set("Accept-Language", "en-US,en;q=0.9") + req.Header.Set("Sec-CH-UA", `"Chromium";v="134", "Google Chrome";v="134", "Not:A-Brand";v="24"`) + req.Header.Set("Sec-CH-UA-Mobile", "?0") + req.Header.Set("Sec-CH-UA-Platform", `"macOS"`) + req.Header.Set("Sec-Fetch-Dest", "document") + req.Header.Set("Sec-Fetch-Mode", "navigate") + req.Header.Set("Sec-Fetch-Site", "cross-site") + req.Header.Set("Sec-Fetch-User", "?1") + req.Header.Set("Upgrade-Insecure-Requests", "1") + if req.Header.Get("Referer") == "" && u.Scheme == "https" { + req.Header.Set("Referer", "https://www.google.com/") + } + } else { + req.Header.Set("User-Agent", "REM-Go/0.1 (public marketing research)") + req.Header.Set("Accept", "text/html, application/json, application/xml, text/plain;q=0.9, */*;q=0.3") + } res, err := c.HTTP.Do(req) if err != nil { return Response{}, err } - if attempt == 0 && (res.StatusCode == 502 || res.StatusCode == 503 || res.StatusCode == 504) { + if attempt == 0 && (res.StatusCode == 429 || res.StatusCode == 502 || res.StatusCode == 503 || res.StatusCode == 504) { res.Body.Close() select { case <-ctx.Done(): return Response{}, ctx.Err() - case <-time.After(200 * time.Millisecond): + case <-time.After(250 * time.Millisecond): } continue } diff --git a/internal/fetch/http_test.go b/internal/fetch/http_test.go index a942dc0..b49dfe0 100644 --- a/internal/fetch/http_test.go +++ b/internal/fetch/http_test.go @@ -42,3 +42,24 @@ func TestBodyBoundAndRetry(t *testing.T) { t.Fatal(err) } } + +func TestStealthHeaders(t *testing.T) { + var gotUA, gotSecUA, gotReferer string + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotUA = r.Header.Get("User-Agent") + gotSecUA = r.Header.Get("Sec-CH-UA") + gotReferer = r.Header.Get("Referer") + fmt.Fprint(w, "ok") + })) + defer server.Close() + + c := &Client{HTTP: server.Client(), allowTest: true} + c.SetStealth(true) + if _, err := c.Get(context.Background(), server.URL); err != nil { + t.Fatal(err) + } + if !strings.Contains(gotUA, "Chrome/") || !strings.Contains(gotSecUA, "Google Chrome") { + t.Errorf("expected stealth chrome headers, got UA=%q SecUA=%q", gotUA, gotSecUA) + } + _ = gotReferer +} diff --git a/internal/mcp/definitions.go b/internal/mcp/definitions.go index 576496d..b9f598f 100644 --- a/internal/mcp/definitions.go +++ b/internal/mcp/definitions.go @@ -52,6 +52,9 @@ var tools = []toolDefinition{ "only": {"type": "array", "description": "Areas to investigate. Website collection always runs when a website is available.", "items": stringEnum("Marketing area", app.Areas)}, "handles": {"type": "object", "description": "Override discovered handles, such as x or tiktok.", "additionalProperties": schema{"type": "string"}}, "max_credits": integerField("Maximum paid Scrape Creators credits for this run. Set to zero to skip paid requests.", 0, 500), + "stealth": {"type": "boolean", "description": "Enable Scrapling-style browser TLS/HTTP-2 fingerprinting & anti-bot bypass."}, + "keyless": {"type": "boolean", "description": "Rely entirely on zero-cost extractors without consuming paid credits."}, + "window": stringField("Recency filter window (e.g. 30d, 90d; default: 30d)."), }, "url"), }, { diff --git a/internal/mcp/server.go b/internal/mcp/server.go index 53271d5..91176f4 100644 --- a/internal/mcp/server.go +++ b/internal/mcp/server.go @@ -122,6 +122,9 @@ func (m marketing) investigate(ctx context.Context, raw json.RawMessage) (string Only []string `json:"only"` Handles map[string]string `json:"handles"` MaxCredits *int `json:"max_credits"` + Stealth *bool `json:"stealth"` + Keyless *bool `json:"keyless"` + Window string `json:"window"` } if err := decodeObject(raw, &args); err != nil { return "", err @@ -134,7 +137,13 @@ func (m marketing) investigate(ctx context.Context, raw json.RawMessage) (string return "", fmt.Errorf("unknown area %q", area) } } - opts := app.Options{Only: args.Only, Handles: args.Handles, Engine: "evidence"} + opts := app.Options{Only: args.Only, Handles: args.Handles, Engine: "evidence", Window: args.Window} + if args.Stealth != nil && *args.Stealth { + opts.Stealth = true + } + if args.Keyless != nil && *args.Keyless { + opts.Keyless = true + } if args.MaxCredits != nil { if *args.MaxCredits < 0 || *args.MaxCredits > 500 { return "", errors.New("max_credits must be between 0 and 500") @@ -166,6 +175,26 @@ func investigationSummary(bundle model.Bundle) string { } summary.WriteByte('\n') } + if len(bundle.Entrepreneur.CreativeAngles) > 0 { + fmt.Fprintf(&summary, "\n## Creative angles (REA): %d identified\n", len(bundle.Entrepreneur.CreativeAngles)) + for _, a := range bundle.Entrepreneur.CreativeAngles { + fmt.Fprintf(&summary, "- [%s] %s: %s", report.Clean(a.Category), report.Clean(a.Format), report.Clean(oneLine(a.Hook))) + for _, id := range a.EvidenceIDs { + fmt.Fprintf(&summary, " [%s]", id) + } + summary.WriteByte('\n') + } + } + if len(bundle.Entrepreneur.Displacement) > 0 { + fmt.Fprintf(&summary, "\n## Day-1 Displacement plays (Replica-Entrepreneur): %d plays\n", len(bundle.Entrepreneur.Displacement)) + for _, p := range bundle.Entrepreneur.Displacement { + fmt.Fprintf(&summary, "- %s -> Counter: %s (%s)", report.Clean(oneLine(p.TargetWeakness)), report.Clean(oneLine(p.CounterHook)), report.Clean(p.ImpactROI)) + for _, id := range p.EvidenceIDs { + fmt.Fprintf(&summary, " [%s]", id) + } + summary.WriteByte('\n') + } + } summary.WriteString("\n## Evidence index\n") for _, item := range bundle.Evidence { fmt.Fprintf(&summary, "- [%s] (%s) %s", item.ID, report.Clean(item.Source), report.Clean(oneLine(item.Title))) diff --git a/internal/model/model.go b/internal/model/model.go index cb1faad..387701e 100644 --- a/internal/model/model.go +++ b/internal/model/model.go @@ -93,6 +93,40 @@ type Identity struct { Founders []Founder `json:"founders"` } +type CreativeAngle struct { + Category string `json:"category"` + Hook string `json:"hook"` + Mechanism string `json:"mechanism"` + Proof string `json:"proof"` + OfferCTA string `json:"offerCta"` + Format string `json:"format"` + EvidenceIDs []string `json:"evidenceIds"` +} + +type Grievance struct { + Theme string `json:"theme"` + Score float64 `json:"score"` + EvidenceCount int `json:"evidenceCount"` + SampleQuotes []string `json:"sampleQuotes"` + Sources []string `json:"sources"` + EvidenceIDs []string `json:"evidenceIds"` +} + +type DisplacementPlay struct { + TargetWeakness string `json:"targetWeakness"` + CounterHook string `json:"counterHook"` + AnglePlaybook string `json:"anglePlaybook"` + RecommendedCTA string `json:"recommendedCta"` + ImpactROI string `json:"impactRoi"` + EvidenceIDs []string `json:"evidenceIds"` +} + +type EntrepreneurAnalysis struct { + TopGrievances []Grievance `json:"topGrievances,omitempty"` + CreativeAngles []CreativeAngle `json:"creativeAngles,omitempty"` + Displacement []DisplacementPlay `json:"displacement,omitempty"` +} + type Metadata struct { Name string `json:"name"` Version string `json:"version"` @@ -108,21 +142,22 @@ type Credits struct { } type Bundle struct { - ID string `json:"id,omitempty"` - Tool Metadata `json:"tool"` - Harness Harness `json:"harness"` - Target Target `json:"target"` - GeneratedAt string `json:"generatedAt"` - Brand string `json:"brand"` - Discovered Discovered `json:"discovered"` - Scorecard []ChannelScore `json:"scorecard"` - Evidence []Evidence `json:"evidence"` - Unknowns []Unknown `json:"unknowns"` - Sources []SourceRun `json:"sources"` - Credits Credits `json:"credits"` - Content []Content `json:"content"` - Identity Identity `json:"identity"` - ReportWarning string `json:"reportWarning,omitempty"` + ID string `json:"id,omitempty"` + Tool Metadata `json:"tool"` + Harness Harness `json:"harness"` + Target Target `json:"target"` + GeneratedAt string `json:"generatedAt"` + Brand string `json:"brand"` + Discovered Discovered `json:"discovered"` + Scorecard []ChannelScore `json:"scorecard"` + Evidence []Evidence `json:"evidence"` + Unknowns []Unknown `json:"unknowns"` + Sources []SourceRun `json:"sources"` + Credits Credits `json:"credits"` + Content []Content `json:"content"` + Identity Identity `json:"identity"` + Entrepreneur EntrepreneurAnalysis `json:"entrepreneur,omitempty"` + ReportWarning string `json:"reportWarning,omitempty"` } type Summary struct { diff --git a/internal/providers/free_community.go b/internal/providers/free_community.go index 3048eca..5239069 100644 --- a/internal/providers/free_community.go +++ b/internal/providers/free_community.go @@ -8,6 +8,7 @@ import ( "regexp" "slices" "strings" + "time" "github.com/funnelscript/rem-from-funnelscript/internal/fetch" "github.com/funnelscript/rem-from-funnelscript/internal/model" @@ -87,3 +88,166 @@ func Community(ctx context.Context, get fetch.Getter, target model.Target) model appendObservation(&result, "hackernews", title, manual, stories) return result } + +type redditPost struct { + ID string `json:"id"` + Title string `json:"title"` + Subreddit string `json:"subreddit"` + Score int `json:"score"` + Comments int `json:"num_comments"` + Permalink string `json:"permalink"` + URL string `json:"url"` + Created float64 `json:"created_utc"` +} + +type redditListing struct { + Data struct { + Children []struct { + Data redditPost `json:"data"` + } `json:"children"` + } `json:"data"` +} + +// RedditPublic queries keyless public search for community discussions about the target. +func RedditPublic(ctx context.Context, get fetch.Getter, target model.Target) model.Result { + var result model.Result + queryTarget := target.Domain + if queryTarget == "" { + queryTarget = target.Brand + } + if queryTarget == "" { + appendUnknown(&result, "reddit", "No domain or brand available for Reddit search", "https://www.reddit.com/search/") + return result + } + searchURL := "https://www.reddit.com/search.json?q=" + url.QueryEscape(queryTarget) + "&sort=top&t=year&limit=25" + manual := "https://www.reddit.com/search/?q=" + url.QueryEscape(queryTarget) + + var listing redditListing + if err := freeJSON(ctx, get, searchURL, &listing); err != nil { + appendUnknown(&result, "reddit", "Reddit public search unavailable: "+err.Error(), manual) + return result + } + + threads := make([]map[string]any, 0, len(listing.Data.Children)) + totalScore := 0 + subreddits := make(map[string]int) + + for _, child := range listing.Data.Children { + p := child.Data + if p.Title == "" { + continue + } + totalScore += p.Score + subreddits[p.Subreddit]++ + if len(threads) < 10 { + permalink := p.Permalink + if !strings.HasPrefix(permalink, "http") { + permalink = "https://www.reddit.com" + permalink + } + threads = append(threads, map[string]any{ + "title": brief(p.Title, 180), + "subreddit": p.Subreddit, + "score": p.Score, + "comments": p.Comments, + "url": permalink, + "created": time.Unix(int64(p.Created), 0).UTC().Format("2006-01-02"), + }) + } + } + + if len(threads) == 0 { + appendObservation(&result, "reddit", "Reddit returned no matching threads in top search results", manual, map[string]any{ + "threads": []any{}, "matchedThreads": 0, "totalScore": 0, + }) + return result + } + + topSubs := make([]string, 0, len(subreddits)) + for sub := range subreddits { + topSubs = append(topSubs, "r/"+sub) + } + slices.Sort(topSubs) + + title := fmt.Sprintf("Reddit: %d public threads discussing %s, total score %d across %d subreddits", len(threads), queryTarget, totalScore, len(topSubs)) + appendObservation(&result, "reddit", title, manual, map[string]any{ + "threads": threads, + "matchedThreads": len(threads), + "totalScore": totalScore, + "subreddits": topSubs, + }) + return result +} + +type gitHubRepo struct { + Name string `json:"name"` + FullName string `json:"full_name"` + Description string `json:"description"` + Stars int `json:"stargazers_count"` + Forks int `json:"forks_count"` + OpenIssues int `json:"open_issues_count"` + Topics []string `json:"topics"` + PushedAt string `json:"pushed_at"` + License *struct { + SPDX string `json:"spdx_id"` + } `json:"license"` +} + +// GitHubPublic extracts public repository intelligence without requiring API tokens. +func GitHubPublic(ctx context.Context, get fetch.Getter, target model.Target, discovered model.Discovered) model.Result { + var result model.Result + gh := discovered.Socials["github"] + if gh == "" { + return result + } + gh = strings.TrimPrefix(gh, "@") + parts := strings.Split(strings.Trim(gh, "/"), "/") + if len(parts) == 0 || parts[0] == "" { + return result + } + + var repoURL, apiURL string + if len(parts) >= 2 { + repoURL = "https://github.com/" + parts[0] + "/" + parts[1] + apiURL = "https://api.github.com/repos/" + parts[0] + "/" + parts[1] + } else { + repoURL = "https://github.com/" + parts[0] + apiURL = "https://api.github.com/users/" + parts[0] + "/repos?sort=updated&per_page=1" + } + + if len(parts) >= 2 { + var repo gitHubRepo + if err := freeJSON(ctx, get, apiURL, &repo); err != nil { + appendUnknown(&result, "community_launches", "GitHub public API unavailable: "+err.Error(), repoURL) + return result + } + license := "" + if repo.License != nil { + license = repo.License.SPDX + } + title := fmt.Sprintf("GitHub: %s with %d stars, %d forks", repo.FullName, repo.Stars, repo.Forks) + appendObservation(&result, "github", title, repoURL, map[string]any{ + "fullName": repo.FullName, + "stars": repo.Stars, + "forks": repo.Forks, + "openIssues": repo.OpenIssues, + "topics": repo.Topics, + "license": license, + "pushedAt": repo.PushedAt, + }) + } else { + var repos []gitHubRepo + if err := freeJSON(ctx, get, apiURL, &repos); err != nil || len(repos) == 0 { + appendUnknown(&result, "community_launches", "GitHub public org/user info: no public repos found", repoURL) + return result + } + repo := repos[0] + title := fmt.Sprintf("GitHub: %s active repo %s with %d stars", parts[0], repo.Name, repo.Stars) + appendObservation(&result, "github", title, repoURL, map[string]any{ + "fullName": repo.FullName, + "stars": repo.Stars, + "forks": repo.Forks, + "pushedAt": repo.PushedAt, + }) + } + return result +} diff --git a/internal/providers/free_test.go b/internal/providers/free_test.go index 4c9e90d..554c0b5 100644 --- a/internal/providers/free_test.go +++ b/internal/providers/free_test.go @@ -179,3 +179,76 @@ func TestAppleStoreLookupUsesStorefrontFromTarget(t *testing.T) { }) _, _, _ = Apps(t.Context(), get, model.Target{Input: "https://apps.apple.com/gb/app/name/id123456", Kind: "app_store", AppleID: "123456"}, model.Discovered{}) } + +func TestRedditPublicParsesThreadsAndSubreddits(t *testing.T) { + get := freeFixture(func(u *url.URL) (int, string, error) { + if !strings.Contains(u.Host, "reddit.com") || u.Path != "/search.json" { + t.Fatalf("unexpected URL: %s", u) + } + return 200, `{ + "data": { + "children": [ + { + "data": { + "id": "t1", + "title": "Why we switched to Linear", + "subreddit": "programming", + "score": 340, + "num_comments": 85, + "permalink": "/r/programming/comments/t1/linear", + "created_utc": 1710000000 + } + } + ] + } + }`, nil + }) + + result := RedditPublic(t.Context(), get, model.Target{Domain: "linear.app"}) + if len(result.Evidence) != 1 { + t.Fatalf("expected 1 evidence item, got %d", len(result.Evidence)) + } + ev := result.Evidence[0] + if ev.Source != "reddit" { + t.Errorf("expected source reddit, got %s", ev.Source) + } + data := ev.Data.(map[string]any) + if data["matchedThreads"] != 1 || data["totalScore"] != 340 { + t.Errorf("unexpected data %+v", data) + } +} + +func TestGitHubPublicParsesRepoMetrics(t *testing.T) { + get := freeFixture(func(u *url.URL) (int, string, error) { + if !strings.Contains(u.Host, "github.com") || !strings.Contains(u.Path, "/repos/owner/repo") { + t.Fatalf("unexpected URL: %s", u) + } + return 200, `{ + "name": "repo", + "full_name": "owner/repo", + "description": "An open source engine", + "stargazers_count": 5200, + "forks_count": 410, + "open_issues_count": 25, + "topics": ["marketing", "growth"], + "pushed_at": "2026-11-10T12:00:00Z", + "license": {"spdx_id": "MIT"} + }`, nil + }) + + discovered := model.Discovered{ + Socials: map[string]string{"github": "owner/repo"}, + } + result := GitHubPublic(t.Context(), get, model.Target{Domain: "example.com"}, discovered) + if len(result.Evidence) != 1 { + t.Fatalf("expected 1 evidence item, got %d", len(result.Evidence)) + } + ev := result.Evidence[0] + if ev.Source != "github" { + t.Errorf("expected source github, got %s", ev.Source) + } + data := ev.Data.(map[string]any) + if data["stars"] != 5200 || data["forks"] != 410 || data["license"] != "MIT" { + t.Errorf("unexpected data %+v", data) + } +} diff --git a/internal/report/entrepreneur.go b/internal/report/entrepreneur.go new file mode 100644 index 0000000..54c99c2 --- /dev/null +++ b/internal/report/entrepreneur.go @@ -0,0 +1,224 @@ +package report + +import ( + "fmt" + "math" + "regexp" + "sort" + "strings" + "time" + + "github.com/funnelscript/rem-from-funnelscript/internal/collect" + "github.com/funnelscript/rem-from-funnelscript/internal/model" +) + +var ( + grievanceKeywords = regexp.MustCompile(`(?i)\b(slow|lag|bloat|crash|freez|expensive|price hike|pricing increase|paywall|forced|complicated|clunky|unusable|confus|broken|sync fail|data loss|buggy|bad support|awful|switched to|deal breaker|worst)\b`) + + themePatterns = map[string]*regexp.Regexp{ + "Performance & Speed": regexp.MustCompile(`(?i)\b(slow|lag|bloat|crash|freez|heavy|loading|sluggish)\b`), + "Pricing & Monetization": regexp.MustCompile(`(?i)\b(expensive|price|pricing|paywall|per seat|forced upgrade|billing|cost)\b`), + "Complexity & Clunky UI": regexp.MustCompile(`(?i)\b(complicated|clunky|unusable|confus|steep learning|mess|chaos|bloated)\b`), + "Sync & Integration Stability": regexp.MustCompile(`(?i)\b(sync|integration|api|rate limit|disconnected|data loss|webhook|broken)\b`), + "Support & Developer Trust": regexp.MustCompile(`(?i)\b(support|service|response|ignored|awful|deal breaker|cancelled|left)\b`), + } +) + +// AnalyzeEntrepreneur applies the Replica-Square Entrepreneur methodology to mine verified competitor +// grievances, score their Bayesian time-decay impact, and generate a Day-1 Challenger Displacement Matrix. +func AnalyzeEntrepreneur(b model.Bundle) model.EntrepreneurAnalysis { + var analysis model.EntrepreneurAnalysis + + // 1. Creative angles deconstruction via REA + analysis.CreativeAngles = collect.DeconstructCreative(b.Content, b.Evidence) + + // 2. Extract verified customer grievances across evidence items + now := time.Now().UTC() + if b.GeneratedAt != "" { + if t, err := time.Parse(time.RFC3339, b.GeneratedAt); err == nil { + now = t + } + } + + themeScores := make(map[string]float64) + themeQuotes := make(map[string][]string) + themeSources := make(map[string]map[string]bool) + themeEvidence := make(map[string]map[string]bool) + + for _, e := range b.Evidence { + textList := extractTextFromEvidence(e) + for _, text := range textList { + if !grievanceKeywords.MatchString(text) { + continue + } + + // Bayesian scoring with 540-day time decay + baseScore := 0.5 // Unrated forum / review default + daysDiff := 30.0 // Default 30 days if date is unspecified + if e.ObservedAt != "" { + if obsTime, err := time.Parse("2006-01-02", e.ObservedAt[:min(10, len(e.ObservedAt))]); err == nil { + days := now.Sub(obsTime).Hours() / 24 + if days > 0 { + daysDiff = days + } + } + } + + timeWeight := math.Pow(0.5, daysDiff/540.0) + score := baseScore * timeWeight + + // Categorize into themes + for theme, pat := range themePatterns { + if pat.MatchString(text) { + themeScores[theme] += score + if themeSources[theme] == nil { + themeSources[theme] = make(map[string]bool) + themeEvidence[theme] = make(map[string]bool) + } + themeSources[theme][e.Source] = true + if e.ID != "" { + themeEvidence[theme][e.ID] = true + } + + cleanQuote := cleanVerbatimQuote(text) + if cleanQuote != "" && len(themeQuotes[theme]) < 3 { + themeQuotes[theme] = append(themeQuotes[theme], cleanQuote) + } + } + } + } + } + + // 3. Build ranked Grievance list + var themes []string + for t := range themeScores { + themes = append(themes, t) + } + sort.Slice(themes, func(i, j int) bool { + return themeScores[themes[i]] > themeScores[themes[j]] + }) + + for _, t := range themes { + var srcList []string + for s := range themeSources[t] { + srcList = append(srcList, s) + } + sort.Strings(srcList) + + var evList []string + for ev := range themeEvidence[t] { + evList = append(evList, ev) + } + sort.Strings(evList) + + analysis.TopGrievances = append(analysis.TopGrievances, model.Grievance{ + Theme: t, + Score: math.Round(themeScores[t]*100) / 100, + EvidenceCount: len(evList), + SampleQuotes: themeQuotes[t], + Sources: srcList, + EvidenceIDs: evList, + }) + } + + // 4. Synthesize Day-1 Displacement Playbook + analysis.Displacement = generateDisplacementPlays(b.Brand, analysis.TopGrievances) + + return analysis +} + +func extractTextFromEvidence(e model.Evidence) []string { + var texts []string + if e.Title != "" { + texts = append(texts, e.Title) + } + + switch data := e.Data.(type) { + case string: + texts = append(texts, data) + case map[string]any: + for _, val := range data { + if s, ok := val.(string); ok && s != "" { + texts = append(texts, s) + } else if slice, ok := val.([]string); ok { + texts = append(texts, slice...) + } + } + case []any: + for _, item := range data { + if m, ok := item.(map[string]any); ok { + for _, val := range m { + if s, ok := val.(string); ok { + texts = append(texts, s) + } + } + } + } + } + return texts +} + +func cleanVerbatimQuote(raw string) string { + s := strings.TrimSpace(raw) + lines := strings.Split(s, "\n") + s = strings.TrimSpace(lines[0]) + if len([]rune(s)) > 160 { + s = string([]rune(s)[:157]) + "..." + } + return s +} + +func generateDisplacementPlays(brand string, grievances []model.Grievance) []model.DisplacementPlay { + var plays []model.DisplacementPlay + brandName := brand + if brandName == "" { + brandName = "Competitor" + } + + for _, g := range grievances { + play := model.DisplacementPlay{ + EvidenceIDs: g.EvidenceIDs, + } + + switch g.Theme { + case "Performance & Speed": + play.TargetWeakness = fmt.Sprintf("%s is weighed down by legacy bloat, causing slow load times and lag.", brandName) + play.CounterHook = fmt.Sprintf("Frustrated with %s running slow? Switch to instant, keyboard-first speed.", brandName) + play.AnglePlaybook = "Side-by-side performance comparison ad. Highlight sub-50ms interaction benchmarks vs competitor loading spinners." + play.RecommendedCTA = "Try instant web demo — no signup or credit card required." + play.ImpactROI = "High (Viral product-led conversion)" + + case "Pricing & Monetization": + play.TargetWeakness = fmt.Sprintf("%s forces complex per-seat tiers and paywalls essential features.", brandName) + play.CounterHook = fmt.Sprintf("Done with %s's surprise price hikes? Get transparent, flat pricing forever.", brandName) + play.AnglePlaybook = "Direct price comparison calculator. Show total annual savings and zero per-seat penalty." + play.RecommendedCTA = "Lock in flat founding rate with 1-click account migration." + play.ImpactROI = "High (Immediate economic switching trigger)" + + case "Complexity & Clunky UI": + play.TargetWeakness = fmt.Sprintf("%s has become bloated with enterprise complexity and steep learning curves.", brandName) + play.CounterHook = fmt.Sprintf("Software shouldn't feel like a chore. The clean, modern alternative to %s.", brandName) + play.AnglePlaybook = "Clean UI walkthrough video comparing intuitive 2-click workflows against competitor's multi-step maze." + play.RecommendedCTA = "Get started in 30 seconds with zero onboarding friction." + play.ImpactROI = "Fast-Win (Appeals to disenfranchised daily operators)" + + case "Sync & Integration Stability": + play.TargetWeakness = fmt.Sprintf("%s suffers from brittle integrations and unexpected sync failures.", brandName) + play.CounterHook = fmt.Sprintf("Tired of broken %s integrations? Bulletproof sync that never drops data.", brandName) + play.AnglePlaybook = "Reliability proof dossier. Emphasize offline-first local cache and 99.99% webhook delivery guarantees." + play.RecommendedCTA = "Free migration audit + 60-day money-back guarantee." + play.ImpactROI = "Strategic (Captures mission-critical enterprise workflows)" + + default: + play.TargetWeakness = fmt.Sprintf("Customer dissatisfaction with %s service responsiveness and support.", brandName) + play.CounterHook = fmt.Sprintf("Experience real human support that actually solves your problems. Meet the %s alternative.", brandName) + play.AnglePlaybook = "Social proof compilation highlighting disgruntled customer switch stories with verbatim quotes." + play.RecommendedCTA = "Book a dedicated 1-on-1 migration concierge call." + play.ImpactROI = "Medium (Trust and relationship building)" + } + + plays = append(plays, play) + } + + return plays +} diff --git a/internal/report/entrepreneur_test.go b/internal/report/entrepreneur_test.go new file mode 100644 index 0000000..7f53ac6 --- /dev/null +++ b/internal/report/entrepreneur_test.go @@ -0,0 +1,120 @@ +package report + +import ( + "testing" + "time" + + "github.com/funnelscript/rem-from-funnelscript/internal/model" +) + +func TestAnalyzeEntrepreneurExtractsRankedGrievancesAndPlays(t *testing.T) { + nowStr := time.Now().UTC().Format(time.RFC3339) + bundle := model.Bundle{ + Brand: "SlowCorp", + GeneratedAt: nowStr, + Evidence: []model.Evidence{ + { + ID: "E1", + Source: "reddit", + Kind: "community_discussion", + Title: "SlowCorp is so slow and bloated now", + ObservedAt: "2026-11-01", + Data: map[string]any{ + "comment": "Ever since the last update, SlowCorp is painfully slow and crashes on large files. Deal breaker.", + }, + }, + { + ID: "E2", + Source: "hackernews", + Kind: "community_discussion", + Title: "SlowCorp pricing increase", + ObservedAt: "2026-10-15", + Data: map[string]any{ + "comment": "They announced a huge price hike and forced upgrade to enterprise tier. Switched to an alternative.", + }, + }, + { + ID: "E3", + Source: "reddit", + Kind: "community_discussion", + Title: "Sync failed in SlowCorp", + ObservedAt: "2026-09-01", + Data: map[string]any{ + "comment": "Integration broken and webhook sync failed, causing data loss.", + }, + }, + }, + Content: []model.Content{ + { + ID: "c1", + Kind: "video", + Text: "Tired of waiting for spreadsheets to load?\nOur engine is 10x faster.\nGet started free.", + Evidence: "E4", + }, + }, + } + + analysis := AnalyzeEntrepreneur(bundle) + + // 1. Verify Creative Angles + if len(analysis.CreativeAngles) == 0 { + t.Fatal("expected creative angles from content, got 0") + } + if analysis.CreativeAngles[0].Category != "agitation" && analysis.CreativeAngles[0].Category != "shock_stat" { + t.Errorf("unexpected creative category: %q", analysis.CreativeAngles[0].Category) + } + + // 2. Verify Grievances + if len(analysis.TopGrievances) == 0 { + t.Fatal("expected top grievances, got 0") + } + + themeMap := make(map[string]model.Grievance) + for _, g := range analysis.TopGrievances { + themeMap[g.Theme] = g + if g.Score <= 0 { + t.Errorf("expected positive score for %q, got %f", g.Theme, g.Score) + } + if len(g.SampleQuotes) == 0 { + t.Errorf("expected verbatim quotes for %q", g.Theme) + } + if len(g.EvidenceIDs) == 0 { + t.Errorf("expected evidence IDs for %q", g.Theme) + } + } + + if _, ok := themeMap["Performance & Speed"]; !ok { + t.Errorf("expected Performance & Speed theme from E1") + } + if _, ok := themeMap["Pricing & Monetization"]; !ok { + t.Errorf("expected Pricing & Monetization theme from E2") + } + + // 3. Verify Displacement Plays + if len(analysis.Displacement) == 0 { + t.Fatal("expected displacement plays, got 0") + } + + var hasPerformancePlay, hasPricingPlay bool + for _, p := range analysis.Displacement { + if p.CounterHook == "" || p.AnglePlaybook == "" || p.RecommendedCTA == "" { + t.Errorf("incomplete displacement play: %#v", p) + } + if len(p.EvidenceIDs) == 0 { + t.Errorf("missing evidence citations in displacement play") + } + if p.ImpactROI == "" { + t.Errorf("missing impact ROI in displacement play") + } + if p.TargetWeakness != "" { + hasPerformancePlay = true + } + if p.ImpactROI == "High (Immediate economic switching trigger)" { + hasPricingPlay = true + } + } + + if !hasPerformancePlay || !hasPricingPlay { + t.Errorf("missing expected displacement play types") + } +} diff --git a/internal/report/report.go b/internal/report/report.go index 8b4ee1a..307d98f 100644 --- a/internal/report/report.go +++ b/internal/report/report.go @@ -63,8 +63,8 @@ var rules = []rule{ {"paid_meta", "meta_ads"}, {"paid_google", "google_ads"}, {"paid_linkedin", "linkedin_ads"}, {"paid_reddit", "reddit_ads"}, {"paid_tiktok", "tiktok_ads"}, {"paid_apple", "apple_ads"}, {"organic_tiktok", "tiktok"}, {"organic_instagram", "instagram"}, {"organic_youtube", "youtube"}, {"organic_x", "x"}, {"organic_linkedin", "linkedin"}, {"seo", "seo"}, {"geo", "seo"}, - {"founder_led", "founders"}, {"reddit_marketing", "reddit"}, {"community", "reddit"}, {"launches", "hackernews"}, - {"app_store", "appstore"}, {"lifecycle", "tracking"}, {"affiliate", "website"}, {"sales_led", "website"}, + {"founder_led", "founders"}, {"reddit_marketing", "reddit"}, {"community", "hackernews"}, {"launches", "hackernews"}, + {"open_source", "github"}, {"app_store", "appstore"}, {"lifecycle", "tracking"}, {"affiliate", "website"}, {"sales_led", "website"}, } func Score(b model.Bundle) []model.ChannelScore { @@ -78,7 +78,19 @@ func Score(b model.Bundle) []model.ChannelScore { } } for _, e := range b.Evidence { - if e.Source != r.source && !(r.channel == "founder_led" && e.Source == "wikidata" && len(dataSlice(dataMap(e.Data)["founders"])) > 0) { + matchSource := e.Source == r.source + if !matchSource { + if r.channel == "founder_led" && e.Source == "wikidata" && len(dataSlice(dataMap(e.Data)["founders"])) > 0 { + matchSource = true + } else if (r.channel == "community" || r.channel == "reddit_marketing") && (e.Source == "reddit" || e.Source == "hackernews" || e.Source == "github") { + matchSource = true + } else if r.channel == "launches" && (e.Source == "hackernews" || e.Source == "github") { + matchSource = true + } else if r.channel == "open_source" && e.Source == "github" { + matchSource = true + } + } + if !matchSource { continue } status, strength := classify(r.channel, e) @@ -168,6 +180,24 @@ func classify(channel string, e model.Evidence) (string, int) { } } } + if channel == "open_source" { + if e.Source != "github" { + return "", 0 + } + if stars, ok := number(d["stars"]); ok && stars > 0 { + return "active", strength(stars) + } + return "likely", 1 + } + if (channel == "reddit_marketing" || channel == "community") && e.Source == "reddit" { + if n, ok := number(d["matchedThreads"]); ok { + if n == 0 { + return "none", 0 + } + score, _ := number(d["totalScore"]) + return "active", strength(score) + } + } if n, ok := number(d["matchedThreads"]); ok && n == 0 { return "none", 0 } @@ -283,6 +313,50 @@ func Markdown(b model.Bundle) string { for _, c := range b.Scorecard { fmt.Fprintf(&w, "| %s | %s | %s | %s |\n", Clean(strings.ReplaceAll(c.Channel, "_", " ")), Clean(c.Status), Clean(c.Signal), references(c.EvidenceIDs)) } + + ent := b.Entrepreneur + if len(ent.TopGrievances) == 0 && len(ent.CreativeAngles) == 0 && len(ent.Displacement) == 0 { + ent = AnalyzeEntrepreneur(b) + } + + if len(ent.CreativeAngles) > 0 { + w.WriteString("\n## Creative angles & ad deconstruction (REA)\n\n| Category | Hook | Mechanism | Proof & CTA | Format | Evidence |\n|---|---|---|---|---|---|\n") + for _, a := range ent.CreativeAngles { + proofCTA := strings.TrimSpace(a.Proof + " " + a.OfferCTA) + if proofCTA == "" { + proofCTA = "—" + } + fmt.Fprintf(&w, "| %s | %s | %s | %s | %s | %s |\n", + Clean(a.Category), Clean(a.Hook), Clean(a.Mechanism), Clean(proofCTA), Clean(a.Format), references(a.EvidenceIDs)) + } + } + + if len(ent.TopGrievances) > 0 || len(ent.Displacement) > 0 { + w.WriteString("\n## Day-1 Displacement Playbook (Replica-Entrepreneur)\n\n") + if len(ent.TopGrievances) > 0 { + w.WriteString("### Competitor grievances & flaws\n\n| Grievance Theme | Severity | Verbatim Customer Quote | Sources | Evidence |\n|---|---|---|---|---|\n") + for _, g := range ent.TopGrievances { + quote := "—" + if len(g.SampleQuotes) > 0 { + quote = g.SampleQuotes[0] + } + sourcesStr := strings.Join(g.Sources, ", ") + if sourcesStr == "" { + sourcesStr = "public_web" + } + fmt.Fprintf(&w, "| %s | %.2f | %s | %s | %s |\n", + Clean(g.Theme), g.Score, Clean(quote), Clean(sourcesStr), references(g.EvidenceIDs)) + } + } + + if len(ent.Displacement) > 0 { + w.WriteString("\n### Challenger brand counter-marketing plays\n\n| Target Flaw | Challenger Counter-Hook | Angle & Copy Script | Recommended CTA | Impact ROI | Evidence |\n|---|---|---|---|---|---|\n") + for _, p := range ent.Displacement { + fmt.Fprintf(&w, "| %s | %s | %s | %s | %s | %s |\n", + Clean(p.TargetWeakness), Clean(p.CounterHook), Clean(p.AnglePlaybook), Clean(p.RecommendedCTA), Clean(p.ImpactROI), references(p.EvidenceIDs)) + } + } + } w.WriteString("\n## Evidence\n\n") for _, e := range b.Evidence { fmt.Fprintf(&w, "- [%s] %s · %s", e.ID, Clean(e.Title), Clean(e.Kind)) @@ -323,7 +397,7 @@ func Markdown(b model.Bundle) string { func describe(e model.Evidence) []string { fields := map[string][]string{ - "website": {"h1", "ctas", "prices", "freePlan", "freeTrial", "contactSales"}, + "website": {"h1", "ctas", "prices", "freePlan", "freeTrial", "noCreditCard", "moneyBack", "annualDiscount", "soc2Certified", "contactSales", "growthRoles"}, "seo": {"totalUrls", "sections", "patterns", "llmsTxt", "blogPostsLast90d"}, "tracking": {"pixels", "tags", "gtmContainersScanned"}, "meta_ads": {"totalInLibrary", "activeNow", "launchedLast30d", "formats"}, @@ -331,6 +405,8 @@ func describe(e model.Evidence) []string { "apple_ads": {"totalAdsLastYear", "active"}, "wayback": {"firstSeen", "monthlySnapshots", "history"}, "appstore": {"ratingCount", "rating", "installs", "seller"}, + "reddit": {"matchedThreads", "totalScore", "subreddits"}, + "github": {"fullName", "stars", "forks", "topics", "license", "pushedAt"}, } d := dataMap(e.Data) var out []string diff --git a/internal/web/app.css b/internal/web/app.css index 0c2e697..1c56f26 100644 --- a/internal/web/app.css +++ b/internal/web/app.css @@ -56,6 +56,8 @@ fieldset legend { padding: 0 0 5px; } fieldset label { display: inline-flex; align-items: center; gap: 3px; } fieldset input { min-height: auto; accent-color: var(--green); } .advanced-fields { grid-template-columns: repeat(2,minmax(0,1fr)); } +.advanced-fields .checkbox-label { grid-column: 1 / -1; display: inline-flex; align-items: center; gap: 8px; font-size: .88rem; cursor: pointer; } +.advanced-fields .checkbox-label input { width: auto; min-height: auto; margin: 0; accent-color: var(--green); } .advanced-fields input, .filterbar input, .filterbar select { display: block; margin-top: 3px; width: 100%; } .recent { margin-top: clamp(36px, 6vw, 72px); } .section-heading, .dialog-head, .run-top, .run-title { display: flex; justify-content: space-between; align-items: center; gap: 12px; flex-wrap: wrap; } @@ -80,7 +82,7 @@ fieldset input { min-height: auto; accent-color: var(--green); } .run-title h1 { font-size: clamp(2.2rem, 6vw, 3.65rem); } .run-title p { color: var(--muted); margin-top: 6px; } .download-links { display: flex; flex-wrap: wrap; gap: 6px; } -.metrics { display: grid; grid-template-columns: repeat(3,minmax(0,1fr)); gap: 9px; } +.metrics { display: grid; grid-template-columns: repeat(auto-fit, minmax(130px, 1fr)); gap: 9px; } .metric { background: var(--paper); border: 1px solid var(--line); border-radius: 7px; padding: 13px; } .metric strong { display: block; font-size: 1.45rem; font-variant-numeric: tabular-nums; margin-bottom: 3px; } .metric span { display: block; color: var(--muted); font-size: .82rem; } diff --git a/internal/web/app.js b/internal/web/app.js index a3d8ac8..29dfec3 100644 --- a/internal/web/app.js +++ b/internal/web/app.js @@ -8,6 +8,8 @@ const labels = { geo: "AI search", paid_meta: "Meta ads", paid_apple: "Apple ads", paid_google: "Google ads", paid_linkedin: "LinkedIn ads", paid_tiktok: "TikTok ads", paid_reddit: "Reddit ads", appstore: "App Store", playstore: "Google Play", llms: "LLMs.txt", + open_source: "Open Source / Developer", reddit_marketing: "Reddit Community", + github: "GitHub Public", reddit: "Reddit Public", }; const defaultAreas = ["website", "seo", "history", "apps", "paid", "social", "founders", "community"]; const panels = ["overview", "report", "evidence", "content", "unknowns"]; @@ -155,6 +157,10 @@ async function begin(event) { if (credits !== "") body.maxCredits = Number(credits); const model = $("model").value.trim(); if (model && body.engine !== "evidence") body.model = model; + if ($("stealth") && $("stealth").checked) body.stealth = true; + if ($("keyless") && $("keyless").checked) body.keyless = true; + const windowVal = $("window") && $("window").value.trim(); + if (windowVal) body.window = windowVal; try { body.handles = getHandles($("handles").value); $("start").disabled = true; @@ -477,9 +483,18 @@ function displayRun(id, data) { }; } $("evidence-link").href = `${path}/evidence.json`; - $("metrics").replaceChildren(...[ - [current.evidence.length, "Evidence items"], [current.content.length, "Ads & posts"], [bundle.unknowns?.length || 0, "Unknowns"], - ].map(([number, name]) => { + const metricCards = [ + [current.evidence.length, "Evidence items"], + [current.content.length, "Ads & posts"], + [bundle.unknowns?.length || 0, "Unknowns"], + ]; + if (bundle.entrepreneur?.creativeAngles?.length) { + metricCards.push([bundle.entrepreneur.creativeAngles.length, "Creative angles"]); + } + if (bundle.entrepreneur?.displacement?.length) { + metricCards.push([bundle.entrepreneur.displacement.length, "Displacement plays"]); + } + $("metrics").replaceChildren(...metricCards.map(([number, name]) => { const metric = el("div", null, "metric"); metric.append(el("strong", number), el("span", name)); return metric; diff --git a/internal/web/index.html b/internal/web/index.html index 0b8401c..355862b 100644 --- a/internal/web/index.html +++ b/internal/web/index.html @@ -42,6 +42,9 @@

Investigate a company’s marketing

+ + + diff --git a/internal/web/server.go b/internal/web/server.go index c11d19b..b7f7664 100644 --- a/internal/web/server.go +++ b/internal/web/server.go @@ -228,6 +228,9 @@ type runInput struct { MaxCredits *int `json:"maxCredits,omitempty"` Engine string `json:"engine,omitempty"` Model string `json:"model,omitempty"` + Stealth bool `json:"stealth,omitempty"` + Keyless bool `json:"keyless,omitempty"` + Window string `json:"window,omitempty"` } func (s *server) startRun(w http.ResponseWriter, r *http.Request) { @@ -281,7 +284,15 @@ func (s *server) startRun(w http.ResponseWriter, r *http.Request) { return } } - opts := app.Options{Only: input.Only, Handles: input.Handles, Engine: input.Engine, Model: input.Model} + opts := app.Options{ + Only: input.Only, + Handles: input.Handles, + Engine: input.Engine, + Model: input.Model, + Stealth: input.Stealth, + Keyless: input.Keyless, + Window: input.Window, + } if input.MaxCredits != nil { opts.BudgetSet = true opts.MaxCredits = *input.MaxCredits diff --git a/internal/web/server_test.go b/internal/web/server_test.go index 7911d81..989a6a1 100644 --- a/internal/web/server_test.go +++ b/internal/web/server_test.go @@ -176,6 +176,39 @@ func TestCompareLatestRuns(t *testing.T) { } } +func TestStartRunAcceptsStealthKeylessAndWindow(t *testing.T) { + receivedOpts := make(chan app.Options, 1) + h := newHandler(backend{ + doctor: func() map[string]any { + return map[string]any{"engines": map[string]bool{"evidence": true}} + }, + investigate: func(ctx context.Context, input string, opts app.Options) (model.Bundle, error) { + receivedOpts <- opts + return model.Bundle{ID: "run-stealth", Brand: "Example"}, nil + }, + }) + + body := `{"url":"example.com","stealth":true,"keyless":true,"window":"30d"}` + w := request(t, h, "POST", "/api/runs", body) + if w.Code != http.StatusAccepted { + t.Fatalf("expected accepted status, got %d: %s", w.Code, w.Body) + } + select { + case opts := <-receivedOpts: + if !opts.Stealth { + t.Errorf("expected Stealth=true, got false") + } + if !opts.Keyless { + t.Errorf("expected Keyless=true, got false") + } + if opts.Window != "30d" { + t.Errorf("expected Window=30d, got %q", opts.Window) + } + case <-time.After(2 * time.Second): + t.Fatal("investigate was not invoked within timeout") + } +} + func readJobID(t *testing.T, w *httptest.ResponseRecorder) string { t.Helper() if w.Code != http.StatusAccepted {