diff --git a/README.md b/README.md index c52aa66..cfa63d4 100644 --- a/README.md +++ b/README.md @@ -85,7 +85,7 @@ Every day it collects signals from 6 data sources, runs them through a smart sco | GitHub Search | New & rising | Repos created in the last 7 days with fast star growth | | [Hacker News](https://news.ycombinator.com/) | Community picks | GitHub repos from Show HN posts | | Rising Detection | Early signal | Repos created in the last 3 days with a fork/star ratio above 0.3 and at most 0.5 — forks as an early usage signal; higher ratios are discarded as fork farms | -| AI/ML Keyword Sweep | AI focus | GitHub Search over an AI/ML keyword list, 5 keywords per day on a rotating schedule (inspired by [OSSInsight trending/ai](https://ossinsight.io/trending/ai), implemented against the GitHub Search API) | +| AI/ML Keyword Sweep | AI focus | GitHub Search over an AI/ML keyword list, 5 keywords per day on a rotating schedule, plus 2 semiconductor / industrial-AI keywords per day over repos pushed in the last 30 days (inspired by [OSSInsight trending/ai](https://ossinsight.io/trending/ai), implemented against the GitHub Search API) | | [HF Daily Papers](https://huggingface.co/papers) | Research signal | GitHub repos linked from trending Hugging Face papers — we assume paper upvotes lead GitHub stars by days, but no committed measurement backs that lead time yet | ### Smart Scoring (100 points) @@ -104,11 +104,22 @@ Every day it collects signals from 6 data sources, runs them through a smart sco - **Star fraud detection**: age ≤ 1 day with 1000+ stars, age ≤ 2 days with 2000+ stars, or 1000+ stars/day with an empty description → flagged - **Batch fraud detection**: same owner with ≥ 3 repos in one batch, or ≥ 2 repos under 7 days old already past 200 stars → flagged - **Content quality**: No description or no README → penalty -- **Cross-day dedup**: 7-day window, no duplicate recommendations across days (a same-day rerun may repeat a pick) +- **Cross-day dedup**: 7-day window for the daily report (a same-day rerun may repeat a pick) +- **Picks never repeat**: a repo that has appeared in a picks issue (or was recommended in chat) is never picked again — see `data/picks_history.json` + +### Picks (every 3 days) + +Collection still runs daily; every `PICKS_EVERY_DAYS` (3) days `scripts/picks.py` publishes `data/picks-YYYY-MM-DD.json`: + +- 8–10 repos (`PICKS_MIN`/`PICKS_MAX`), at least `SEMI_MIN` (2) tagged semiconductor / industrial AI when the pool has them (`semi_min_met` says whether it did); AI-tagged repos rank ahead of untagged ones +- every pick is re-read from `GET /repos/{owner}/{repo}` right before publishing; missing, archived or disabled repos are dropped +- facts only: stars, forks, topics, dates and the sources that surfaced it; `growth` is a measured delta from the committed star snapshots, or `null` — never a lifetime average +- no repeat performers, and no LLM in the pipeline; fewer than 8 verified candidates means no issue that day +- `picks-never-repeat` and `picks-count-in-range` in `docs/claims.json` re-check every committed issue ### Email Subscription -- Daily curated repos delivered to your inbox +- Each picks issue (the same 8–10 repos) is delivered to your inbox; non-picks days send no email - Rendered dark-only, with `color-scheme` declared for dark-aware clients (Apple Mail / iOS) - Powered by Resend API @@ -245,6 +256,7 @@ github-discovery/ │ ├── quality.py # Code quality detection │ ├── anti_spam.py # Anti-spam scoring dimension │ ├── dedup.py # Cross-day deduplication (7-day window) +│ ├── picks.py # Picks issue every 3 days (permanent dedup, facts only) │ ├── fraud_detection.py # Batch fraud detection │ ├── snapshots.py # Daily star snapshots (real growth) │ ├── verify_scoring.py # Scoring verification / backtesting @@ -324,7 +336,7 @@ have since moved. **Do I need any paid service?** Only for email. `RESEND_API_KEY` sends the digest through Resend; `FIRECRAWL_API_KEY` is optional and only makes GitHub Trending parsing more robust than regex-matching raw HTML. Everything else runs on the GitHub Actions free tier with the default `GITHUB_TOKEN`. -**Will the same repo be recommended every day while it is hot?** No. Cross-day dedup blocks any repo recommended in the previous 7 days, using the history committed at `data/recommend_history.json`; records older than 30 days are cleaned up. The history has to be a committed file because every CI run starts from a fresh checkout with no local state. +**Will the same repo be recommended every day while it is hot?** No. Picks never repeat a repo (permanent history in `data/picks_history.json`). The daily report's cross-day dedup blocks any repo recommended in the previous 7 days, using the history committed at `data/recommend_history.json`; records older than 30 days are cleaned up. The history has to be a committed file because every CI run starts from a fresh checkout with no local state. **Can I trust the numbers on the published page?** The page is a build artifact: `scripts/generate_site.py` renders `docs/index.html` from `docs/template.html` plus the committed reports, and the workflow publishes `docs/` to the `gh-pages` branch. The authoritative copies are the committed JSON under `data/`, which is what `verify_scoring.py` reads; the Markdown digests under `output/` are the human-readable copies, and the site reads the JSON first and falls back to those digests. So any claim on the page can be recomputed from a fresh clone. diff --git a/README.zh-CN.md b/README.zh-CN.md index 0fe65d6..4adf0d1 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -85,7 +85,7 @@ GitHub Discovery 告诉你什么**即将火起来**——增长曲线异常的 | GitHub Search | 新晋上升 | 最近 7 天创建、星标增长迅猛的仓库 | | [Hacker News](https://news.ycombinator.com/) | 社区精选 | Show HN 帖子中出现的 GitHub 仓库 | | Rising Detection | 早期信号 | 最近 3 天创建、fork/star 比大于 0.3 且不超过 0.5 的仓库——fork 作为真实使用的早期信号；比率更高的按 fork farm 直接排除 | -| AI/ML 关键词轮扫 | AI 方向 | 用一组 AI/ML 关键词扫 GitHub Search，每天轮换 5 个词（思路来自 [OSSInsight trending/ai](https://ossinsight.io/trending/ai)，实现走的是 GitHub Search API） | +| AI/ML 关键词轮扫 | AI 方向 | 用一组 AI/ML 关键词扫 GitHub Search，每天轮换 5 个词，另加每天 2 个半导体/工业 AI 关键词（看近 30 天有推送的仓库）（思路来自 [OSSInsight trending/ai](https://ossinsight.io/trending/ai)，实现走的是 GitHub Search API） | | [HF Daily Papers](https://huggingface.co/papers) | 研究信号 | Hugging Face 热门论文关联的 GitHub 仓库——我们假设论文点赞领先 GitHub 星标几天，但仓库里还没有可提交的测量来支撑这个领先时间 | ### 智能评分（满分 100 分） @@ -104,11 +104,22 @@ GitHub Discovery 告诉你什么**即将火起来**——增长曲线异常的 - **刷星检测**：年龄 ≤ 1 天且 1000+ 星、年龄 ≤ 2 天且 2000+ 星，或日增 1000+ 星且描述为空 → 标记 - **批量刷量检测**：同一作者一批里出现 ≥ 3 个仓库，或 ≥ 2 个上线不足 7 天的仓库已过 200 星 → 标记 - **内容质量**：没有描述或没有 README → 扣分 -- **跨日去重**：7 天窗口内跨天不重复推荐（同日重跑可能重复选到同一个仓库） +- **跨日去重**：每日报告 7 天窗口内跨天不重复推荐（同日重跑可能重复选到同一个仓库） +- **推荐期永不重复**：进过推荐期（或在聊天里推过）的仓库永不再推，见 `data/picks_history.json` + +### 推荐期（每 3 天一期） + +采集仍是每天一次；每 `PICKS_EVERY_DAYS`（3）天，`scripts/picks.py` 发布一期 `data/picks-YYYY-MM-DD.json`： + +- 每期 8–10 个（`PICKS_MIN`/`PICKS_MAX`），候选里有的话至少 `SEMI_MIN`（2）个半导体/工业 AI 仓库（是否达标见 `semi_min_met`）；带 AI 标签的排在无主题的前面 +- 发布前逐个用 `GET /repos/{owner}/{repo}` 重新读取，不存在、已归档或已禁用的剔除 +- 只放事实：star、fork、topics、日期和发现它的数据源；`growth` 是按已提交 star 快照算出的实测增量，拿不到就是 `null`，绝不用终身平均值顶替 +- 不带 repeat，流程里不接大模型；核实后不足 8 个当天就不出刊 +- `docs/claims.json` 里的 `picks-never-repeat` 和 `picks-count-in-range` 会复核每一期 ### 邮件订阅 -- 每日精选仓库直达你的收件箱 +- 每期推荐（同一份 8–10 个）直达你的收件箱；非推荐日不发邮件 - 只渲染深色版本，并声明 `color-scheme` 供支持深色的客户端使用（Apple Mail / iOS） - 基于 Resend API 发送 @@ -198,6 +209,7 @@ github-discovery/ │ ├── quality.py # Code quality detection │ ├── anti_spam.py # Anti-spam scoring dimension │ ├── dedup.py # Cross-day deduplication (7-day window) +│ ├── picks.py # Picks issue every 3 days (permanent dedup, facts only) │ ├── fraud_detection.py # Batch fraud detection │ ├── snapshots.py # Daily star snapshots (real growth) │ ├── verify_scoring.py # Scoring verification / backtesting @@ -274,7 +286,7 @@ python scripts/verify_scoring.py --days 30 **需要付费服务吗？** 只有邮件需要。`RESEND_API_KEY` 用于通过 Resend 发送日报；`FIRECRAWL_API_KEY` 是可选的，只是让 GitHub Trending 的解析比正则匹配原始 HTML 更健壮。其余部分都跑在 GitHub Actions 免费额度上，用默认的 `GITHUB_TOKEN` 即可。 -**一个仓库火着的时候会天天被推荐吗？** 不会。跨日去重会拦掉过去 7 天内推荐过的仓库，依据是提交在 `data/recommend_history.json` 里的推荐历史，超过 30 天的记录会被清理。这份历史必须是提交进仓库的文件，因为每次 CI 都是全新 checkout，没有任何本地状态。 +**一个仓库火着的时候会天天被推荐吗？** 不会。推荐期永不重复推同一个仓库（永久历史见 `data/picks_history.json`）；每日报告的跨日去重会拦掉过去 7 天内推荐过的仓库，依据是提交在 `data/recommend_history.json` 里的推荐历史，超过 30 天的记录会被清理。这份历史必须是提交进仓库的文件，因为每次 CI 都是全新 checkout，没有任何本地状态。 **网页上的数字可信吗？** 网页是构建产物：`scripts/generate_site.py` 用 `docs/template.html` 加上已提交的报告渲染出 `docs/index.html`，工作流再把整个 `docs/` 发布到 `gh-pages` 分支。真正权威的是提交在 `data/` 下的 JSON，`verify_scoring.py` 读的就是它；`output/` 下的 Markdown 日报是给人看的副本，站点优先读 JSON、读不到才回退到日报。所以网页上的任何结论都能在全新 clone 上重算一遍。 diff --git a/data/picks_history.json b/data/picks_history.json new file mode 100644 index 0000000..372a9de --- /dev/null +++ b/data/picks_history.json @@ -0,0 +1,15 @@ +{ + "repos": { + "openai/math": {"first_picked": "2026-10-16", "source": "chat"}, + "NandhaKishorM/laya": {"first_picked": "2026-10-16", "source": "chat"}, + "morluto/rea": {"first_picked": "2026-10-16", "source": "chat"}, + "zai-org/ZCode": {"first_picked": "2026-10-16", "source": "chat"}, + "feder-cr/git-ai": {"first_picked": "2026-10-16", "source": "chat"}, + "anthropics/oss-scanner": {"first_picked": "2026-10-16", "source": "chat"}, + "rociiu/talorys": {"first_picked": "2026-10-16", "source": "chat"}, + "chenwei810/Semiconductor_Manufacturing_Introduction": {"first_picked": "2026-10-16", "source": "chat"}, + "hdl-tools/digital-chip-design-agents": {"first_picked": "2026-10-16", "source": "chat"}, + "BLANK2077/xverif": {"first_picked": "2026-10-16", "source": "chat"} + }, + "updated_at": "2026-10-16T17:00:00-07:00" +} diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 99a2f71..803a4d0 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -8,7 +8,7 @@ - **100-point scorer** — acceleration (40) + quality (30) + anti-spam (30), with code-quality bonus, suspicious-star and batch-fraud penalties, and an explainable reason line on every card - **Cross-day dedup** with a 7-day window, deep checks budgeted to the top-K candidates after coarse ranking - **Delivery** — daily email digest via Resend (dark-mode HTML) + GitHub Pages site with date/language filters -- **Current quality baseline** — 202 tests and 20 executable receipts; the site, feed and daily JSON are generated from committed reports. The presentation pass (light/dark themes, two-band header, merged report header, lead-card treatment) is complete; further UI work is gated by a user task or a measured defect. +- **Current quality baseline** — 210 tests and 22 executable receipts; the site, feed and daily JSON are generated from committed reports. The presentation pass (light/dark themes, two-band header, merged report header, lead-card treatment) is complete; further UI work is gated by a user task or a measured defect. ## Long-term operating plan diff --git a/docs/claims.json b/docs/claims.json index c76e7ce..98db003 100644 --- a/docs/claims.json +++ b/docs/claims.json @@ -213,7 +213,9 @@ "verified": "2026-09-26", "check": { "cmd": "python3 - <<'PY'\nimport glob, json\npaths = sorted(glob.glob('data/candidate-pool-*.json'))\nif not paths:\n print('candidate pool: not-ready (no production pool yet)')\n raise SystemExit(0)\npath = paths[-1]\ndoc = json.load(open(path, encoding='utf-8'))\nrows = doc.get('candidates') or []\nallowed = {'candidate', 'deduplicated', 'content_blocked', 'eligible', 'recommended'}\nnames = [r.get('full_name') for r in rows]\nproblems = []\nif len(names) != len(set(names)): problems.append('duplicate names')\nfor row in rows:\n if not row.get('full_name'): problems.append('missing name')\n if not (row.get('sources') or row.get('source')): problems.append('missing source')\n if row.get('pool_status') not in allowed: problems.append('bad status')\nprint('candidate pool', path, 'rows', len(rows), 'problems', sorted(set(problems)))\nassert rows and not problems\nPY", - "expect": {"regex": "candidate pool: not-ready|candidate pool .* problems \\[\\]"}, + "expect": { + "regex": "candidate pool: not-ready|candidate pool .* problems \\[\\]" + }, "timeout": 60 } }, @@ -228,7 +230,9 @@ "as_of": "2026-09-26", "check": { "cmd": "python3 -c \"import re,subprocess,pathlib;out=subprocess.run(['python3','-m','pytest','--collect-only','-q'],capture_output=True,text=True).stdout;got=int(re.search(r'(\\d+) tests? collected', out).group(1));doc=int(re.search(r'(\\d+) tests', pathlib.Path('docs/ROADMAP.md').read_text()).group(1));print('pytest', got, 'roadmap', doc);assert got==doc\"", - "expect": {"regex": "pytest \\d+ roadmap \\d+$"}, + "expect": { + "regex": "pytest \\d+ roadmap \\d+$" + }, "timeout": 180 } }, @@ -351,6 +355,40 @@ }, "timeout": 120 } + }, + { + "id": "picks-never-repeat", + "claim": "No repository appears in more than one picks issue, every pick is recorded in the permanent history, and none repeats a repository already recommended in chat.", + "value": "0 repeats", + "metric": "duplicate full_name across data/picks-*.json, plus picks missing from or colliding with data/picks_history.json", + "method": "scripts/picks.py filters candidates against data/picks_history.json, which is never pruned; this receipt re-derives the property from the committed files alone.", + "repro": "ls data/picks-*.json", + "evidence": "data/picks_history.json", + "as_of": "2026-10-16", + "check": { + "cmd": "python3 - <<'PY'\nimport glob, json\npaths = sorted(glob.glob('data/picks-*.json'))\nhist = json.load(open('data/picks_history.json', encoding='utf-8'))['repos']\nseen, problems = {}, []\nfor path in paths:\n for p in json.load(open(path, encoding='utf-8'))['picks']:\n k = p['full_name'].lower()\n if k in seen: problems.append(f\"{p['full_name']} in {seen[k]} and {path}\")\n seen[k] = path\n if p['full_name'] not in hist: problems.append(f\"{p['full_name']} missing from history\")\n elif hist[p['full_name']].get('source') == 'chat': problems.append(f\"{p['full_name']} repeats a chat pick\")\nprint('picks files', len(paths), 'repos', len(seen), 'problems', problems)\nassert not problems\nPY", + "expect": { + "regex": "problems \\[\\]$" + }, + "timeout": 60 + } + }, + { + "id": "picks-count-in-range", + "claim": "Every picks issue holds 8 to 10 repositories with GitHub URLs, integer star counts, and a growth field that is either a measured snapshot delta or null.", + "value": "8-10 per issue", + "metric": "len(picks) per data/picks-*.json and the shape of its url, stars and growth fields", + "method": "scripts/picks.py publishes an issue only when at least PICKS_MIN verified repositories exist and never substitutes lifetime averages for growth; this receipt checks every committed issue.", + "repro": "ls data/picks-*.json", + "evidence": "data/picks-*.json", + "as_of": "2026-10-16", + "check": { + "cmd": "python3 - <<'PY'\nimport glob, json\npaths = sorted(glob.glob('data/picks-*.json'))\nproblems = []\nfor path in paths:\n d = json.load(open(path, encoding='utf-8'))\n n = len(d['picks'])\n if not 8 <= n <= 10 or d['count'] != n: problems.append(f'{path}: count {n}')\n for p in d['picks']:\n if not p['url'].startswith('https://github.com/'): problems.append(f\"{p['full_name']}: url\")\n if not isinstance(p['stars'], int): problems.append(f\"{p['full_name']}: stars\")\n g = p['growth']\n if g is not None and (g.get('source') != 'star_snapshots' or not isinstance(g.get('stars_gained'), int)):\n problems.append(f\"{p['full_name']}: growth not measured\")\nprint('picks files', len(paths), 'problems', problems)\nassert not problems\nPY", + "expect": { + "regex": "problems \\[\\]$" + }, + "timeout": 60 + } } ] } diff --git a/scripts/main.py b/scripts/main.py index e4d8bbf..153a6e8 100644 --- a/scripts/main.py +++ b/scripts/main.py @@ -27,6 +27,9 @@ from fraud_detection import detect_batch_fraud, apply_fraud_penalty from snapshots import record_snapshots, get_growth from watchlist import add_discoveries +import picks as picks_mod +from sources import _gh_api +from snapshots import _load as load_snapshots def get_subscribers() -> list[str]: @@ -586,7 +589,24 @@ def main(): # been sent (or benignly skipped). If the email fails, we exit non-zero # without writing the marker, so the next scheduled run retries. date_str = datetime.now().strftime("%Y-%m-%d") - email_result = send_digest_email(date_str, top_new) + + # ── 推荐期（picks）：每 PICKS_EVERY_DAYS 天一期，邮件和聊天共用这一份 ── + # 非推荐日只采集（报告/站点/快照照常），不发邮件。 + chosen = [] + if picks_mod.is_picks_day(): + ordered = picks_mod.select_candidates(scored) + chosen = picks_mod.verify_picks(ordered, _gh_api) + if len(chosen) < picks_mod.PICKS_MIN: + print(f"[Picks] only {len(chosen)} verified candidates (< {picks_mod.PICKS_MIN}); " + "no picks issue today, no email.") + chosen = [] + else: + print("[Picks] not a picks day — collect only, no email.") + + if chosen: + email_result = send_digest_email(date_str, chosen) + else: + email_result = "skipped" if email_result == "failed": print("[ERROR] Email send failed — not writing today's report so the " "next run retries. Exiting non-zero to flag the failure.") @@ -599,6 +619,11 @@ def main(): f.write(md) print(f"\n[Saved] Report written to {out_path}") + if chosen: + picks_path = picks_mod.write_picks(date_str, chosen, load_snapshots()) + picks_mod.record_picks([r for r, _ in chosen], date_str) + print(f"[Saved] Picks written to {picks_path}") + # Persist source health only after ranking, dedup and recommendation selection are complete. set_source_recommendations([repo for repo, _scores in top_new + top_repeat]) source_health = get_source_health() diff --git a/scripts/picks.py b/scripts/picks.py new file mode 100644 index 0000000..dc1e7fa --- /dev/null +++ b/scripts/picks.py @@ -0,0 +1,219 @@ +"""推荐期（picks）—— 聊天推荐和邮件共用的唯一来源。 + +每 PICKS_EVERY_DAYS 天出一期，每期 PICKS_MIN..PICKS_MAX 个仓库，至少 SEMI_MIN 个 +半导体/工业 AI 仓库。和每日的 discovery 报告不同： + - 永久去重：data/picks_history.json 里出现过的仓库永不再进 picks（不像 + recommend_history 那样 7 天后放行、30 天后清除）； + - 只写事实字段：数字全部来自 GitHub API（verify_picks 当场重新读取）或本仓库 + 提交的 star 快照，增长拿不到就写 null，不用终身平均值顶替； + - 不接大模型：「做什么 / 为什么火」的文字由读取方根据这些字段来写。 +""" + +import json +import re +from datetime import date, datetime, timezone, timedelta +from pathlib import Path + +DATA_DIR = Path(__file__).parent.parent / "data" +PICKS_HISTORY_FILE = DATA_DIR / "picks_history.json" + +PICKS_EVERY_DAYS = 3 +# 推荐日 = 与锚点相差 PICKS_EVERY_DAYS 整数倍的日期（UTC）。锚点是第一期聊天推荐的日期。 +PICKS_ANCHOR = "2026-10-16" +PICKS_MIN = 8 +PICKS_MAX = 10 +SEMI_MIN = 2 + +AI_TERMS = [ + "llm", "large language model", "language model", "agent", "agentic", "ai", + "machine learning", "deep learning", "transformer", "diffusion", "rag", + "embedding", "fine-tuning", "lora", "inference", "mcp", "model context protocol", + "reinforcement learning", "multimodal", "vlm", "world model", "neural", + "gpt", "claude", "gemini", "qwen", "deepseek", "llama", "computer vision", "nlp", +] +SEMI_TERMS = [ + "semiconductor", "chip design", "asic", "fpga", "rtl", "verilog", "systemverilog", + "vhdl", "eda", "hdl", "pdk", "wafer", "lithography", "tapeout", "risc-v", "riscv", + "uvm", "formal verification", "timing closure", "place and route", "openroad", + "yosys", "chisel", "industrial ai", "predictive maintenance", "plc", "scada", + "manufacturing", "digital twin", "industrial iot", "iiot", "process control", +] + + +def is_picks_day(day: date | None = None) -> bool: + day = day or datetime.now(timezone.utc).date() + anchor = date.fromisoformat(PICKS_ANCHOR) + return (day - anchor).days % PICKS_EVERY_DAYS == 0 + + +def _has_term(text: str, term: str) -> bool: + return re.search(r"(? str | None: + """'semi' / 'ai' / None，依据仓库描述、topics 和名字。半导体优先。""" + text = " ".join([ + repo.get("full_name", "").replace("-", " ").replace("_", " "), + repo.get("description") or "", + " ".join(t.replace("-", " ") for t in repo.get("topics") or []), + ]).lower() + if any(_has_term(text, t) for t in SEMI_TERMS): + return "semi" + if any(_has_term(text, t) for t in AI_TERMS): + return "ai" + return None + + +# ── 永久历史 ───────────────────────────────────────────────────────── + +def load_history() -> dict: + if PICKS_HISTORY_FILE.exists(): + return json.loads(PICKS_HISTORY_FILE.read_text()) + return {"repos": {}} + + +def was_ever_picked(full_name: str, history: dict | None = None) -> bool: + history = history if history is not None else load_history() + key = full_name.lower() + return any(k.lower() == key for k in history.get("repos", {})) + + +def record_picks(picks: list[dict], day: str, source: str = "github-discovery"): + history = load_history() + repos = history.setdefault("repos", {}) + for p in picks: + repos.setdefault(p["full_name"], {"first_picked": day, "source": source}) + history["updated_at"] = datetime.now(timezone.utc).isoformat() + PICKS_HISTORY_FILE.parent.mkdir(parents=True, exist_ok=True) + PICKS_HISTORY_FILE.write_text(json.dumps(history, ensure_ascii=False, indent=2) + "\n") + + +# ── 选择 ───────────────────────────────────────────────────────────── + +def select_candidates(scored: list[tuple[dict, dict]], history: dict | None = None) -> list[tuple[dict, dict]]: + """按 (半导体配额, 有主题优先, 分数) 给出有序候选（多于 PICKS_MAX，留给核实失败时补位）。""" + history = history if history is not None else load_history() + pool = [] + for repo, scores in scored: + if was_ever_picked(repo["full_name"], history): + continue + if repo.get("star_suspicious") or scores.get("fraud_penalty"): + continue + repo["topic"] = topic_of(repo) + pool.append((repo, scores)) + pool.sort(key=lambda x: x[1].get("total", 0), reverse=True) + semi = [x for x in pool if x[0]["topic"] == "semi"] + rest = [x for x in pool if x[0]["topic"] != "semi"] + rest.sort(key=lambda x: (x[0]["topic"] is None, -x[1].get("total", 0))) + return semi[:SEMI_MIN] + rest + semi[SEMI_MIN:] + + +def finalize(verified: list[tuple[dict, dict]]) -> list[tuple[dict, dict]]: + """核实后的有序候选 → 最多 PICKS_MAX 个，尽量满足 SEMI_MIN。""" + semi = [x for x in verified if x[0].get("topic") == "semi"] + chosen = semi[:SEMI_MIN] + for x in verified: + if len(chosen) >= PICKS_MAX: + break + if x not in chosen: + chosen.append(x) + return chosen + + +# ── 核实与增长 ─────────────────────────────────────────────────────── + +def verify_picks(ordered: list[tuple[dict, dict]], gh_api, need: int = PICKS_MAX) -> list[tuple[dict, dict]]: + """逐个用 GET /repos/{full_name} 重新读取；404、归档、禁用的丢弃。 + + 凑够 need 个、且其中半导体达到 SEMI_MIN（或候选里已无半导体）就停，省 API 配额。 + """ + out = [] + semi_left = sum(1 for r, _ in ordered if r.get("topic") == "semi") + for repo, scores in ordered: + is_semi = repo.get("topic") == "semi" + semi_have = sum(1 for r, _ in out if r.get("topic") == "semi") + if len(out) >= need and (semi_have >= SEMI_MIN or semi_left == 0): + break + if is_semi: + semi_left -= 1 + data = gh_api(f"/repos/{repo['full_name']}") + if not data or data.get("archived") or data.get("disabled") or data.get("private"): + print(f" [Picks] drop {repo['full_name']}: not reachable/archived") + continue + repo.update({ + "full_name": data["full_name"], + "url": data["html_url"], + "description": data.get("description") or "", + "stars": data.get("stargazers_count", 0), + "forks": data.get("forks_count", 0), + "language": data.get("language") or "", + "topics": data.get("topics") or [], + "created_at": data.get("created_at", ""), + "pushed_at": data.get("pushed_at", ""), + "license": (data.get("license") or {}).get("spdx_id") or "", + "verified_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), + }) + repo["topic"] = topic_of(repo) or repo.get("topic") + out.append((repo, scores)) + return finalize(out) + + +def measured_growth(full_name: str, current_stars: int, snapshots: dict, today: str) -> dict | None: + """只用提交过的快照算实测增长；优先取 7 天前（或更早、14 天内）的点，否则取最近的一个更早的点。""" + hist = [p for p in snapshots.get("repos", {}).get(full_name, []) if p[0] < today] + if not hist: + return None + t = date.fromisoformat(today) + week = [p for p in hist if 7 <= (t - date.fromisoformat(p[0])).days <= 14] + prev_date, prev_stars = (week[-1] if week else hist[-1]) + span = (t - date.fromisoformat(prev_date)).days + return { + "since": prev_date, + "window_days": span, + "stars_gained": current_stars - prev_stars, + "source": "star_snapshots", + } + + +def build_pick(rank: int, repo: dict, scores: dict, snapshots: dict, today: str) -> dict: + return { + "rank": rank, + "full_name": repo["full_name"], + "url": repo["url"], + "description": repo.get("description") or "", + "topic": repo.get("topic"), + "language": repo.get("language") or "", + "topics": repo.get("topics") or [], + "license": repo.get("license") or "", + "stars": repo.get("stars", 0), + "forks": repo.get("forks", 0), + "created_at": repo.get("created_at", ""), + "pushed_at": repo.get("pushed_at", ""), + "growth": measured_growth(repo["full_name"], repo.get("stars", 0), snapshots, today), + "evidence": { + "sources": repo.get("sources") or [repo.get("source", "unknown")], + "hn_title": repo.get("hn_title"), + "hn_score": repo.get("hn_score"), + "hf_title": repo.get("hf_title"), + "hf_upvotes": repo.get("hf_upvotes"), + }, + "score": scores.get("total"), + "verified_at": repo.get("verified_at"), + } + + +def write_picks(day: str, chosen: list[tuple[dict, dict]], snapshots: dict) -> str: + picks = [build_pick(i, r, s, snapshots, day) for i, (r, s) in enumerate(chosen, 1)] + semi = sum(1 for p in picks if p["topic"] == "semi") + doc = { + "date": day, + "generated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "every_days": PICKS_EVERY_DAYS, + "count": len(picks), + "semi_count": semi, + "semi_min_met": semi >= SEMI_MIN, + "picks": picks, + } + path = DATA_DIR / f"picks-{day}.json" + path.write_text(json.dumps(doc, ensure_ascii=False, indent=2) + "\n") + return str(path) diff --git a/scripts/sources.py b/scripts/sources.py index b05949d..eb3dea4 100644 --- a/scripts/sources.py +++ b/scripts/sources.py @@ -501,6 +501,28 @@ def fetch_ai_trending() -> list[dict]: print(f" [WARN] AI trending error for {keyword}: {e}") continue + # 半导体/工业 AI：这个领域涨得慢，"近 7 天新建"几乎捞不到，所以改看近 30 天 + # 有推送、star 过百的仓库。仍属于 ai-trending 这一个数据源（不新增 fetcher）。 + semi_keywords = [ + "chip design", "verilog", "systemverilog", "fpga", "rtl", "eda", + "semiconductor", "risc-v", "formal verification", "industrial ai", + "predictive maintenance", "plc", + ] + s_start = (day * 2) % len(semi_keywords) + pushed_from = (datetime.now(timezone.utc) - timedelta(days=30)).strftime('%Y-%m-%d') + for keyword in [semi_keywords[(s_start + i) % len(semi_keywords)] for i in range(2)]: + try: + data = _gh_api("/search/repositories", { + "q": f'"{keyword}" pushed:>{pushed_from} stars:>100', + "sort": "stars", "order": "desc", "per_page": 10, + }) + if data and "items" in data: + for repo in data["items"]: + all_repos.append(_normalize_repo(repo)) + time.sleep(API_DELAY) + except Exception as e: + print(f" [WARN] semi sweep error for {keyword}: {e}") + # 去重 seen = set() unique = [] @@ -513,4 +535,4 @@ def fetch_ai_trending() -> list[dict]: unique.sort(key=lambda x: x.get("stars", 0), reverse=True) print(f"[AI Trending] {len(unique)} repos") - return unique[:30] # 返回前 30 个 + return unique[:40] # 返回前 40 个（含半导体扫描） diff --git a/tests/test_picks.py b/tests/test_picks.py new file mode 100644 index 0000000..fa87ba2 --- /dev/null +++ b/tests/test_picks.py @@ -0,0 +1,94 @@ +"""picks：推荐期的选择、永久去重、核实与增长字段。""" + +import json +from datetime import date + +import pytest + +import picks + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.setattr(picks, "DATA_DIR", tmp_path) + monkeypatch.setattr(picks, "PICKS_HISTORY_FILE", tmp_path / "picks_history.json") + + +def R(name, desc="", topics=None, **kw): + return {"full_name": name, "url": f"https://github.com/{name}", "description": desc, + "topics": topics or [], **kw} + + +def api_ok(path): + name = path.split("/repos/", 1)[1] + return {"full_name": name, "html_url": f"https://github.com/{name}", "description": "d", + "stargazers_count": 100, "forks_count": 5, "topics": []} + + +def test_picks_day_every_three_days(): + assert picks.is_picks_day(date(2026, 10, 16)) + assert not picks.is_picks_day(date(2026, 10, 17)) + assert not picks.is_picks_day(date(2026, 10, 18)) + assert picks.is_picks_day(date(2026, 10, 19)) + + +def test_topic_tagging(): + assert picks.topic_of(R("a/b", "An RTL to GDS flow")) == "semi" + assert picks.topic_of(R("a/b", "LLM agent framework")) == "ai" + assert picks.topic_of(R("a/b", "a retro game port")) is None + # 子串不算：'paint' 里有 'ai'，'credit' 里没有 'eda' 边界 + assert picks.topic_of(R("a/b", "paint program for credit cards")) is None + + +def test_never_repeats_even_after_months(): + picks.record_picks([R("old/one")], "2026-01-01", source="chat") + scored = [(R("old/one", "llm"), {"total": 99}), (R("new/two", "llm"), {"total": 50})] + names = [r["full_name"] for r, _ in picks.select_candidates(scored)] + assert names == ["new/two"] + assert picks.was_ever_picked("OLD/ONE") + + +def test_semi_quota_comes_first_and_suspicious_dropped(): + scored = [(R(f"ai/{i}", "llm agent"), {"total": 100 - i}) for i in range(12)] + scored += [(R("chip/a", "verilog linter"), {"total": 10}), (R("chip/b", "fpga toolkit"), {"total": 9}), + (R("bad/x", "llm", star_suspicious=True), {"total": 100})] + ordered = picks.select_candidates(scored) + chosen = picks.verify_picks(ordered, api_ok) + names = [r["full_name"] for r, _ in chosen] + assert len(names) == picks.PICKS_MAX + assert {"chip/a", "chip/b"} <= set(names) + assert "bad/x" not in names + + +def test_verify_drops_missing_and_archived(): + def api(path): + if path.endswith("gone/x"): + return None + if path.endswith("arch/x"): + return {**api_ok(path), "archived": True} + return api_ok(path) + ordered = [(R("gone/x", topic="ai"), {}), (R("arch/x", topic="ai"), {}), (R("ok/x", topic="ai"), {})] + assert [r["full_name"] for r, _ in picks.verify_picks(ordered, api)] == ["ok/x"] + + +def test_growth_measured_or_null(): + snaps = {"repos": {"a/b": [["2026-10-08", 100], ["2026-10-15", 180]]}} + g = picks.measured_growth("a/b", 200, snaps, "2026-10-16") + assert g == {"since": "2026-10-08", "window_days": 8, "stars_gained": 100, "source": "star_snapshots"} + assert picks.measured_growth("c/d", 50, snaps, "2026-10-16") is None + + +def test_write_picks_shape(tmp_path): + chosen = [(R("a/b", "llm", topic="ai", stars=10), {"total": 90})] + path = picks.write_picks("2026-10-19", chosen, {"repos": {}}) + doc = json.loads(open(path).read()) + assert doc["count"] == 1 and doc["picks"][0]["growth"] is None + assert doc["picks"][0]["url"] == "https://github.com/a/b" + assert doc["semi_min_met"] is False + + +def test_committed_history_seeds_chat_picks(): + import pathlib + real = json.loads((pathlib.Path(__file__).parent.parent / "data" / "picks_history.json").read_text()) + chat = [k for k, v in real["repos"].items() if v.get("source") == "chat"] + assert len(chat) >= 10 diff --git a/tests/test_sources.py b/tests/test_sources.py index 4c3a2e4..a171f22 100644 --- a/tests/test_sources.py +++ b/tests/test_sources.py @@ -388,8 +388,11 @@ class TestAITrendingRotation: @patch("time.sleep") @patch("sources._gh_api", return_value={"items": []}) def test_uses_five_keywords_per_day(self, mock_api, mock_sleep): + """5 个 AI 关键词（近 7 天新建）+ 2 个半导体/工业关键词（近 30 天有推送）。""" sources.fetch_ai_trending() - assert mock_api.call_count == 5 + qs = [c.args[1]["q"] for c in mock_api.call_args_list] + assert sum("created:>" in q for q in qs) == 5 + assert sum("pushed:>" in q for q in qs) == 2 @patch("time.sleep") @patch("sources._gh_api", return_value={"items": []})