Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ Every day it collects signals from 6 data sources, runs them through a smart sco
| GitHub Search | New & rising | Repos created in the last 7 days with fast star growth |
| [Hacker News](https://news.ycombinator.com/) | Community picks | GitHub repos from Show HN posts |
| Rising Detection | Early signal | Repos created in the last 3 days with a fork/star ratio above 0.3 and at most 0.5 — forks as an early usage signal; higher ratios are discarded as fork farms |
| AI/ML Keyword Sweep | AI focus | GitHub Search over an AI/ML keyword list, 5 keywords per day on a rotating schedule (inspired by [OSSInsight trending/ai](https://ossinsight.io/trending/ai), implemented against the GitHub Search API) |
| AI/ML Keyword Sweep | AI focus | GitHub Search over an AI/ML keyword list, 5 keywords per day on a rotating schedule, plus 2 semiconductor / industrial-AI keywords per day over repos pushed in the last 30 days (inspired by [OSSInsight trending/ai](https://ossinsight.io/trending/ai), implemented against the GitHub Search API) |
| [HF Daily Papers](https://huggingface.co/papers) | Research signal | GitHub repos linked from trending Hugging Face papers — we assume paper upvotes lead GitHub stars by days, but no committed measurement backs that lead time yet |

### Smart Scoring (100 points)
Expand All @@ -104,11 +104,22 @@ Every day it collects signals from 6 data sources, runs them through a smart sco
- **Star fraud detection**: age ≤ 1 day with 1000+ stars, age ≤ 2 days with 2000+ stars, or 1000+ stars/day with an empty description → flagged
- **Batch fraud detection**: same owner with ≥ 3 repos in one batch, or ≥ 2 repos under 7 days old already past 200 stars → flagged
- **Content quality**: No description or no README → penalty
- **Cross-day dedup**: 7-day window, no duplicate recommendations across days (a same-day rerun may repeat a pick)
- **Cross-day dedup**: 7-day window for the daily report (a same-day rerun may repeat a pick)
- **Picks never repeat**: a repo that has appeared in a picks issue (or was recommended in chat) is never picked again — see `data/picks_history.json`

### Picks (every 3 days)

Collection still runs daily; every `PICKS_EVERY_DAYS` (3) days `scripts/picks.py` publishes `data/picks-YYYY-MM-DD.json`:

- 8–10 repos (`PICKS_MIN`/`PICKS_MAX`), at least `SEMI_MIN` (2) tagged semiconductor / industrial AI when the pool has them (`semi_min_met` says whether it did); AI-tagged repos rank ahead of untagged ones
- every pick is re-read from `GET /repos/{owner}/{repo}` right before publishing; missing, archived or disabled repos are dropped
- facts only: stars, forks, topics, dates and the sources that surfaced it; `growth` is a measured delta from the committed star snapshots, or `null` — never a lifetime average
- no repeat performers, and no LLM in the pipeline; fewer than 8 verified candidates means no issue that day
- `picks-never-repeat` and `picks-count-in-range` in `docs/claims.json` re-check every committed issue

### Email Subscription

- Daily curated repos delivered to your inbox
- Each picks issue (the same 8–10 repos) is delivered to your inbox; non-picks days send no email
- Rendered dark-only, with `color-scheme` declared for dark-aware clients (Apple Mail / iOS)
- Powered by Resend API

Expand Down Expand Up @@ -245,6 +256,7 @@ github-discovery/
│ ├── quality.py # Code quality detection
│ ├── anti_spam.py # Anti-spam scoring dimension
│ ├── dedup.py # Cross-day deduplication (7-day window)
│ ├── picks.py # Picks issue every 3 days (permanent dedup, facts only)
│ ├── fraud_detection.py # Batch fraud detection
│ ├── snapshots.py # Daily star snapshots (real growth)
│ ├── verify_scoring.py # Scoring verification / backtesting
Expand Down Expand Up @@ -324,7 +336,7 @@ have since moved.

**Do I need any paid service?** Only for email. `RESEND_API_KEY` sends the digest through Resend; `FIRECRAWL_API_KEY` is optional and only makes GitHub Trending parsing more robust than regex-matching raw HTML. Everything else runs on the GitHub Actions free tier with the default `GITHUB_TOKEN`.

**Will the same repo be recommended every day while it is hot?** No. Cross-day dedup blocks any repo recommended in the previous 7 days, using the history committed at `data/recommend_history.json`; records older than 30 days are cleaned up. The history has to be a committed file because every CI run starts from a fresh checkout with no local state.
**Will the same repo be recommended every day while it is hot?** No. Picks never repeat a repo (permanent history in `data/picks_history.json`). The daily report's cross-day dedup blocks any repo recommended in the previous 7 days, using the history committed at `data/recommend_history.json`; records older than 30 days are cleaned up. The history has to be a committed file because every CI run starts from a fresh checkout with no local state.

**Can I trust the numbers on the published page?** The page is a build artifact: `scripts/generate_site.py` renders `docs/index.html` from `docs/template.html` plus the committed reports, and the workflow publishes `docs/` to the `gh-pages` branch. The authoritative copies are the committed JSON under `data/`, which is what `verify_scoring.py` reads; the Markdown digests under `output/` are the human-readable copies, and the site reads the JSON first and falls back to those digests. So any claim on the page can be recomputed from a fresh clone.

Expand Down
20 changes: 16 additions & 4 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ GitHub Discovery 告诉你什么**即将火起来**——增长曲线异常的
| GitHub Search | 新晋上升 | 最近 7 天创建、星标增长迅猛的仓库 |
| [Hacker News](https://news.ycombinator.com/) | 社区精选 | Show HN 帖子中出现的 GitHub 仓库 |
| Rising Detection | 早期信号 | 最近 3 天创建、fork/star 比大于 0.3 且不超过 0.5 的仓库——fork 作为真实使用的早期信号;比率更高的按 fork farm 直接排除 |
| AI/ML 关键词轮扫 | AI 方向 | 用一组 AI/ML 关键词扫 GitHub Search,每天轮换 5 个词(思路来自 [OSSInsight trending/ai](https://ossinsight.io/trending/ai),实现走的是 GitHub Search API) |
| AI/ML 关键词轮扫 | AI 方向 | 用一组 AI/ML 关键词扫 GitHub Search,每天轮换 5 个词,另加每天 2 个半导体/工业 AI 关键词(看近 30 天有推送的仓库)(思路来自 [OSSInsight trending/ai](https://ossinsight.io/trending/ai),实现走的是 GitHub Search API) |
| [HF Daily Papers](https://huggingface.co/papers) | 研究信号 | Hugging Face 热门论文关联的 GitHub 仓库——我们假设论文点赞领先 GitHub 星标几天,但仓库里还没有可提交的测量来支撑这个领先时间 |

### 智能评分(满分 100 分)
Expand All @@ -104,11 +104,22 @@ GitHub Discovery 告诉你什么**即将火起来**——增长曲线异常的
- **刷星检测**:年龄 ≤ 1 天且 1000+ 星、年龄 ≤ 2 天且 2000+ 星,或日增 1000+ 星且描述为空 → 标记
- **批量刷量检测**:同一作者一批里出现 ≥ 3 个仓库,或 ≥ 2 个上线不足 7 天的仓库已过 200 星 → 标记
- **内容质量**:没有描述或没有 README → 扣分
- **跨日去重**:7 天窗口内跨天不重复推荐(同日重跑可能重复选到同一个仓库)
- **跨日去重**:每日报告 7 天窗口内跨天不重复推荐(同日重跑可能重复选到同一个仓库)
- **推荐期永不重复**:进过推荐期(或在聊天里推过)的仓库永不再推,见 `data/picks_history.json`

### 推荐期(每 3 天一期)

采集仍是每天一次;每 `PICKS_EVERY_DAYS`(3)天,`scripts/picks.py` 发布一期 `data/picks-YYYY-MM-DD.json`:

- 每期 8–10 个(`PICKS_MIN`/`PICKS_MAX`),候选里有的话至少 `SEMI_MIN`(2)个半导体/工业 AI 仓库(是否达标见 `semi_min_met`);带 AI 标签的排在无主题的前面
- 发布前逐个用 `GET /repos/{owner}/{repo}` 重新读取,不存在、已归档或已禁用的剔除
- 只放事实:star、fork、topics、日期和发现它的数据源;`growth` 是按已提交 star 快照算出的实测增量,拿不到就是 `null`,绝不用终身平均值顶替
- 不带 repeat,流程里不接大模型;核实后不足 8 个当天就不出刊
- `docs/claims.json` 里的 `picks-never-repeat` 和 `picks-count-in-range` 会复核每一期

### 邮件订阅

- 每日精选仓库直达你的收件箱
- 每期推荐(同一份 8–10 个)直达你的收件箱;非推荐日不发邮件
- 只渲染深色版本,并声明 `color-scheme` 供支持深色的客户端使用(Apple Mail / iOS)
- 基于 Resend API 发送

Expand Down Expand Up @@ -198,6 +209,7 @@ github-discovery/
│ ├── quality.py # Code quality detection
│ ├── anti_spam.py # Anti-spam scoring dimension
│ ├── dedup.py # Cross-day deduplication (7-day window)
│ ├── picks.py # Picks issue every 3 days (permanent dedup, facts only)
│ ├── fraud_detection.py # Batch fraud detection
│ ├── snapshots.py # Daily star snapshots (real growth)
│ ├── verify_scoring.py # Scoring verification / backtesting
Expand Down Expand Up @@ -274,7 +286,7 @@ python scripts/verify_scoring.py --days 30

**需要付费服务吗?** 只有邮件需要。`RESEND_API_KEY` 用于通过 Resend 发送日报;`FIRECRAWL_API_KEY` 是可选的,只是让 GitHub Trending 的解析比正则匹配原始 HTML 更健壮。其余部分都跑在 GitHub Actions 免费额度上,用默认的 `GITHUB_TOKEN` 即可。

**一个仓库火着的时候会天天被推荐吗?** 不会。跨日去重会拦掉过去 7 天内推荐过的仓库,依据是提交在 `data/recommend_history.json` 里的推荐历史,超过 30 天的记录会被清理。这份历史必须是提交进仓库的文件,因为每次 CI 都是全新 checkout,没有任何本地状态。
**一个仓库火着的时候会天天被推荐吗?** 不会。推荐期永不重复推同一个仓库(永久历史见 `data/picks_history.json`);每日报告的跨日去重会拦掉过去 7 天内推荐过的仓库,依据是提交在 `data/recommend_history.json` 里的推荐历史,超过 30 天的记录会被清理。这份历史必须是提交进仓库的文件,因为每次 CI 都是全新 checkout,没有任何本地状态。

**网页上的数字可信吗?** 网页是构建产物:`scripts/generate_site.py` 用 `docs/template.html` 加上已提交的报告渲染出 `docs/index.html`,工作流再把整个 `docs/` 发布到 `gh-pages` 分支。真正权威的是提交在 `data/` 下的 JSON,`verify_scoring.py` 读的就是它;`output/` 下的 Markdown 日报是给人看的副本,站点优先读 JSON、读不到才回退到日报。所以网页上的任何结论都能在全新 clone 上重算一遍。

Expand Down
15 changes: 15 additions & 0 deletions data/picks_history.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"repos": {
"openai/math": {"first_picked": "2026-10-16", "source": "chat"},
"NandhaKishorM/laya": {"first_picked": "2026-10-16", "source": "chat"},
"morluto/rea": {"first_picked": "2026-10-16", "source": "chat"},
"zai-org/ZCode": {"first_picked": "2026-10-16", "source": "chat"},
"feder-cr/git-ai": {"first_picked": "2026-10-16", "source": "chat"},
"anthropics/oss-scanner": {"first_picked": "2026-10-16", "source": "chat"},
"rociiu/talorys": {"first_picked": "2026-10-16", "source": "chat"},
"chenwei810/Semiconductor_Manufacturing_Introduction": {"first_picked": "2026-10-16", "source": "chat"},
"hdl-tools/digital-chip-design-agents": {"first_picked": "2026-10-16", "source": "chat"},
"BLANK2077/xverif": {"first_picked": "2026-10-16", "source": "chat"}
},
"updated_at": "2026-10-16T17:00:00-07:00"
}
2 changes: 1 addition & 1 deletion docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
- **100-point scorer** — acceleration (40) + quality (30) + anti-spam (30), with code-quality bonus, suspicious-star and batch-fraud penalties, and an explainable reason line on every card
- **Cross-day dedup** with a 7-day window, deep checks budgeted to the top-K candidates after coarse ranking
- **Delivery** — daily email digest via Resend (dark-mode HTML) + GitHub Pages site with date/language filters
- **Current quality baseline** — 202 tests and 20 executable receipts; the site, feed and daily JSON are generated from committed reports. The presentation pass (light/dark themes, two-band header, merged report header, lead-card treatment) is complete; further UI work is gated by a user task or a measured defect.
- **Current quality baseline** — 210 tests and 22 executable receipts; the site, feed and daily JSON are generated from committed reports. The presentation pass (light/dark themes, two-band header, merged report header, lead-card treatment) is complete; further UI work is gated by a user task or a measured defect.

## Long-term operating plan

Expand Down
42 changes: 40 additions & 2 deletions docs/claims.json
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,9 @@
"verified": "2026-09-26",
"check": {
"cmd": "python3 - <<'PY'\nimport glob, json\npaths = sorted(glob.glob('data/candidate-pool-*.json'))\nif not paths:\n print('candidate pool: not-ready (no production pool yet)')\n raise SystemExit(0)\npath = paths[-1]\ndoc = json.load(open(path, encoding='utf-8'))\nrows = doc.get('candidates') or []\nallowed = {'candidate', 'deduplicated', 'content_blocked', 'eligible', 'recommended'}\nnames = [r.get('full_name') for r in rows]\nproblems = []\nif len(names) != len(set(names)): problems.append('duplicate names')\nfor row in rows:\n if not row.get('full_name'): problems.append('missing name')\n if not (row.get('sources') or row.get('source')): problems.append('missing source')\n if row.get('pool_status') not in allowed: problems.append('bad status')\nprint('candidate pool', path, 'rows', len(rows), 'problems', sorted(set(problems)))\nassert rows and not problems\nPY",
"expect": {"regex": "candidate pool: not-ready|candidate pool .* problems \\[\\]"},
"expect": {
"regex": "candidate pool: not-ready|candidate pool .* problems \\[\\]"
},
"timeout": 60
}
},
Expand All @@ -228,7 +230,9 @@
"as_of": "2026-09-26",
"check": {
"cmd": "python3 -c \"import re,subprocess,pathlib;out=subprocess.run(['python3','-m','pytest','--collect-only','-q'],capture_output=True,text=True).stdout;got=int(re.search(r'(\\d+) tests? collected', out).group(1));doc=int(re.search(r'(\\d+) tests', pathlib.Path('docs/ROADMAP.md').read_text()).group(1));print('pytest', got, 'roadmap', doc);assert got==doc\"",
"expect": {"regex": "pytest \\d+ roadmap \\d+$"},
"expect": {
"regex": "pytest \\d+ roadmap \\d+$"
},
"timeout": 180
}
},
Expand Down Expand Up @@ -351,6 +355,40 @@
},
"timeout": 120
}
},
{
"id": "picks-never-repeat",
"claim": "No repository appears in more than one picks issue, every pick is recorded in the permanent history, and none repeats a repository already recommended in chat.",
"value": "0 repeats",
"metric": "duplicate full_name across data/picks-*.json, plus picks missing from or colliding with data/picks_history.json",
"method": "scripts/picks.py filters candidates against data/picks_history.json, which is never pruned; this receipt re-derives the property from the committed files alone.",
"repro": "ls data/picks-*.json",
"evidence": "data/picks_history.json",
"as_of": "2026-10-16",
"check": {
"cmd": "python3 - <<'PY'\nimport glob, json\npaths = sorted(glob.glob('data/picks-*.json'))\nhist = json.load(open('data/picks_history.json', encoding='utf-8'))['repos']\nseen, problems = {}, []\nfor path in paths:\n for p in json.load(open(path, encoding='utf-8'))['picks']:\n k = p['full_name'].lower()\n if k in seen: problems.append(f\"{p['full_name']} in {seen[k]} and {path}\")\n seen[k] = path\n if p['full_name'] not in hist: problems.append(f\"{p['full_name']} missing from history\")\n elif hist[p['full_name']].get('source') == 'chat': problems.append(f\"{p['full_name']} repeats a chat pick\")\nprint('picks files', len(paths), 'repos', len(seen), 'problems', problems)\nassert not problems\nPY",
"expect": {
"regex": "problems \\[\\]$"
},
"timeout": 60
}
},
{
"id": "picks-count-in-range",
"claim": "Every picks issue holds 8 to 10 repositories with GitHub URLs, integer star counts, and a growth field that is either a measured snapshot delta or null.",
"value": "8-10 per issue",
"metric": "len(picks) per data/picks-*.json and the shape of its url, stars and growth fields",
"method": "scripts/picks.py publishes an issue only when at least PICKS_MIN verified repositories exist and never substitutes lifetime averages for growth; this receipt checks every committed issue.",
"repro": "ls data/picks-*.json",
"evidence": "data/picks-*.json",
"as_of": "2026-10-16",
"check": {
"cmd": "python3 - <<'PY'\nimport glob, json\npaths = sorted(glob.glob('data/picks-*.json'))\nproblems = []\nfor path in paths:\n d = json.load(open(path, encoding='utf-8'))\n n = len(d['picks'])\n if not 8 <= n <= 10 or d['count'] != n: problems.append(f'{path}: count {n}')\n for p in d['picks']:\n if not p['url'].startswith('https://github.com/'): problems.append(f\"{p['full_name']}: url\")\n if not isinstance(p['stars'], int): problems.append(f\"{p['full_name']}: stars\")\n g = p['growth']\n if g is not None and (g.get('source') != 'star_snapshots' or not isinstance(g.get('stars_gained'), int)):\n problems.append(f\"{p['full_name']}: growth not measured\")\nprint('picks files', len(paths), 'problems', problems)\nassert not problems\nPY",
"expect": {
"regex": "problems \\[\\]$"
},
"timeout": 60
}
}
]
}
27 changes: 26 additions & 1 deletion scripts/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@
from fraud_detection import detect_batch_fraud, apply_fraud_penalty
from snapshots import record_snapshots, get_growth
from watchlist import add_discoveries
import picks as picks_mod
from sources import _gh_api
from snapshots import _load as load_snapshots


def get_subscribers() -> list[str]:
Expand Down Expand Up @@ -586,7 +589,24 @@ def main():
# been sent (or benignly skipped). If the email fails, we exit non-zero
# without writing the marker, so the next scheduled run retries.
date_str = datetime.now().strftime("%Y-%m-%d")
email_result = send_digest_email(date_str, top_new)

# ── 推荐期(picks):每 PICKS_EVERY_DAYS 天一期,邮件和聊天共用这一份 ──
# 非推荐日只采集(报告/站点/快照照常),不发邮件。
chosen = []
if picks_mod.is_picks_day():
ordered = picks_mod.select_candidates(scored)
chosen = picks_mod.verify_picks(ordered, _gh_api)
if len(chosen) < picks_mod.PICKS_MIN:
print(f"[Picks] only {len(chosen)} verified candidates (< {picks_mod.PICKS_MIN}); "
"no picks issue today, no email.")
chosen = []
else:
print("[Picks] not a picks day — collect only, no email.")

if chosen:
email_result = send_digest_email(date_str, chosen)
else:
email_result = "skipped"
if email_result == "failed":
print("[ERROR] Email send failed — not writing today's report so the "
"next run retries. Exiting non-zero to flag the failure.")
Expand All @@ -599,6 +619,11 @@ def main():
f.write(md)
print(f"\n[Saved] Report written to {out_path}")

if chosen:
picks_path = picks_mod.write_picks(date_str, chosen, load_snapshots())
picks_mod.record_picks([r for r, _ in chosen], date_str)
print(f"[Saved] Picks written to {picks_path}")

# Persist source health only after ranking, dedup and recommendation selection are complete.
set_source_recommendations([repo for repo, _scores in top_new + top_repeat])
source_health = get_source_health()
Expand Down
Loading
Loading