Skip to content

fix(ingest): 파싱 리뷰 Stage 3 — MED/LOW 폴리시 (#9~13·#17·#18·#20~22 + display) - #140

Merged
hang-in merged 2 commits into
mainfrom
fix/parsing-stage3-polish
Jul 6, 2026
Merged

fix(ingest): 파싱 리뷰 Stage 3 — MED/LOW 폴리시 (#9~13·#17·#18·#20~22 + display)#140
hang-in merged 2 commits into
mainfrom
fix/parsing-stage3-polish

Conversation

@hang-in

@hang-in hang-in commented Jul 6, 2026

Copy link
Copy Markdown
Owner

파싱 리뷰(23 confirmed)의 마지막 스테이지 — MED/LOW 폴리시. 8-에이전트 병렬 수정, 중앙 검증.

수정

드롭 (별도 재설계)

검증

  • cargo test 742 pass (신규 15+: injected_context / tool_only / nontext_part / earliest_ts / code_content / user_text / ansi_stripped / multibyte 등).
  • clippy --workspace --all-targets (-Dwarnings) clean, fmt clean.

후속 (별건)

머지 전

  • CI green + CodeRabbit 확인

Summary by CodeRabbit

  • Bug Fixes
    • 시작 시간이 누락된 경우 가장 이른 메시지 시각으로 표시됩니다.
    • Gemini 토큰 누적과 Gemini Web/Claude/ChatGPT/Opencode 인제스트가 더 정확해졌고, 도구 호출 시 사용자 텍스트와 도구 결과가 함께 유지됩니다.
    • ANSI 및 제어 문자를 제거해 가독성이 개선됩니다.
  • New Features
    • 도구 사용 내역 반영 범위가 확대되었습니다.
  • Refactor
    • 세션 ID 단축/표시를 문자 경계 안전 방식으로 통일했습니다.

파싱 리뷰(23 confirmed) Stage 3 — 파서 정확성/노이즈 폴리시.

- #9 (MED) codex.rs: 주입 컨텍스트 user 메시지(AGENTS.md/<environment_context>/
  <user_instructions>/<permissions>) skip (developer 와 동일).
- #20 (LOW) codex.rs: 파일명 fallback id 를 rollout-<timestamp>-<uuid> 에서
  trailing uuid(8-4-4-4-12) 추출 (타임스탬프 오염 제거).
- #10 (MED) opencode.rs: tool 파트(type=tool, state.input/output)를 Action::ToolUse
  로 매핑 + tool-only 어시스턴트 메시지도 턴으로 보존.
- #11 (MED) gemini_web.rs: 비텍스트 파트/누락 필드 시 대화 전체 drop → text
  Option + best-effort 로 lenient 파싱.
- #12 (MED) gemini.rs: startTime 없으면 earliest 턴 ts 로 (now() 오날짜 방지).
- #21 (LOW) gemini.rs: per-turn 토큰을 total_tokens 로 누적.
- #13 (MED) chatgpt.rs: content_type "code" 를 text 필드에서 읽도록 (빈 추출 방지).
- #22 (LOW) chatgpt.rs: create_time null 시 earliest 메시지 ts fallback.
- #17 (LOW) claude.rs: user 메시지의 tool_result+text 공존 시 text 보존.
- #18 (LOW) claude.rs: tool 출력의 ANSI/제어 escape strip (strip_ansi).
- display byte-slice: classify/graph/lint/recall/wiki/output 의 id[..8] →
  chars().take(8) (비ASCII panic 방지, CodeRabbit Major 잔여분).

드롭: #14/#15 (markdown parse_session_turns) — 제안된 render-이스케이프 방식이
기존 vault 파일(코드블록 내 미이스케이프 헤더) reindex 시 phantom turn 을 유발해
백워드 호환 불가. 별도 재설계 필요(기존 fence 추적 유지).

검증: cargo test 742 pass(신규 15+), clippy -Dwarnings clean.
후속(별건): codex fast_dedup_key 를 #20 fallback_session_id 와 정합(최적화).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVEL1SnCQBtUMeWVy99ebj
@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 02e10f8c-c3d3-416d-8802-1ac82dc5f5ba

📥 Commits

Reviewing files that changed from the base of the PR and between dcbbc9c and c7f6ccc.

📒 Files selected for processing (3)
  • crates/secall-core/src/ingest/codex.rs
  • crates/secall-core/src/ingest/gemini.rs
  • crates/secall/src/commands/ingest.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/secall-core/src/ingest/gemini.rs

📝 Walkthrough

Walkthrough

여러 인제스트 파서에서 콘텐츠 추출, 토큰 집계, start_time 폴백, 도구 결과 처리 로직을 수정했고, CLI 여러 명령과 출력에서 세션 ID 8자 축약을 문자 단위로 바꿨다.

Changes

인제스트 파서 수정

Layer / File(s) Summary
ChatGPT 콘텐츠 추출 및 start_time 폴백
crates/secall-core/src/ingest/chatgpt.rs
content_type: "code"text 필드 기반으로 읽도록 분리하고, conv.create_time이 없을 때 체인 내 가장 이른 메시지 시각으로 start_time을 폴백하도록 변경했으며 테스트를 추가했다.
Claude tool_result 사용자 텍스트 보존 및 ANSI 제거
crates/secall-core/src/ingest/claude.rs
tool_result 처리 후 함께 담긴 사용자 텍스트를 별도 User 턴으로 보존하고, 도구 출력의 ANSI/제어문자를 strip_ansi로 제거했으며 테스트를 추가했다.
Codex 세션 ID 폴백 및 인젝션 컨텍스트 스킵
crates/secall-core/src/ingest/codex.rs
파일명에서 UUID tail을 추출하는 fallback_session_id를 도입하고, Codex가 주입한 합성 컨텍스트 메시지를 감지해 스킵하는 로직과 테스트를 추가했다.
Gemini 토큰 누적 및 start_time 폴백
crates/secall-core/src/ingest/gemini.rs
세션 단위 total_tokens 누적 로직과 startTime 부재 시 최초 턴 timestamp 폴백을 추가하고, 관련 테스트를 보강했다.
Gemini Web non-text 파트 처리
crates/secall-core/src/ingest/gemini_web.rs
text 필드를 Optional로 바꿔 non-text 파트로 인한 역직렬화 실패를 방지하고, 콘텐츠 결합 시 누락/빈 텍스트를 제외하도록 변경했으며 테스트를 추가했다.
OpenCode 도구 액션 파싱
crates/secall-core/src/ingest/opencode.rs
tool 파트를 Action::ToolUse로 변환하는 로직을 추가하고, tool-only assistant 메시지도 turn으로 보존하도록 변경했으며 테스트를 추가했다.

세션 ID 표시 유니코드 안전화

Layer / File(s) Summary
short_id 헬퍼 도입 및 output.rs 적용
crates/secall/src/output.rs
short_id 헬퍼를 추가해 print_ingest_result에서 사용하고, ASCII/멀티바이트 입력 검증 테스트를 추가했다.
CLI 커맨드 세션 ID 축약 로직 통일
crates/secall/src/commands/classify.rs, crates/secall/src/commands/graph.rs, crates/secall/src/commands/lint.rs, crates/secall/src/commands/recall.rs, crates/secall/src/commands/wiki.rs
여러 커맨드의 로그와 출력에서 세션 ID 앞 8자 표시를 chars().take(8) 기반으로 통일했다.

Estimated code review effort: 3 (Moderate) | ~30 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed 제목이 파싱 리뷰 Stage 3의 MED/LOW 정책 및 display 관련 변경을 잘 요약합니다.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/parsing-stage3-polish

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request improves robustness across multiple ingest parsers (ChatGPT, Claude, Codex, Gemini, Gemini Web, and OpenCode) by handling missing timestamps, synthetic contexts, non-text message parts, and tool execution actions. It also fixes a potential panic when slicing multibyte session IDs by switching from byte-slicing to character-based slicing. The feedback suggests pre-truncating large tool outputs in the Claude parser before stripping ANSI escape sequences to prevent regex performance bottlenecks.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines 361 to +382
fn extract_tool_result_content(content: &Value) -> String {
if content.is_string() {
return content.as_str().unwrap_or("").to_string();
}
if let Some(arr) = content.as_array() {
let parts: Vec<String> = arr
.iter()
let raw = if content.is_string() {
content.as_str().unwrap_or("").to_string()
} else if let Some(arr) = content.as_array() {
arr.iter()
.filter_map(|item| {
if item["type"].as_str() == Some("text") {
item["text"].as_str().map(String::from)
} else {
None
}
})
.collect();
return parts.join("\n");
}
String::new()
.collect::<Vec<_>>()
.join("\n")
} else {
String::new()
};

// Tool output (PowerShell/Bash) frequently carries ANSI color/cursor escapes
// and stray control bytes. Strip them before the content is stored/truncated.
strip_ansi(&raw)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

도구 출력(예: 빌드 로그, 대용량 파일 출력 등)이 매우 클 경우, strip_ansi에서 전체 문자열에 대해 정규식 매칭 및 문자 단위 필터링을 수행하면 심각한 성능 저하(CPU 및 메모리 낭비)가 발생할 수 있습니다. TOOL_OUTPUT_MAX_CHARS가 1000이므로, ANSI 이스케이프 시퀀스를 제거하기 전에 raw 문자열을 적절한 크기(예: 5000자)로 먼저 자른 후 strip_ansi를 호출하는 것이 효율적입니다.

fn extract_tool_result_content(content: &Value) -> String {
    let raw = if content.is_string() {
        content.as_str().unwrap_or("").to_string()
    } else if let Some(arr) = content.as_array() {
        arr.iter()
            .filter_map(|item| {
                if item["type"].as_str() == Some("text") {
                    item["text"].as_str().map(String::from)
                } else {
                    None
                }
            })
            .collect::<Vec<_>>()
            .join("\n")
    } else {
        String::new()
    };

    // Tool output (PowerShell/Bash) frequently carries ANSI color/cursor escapes
    // and stray control bytes. Strip them before the content is stored/truncated.
    // Pre-truncate to prevent regex performance bottlenecks on huge outputs.
    let limited = truncate_str(&raw, 5000);
    strip_ansi(&limited)
}

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
crates/secall/src/commands/wiki.rs (1)

1054-1059: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

turn.content 트렁케이션을 문자 경계 기준으로 바꿔야 합니다.

turn.content는 자유 텍스트라 &turn.content[..1000]가 멀티바이트 문자에서 패닉할 수 있습니다. crates/secall/src/commands/wiki.rs:1055-1059build_haiku_single_project_prompt의 동일한 패턴(crates/secall/src/commands/wiki.rs:1387-1391)을 chars().take(1000) 같은 방식으로 맞춰주세요.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/secall/src/commands/wiki.rs` around lines 1054 - 1059, The snippet
truncation in turn.content uses byte slicing and can panic on multibyte text;
update the snippet-building logic in the affected wiki command codepaths,
including the same pattern in build_haiku_single_project_prompt, to truncate by
character boundary instead. Replace the direct &turn.content[..1000] approach
with a chars()-based limit (or equivalent safe Unicode-aware truncation) so the
snippet stays valid for arbitrary free text while preserving the existing
1000-character behavior.
🧹 Nitpick comments (4)
crates/secall-core/src/ingest/opencode.rs (1)

161-188: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

툴 요약에 길이 상한을 두세요. input/output를 그대로 input_summary/output_summary에 담고 chunker.rs::build_turn_text가 이를 그대로 이어붙입니다. 대용량 툴 출력이 들어오면 검색 텍스트가 불필요하게 비대해질 수 있으니, output_summary(필요하면 input_summary도)에 문자 수 제한을 두는 편이 좋습니다.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/secall-core/src/ingest/opencode.rs` around lines 161 - 188,
`opencode.rs`의 `Action::ToolUse` 생성 시 `input_summary`와 특히 `output_summary`가
`state.input`/`state.output` 전체를 그대로 반영하지 않도록 제한을 추가하세요. `value_to_summary` 결과를
`build_turn_text`로 넘기기 전에 일정 문자 수로 잘라 대용량 툴 출력이 검색 텍스트를 과도하게 키우지 않게 하고, 필요하면
`input_summary`에도 같은 상한을 적용하세요. `msg.parts`를 순회해 `Action::ToolUse`를 만드는 로직을 중심으로
수정하면 됩니다.
crates/secall/src/output.rs (1)

17-23: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

short_id 로직 정상 동작, 다만 5개 커맨드 파일에서 동일 로직이 중복 구현됨.

id.chars().take(8).collect() 구현 자체는 UTF-8 경계 안전성 관점에서 올바릅니다. 다만 classify.rs, graph.rs, lint.rs, recall.rs, wiki.rs에서 동일한 s.chars().take(8).collect::<String>() 로직을 각각 인라인으로 재구현하고 있습니다. 이 함수를 pub으로 노출(혹은 secall-core의 공용 유틸로 이동)하고 나머지 커맨드에서 재사용하면 향후 이런 종류의 수정이 필요할 때 한 곳만 고치면 됩니다.

♻️ 제안: 공용 헬퍼 노출
-fn short_id(id: &str) -> String {
+pub fn short_id(id: &str) -> String {
     id.chars().take(8).collect()
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/secall/src/output.rs` around lines 17 - 23, The UTF-8-safe 8-character
prefix logic in short_id is correct, but the same
s.chars().take(8).collect::<String>() code is duplicated across classify, graph,
lint, recall, and wiki command files. Expose short_id as a shared public helper
(or move it into a common secall-core utility) and update those command
implementations to call it instead of inlining the logic, using short_id as the
unique symbol to locate the reusable code.
crates/secall-core/src/ingest/chatgpt.rs (1)

182-195: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

"code"/"execution_output" 분기에 parts 폴백 부재.

text 필드가 비어있거나 없는 경우 빈 문자열을 반환하며 parts 기반 폴백이 없습니다. 실제 ChatGPT 익스포트에서 일부 code 페이로드가 parts 형식으로 올 가능성이 있다면 콘텐츠 유실이 발생할 수 있습니다. _ 기본 분기처럼 textcontent 순 폴백 체인을 적용하는 것도 고려해볼 만합니다.

♻️ 폴백 강화 제안
         "code" | "execution_output" => message
             .content
             .get("text")
             .and_then(|v| v.as_str())
-            .unwrap_or_default()
-            .to_string(),
+            .map(ToOwned::to_owned)
+            .or_else(|| {
+                message
+                    .content
+                    .get("parts")
+                    .and_then(|v| v.as_array())
+                    .map(|parts| extract_parts(parts))
+            })
+            .unwrap_or_default(),
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/secall-core/src/ingest/chatgpt.rs` around lines 182 - 195, The
"code"/"execution_output" handling in the ChatGPT ingest logic is too strict
because it only reads the text field and drops content when that field is
missing or empty. Update the match arm in ingest/chatgpt.rs around the message
content parsing to add a fallback chain like the default text handling, so it
can fall back from text to parts (using extract_parts) and preserve payloads
that arrive in parts form. Keep the fix localized to the existing
message.content parsing branch for "code" and "execution_output".
crates/secall-core/src/ingest/gemini_web.rs (1)

82-90: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

모든 파트가 non-text인 경우 빈 콘텐츠 턴이 생성될 수 있음

filter_mapfilter(!is_empty)를 통과한 뒤 파트가 하나도 남지 않으면 content는 빈 문자열이 됩니다. session_repo::insert_turnturn.content를 그대로 DB에 저장하므로, 텍스트가 전혀 없는 메시지(예: 이미지/thought만 있는 assistant 응답)가 빈 콘텐츠 턴으로 저장될 수 있습니다. 이런 턴을 필터링하거나 플레이스홀더 텍스트(예: "[non-text content]")로 대체하는 것을 고려해볼 수 있습니다.

♻️ 예시 개선안 (턴 자체를 제외)
     let turns: Vec<Turn> = export
         .messages
         .into_iter()
         .enumerate()
-        .map(|(idx, msg)| {
+        .filter_map(|(idx, msg)| {
             let role = match msg.msg_type.as_str() {
                 "user" => Role::User,
                 "gemini" => Role::Assistant,
                 _ => Role::System,
             };
             let content = match msg.content {
                 GeminiWebContent::Text(s) => s,
                 GeminiWebContent::Parts(parts) => parts
                     .into_iter()
                     .filter_map(|p| p.text)
                     .filter(|t| !t.is_empty())
                     .collect::<Vec<_>>()
                     .join("\n"),
             };
+            if content.is_empty() {
+                return None;
+            }
             let timestamp = DateTime::parse_from_rfc3339(&msg.timestamp)
                 .map(|dt| dt.with_timezone(&Utc))
                 .ok();
-            Turn {
+            Some(Turn {
                 index: idx as u32,
                 role,
                 timestamp,
                 content,
                 actions: Vec::new(),
                 tokens: None,
                 thinking: None,
                 is_sidechain: false,
-            }
+            })
         })
         .collect();
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/secall-core/src/ingest/gemini_web.rs` around lines 82 - 90, The Gemini
web ingest path can produce an empty content string when all Parts are non-text,
which then gets stored as a blank turn by session_repo::insert_turn. Update the
GeminiWebContent::Parts handling in gemini_web.rs so that msg.content is either
filtered out entirely when no text remains or replaced with a clear placeholder
before it reaches insert_turn, and keep the logic localized around the content
match branch.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/secall-core/src/ingest/codex.rs`:
- Around line 77-84: fast_dedup_key still normalizes Codex session filenames
differently than the new session_id extraction, so dedup checks can miss rollout
files. Update the downstream normalization in fast_dedup_key to use the same
fallback rule as fallback_session_id, extracting the trailing UUID from
rollout-<timestamp>-<uuid> filenames instead of only stripping rollout-. Prefer
sharing the same helper or matching logic between
crates/secall-core/src/ingest/codex.rs and crates/secall/src/commands/ingest.rs
so session.id and dedup keys stay consistent.

In `@crates/secall-core/src/ingest/gemini.rs`:
- Around line 178-187: Add overflow/range checks around token accumulation and
persistence: in gemini.rs, the token updates on total_tokens.input/output/cached
should use checked_add and return an error instead of allowing u64 wrap or
panic; then ensure session_repo.rs validates token totals before casting to i64
so values above i64::MAX are rejected rather than persisted incorrectly. Use the
token handling paths in the gemini ingest flow and the session repository save
logic to locate the changes.

---

Outside diff comments:
In `@crates/secall/src/commands/wiki.rs`:
- Around line 1054-1059: The snippet truncation in turn.content uses byte
slicing and can panic on multibyte text; update the snippet-building logic in
the affected wiki command codepaths, including the same pattern in
build_haiku_single_project_prompt, to truncate by character boundary instead.
Replace the direct &turn.content[..1000] approach with a chars()-based limit (or
equivalent safe Unicode-aware truncation) so the snippet stays valid for
arbitrary free text while preserving the existing 1000-character behavior.

---

Nitpick comments:
In `@crates/secall-core/src/ingest/chatgpt.rs`:
- Around line 182-195: The "code"/"execution_output" handling in the ChatGPT
ingest logic is too strict because it only reads the text field and drops
content when that field is missing or empty. Update the match arm in
ingest/chatgpt.rs around the message content parsing to add a fallback chain
like the default text handling, so it can fall back from text to parts (using
extract_parts) and preserve payloads that arrive in parts form. Keep the fix
localized to the existing message.content parsing branch for "code" and
"execution_output".

In `@crates/secall-core/src/ingest/gemini_web.rs`:
- Around line 82-90: The Gemini web ingest path can produce an empty content
string when all Parts are non-text, which then gets stored as a blank turn by
session_repo::insert_turn. Update the GeminiWebContent::Parts handling in
gemini_web.rs so that msg.content is either filtered out entirely when no text
remains or replaced with a clear placeholder before it reaches insert_turn, and
keep the logic localized around the content match branch.

In `@crates/secall-core/src/ingest/opencode.rs`:
- Around line 161-188: `opencode.rs`의 `Action::ToolUse` 생성 시 `input_summary`와 특히
`output_summary`가 `state.input`/`state.output` 전체를 그대로 반영하지 않도록 제한을 추가하세요.
`value_to_summary` 결과를 `build_turn_text`로 넘기기 전에 일정 문자 수로 잘라 대용량 툴 출력이 검색 텍스트를
과도하게 키우지 않게 하고, 필요하면 `input_summary`에도 같은 상한을 적용하세요. `msg.parts`를 순회해
`Action::ToolUse`를 만드는 로직을 중심으로 수정하면 됩니다.

In `@crates/secall/src/output.rs`:
- Around line 17-23: The UTF-8-safe 8-character prefix logic in short_id is
correct, but the same s.chars().take(8).collect::<String>() code is duplicated
across classify, graph, lint, recall, and wiki command files. Expose short_id as
a shared public helper (or move it into a common secall-core utility) and update
those command implementations to call it instead of inlining the logic, using
short_id as the unique symbol to locate the reusable code.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 560d3962-a3ee-4633-9083-52f47531a024

📥 Commits

Reviewing files that changed from the base of the PR and between 57b35b6 and dcbbc9c.

📒 Files selected for processing (12)
  • crates/secall-core/src/ingest/chatgpt.rs
  • crates/secall-core/src/ingest/claude.rs
  • crates/secall-core/src/ingest/codex.rs
  • crates/secall-core/src/ingest/gemini.rs
  • crates/secall-core/src/ingest/gemini_web.rs
  • crates/secall-core/src/ingest/opencode.rs
  • crates/secall/src/commands/classify.rs
  • crates/secall/src/commands/graph.rs
  • crates/secall/src/commands/lint.rs
  • crates/secall/src/commands/recall.rs
  • crates/secall/src/commands/wiki.rs
  • crates/secall/src/output.rs

Comment thread crates/secall-core/src/ingest/codex.rs
Comment on lines +178 to +187
// tokens — 턴별 사용량을 세션 total_tokens 에 누적 (claude.rs 미러)
let tokens = msg.tokens.as_ref().map(|t| {
total_tokens.input += t.input;
total_tokens.output += t.output;
total_tokens.cached += t.cached;
TokenUsage {
input: t.input,
output: t.output,
cached: t.cached,
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# TokenUsage 저장/캐스팅 계약 확인
rg -n -C3 'total_tokens\.(input|output).*as i64|tokens_in|tokens_out|struct TokenUsage' crates/secall-core/src

Repository: hang-in/seCall

Length of output: 18917


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '--- gemini.rs outline ---'
ast-grep outline crates/secall-core/src/ingest/gemini.rs --view expanded | sed -n '1,220p'

echo
echo '--- gemini.rs around target lines ---'
cat -n crates/secall-core/src/ingest/gemini.rs | sed -n '150,230p'

echo
echo '--- types.rs TokenUsage definition ---'
cat -n crates/secall-core/src/ingest/types.rs | sed -n '90,120p'

echo
echo '--- session_repo.rs write/read token paths ---'
cat -n crates/seall-core/src/store/session_repo.rs | sed -n '45,75p'
echo
cat -n crates/seall-core/src/store/session_repo.rs | sed -n '540,655p'

Repository: hang-in/seCall

Length of output: 6443


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '--- gemini.rs outline ---'
ast-grep outline crates/secall-core/src/ingest/gemini.rs --view expanded | sed -n '1,220p'

echo
echo '--- gemini.rs around target lines ---'
cat -n crates/secall-core/src/ingest/gemini.rs | sed -n '150,230p'

echo
echo '--- types.rs TokenUsage definition ---'
cat -n crates/secall-core/src/ingest/types.rs | sed -n '90,120p'

echo
echo '--- session_repo.rs write/read token paths ---'
cat -n crates/secall-core/src/store/session_repo.rs | sed -n '45,75p'
echo
cat -n crates/secall-core/src/store/session_repo.rs | sed -n '540,655p'

Repository: hang-in/seCall

Length of output: 12801


토큰 누적/저장에 범위 검사를 추가하세요. crates/secall-core/src/ingest/gemini.rs:178-182에서 u64를 그대로 += 누적하면 overflow 시 wrap/panic이 날 수 있고, crates/secall-core/src/store/session_repo.rs:67-68as i64 저장 때문에 i64::MAX를 넘는 값은 잘못된 토큰 값으로 persisted 됩니다. checked_add로 에러를 반환하고, 저장 전 i64 범위를 막아 주세요.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/secall-core/src/ingest/gemini.rs` around lines 178 - 187, Add
overflow/range checks around token accumulation and persistence: in gemini.rs,
the token updates on total_tokens.input/output/cached should use checked_add and
return an error instead of allowing u64 wrap or panic; then ensure
session_repo.rs validates token totals before casting to i64 so values above
i64::MAX are rejected rather than persisted incorrectly. Use the token handling
paths in the gemini ingest flow and the session repository save logic to locate
the changes.

…eRabbit Major)

- codex fast_dedup_key 를 fallback_session_id 와 공유(pub화) — #20 이후 저장 id 가
  후행 uuid 인데 dedup 힌트는 rollout- 만 벗겨 timestamp-uuid 로 어긋나 codex 세션이
  매번 재인제스트되던 문제. 이제 동일 규칙(trailing uuid)으로 일치.
- gemini 토큰 누적을 saturating_add 로 — 비현실적 대용량 합에서 u64 overflow
  wrap/panic 방지.

검증: cargo test 742 pass, clippy -Dwarnings clean. fast_dedup_key 타임스탬프 uuid
테스트 갱신.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVEL1SnCQBtUMeWVy99ebj
@hang-in
hang-in merged commit 3f4973a into main Jul 6, 2026
4 checks passed
hang-in added a commit that referenced this pull request Jul 6, 2026
v0.6.5 이후 28커밋 minor 릴리즈.

- version 0.6.5 → 0.7.0 (Cargo.toml + lock)
- CHANGELOG.md v0.7.0 섹션
- README ×4: 기본 임베딩 bge-m3 → qwen3-embedding:0.6b 정정(이력 보존) + ko Web UI 신규 기능

하이라이트: Web UI 대개편(#137/#128/#129/#132) · Graph Insights(#126)/AA(#124) ·
ingest 파싱 서브시스템 리뷰 21건(#138/#139/#140) · do_get 복구(#135) · 성능(#134/#136).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant