Skip to content

fix: normalize OpenAI citation markers #815

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alphaxiv-bot merged 2 commits into from
Oct 28, 2026
Merged

fix: normalize OpenAI citation markers #815

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
028e051
fix: normalize OpenAI citation markers
paperclover Oct 28, 2026
8bbc6c4
Merge branch 'main' into clo/normalize-openai-citation-markers
alphaxiv-bot[bot] Oct 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion ui/src/markdownNormalization.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,17 +232,23 @@ export function normalizeCurrencyDollars(

function normalizeProse(text: string, options: NormalizationOptions): string {
let normalizedMath = text
.replace(/\uE200cite\uE202([^\uE200\uE201]*)\uE201/g, (_, sources: string) =>
sources.split("\uE202")
.filter((source) => /^https?:\/\/[^\s<>]+$/i.test(source))
.map((source) => `<${source}>`)
.join(" "))
Comment on lines +235 to +239

@greptile-apps greptile-apps Bot Oct 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Citation URLs can change

A completed citation for https://example.org/prices/$100 becomes <https://example.org/prices/\$100>. This replacement creates the link, then normalizeCurrencyDollars inserts a backslash. Backslashes are literal inside Markdown autolinks, so clicking the citation opens a different URL.

Keep citation URLs out of the later math and currency changes, and add a test for a URL containing a numeric dollar amount.

Prompt To Fix With AI 
This is a comment left during a code review.
Path: ui/src/markdownNormalization.ts
Line: 235-239

Comment:
**Citation URLs can change**

A completed citation for `https://example.org/prices/$100` becomes `<https://example.org/prices/\$100>`. This replacement creates the link, then `normalizeCurrencyDollars` inserts a backslash. Backslashes are literal inside Markdown autolinks, so clicking the citation opens a different URL.

Keep citation URLs out of the later math and currency changes, and add a test for a URL containing a numeric dollar amount.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

.replace(/\\\[([\s\S]+?)\\\]/g, (_, inner: string) => `$$${inner}$$`)
.replace(/\\\(([\s\S]+?)\\\)/g, (_, inner: string) => `$$${inner}$$`);
if (options.predictMath) {
normalizedMath = normalizedMath
.replace(/\uE200(?:c(?:i(?:t(?:e(?:\uE202[^\uE200\uE201]*)?)?)?)?)?$/, "")
.replace(/\\\[([\s\S]*)$/, (_, inner: string) => `$$${inner}`)
.replace(/\\\(([\s\S]*)$/, (_, inner: string) => `$$${inner}`);
}
return normalizeCurrencyDollars(normalizedMath, options);
}

/** Normalize math and legacy currency while leaving code and mention tags opaque. */
/** Normalize citations, math, and legacy currency while leaving code and mention tags opaque. */
export function normalizeMarkdownForRendering(
text: string,
options: NormalizationOptions = {},
Expand Down
33 changes: 33 additions & 0 deletions ui/tests/markdownNormalization.test.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
import assert from "node:assert/strict";
import test from "node:test";
import { normalizeMarkdownForRendering } from "../src/markdownNormalization.ts";

test("renders citation URLs and removes unresolved internal references", () => {
assert.equal(
normalizeMarkdownForRendering("结果。\uE200cite\uE202turn421719view1\uE201"),
"结果。",
);
assert.equal(
normalizeMarkdownForRendering("Result. \uE200cite\uE202https://www.alphaxiv.org/pdf/2604.26417\uE201"),
"Result. <https://www.alphaxiv.org/pdf/2604.26417>",
);
assert.equal(
normalizeMarkdownForRendering("\uE200cite\uE202turn1view0\uE202https://example.org/a\uE202https://example.org/b\uE201"),
"<https://example.org/a> <https://example.org/b>",
);
});

test("hides incomplete citation markers while streaming", () => {
const citation = "\uE200cite\uE202https://example.org/paper\uE201";
for (let end = 1; end < citation.length; end += 1) {
assert.equal(normalizeMarkdownForRendering(`Result. ${citation.slice(0, end)}`, { predictMath: true }), "Result. ");
}
assert.equal(normalizeMarkdownForRendering(`Result. ${citation}`, { predictMath: true }), "Result. <https://example.org/paper>");
assert.equal(normalizeMarkdownForRendering("\uE200cider", { predictMath: true }), "\uE200cider");
});

test("leaves citation syntax in code and mention tags untouched", () => {
const citation = "\uE200cite\uE202turn1view0\uE201";
const markdown = `\`${citation}\`\n\n\`\`\`\n${citation}\n\`\`\`\n\n<file path="${citation}">`;
assert.equal(normalizeMarkdownForRendering(markdown), markdown);
});
Loading