Problem
A team turns on the GitHub Issues inbox source, the card reads Synced, the issues table syncs every few hours, and no report ever comes out of it. Nothing in the product says anything went wrong.
The warehouse side is healthy. What dies is the signal-emission child workflow that runs after each sync, and it dies before it can log anything. It has done so on every sync, for every team whose GitHub repository name contains a hyphen, since repo-qualified GitHub schema names landed in #71553.
A community member reported it in Discord ("I have it already setup but it never created a PR from a GH issue yet"). Internal thread: https://posthog.slack.com/archives/C09SK2PAGKF/p1788542132947169
What happens today
After a warehouse sync completes, external-data-job starts emit-data-import-signals-<job_id> on the signals worker (video-export-task-queue). The activity calls data_warehouse_record_fetcher in products/signals/backend/emission/fetchers/data_warehouse.py, which builds the HogQL query by f-string:
query = f"""
SELECT {fields_sql}
FROM {table_name}
WHERE {where_sql}
LIMIT {config.max_records}
"""
table_name comes from get_data_warehouse_table_name(schema.source, schema.table.name). For a GitHub source that is github.<owner>_<repo>__issues, so a repository such as andrewm4894/andys-daily-factoids produces:
FROM github.andrewm4894_andys-daily-factoids__issues
HogQL tokenises the hyphens as subtraction and parse_select raises:
SyntaxError: trailing tokens after expression: '-' (Dash)
parse_select runs before the try/except that logs Error querying new records, so:
- nothing is logged and no PostHog event fires
- Temporal retries the activity three times and the child workflow fails
- the parent sync has already succeeded and advanced
last_synced_at, so the issues in that window are never revisited
The only trace is three Starting signal emission for Github/... lines one to two seconds apart per sync on temporal-worker-video-export, each followed by Querying new records for signal emission and then silence.
The comment above the f-string says none of the data comes from external input. That is no longer true: the customer's repository name is in the table key.
Most enabled GitHub Issues sources have a hyphen in the repository name, and none of those teams has ever emitted a github signal. Teams with plain repository names emit normally.
Repository names with a dot (owner/my.site) probably fail a second way: Database.get_table_node splits the joined name on ., so the key resolves to a three-part chain and Unknown table. That one is inside the try and would be logged. Unverified.
Same shape, different source: Intercom
products/signals/backend/emission/intercom_conversations.py sets:
partition_field="fromUnixTimestamp(toUInt32(created_at))",
toUInt32 is not a HogQL function, so every emission run for the Intercom conversations source fails with:
QueryError: Unsupported function call 'toUInt32(...)'. Perhaps you meant 'toInt(...)'?
This one is logged at error level with team_id (it is inside the try), three attempts per sync, on every team with the source enabled, since #72535.
What we want
- A synced data-import source emits signals whatever the repository name looks like.
- A fetch that cannot even be parsed is logged and counted like a fetch that fails at execution.
- Every registered source's
partition_field and where_clause is known to print through HogQL before it ships.
- Teams that were silently skipped can be caught up without waiting for a new issue to be opened.
Design
Quote the table identifier
In data_warehouse_record_fetcher, split table_name on . and pass each segment through escape_hogql_identifier (posthog/hogql/escape_sql.py), then join with .. The escaper leaves plain identifiers alone and backticks anything else, so existing sources print byte-identically and github.andrewm4894_andys-daily-factoids__issues becomes:
FROM github.`andrewm4894_andys-daily-factoids__issues`
Apply the same to google_search_console_record_fetcher in google_search_console_opportunities.py, which has the same f-string.
Check whether a two-part chain whose second segment contains a dot resolves through Database.get_table_node; if not, the dotted-repo shape needs its own handling and the test below should cover it.
Move parse_select inside the try
So a parse failure reaches logger.exception("Error querying new records: ...") with the run's labels, and the same re-raise semantics apply.
Fix the Intercom partition field
Replace toUInt32 with toInt (or whichever HogQL cast prints for the column's type), and confirm the printed expression against a real Intercom conversations table.
Guard the registry
Add a test that iterates every entry in _SIGNAL_TABLE_CONFIGS, builds the continuous-sync query the fetcher would build (with a placeholder table name), and runs it through parse_select plus print_ast against a HogQL database. Any source with a non-HogQL function in partition_field or where_clause fails the suite instead of failing in prod.
Correct the comment
Replace the "none of the data comes externally" comment with why the identifier is escaped.
Tests
products/signals/backend/emission/tests/test_emit_signals.py: parameterize the existing continuous-sync fetcher test over table names test_table, github.owner_my-repo__issues, and github.owner_my.repo__issues; assert the query parses and the FROM clause is quoted.
- A parse failure (feed a table name that still cannot be quoted, or patch
parse_select to raise) is logged through the same Error querying new records path and re-raised.
- Registry-wide print test as above.
test_github_issues.py / a new Intercom test: the config's partition_field prints through HogQL.
Recovery
python manage.py emit_signals_from_warehouse --team-id <id> --source github --last-synced-at <iso> runs the child workflow for one team without a sync. After the fix ships, run it for affected teams with a --last-synced-at far enough back to cover the time the source was on. first_sync_lookback_days is one day, so nothing catches up by itself.
Acceptance
- A GitHub Issues source on a repository named with a hyphen emits a signal for a newly opened issue on the next sync.
- A parse failure in the fetcher shows up as an error log line carrying
team_id, schema_id and source_type.
- The Intercom conversations source emits on its next sync.
- The registry print test is green for every registered source.
Out of scope
- Surfacing emission failures on the source card (it reads Synced while every emission dies). Worth its own issue.
- A metric or event on emission failure. Same.
References
Problem
A team turns on the GitHub Issues inbox source, the card reads Synced, the issues table syncs every few hours, and no report ever comes out of it. Nothing in the product says anything went wrong.
The warehouse side is healthy. What dies is the signal-emission child workflow that runs after each sync, and it dies before it can log anything. It has done so on every sync, for every team whose GitHub repository name contains a hyphen, since repo-qualified GitHub schema names landed in #71553.
A community member reported it in Discord ("I have it already setup but it never created a PR from a GH issue yet"). Internal thread: https://posthog.slack.com/archives/C09SK2PAGKF/p1788542132947169
What happens today
After a warehouse sync completes,
external-data-jobstartsemit-data-import-signals-<job_id>on the signals worker (video-export-task-queue). The activity callsdata_warehouse_record_fetcherinproducts/signals/backend/emission/fetchers/data_warehouse.py, which builds the HogQL query by f-string:table_namecomes fromget_data_warehouse_table_name(schema.source, schema.table.name). For a GitHub source that isgithub.<owner>_<repo>__issues, so a repository such asandrewm4894/andys-daily-factoidsproduces:HogQL tokenises the hyphens as subtraction and
parse_selectraises:parse_selectruns before thetry/exceptthat logsError querying new records, so:last_synced_at, so the issues in that window are never revisitedThe only trace is three
Starting signal emission for Github/...lines one to two seconds apart per sync ontemporal-worker-video-export, each followed byQuerying new records for signal emissionand then silence.The comment above the f-string says none of the data comes from external input. That is no longer true: the customer's repository name is in the table key.
Most enabled GitHub Issues sources have a hyphen in the repository name, and none of those teams has ever emitted a
githubsignal. Teams with plain repository names emit normally.Repository names with a dot (
owner/my.site) probably fail a second way:Database.get_table_nodesplits the joined name on., so the key resolves to a three-part chain andUnknown table. That one is inside thetryand would be logged. Unverified.Same shape, different source: Intercom
products/signals/backend/emission/intercom_conversations.pysets:toUInt32is not a HogQL function, so every emission run for the Intercom conversations source fails with:This one is logged at error level with
team_id(it is inside thetry), three attempts per sync, on every team with the source enabled, since #72535.What we want
partition_fieldandwhere_clauseis known to print through HogQL before it ships.Design
Quote the table identifier
In
data_warehouse_record_fetcher, splittable_nameon.and pass each segment throughescape_hogql_identifier(posthog/hogql/escape_sql.py), then join with.. The escaper leaves plain identifiers alone and backticks anything else, so existing sources print byte-identically andgithub.andrewm4894_andys-daily-factoids__issuesbecomes:Apply the same to
google_search_console_record_fetcheringoogle_search_console_opportunities.py, which has the same f-string.Check whether a two-part chain whose second segment contains a dot resolves through
Database.get_table_node; if not, the dotted-repo shape needs its own handling and the test below should cover it.Move
parse_selectinside thetrySo a parse failure reaches
logger.exception("Error querying new records: ...")with the run's labels, and the same re-raise semantics apply.Fix the Intercom partition field
Replace
toUInt32withtoInt(or whichever HogQL cast prints for the column's type), and confirm the printed expression against a real Intercomconversationstable.Guard the registry
Add a test that iterates every entry in
_SIGNAL_TABLE_CONFIGS, builds the continuous-sync query the fetcher would build (with a placeholder table name), and runs it throughparse_selectplusprint_astagainst a HogQL database. Any source with a non-HogQL function inpartition_fieldorwhere_clausefails the suite instead of failing in prod.Correct the comment
Replace the "none of the data comes externally" comment with why the identifier is escaped.
Tests
products/signals/backend/emission/tests/test_emit_signals.py: parameterize the existing continuous-sync fetcher test over table namestest_table,github.owner_my-repo__issues, andgithub.owner_my.repo__issues; assert the query parses and theFROMclause is quoted.parse_selectto raise) is logged through the sameError querying new recordspath and re-raised.test_github_issues.py/ a new Intercom test: the config'spartition_fieldprints through HogQL.Recovery
python manage.py emit_signals_from_warehouse --team-id <id> --source github --last-synced-at <iso>runs the child workflow for one team without a sync. After the fix ships, run it for affected teams with a--last-synced-atfar enough back to cover the time the source was on.first_sync_lookback_daysis one day, so nothing catches up by itself.Acceptance
team_id,schema_idandsource_type.Out of scope
References