Skip to content

Conversation

@suddendust
Copy link
Contributor

@suddendust suddendust commented Dec 28, 2025

Description

This PR introduces a way to get table schema information using the SchemaRegistry interface. This interface provides methods to retrieve a table's metadata from the underlying DB. For PG, this is the query that is used:

postgres=# SELECT "column_name", "udt_name", "is_nullable" FROM information_schema.columns WHERE table_schema = 'public' AND "table_name" = 'myTestFlat';
 column_name  |  udt_name   | is_nullable 
--------------+-------------+-------------
 _id          | int4        | NO
 item         | text        | YES
 price        | int4        | YES
 quantity     | int4        | YES
 date         | timestamptz | YES
 in_stock     | bool        | YES
 tags         | _text       | YES
 categoryTags | _text       | YES
 props        | jsonb       | YES
 sales        | jsonb       | YES
 numbers      | _int4       | YES
 scores       | _float8     | YES
 flags        | _bool       | YES

Cache Refresh and Expiry

This registry fetches and caches column metadata from PostgreSQL's information_schema on demand. It provides:

  1. Lazy loading: Schema metadata is fetched only when first requested for the particular table.
  2. TTL-based caching: Cached schemas expire after a configurable duration (default: 24 hours).
  3. Circuit breaker: Prevents excessive database calls by enforcing a cooldown period between refresh attempts for missing columns (default: 15 minutes).

Testing

[x] Added UTs.
[x] Added integration tests.

Checklist:

  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • Any dependent changes have been merged and published in downstream modules

@codecov
Copy link

codecov bot commented Dec 28, 2025

Codecov Report

❌ Patch coverage is 94.62366% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.47%. Comparing base (e434dfa) to head (602037b).

Files with missing lines Patch % Lines
...documentstore/postgres/PostgresSchemaRegistry.java 84.00% 3 Missing and 1 partial ⚠️
...a/org/hypertrace/core/documentstore/Datastore.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main     #263      +/-   ##
============================================
+ Coverage     80.26%   80.47%   +0.21%     
- Complexity     1337     1380      +43     
============================================
  Files           231      234       +3     
  Lines          6080     6171      +91     
  Branches        545      551       +6     
============================================
+ Hits           4880     4966      +86     
- Misses          826      830       +4     
- Partials        374      375       +1     
Flag Coverage Δ
integration 80.47% <94.62%> (+0.21%) ⬆️
unit 58.54% <93.54%> (+0.52%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@suddendust suddendust changed the title [PoC] Add Schema Registry to Provide Table Metadata Add Schema Registry to Provide Table Metadata Dec 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant