Skip to content

timestream: every query column is VARCHAR, and neither published endpoint is answered or enforced #1209

Description

@scttfrdmn

What

Timestream's Query reconstructs a result set from stored records rather than evaluating the query,
and the reconstruction reports one column type, one column order and one row shape regardless of what
was asked. Separately, DescribeEndpoints answers neither published endpoint, and neither endpoint is
enforced.

1. Every column is VARCHAR and every value is fmt.Sprintf("%v", v)

The reconstructed ColumnInfo gives every column {"ScalarType": "VARCHAR"} and every Datum a
ScalarValue formatted with %v. AWS publishes eight scalar types (BIGINT, BOOLEAN, DOUBLE,
DATE, TIME, TIMESTAMP, INTERVAL_DAY_TO_SECOND, INTERVAL_YEAR_TO_MONTH) alongside VARCHAR,
and a Timestream table's measure_value::double column is a DOUBLE. A consumer that switches on
ScalarType — which is the documented way to read a Timestream result, since Datum carries no type
of its own — takes the string branch for every column, and a consumer parsing %v output gets Go's
float formatting rather than the JSON number form.

2. Columns are sorted alphabetically, and one shape serves every query

The reconstruction sorts column names alphabetically and emits every dimension and measure it finds.
AWS returns the columns the SELECT list names, in the order it names them. So:

  • SELECT time, measure_value::double FROM t gets columns in alphabetical order, not the requested
    order — and a consumer indexing row.Data[0] reads the wrong column.
  • SELECT count(*) FROM t gets a row per stored record rather than one row with one BIGINT.
  • A WHERE clause is not evaluated, so a filtered query returns everything.
  • GROUP BY, ORDER BY and LIMIT are likewise not evaluated.

This is not a bug to be fixed by implementing SQL — it is the reason seeding exists, and the seed
path is the correct answer for any query whose shape matters. What is wrong is that the unseeded
fallback silently answers a plausible-looking wrong result instead of making its limits visible. A
consumer who forgets to seed gets rows, not an error, and writes an assertion against them.

3. QueryStatus and the row count are fabricated

Query reports a QueryStatus with fixed progress and byte counts unrelated to the result, and
emits "NextToken": "" — an empty string where AWS omits the member, so
while "NextToken" in response never terminates. (The pagination issue in this batch owns the token;
it is noted here because it is in these lines.)

4. DescribeEndpoints answers neither published endpoint, and neither is enforced

AWS publishes two distinct endpoint discovery surfaces —
query.timestream.<region>.amazonaws.com for the Query API and
ingest.timestream.<region>.amazonaws.com for the Write API — and both APIs require endpoint
discovery: a request to the wrong one fails. Substrate answers DescribeEndpoints with neither
published host, and routes Query and WriteRecords identically regardless of which host a request
arrives on. So a consumer whose endpoint-discovery code is broken — sending writes to the query
endpoint, or skipping discovery entirely — passes against substrate and fails against AWS. Endpoint
discovery is unusually failure-prone in real use, which makes it exactly the thing a consumer would
want covered.

Why this matters

CLAUDE.md names Timestream's seedable results as a headline capability — it is one of the four
examples cited for the seed pattern — so the query surface is advertised, which raises the bar for
what the unseeded path may quietly get wrong. A result set with every column typed VARCHAR, in
alphabetical order, ignoring the WHERE clause, is not a degraded answer a consumer can allow for:
it is an answer they will write assertions against, because it looks like a result.

The endpoint half matters for a different reason. Timestream is one of the few AWS services where
using the wrong endpoint is a documented, mandatory failure, so it is one of the few where a
consumer's discovery logic needs testing at all. Substrate currently guarantees that logic passes
whether it is right or wrong.

Acceptance criteria

  • ColumnInfo reports the scalar type the stored value actually has, at least across
    VARCHAR/DOUBLE/BIGINT/BOOLEAN/TIMESTAMP, and ScalarValue is formatted per type
    rather than with %v.
  • The unseeded fallback's limits are observable: either it refuses a query it cannot evaluate
    with Timestream's published ValidationException/400, or QueryStatus reports that the result
    is reconstructed rather than evaluated. Decide which and record the reason — silently answering
    a plausible wrong result is the behaviour to remove.
  • The seed path is documented as the mechanism for any query whose columns, order, or row count
    matter, with a worked example in docs/services.md, following the Athena and RedshiftData seed
    sections.
  • If the SELECT list is parsed at all, columns appear in its order; if it is not, say so rather
    than sorting alphabetically and appearing to have an opinion.
  • QueryStatus' progress and byte counts either reflect the returned result or are documented as
    fixed.
  • DescribeEndpoints answers the published query. or ingest. host for the API it was called
    on, and a request arriving on the wrong one is refused as AWS documents. A test asserts both
    directions.
  • docs/services.md's Timestream section states which column types are reported, what the
    unseeded fallback does, and that endpoint discovery is enforced.

Provenance

The nine ScalarType values, the Datum/ColumnInfo/Row shapes, and the QueryStatus members are
from Timestream Query's API_Query, API_ColumnInfo, API_Datum and API_QueryStatus. The
two-endpoint requirement and the query./ingest. host forms are from
Using-API.endpoint-discovery.html and each API's API_DescribeEndpoints, which state that endpoint
discovery is required. The seed-endpoint pattern this issue points consumers at is CLAUDE.md's, with
Athena and RedshiftData as the in-tree precedents.

In-tree: emulator/timestream_plugin.go — the Query handler, its reconstruction, its QueryStatus,
its "NextToken": "", and describeEndpoints. Line citations are from the tree at the commit this
issue was filed against.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions