[AHeise]

What is the purpose of the change

HttpTableLookupFunction decided whether a response without data yields a row based on whether producedDataType was set. HttpLookupTableSource sets it only in applyReadableMetadata, and for a table without metadata columns the planner calls that only when it pushes a projection. So the same failed call (with http.source.lookup.continue-on-error) or ignored status code behaved differently depending on the query:

Table	Response	Pushed projection	Before	After
no metadata columns	no data (failed, ignored, empty)	no	no row	no row
no metadata columns	no data (failed, ignored, empty)	yes	all-null row	no row
metadata columns	no data	any	row with metadata	row with metadata
any	data	any	row	row

FLINK-38569 stops the planner from recording the empty ReadingMetadata ability, which would have made the planner-dependent path the only one.

Without metadata columns, a response without data is a lookup miss: an inner lookup join drops the row, and a LEFT JOIN pads it with nulls. To see failed calls, declare metadata columns.

Brief change log

1. Decide lookup row emission independently of applyReadableMetadata: a response without data yields a row exactly when metadata columns are declared, and the row is sized from the physical row type. The unused producedDataType is removed from the source and the lookup function. The docs (en + zh) state the inner join / LEFT JOIN behavior explicitly, and that the join key columns of such a metadata row stay null.
2. Retain readable metadata in HttpLookupTableSource#copy: copy() dropped metadataKeys. The source also defines equals/hashCode.

Verifying this change

• HttpTableLookupFunctionTest (new): for every HttpCompletionState, a response without data emits no row without metadata columns, and a row with metadata (all physical columns null) otherwise.
• HttpLookupTableSourceTest#copyRetainsReadableMetadata: copy() equals the source after applyProjection and applyReadableMetadata.
• HttpLookupTableSourceITCaseTest#testHttpLookupJoinParameterized: new bad-status specs without metadata columns for GET/POST/PUT, sync/async, continue-on-error on/off and inner/LEFT JOIN. With continue-on-error, the inner join yields no rows and the LEFT JOIN yields the four probe rows with null enrichment.
• mvn verify -pl flink-connector-http passes on each commit (JDK 17, Flink 2.2.0).

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no (the HttpTableLookupFunction constructor loses its producedDataType parameter; the class is not annotated)
• The serializers: no
• The runtime per-record code paths (performance sensitive): yes (lookup function result assembly; no extra work per record)
• Anything that affects deployment or recovery: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable (docs clarify the existing continue-on-error behavior)

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Generated-by: Claude Code (Opus 5.5)

[davidradl]

@AHeise thanks for the contribution.

So we are issuing a lookup join, we only expect a row if there is a match - there is no a match, no response or there is an error we cannot tell the difference so we treat that as a non match. In this way we only get enrichment rows for the API table when there is something there. It would be unexpected for our user to see rows if there were errors. This is our main use case.

If our user wants to see the complete rows then they can use a LEFT join. The left join allows the user to get rows back in all error cases. The Null enrichment rows are provided by the Flink framework.

The copy change looks good.

[davidradl]

What we were intending for our users is

We would need a config option to introduce any new behaviour; that defaults to the existing behaviour

[AHeise]

@davidradl fair, that is the cleaner contract, and it is planner-independent as well: a response without data yields a row exactly when metadata columns are declared, so an inner join drops failed lookups and a LEFT JOIN pads them with nulls. I reworked the main commit that way, added inner/LEFT JOIN IT specs, and made the docs say it explicitly; the old "lookup join will complete without content in the expected enrichment columns" read as if inner joins kept the row. Description has a before/after table.

[davidradl]

@AHeise I am not convinced with should override the id column with the probe column value if we get a null back.
So the table to be enriched has id column an event with id=1 comes in, we issue a lookup using id=1, but the response gives us a null value in id, since the id here is a response field. In this case I think we should honour the null. In the same way we would honour values that were no null and did not match the request join key value. Or is there some side effect with this approach?

We could add to the docs to say something like: if the user wants to process field names that are the same and the same type and exist in the the API request and response, it is possible to define one column; in this case the response value will be populated after the API call. If the user would like to keep the request and response values separate then the response column should be defined with the name in the response, the path, query param or body field should be mapped from a differently named column name. In this way you can keep both the request and the response values in SQL columns. This is the pattern we use.

[AHeise]

The fill-in predates this PR (it came with #39); I dropped the commit that applied it to metadata rows without response data, so those keep the join key columns null, and the docs say so. For rows with response data the behavior stays as is: honoring the null there is a behavior change that, as you said above, needs an option defaulting to today's behavior. No planner side effect as far as I can tell: with the fill-in disabled entirely, all 121 HttpLookupTableSourceITCaseTest cases pass, only HttpTableLookupFunctionPopulateJoinKeysTest pins it. Happy to open a follow-up for the option plus your docs note on separate request/response columns.