[FLINK-40496][table-planner] Apply append-only column rules consistently when altering a materialized table

[AHeise]

What is the purpose of the change

CREATE OR ALTER MATERIALIZED TABLE derived its column changes differently from ALTER ... AS and did not surface a query-driven column reorder to the append-only validation, so reordering existing columns behaved inconsistently: silently applied when the query text was unchanged (rewriting the stored column order with no error), rejected otherwise. The silent path left the stored schema disagreeing with the query and miscompiled the positional refresh INSERT.

Given users_shops = (user_id, shop_id, ds, order_cnt):

CREATE OR ALTER MATERIALIZED TABLE users_shops (shop_id, user_id, ds, order_cnt)
  AS SELECT user_id, shop_id, ds, COUNT(order_id) AS order_cnt FROM ...

The bare column list reorders the columns but the query text is unchanged, so no ModifyDefinitionQuery is produced and the append-only check is skipped — before this change the reorder was applied silently.

Brief change log

• CREATE OR ALTER computes its column diff from the query (via validateAndExtractColumnChanges), positioning old columns by their rank among the columns that survive into the new schema so retained non-persisted columns do not skew the diff.
• The append-only column rules apply to every query-carrying alter (CREATE OR ALTER and ALTER ... AS): existing columns may not be reordered or retyped even when the query text is unchanged. Metadata-only DDL alters are unaffected.
• Second commit: ALTER ... AS and CREATE OR ALTER now share one diff implementation; buildSchemaTableChanges is removed.

Verifying this change

Added and updated tests:

• MaterializedTableStatementITCase — CREATE OR ALTER and ALTER ... AS reject reordering existing columns end-to-end, including a bare-list reorder with an unchanged query.
• SqlMaterializedTableNodeToOperationConverterTest, SqlNodeToOperationSqlCreateOrAlterMaterializedTableConverterTest, AlterMaterializedTableAsQueryOperationValidationTest, ValidateAndExtractColumnChangesTest — operation- and guard-level coverage of the reject/accept cases.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Generated-by: Claude Code (Claude Opus 4.8)

[flinkbot]

CI report:

• 745323d Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build