DLQ replication tasks inaccessible when source/target shard counts differ

[NasitSony]

Summary

In convertTaskToDLQTask() in task_processor.go, failed replication
tasks are stored in the DLQ using p.shard.GetShardID() (the local
shard ID). When source and target clusters have different shard counts
(e.g., 1:2 ratio), GetDLQReplicationMessages cannot locate these
tasks because it looks up DLQ entries by a different shard ID mapping.

Root Cause

convertTaskToDLQTask() always uses the local shard ID:

// TODO: GetShardID will break GetDLQReplicationMessages 
// we need to handle DLQ for cross shard replication.
return &persistence.PutReplicationTaskToDLQRequest{
    ShardID: p.shard.GetShardID(),  // always local shard ID

This assumption breaks when source shard count != target shard count.

Impact

• Failed replication tasks stored in DLQ are permanently inaccessible
• GetDLQReplicationMessages (handler.go:1536) cannot retrieve them
• DLQ merging operations silently fail to find affected tasks
• No error is logged — silent data loss under cross-shard replication
• Affects all 4 task types: SYNC_ACTIVITY, HISTORY_V2,
  SYNC_WORKFLOW_STATE, SYNC_HSM

Relationship to #10224

This is related to #10224 (replication tasks not cleaned up with
mismatched shard counts) — same root cause, different symptom:

• Replication tasks not cleaned up on source when source/target shard counts differ (pull replication) #10224: ack level tracking broken → tasks not cleaned up
• This issue: DLQ shard ID wrong → failed tasks permanently lost

Steps to Reproduce

1. Set up two clusters with different shard counts
   • Source: history.numberOfShards = 8192
   • Target: history.numberOfShards = 16384
2. Run workflows that generate replication failures
3. Attempt DLQ merge via GetDLQReplicationMessages
4. Observe: DLQ tasks not found despite being written

Suggested Fix

Similar to the fix in PR #10226, use common.MapShardID to determine
the correct target shard IDs when writing to DLQ, ensuring consistency
with how GetDLQReplicationMessages retrieves them.

References

• service/history/replication/task_processor.go lines 372, 405, 437, 455
• service/history/handler.go lines 1535-1601
• service/history/replication/dlq_handler.go line 301
• Related: Replication tasks not cleaned up on source when source/target shard counts differ (pull replication) #10224, PR [Issue-10224] Fix replication ack propagation across mismatched shard count #10226
  cc @temporalio/server — would appreciate potential-bug label if appropriate

[NasitSony]

While tracing this, the actual write/read key mismatch is in handleReplicationDLQTask (the writeTaskToDLQ call), not the four TODO-marked sites in convertTaskToDLQTask — those ShardID assignments are overwritten downstream by the targetShardID argument that executionManagerDLQWriter persists, so they're effectively dead for the write key.

Concretely: DLQ tasks are written keyed by p.shard.GetShardID() (the local processing shard), but GetDLQReplicationMessages looks them up via config.GetShardID(namespaceId, workflowId) (the workflow hash against this cluster's shard count). When source/target shard counts differ these diverge, so the task is stored under one shard and searched for under another — silently unreachable, no error logged.

Before I build a fix + failing test: is replication across clusters with different shard counts a configuration you intend to support? Multi-cluster replication is marked experimental and the docs don't mention mismatched shard counts, so I want to confirm this is in scope before investing. Happy to put up a PR with a reproducing test if so.

[NasitSony]

Thanks for picking this up! I've been reading through the
replication layer and have some familiarity with the codebase
around this area. Happy to help with tests, documentation,
or any aspect of the fix if that would be useful — just let
me know.