feat: emit durable execution log to stdout at invocation start [SVLS-8582]

[lym953]

Summary

At the beginning of an invocation of a durable function, emit a log like:

{
  "request_id": "123",
  "durable_execution_name": "abc",
  "durable_execution_id": "def",
  "schema_version": "1.0.0"
}

The Lambda extension layer uses this log to map request ID to durable_execution_name and durable_execution_id.

Passing this information using logs instead of trace tags so the architecture is cleaner on extension side: the logs agent can use logs to extract this information and add attribute to other logs, instead of depending on traces and receiving information from trace agent. Discussion: DataDog/datadog-lambda-extension#1053 (comment)

Test plan

1. Passed the added unit tests
2. Build a layer, install it on a function, and invoke the function. The message was logged as expected:

🤖 Generated with Claude Code

[joeyzhao2018]

why not using the logger?

[duncanista]

@joeyzhao2018 probably logger injects other things which we don't want

[lym953]

I think this message should always be logged, and using logger introduces a risk that this message is dropped due to log level. Do you think it's safe to use logger.info()?

[joeyzhao2018]

ah, you mean the "parsing" part would be a big unpredictable? can we use regex to solve that part?
using logger can make sure it still follows the existing rules such as extra injections or if customers want to change the level.

[joeyzhao2018]

oh, i see, you don't want this to be controlled by log level.

[duncanista]

Although your change says "to use logs instead of traces to enrich logs" still creates a dependency where you need a tracing layer to have complete observability in logs – does that even make sense from a dependency standpoint?

[lym953]

The dependency refers to the one between trace agent and log agent in lambda extension. Using traces adds coupling between them, making extension code harder to maintain. See discussion here DataDog/datadog-lambda-extension#1053 (comment)

[joeyzhao2018]

I thought extension side has the information of the request_id

[lym953]

Race conditions often happen (e.g. logs can be delayed), and given a log message, it's hard for the extension to know which request_id it was for, so I think it's safer and easier to add request_id to the log message.

[joeyzhao2018]

i see. that makes sense.

[joeyzhao2018]

why we need a schema version here?

[lym953]

I'm worried that we may need to add more fields in the future, though I'm not sure how yet. Adding schema version will make it easier for extension to identify incompatibility (e.g. missing field) and print an error to ask the user to update tracer.

[joeyzhao2018]

adding more fields shouldn't require a schema. i'm not very convinced that we need it.

[jchrostek-dd]

I would agree that we don't need schema version. If we need to change this in the future, which I think is very unlikely, then we can add schema version at that point.

[joeyzhao2018]

the logs agent can use logs to extract this information and add attribute to other logs

1. Would there be any race conditions such that this piece of log gets processed too late?
2. Would there be any performance impact since you need to look for one specific log? (edit: i look up in the current code, it's already parsing all the logs. so not a big concern here)