[Feature] Support per-event-type configurable log levels for event log

[weiqingy]

Search before asking

• I searched in the issues and found nothing similar.

Description

Summary

Add per-event-type configurable log levels to the event log system, as identified in the 0.3 planning discussion (#516).

Motivation

The current event log system only supports binary accept/reject filtering via EventFilter. There is no way to:

• Configure different log levels for different event types (e.g., VERBOSE for ChatRequestEvent, OFF for ContextRetrievalRequestEvent)
• Distinguish between standard and verbose logging detail (both behave identically)
• Filter logged events by level or event type in downstream tools without parsing nested JSON

Changes

• New EventLogLevel enum: OFF, STANDARD, VERBOSE
• Per-event-type levels configured via EventLoggerConfig builder and string-based config options
• STANDARD truncates long string fields (configurable eventLogMaxFieldLength, default 1024); VERBOSE logs full content
• Top-level eventType field in JSON output for easier downstream filtering
• Startup validation warns about unrecognized event type names in config
• Backward compatible: existing configs and JSON format continue to work

Config Options

Key	Default	Description
eventLogLevel	STANDARD	Global default level (OFF, STANDARD, VERBOSE)
eventLogLevels	(none)	Per-type overrides. Format: ChatRequestEvent=VERBOSE,InputEvent=OFF
eventLogMaxFieldLength	1024	Max chars for string fields at STANDARD level. 0 disables truncation.

Related

• Discussion Planning Flink Agents 0.3 #516 (Planning Flink Agents 0.3)

Are you willing to submit a PR?

• I'm willing to submit a PR!

[xintongsong]

Hi @weiqingy, Thanks for working on this.

The event log enhancement is an important feature that directly affects user experience, and is also relatively complex. I think this deserves a design doc, based on which we can first discuss and align on the expected behaviors before moving to the implementation. Would you like to open a discussion thread?

Based on the issue description, I have a few comments:

• I like the idea of truncating long fields when the level is STANDARD. In addition to string fields, I think long lists or deeply nested structures can also be truncated. Maybe we should define the semantic of STANDARD as "details might be omitted to keep the logs concise". This would give us the flexibility to gradually optimize the truncating strategy w/o breaking the semantics.
  • This also means the per-string-field length limitation, which exposes the truncating strategy, might not be ideal. I'd suggest an per-event overall length limitation.
• For per-class log level, a single config option is probably not enough.
  • Imagine you have configured a list of log levels in config.yaml, which is shared by all your jobs. Then for a certain job you want to overwrite the config option in the job submitting command, setting a certain class to VERBOSE for debugging. In this case, you can only overwrite the entire log level list, which means either include everything from config.yaml in the command, or lose the log levels configured in config.yaml.
  • I think we should introduce a config key pattern. E.g., event-log.level.foo.bar: OFF|STANDARD|VERBOSE, where foo.bar` is the event type. In this way, each class will have its own option that can be overwritten separately.

WDYT?

[weiqingy]

Hi @xintongsong, thanks for the thoughtful review! Great points across the board. Let me address each:

• Design doc / discussion: Absolutely, I'll open a discussion thread with the design doc so we can align before moving to implementation.
• Broader truncation semantics for STANDARD: Agreed. Defining STANDARD as "details might be omitted to keep the logs concise" is a better semantic - it gives us room to evolve the truncation strategy (e.g., truncating long lists, capping nested depth) without breaking the contract. I'll update the design to use this framing.
• Per-event overall length limit vs per-field: Makes sense. Exposing per-string-field length ties the config too tightly to the truncation implementation. I'll replace eventLogMaxFieldLength with a per-event overall length limit (e.g., event-log.max-length). The internal strategy for how we truncate to fit that limit can evolve independently.
• Config key pattern for per-class levels: This is a great call. The single comma-separated eventLogLevels option has exactly the override composability problem you described. I'll switch to a config key pattern like:

  event-log.level.ChatRequestEvent: VERBOSE
  event-log.level.ContextRetrievalRequestEvent: OFF
  

  This way each event type has its own independently overridable config key, and the global default can remain as event-log.level.default: STANDARD.

I'll update the design doc with these changes and open a discussion thread. Thanks again for the feedback.

[weiqingy]

Discussion thread: #552