Add Kafka topic pattern matching and metadata-based dynamic index naming

[Copilot]

Implements regex-based topic discovery and dynamic index naming using Kafka metadata, similar to Logstash's topics_pattern and %{[@metadata][kafka][topic]} functionality.

Changes

Configuration (stash/config/)

• Added TopicsPattern field to KafkaConf for regex-based topic matching
• Implemented FetchMatchingTopics() with broker failover to discover topics dynamically

Metadata Injection (stash/handler/)

• Enhanced MessageHandler to inject Kafka topic name into @metadata.kafka.topic
• Extracted ensureMetadataStructure() helper for cleaner metadata initialization

Index Formatting (stash/es/)

• Added getNestedValue() to support dot-notation field access (e.g., @metadata.kafka.topic)
• Updated buildIndexFormatter() to resolve nested fields in index templates

Main (stash/stash.go)

• Modified toKqConf() to fetch topics via pattern matching
• Updated handler initialization to pass topic context per consumer

Usage

Topic pattern matching:

Input:
  Kafka:
    TopicsPattern: "^app-.*-logs$"  # Matches app-service1-logs, app-service2-logs, etc.

Dynamic index naming:

Output:
  ElasticSearch:
    Index: "{.@metadata.kafka.topic}-{{yyyy-MM-dd}}"  # Creates my-topic-2024-01-15

Both features work independently or together. Existing explicit Topics configurations remain fully supported.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>能实现像logstash中topics_pattern功能么？</issue_title>
<issue_description>目前go-stash只支持配置单个topic,如果有上千个topic,需要一个一个配置还是可以支持正则？
另外，output也可以实现像logstash中的index => "%{[@metadata][kafka][topic]}-%{+YYYY-MM-dd}" 类似功能么？</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes 能实现像logstash中topics_pattern功能么？ #14

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[drings-liu]

这个功能还没集成到最新的版本中么

[Copilot]

Pull request overview

This PR adds Logstash-like Kafka topic pattern discovery (TopicsPattern) and enables dynamic Elasticsearch index naming using nested metadata placeholders (e.g., {.@metadata.kafka.topic}), so indices can be derived from the Kafka topic at ingest time.

Changes:

• Added KafkaConf.TopicsPattern and implemented Kafka topic discovery via regex (FetchMatchingTopics).
• Injected Kafka topic into message metadata (@metadata.kafka.topic) via a topic-aware handler.
• Extended index template resolution to support nested fields via dot notation, with new test coverage.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
stash/stash.go	Fetches topics via pattern matching and instantiates per-topic handlers for metadata injection.
stash/handler/handler.go	Adds topic-aware handler and injects @metadata.kafka.topic.
stash/etc/config.yaml	Documents TopicsPattern and nested metadata index placeholder usage.
stash/es/index.go	Adds nested (dot-notation) field lookup for index template formatting.
stash/es/index_test.go	Adds tests for nested metadata placeholders and getNestedValue.
stash/config/topics.go	Implements regex-based topic discovery via broker metadata.
stash/config/topics_test.go	Adds unit tests for explicit topics, empty pattern, and invalid regex.
stash/config/config.go	Adds TopicsPattern to KafkaConf.
readme.md	Documents TopicsPattern and {.@metadata.kafka.topic} index formatting.
readme-cn.md	Chinese documentation for topic pattern and dynamic index naming.
go.mod	Promotes segmentio/kafka-go to a direct dependency.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[drings-liu]

下载编译后，用TopicsPattern: 配置，提示： [type mismatch for field "Clusters[0].Input.Kafka.Topics"](error: config file etc/test.yaml, type mismatch for field "Clusters[0].Input.Kafka.Topics")