feat(plugin-elasticsearch): add Elasticsearch driver plugin (#1529)#1751
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d2f4891fb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if isTextField(column, fields: fields) { | ||
| return ["match": [column: value]] | ||
| } | ||
| return ["wildcard": [keywordField(column, fields: fields): ["value": "*\(escapeWildcard(value))*", "case_insensitive": true]]] |
There was a problem hiding this comment.
This plugin advertises Elasticsearch 7.x support, but keyword CONTAINS always emits case_insensitive; Elastic documents this wildcard parameter as added in 7.10.0, and the prefix branch below uses the same 7.10-only option, so 7.0-7.9 clusters reject these grid filters with a parsing error. Omit the option or gate it on serverVersion for pre-7.10 clusters.
Useful? React with 👍 / 👎.
| default: | ||
| return equalsClause(column: column, value: value, fields: fields) |
There was a problem hiding this comment.
When the filter UI sends operators already defined by TableFilter such as NOT CONTAINS, NOT IN, BETWEEN, REGEX, IS EMPTY, or IS NOT EMPTY, this falls through to equality instead of either translating or rejecting them. For example a numeric BETWEEN filter is passed as op == "BETWEEN" with value "10,20", which becomes a term query for that string rather than a range, so filtered browsing can return incorrect results without any error.
Useful? React with 👍 / 👎.
| private func encodePathComponent(_ value: String) -> String { | ||
| value.addingPercentEncoding(withAllowedCharacters: .urlPathAllowed) ?? value |
There was a problem hiding this comment.
urlPathAllowed leaves / unescaped, so an Elasticsearch _id such as tenant/123 is encoded into /index/_doc/tenant/123?refresh=true instead of a single path segment. In that case explicit inserts, updates, and deletes for otherwise valid document IDs fail because Elasticsearch sees the extra path component; use a path-component character set that excludes /.
Useful? React with 👍 / 👎.
|
|
||
| static func flattenSource(_ source: [String: Any]) -> [String: PluginCellValue] { | ||
| var result: [String: PluginCellValue] = [:] | ||
| flatten(value: source, prefix: "", into: &result) | ||
| return result |
There was a problem hiding this comment.
For mapped fields whose value is an object but whose mapping column is the parent field itself, such as Elasticsearch flattened fields, columns(forHits:) includes labels from the mapping but flattenSource only emits labels.foo keys and never stores a value for labels. Those columns therefore render as null even when the document contains data; keep the parent JSON value when the mapping expects the parent column, or pass mapping/type context into flattening.
Useful? React with 👍 / 👎.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a344b9e7a9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| var document: [String: Any] = [:] | ||
| for column in columns where !metaColumns.contains(column) { | ||
| guard let value = values[column], let text = value.asText, !text.isEmpty else { continue } |
There was a problem hiding this comment.
When a new row contains .text("") for a non-meta field, this guard treats it like no value and omits the key from the _doc body. Elasticsearch preserves empty strings in _source distinctly from missing fields, and the update path below does send empty strings, so inserting a document with an intentionally blank keyword/text field saves it as absent/null in TablePro. Only skip .null values, not empty text.
Useful? React with 👍 / 👎.
| if let json = response.json as? [String: Any], json["hits"] is [String: Any] { | ||
| let hits = extractHits(response) | ||
| return renderHits(hits, mappingColumns: [], fields: [:], startTime: startTime) |
There was a problem hiding this comment.
In the console, any response with a top-level hits object is forced through renderHits. Elasticsearch search requests with aggs include both hits and aggregations; for common size: 0 aggregation queries this returns an empty grid and discards the aggregation payload entirely, despite the plugin offering an aggs completion. Render raw JSON or otherwise preserve the aggregation/suggest sections when they are present.
Useful? React with 👍 / 👎.
| if let data = text.data(using: .utf8), | ||
| let parsed = try? JSONSerialization.jsonObject(with: data), | ||
| parsed is [Any] || parsed is [String: Any] { | ||
| return parsed |
There was a problem hiding this comment.
This coerces any valid JSON object or array text into a structured value regardless of the mapped column type. If a scalar text, keyword, date, or ip field contains a literal string like { "event": "login" }, saving an edit sends an object/array instead of the original string and Elasticsearch will either reject it as a type conflict or change the stored shape. Gate JSON parsing to structured field types instead of applying it to every non-numeric/non-boolean column.
Useful? React with 👍 / 👎.
…ng, type coercion, aggs)
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6a20db4c0d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| hidesPassword: true, | ||
| visibleWhen: FieldVisibilityRule(fieldId: "esAuthMethod", values: ["apiKey"]) |
There was a problem hiding this comment.
With the current ConnectionField+PasswordHiding helper, any authentication field marked hidesPassword hides the standard password control without checking visibleWhen; because this API-key field is always present in the Elasticsearch auth fields, the default Username & Password mode shows a username box but no password box, making Basic-auth Elasticsearch clusters impossible to configure from a new connection. The registry default declares the same field too, so both descriptors need the same treatment.
Useful? React with 👍 / 👎.
| for: parsed, fields: fields, size: Self.deepPageBatchSize, | ||
| tiebreaker: true, searchAfter: searchAfter, caseInsensitive: supportsCaseInsensitiveSearch | ||
| ) | ||
| body["pit"] = ["id": pit, "keep_alive": Self.pitKeepAlive] |
There was a problem hiding this comment.
When browsing past max_result_window, Elasticsearch can return a new PIT id on each search response and expects the next search_after request to use the most recent id; this loop keeps sending the original pit value and also closes only that original id. On clusters that rotate PIT ids, deep paging can fail or leave the latest PIT context unclosed after the first batch.
Useful? React with 👍 / 👎.
| return session | ||
| } | ||
|
|
||
| guard let url = URL(string: path, relativeTo: baseURL) else { |
There was a problem hiding this comment.
For console input like GET //attacker.example/_search, URL(string:relativeTo:) treats the path as a network-path URL and sends the request to that host while still attaching this connection's Authorization header. The console should be constrained to the configured Elasticsearch base URL, e.g. by rejecting // paths or verifying the resolved host before building the request.
Useful? React with 👍 / 👎.
| case "IS NULL", "IS_NULL", "IS EMPTY", "IS_EMPTY": | ||
| return ["bool": [ | ||
| "should": [mustNot(["exists": ["field": column]]), ["term": [keywordField(column, fields: fields): ""]]], |
There was a problem hiding this comment.
The UI has separate IS NULL and IS EMPTY operators, but this branch makes IS NULL match present empty strings and makes IS NOT NULL exclude them; for keyword/text fields that legitimately store "", the grid counts and rows disagree with the selected operator, and for numeric/date fields the added term for an empty string can make Elasticsearch reject the filter. Only the empty operators should add the empty-string term.
Useful? React with 👍 / 👎.
1 participant
Closes #1529.
Adds Elasticsearch as a registry-only driver plugin (7.x/8.x), pure-Swift over the REST API with no C bridge.
What it does
_id/_index/_scoremeta columns. Nestedobject/nestedfields flatten to dotted columns; arrays render as JSON cells.METHOD /pathplus an optional JSON body. Search responses tabularize;_cat/array responses become a grid; everything else shows formatted JSON.textfield targets its.keywordsub-field;_idand other non-sortable fields are skipped so the query doesn't error._update/_doc), keyed by_id.index.max_result_window(10k) usessearch_afterwith a point-in-time.Approach
DSL-first, matching how mature ES clients (Kibana, Elasticvue, Dejavu, EDQL) work. ES SQL (
_sql) is read-only and blind to arrays/nested fields, so it is not the foundation. Built on the existing tagged-query dispatch pattern (like DynamoDB).Included fixes
asPluginFilterTupleforwardedvalue, but a raw filter stores its text inrawSQL). This broke raw filters on every NoSQL plugin, not just Elasticsearch. The Elasticsearch plugin maps the raw filter to aquery_stringfull-text search.ORDER BYto the plugin query string when sorting; the plugin now parses and applies it as an ES sort.Integration
DatabaseType.elasticsearch, new pbxproj bundle target,build-plugin.ymlcase,PluginMetadataRegistryseed (so it shows in the picker pre-install), icon imageset, docs page, CHANGELOG, version bump to 0.53.0 (the release this ships in).Tests
Unit tests for the console parser, query builder + DSL, mapping flattener, statement generator, the appended-
ORDER BYparsing, the__RAW__mapping, and theasPluginFilterTupleraw-filter fix.Notes
scripts/install-plugin-dev.shinstalls a local build for testing.