Add analysis-kuromoji plugin documentation

[WojciechKrakowiak]

Description

Added quite complete documentation for analysis-kuromoji plugin, that covers analyzer, tokenizer and all token and character filters.

Documentation was created with help of IBM Bob, with analysis of OpenSearch/Lucene/kuromoji source codes. I carefully reviewed generated docs, verifying, testing and fixing nomenclature, every single example and available parameter. I checked translations and verified that I cannot google generated examples or pieces of documentation (to make sure they aren't taken from other websites).

I did my best to verify the content, though I don't speak japan and any additional verification is welcome (though it also may come later from users).

Version

List the OpenSearch version to which this PR applies, e.g. 2.14, 2.12--2.14, or all.

kuromoji-completion analyzer and tokenizer was added in 3.0 (basing on opensearch-project/OpenSearch#4835)
Other components have over 5 years.

Checklist

• By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
  For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference).

[WojciechKrakowiak]

I fixed all style issues reported by reviewdog that I considered relevant.

Remaining ones are treating names or japanese-related terms as options and I consider them false positive.

[kolchfa-aws]

Thank you, @WojciechKrakowiak! This is great. Trying to find a Japanese-speaking reviewer now.

[lawofcycles]

I will review this from the perspective of a native Japanese speaker. Please wait a moment.

[lawofcycles]

Kuromoji splits 行きました into three tokens (行き, まし, た), so ました is not a single auxiliary verb. kuromoji_baseform normalizes 行き to 行く, and kuromoji_part_of_speech removes まし and た separately.

[lawofcycles]

This reads as if only index mode generates variants. Actually both modes emit variants; query mode additionally merges Kana with trailing lowercase letters (mid-typing IME input, e.g. コンピュ + t → konpyut) so autocomplete matches on in-progress input.

[lawofcycles]

To clarify my earlier comment: both modes emit Romaji variants when given complete text like コンピューター. The actual difference is how they handle partial IME input like コンピュt (Kana + trailing alphabet from mid-typing): index mode splits it into [コンピュ, t], while query mode merges it as コンピュt → konpyut. This is why the pair is needed — index mode indexes completed text, query mode handles in-progress input. The current description doesn't convey this split.

[WojciechKrakowiak]

That's a good point. I think I have it well described on the kuromoji-completion token filter page, I'll simply reuse the same description.

Controls how tokens are generated. Valid values are `index` (default) and `query`. Both modes expand Katakana tokens into their original form plus all Romaji variants. The `query` mode adds two additional behaviors for handling partial IME input: it concatenates consecutive Kana tokens into a single token before romanizing, and it merges a Kana token with a trailing lowercase alphabet token that represents a partially typed IME keystroke (for example, `サッ` followed by `k` becomes `サッk`).

[lawofcycles]

Ex1 (ja_stop only) and Ex2 (full pipeline) produce nearly identical results on this input — both remove な, こと, を, だ. The only observable difference is 学ん vs 学ぶ (from baseform). The claim "leaving ja_stop to handle こと" doesn't hold in this pairing since Ex1 already removes こと.

[WojciechKrakowiak]

I will provide a better example, in which results differ by number of resulting tokens and a baseform of one token.

The sentence `新聞を読んでいるばかりだ` ("I do nothing but read the newspaper") produces the tokens `新聞`, `を`, `読ん`, `で`, `いる`, `ばかり`, `だ`.

In minimal example:

{
  "tokens": [
    {
      "token": "新聞",
      "start_offset": 0,
      "end_offset": 2,
      "type": "word",
      "position": 0
    },
    {
      "token": "読ん",
      "start_offset": 3,
      "end_offset": 5,
      "type": "word",
      "position": 2
    },
    {
      "token": "ばかり",
      "start_offset": 8,
      "end_offset": 11,
      "type": "word",
      "position": 5
    }
  ]
}


`を`, `で`, `いる`, and `だ` are in the `_japanese_` stop set and are removed.

In pipeline example:

{
  "tokens": [
    {
      "token": "新聞",
      "start_offset": 0,
      "end_offset": 2,
      "type": "word",
      "position": 0
    },
    {
      "token": "読む",
      "start_offset": 3,
      "end_offset": 5,
      "type": "word",
      "position": 2
    }
  ]
}

- **`kuromoji_baseform`** normalises the inflected verb stem `読ん` to its dictionary form `読む`.
- **`kuromoji_part_of_speech`** removes `ばかり` (adverbial particle) that would not be handled by `ja_stop`, and removes particles `を`, `で`, and `だ` (which would also be removed by `ja_stop`).
- **`ja_stop`** removes remaining `いる`, which is listed in the built-in Japanese stop set.

[lawofcycles]

食べて is te-form (a conjunctive form), not past tense — past tense would be 食べた. 飲んだ is past tense. Also て is a conjunctive particle, not an auxiliary verb.

[lawofcycles]

"Katakana tokens" is inaccurate — Romaji variants are generated for any Japanese token with reading info, not just Katakana. In the example on L89, the Kanji+Hiragana token 使う produces tukau/tsukau.

[lawofcycles]

To clarify my earlier comment: both modes emit Romaji variants when given complete text like コンピューター. The actual difference is how they handle partial IME input like コンピュt (Kana + trailing alphabet from mid-typing): index mode splits it into [コンピュ, t], while query mode merges it as コンピュt → konpyut. This is why the pair is needed — index mode indexes completed text, query mode handles in-progress input. The current description doesn't convey this split.

[lawofcycles]

The stoptags matching is exact, not prefix. Lucene's JapanesePartOfSpeechStopFilter uses stopTags.contains(pos) (Set equality). An entry of 助詞 does NOT match 助詞-連体化 or 助詞-格助詞-一般. To remove all particles, every subcategory (助詞, 助詞-格助詞, 助詞-格助詞-一般, ...) must be listed explicitly — this is why the default stoptags.txt enumerates them.

[WojciechKrakowiak]

I'll remove this whole note and update the parameter description.

[lawofcycles]

The (の, ...) looks like a typo — the romanized form was likely intended. Also "location marker" / "object marker" are informal; standard terms are "locative" / "accusative".

[lawofcycles]

@WojciechKrakowiak Thank you for well written documentation! I left some comments.

[lawofcycles]

This paragraph and L16 make the same point twice — Kuromoji splits inflected forms and kuromoji_baseform normalizes only the stem. Consider consolidating.

[WojciechKrakowiak]

@lawofcycles thank you for the throughout review! I'll work with it and apply changes :)