diff --git a/README.md b/README.md index ffbc6a8..7798fef 100644 --- a/README.md +++ b/README.md @@ -160,6 +160,27 @@ embedder: Termite auto-selects the best available variant if not specified. +### T5Gemma-2 (Multimodal) + +T5Gemma-2 is Google's multimodal encoder-decoder model supporting text embeddings, image embeddings, and text generation. + +```bash +# Export model to ONNX (requires uv and HuggingFace login) +./scripts/export_t5gemma2.py \ + --model google/t5gemma-2-270m-270m \ + --output ~/.termite/models/rewriters/google/t5gemma-2-270m + +# Use for embeddings +curl -X POST http://localhost:8080/v1/embeddings \ + -d '{"model": "google/t5gemma-2-270m", "input": ["Hello world"]}' + +# Use for text generation +curl -X POST http://localhost:8080/api/rewrite \ + -d '{"model": "google/t5gemma-2-270m", "texts": ["Summarize: ..."]}' +``` + +See [docs/T5GEMMA2.md](docs/T5GEMMA2.md) for full setup instructions. + ## Kubernetes Operator Deploy on GKE with TPU support using the Termite Operator. diff --git a/docs/T5GEMMA2.md b/docs/T5GEMMA2.md new file mode 100644 index 0000000..f026791 --- /dev/null +++ b/docs/T5GEMMA2.md @@ -0,0 +1,301 @@ +# T5Gemma-2 Model Support + +T5Gemma-2 is Google's multimodal encoder-decoder model from the Gemma 3 family, trained with the UL2 objective. This document explains how to export and run T5Gemma-2 models in Termite. + +## Features + +- **Text Embeddings**: 640-dimensional vectors from the encoder +- **Image Embeddings**: 640-dimensional vectors from SigLIP vision encoder +- **Text Generation**: Seq2seq generation for summarization, Q&A, rewriting +- **Multimodal**: Combined text + image understanding + +## Prerequisites + +### 1. Install uv (Python Package Manager) + +The export script uses [uv](https://github.com/astral-sh/uv) for hermetic, reproducible Python environments: + +```bash +# macOS +brew install uv + +# Linux/WSL +curl -LsSf https://astral.sh/uv/install.sh | sh +``` + +### 2. About optimum-onnx (Optional) + +**The export script is self-contained** - it applies ONNX compatibility patches manually and does NOT require optimum-onnx. + +If you prefer using `optimum-cli export onnx` instead of our script, we have a fork with T5Gemma-2 support: + +```bash +# Only needed if using optimum-cli (not our export script) +git clone https://github.com/timkaye11/optimum-onnx.git +cd optimum-onnx +git checkout add-t5gemma2-support +pip install -e . + +# Then export with optimum-cli +optimum-cli export onnx --model google/t5gemma-2-270m-270m ./output +``` + +The fork adds: +- `T5Gemma2OnnxConfig` in `optimum/exporters/onnx/model_configs.py` +- Removes transformers version upper bound (T5Gemma-2 requires transformers >= 4.58.0) + +### 3. HuggingFace Access + +T5Gemma-2 models require accepting Google's license on HuggingFace: + +1. Go to https://huggingface.co/google/t5gemma-2-270m-270m +2. Accept the license agreement +3. Get a HuggingFace token from https://huggingface.co/settings/tokens + +```bash +# Login to HuggingFace +huggingface-cli login +``` + +## Exporting Models + +### Quick Start + +```bash +cd termite + +# Export the 270M model (recommended for development, ~6.7GB) +# Uses default output: ~/.termite/models/rewriters/google/t5gemma-2-270m +./scripts/export_t5gemma2.py --model google/t5gemma-2-270m-270m + +# Or export with a test to verify the output +./scripts/export_t5gemma2.py --model google/t5gemma-2-270m-270m --test +``` + +### Available Models + +| Model | Total Params | Size | Use Case | +|-------|--------------|------|----------| +| `google/t5gemma-2-270m-270m` | ~800M | ~6.7GB | Development, testing | +| `google/t5gemma-2-1b-1b` | ~2B | ~15GB | Balanced quality/speed | +| `google/t5gemma-2-4b-4b` | ~8B | ~50GB | Best quality | + +### Export Options + +```bash +./scripts/export_t5gemma2.py --help + +Options: + --model MODEL HuggingFace model ID (required) + --output DIR Output directory for ONNX files (required) + --test Run validation tests after export + --test-input TEXT Custom test input for generation + --skip-encoder Skip encoder export (if already exported) + --skip-decoder Skip decoder export (if already exported) + --skip-vision Skip vision encoder export +``` + +### Exported Files + +After export, the output directory contains: + +``` +~/.termite/models/rewriters/google/t5gemma-2-270m/ +├── encoder.onnx # Text encoder (1.07 GB) +├── decoder.onnx # Decoder with KV cache (1.74 GB) +├── decoder-init.onnx # Decoder for first token (1.74 GB) +├── vision_encoder.onnx # SigLIP vision encoder (1.67 GB) +├── embedding_layer.onnx # Word embeddings fallback (671 MB) +├── config.json # HuggingFace model config +├── t5gemma2_config.json # Termite-specific config +├── tokenizer.json # SentencePiece tokenizer (33 MB) +├── tokenizer_config.json # Tokenizer settings +└── preprocessor_config.json # Vision preprocessing +``` + +## Running in Termite + +### Configuration + +Set the rewriters directory in your termite config: + +```yaml +# config.yaml +rewriters_dir: ~/.termite/models/rewriters +keep_alive: 5m +max_loaded_models: 2 +``` + +Or via environment variable: + +```bash +export TERMITE_REWRITERS_DIR=~/.termite/models/rewriters +``` + +### Starting the Server + +```bash +# Build with ONNX support +CGO_ENABLED=1 go build -tags="onnx,ORT" -o termite ./pkg/termite/cmd + +# Run +./termite run --config config.yaml +``` + +### Verifying Model Discovery + +```bash +# List available models +curl http://localhost:8080/api/models | jq + +# Should show: +# { +# "rewriters": ["google/t5gemma-2-270m"], +# ... +# } +``` + +## API Usage + +### Text Embeddings + +```bash +curl -X POST http://localhost:8080/v1/embeddings \ + -H "Content-Type: application/json" \ + -d '{ + "model": "google/t5gemma-2-270m", + "input": ["What is machine learning?", "How do neural networks work?"] + }' +``` + +Response: +```json +{ + "data": [ + {"embedding": [0.123, -0.456, ...], "index": 0}, + {"embedding": [0.789, -0.012, ...], "index": 1} + ], + "model": "google/t5gemma-2-270m", + "usage": {"prompt_tokens": 12, "total_tokens": 12} +} +``` + +### Image Embeddings + +```bash +# Base64-encoded image +curl -X POST http://localhost:8080/v1/embeddings \ + -H "Content-Type: application/json" \ + -d '{ + "model": "google/t5gemma-2-270m", + "input": [{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}] + }' +``` + +### Text Generation (Rewriting) + +```bash +curl -X POST http://localhost:8080/api/rewrite \ + -H "Content-Type: application/json" \ + -d '{ + "model": "google/t5gemma-2-270m", + "texts": ["Summarize: The quick brown fox jumps over the lazy dog."] + }' +``` + +Response: +```json +{ + "outputs": [ + {"text": "A fox jumps over a dog.", "tokens": [...]} + ], + "model": "google/t5gemma-2-270m" +} +``` + +## Troubleshooting + +### Export Fails with "transformers version" Error + +Ensure you're using the forked optimum-onnx: +```bash +cd /path/to/optimum-onnx +git checkout add-t5gemma2-support +pip install -e . +``` + +### "Model not found" in Termite + +Check the directory structure: +```bash +ls -la ~/.termite/models/rewriters/google/t5gemma-2-270m/ + +# Must contain: +# - encoder.onnx +# - decoder.onnx +# - decoder-init.onnx +# - config.json with "model_type": "t5gemma2" +``` + +### Out of Memory During Export + +T5Gemma-2 requires significant RAM for export: +- 270M model: ~8GB RAM +- 1B model: ~16GB RAM +- 4B model: ~32GB RAM + +Use `--skip-vision` if you only need text capabilities. + +### Out of Memory During Inference + +The 270M model requires significant RAM for inference due to ONNX model sizes: +- **Text embeddings only**: ~2GB RAM (encoder.onnx: 1.07GB + overhead) +- **Text generation (seq2seq)**: ~6GB RAM (encoder + 2x decoder ~4.5GB + overhead) +- **With vision encoder**: Add ~2GB RAM (vision_encoder.onnx: 1.67GB) + +**Memory reduction tips:** +1. **Limit max_loaded_models**: Set `max_loaded_models: 1` in config +2. **Reduce keep_alive**: Set `keep_alive: 1m` to unload models quickly +3. **Test on a machine with more RAM**: The 270M model needs at least 8GB free RAM + +**Note:** The vision encoder is loaded eagerly at initialization time as it is required for proper text generation behavior. + +### Slow Inference + +Ensure you're using the ONNX build: +```bash +# Check build tags +./termite version + +# Should show: Backend: onnx +``` + +## Technical Details + +### Model Architecture + +- **Encoder**: 18 layers, 4 attention heads (GQA: 1 KV head), 640 hidden size +- **Decoder**: 18 layers, merged self/cross attention, alternating sliding window +- **Vision**: SigLIP with 27 layers, 896x896 input, 256 tokens per image +- **Context**: 128K encoder tokens, 32K decoder tokens + +### ONNX Export Patches + +The export script applies several patches for ONNX compatibility: +1. Disables torch.dynamo (incompatible with tracing) +2. Replaces vmap-based attention masks with broadcasting +3. Sets sliding window to large value during export +4. Forces eager attention mode + +### Known Limitations + +1. **GenerateMultimodal**: Currently validates images but doesn't inject vision embeddings into generation (text generation only uses `` placeholder tokens) +2. **KV Cache**: decoder.onnx currently copies decoder-init.onnx (full KV cache optimization is complex) +3. **Fused Embeddings**: Text and image embeddings are separate (no fused text+image vectors yet) + +## References + +- [HuggingFace Model Card](https://huggingface.co/google/t5gemma-2-270m-270m) +- [Transformers Documentation](https://huggingface.co/docs/transformers/model_doc/t5gemma2) +- [Google Blog Post](https://blog.google/technology/developers/t5gemma-2/) +- [optimum-onnx Fork](https://github.com/timkaye11/optimum-onnx/tree/add-t5gemma2-support) diff --git a/e2e/t5gemma2_test.go b/e2e/t5gemma2_test.go new file mode 100644 index 0000000..4b9149a --- /dev/null +++ b/e2e/t5gemma2_test.go @@ -0,0 +1,380 @@ +//go:build onnx && ORT + +// Copyright 2025 Antfly, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package e2e + +import ( + "context" + "os" + "path/filepath" + "strconv" + "testing" + "time" + + "github.com/antflydb/termite/pkg/client" + "github.com/antflydb/termite/pkg/termite" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.uber.org/zap/zaptest" +) + +// T5Gemma-2 model name using owner/model format +const t5Gemma2ModelName = "google/t5gemma-2-270m" + +// T5Gemma-2 hidden size (640). The actual embedding dimension may vary +// based on sequence length as the encoder returns token-level embeddings. +const t5Gemma2HiddenSize = 640 + +// getT5Gemma2ModelsDir returns the models directory, preferring ~/.termite/models +// if TERMITE_MODELS_DIR is not set and the model exists there. +func getT5Gemma2ModelsDir(t *testing.T) string { + // First check TERMITE_MODELS_DIR + if dir := os.Getenv("TERMITE_MODELS_DIR"); dir != "" { + return dir + } + + // Check test models dir from harness + testDir := getTestModelsDir() + if testDir != "" { + return testDir + } + + // Fall back to ~/.termite/models + homeDir, err := os.UserHomeDir() + if err != nil { + t.Logf("Could not get home directory: %v", err) + return "" + } + return filepath.Join(homeDir, ".termite", "models") +} + +// TestT5Gemma2EmbedderE2E tests the T5Gemma-2 embedder end-to-end. +// This test requires the model to be downloaded to the models directory. +func TestT5Gemma2EmbedderE2E(t *testing.T) { + if testing.Short() { + t.Skip("Skipping E2E test in short mode") + } + + // Check if model exists + modelsDir := getT5Gemma2ModelsDir(t) + if modelsDir == "" { + t.Skip("No models directory available") + } + + embeddersDir := filepath.Join(modelsDir, "embedders") + modelPath := filepath.Join(embeddersDir, "google", "t5gemma-2-270m") + if !fileExists(filepath.Join(modelPath, "encoder.onnx")) { + t.Skipf("T5Gemma-2 model not found at %s. Skipping E2E test.", modelPath) + } + + t.Logf("Using models directory: %s", modelsDir) + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute) + defer cancel() + + logger := zaptest.NewLogger(t) + port := findAvailablePort(t) + serverURL := "http://localhost:" + itoa(port) + + config := termite.Config{ + ApiUrl: serverURL, + ModelsDir: modelsDir, + } + + serverCtx, serverCancel := context.WithCancel(ctx) + defer serverCancel() + + readyC := make(chan struct{}) + serverDone := make(chan struct{}) + + go func() { + defer close(serverDone) + termite.RunAsTermite(serverCtx, logger, config, readyC) + }() + + select { + case <-readyC: + t.Log("Server is ready") + case <-time.After(120 * time.Second): + t.Fatal("Timeout waiting for server to be ready") + } + + termiteClient, err := client.NewTermiteClient(serverURL, nil) + require.NoError(t, err, "Creating client failed") + + // Run sub-tests + t.Run("ListModels", func(t *testing.T) { + testT5Gemma2ListModelsEmbedder(t, ctx, termiteClient) + }) + + t.Run("EmbedText", func(t *testing.T) { + testT5Gemma2EmbedText(t, ctx, termiteClient) + }) + + t.Run("EmbedMultipleTexts", func(t *testing.T) { + testT5Gemma2EmbedMultipleTexts(t, ctx, termiteClient) + }) + + t.Run("EmbeddingDimensions", func(t *testing.T) { + testT5Gemma2EmbeddingDimensions(t, ctx, termiteClient) + }) + + // Graceful shutdown + serverCancel() + select { + case <-serverDone: + t.Log("Server shutdown complete") + case <-time.After(30 * time.Second): + t.Log("Server shutdown timeout (may still be cleaning up)") + } +} + +func testT5Gemma2ListModelsEmbedder(t *testing.T, ctx context.Context, c *client.TermiteClient) { + t.Helper() + + models, err := c.ListModels(ctx) + require.NoError(t, err, "ListModels failed") + + // T5Gemma-2 should appear in embedders list + found := false + for _, name := range models.Embedders { + if name == t5Gemma2ModelName { + found = true + break + } + } + + require.True(t, found, "T5Gemma-2 should be in embedders list. Available: %v", models.Embedders) + t.Logf("Found T5Gemma-2 in embedders: %s", t5Gemma2ModelName) +} + +func testT5Gemma2EmbedText(t *testing.T, ctx context.Context, c *client.TermiteClient) { + t.Helper() + + inputs := []string{ + "The quick brown fox jumps over the lazy dog.", + } + + embeddings, err := c.Embed(ctx, t5Gemma2ModelName, inputs) + require.NoError(t, err, "T5Gemma-2 embedding failed") + require.Len(t, embeddings, 1, "Should return one embedding") + require.NotEmpty(t, embeddings[0], "Embedding should not be empty") + + t.Logf("Single text embedding: %d dimensions", len(embeddings[0])) +} + +func testT5Gemma2EmbedMultipleTexts(t *testing.T, ctx context.Context, c *client.TermiteClient) { + t.Helper() + + inputs := []string{ + "Machine learning is transforming artificial intelligence.", + "Neural networks can learn complex patterns from data.", + "Natural language processing enables computers to understand human language.", + } + + embeddings, err := c.Embed(ctx, t5Gemma2ModelName, inputs) + require.NoError(t, err, "T5Gemma-2 batch embedding failed") + require.Len(t, embeddings, len(inputs), "Should return embedding for each input") + + for i, emb := range embeddings { + require.NotEmpty(t, emb, "Embedding %d should not be empty", i) + t.Logf("Embedding %d: %d dimensions", i, len(emb)) + } +} + +func testT5Gemma2EmbeddingDimensions(t *testing.T, ctx context.Context, c *client.TermiteClient) { + t.Helper() + + inputs := []string{"Test embedding dimensions."} + + embeddings, err := c.Embed(ctx, t5Gemma2ModelName, inputs) + require.NoError(t, err, "Embedding failed") + require.Len(t, embeddings, 1, "Should return one embedding") + + // T5Gemma-2 returns token-level embeddings (seq_len * hidden_size). + // The dimension should be a multiple of the hidden size (640). + embDim := len(embeddings[0]) + assert.Greater(t, embDim, 0, "Embedding dimension should be positive") + assert.Equal(t, 0, embDim%t5Gemma2HiddenSize, + "T5Gemma-2 embedding dimension (%d) should be a multiple of hidden size (%d)", + embDim, t5Gemma2HiddenSize) + + seqLen := embDim / t5Gemma2HiddenSize + t.Logf("Verified embedding dimension: %d (seq_len=%d × hidden=%d)", embDim, seqLen, t5Gemma2HiddenSize) +} + +// TestT5Gemma2GeneratorE2E tests the T5Gemma-2 seq2seq generator end-to-end. +func TestT5Gemma2GeneratorE2E(t *testing.T) { + if testing.Short() { + t.Skip("Skipping E2E test in short mode") + } + + // Check if model exists in rewriters directory (seq2seq models) + modelsDir := getT5Gemma2ModelsDir(t) + if modelsDir == "" { + t.Skip("No models directory available") + } + + rewritersDir := filepath.Join(modelsDir, "rewriters") + modelPath := filepath.Join(rewritersDir, "google", "t5gemma-2-270m") + if !fileExists(filepath.Join(modelPath, "encoder.onnx")) { + t.Skipf("T5Gemma-2 generator model not found at %s. Skipping E2E test.", modelPath) + } + + t.Logf("Using models directory: %s", modelsDir) + + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute) + defer cancel() + + logger := zaptest.NewLogger(t) + port := findAvailablePort(t) + serverURL := "http://localhost:" + itoa(port) + + config := termite.Config{ + ApiUrl: serverURL, + ModelsDir: modelsDir, + } + + serverCtx, serverCancel := context.WithCancel(ctx) + defer serverCancel() + + readyC := make(chan struct{}) + serverDone := make(chan struct{}) + + go func() { + defer close(serverDone) + termite.RunAsTermite(serverCtx, logger, config, readyC) + }() + + select { + case <-readyC: + t.Log("Server is ready") + case <-time.After(120 * time.Second): + t.Fatal("Timeout waiting for server to be ready") + } + + termiteClient, err := client.NewTermiteClient(serverURL, nil) + require.NoError(t, err, "Creating client failed") + + t.Run("ListModels", func(t *testing.T) { + testT5Gemma2ListModelsRewriter(t, ctx, termiteClient) + }) + + t.Run("RewriteText", func(t *testing.T) { + testT5Gemma2RewriteText(t, ctx, termiteClient) + }) + + t.Run("RewriteWithSummarizePrompt", func(t *testing.T) { + testT5Gemma2RewriteSummarize(t, ctx, termiteClient) + }) + + t.Run("RewriteMultipleInputs", func(t *testing.T) { + testT5Gemma2RewriteMultiple(t, ctx, termiteClient) + }) + + serverCancel() + select { + case <-serverDone: + t.Log("Server shutdown complete") + case <-time.After(30 * time.Second): + t.Log("Server shutdown timeout") + } +} + +func testT5Gemma2ListModelsRewriter(t *testing.T, ctx context.Context, c *client.TermiteClient) { + t.Helper() + + models, err := c.ListModels(ctx) + require.NoError(t, err, "ListModels failed") + + // T5Gemma-2 should appear in rewriters list + found := false + for _, name := range models.Rewriters { + if name == t5Gemma2ModelName { + found = true + break + } + } + + require.True(t, found, "T5Gemma-2 should be in rewriters list. Available: %v", models.Rewriters) + t.Logf("Found T5Gemma-2 in rewriters: %s", t5Gemma2ModelName) +} + +func testT5Gemma2RewriteText(t *testing.T, ctx context.Context, c *client.TermiteClient) { + t.Helper() + + inputs := []string{ + "The T5Gemma-2 model is a multimodal encoder-decoder architecture.", + } + + resp, err := c.RewriteText(ctx, t5Gemma2ModelName, inputs) + require.NoError(t, err, "T5Gemma-2 rewrite failed") + require.NotNil(t, resp, "Response should not be nil") + require.Equal(t, t5Gemma2ModelName, resp.Model, "Response model should match") + require.NotEmpty(t, resp.Texts, "Should return generated text") + + for i, texts := range resp.Texts { + require.NotEmpty(t, texts, "Output %d should have generated sequences", i) + t.Logf("Input %d generated: %q", i, texts[0]) + } +} + +func testT5Gemma2RewriteSummarize(t *testing.T, ctx context.Context, c *client.TermiteClient) { + t.Helper() + + // T5-style models typically work with task prefixes + inputs := []string{ + "Summarize: The quick brown fox jumps over the lazy dog. This is a classic pangram that contains every letter of the English alphabet at least once. It has been used for testing typewriters and fonts since the late 19th century.", + } + + resp, err := c.RewriteText(ctx, t5Gemma2ModelName, inputs) + require.NoError(t, err, "T5Gemma-2 summarize failed") + require.NotNil(t, resp, "Response should not be nil") + require.NotEmpty(t, resp.Texts, "Should return summarized text") + require.NotEmpty(t, resp.Texts[0], "First output should have generated text") + + t.Logf("Summarization result: %q", resp.Texts[0][0]) +} + +func testT5Gemma2RewriteMultiple(t *testing.T, ctx context.Context, c *client.TermiteClient) { + t.Helper() + + inputs := []string{ + "Hello world", + "Translate this to French", + "What is machine learning", + } + + resp, err := c.RewriteText(ctx, t5Gemma2ModelName, inputs) + require.NoError(t, err, "T5Gemma-2 batch rewrite failed") + require.NotNil(t, resp, "Response should not be nil") + require.Len(t, resp.Texts, len(inputs), "Should return output for each input") + + for i, texts := range resp.Texts { + require.NotEmpty(t, texts, "Output %d should have generated text", i) + t.Logf("Input %d: %q -> %q", i, inputs[i], texts[0]) + } +} + +// Helper functions +func fileExists(path string) bool { + _, err := os.Stat(path) + return err == nil +} + +func itoa(i int) string { + return strconv.Itoa(i) +} diff --git a/go.work b/go.work index b671882..f18056d 100644 --- a/go.work +++ b/go.work @@ -12,6 +12,6 @@ replace github.com/gomlx/gomlx => github.com/timkaye11/gomlx v0.0.0-202512210449 replace github.com/gomlx/onnx-gomlx => github.com/timkaye11/onnx-gomlx v0.0.0-20251217194823-a871bb8cc687 -replace github.com/knights-analytics/hugot => github.com/ajroetker/hugot v0.0.0-20260107061453-f83ae7b7c356 +replace github.com/knights-analytics/hugot => github.com/timkaye11/hugot v0.0.0-20260108205439-6127b8a4c499 replace github.com/knights-analytics/ortgenai => github.com/ajroetker/ortgenai v0.0.0-20260102002309-90abcfc27670 diff --git a/go.work.sum b/go.work.sum index 0bc772b..e9931bd 100644 --- a/go.work.sum +++ b/go.work.sum @@ -49,8 +49,6 @@ github.com/NYTimes/gziphandler v1.1.1/go.mod h1:n/CVRwUEOgIxrgPvAQhUUr9oeUtvrhMo github.com/Shopify/goreferrer v0.0.0-20220729165902-8cddb4f5de06/go.mod h1:7erjKLwalezA0k99cWs5L11HWOAPNjdUZ6RxH1BXbbM= github.com/ajroetker/hugot v0.0.0-20251229040509-3dc1031934d6 h1:sMCUPp1t4ClAZ7RoBfbF9mFPZ4fwu3/TktqtXj7dyrs= github.com/ajroetker/hugot v0.0.0-20251229040509-3dc1031934d6/go.mod h1:HM0TApRamY//nlzNqZCIlP1muD8mvkdYSka3bmRrm7Y= -github.com/ajroetker/hugot v0.0.0-20260107061453-f83ae7b7c356 h1:opUqf468UDMKDFEmmQZybIi5s7egT0RZGZX+WeylE8A= -github.com/ajroetker/hugot v0.0.0-20260107061453-f83ae7b7c356/go.mod h1:I3/sobJjSQecs1xBUKnbjBKcbaX03uQ7gfkDW55zAU8= github.com/ajroetker/ortgenai v0.0.0-20251229052243-d2d8091589c4/go.mod h1:zolhST8PioaCCsnwHBUesIBKU4CW/ZrF169YjFfg7BY= github.com/ajroetker/ortgenai v0.0.0-20260101101217-6423af4e4878 h1:5tWcPKi9eDbiaW3hDw+vwIupoz1nJT7xEFtLKB60Vxk= github.com/ajroetker/ortgenai v0.0.0-20260101101217-6423af4e4878/go.mod h1:zolhST8PioaCCsnwHBUesIBKU4CW/ZrF169YjFfg7BY= @@ -87,7 +85,6 @@ github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7 github.com/erkkah/margaid v0.3.0/go.mod h1:rf8vNecfnMZbhvzml9y+UspzrsmvUyS0bfkLVNPNCQY= github.com/fatih/structs v1.1.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M= github.com/flosch/pongo2/v4 v4.0.2/go.mod h1:B5ObFANs/36VwxxlgKpdchIJHMvHB562PW+BWPhwZD8= -github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw= github.com/fsnotify/fsnotify v1.8.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0= github.com/gabriel-vasile/mimetype v1.4.2/go.mod h1:zApsH/mKG4w07erKIaJPFiX0Tsq9BFQgN3qGY5GnNgA= github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI= @@ -221,6 +218,8 @@ github.com/timkaye11/gomlx v0.0.0-20251218235241-74992cedba51 h1:esopQIaYzkoJWVm github.com/timkaye11/gomlx v0.0.0-20251218235241-74992cedba51/go.mod h1:HKSGc4Btb3VntH/BV/UsJYzgTMbLT/Gk69kv95rjruM= github.com/timkaye11/hugot v0.0.0-20251229042211-73e5b84f666d h1:NpwiXABAYQogTUTHHa70UzICllLEjZNfJHQmEm00854= github.com/timkaye11/hugot v0.0.0-20251229042211-73e5b84f666d/go.mod h1:HM0TApRamY//nlzNqZCIlP1muD8mvkdYSka3bmRrm7Y= +github.com/timkaye11/hugot v0.0.0-20260108205439-6127b8a4c499 h1:f5+W2JKwNwLLkYzZ+A+Wcv9shdFSffo1L8zasfaIUAw= +github.com/timkaye11/hugot v0.0.0-20260108205439-6127b8a4c499/go.mod h1:F8q/Zn0G8jUFUCp6KAuUvQRRsHZk3VEb0Efo0c6z/N8= github.com/timkaye11/onnx-gomlx v0.0.0-20251216183743-5f89a03da602 h1:+5JuKeEiKoZ8BONRE4HDEkdOGl62X7CTv18wk3N/uiA= github.com/timkaye11/onnx-gomlx v0.0.0-20251216183743-5f89a03da602/go.mod h1:RjCGoxroeaa4IQc7TN+XeUJRwaVRfZ7TjRm+gtS/Z/U= github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75/go.mod h1:KO6IkyS8Y3j8OdNO85qEYBsRPuteD+YciPomcXdrMnk= @@ -372,7 +371,6 @@ golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20220310020820-b874c991c1a5/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.26.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= diff --git a/openapi.yaml b/openapi.yaml index dd99992..22f92c1 100644 --- a/openapi.yaml +++ b/openapi.yaml @@ -679,6 +679,12 @@ components: type: boolean default: true description: Truncate input to fit model context length + return_hidden_states: + type: boolean + default: false + description: | + If true, returns raw encoder hidden states [seq_len, hidden_size] per input + instead of mean-pooled embeddings. Useful for passing to the decode API. EmbedResponse: type: object example: @@ -692,7 +698,6 @@ components: - 0.089 required: - model - - embeddings properties: model: type: string @@ -705,7 +710,30 @@ components: items: type: number format: float - description: Array of embedding vectors (one per input string) + description: | + Array of mean-pooled embedding vectors [hidden_size] per input. + Present when return_hidden_states is false (default). + hidden_states: + type: array + items: + type: array + items: + type: array + items: + type: number + format: float + description: | + Raw encoder hidden states [seq_len, hidden_size] per input. + Present when return_hidden_states is true. + attention_mask: + type: array + items: + type: array + items: + type: integer + description: | + Attention mask [seq_len] per input indicating valid tokens. + Present when return_hidden_states is true. Chunk: type: object description: A chunk of text with position information. diff --git a/pkg/client/oapi/client.gen.go b/pkg/client/oapi/client.gen.go index b09a85b..dacef44 100644 --- a/pkg/client/oapi/client.gen.go +++ b/pkg/client/oapi/client.gen.go @@ -303,6 +303,10 @@ type EmbedRequest struct { // Model Name of the embedder model from models_dir/embedders/ Model string `json:"model"` + // ReturnHiddenStates If true, returns raw encoder hidden states [seq_len, hidden_size] per input + // instead of mean-pooled embeddings. Useful for passing to the decode API. + ReturnHiddenStates bool `json:"return_hidden_states,omitempty,omitzero"` + // Truncate Truncate input to fit model context length Truncate bool `json:"truncate,omitempty,omitzero"` } @@ -326,8 +330,17 @@ type EmbedRequest_Input struct { // EmbedResponse defines model for EmbedResponse. type EmbedResponse struct { - // Embeddings Array of embedding vectors (one per input string) - Embeddings [][]float32 `json:"embeddings"` + // AttentionMask Attention mask [seq_len] per input indicating valid tokens. + // Present when return_hidden_states is true. + AttentionMask [][]int `json:"attention_mask,omitempty,omitzero"` + + // Embeddings Array of mean-pooled embedding vectors [hidden_size] per input. + // Present when return_hidden_states is false (default). + Embeddings [][]float32 `json:"embeddings,omitempty,omitzero"` + + // HiddenStates Raw encoder hidden states [seq_len, hidden_size] per input. + // Present when return_hidden_states is true. + HiddenStates [][][]float32 `json:"hidden_states,omitempty,omitzero"` // Model Model used for embedding Model string `json:"model"` @@ -2410,197 +2423,200 @@ func ParseGetVersionResponse(rsp *http.Response) (*GetVersionResponse, error) { // Base64 encoded, gzipped, json marshaled Swagger object var swaggerSpec = []string{ - "H4sIAAAAAAAC/+y9C28ct5Io/Ff4dT7AUrbnIclOnLkIFrIs+2hXtnUk+SR7PYbE6ebM8KiHbDfZkiaB", - "7m+/YBXJZr9GoyTOHtwNECDWNFkki8V6sar4a5TIVS4FE1pFk18jlSzZisI/j5ZUv2NK0QUzf+aFzFmh", - "OYOPiRSaCW3++f8XbB5Nom9GFaSRBTMKYBzZHg9xVMiMPdbz3LR5iCMtZXaV0Cy74qnpkzKVFDzXXIpo", - "Ep28JnJO9JIR046YdkQvuSIrHJRwRQqmcilSLhZES7IjRbYmc1kQM4sfTbfdKI70OmfRJFK64GJRG1e1", - "R710YymyoikjszVMgSrFlaZCNwfxH8xIXLOVemz1ZoQjmmUwE5waLQq6jh4M+tiXkhcsjSafEJWffSM5", - "+ydLAMcdmG+tw34ndjOH5KLMc1loRfSdNNNfUa0mUzEgF3yVZ4wgeiZkGv2NZZmMyVLeEVowspblv08j", - "0/LQTNNsigVKcmoA7nzImTg8Iasy03wlU5pZ+LsT8unXKcx/GhnImt3raRT7f/nRptHD56mI4kgK9mEe", - "TT41l2Mnabq50Tt2ttnryRPedg8t2s9oodvb+Bm2qBQ37V05JIn5AIRtlnLH9ZLkUnHznXCB0+BSDKO4", - "cSqZSK+SJS3aQI+WtKCJZkUIiciCL7igmR1oyQpmB2ciVWSH3SdZqfgtC44IF5otWGGW1HUiL9iXkomE", - "EVGuZqxwxxOh7oxjsheT/ZgMh8MOmHF0P1jIgf215EIf7JuBlKaF/oNWBrBU53pM247T7qffS1SNQ8nT", - "yAKrTT2u9qf7xJbi5kiKOV90rBJ+LwvYeOAsMCXD1MzITJlDK8klK1ZcM3J4djKcikvDCbkilChzNPic", - "s9QsYs4XAMJszN8uL89MczIgKZ/PWaHIvJAr+DYvs4zAtFiBE5iKuyVPloSLJCtTpkheyFuesoIolrEE", - "JkdFShKaLM3cknDaQzi9dYpd0fsrWIlls3NaZjqavBjHTV5F7/mqXAVkhd3Mqgumy8LAZvfUsADs397f", - "lUxZVhsnmvN7ZnarZ8vNGqCXGaZUbEiOuV6ygjyDjs8AjcryHXnDxGBGlUGy7RwTWRBqQQi6Yohc+FuN", - "EkStGv1qPj2MhuES/NRaoknesiKj+RUM+Bje3nt82W45CkLTlcyYvmNMWFQ+jkDFclpQLYs6EqcCdraB", - "Q3PwfAdAFKzI46a2WAuiLYZpsWC6e6mttV5C45Dz4DJzZo9LfYWdS9TLgqmlzNLaYOPhi7jrRKbA6nwf", - "WOWH9+9/tjtMdsbD8WBvON4NRwZgyMXNNmeSBiwFJw8spZtDnONx71TJLOvYrJFVXKaf5Zm907KNtKhi", - "KZkUC5LKpFwZoamXVBPBWAoHcsaIyjOuCRdaErWiWea2QA2Hw0cZKMzqcz8GjEKnQIX0M/s1MjyHXS25", - "jiZzmikWR46xfAol457ZdyO5xnW5MnbI8GsEFsgLpXHmZuJGfQhA/WBB7dVB/dANS7FEijQA9tmzJHvY", - "H5rsMVhTc49+WjLgRAVTZabJHVVEseKWpchioGeF6JmUGaPCjBCy205FCBUoxxK203WAUFpaTo3l1oV2", - "jbnSRJc0y9aGxaZkZ0XXVhjhWqyEYynhczKnWTajyQ2RSVIWBUt3t+GaDQLzq8P5xQGiO8nOH6369tCc", - "X5VFx/o+np86zceJZLaasdRYISMvWcx+8YTV5r/UOp+MRplMaLaUSk9ejl+Oo4BflAXvYpQGJYYy84LL", - "gut1jX19iqQQ90a9ymgURwtpVlmf8SvsT1x/Igsj1g1Dwz3KJAUbCtRRCd1oRlJmVkBUzhKjXhRqOBVv", - "YKoTcm3ndG2EoPtjgj2urU5m+9uZAvO4pqWW10MjERzudMGZIkzg/0G9M5MzmkapWHhS6S3lGZ1ljNjx", - "/g1HmIpErmZcoP4E3Ep5Y2fJcI0w5rffWlSob78lOynLDVakILOSZynRdKF2wSS6XshrMiBnZcHIW2n0", - "claANNih2R1dq2oqMTk6+0iMSRgTlck7pvQuQDCbYmCAxDgvheYrRnYsnSoyMIP9OIW9iz+cXxqraE6V", - "9v3vM2q6v5XvTn8mP58edvS9z2j88+mh6Xp59nF09PH14ejo7OMurvQ14EZ9+y2uCBBPBuSw1HKQMs0S", - "TWashtUdu1M4gaRMYQbv/3Hy+uSQGOjk7dlH/CYLtsoAXG7UoyNZsHen5nAnHy4sNuC4z9aAAISo8xKX", - "JBcZM1MmO67Rz6eHdlRs80YWCfOoxQUd4znyK/qE2EN78j6j+I+FnEafDYjLYu3oRCFhGxRURKlRpQ1B", - "TcyaPbyJzkv8A7+5v/wIZwUzbOzt2ccYOBcMZ6j8yKLJgwV0NXojrmBiFn00SVjGUJ+e1lTeT9X08KCb", - "uYG1jBOzP+K/kAN41t5Wuxps3Fo+V4olpeMum+QBFXqerQcLeZXxGZ1fqaSgRvG8kjkThmta4/jCwqsU", - "khvG8iua8VtW1zFfrFoa5t/kHWohWhLTzSldhlMZWSHIiq1ksSZ0bszDjCptKI7sfMgyuqIDM2+q+Sxj", - "u8OpeIedacGACIyJnaBMEhYgggH3UuosMTknXNBE81uu18Op+KiAG6SBpQa8cBq9WE0jsvOCrLgoNVO7", - "ZqP3lua3PbKUZQE/jM3fjC7MdJHfmqm9DrijA5TRX9a+Dcn4DSO4rCG5YBpbGmhaEnafZzzhOlsTJuAU", - "10YwNp1hxYY6VxUS3KI1WsxlDuxWsFtWeJQM6ySIm9QipkwunkovmVwsGuRi6QMMRilA9At95Yzfura+", - "hfHoQQSc21vSNDNs2izvME25FXX+6x0355iRLyUrWUrK3CDZzAt+uFL8FzacCrsLYxChpci4kWRpxUAb", - "uHveabDS+yvE9BXuzFOXaffTHZDWyVB8VWaaCiZLlRn6/WnJBJI4TBj9p0Y3SmOQkxkzp6hgCRPa6Wuo", - "H3BVnZSd0/OPxDBRM7HdbZBBPghDnvO54bi3zGgHglS8wEAXUgx+YYVsIO6gD3G4xKvVbDukWYzscEHe", - "vdq19j7M1y4KcdmNI5rnhbRo6kURnqvg7DyOFW9uCULTW67MDHHQgVWa7bynogRXLuoryigsuC2GGhUc", - "XaNVa7bKZUELbpB9nzCW4kJuaVYaov1JFjeG/KVYKJ4y0iJAa8cLNlgUlAv0J+lCZk1yHv/wXd/GVMfk", - "qeQcuroACtJJD08IiLfaNXtszbd5mWUxEeyugmt2zZDbi/EBuUDtnHwUlfoDovic6WI9OARpsGQ0ZUX/", - "XuJgj9B53/yn5Xh8wMi4gdu9cb9360rpgmq2cAaKZ19nDUdxuTIqAwiCKI6MNIniaCZLkbI0sIEqLl7f", - "mzNWDOp2gR14DX6mgqdMDck7mqvA+QX7ppeMF61elQAW0rBke4tDcziFZtssCqtx0NMmQzYBSt/JPPgF", - "UTj2O2Dl5+6EhAKU7Jh/GFgpW1GRxvaUWpHP04zttkD/CAJ2UhemFlIlNeOayAS9GyjROQWXxm53S5ri", - "hkyjEAO4KGAzbZm8QxVcWJjTkRfMjuHa16V1TNgtE03W6jGUcyFC4QBzBQYE4lCRFb83K8T1Go4CKLR8", - "gePhUnTFSC5b/ODX6NXh4clotmADcAcNmBjc7g1fRBNPgyte6OUoWUpxsx6szJHXti3cwWRcLEqaDfai", - "CZJrl4cMJ3OV8o6LgldUMZLygiXasFHDtygXlRNClTP3lRvidcYnBWuIq8QQtnLrnRdyhTbGr9WgDyO0", - "8lmhRkZ/P3Y2v/cHVpZOrZv3BJtededIf6eCFbTqdQ5/bdUtkQvBf6k6wt+grj7a9a7g2nW8YF/2L9gX", - "4n5t9q4rrv9nNNSI0ZFrp5gmt5ySW56zYndojpc5/YrpGGw6sLn1gIuG5xgkouPJTR20NU6XSmoodAsJ", - "9H6zvpjznGVcMPQwowuBHNNk6b/AiVVTQUki87VzCUHLmChJlnyxZAWKXwWsbiUL5nSRWanx7MGvUxFM", - "wskIr+2PncYS8sqdFRc778vVkTE/n+82dc69LjFiuUj7/Jxypb1OWfFz277GlDoNrMslU6zDwuCrFUs5", - "1SxbI29yBw9v6mJCbyWHQwQunoFdOsmoZiLxksPOSC1lmRllRydLopdSMeBonoiDE0p22HAxNGZXN2+a", - "Rmba2yumdaMMxmxi/FMfG4wjMHX4KMl4PrjlGi6SBrlZxsH+04x1i6ArzVdMlrpuR4+jPjULvE9akjvK", - "NWgx1FEZXAVIswLNYnv3Z1ZoVTDT3nTeaP4ejBU6N/ZW+H8wdYUkdpYxCQyjc6cBoY4KN1ZGENm2gZr2", - "nLylmt3RNbnEb012cDDuZADq4CopWMqE5jRTT3ZnHFSWaQCl6WV27uEel7KPDugKAQiCEdAJW8UheE8y", - "2QFXvSwIX9EFXNJXsREbw0vYva6HJ2xuf2LAfzw/rfX5/BBHIOB6b6S4yMuO1Z2Yn/0KtcQFhdEny4Kx", - "RvyJWLjQDheEcj2NlizLJLmTRZZOo+ta+EnQVE3AzbbEIBJDe7bH5+tNESsVxncNgEeDVCz8h5jUmsLW", - "GDLA9sGfE9PQ/msapVTTCXwd5WLxv8zx/+55PBwOp9HDg5lpnY8ESze6E9y3g3leGJXGkFx/mEwTl2Rn", - "RpObO1qkJGDZT4mdCbHdC21rDtY7THAIGptVOwjqj4vR6bm/em/UXCvNnUCxgjG44G/Imxpf6pUDbeQU", - "pUiorqsquihZ6/rbNiRw7szBmnNtZwXoutckY2Khlx33gg3W5e7F8Ah3MTB79DuvYj2HUtHk06fxcLy3", - "fxAPxsPx8xffxePh+PuXP3yOze/7B8/h9xfffW9+f/nD5+BOtAdFrUvScLRe0qmY5i2o+IrsSMFAcUN0", - "IcJrlOP/8dhlfVsGb3kPipoLuIgMk/eT/E2k0rOFAXo697EoMJ6jgVT3c33C0NpFVz46BQTSNeobLrha", - "njOqDNjmKPg7uVuuyYIJe91BlJZ5Dpe7zn1hfoniqCLqKmizurOY80yDgTkvBTgkoUWnj+ONbfGazTla", - "RO25Vd8MWVHioOKlYkIFmTEIDsVbK6/vt0KfamDbGkDwtxnozkDHoCw7XipZLWQnesuwhbMR7hiFCAEu", - "IFwjASx2cRijOXfHvYmAz1ULlbC+2tgLpq/seJ32Fi3oihnLsd8r1cXS/uPiw3tyAcyaIPFYvMxQJwxm", - "FQxR8zvUke7R0B3mx/UaPBtKU6PqGvuAXFBB3hRUJFwlMiZHh4+SvR+ki/JbPxggSV1Lt/Er3QEfRmES", - "c7j8xK7V0o0hydPGPvexeNj1rhm+xSPHjpaSJx04nMPZvSr84d0kYWsH/cEIlJTdd+mEKbtHSuOKJDCy", - "8yllXOnOeM1MLvJCzjrY/qlcEPOJznhm9jSImjVGh7/8Z2lM7HW9KDOIBDf/pzNDO0iQbR9TFRO/aeEO", - "iy6EvhUkCoiowMUNvG7emc7I4QtdMLoyJyPgmjby9uLiGNx/mhhFc7fFjRDldbG3zfIwngwppeviuGBU", - "szTQ+lrxwy1m4WVlO/IRMRE4sJMl1UNrm3Iphhix9nmrCF0LrZpkEA5ksfHoJvSdkZRlmm6Lw9fQ+CH+", - "g05WE9E9hIdT3LTC124RTQmYaeoVcIiAdYTXJqu+3INLecOEB4Jz6T18FQX8ptSRTSkcOLSqL6PKJlFP", - "zdjwe9lK2+hF8zZJNm1ZZY84S32ui0PmjsEiOqoqFBCuSF4wxTAL5c9HdD1XBq0SzJOx08IZ144ARE56", - "4H9i9ozbml7XxoredwUl77/4rte9hrHIWvqdC/UU6Nhxp4dbqzoTAm5ZoZDHu2YuayXwvD45ayXI+tre", - "jgntYbs82WcQ++8Ni3gBMV+jBVut6OBgsDcbcN3puoNT+ri+dDInQNvgIuBwU67sHTmceBfGAbRHFfES", - "UnWGzWq2ys28y6Juie81x70wCwImUvWAQHDyI4SUFStD4ponsXP//4he/4KKVK4aoeLfx9tYn3BGvCh6", - "9GxU4lrKrkPrXP3wmew4HVvtBmfXWDrmWDoXudeozBG3Xn64Lq8y+HAhPinExQ9tfaY7DW2ZX908kgNx", - "KfPBDVFuX3Z8SO69ZkJxKWJ362yTvg7PTroznsxo+ebtf18mGStVNVygh0Zbxf13GfGeEWzmVpU3pj6p", - "NltIwFr1ipPNklRsk2LYTSQBENuW7FgTNFs77XpvawbUsD4265SNqGvB7+GOQGm6yskOBtyrXSsLjYFX", - "zfWOKlIpftWd2Pfj53v7B8+7+XFXxtshKQX/UjLC4SJgzm3YNKa91bBc43fmW7LKswGdJXv7B12sbjuv", - "UaXsd2YLeZ25rUNYk9r08XbQtKlTT6PA49L41ulFKZ9iHn3sNo621M/dYJtOxcc+5cotoibIu69/rfDm", - "TSqKum9Q5SrXT4aK3XrYjqZZL8BL89UBwwwKhEX+LZhqF0NrYL0+8bgDRY2pdKHd3RV12PjmC4HUiAKs", - "UPLx/KTFb/rzKapOZKf3tqSeDdLdjP/j1Yfzu/F/vl3IbbKR+q7wum7FehYdGkvB9d0mba1xleaujR47", - "Vx79nnFWxnIF5FEDGb7GnT0qBGDEVih1mqIDo1o6eKaPqEuawS6WDfkM0ymm9Eyj3foFmEv02Tps6El3", - "6P7WZtPcWTO8Z7u7nw1BAN0fJvzlE2d/rwsKtxybpv/2lL8/Pie+sd+CX1ghB2op4aLdxwaBMpeUSssV", - "yeiMoUHWyD8wtLwqBdfr0SLjghVXuJLb/eGeTUZoNcGN0lTdPG2VlSGxaZW+1ej09F3fRvXYJJ/6jBKz", - "jgITdkZv7bcP79///NRADReJdcXFXG4K2NxohnswcCBPDKhWwOZrpinPIL6vcoXSmY2msP2rYD4X4pHQ", - "HHVXztRwKt7RvB4CBKHvGP8t5nJIPkIEEpjt3thheHkRgpoKRpNlOLLPxtpB2oqJoUJDhDEpWAbzVTGh", - "Qt2xQu22wgtTlfHVCE4/RM68Pz7HBMpqUHDNA/Do88PjtNjfO47c3CKziXZyURzZ2fWDD0/DVvA/d3qO", - "ghC+TaTf3tgm3YexgHXC78JnHP6a0WLhfn50rU88FjaucfPa6uGOPQzWwRrc7g9WB0+dh4123DSPviDI", - "NqZd6GR9ptnqy2I0z6gY6BeW66svJU0HXxZRXP8K++A/PmEpXcmveE9WoToUeTXWWqe2ECtdKoHnRcdC", - "2zyxVomSTTU85HyumCvYwQBGVy2SIBa+82LInKOOy2uEZ9qTnbPj85h8OH8bk9MPRzF5d3JxVNcdz4C0", - "2+6nRBasp0AHlgOAFujx0ZK0s/9/+GEbnw5EPz4VVTO24KJ2FsZPK3Ri4djiJRUy/kMuBblY8TB2ZFPy", - "vtsDtw6ofRI57G2knF6fq+WObZSgTmKnjq0w7w0UG7JjFR17MqXI1rsuIaRUhoVQEjaJyVqW4N/CtOY1", - "oWLtMbPOWQg9ngpIqjNgKr3JwDL6khGvKB4LpgsMNx9OhQ10tsVTbGoVtHumfPCuXQjkDKlWrt2nKGeF", - "AgsUzAexRpshLcFkTqlmT+N2j3h2m7Jka1HSI0naFw5Wjl71bfO5bdDaArIzr5RX0FKfeaH8rFI71j6a", - "F2xk7xNy0f9W/3BdHXS4Mnbk4ZSDmJwfvzo+3d24lw4CJFBUSgzhGCaC6cWtbZ3bHJw4upPFjboC7gAR", - "BSy94uJpm2rOouqu71HDIBA3Zyi26vOpTj65w+wwbdOzjeWMud0nIhkSDp6MoxJOrJDD3yGhnJsHp/8I", - "t+izPt2aNgWF4ZmGSbQCwszg3eFgWynjVgD+9tgwd/JWdW/fI8dn03I9aW9YsDsjtUs5xyVxKhuO2MaT", - "8htwiUM8jsS+0DdHAhtJqDjy8+8QLtXaIMCLtjmhYx0QqYzca0KsrhEykffH57tkUHEFaMpSonIqFKZw", - "15QSA84ZApMqOpoWM64LWnhRR8OcFKwhUZN4kMjjt2tCzjs4nKsE5RlB70YaYNbOmZBjC+EW0iqVrf9l", - "PoJ/qwljF5mddUdtbU11kHuXzdsRqNBr87a3sX3xUTPOem8/glZo+fbSR0Pxb9t68RNZTEi4W0Uc+APV", - "YpZLhik/T2RtPYq20ST9Ti55Tqw7MSjT46Xcb1aw3R2L52obNO4XW92iUp49GQcNzgN4tKACHXiD1msM", - "sP4wg8eUspoR3KGTuQzB7oDihnXcFbgJVwId9H9WmO4iZQVLgzJgTrFAyEMCUZUZh7w5V4pVOU+3Z0Bi", - "MRVUpAQB8rCw2JyzLFUjzVZ5RjVTZMbmZvcTmmU+L4iJNJdctJWpE6FRJbZxq808CXT/v4baCPYnMi9F", - "Ss3YNINiZU9SuL6UrFh3VcWkRbIk8BXrwrKM3VJHzM2I7+Y0Cc3zjCeeM24X+I1zqfZwE/31KVFP0008", - "Pfae6269PkAFC9QSnHiMScW25JNCZaWmmD01PL8HX3aC3WgCp8fmTCfVl+oUnAoAM9IFFcrMuk6tl+Zn", - "Q+Wm7VtWrKiYkI5Ku3+oYdftwNrSf/WI+2o7OrXI24j3P4o+Dbge+uyxlXAK2hiLsI071KnTqEXHpKZF", - "x6jtGKvI/DZjdNVtRDxuu/1OC8kG8HVIZpkxzF1wEVEK+C4mCiRBYFmYarFWmq3gwh1C/H1Na5t20aml", - "NfMMuwtNdl6OttSxDZUqm6mEnbvbuAttlJjcfA3aW48SopI6IkIgftTHTe2i6dCOnWot0vV4NPa3nagS", - "rLG95eDytAFdYXQOxF9O/bDTyEhpH8m1G+y/n9m2OOvoUMcbRGj24Q5Cc1sBo78ZX248h7e+WJ6PPZE8", - "Ng61nfcCcW39ATxb7gisdtO2/Lad4P4kbLkdPQHfJyIpGGpEwXwhGGNz9Pdv3R4fRL3dHmEYMZSJKJQN", - "Jq82DuLdO2v8P56LEr4rYMN0kB93etWfstkdU95+k3s38E2A8PY0fK7UNulptFiABt4hEiEry6YOWzT5", - "1rbEjx+rdlp+nfrcKEx6buVWTaOHPyBDDde3fYJaV2ZUHGDg8xYo3+Ls1Oa3zfnZsAch3Ar5wc5sj0a3", - "AsRlH2VuS4U+CrllvutCZgoU2UoSYvlWKTM1JB9ydFBNxYBMI1pqaYgEgx1TlvCUKXJXJcEBEqkVaWFZ", - "0mkkpGBVXwj0FlLbHmKNA2JTt+9V81WpXFNti7kZDc/0MV1wuRNbfpS60rdJLT/Sp4A2Hqxwx9ssLoph", - "nlFAe58fT7nfbtiNnLj+5Tcnf25xhvrzH3/9KorF54c4+gcrFJei32SAQr5QHKWjRhQW+XUhxDUOsj/e", - "fz4Y7w32XlzujScH48l4/L+7DtmC66tErlZdRbvfcggHXnFNllQta/BRi3jeCVJe3eKyOkBKYiORiGtT", - "z6rYG+6/GI67wPbCdAHyXQBv94bjLnCNzaq6BviIQ+TXltVFKc2qKy5UsKq98tcDKn89oNJPLz1FZLuS", - "hRetx0pAMvtKVFgKUrXoJWO3LPu9xW1PAQjs0vrxpLvHoF0AkE75vN1EauWiIC4x7kHYLStmhmTWBPFQ", - "qa8pm5ULcO1A9zuKz7O4chAVN7ENWqxpu1XWporBhjTrna4sdV5qG2JNANlD8sx1w7dcEpnJgv+CdU2V", - "zFhMnv1TSfHMaWplosuCpaBpxeRZJhfzlcav+AAMm895Ag7vG7b+ESq7kZzyQsXkmZAyd2/GZEzoYYCy", - "YPr/xLgOhA1qgszraAsaP4q6nmJV7ccMkoQpdXXD1p1PrR3+dEGwiVkYOXkdFGy6YWulZcGIWgtN73GF", - "LCmYJpmUN2U+JG8gBdRVPz/86eLq8Ojo+OLi6j+P/+vq5DVh4pYXUoDP/5YWHILruC8IWH8aZy3LYoCT", - "Gdyw9YB3unzcrUAHjz0IY+5dO1cP7pk6GNIV/UUKeqeGiVw9I7IwW+2eZPhhPB7jNr7j4uRDPWys2RlC", - "1MUpligJSu4FjnHA1FWF/27kW4RWe/B7N+Di+Oj8+DLYh9+wCThIsBednn+mlE8i6UpOsxW2cZXQFg8T", - "Hitbs3hNgoptT1p717RhlAHOqDOHiV0plT1ac+kYC6pfXJyOLk8vYOyLA8M7BD4doLxpMoFsVGhx+NMF", - "vAbg60snNKtIqaNsx6OcfMv6+u0zj4XOrwxZq643Zrhmmb3ptm2JaQu1FkcnZ5g3n3FxQ1J5B3VmFdTx", - "ZKtcr2MIfALYWFffQjBqEcs1yQt+SzUjBg6fk1kmk5sr++MVz/HhnKJku8P6PYn9pz1dSSqG9V/2ftgf", - "jodPjjt2yMipXm6LDNOW5AWb83vmimNnbDIaYRmZA/Ovj+enLaTAGCFShsag9J1LxQidKZmVmtm2ljmN", - "PipWqFFKNR3tYiccxXSZlckN0yOcj+uxWg/s72UOGzRq4jOEadhVq8PT8Njax0dP0SvTo1bduyINUlAB", - "qe17+98by2M4Hr2Myd44+Pf3+8O97+Cvvf2YmN3f++4l/v1dTPa++2G4/+K5/Xu3M7nbEa+rmXllU0dr", - "Mz9oJxbbepRm37lI+S1PS5r5o0BkbhMzITTOwQyL149BOvCV0QH2+gqm+9kp/gu7mq01q09sb/z85Yvv", - "vxv3VlA3/QzVOkC2bDtWsyUIsBZU7eH5yY0fsTW40N89dxPGtLGUrzC9ul4iYfz8Zd88oR+546lejpaM", - "L5Ywv5zfQ1oS1vbzefsFM8uqP/+EwDdhtM1NHx6snmorblDMlkW/SHQInDaKMS8RnmdSk9FowfWynBl+", - "YxXydDayZYfbuTfOjMBC/rY4LsR7IeuvorfgBa/CPyNnXxN8d1o9PjAV33xDflpSiKewgMOnkmwcqnJS", - "5TSAji/c+BkEKtDh2Qm43b79tipZ/danFX/77YTARRs8I1DVYNw5Oj05222lwSEg6OBKWRsIF2xFheZJ", - "lfIH8wmfy3NvBA6AYN1LXwjP17c2sKqYgYINbASFFfwQaIGPotmZvKcrlrqAvKDatQFkA9gIBlOrmMhi", - "QQX/xeUbOXe1exWT3etgeefuPhkw5C710WTGmEl3u+5S63yUXJW0HduwkPAXppMhVtB2rAQKYVHN4GmJ", - "jFEjIDRx5Ig0OORylMpEjbws9vQCtWAgO6uDZhIqSFEKY9mLlGZSMHi2sqosTwXBc0CSrFSaFUAsp0Bh", - "1f41KM0wRnavWQGq3dmJfZuCiQQSwAYdZHg9ojnHrMnrSi2v3QtDT09K1VN1jgDOD99WdblN25BUXLQU", - "8OuVOSosdSTzpaQZ12vT5YgJXdAMzEAbwseKW4jDSQqpFEm5kX6zUrOUCJniQGdGZCXrAUTOYvPa6YOc", - "ZVuSP2P0lilidFHToqDesty1W/aGQeEQu4PfkK5ziZSIWbiGAsOjVKta7x4F7K1Vj5AOz04AzHb74o4l", - "+lkJvvoG83jFhVHXvVM+BmvZreRdxUCsDm8ZCQLEhGm/XABoPpPgAZJ/Al2YzR+gyKhYkMppwiwkCL4J", - "5wXp3z6LXJGd69488utdcwaM6obAbKr2kUeKAfhRMdWoRGu9CzvXv6kIsC33e21xYc7rEVXwnhlBxCC1", - "xgQIcYBoLJguOLulWUxuuTIaiOIrntEC6BmxXmPHTcJ5UzFdf5haz9jtkn8LKSyAQV5bOlsbYIe97yX0", - "PnqAsI7w0WADY3+Az2SRy8tTVw0fnmjElm+sZIC51+za5kMBLgRoTnnmz5IXJduuITxYHQsJHmJAiH8v", - "DXv6xQvPw/qDYoZqvmCT6m0gPq9QHdCv6V6LtQSWZ0P6yI6NrVxSkWbGAOEsS4O4bgy4znjC7FWI02qy", - "jJwb/UqRc4bPhrZUnEqmZGxBsfgO1/gKbPXMdBRcI0S3ezTLl3QP6o6gJRpNooPheHgQxZG3q3DnwRqV", - "qss9k2fcmAFmpR0vyFrRqpwQqGsNjYICTmV6Z4kWKQAIvnpoo03rwaP34bvO8Kqtrp7BxCeVTeOP7m2D", - "H33BAvPzG6pQbapVfnLTALp6549TW0Pyb9w4JhMe7AF5t4nXB8F79ZP25BND7Mk0f52jZgWmK5zI8Dkx", - "w5v33Qt3Q+jGwFvoqznMpF66J8gNWsLH+t1W2UcczV/X19f/VEbQ/ToVpFZtvechYmCgMTbGYZDFCkLM", - "T7A0BGA3KXafai9NmyYvxmP/sf7kNn71H/0j1wh4OhXmv8h8fpiKB1gF3gM7c/AkdW/gXuIlhzV9X8l0", - "3Sj8F0T+jv5pq0LiVcBWb/K6iNWH+hWLMcLhBxTecCr3x+M/emx7BQuDd8XtAVmmRJXgPpyXWQYOhOd/", - "4EywvHXHDE4EFMt1Xgcz7os/Z1yrEltbj9mGcaTK1YoWa0caHfxPsQWGg5jmqI/1c1FX/0jZiueBDm0d", - "FuEzDMhTs4ZKr4b2CSivjHMVVPW3Fhzoi89UXU20ihB6cO91jBH+UHIthRxI7NuqgBPYlzYQEWAEKmCb", - "pTfe7tzIC8MXmqo3a+zbJcbKwlVgj8AK0RIKEQdv4QazcRbhAIJnVlaxch647qouuxNymIAfNHw9waok", - "FRKawEBVrnc1iLXOkVtOK/3W4qmmB+OjXsH7GoA1H/L5VV/b6HoYqP8Fjq0e3UCF+es8udFRn2n3dwrE", - "4BUpuMlIwndFKbyHnZbIclg6FVbRge2YZ+APwxeXbvENqLQUKRUaEnPdqWgaZOZnf1fiY9N9yDTkUhuq", - "sQSFDxra96AD7icTzfQAQ9WuvYmnWMGNoWwTsKzBF5OVVJr4m9DdFjTDS68nGK7mhBAyhKC0DPCLwO4P", - "lQOkY0N1YTmtFnVN2ipEoAL0Pftk9QdgkKblpwbtG6JqpuhMo8+VrJ+Kd53P9bTpafMEN70J1TVNUEge", - "PTOU5EupJaYAJFSbA9TR9fedIltjzR4mA/7zBl3IyarK7P9KSlHtwaI/WSmqv5hixu47YHWYjbsSOHkD", - "d/JY2n7aJCwzOoPmHRE3LcWkwn1QW/pfRTV7Pn7+9ce178hJo3GUIv2XUgndCQk4ImqBvpr0Fopg4JhG", - "TS8oiWbVC6hmRg52h1Pxxmx6S9szKpItk3u0pJoc+SKRihyenfw+5Syow/YHamfffEP80wggDpkm13jS", - "8Fb+GpPz8FG5C9iEwYURj8dQDxpeT9jF5fMq7noqMEgB60nblx9RNNvHRY15em2wVCEJVPtrG0WMryLa", - "AHAMGYRB0C1JPr3+8P748/WwQ48zP4H3saeyr9W+UaS66uBwQ7tZ4nTVu/OSxgGqC5tCZlZiYI4Yigt7", - "YvDDf8kS958sWZbPy4z49LFhTfpUsErFig5IkA35/00jlChuXr4Yu2m0/+I7+zvi1fwGwRahfK40CbLz", - "XoqBj8HftW6HGVXLqdHWMjL4mZx9uLiEy5bJaFTFAb0cvxwDObozSKZTM/Lgb2TqHvgbXK5zNiFNrjGN", - "XNuUPOtyT2zYiNZWbIM4d2XY1Fv+fRo9fK7A1jC5Nx6DE+NZJ9ou/l9Hmae1Cj8VRRm2UUNOr05z5INe", - "v4ZC03yp4E/WaVqlxx9sGPAIGGGnPrP1szJdsvKop275kJwzXRbC5sAYw0uE59qbWfFUyKKLyddfAPFW", - "2U4oKKCMzV9q0J+qBpnRD77+6G+Dh+bsy/9l8PL/jpBOU6reL9jtU9ICRcuoWB2GHypvtnbl5NdowXRX", - "7jmStFFtFNhr7drDsffdx9UNd1yVvY3J++PzmGB5jVqNSu9vdx4x9GS1L7u6vWem7d/9LRaEi5pjtKS3", - "jFwP+Mtrosr5nN87l4m9g8BBDvuqPJMd9+g3eA3OslJBytbGWYX3G9YJYm/ktlhS4/bueJXrNQa9tjfc", - "gn/r1VSE36ws3KfSbgu/qvCz3QKCB93h8X+H1XrhQ8PgOks7+1FtiQccsxG60jGsfwzeNL98EZM3p4fv", - "B+Yfrw7PL5HmTt/9/S3piHrxt8g1AyGwDYz2b8+he1fcwEMX27BD3p5ypd+5WrBfTeA1ap73sU/lLuX/", - "u2zoV/Rf82rjtMtKQ2bo6bjflPUHQ4GLMA3qlT0awWWjqqpALsug4WyFhc4b3BHjxoxOir5UCJIqUnIk", - "35+e1sqC2hC6s+NzuLWHCdln6f1T81U5xWlkozw+nL+FsJtw3lUHrLWI6mlVbdF3Pv1wZDqf+jg13/E9", - "uyP/JYsb7ApZ1sx3e3dyAf3ecZWwLKOCyVJ5bLZs+OqqeRsrvsWQvP/5S1te7EDHK/5yKIW4h+kdNVwO", - "fyshgecNTRh5dXx+aS/j23tmOtd5nrtCSrpqxN5yij4F/Pu6ekjz8ZvowO7oKHDqTWaoz2J9yBsqaTa3", - "t1VMs+ZebrG+elUyzr6W97RVo/dPtjbaVT+7/JiOJdgwmL+8mP8T1Pcgrve36++evhrSxcknoyhuEk4D", - "W8Es7w3Y8gE9VQE4LQm1IcA1gVSvb+/E0REGfJ27Snr+QQV0Jdp4sFWp9GQq9oY+wNmO5wrqIdN2UTNq", - "KvaNAQ1VoKBAgiu3NxUHQ3LBRNqxJvfeP1Xk2q7vGufBFUmZ4gtBbhjLVZVHq6lmGVMKVDnIfFF2ypD1", - "gVza4L+qBJjJBU+Gv1si1XT8DnnkvODY68p/AKmE8Zj1KL68YHNWdIq0eijfEyRJT2FGL02ASrDpxhKB", - "voPdlsBhO23VRHxnIZ1aSBMCG7goecoIBkBWUVUGAJRLdK3Jm6Bc4oS8Z2VBMyKYtiKuYNC5cQE4FW9k", - "4cnPy3NH8iD1m7GMMaQ4aU/kA8VTNhXXGZ+NfFcjw5MbyCJZ8mTpAyMrenq8fmRTthrQZ7aG4teSq2EV", - "0D9dqNZKQHbwVrt4u0F/ydP/GfLUHcbfIU0NiEr0rSuh5wQqeBA2SdRGvcxNqTUdDggw+H6/7Kj5ORrD", - "gruLiUSmrABZEZOUwV8DLri2P2EEcNUIEj1Vl62zhb/km2+ITwxwwWd/dz3qWVveIekgWsGPpbUHQEJG", - "O6G8UB5Qse6Doqoytu6xgFCHAOOXFjRfFlT5TC3AngtmJCmfQ0IMBC2lLlfMFj51uWZVHVTTy9Umz6hY", - "lHCD05KpRpp0Ia1CKaJv8+XnhmKm9WgbZ9EtmuidkGk5Hh8kywz+z8grtpZGyev5WWlagLNLkWOtKfkP", - "cBYYy4+mPMtoAq7AIlWPGoCA5a8YTdyogPuny6h6Hdi+iOLCF2v9S0T9dWOzlYwK2FOXXEFJFVTK2nhR", - "0yicFZOFL/gVk5kvLoYCoV25a9hxiaz/4StpfbXj1ayZ1oFs2yQsn/Xf48tuXLhph7v6zKAZEKWCwnvd", - "CdmHZyeWdH3204jmPHr4/PB/AwAA//8FXdNIGbUAAA==", + "H4sIAAAAAAAC/+y9C3Mbt5Io/FfwTb4qS9nhQ5adOLyV2pJl2Ue7sq0jySfZa7oocAYkcTwExgOMJCal", + "+9tvoRvAYF4U5Tz21N1UpSoWB2gAjX4C3Y1fo0SucymY0Cqa/BqpZMXWFP55vKL6LVOKLpn5My9kzgrN", + "GXxMpNBMaPPP/79gi2gSfTOqII0smFEA49j2uI+jQmbsoZ4Xps19HGkps1lCs2zGU9MnZSopeK65FNEk", + "On1F5ILoFSOmHTHtiF5xRdY4KOGKFEzlUqRcLImWZE+KbEMWsiBmFj+abvtRHOlNzqJJpHTBxbI2rmqP", + "euXGUmRNU0bmG5gCVYorTYVuDuI/mJG4Zmv10OrNCMc0y2AmODVaFHQT3Rv0sS8lL1gaTT4iKj/5RnL+", + "T5YAjjsw31qH/U7sZg7JZZnnstCK6Ftppr+mWk2mYkAu+TrPGEH0TMg0+hvLMhmTlbwltGBkI8t/n0am", + "5ZGZptkUC5Tk1ADce58zcXRK1mWm+VqmNLPw9yfk469TmP80MpA1u9PTKPb/8qNNo/tPUxHFkRTs/SKa", + "fGwux07SdHOjd+xss9ejJ7zrHlq0n9NCt7fxE2xRKT63d+WIJOYDELZZyi3XK5JLxc13wgVOg0sxjOIG", + "VzKRzpIVLdpAj1e0oIlmRQiJyIIvuaCZHWjFCmYHZyJVZI/dJVmp+A0LWIQLzZasMEvq4shL9qVkImFE", + "lOs5Kxx7ItS9cUwOYvI0JsPhsANmHN0NlnJgfy250IdPzUBK00L/TisDWKpzPaZtB7f76fcSVYMpeRpZ", + "YLWpx9X+dHNsKT4fS7Hgy45Vwu9lARsPkgWmZISaGZkpw7SSXLFizTUjR+enw6m4MpKQK0KJMqzBF5yl", + "ZhELvgQQZmP+dnV1bpqTAUn5YsEKRRaFXMO3RZllBKbFCpzAVNyueLIiXCRZmTJF8kLe8JQVRLGMJTA5", + "KlKS0GRl5paE0x4C99Ypdk3vZrASK2YXtMx0NHk+jpuyit7xdbkOyAq7mVUXTJeFgc3uqBEB2L+9v2uZ", + "sqw2TrTgd8zsVs+WmzVALzNMqdiQnHC9YgV5Ah2fABqVlTvyMxODOVUGybZzTGRBqAUh6JohcuFvNUoQ", + "tWr0q/l0PxqGS/BTa6kmecOKjOYzGPAhvL3z+LLdclSEpiuZM33LmLCofBiBiuW0oFoWdSROBexsA4eG", + "8XwHQBSsyOOmtlgLoq2GabFkunuprbVeQeNQ8uAyc2bZpb7CziXqVcHUSmZpbbDx8HncxZEpiDrfB1b5", + "/t27n+0Ok73xcDw4GI73w5EBGEpxs82ZpIFIwcmDSOmWEBfI7p0mmRUd2y2ySsr0izyzd1q2kRZVIiWT", + "YklSmZRrozT1imoiGEuBIeeMqDzjmnChJVFrmmVuC9RwOHxQgMKsPvVjwBh0CkxIP7NfIyNz2GzFdTRZ", + "0EyxOHKC5WOoGQ/MvhvNNa7rlbFDhl8jiEBeKI0zNxM35kMA6gcL6qAO6oduWIolUqQBsE9eJFlmv2+K", + "x2BNzT36acVAEhVMlZkmt1QRxYoblqKIgZ4VoudSZowKM0IobjsNITSgnEjYzdYBQmlZOTWRW1faNeFK", + "E13SLNsYEZuSvTXdWGWEa7EajqWEL8iCZtmcJp+JTJKyKFi6v4vUbBCYXx3OLw4Q3Ul2nrXq20NzPiuL", + "jvV9uDhzlo9TyWw9Z6nxQkZes5j94gmrzX+ldT4ZjTKZ0GwllZ68GL8YR4G8KAveJSgNSgxl5gWXBdeb", + "mvj6GEkh7ox5ldEojpbSrLI+45fYn7j+RBZGrRuBhnuUSQo+FJijErrRjKTMrIConCXGvCjUcCpew1Qn", + "5NrO6dooQffHBHtcW5vM9rczBeFxTUstr4dGIzjc6YIzRZjA/4N5ZyZnLI1SsZBT6Q3lGZ1njNjx/g1H", + "mIpErudcoP0E0kp5Z2fFcI0w5rffWlSob78leynLDVakIPOSZynRdKn2wSW6XsprMiDnZcHIG2nsclaA", + "Ntij2S3dqGoqMTk+/0CMSxgTlclbpvQ+QDCbYmCAxrgoheZrRvYsnSoyMIP9OIW9i99fXBmvaEGV9v3v", + "Mmq6v5Fvz34mP58ddfS9y2j889mR6Xp1/mF0/OHV0ej4/MM+rvQV4EZ9+y2uCBBPBuSo1HKQMs0STeas", + "htU9u1M4gaRMYQbv/nH66vSIGOjkzfkH/CYLts4AXG7Mo2NZsLdnhrmT95cWG8Du8w0gACHqvMQlyWXG", + "zJTJnmv089mRHRXbvJZFwjxqcUEnyEd+RR8Re+hP3mUU/7GU0+iTAXFVbBydKCRsg4KKKDWatCGoiVmz", + "hzfReYl/4Df3lx/hvGBGjL05/xCD5ILhDJUfWzR5sICuRm/EFUzMoo8mCcsY2tPTmsn7sZoeMrqZG3jL", + "ODH7I/4LJYAX7W2zqyHGreczUywpnXTZpg+o0ItsM1jKWcbndDFTSUGN4TmTORNGalrn+NLCqwySz4zl", + "M5rxG1a3MZ+vWxbm3+QtWiFaEtPNGV1GUhldIciarWWxIXRh3MOMKm0ojuy9zzK6pgMzb6r5PGP7w6l4", + "i51pwYAIjIudoE4SFiCCgeOl1HlickG4oInmN1xvhlPxQYE0SANPDWThNHq+nkZk7zlZc1FqpvbNRh+s", + "zG8HZCXLAn4Ym78ZXZrporw1U3sVSEcHKKO/bHwbkvHPjOCyhuSSaWxpoGlJ2F2e8YTrbEOYAC6ujWB8", + "OiOKDXWuKyS4RWv0mMscxK1gN6zwKBnWSRA3qUVMmVw+ll4yuVw2yMXSBziMUoDqF3rmnN+6tb6D8+hB", + "BJLbe9I0M2LaLO8oTblVdf7rLTd8zMiXkpUsJWVukGzmBT/MFP+FDafC7sIYVGgpMm40WVoJ0AbunnU6", + "rPRuhpie4c48dpl2Px2DtDhD8XWZaSqYLFVm6PenFRNI4jBhPD81tlEag57MmOGigiVMaGevoX3AVcUp", + "e2cXH4gRomZi+7sgg7wXhjwXCyNxb5ixDgSpZIGBLqQY/MIK2UDcYR/icImz9Xw3pFmM7HFB3r7ct/4+", + "zNcuCnHZjSOa54W0aOpFEfJVwDsPY8W7W4LQ9IYrM0McdGCNZjvvqSjhKBftFWUMFtwWQ40KWNdY1Zqt", + "c1nQghtk3yWMpbiQG5qVhmh/ksVnQ/5SLBVPGWkRoPXjBRssC8oFnifpQmZNch7/8F3fxlRs8lhyDo+6", + "AArSSY9MCIi32jXLtubbosyymAh2W8E1u2bI7fn4kFyidU4+iMr8AVV8wXSxGRyBNlgxmrKify9xsAfo", + "vG/+03I8PmRk3MDtwbj/dGumdEE1WzoHxYuv88ZBcbk2JgMogiiOjDaJ4mguS5GyNPCBKile35tzVgzq", + "foEdeAPnTAVPmRqStzRXweEX7JteMV60elUKWEgjku0tDs2BC822WRRW4+BJmwzFBBh9p4vgF0Th2O+A", + "1Z/7ExIqULJn/mFgpWxNRRpbLrUqn6cZ22+B/hEU7KSuTC2kSmvGNZUJdjdQojsUXBm/3S1pihsyjUIM", + "4KJAzLR18h5VcGFhuCMvmB3Dta9r65iwGyaaotVjKOdChMoB5goCCNShImt+Z1aI6zUSBVBo5QJH5lJ0", + "zUguW/Lg1+jl0dHpaL5kAzgOGjAxuDkYPo8mngbXvNCrUbKS4vNmsDYsr21buIPJuFiWNBscRBMk164T", + "MpzMLOUdFwUvqWIk5QVLtBGjRm5RLqpDCFXO3VduiNc5nxS8Ia4SQ9jKrXdRyDX6GL9Wg96P0MtnhRoZ", + "+/3E+fz+PLDydGrd/Emw6VU/HOnvVLCCVr0u4K+duiVyKfgvVUf4G8zVB7veFly7jpfsy9NL9oW4X5u9", + "64br/xkNNWJ05NoppskNp+SG56zYHxr2MtyvmI7BpwOfWw+4aJwcg0Z0Mrlpg7bG6TJJDYXuoIHebbcX", + "c56zjAuGJ8x4hEBOaLLyX4Bj1VRQksh8446EoGVMlCQrvlyxAtWvAlG3lgVztsi81Mh78OtUBJNwOsJb", + "+2NnsYSycm/Nxd67cn1s3M9n+02b86BLjVgp0uafM660tykreW7b14RSp4N1tWKKdXgYfL1mKaeaZRuU", + "TY7x8KYuJvRGcmAiOOIZ2KWTjGomEq857IzUSpaZMXZ0siJ6JRUDieaJOOBQsseGy6Fxu7pl0zQy097d", + "MK07ZTBmE+Mf+8RgHIGrw0dJxvPBDddwkTTIzTIOnz7OWbcImmm+ZrLUdT96HPWZWXD6pCW5pVyDFUMd", + "lcFVgDQr0Cy2d39mhdYEM+1N563u7+FY4eHGwRr/D66ukMTOMiaBY3ThLCC0UeHGyigi2zYw056RN1Sz", + "W7ohV/itKQ4Ox50CQB3OkoKlTGhOM/Xo44zDyjMNoDRPmd3xcM+Rso8O6AoBCIIR8BC2ikPwJ8lkD47q", + "ZUH4mi7hkr6KjdgaXsLudD08YXv7UwP+w8VZrc+n+zgCBdd7I8VFXnas7tT87FeoJS4ojD5ZFYw14k/E", + "0oV2uCCU62m0Ylkmya0ssnQaXdfCT4KmagLHbCsMIjG0Z3t8ut4WsVJhfN8AeDBIxcK/j0mtKWyNIQNs", + "H/w5MQ3tv6ZRSjWdwNdRLpb/y7D/d8/i4XA4je7vzUzrciRYurGd4L4d3PPCmDSG5PrDZJq4JHtzmny+", + "pUVKApH9mNiZENu90HaWYL3DBEzQ2KwaI6jfL0an5/7qnTFzrTZ3CsUqxuCCv6FvanKpVw+0kIPRDbMV", + "T1MmZkpTzernQPa2s8FkC6KLksU2OEKRgt4SJhJpZoqwCMIiHxX7MsuYiIkbg//CPoFNAxw8FVwozYyD", + "tCBrRsXAmE8srQSRGpIPii1KiJEiOVXKRtuhMWIGtSEpnTeSuihFQnXdGIPZN++lbUOclxlgwbXFOxDE", + "nSYZE0u96hinIZzdzR8KqS4RbYVb52VztfRo8vHjeDg+eHoYD8bD8bPn38Xj4fj7Fz98is3vTw+fwe/P", + "v/ve/P7ih0/BrW8PEbSugak2VMqlmK2p6goZc9+J+e43NNhDwkXKE6rNvtzQjKc2NmI4FecFU4aTwLLp", + "Ija4vy5Ka2l7vmqycxhE0bBImn+H2Otn9i5KIzfgoCnysZtWd10P8EzzMLa9tIeiNB5eagfbhqu9+Gqu", + "/D127ndc544X/2iqw5moERV+X79KNnZydCcnFwXGLDXCJd3P9TlCaxdB/OCoCKRr1NdccLW6YFQZsK2d", + "h9/J7WpDlkzYKz2itMxzCGBwR3TmlyiOKrFWBSZX93ILnmk4RFmUAg7doUXnOd5r2+IVW3D0+ttzq74Z", + "RqTEQcWL84QKMmcQAI03s96nbYX31cC2rdzgbzPQrYGOgYd2vFSyWlha9IZhC+cH3zIKUTBcQEhSAljs", + "0qLGO+yO7RSBLq8WKmF9tbGXTM/seJ1nCrSga6ZZseXktUup/cfl+3fkEgwSgsRj8TJHvyeYVTBE7Wyt", + "jnSPhu5QVq43cHoHIiImxgcml1SQ1wUVCVeJjMnx0YNk7wfpovzWDwZIoh+2WlxQk3EKxAIu+LFrtXRU", + "X4197lPysOtdM3yDLMeOV5InHThcAO/OCs+826zIGqPfG5MiZXddfk/K7pDSuCIJjOzOTTOudGdMciaX", + "eSHnHarjTC6J+UTnPDN7GkSGG8faB7iwNCY2JEWUGWQ7mP/TuaEdJMj2OWqV97Ft4Q6LLk2kFQgNiKjA", + "xQ28bt+Zzuj4S10wujacEUhNG11+eXkCR9yaGGdqvyWNEOV1zbfL8jBmEimlKziiYFSztNsU4mmnx+PV", + "Yzu6FzERXNIkK6qH9vyFSzHEqMxPO0WhW2jVJIOQN4uNBzehj0dSlmm6Kw5fQeP7+HfirCaiewgPp7ht", + "ha/cIpoaMNPUO5kQ5e0Ir01Wffk1V8bE9kBwLr3MFzh8X5MetS1NCYdW9WVUGVPqsVlJfi9bqUm9aN4l", + "kaytqyyLs9Tnczlk7hksouFbocCYuznaxPv/LYiu54OhX4q5YHloqtdYAGx0D/xPzBBzW9N7fLemd12B", + "90+ff9d7hIzx9lr6nQvtFOjYcW+NW6s6k15uWKFQxrtmLjMruF14dGZWkNm4u+sSnvnY5cm+Qx//vXHq", + "s4S4xtGSrdd0cDg4mA+47jyeBi59xClPTgvNIRpE2TgQ4HgXqgS0RxXxGlJ1H8SwdW7mXRb1s5iD5riX", + "ZkEgRKoekOxAfoSwyWJtSFzzJHZXXD/izVZBRSrXjXSI7+NdHFDgEa+KHuSNSl1L2cW07joLPpM9Z2Or", + "/YB3jadj2NJdA3mLyrC4vcmCkJAqSxUX4hOfXIzczjzd6VvLfPb5gTyfK5kPPhPl9mXPh53faSYUlyJ2", + "kRU2sfHo/LQ7q8+Mlm/f/ndlkrFSVcMFdmi0U25L10mcFwTbpVV1HlefVFssJOCtesPJZgIrts0w7CaS", + "AIhtS/asC5ptnHV9sLMAangf223KRmaB4HdwD6Y0XedkD5NK1L7VhcbBq+Z6SxWpDL/q3vf78bODp4fP", + "uuVxV1bnESkF/1IywuGya8FtagCmdtawXJN35luyzrMBnScHTw+7RN1uB0WVsd+ZEedt5rYNYV1q08f7", + "QdOmTT2NghOXxrfOU5TyMe7Rh27naEf73A22jSs+9BlXbhE1Rd4d4mCVN29SUdQdJSDXuX40VOzWI3Y0", + "zXoBXpmvDhhmCSEs8m/BVLsEWgPr9YnHHShqTKUL7e4+tMPHN18IpP8U4IWSDxenLXnTnzNUdSJ7vTeC", + "9Yyn7mb8Hy/fX9yO//PNUu6Scdd3Td1189uz6NBZCq6ot1lrjetidzX6EF959HvBWTnLFZAHHWT4Gnf2", + "qBCAUYmh1mmqDozc6pCZPmo0aQZ0WTHks6inmLY2jfbrl7wumW3n0LhHxYn4m8ltc2fNELbd7je3BLp0", + "f5jwF4+c/Z0uKNwFbZv+mzP+7uSC+MZ+C35hhRyolYRgEh//BsZcUiot1ySjc4YOWSPHxtDyuhRcb0bL", + "jAtWzHAlN0+HBzbhptUEN0pT9flxq6wciW2r9K1GZ2dv+zaqxyf52OeUmHUUmJQ2emO/vX/37ufHBiO5", + "aMMZFwu5LSh5qxvuwQBDnhpQraDkV0xTnkEMa3UUSuc2Ysj2rwJWXRhTQnO0XTlTw6l4S/N6mBukd2CO", + "g1hIuOq24UjSOzsMLy9CUFPBaLIKR/YZh3tIWzExVGiIMCYFy2C+KiZUqFtWqP1WCG2qMr4eAfdDdNi7", + "kwtMEq4GhaN5AB59un+YFvt7x5GbW2Q20U4uiiM7u37wITfsBP9T58lREKa6jfTbG9uk+zDetU74XfiM", + "w18zWizdzw+u9ZFsYWN3t6+tHtLbI2AdrMHN08H68LHzsBG92+bRF+jbxrQLD67PNFt/WY4WGRUD/dxK", + "ffWlpOngyzKK619hH/zHRyylK8Eb78kqVIcqryZa69QWYqXLJPCy6ERomwvZKsOzrU6NXCwUc0VpGMDo", + "qrcT5Ht0XgwZPuq4vEZ4pj3ZOz+5iMn7izcxOXt/HJO3p5fHddvxHEi7ffyUyIL1FKHBkhfQAk98tCTt", + "Chc//LDLmQ5E+D4WVXO25KLGC+PHFfOxcGyBngoZ/yFXglyueRg9tK1AhdsDtw6o7xM57G2lnN4zVysd", + "2yhBm8ROHVthbicYNmTPGjqWM6XINvsu6amEaCxKwiYx2cgSzrcwdX9DqNh4zGxyFkKPpwISRw2Yym4y", + "sIy9ZNQrqseC6QJTKoZTYYP5bYEgmz4I7Z4oH6BuFwJ5caqVT/oxylmhwAMF90Fs0GdIS3CZU6rZ46Td", + "Aye7TV2ysyrp0SQdoXyoR2d923xhG7S2gOwtKuMVrNQnXik/qcyOjY9YBx/Znwm5DBdrf7iuDjpcGTvy", + "cMZBTC5OXp6c7W/dSwcBkoQqI4ZwDBPBFPrWti5snlkc3cris5qBdICIApbOuHjcphpeVN01bGoYBOLm", + "DNVWfT4V55NbzIDUtgSB8ZyxfsGpSIaEw0nGcQkcK+TwN2god8yD039AWvR5n25NW8LoLE/DJBTZk4IF", + "UYFm8P3O0LCdjHGrAL8+HMxx3rp+2vcA+2xbriftLQt2PFK7lHNSEqeyhcW2cspX4BKHeBiJPfTjSWAr", + "CRXHfv4dyqVaGwR40bYkdKIDovFRek2ItTVCIfLu5GKfDCqpAE1ZSlROhcIyBTWjxIBzjsCkygCgxZzr", + "ghZe1dEw7wrrpNQ0HiSr+e2akIsOCeeqnXlB0LuRBpj1cybkxEK4gdRhZWvcmY9wvtWEsY/Czh5H7exN", + "dZB7l8/bEajQ6/O2t7F98VFzznpvP4JW6Pn20kfD8G/7evEjRUxIuDtFHHiGagnLFcO0tkeKth5D21iS", + "fidXPCf2ODEoReW13Fcb2O6OxUu1LRb3851uUSnPHo2DhuQBPFpQgQ28xeo1Dlh/mMFDRlnNCe6wyVwW", + "bHcMccM77grchCuBDvo/L0x3kbKCpUGpO2dYIOQhgajKjENuqCs3rNxJtxdAYjkVVKQEAfKweN6CsyxV", + "I83WeQZh23O2MLuf0CzzuW9MpLnkom1MnQqNJrGNW23mAuHx/yuo/2F/IotSpNSMTTMoyPcog+tLyYpN", + "V+VXWiQrAl+x9jHL2A11xNwM8m5Ok9A8z3jiJeMusd5uLtUebqO/PiPqcbaJp8devu626wNUsMAswYnH", + "mDhvy5opNFZqhtljI/R78GUn2I0mOPTYns2n+tL5Aq4AMCNdUKHMrOvUemV+NlRu2r5hxZqKCemoJv27", + "OnbdB1g7nl89cHy1G51a5G3F++9FnwZcD332+Eo4BW2cRdjGPerMabSiY1KzomO0doxXZH6bM7re35YZ", + "tMV3+40ekg3g69DMMmOYu+AiohTIXUwUSILAsjDVYqM0W8OFO4T4+7rtNu2i00pr5tJ2F1PtvBxtmWNb", + "qrE202U7d7dxF9ooo7r9GrS35ipEJXVEhED8qI+b2kfXoR071Vqk6/Fg7G87USVYY3vL4cjTBnSF0TkQ", + "fzn1w04jo6V9JNd+sP9+ZrvirKNDHW8QodmHOwjNbQWMfjW+3HgOb32xPB96InlsHGo77wXi2voDeHbc", + "EVjttm35up3gnhN23I6egO9TkRQMLaJgvhCMsT36+2u3xwdR77ZHGEYMpVAKZYPJq42DePfOdywezkUJ", + "386wYTooj+PezM4dN7tjyrtvcu8Gvg4Q3p6Gz5XaJT2NFkuwwDtUImRl2fR4iybf2mU2+wWE3PLr1OdG", + "YWJ/K7dqGt3/DhlquL7dE9S6MqPiAAOfdkD5DrxTm98u/LNlD0K4FfKDndkdjW4FiMs+ytyVCn0Ucst9", + "14XMFBiylSbEEsVSZmpI3ud4QDUVAzKNaKmlIRIMdkxZwlOmyG2VBAdIpFalhaV3p5GQglV9IdBbSG17", + "iA0OiE3dvlfN16VyTbUtWGgsPNPHdMHlTmyJXerKOye1/EifAtp4lMWxt1lcFMM8o4D2Pj1cVmK3YbdK", + "4vqXr07+3IGH+vMff/1DDItP93H0D1YoLkW/ywDFqqEAUEcdNCxk7UKIaxLk6fjps8H4YHDw/OpgPDkc", + "T8bj/93FZEuuZ4lcr7sK07/hEA685pqsqFrV4KMV8awTpJzd4LI6QEpiI5GIa1PPqjgYPn0+HHeB7YXp", + "AuS7AN4cDMdd4BqbVXUN8BGHyK8tq4tSmpWFXKhgVV/or0eC/nokqJ9eegoldyULL1sP8oBm9tXWsNyp", + "atFLxm5Y9lsLOJ8BENilzcNJdw9BuwQgnfp5t4nUSqJBXGLcg7AbVswNyWwI4qEyX1M2L5dwtAPdbyk+", + "QeTKQVTSxDZoiabdVlmbKgYb0qx3urLUealtiDUBZA/JE9cN3ytKZCYL/gvW7lUyYzF58k8lxRNnqZWJ", + "LguWgqUVkyeZXC7WGr/iI0dsseAJHHh/ZpsfoXohySkvVEyeCClz9y5SxoQeBigLpv9PjOtA2GAmyLyO", + "tqDxg6jrKcjWfrAjSZhSs89s0/mc4NFPlwSbmIWR01dBUbLPbKO0LBhRG6HpHa6QJQXTJJPyc5kPyWtI", + "AXUV/o9+upwdHR+fXF7O/vPkv2anrwgTN7yQAs78b2jBIbiO+6KX9eefNrIsBjiZwWe2GfDOIx93K9Ah", + "Yw/DmHvXztU8fKIOh3RNf5GC3qphItdPiCzMVrtnR34Yj8e4jW+5OH1fDxtrdoYQdXGGJUqCspLBwThg", + "albhvxv5FqHVHvzWDbg8Ob44uQr24Ss2AQcJ9qLz5J8p5ZNIupLTbBV5XCW0RWZCtrJ1uTckqEr4qLV3", + "TRtGGeCMOnOY2Eyp7MGqWyf4aMDl5dno6uwSxr48NLJD4PMYyrsmE8hGhRZHP13GtsoS1lBPaFaRUkfZ", + "jgcl+Y5vSLR5Hov5zwxZq653lLhmmb3ptm2JaQv1REen55g3n3HxmaTyFmopK6hVy9a53sQQ+ASw8e0I", + "C8GYRSzXJC/4DdWMGDh8QeaZTD7P7I8znvsSTfvD+j2J/aflriQVw/ovBz88HY6Hj447dsjIqV7tigzT", + "luQFW/A75grAZ2wyGmEZmUPzrw8XZy2kwBghUobGofSdS8UInSuZlZrZtlY4jT4oVqhRSjUd7WMnHMV0", + "mZfJZ6ZHOB/XY70Z2N/LHDZo1MRnCNOIq1aHx+GxtY8PctFL06NWwb4iDVJQAantB0+/N57HcDx6EZOD", + "cfDv758OD76Dvw6exsTs/sF3L/Dv72Jy8N0Pw6fPn9m/9zuTux3xurqwM5s6Wpv5YTux2NZcNfvORcpv", + "eFrSzLMCkblNzITQOAczfKBhDNqBr40NcND3KICfneK/sNl80yxyeDB+9uL599+Ne18JMP0M1TpA9mkC", + "rNhMEGAtqNrD85MbP+BrcKG/e+YmjGljKV9jenW9RML42Yu+eUI/cstTvRqtGF+uYH45v4O0JKxf6fP2", + "C2aWVX/iDIFvw2hbmt7fWzvVVtygmC2L5yLREUjaKMa8RHiCTE1GoyXXq3Ju5I01yNP5yJbWbufeODcC", + "H6uwBaAh3gtFfxW9Ba/UFf6pRPti5tuz6oGNqfjmG/LTikI8hQUcPgdm41CV0ypnAXR8xcnPIDCBjs5P", + "4djt22+rsuxvfFrxt99OCFy0wVMZVZ3RveOz0/P9VhocAoIOrly7gXDJ1lRonlQpfzCf8ElI9w7mAAjW", + "vWaH8HwNdwOrihko2MBGUFjFD4EW+PCfnck7umapC8gLKrobQDaAjWAwtYqJLJZU8F9cvpE7rnYvv7I7", + "HSzvwt0nA4bcpT66zBgz6W7XXWqdj5KrkrZjGxYS/sJ0MsQq8U6UQCEsqhk8n5IxahSEJo4ckQaHXI5S", + "maiR18WeXqAWDGRnddBMQgUpSqiuKFKaScHgadbq9QQqCPIBSbJSaVYAsZwBhVX716A0IxjZnWYFmHbn", + "p/b9FSYSSAAbdJDh9YjmHLMmryuzvHYvDD09KVXPMToCuDh6U9WeN21DUnHRUiCv14ZVWOpI5ktJM643", + "pssxE7qgGbiBNoSPFTcQh5MUUimScqP95qVmKREyxYHOjcpKNgOInMXmNe6DnGX77ETG6A1TxNiipkVB", + "vWe5b7fsNYPCIXYHvyFdfImUiFm4hgJDVqq9zOAevux9jwEhHZ2fApjd9sWxJZ6zEnzZEObxkgtjrvtD", + "+Ri8ZbeSt5UAsTa8FSQIEBOm/XIBoPlMgkd2/gl0YTZ/gCqjEkEqpwmzkCD4JpwXpH/7LHJF9q5788iv", + "9w0PGNMNgdlU7WOPFAPwg2KqUW3Zni7sXX9VoWtb0vra4sLw6zFV8GYfQcQgtcYECHGAaCyYLji7oVlM", + "brgyFojia57RAugZsV4Tx03CeV0JXc9Mraca98m/hRQWwCCvLJ1tDLCj3jdBeh/2QFjH+DC2gfF0gE/B", + "kaurM/fiAzxDii1fW80Ac6/5tc3HMFwI0ILyzPOSVyW7riFkrI6FBI+NIMS/l0Y8/eKV51H90TxDNV+w", + "SfX+FV9UqA7o13SvxVqCyLMhfWTPxlauqEgz44BwlqVBXDcGXGc8YfYqxFk1WUYujH2lyAXDp3FbJk6l", + "UzK2pFh8h2t86bh6Sj0KrhGimwOa5St6AHVH0BONJtHhcDw8jOLI+1W48+CNStV1PJNn3LgBZqUdryRb", + "1aqcEqhbDY2CAs5kemuJFikACL56TKZN61hYv/V2ObzcrKunXvHZcNP4g3u/40dfsMD8/JoqNJtqlZ/c", + "NICu3np2altI/h0nJ2RCxh6Qt9tkfRC8V+e0R3MMsZxp/rpAywpcV+DI8Mk8I5ufulcch9CNwWmhr+Yw", + "l3rlntk3aNFB/JbbKvtQqfnr+vr6n8ooul+ngtReFOh5bBsEaIyNcRgUsYIQ8xMsDQHYTYrdp9pr6qbJ", + "8/HYf6w/K49f/Uf/kDsCnk6F+S8yn++n4h5WgffAzh08Td07z1d4yWFd35cy3TQK/wWRv6N/2qqQeBWw", + "07vTLmL1vn7FYpxw+AGVN3Dl0/H49x7bXsHC4F1xe0CWKVElHB8uyiyDA4Rnv+NMsLx1xwxOBdZ6LxyG", + "4uj5nzOuNYmtr8dswzhS5XpNi40jjQ75p9gSw0FMc7TH+qWoq3+kbF34wIa2BxbhUyMoU7OGSa+G9pkz", + "b4xzFbxcYT04sBefqLqZaA0hPMG90zFG+EPJtRRyILFvqwJO4F/aQESAEZiAbZHeeJ92qywMXyGr3mWy", + "7/MYLwtXgT0CL0RLKEQcvPcczMZ5hAMInllbw8qdwHVXddmfkKMEzkHDF0KsSVIhoQkMTOV6V4NYezhy", + "w2ll31o81exgfLgueEMGsOZDPv/QF2W6Hr/qf2Vmp4dl0GD+Y56V6ajPtP8bFWLwUhrcZCTh27kU3nxP", + "SxQ5LJ0Ka+jAdiwyOA/DV8Vu8J2ztBQpFRoScx1XNB0y87O/K/Gx6T5kGnKpDdVYgsJHO+2b54H0k4lm", + "eoChatfexVOs4MZRtglY1uGLyVoqTfxN6H4LmpGl1xMMV3NKCAVCUFoG5EXg94fGAdKxobqwnFaLuiZt", + "EyIwAfqeNrP2Az7wEk3IxwbtG6JqpuhMo0+Vrp+Kt51PUrXpafsEt7171jVNMEge5BlK8pXUElMAEqoN", + "A3V0/W1cZGusWWYy4D9tsYWcrqrc/j/IKKo9yvUnG0X1N3PM2H0MVofZuCsBzhs4zut6ACYsMzqH5h0R", + "Ny3DpMJ9UFv6X8U0ezZ+9sePa99KlMbiKEX6L2USOg4JJCJagb6a9A6GYHAwjZZeUBLNmhdQzYwc7g+n", + "4rXZ9Ja1Z0wkWyb3eEU1OfZFIpV9x+o3GGdBHbbf0Tr75hvin0YAdcg0uUZOw1v5a0zOw4cTL2ETBpdG", + "PZ5APWh4PWEfl8+ruOupwCAFrCdtXzdF1Wwf0DXu6bXBUoUkMO2vbRQxvvxpA8AxZBAGwWNJ8vHV+3cn", + "n66HHXac+QlOH3sq+1rrG1Wqqw4ON7TbNU5XvTuvaRygurIpZGY1BuaIobqwHIMf/kuWuP9kxbJ8UWbE", + "p48Na9qnglUqVnRAgmzI/28aoUZx8/LF2E2jp8+/s78jXs1vEGwR6ufKkiB776QY+Bj8fXvsMKdqNTXW", + "WkYGP5Pz95dXcNkyGY2qOKAX4xdjIEfHg2Q6NSMP/kam7hHLwdUmZxPSlBrTyLVNyZOu44ktG9Hail0Q", + "564Mm3bLv0+j+08V2BomD8ZjOMR40om2y//XUeZprcJPRVFGbNSQ02vTHPug1z/CoGm+VPAn2zSt0uP3", + "Ngx4BIKw057Z+VmZLl153FO3fEgu7GOO4FQYx0uEfO3drHgqZNEl5OsvgHivbC9UFFDG5i8z6E81g8zo", + "h3/86G+Ch+Yw3JuUojIn9oR0llL1fsF+n5EWGFrGxOpw/NB4s7UrJ79GS6a7cs+RpI1po8Bfa9cejv3Z", + "fVzdcMdV2duYvDu5iAmW16jVqPTn7e5EDE+y2pdd3adnpu3f/S0WhIsaNlrRG0auB/zFNVHlYsHv3JGJ", + "vYPAQY76qjyTPfewPZwanGelgpStrbMK7zfsIYi9kdthSY3bu5N1rjcY9NrecAv+jTdTEX6zsnCfSbsr", + "/KrCz24LqCrvwRCnDqv1wodGwHWWdvaj2hIPOGYjdKVjWFcRAga9eh6T12dH7wbmHy+PLq6Q5s7e/v0N", + "6Yh68bfINQch8A2M9W/50L2db+DhEduwQ9+ecaXfulqwf5jCa9Q87xOfyl3K/3f50C/pv+bVxlmXl4bC", + "0NNxvyvrGUPBEWEa1Ct7MILLRlVVgVxWQANvhYXOG9IR48aMTYpnqRAkVaTkWL47O6uVBbUhdOcnF3Br", + "DxOCeXrvehqUU5xGNsrj/cUbCLsJ5111wFqLaJ5W1RZ957P3x6bzmY9T8x3fsVvyX7L4jF0hy5r5bm9P", + "L6HfW64SlmVUMFkqj82WD19dNe/ixbcEkj9//tLWF3vQccZfDKUQdzC948aRw99KSOB5TRNGXp5cXNnL", + "+Paemc51meeukJKuGrE3nOKZAv59XT2k+fBNdOB3dBQ49S4z1GexZ8hbKmk2t7dVTLN2vNwSffWqZJz9", + "UaenrRq9f7K30a762XWO6USCDYP56xTzf4L5HsT1fr397umroV2cfjKG4jblNLAVzPLegC0f0FMVgNOS", + "UBsCXFNI9fr2Th0dY8DXhauk5x9UwKNEGw+2LpWeTMXB0Ac42/FcQT0U2i5qRk3FU+NAQxUoKJDgyu1N", + "xeGQXDKRdqzJ3vcSqsi1Xd81zoMrkjLFl4J8ZixXVR6tppplTCkw5SDzRdkpQ9YHSmmD/6oSYCaXPBn+", + "Zo1Us/E79JE7BcdeM/8BtBLGY9aj+PKCLVjRqdLqoXyP0CQ9hRm9NgEqwaZbSwT6DnZbggPbaasm4lsL", + "6cxCmhDYwGXJU0YwALKKqjIAoFyia01eB+USJ+QdKwuaEcG0VXEFg86NC8CpeC0LT35enzuSB63fjGWM", + "IcVJeyIfKJ6yqbjO+HzkuxodnnyGLJIVT1Y+MLKip4frRzZ1qwF9bmso/lF6NawC+qcr1VoJyA7Zahdv", + "N+gvffo/Q586ZvwN2tSAqFTfplJ6TqHCCcI2jdqol7kttabjAAIcvt+uO2rnHI1h4biLiUSmrABdEZOU", + "wV8DLri2P2EEcNUIEj1Vl6+zw3nJN98Qnxjggs/+7nrUs7b8gaSDaBU/ltYeAAkZ64TyQnlAxaYPiqrK", + "2LrHAkIbApxfWtB8VVDlM7UAey6YkaR8AQkxELSUulwxW/jU5ZpVdVBNL1ebPKNiWcINTkunGm3ShbQK", + "pYi+7ZefW4qZ1qNtnEe3bKJ3QqbleHyYrDL4PyMv2UYaI6/nZ6VpAYddipxoTcl/wGGB8fxoyrOMJnAU", + "WKTqQQcQsPwHRhM3KuD+6TqqXge2L6K48MVa/1JRf93Y7KSjAvHUpVdQUwWVsrZe1DQKZ8Vk6Qt+xWTu", + "i4uhQmhX7hp2XCLrf/hKWn8YezVrpnUg2zYJy2f995xlNy7ctMNdfWbQDIhSQeG97oTso/NTS7o++2lE", + "cx7df7r/vwEAAP//LcZSyv23AAA=", } // GetSwagger returns the content of the embedded swagger specification file diff --git a/pkg/termite/api.gen.go b/pkg/termite/api.gen.go index 12561ee..b52bf1c 100644 --- a/pkg/termite/api.gen.go +++ b/pkg/termite/api.gen.go @@ -303,6 +303,10 @@ type EmbedRequest struct { // Model Name of the embedder model from models_dir/embedders/ Model string `json:"model"` + // ReturnHiddenStates If true, returns raw encoder hidden states [seq_len, hidden_size] per input + // instead of mean-pooled embeddings. Useful for passing to the decode API. + ReturnHiddenStates bool `json:"return_hidden_states,omitempty,omitzero"` + // Truncate Truncate input to fit model context length Truncate bool `json:"truncate,omitempty,omitzero"` } @@ -326,8 +330,17 @@ type EmbedRequest_Input struct { // EmbedResponse defines model for EmbedResponse. type EmbedResponse struct { - // Embeddings Array of embedding vectors (one per input string) - Embeddings [][]float32 `json:"embeddings"` + // AttentionMask Attention mask [seq_len] per input indicating valid tokens. + // Present when return_hidden_states is true. + AttentionMask [][]int `json:"attention_mask,omitempty,omitzero"` + + // Embeddings Array of mean-pooled embedding vectors [hidden_size] per input. + // Present when return_hidden_states is false (default). + Embeddings [][]float32 `json:"embeddings,omitempty,omitzero"` + + // HiddenStates Raw encoder hidden states [seq_len, hidden_size] per input. + // Present when return_hidden_states is true. + HiddenStates [][][]float32 `json:"hidden_states,omitempty,omitzero"` // Model Model used for embedding Model string `json:"model"` @@ -1337,197 +1350,200 @@ func HandlerWithOptions(si ServerInterface, options StdHTTPServerOptions) http.H // Base64 encoded, gzipped, json marshaled Swagger object var swaggerSpec = []string{ - "H4sIAAAAAAAC/+y9C28ct5Io/Ff4dT7AUrbnIclOnLkIFrIs+2hXtnUk+SR7PYbE6ebM8KiHbDfZkiaB", - "7m+/YBXJZr9GoyTOHtwNECDWNFkki8V6sar4a5TIVS4FE1pFk18jlSzZisI/j5ZUv2NK0QUzf+aFzFmh", - "OYOPiRSaCW3++f8XbB5Nom9GFaSRBTMKYBzZHg9xVMiMPdbz3LR5iCMtZXaV0Cy74qnpkzKVFDzXXIpo", - "Ep28JnJO9JIR046YdkQvuSIrHJRwRQqmcilSLhZES7IjRbYmc1kQM4sfTbfdKI70OmfRJFK64GJRG1e1", - "R710YymyoikjszVMgSrFlaZCNwfxH8xIXLOVemz1ZoQjmmUwE5waLQq6jh4M+tiXkhcsjSafEJWffSM5", - "+ydLAMcdmG+tw34ndjOH5KLMc1loRfSdNNNfUa0mUzEgF3yVZ4wgeiZkGv2NZZmMyVLeEVowspblv08j", - "0/LQTNNsigVKcmoA7nzImTg8Iasy03wlU5pZ+LsT8unXKcx/GhnImt3raRT7f/nRptHD56mI4kgK9mEe", - "TT41l2Mnabq50Tt2ttnryRPedg8t2s9oodvb+Bm2qBQ37V05JIn5AIRtlnLH9ZLkUnHznXCB0+BSDKO4", - "cSqZSK+SJS3aQI+WtKCJZkUIiciCL7igmR1oyQpmB2ciVWSH3SdZqfgtC44IF5otWGGW1HUiL9iXkomE", - "EVGuZqxwxxOh7oxjsheT/ZgMh8MOmHF0P1jIgf215EIf7JuBlKaF/oNWBrBU53pM247T7qffS1SNQ8nT", - "yAKrTT2u9qf7xJbi5kiKOV90rBJ+LwvYeOAsMCXD1MzITJlDK8klK1ZcM3J4djKcikvDCbkilChzNPic", - "s9QsYs4XAMJszN8uL89MczIgKZ/PWaHIvJAr+DYvs4zAtFiBE5iKuyVPloSLJCtTpkheyFuesoIolrEE", - "JkdFShKaLM3cknDaQzi9dYpd0fsrWIlls3NaZjqavBjHTV5F7/mqXAVkhd3Mqgumy8LAZvfUsADs397f", - "lUxZVhsnmvN7ZnarZ8vNGqCXGaZUbEiOuV6ygjyDjs8AjcryHXnDxGBGlUGy7RwTWRBqQQi6Yohc+FuN", - "EkStGv1qPj2MhuES/NRaoknesiKj+RUM+Bje3nt82W45CkLTlcyYvmNMWFQ+jkDFclpQLYs6EqcCdraB", - "Q3PwfAdAFKzI46a2WAuiLYZpsWC6e6mttV5C45Dz4DJzZo9LfYWdS9TLgqmlzNLaYOPhi7jrRKbA6nwf", - "WOWH9+9/tjtMdsbD8WBvON4NRwZgyMXNNmeSBiwFJw8spZtDnONx71TJLOvYrJFVXKaf5Zm907KNtKhi", - "KZkUC5LKpFwZoamXVBPBWAoHcsaIyjOuCRdaErWiWea2QA2Hw0cZKMzqcz8GjEKnQIX0M/s1MjyHXS25", - "jiZzmikWR46xfAol457ZdyO5xnW5MnbI8GsEFsgLpXHmZuJGfQhA/WBB7dVB/dANS7FEijQA9tmzJHvY", - "H5rsMVhTc49+WjLgRAVTZabJHVVEseKWpchioGeF6JmUGaPCjBCy205FCBUoxxK203WAUFpaTo3l1oV2", - "jbnSRJc0y9aGxaZkZ0XXVhjhWqyEYynhczKnWTajyQ2RSVIWBUt3t+GaDQLzq8P5xQGiO8nOH6369tCc", - "X5VFx/o+np86zceJZLaasdRYISMvWcx+8YTV5r/UOp+MRplMaLaUSk9ejl+Oo4BflAXvYpQGJYYy84LL", - "gut1jX19iqQQ90a9ymgURwtpVlmf8SvsT1x/Igsj1g1Dwz3KJAUbCtRRCd1oRlJmVkBUzhKjXhRqOBVv", - "YKoTcm3ndG2EoPtjgj2urU5m+9uZAvO4pqWW10MjERzudMGZIkzg/0G9M5MzmkapWHhS6S3lGZ1ljNjx", - "/g1HmIpErmZcoP4E3Ep5Y2fJcI0w5rffWlSob78lOynLDVakILOSZynRdKF2wSS6XshrMiBnZcHIW2n0", - "claANNih2R1dq2oqMTk6+0iMSRgTlck7pvQuQDCbYmCAxDgvheYrRnYsnSoyMIP9OIW9iz+cXxqraE6V", - "9v3vM2q6v5XvTn8mP58edvS9z2j88+mh6Xp59nF09PH14ejo7OMurvQ14EZ9+y2uCBBPBuSw1HKQMs0S", - "TWashtUdu1M4gaRMYQbv/3Hy+uSQGOjk7dlH/CYLtsoAXG7UoyNZsHen5nAnHy4sNuC4z9aAAISo8xKX", - "JBcZM1MmO67Rz6eHdlRs80YWCfOoxQUd4znyK/qE2EN78j6j+I+FnEafDYjLYu3oRCFhGxRURKlRpQ1B", - "TcyaPbyJzkv8A7+5v/wIZwUzbOzt2ccYOBcMZ6j8yKLJgwV0NXojrmBiFn00SVjGUJ+e1lTeT9X08KCb", - "uYG1jBOzP+K/kAN41t5Wuxps3Fo+V4olpeMum+QBFXqerQcLeZXxGZ1fqaSgRvG8kjkThmta4/jCwqsU", - "khvG8iua8VtW1zFfrFoa5t/kHWohWhLTzSldhlMZWSHIiq1ksSZ0bszDjCptKI7sfMgyuqIDM2+q+Sxj", - "u8OpeIedacGACIyJnaBMEhYgggH3UuosMTknXNBE81uu18Op+KiAG6SBpQa8cBq9WE0jsvOCrLgoNVO7", - "ZqP3lua3PbKUZQE/jM3fjC7MdJHfmqm9DrijA5TRX9a+Dcn4DSO4rCG5YBpbGmhaEnafZzzhOlsTJuAU", - "10YwNp1hxYY6VxUS3KI1WsxlDuxWsFtWeJQM6ySIm9QipkwunkovmVwsGuRi6QMMRilA9At95Yzfura+", - "hfHoQQSc21vSNDNs2izvME25FXX+6x0355iRLyUrWUrK3CDZzAt+uFL8FzacCrsLYxChpci4kWRpxUAb", - "uHveabDS+yvE9BXuzFOXaffTHZDWyVB8VWaaCiZLlRn6/WnJBJI4TBj9p0Y3SmOQkxkzp6hgCRPa6Wuo", - "H3BVnZSd0/OPxDBRM7HdbZBBPghDnvO54bi3zGgHglS8wEAXUgx+YYVsIO6gD3G4xKvVbDukWYzscEHe", - "vdq19j7M1y4KcdmNI5rnhbRo6kURnqvg7DyOFW9uCULTW67MDHHQgVWa7bynogRXLuoryigsuC2GGhUc", - "XaNVa7bKZUELbpB9nzCW4kJuaVYaov1JFjeG/KVYKJ4y0iJAa8cLNlgUlAv0J+lCZk1yHv/wXd/GVMfk", - "qeQcuroACtJJD08IiLfaNXtszbd5mWUxEeyugmt2zZDbi/EBuUDtnHwUlfoDovic6WI9OARpsGQ0ZUX/", - "XuJgj9B53/yn5Xh8wMi4gdu9cb9360rpgmq2cAaKZ19nDUdxuTIqAwiCKI6MNIniaCZLkbI0sIEqLl7f", - "mzNWDOp2gR14DX6mgqdMDck7mqvA+QX7ppeMF61elQAW0rBke4tDcziFZtssCqtx0NMmQzYBSt/JPPgF", - "UTj2O2Dl5+6EhAKU7Jh/GFgpW1GRxvaUWpHP04zttkD/CAJ2UhemFlIlNeOayAS9GyjROQWXxm53S5ri", - "hkyjEAO4KGAzbZm8QxVcWJjTkRfMjuHa16V1TNgtE03W6jGUcyFC4QBzBQYE4lCRFb83K8T1Go4CKLR8", - "gePhUnTFSC5b/ODX6NXh4clotmADcAcNmBjc7g1fRBNPgyte6OUoWUpxsx6szJHXti3cwWRcLEqaDfai", - "CZJrl4cMJ3OV8o6LgldUMZLygiXasFHDtygXlRNClTP3lRvidcYnBWuIq8QQtnLrnRdyhTbGr9WgDyO0", - "8lmhRkZ/P3Y2v/cHVpZOrZv3BJtededIf6eCFbTqdQ5/bdUtkQvBf6k6wt+grj7a9a7g2nW8YF/2L9gX", - "4n5t9q4rrv9nNNSI0ZFrp5gmt5ySW56zYndojpc5/YrpGGw6sLn1gIuG5xgkouPJTR20NU6XSmoodAsJ", - "9H6zvpjznGVcMPQwowuBHNNk6b/AiVVTQUki87VzCUHLmChJlnyxZAWKXwWsbiUL5nSRWanx7MGvUxFM", - "wskIr+2PncYS8sqdFRc778vVkTE/n+82dc69LjFiuUj7/Jxypb1OWfFz277GlDoNrMslU6zDwuCrFUs5", - "1SxbI29yBw9v6mJCbyWHQwQunoFdOsmoZiLxksPOSC1lmRllRydLopdSMeBonoiDE0p22HAxNGZXN2+a", - "Rmba2yumdaMMxmxi/FMfG4wjMHX4KMl4PrjlGi6SBrlZxsH+04x1i6ArzVdMlrpuR4+jPjULvE9akjvK", - "NWgx1FEZXAVIswLNYnv3Z1ZoVTDT3nTeaP4ejBU6N/ZW+H8wdYUkdpYxCQyjc6cBoY4KN1ZGENm2gZr2", - "nLylmt3RNbnEb012cDDuZADq4CopWMqE5jRTT3ZnHFSWaQCl6WV27uEel7KPDugKAQiCEdAJW8UheE8y", - "2QFXvSwIX9EFXNJXsREbw0vYva6HJ2xuf2LAfzw/rfX5/BBHIOB6b6S4yMuO1Z2Yn/0KtcQFhdEny4Kx", - "RvyJWLjQDheEcj2NlizLJLmTRZZOo+ta+EnQVE3AzbbEIBJDe7bH5+tNESsVxncNgEeDVCz8h5jUmsLW", - "GDLA9sGfE9PQ/msapVTTCXwd5WLxv8zx/+55PBwOp9HDg5lpnY8ESze6E9y3g3leGJXGkFx/mEwTl2Rn", - "RpObO1qkJGDZT4mdCbHdC21rDtY7THAIGptVOwjqj4vR6bm/em/UXCvNnUCxgjG44G/Imxpf6pUDbeQU", - "pUiorqsquihZ6/rbNiRw7szBmnNtZwXoutckY2Khlx33gg3W5e7F8Ah3MTB79DuvYj2HUtHk06fxcLy3", - "fxAPxsPx8xffxePh+PuXP3yOze/7B8/h9xfffW9+f/nD5+BOtAdFrUvScLRe0qmY5i2o+IrsSMFAcUN0", - "IcJrlOP/8dhlfVsGb3kPipoLuIgMk/eT/E2k0rOFAXo697EoMJ6jgVT3c33C0NpFVz46BQTSNeobLrha", - "njOqDNjmKPg7uVuuyYIJe91BlJZ5Dpe7zn1hfoniqCLqKmizurOY80yDgTkvBTgkoUWnj+ONbfGazTla", - "RO25Vd8MWVHioOKlYkIFmTEIDsVbK6/vt0KfamDbGkDwtxnozkDHoCw7XipZLWQnesuwhbMR7hiFCAEu", - "IFwjASx2cRijOXfHvYmAz1ULlbC+2tgLpq/seJ32Fi3oihnLsd8r1cXS/uPiw3tyAcyaIPFYvMxQJwxm", - "FQxR8zvUke7R0B3mx/UaPBtKU6PqGvuAXFBB3hRUJFwlMiZHh4+SvR+ki/JbPxggSV1Lt/Er3QEfRmES", - "c7j8xK7V0o0hydPGPvexeNj1rhm+xSPHjpaSJx04nMPZvSr84d0kYWsH/cEIlJTdd+mEKbtHSuOKJDCy", - "8yllXOnOeM1MLvJCzjrY/qlcEPOJznhm9jSImjVGh7/8Z2lM7HW9KDOIBDf/pzNDO0iQbR9TFRO/aeEO", - "iy6EvhUkCoiowMUNvG7emc7I4QtdMLoyJyPgmjby9uLiGNx/mhhFc7fFjRDldbG3zfIwngwppeviuGBU", - "szTQ+lrxwy1m4WVlO/IRMRE4sJMl1UNrm3Iphhix9nmrCF0LrZpkEA5ksfHoJvSdkZRlmm6Lw9fQ+CH+", - "g05WE9E9hIdT3LTC124RTQmYaeoVcIiAdYTXJqu+3INLecOEB4Jz6T18FQX8ptSRTSkcOLSqL6PKJlFP", - "zdjwe9lK2+hF8zZJNm1ZZY84S32ui0PmjsEiOqoqFBCuSF4wxTAL5c9HdD1XBq0SzJOx08IZ144ARE56", - "4H9i9ozbml7XxoredwUl77/4rte9hrHIWvqdC/UU6Nhxp4dbqzoTAm5ZoZDHu2YuayXwvD45ayXI+tre", - "jgntYbs82WcQ++8Ni3gBMV+jBVut6OBgsDcbcN3puoNT+ri+dDInQNvgIuBwU67sHTmceBfGAbRHFfES", - "UnWGzWq2ys28y6Juie81x70wCwImUvWAQHDyI4SUFStD4ponsXP//4he/4KKVK4aoeLfx9tYn3BGvCh6", - "9GxU4lrKrkPrXP3wmew4HVvtBmfXWDrmWDoXudeozBG3Xn64Lq8y+HAhPinExQ9tfaY7DW2ZX908kgNx", - "KfPBDVFuX3Z8SO69ZkJxKWJ362yTvg7PTroznsxo+ebtf18mGStVNVygh0Zbxf13GfGeEWzmVpU3pj6p", - "NltIwFr1ipPNklRsk2LYTSQBENuW7FgTNFs77XpvawbUsD4265SNqGvB7+GOQGm6yskOBtyrXSsLjYFX", - "zfWOKlIpftWd2Pfj53v7B8+7+XFXxtshKQX/UjLC4SJgzm3YNKa91bBc43fmW7LKswGdJXv7B12sbjuv", - "UaXsd2YLeZ25rUNYk9r08XbQtKlTT6PA49L41ulFKZ9iHn3sNo621M/dYJtOxcc+5cotoibIu69/rfDm", - "TSqKum9Q5SrXT4aK3XrYjqZZL8BL89UBwwwKhEX+LZhqF0NrYL0+8bgDRY2pdKHd3RV12PjmC4HUiAKs", - "UPLx/KTFb/rzKapOZKf3tqSeDdLdjP/j1Yfzu/F/vl3IbbKR+q7wum7FehYdGkvB9d0mba1xleaujR47", - "Vx79nnFWxnIF5FEDGb7GnT0qBGDEVih1mqIDo1o6eKaPqEuawS6WDfkM0ymm9Eyj3foFmEv02Tps6El3", - "6P7WZtPcWTO8Z7u7nw1BAN0fJvzlE2d/rwsKtxybpv/2lL8/Pie+sd+CX1ghB2op4aLdxwaBMpeUSssV", - "yeiMoUHWyD8wtLwqBdfr0SLjghVXuJLb/eGeTUZoNcGN0lTdPG2VlSGxaZW+1ej09F3fRvXYJJ/6jBKz", - "jgITdkZv7bcP79///NRADReJdcXFXG4K2NxohnswcCBPDKhWwOZrpinPIL6vcoXSmY2msP2rYD4X4pHQ", - "HHVXztRwKt7RvB4CBKHvGP8t5nJIPkIEEpjt3thheHkRgpoKRpNlOLLPxtpB2oqJoUJDhDEpWAbzVTGh", - "Qt2xQu22wgtTlfHVCE4/RM68Pz7HBMpqUHDNA/Do88PjtNjfO47c3CKziXZyURzZ2fWDD0/DVvA/d3qO", - "ghC+TaTf3tgm3YexgHXC78JnHP6a0WLhfn50rU88FjaucfPa6uGOPQzWwRrc7g9WB0+dh4123DSPviDI", - "NqZd6GR9ptnqy2I0z6gY6BeW66svJU0HXxZRXP8K++A/PmEpXcmveE9WoToUeTXWWqe2ECtdKoHnRcdC", - "2zyxVomSTTU85HyumCvYwQBGVy2SIBa+82LInKOOy2uEZ9qTnbPj85h8OH8bk9MPRzF5d3JxVNcdz4C0", - "2+6nRBasp0AHlgOAFujx0ZK0s/9/+GEbnw5EPz4VVTO24KJ2FsZPK3Ri4djiJRUy/kMuBblY8TB2ZFPy", - "vtsDtw6ofRI57G2knF6fq+WObZSgTmKnjq0w7w0UG7JjFR17MqXI1rsuIaRUhoVQEjaJyVqW4N/CtOY1", - "oWLtMbPOWQg9ngpIqjNgKr3JwDL6khGvKB4LpgsMNx9OhQ10tsVTbGoVtHumfPCuXQjkDKlWrt2nKGeF", - "AgsUzAexRpshLcFkTqlmT+N2j3h2m7Jka1HSI0naFw5Wjl71bfO5bdDaArIzr5RX0FKfeaH8rFI71j6a", - "F2xk7xNy0f9W/3BdHXS4Mnbk4ZSDmJwfvzo+3d24lw4CJFBUSgzhGCaC6cWtbZ3bHJw4upPFjboC7gAR", - "BSy94uJpm2rOouqu71HDIBA3Zyi26vOpTj65w+wwbdOzjeWMud0nIhkSDp6MoxJOrJDD3yGhnJsHp/8I", - "t+izPt2aNgWF4ZmGSbQCwszg3eFgWynjVgD+9tgwd/JWdW/fI8dn03I9aW9YsDsjtUs5xyVxKhuO2MaT", - "8htwiUM8jsS+0DdHAhtJqDjy8+8QLtXaIMCLtjmhYx0QqYzca0KsrhEykffH57tkUHEFaMpSonIqFKZw", - "15QSA84ZApMqOpoWM64LWnhRR8OcFKwhUZN4kMjjt2tCzjs4nKsE5RlB70YaYNbOmZBjC+EW0iqVrf9l", - "PoJ/qwljF5mddUdtbU11kHuXzdsRqNBr87a3sX3xUTPOem8/glZo+fbSR0Pxb9t68RNZTEi4W0Uc+APV", - "YpZLhik/T2RtPYq20ST9Ti55Tqw7MSjT46Xcb1aw3R2L52obNO4XW92iUp49GQcNzgN4tKACHXiD1msM", - "sP4wg8eUspoR3KGTuQzB7oDihnXcFbgJVwId9H9WmO4iZQVLgzJgTrFAyEMCUZUZh7w5V4pVOU+3Z0Bi", - "MRVUpAQB8rCw2JyzLFUjzVZ5RjVTZMbmZvcTmmU+L4iJNJdctJWpE6FRJbZxq808CXT/v4baCPYnMi9F", - "Ss3YNINiZU9SuL6UrFh3VcWkRbIk8BXrwrKM3VJHzM2I7+Y0Cc3zjCeeM24X+I1zqfZwE/31KVFP0008", - "Pfae6269PkAFC9QSnHiMScW25JNCZaWmmD01PL8HX3aC3WgCp8fmTCfVl+oUnAoAM9IFFcrMuk6tl+Zn", - "Q+Wm7VtWrKiYkI5Ku3+oYdftwNrSf/WI+2o7OrXI24j3P4o+Dbge+uyxlXAK2hiLsI071KnTqEXHpKZF", - "x6jtGKvI/DZjdNVtRDxuu/1OC8kG8HVIZpkxzF1wEVEK+C4mCiRBYFmYarFWmq3gwh1C/H1Na5t20aml", - "NfMMuwtNdl6OttSxDZUqm6mEnbvbuAttlJjcfA3aW48SopI6IkIgftTHTe2i6dCOnWot0vV4NPa3nagS", - "rLG95eDytAFdYXQOxF9O/bDTyEhpH8m1G+y/n9m2OOvoUMcbRGj24Q5Cc1sBo78ZX248h7e+WJ6PPZE8", - "Ng61nfcCcW39ATxb7gisdtO2/Lad4P4kbLkdPQHfJyIpGGpEwXwhGGNz9Pdv3R4fRL3dHmEYMZSJKJQN", - "Jq82DuLdO2v8P56LEr4rYMN0kB93etWfstkdU95+k3s38E2A8PY0fK7UNulptFiABt4hEiEry6YOWzT5", - "1rbEjx+rdlp+nfrcKEx6buVWTaOHPyBDDde3fYJaV2ZUHGDg8xYo3+Ls1Oa3zfnZsAch3Ar5wc5sj0a3", - "AsRlH2VuS4U+CrllvutCZgoU2UoSYvlWKTM1JB9ydFBNxYBMI1pqaYgEgx1TlvCUKXJXJcEBEqkVaWFZ", - "0mkkpGBVXwj0FlLbHmKNA2JTt+9V81WpXFNti7kZDc/0MV1wuRNbfpS60rdJLT/Sp4A2Hqxwx9ssLoph", - "nlFAe58fT7nfbtiNnLj+5Tcnf25xhvrzH3/9KorF54c4+gcrFJei32SAQr5QHKWjRhQW+XUhxDUOsj/e", - "fz4Y7w32XlzujScH48l4/L+7DtmC66tErlZdRbvfcggHXnFNllQta/BRi3jeCVJe3eKyOkBKYiORiGtT", - "z6rYG+6/GI67wPbCdAHyXQBv94bjLnCNzaq6BviIQ+TXltVFKc2qKy5UsKq98tcDKn89oNJPLz1FZLuS", - "hRetx0pAMvtKVFgKUrXoJWO3LPu9xW1PAQjs0vrxpLvHoF0AkE75vN1EauWiIC4x7kHYLStmhmTWBPFQ", - "qa8pm5ULcO1A9zuKz7O4chAVN7ENWqxpu1XWporBhjTrna4sdV5qG2JNANlD8sx1w7dcEpnJgv+CdU2V", - "zFhMnv1TSfHMaWplosuCpaBpxeRZJhfzlcav+AAMm895Ag7vG7b+ESq7kZzyQsXkmZAyd2/GZEzoYYCy", - "YPr/xLgOhA1qgszraAsaP4q6nmJV7ccMkoQpdXXD1p1PrR3+dEGwiVkYOXkdFGy6YWulZcGIWgtN73GF", - "LCmYJpmUN2U+JG8gBdRVPz/86eLq8Ojo+OLi6j+P/+vq5DVh4pYXUoDP/5YWHILruC8IWH8aZy3LYoCT", - "Gdyw9YB3unzcrUAHjz0IY+5dO1cP7pk6GNIV/UUKeqeGiVw9I7IwW+2eZPhhPB7jNr7j4uRDPWys2RlC", - "1MUpligJSu4FjnHA1FWF/27kW4RWe/B7N+Di+Oj8+DLYh9+wCThIsBednn+mlE8i6UpOsxW2cZXQFg8T", - "Hitbs3hNgoptT1p717RhlAHOqDOHiV0plT1ac+kYC6pfXJyOLk8vYOyLA8M7BD4doLxpMoFsVGhx+NMF", - "vAbg60snNKtIqaNsx6OcfMv6+u0zj4XOrwxZq643Zrhmmb3ptm2JaQu1FkcnZ5g3n3FxQ1J5B3VmFdTx", - "ZKtcr2MIfALYWFffQjBqEcs1yQt+SzUjBg6fk1kmk5sr++MVz/HhnKJku8P6PYn9pz1dSSqG9V/2ftgf", - "jodPjjt2yMipXm6LDNOW5AWb83vmimNnbDIaYRmZA/Ovj+enLaTAGCFShsag9J1LxQidKZmVmtm2ljmN", - "PipWqFFKNR3tYiccxXSZlckN0yOcj+uxWg/s72UOGzRq4jOEadhVq8PT8Njax0dP0SvTo1bduyINUlAB", - "qe17+98by2M4Hr2Myd44+Pf3+8O97+Cvvf2YmN3f++4l/v1dTPa++2G4/+K5/Xu3M7nbEa+rmXllU0dr", - "Mz9oJxbbepRm37lI+S1PS5r5o0BkbhMzITTOwQyL149BOvCV0QH2+gqm+9kp/gu7mq01q09sb/z85Yvv", - "vxv3VlA3/QzVOkC2bDtWsyUIsBZU7eH5yY0fsTW40N89dxPGtLGUrzC9ul4iYfz8Zd88oR+546lejpaM", - "L5Ywv5zfQ1oS1vbzefsFM8uqP/+EwDdhtM1NHx6snmorblDMlkW/SHQInDaKMS8RnmdSk9FowfWynBl+", - "YxXydDayZYfbuTfOjMBC/rY4LsR7IeuvorfgBa/CPyNnXxN8d1o9PjAV33xDflpSiKewgMOnkmwcqnJS", - "5TSAji/c+BkEKtDh2Qm43b79tipZ/danFX/77YTARRs8I1DVYNw5Oj05222lwSEg6OBKWRsIF2xFheZJ", - "lfIH8wmfy3NvBA6AYN1LXwjP17c2sKqYgYINbASFFfwQaIGPotmZvKcrlrqAvKDatQFkA9gIBlOrmMhi", - "QQX/xeUbOXe1exWT3etgeefuPhkw5C710WTGmEl3u+5S63yUXJW0HduwkPAXppMhVtB2rAQKYVHN4GmJ", - "jFEjIDRx5Ig0OORylMpEjbws9vQCtWAgO6uDZhIqSFEKY9mLlGZSMHi2sqosTwXBc0CSrFSaFUAsp0Bh", - "1f41KM0wRnavWQGq3dmJfZuCiQQSwAYdZHg9ojnHrMnrSi2v3QtDT09K1VN1jgDOD99WdblN25BUXLQU", - "8OuVOSosdSTzpaQZ12vT5YgJXdAMzEAbwseKW4jDSQqpFEm5kX6zUrOUCJniQGdGZCXrAUTOYvPa6YOc", - "ZVuSP2P0lilidFHToqDesty1W/aGQeEQu4PfkK5ziZSIWbiGAsOjVKta7x4F7K1Vj5AOz04AzHb74o4l", - "+lkJvvoG83jFhVHXvVM+BmvZreRdxUCsDm8ZCQLEhGm/XABoPpPgAZJ/Al2YzR+gyKhYkMppwiwkCL4J", - "5wXp3z6LXJGd69488utdcwaM6obAbKr2kUeKAfhRMdWoRGu9CzvXv6kIsC33e21xYc7rEVXwnhlBxCC1", - "xgQIcYBoLJguOLulWUxuuTIaiOIrntEC6BmxXmPHTcJ5UzFdf5haz9jtkn8LKSyAQV5bOlsbYIe97yX0", - "PnqAsI7w0WADY3+Az2SRy8tTVw0fnmjElm+sZIC51+za5kMBLgRoTnnmz5IXJduuITxYHQsJHmJAiH8v", - "DXv6xQvPw/qDYoZqvmCT6m0gPq9QHdCv6V6LtQSWZ0P6yI6NrVxSkWbGAOEsS4O4bgy4znjC7FWI02qy", - "jJwb/UqRc4bPhrZUnEqmZGxBsfgO1/gKbPXMdBRcI0S3ezTLl3QP6o6gJRpNooPheHgQxZG3q3DnwRqV", - "qss9k2fcmAFmpR0vyFrRqpwQqGsNjYICTmV6Z4kWKQAIvnpoo03rwaP34bvO8Kqtrp7BxCeVTeOP7m2D", - "H33BAvPzG6pQbapVfnLTALp6549TW0Pyb9w4JhMe7AF5t4nXB8F79ZP25BND7Mk0f52jZgWmK5zI8Dkx", - "w5v33Qt3Q+jGwFvoqznMpF66J8gNWsLH+t1W2UcczV/X19f/VEbQ/ToVpFZtvechYmCgMTbGYZDFCkLM", - "T7A0BGA3KXafai9NmyYvxmP/sf7kNn71H/0j1wh4OhXmv8h8fpiKB1gF3gM7c/AkdW/gXuIlhzV9X8l0", - "3Sj8F0T+jv5pq0LiVcBWb/K6iNWH+hWLMcLhBxTecCr3x+M/emx7BQuDd8XtAVmmRJXgPpyXWQYOhOd/", - "4EywvHXHDE4EFMt1Xgcz7os/Z1yrEltbj9mGcaTK1YoWa0caHfxPsQWGg5jmqI/1c1FX/0jZiueBDm0d", - "FuEzDMhTs4ZKr4b2CSivjHMVVPW3Fhzoi89UXU20ihB6cO91jBH+UHIthRxI7NuqgBPYlzYQEWAEKmCb", - "pTfe7tzIC8MXmqo3a+zbJcbKwlVgj8AK0RIKEQdv4QazcRbhAIJnVlaxch647qouuxNymIAfNHw9waok", - "FRKawEBVrnc1iLXOkVtOK/3W4qmmB+OjXsH7GoA1H/L5VV/b6HoYqP8Fjq0e3UCF+es8udFRn2n3dwrE", - "4BUpuMlIwndFKbyHnZbIclg6FVbRge2YZ+APwxeXbvENqLQUKRUaEnPdqWgaZOZnf1fiY9N9yDTkUhuq", - "sQSFDxra96AD7icTzfQAQ9WuvYmnWMGNoWwTsKzBF5OVVJr4m9DdFjTDS68nGK7mhBAyhKC0DPCLwO4P", - "lQOkY0N1YTmtFnVN2ipEoAL0Pftk9QdgkKblpwbtG6JqpuhMo8+VrJ+Kd53P9bTpafMEN70J1TVNUEge", - "PTOU5EupJaYAJFSbA9TR9fedIltjzR4mA/7zBl3IyarK7P9KSlHtwaI/WSmqv5hixu47YHWYjbsSOHkD", - "d/JY2n7aJCwzOoPmHRE3LcWkwn1QW/pfRTV7Pn7+9ce178hJo3GUIv2XUgndCQk4ImqBvpr0Fopg4JhG", - "TS8oiWbVC6hmRg52h1Pxxmx6S9szKpItk3u0pJoc+SKRihyenfw+5Syow/YHamfffEP80wggDpkm13jS", - "8Fb+GpPz8FG5C9iEwYURj8dQDxpeT9jF5fMq7noqMEgB60nblx9RNNvHRY15em2wVCEJVPtrG0WMryLa", - "AHAMGYRB0C1JPr3+8P748/WwQ48zP4H3saeyr9W+UaS66uBwQ7tZ4nTVu/OSxgGqC5tCZlZiYI4Yigt7", - "YvDDf8kS958sWZbPy4z49LFhTfpUsErFig5IkA35/00jlChuXr4Yu2m0/+I7+zvi1fwGwRahfK40CbLz", - "XoqBj8HftW6HGVXLqdHWMjL4mZx9uLiEy5bJaFTFAb0cvxwDObozSKZTM/Lgb2TqHvgbXK5zNiFNrjGN", - "XNuUPOtyT2zYiNZWbIM4d2XY1Fv+fRo9fK7A1jC5Nx6DE+NZJ9ou/l9Hmae1Cj8VRRm2UUNOr05z5INe", - "v4ZC03yp4E/WaVqlxx9sGPAIGGGnPrP1szJdsvKop275kJwzXRbC5sAYw0uE59qbWfFUyKKLyddfAPFW", - "2U4oKKCMzV9q0J+qBpnRD77+6G+Dh+bsy/9l8PL/jpBOU6reL9jtU9ICRcuoWB2GHypvtnbl5NdowXRX", - "7jmStFFtFNhr7drDsffdx9UNd1yVvY3J++PzmGB5jVqNSu9vdx4x9GS1L7u6vWem7d/9LRaEi5pjtKS3", - "jFwP+Mtrosr5nN87l4m9g8BBDvuqPJMd9+g3eA3OslJBytbGWYX3G9YJYm/ktlhS4/bueJXrNQa9tjfc", - "gn/r1VSE36ws3KfSbgu/qvCz3QKCB93h8X+H1XrhQ8PgOks7+1FtiQccsxG60jGsfwzeNL98EZM3p4fv", - "B+Yfrw7PL5HmTt/9/S3piHrxt8g1AyGwDYz2b8+he1fcwEMX27BD3p5ypd+5WrBfTeA1ap73sU/lLuX/", - "u2zoV/Rf82rjtMtKQ2bo6bjflPUHQ4GLMA3qlT0awWWjqqpALsug4WyFhc4b3BHjxoxOir5UCJIqUnIk", - "35+e1sqC2hC6s+NzuLWHCdln6f1T81U5xWlkozw+nL+FsJtw3lUHrLWI6mlVbdF3Pv1wZDqf+jg13/E9", - "uyP/JYsb7ApZ1sx3e3dyAf3ecZWwLKOCyVJ5bLZs+OqqeRsrvsWQvP/5S1te7EDHK/5yKIW4h+kdNVwO", - "fyshgecNTRh5dXx+aS/j23tmOtd5nrtCSrpqxN5yij4F/Pu6ekjz8ZvowO7oKHDqTWaoz2J9yBsqaTa3", - "t1VMs+ZebrG+elUyzr6W97RVo/dPtjbaVT+7/JiOJdgwmL+8mP8T1Pcgrve36++evhrSxcknoyhuEk4D", - "W8Es7w3Y8gE9VQE4LQm1IcA1gVSvb+/E0REGfJ27Snr+QQV0Jdp4sFWp9GQq9oY+wNmO5wrqIdN2UTNq", - "KvaNAQ1VoKBAgiu3NxUHQ3LBRNqxJvfeP1Xk2q7vGufBFUmZ4gtBbhjLVZVHq6lmGVMKVDnIfFF2ypD1", - "gVza4L+qBJjJBU+Gv1si1XT8DnnkvODY68p/AKmE8Zj1KL68YHNWdIq0eijfEyRJT2FGL02ASrDpxhKB", - "voPdlsBhO23VRHxnIZ1aSBMCG7goecoIBkBWUVUGAJRLdK3Jm6Bc4oS8Z2VBMyKYtiKuYNC5cQE4FW9k", - "4cnPy3NH8iD1m7GMMaQ4aU/kA8VTNhXXGZ+NfFcjw5MbyCJZ8mTpAyMrenq8fmRTthrQZ7aG4teSq2EV", - "0D9dqNZKQHbwVrt4u0F/ydP/GfLUHcbfIU0NiEr0rSuh5wQqeBA2SdRGvcxNqTUdDggw+H6/7Kj5ORrD", - "gruLiUSmrABZEZOUwV8DLri2P2EEcNUIEj1Vl62zhb/km2+ITwxwwWd/dz3qWVveIekgWsGPpbUHQEJG", - "O6G8UB5Qse6Doqoytu6xgFCHAOOXFjRfFlT5TC3AngtmJCmfQ0IMBC2lLlfMFj51uWZVHVTTy9Umz6hY", - "lHCD05KpRpp0Ia1CKaJv8+XnhmKm9WgbZ9EtmuidkGk5Hh8kywz+z8grtpZGyev5WWlagLNLkWOtKfkP", - "cBYYy4+mPMtoAq7AIlWPGoCA5a8YTdyogPuny6h6Hdi+iOLCF2v9S0T9dWOzlYwK2FOXXEFJFVTK2nhR", - "0yicFZOFL/gVk5kvLoYCoV25a9hxiaz/4StpfbXj1ayZ1oFs2yQsn/Xf48tuXLhph7v6zKAZEKWCwnvd", - "CdmHZyeWdH3204jmPHr4/PB/AwAA//8FXdNIGbUAAA==", + "H4sIAAAAAAAC/+y9C3Mbt5Io/FfwTb4qS9nhQ5adOLyV2pJl2Ue7sq0jySfZa7oocAYkcTwExgOMJCal", + "+9tvoRvAYF4U5Tz21N1UpSoWB2gAjX4C3Y1fo0SucymY0Cqa/BqpZMXWFP55vKL6LVOKLpn5My9kzgrN", + "GXxMpNBMaPPP/79gi2gSfTOqII0smFEA49j2uI+jQmbsoZ4Xps19HGkps1lCs2zGU9MnZSopeK65FNEk", + "On1F5ILoFSOmHTHtiF5xRdY4KOGKFEzlUqRcLImWZE+KbEMWsiBmFj+abvtRHOlNzqJJpHTBxbI2rmqP", + "euXGUmRNU0bmG5gCVYorTYVuDuI/mJG4Zmv10OrNCMc0y2AmODVaFHQT3Rv0sS8lL1gaTT4iKj/5RnL+", + "T5YAjjsw31qH/U7sZg7JZZnnstCK6Ftppr+mWk2mYkAu+TrPGEH0TMg0+hvLMhmTlbwltGBkI8t/n0am", + "5ZGZptkUC5Tk1ADce58zcXRK1mWm+VqmNLPw9yfk469TmP80MpA1u9PTKPb/8qNNo/tPUxHFkRTs/SKa", + "fGwux07SdHOjd+xss9ejJ7zrHlq0n9NCt7fxE2xRKT63d+WIJOYDELZZyi3XK5JLxc13wgVOg0sxjOIG", + "VzKRzpIVLdpAj1e0oIlmRQiJyIIvuaCZHWjFCmYHZyJVZI/dJVmp+A0LWIQLzZasMEvq4shL9qVkImFE", + "lOs5Kxx7ItS9cUwOYvI0JsPhsANmHN0NlnJgfy250IdPzUBK00L/TisDWKpzPaZtB7f76fcSVYMpeRpZ", + "YLWpx9X+dHNsKT4fS7Hgy45Vwu9lARsPkgWmZISaGZkpw7SSXLFizTUjR+enw6m4MpKQK0KJMqzBF5yl", + "ZhELvgQQZmP+dnV1bpqTAUn5YsEKRRaFXMO3RZllBKbFCpzAVNyueLIiXCRZmTJF8kLe8JQVRLGMJTA5", + "KlKS0GRl5paE0x4C99Ypdk3vZrASK2YXtMx0NHk+jpuyit7xdbkOyAq7mVUXTJeFgc3uqBEB2L+9v2uZ", + "sqw2TrTgd8zsVs+WmzVALzNMqdiQnHC9YgV5Ah2fABqVlTvyMxODOVUGybZzTGRBqAUh6JohcuFvNUoQ", + "tWr0q/l0PxqGS/BTa6kmecOKjOYzGPAhvL3z+LLdclSEpiuZM33LmLCofBiBiuW0oFoWdSROBexsA4eG", + "8XwHQBSsyOOmtlgLoq2GabFkunuprbVeQeNQ8uAyc2bZpb7CziXqVcHUSmZpbbDx8HncxZEpiDrfB1b5", + "/t27n+0Ok73xcDw4GI73w5EBGEpxs82ZpIFIwcmDSOmWEBfI7p0mmRUd2y2ySsr0izyzd1q2kRZVIiWT", + "YklSmZRrozT1imoiGEuBIeeMqDzjmnChJVFrmmVuC9RwOHxQgMKsPvVjwBh0CkxIP7NfIyNz2GzFdTRZ", + "0EyxOHKC5WOoGQ/MvhvNNa7rlbFDhl8jiEBeKI0zNxM35kMA6gcL6qAO6oduWIolUqQBsE9eJFlmv2+K", + "x2BNzT36acVAEhVMlZkmt1QRxYoblqKIgZ4VoudSZowKM0IobjsNITSgnEjYzdYBQmlZOTWRW1faNeFK", + "E13SLNsYEZuSvTXdWGWEa7EajqWEL8iCZtmcJp+JTJKyKFi6v4vUbBCYXx3OLw4Q3Ul2nrXq20NzPiuL", + "jvV9uDhzlo9TyWw9Z6nxQkZes5j94gmrzX+ldT4ZjTKZ0GwllZ68GL8YR4G8KAveJSgNSgxl5gWXBdeb", + "mvj6GEkh7ox5ldEojpbSrLI+45fYn7j+RBZGrRuBhnuUSQo+FJijErrRjKTMrIConCXGvCjUcCpew1Qn", + "5NrO6dooQffHBHtcW5vM9rczBeFxTUstr4dGIzjc6YIzRZjA/4N5ZyZnLI1SsZBT6Q3lGZ1njNjx/g1H", + "mIpErudcoP0E0kp5Z2fFcI0w5rffWlSob78leynLDVakIPOSZynRdKn2wSW6XsprMiDnZcHIG2nsclaA", + "Ntij2S3dqGoqMTk+/0CMSxgTlclbpvQ+QDCbYmCAxrgoheZrRvYsnSoyMIP9OIW9i99fXBmvaEGV9v3v", + "Mmq6v5Fvz34mP58ddfS9y2j889mR6Xp1/mF0/OHV0ej4/MM+rvQV4EZ9+y2uCBBPBuSo1HKQMs0STeas", + "htU9u1M4gaRMYQbv/nH66vSIGOjkzfkH/CYLts4AXG7Mo2NZsLdnhrmT95cWG8Du8w0gACHqvMQlyWXG", + "zJTJnmv089mRHRXbvJZFwjxqcUEnyEd+RR8Re+hP3mUU/7GU0+iTAXFVbBydKCRsg4KKKDWatCGoiVmz", + "hzfReYl/4Df3lx/hvGBGjL05/xCD5ILhDJUfWzR5sICuRm/EFUzMoo8mCcsY2tPTmsn7sZoeMrqZG3jL", + "ODH7I/4LJYAX7W2zqyHGreczUywpnXTZpg+o0ItsM1jKWcbndDFTSUGN4TmTORNGalrn+NLCqwySz4zl", + "M5rxG1a3MZ+vWxbm3+QtWiFaEtPNGV1GUhldIciarWWxIXRh3MOMKm0ojuy9zzK6pgMzb6r5PGP7w6l4", + "i51pwYAIjIudoE4SFiCCgeOl1HlickG4oInmN1xvhlPxQYE0SANPDWThNHq+nkZk7zlZc1FqpvbNRh+s", + "zG8HZCXLAn4Ym78ZXZrporw1U3sVSEcHKKO/bHwbkvHPjOCyhuSSaWxpoGlJ2F2e8YTrbEOYAC6ujWB8", + "OiOKDXWuKyS4RWv0mMscxK1gN6zwKBnWSRA3qUVMmVw+ll4yuVw2yMXSBziMUoDqF3rmnN+6tb6D8+hB", + "BJLbe9I0M2LaLO8oTblVdf7rLTd8zMiXkpUsJWVukGzmBT/MFP+FDafC7sIYVGgpMm40WVoJ0AbunnU6", + "rPRuhpie4c48dpl2Px2DtDhD8XWZaSqYLFVm6PenFRNI4jBhPD81tlEag57MmOGigiVMaGevoX3AVcUp", + "e2cXH4gRomZi+7sgg7wXhjwXCyNxb5ixDgSpZIGBLqQY/MIK2UDcYR/icImz9Xw3pFmM7HFB3r7ct/4+", + "zNcuCnHZjSOa54W0aOpFEfJVwDsPY8W7W4LQ9IYrM0McdGCNZjvvqSjhKBftFWUMFtwWQ40KWNdY1Zqt", + "c1nQghtk3yWMpbiQG5qVhmh/ksVnQ/5SLBVPGWkRoPXjBRssC8oFnifpQmZNch7/8F3fxlRs8lhyDo+6", + "AArSSY9MCIi32jXLtubbosyymAh2W8E1u2bI7fn4kFyidU4+iMr8AVV8wXSxGRyBNlgxmrKify9xsAfo", + "vG/+03I8PmRk3MDtwbj/dGumdEE1WzoHxYuv88ZBcbk2JgMogiiOjDaJ4mguS5GyNPCBKile35tzVgzq", + "foEdeAPnTAVPmRqStzRXweEX7JteMV60elUKWEgjku0tDs2BC822WRRW4+BJmwzFBBh9p4vgF0Th2O+A", + "1Z/7ExIqULJn/mFgpWxNRRpbLrUqn6cZ22+B/hEU7KSuTC2kSmvGNZUJdjdQojsUXBm/3S1pihsyjUIM", + "4KJAzLR18h5VcGFhuCMvmB3Dta9r65iwGyaaotVjKOdChMoB5goCCNShImt+Z1aI6zUSBVBo5QJH5lJ0", + "zUguW/Lg1+jl0dHpaL5kAzgOGjAxuDkYPo8mngbXvNCrUbKS4vNmsDYsr21buIPJuFiWNBscRBMk164T", + "MpzMLOUdFwUvqWIk5QVLtBGjRm5RLqpDCFXO3VduiNc5nxS8Ia4SQ9jKrXdRyDX6GL9Wg96P0MtnhRoZ", + "+/3E+fz+PLDydGrd/Emw6VU/HOnvVLCCVr0u4K+duiVyKfgvVUf4G8zVB7veFly7jpfsy9NL9oW4X5u9", + "64br/xkNNWJ05NoppskNp+SG56zYHxr2MtyvmI7BpwOfWw+4aJwcg0Z0Mrlpg7bG6TJJDYXuoIHebbcX", + "c56zjAuGJ8x4hEBOaLLyX4Bj1VRQksh8446EoGVMlCQrvlyxAtWvAlG3lgVztsi81Mh78OtUBJNwOsJb", + "+2NnsYSycm/Nxd67cn1s3M9n+02b86BLjVgp0uafM660tykreW7b14RSp4N1tWKKdXgYfL1mKaeaZRuU", + "TY7x8KYuJvRGcmAiOOIZ2KWTjGomEq857IzUSpaZMXZ0siJ6JRUDieaJOOBQsseGy6Fxu7pl0zQy097d", + "MK07ZTBmE+Mf+8RgHIGrw0dJxvPBDddwkTTIzTIOnz7OWbcImmm+ZrLUdT96HPWZWXD6pCW5pVyDFUMd", + "lcFVgDQr0Cy2d39mhdYEM+1N563u7+FY4eHGwRr/D66ukMTOMiaBY3ThLCC0UeHGyigi2zYw056RN1Sz", + "W7ohV/itKQ4Ox50CQB3OkoKlTGhOM/Xo44zDyjMNoDRPmd3xcM+Rso8O6AoBCIIR8BC2ikPwJ8lkD47q", + "ZUH4mi7hkr6KjdgaXsLudD08YXv7UwP+w8VZrc+n+zgCBdd7I8VFXnas7tT87FeoJS4ojD5ZFYw14k/E", + "0oV2uCCU62m0Ylkmya0ssnQaXdfCT4KmagLHbCsMIjG0Z3t8ut4WsVJhfN8AeDBIxcK/j0mtKWyNIQNs", + "H/w5MQ3tv6ZRSjWdwNdRLpb/y7D/d8/i4XA4je7vzUzrciRYurGd4L4d3PPCmDSG5PrDZJq4JHtzmny+", + "pUVKApH9mNiZENu90HaWYL3DBEzQ2KwaI6jfL0an5/7qnTFzrTZ3CsUqxuCCv6FvanKpVw+0kIPRDbMV", + "T1MmZkpTzernQPa2s8FkC6KLksU2OEKRgt4SJhJpZoqwCMIiHxX7MsuYiIkbg//CPoFNAxw8FVwozYyD", + "tCBrRsXAmE8srQSRGpIPii1KiJEiOVXKRtuhMWIGtSEpnTeSuihFQnXdGIPZN++lbUOclxlgwbXFOxDE", + "nSYZE0u96hinIZzdzR8KqS4RbYVb52VztfRo8vHjeDg+eHoYD8bD8bPn38Xj4fj7Fz98is3vTw+fwe/P", + "v/ve/P7ih0/BrW8PEbSugak2VMqlmK2p6goZc9+J+e43NNhDwkXKE6rNvtzQjKc2NmI4FecFU4aTwLLp", + "Ija4vy5Ka2l7vmqycxhE0bBImn+H2Otn9i5KIzfgoCnysZtWd10P8EzzMLa9tIeiNB5eagfbhqu9+Gqu", + "/D127ndc544X/2iqw5moERV+X79KNnZydCcnFwXGLDXCJd3P9TlCaxdB/OCoCKRr1NdccLW6YFQZsK2d", + "h9/J7WpDlkzYKz2itMxzCGBwR3TmlyiOKrFWBSZX93ILnmk4RFmUAg7doUXnOd5r2+IVW3D0+ttzq74Z", + "RqTEQcWL84QKMmcQAI03s96nbYX31cC2rdzgbzPQrYGOgYd2vFSyWlha9IZhC+cH3zIKUTBcQEhSAljs", + "0qLGO+yO7RSBLq8WKmF9tbGXTM/seJ1nCrSga6ZZseXktUup/cfl+3fkEgwSgsRj8TJHvyeYVTBE7Wyt", + "jnSPhu5QVq43cHoHIiImxgcml1SQ1wUVCVeJjMnx0YNk7wfpovzWDwZIoh+2WlxQk3EKxAIu+LFrtXRU", + "X4197lPysOtdM3yDLMeOV5InHThcAO/OCs+826zIGqPfG5MiZXddfk/K7pDSuCIJjOzOTTOudGdMciaX", + "eSHnHarjTC6J+UTnPDN7GkSGG8faB7iwNCY2JEWUGWQ7mP/TuaEdJMj2OWqV97Ft4Q6LLk2kFQgNiKjA", + "xQ28bt+Zzuj4S10wujacEUhNG11+eXkCR9yaGGdqvyWNEOV1zbfL8jBmEimlKziiYFSztNsU4mmnx+PV", + "Yzu6FzERXNIkK6qH9vyFSzHEqMxPO0WhW2jVJIOQN4uNBzehj0dSlmm6Kw5fQeP7+HfirCaiewgPp7ht", + "ha/cIpoaMNPUO5kQ5e0Ir01Wffk1V8bE9kBwLr3MFzh8X5MetS1NCYdW9WVUGVPqsVlJfi9bqUm9aN4l", + "kaytqyyLs9Tnczlk7hksouFbocCYuznaxPv/LYiu54OhX4q5YHloqtdYAGx0D/xPzBBzW9N7fLemd12B", + "90+ff9d7hIzx9lr6nQvtFOjYcW+NW6s6k15uWKFQxrtmLjMruF14dGZWkNm4u+sSnvnY5cm+Qx//vXHq", + "s4S4xtGSrdd0cDg4mA+47jyeBi59xClPTgvNIRpE2TgQ4HgXqgS0RxXxGlJ1H8SwdW7mXRb1s5iD5riX", + "ZkEgRKoekOxAfoSwyWJtSFzzJHZXXD/izVZBRSrXjXSI7+NdHFDgEa+KHuSNSl1L2cW07joLPpM9Z2Or", + "/YB3jadj2NJdA3mLyrC4vcmCkJAqSxUX4hOfXIzczjzd6VvLfPb5gTyfK5kPPhPl9mXPh53faSYUlyJ2", + "kRU2sfHo/LQ7q8+Mlm/f/ndlkrFSVcMFdmi0U25L10mcFwTbpVV1HlefVFssJOCtesPJZgIrts0w7CaS", + "AIhtS/asC5ptnHV9sLMAangf223KRmaB4HdwD6Y0XedkD5NK1L7VhcbBq+Z6SxWpDL/q3vf78bODp4fP", + "uuVxV1bnESkF/1IywuGya8FtagCmdtawXJN35luyzrMBnScHTw+7RN1uB0WVsd+ZEedt5rYNYV1q08f7", + "QdOmTT2NghOXxrfOU5TyMe7Rh27naEf73A22jSs+9BlXbhE1Rd4d4mCVN29SUdQdJSDXuX40VOzWI3Y0", + "zXoBXpmvDhhmCSEs8m/BVLsEWgPr9YnHHShqTKUL7e4+tMPHN18IpP8U4IWSDxenLXnTnzNUdSJ7vTeC", + "9Yyn7mb8Hy/fX9yO//PNUu6Scdd3Td1189uz6NBZCq6ot1lrjetidzX6EF959HvBWTnLFZAHHWT4Gnf2", + "qBCAUYmh1mmqDozc6pCZPmo0aQZ0WTHks6inmLY2jfbrl7wumW3n0LhHxYn4m8ltc2fNELbd7je3BLp0", + "f5jwF4+c/Z0uKNwFbZv+mzP+7uSC+MZ+C35hhRyolYRgEh//BsZcUiot1ySjc4YOWSPHxtDyuhRcb0bL", + "jAtWzHAlN0+HBzbhptUEN0pT9flxq6wciW2r9K1GZ2dv+zaqxyf52OeUmHUUmJQ2emO/vX/37ufHBiO5", + "aMMZFwu5LSh5qxvuwQBDnhpQraDkV0xTnkEMa3UUSuc2Ysj2rwJWXRhTQnO0XTlTw6l4S/N6mBukd2CO", + "g1hIuOq24UjSOzsMLy9CUFPBaLIKR/YZh3tIWzExVGiIMCYFy2C+KiZUqFtWqP1WCG2qMr4eAfdDdNi7", + "kwtMEq4GhaN5AB59un+YFvt7x5GbW2Q20U4uiiM7u37wITfsBP9T58lREKa6jfTbG9uk+zDetU74XfiM", + "w18zWizdzw+u9ZFsYWN3t6+tHtLbI2AdrMHN08H68LHzsBG92+bRF+jbxrQLD67PNFt/WY4WGRUD/dxK", + "ffWlpOngyzKK619hH/zHRyylK8Eb78kqVIcqryZa69QWYqXLJPCy6ERomwvZKsOzrU6NXCwUc0VpGMDo", + "qrcT5Ht0XgwZPuq4vEZ4pj3ZOz+5iMn7izcxOXt/HJO3p5fHddvxHEi7ffyUyIL1FKHBkhfQAk98tCTt", + "Chc//LDLmQ5E+D4WVXO25KLGC+PHFfOxcGyBngoZ/yFXglyueRg9tK1AhdsDtw6o7xM57G2lnN4zVysd", + "2yhBm8ROHVthbicYNmTPGjqWM6XINvsu6amEaCxKwiYx2cgSzrcwdX9DqNh4zGxyFkKPpwISRw2Yym4y", + "sIy9ZNQrqseC6QJTKoZTYYP5bYEgmz4I7Z4oH6BuFwJ5caqVT/oxylmhwAMF90Fs0GdIS3CZU6rZ46Td", + "Aye7TV2ysyrp0SQdoXyoR2d923xhG7S2gOwtKuMVrNQnXik/qcyOjY9YBx/Znwm5DBdrf7iuDjpcGTvy", + "cMZBTC5OXp6c7W/dSwcBkoQqI4ZwDBPBFPrWti5snlkc3cris5qBdICIApbOuHjcphpeVN01bGoYBOLm", + "DNVWfT4V55NbzIDUtgSB8ZyxfsGpSIaEw0nGcQkcK+TwN2god8yD039AWvR5n25NW8LoLE/DJBTZk4IF", + "UYFm8P3O0LCdjHGrAL8+HMxx3rp+2vcA+2xbriftLQt2PFK7lHNSEqeyhcW2cspX4BKHeBiJPfTjSWAr", + "CRXHfv4dyqVaGwR40bYkdKIDovFRek2ItTVCIfLu5GKfDCqpAE1ZSlROhcIyBTWjxIBzjsCkygCgxZzr", + "ghZe1dEw7wrrpNQ0HiSr+e2akIsOCeeqnXlB0LuRBpj1cybkxEK4gdRhZWvcmY9wvtWEsY/Czh5H7exN", + "dZB7l8/bEajQ6/O2t7F98VFzznpvP4JW6Pn20kfD8G/7evEjRUxIuDtFHHiGagnLFcO0tkeKth5D21iS", + "fidXPCf2ODEoReW13Fcb2O6OxUu1LRb3851uUSnPHo2DhuQBPFpQgQ28xeo1Dlh/mMFDRlnNCe6wyVwW", + "bHcMccM77grchCuBDvo/L0x3kbKCpUGpO2dYIOQhgajKjENuqCs3rNxJtxdAYjkVVKQEAfKweN6CsyxV", + "I83WeQZh23O2MLuf0CzzuW9MpLnkom1MnQqNJrGNW23mAuHx/yuo/2F/IotSpNSMTTMoyPcog+tLyYpN", + "V+VXWiQrAl+x9jHL2A11xNwM8m5Ok9A8z3jiJeMusd5uLtUebqO/PiPqcbaJp8devu626wNUsMAswYnH", + "mDhvy5opNFZqhtljI/R78GUn2I0mOPTYns2n+tL5Aq4AMCNdUKHMrOvUemV+NlRu2r5hxZqKCemoJv27", + "OnbdB1g7nl89cHy1G51a5G3F++9FnwZcD332+Eo4BW2cRdjGPerMabSiY1KzomO0doxXZH6bM7re35YZ", + "tMV3+40ekg3g69DMMmOYu+AiohTIXUwUSILAsjDVYqM0W8OFO4T4+7rtNu2i00pr5tJ2F1PtvBxtmWNb", + "qrE202U7d7dxF9ooo7r9GrS35ipEJXVEhED8qI+b2kfXoR071Vqk6/Fg7G87USVYY3vL4cjTBnSF0TkQ", + "fzn1w04jo6V9JNd+sP9+ZrvirKNDHW8QodmHOwjNbQWMfjW+3HgOb32xPB96InlsHGo77wXi2voDeHbc", + "EVjttm35up3gnhN23I6egO9TkRQMLaJgvhCMsT36+2u3xwdR77ZHGEYMpVAKZYPJq42DePfOdywezkUJ", + "386wYTooj+PezM4dN7tjyrtvcu8Gvg4Q3p6Gz5XaJT2NFkuwwDtUImRl2fR4iybf2mU2+wWE3PLr1OdG", + "YWJ/K7dqGt3/DhlquL7dE9S6MqPiAAOfdkD5DrxTm98u/LNlD0K4FfKDndkdjW4FiMs+ytyVCn0Ucst9", + "14XMFBiylSbEEsVSZmpI3ud4QDUVAzKNaKmlIRIMdkxZwlOmyG2VBAdIpFalhaV3p5GQglV9IdBbSG17", + "iA0OiE3dvlfN16VyTbUtWGgsPNPHdMHlTmyJXerKOye1/EifAtp4lMWxt1lcFMM8o4D2Pj1cVmK3YbdK", + "4vqXr07+3IGH+vMff/1DDItP93H0D1YoLkW/ywDFqqEAUEcdNCxk7UKIaxLk6fjps8H4YHDw/OpgPDkc", + "T8bj/93FZEuuZ4lcr7sK07/hEA685pqsqFrV4KMV8awTpJzd4LI6QEpiI5GIa1PPqjgYPn0+HHeB7YXp", + "AuS7AN4cDMdd4BqbVXUN8BGHyK8tq4tSmpWFXKhgVV/or0eC/nokqJ9eegoldyULL1sP8oBm9tXWsNyp", + "atFLxm5Y9lsLOJ8BENilzcNJdw9BuwQgnfp5t4nUSqJBXGLcg7AbVswNyWwI4qEyX1M2L5dwtAPdbyk+", + "QeTKQVTSxDZoiabdVlmbKgYb0qx3urLUealtiDUBZA/JE9cN3ytKZCYL/gvW7lUyYzF58k8lxRNnqZWJ", + "LguWgqUVkyeZXC7WGr/iI0dsseAJHHh/ZpsfoXohySkvVEyeCClz9y5SxoQeBigLpv9PjOtA2GAmyLyO", + "tqDxg6jrKcjWfrAjSZhSs89s0/mc4NFPlwSbmIWR01dBUbLPbKO0LBhRG6HpHa6QJQXTJJPyc5kPyWtI", + "AXUV/o9+upwdHR+fXF7O/vPkv2anrwgTN7yQAs78b2jBIbiO+6KX9eefNrIsBjiZwWe2GfDOIx93K9Ah", + "Yw/DmHvXztU8fKIOh3RNf5GC3qphItdPiCzMVrtnR34Yj8e4jW+5OH1fDxtrdoYQdXGGJUqCspLBwThg", + "albhvxv5FqHVHvzWDbg8Ob44uQr24Ss2AQcJ9qLz5J8p5ZNIupLTbBV5XCW0RWZCtrJ1uTckqEr4qLV3", + "TRtGGeCMOnOY2Eyp7MGqWyf4aMDl5dno6uwSxr48NLJD4PMYyrsmE8hGhRZHP13GtsoS1lBPaFaRUkfZ", + "jgcl+Y5vSLR5Hov5zwxZq653lLhmmb3ptm2JaQv1REen55g3n3HxmaTyFmopK6hVy9a53sQQ+ASw8e0I", + "C8GYRSzXJC/4DdWMGDh8QeaZTD7P7I8znvsSTfvD+j2J/aflriQVw/ovBz88HY6Hj447dsjIqV7tigzT", + "luQFW/A75grAZ2wyGmEZmUPzrw8XZy2kwBghUobGofSdS8UInSuZlZrZtlY4jT4oVqhRSjUd7WMnHMV0", + "mZfJZ6ZHOB/XY70Z2N/LHDZo1MRnCNOIq1aHx+GxtY8PctFL06NWwb4iDVJQAantB0+/N57HcDx6EZOD", + "cfDv758OD76Dvw6exsTs/sF3L/Dv72Jy8N0Pw6fPn9m/9zuTux3xurqwM5s6Wpv5YTux2NZcNfvORcpv", + "eFrSzLMCkblNzITQOAczfKBhDNqBr40NcND3KICfneK/sNl80yxyeDB+9uL599+Ne18JMP0M1TpA9mkC", + "rNhMEGAtqNrD85MbP+BrcKG/e+YmjGljKV9jenW9RML42Yu+eUI/cstTvRqtGF+uYH45v4O0JKxf6fP2", + "C2aWVX/iDIFvw2hbmt7fWzvVVtygmC2L5yLREUjaKMa8RHiCTE1GoyXXq3Ju5I01yNP5yJbWbufeODcC", + "H6uwBaAh3gtFfxW9Ba/UFf6pRPti5tuz6oGNqfjmG/LTikI8hQUcPgdm41CV0ypnAXR8xcnPIDCBjs5P", + "4djt22+rsuxvfFrxt99OCFy0wVMZVZ3RveOz0/P9VhocAoIOrly7gXDJ1lRonlQpfzCf8ElI9w7mAAjW", + "vWaH8HwNdwOrihko2MBGUFjFD4EW+PCfnck7umapC8gLKrobQDaAjWAwtYqJLJZU8F9cvpE7rnYvv7I7", + "HSzvwt0nA4bcpT66zBgz6W7XXWqdj5KrkrZjGxYS/sJ0MsQq8U6UQCEsqhk8n5IxahSEJo4ckQaHXI5S", + "maiR18WeXqAWDGRnddBMQgUpSqiuKFKaScHgadbq9QQqCPIBSbJSaVYAsZwBhVX716A0IxjZnWYFmHbn", + "p/b9FSYSSAAbdJDh9YjmHLMmryuzvHYvDD09KVXPMToCuDh6U9WeN21DUnHRUiCv14ZVWOpI5ktJM643", + "pssxE7qgGbiBNoSPFTcQh5MUUimScqP95qVmKREyxYHOjcpKNgOInMXmNe6DnGX77ETG6A1TxNiipkVB", + "vWe5b7fsNYPCIXYHvyFdfImUiFm4hgJDVqq9zOAevux9jwEhHZ2fApjd9sWxJZ6zEnzZEObxkgtjrvtD", + "+Ri8ZbeSt5UAsTa8FSQIEBOm/XIBoPlMgkd2/gl0YTZ/gCqjEkEqpwmzkCD4JpwXpH/7LHJF9q5788iv", + "9w0PGNMNgdlU7WOPFAPwg2KqUW3Zni7sXX9VoWtb0vra4sLw6zFV8GYfQcQgtcYECHGAaCyYLji7oVlM", + "brgyFojia57RAugZsV4Tx03CeV0JXc9Mraca98m/hRQWwCCvLJ1tDLCj3jdBeh/2QFjH+DC2gfF0gE/B", + "kaurM/fiAzxDii1fW80Ac6/5tc3HMFwI0ILyzPOSVyW7riFkrI6FBI+NIMS/l0Y8/eKV51H90TxDNV+w", + "SfX+FV9UqA7o13SvxVqCyLMhfWTPxlauqEgz44BwlqVBXDcGXGc8YfYqxFk1WUYujH2lyAXDp3FbJk6l", + "UzK2pFh8h2t86bh6Sj0KrhGimwOa5St6AHVH0BONJtHhcDw8jOLI+1W48+CNStV1PJNn3LgBZqUdryRb", + "1aqcEqhbDY2CAs5kemuJFikACL56TKZN61hYv/V2ObzcrKunXvHZcNP4g3u/40dfsMD8/JoqNJtqlZ/c", + "NICu3np2altI/h0nJ2RCxh6Qt9tkfRC8V+e0R3MMsZxp/rpAywpcV+DI8Mk8I5ufulcch9CNwWmhr+Yw", + "l3rlntk3aNFB/JbbKvtQqfnr+vr6n8ooul+ngtReFOh5bBsEaIyNcRgUsYIQ8xMsDQHYTYrdp9pr6qbJ", + "8/HYf6w/K49f/Uf/kDsCnk6F+S8yn++n4h5WgffAzh08Td07z1d4yWFd35cy3TQK/wWRv6N/2qqQeBWw", + "07vTLmL1vn7FYpxw+AGVN3Dl0/H49x7bXsHC4F1xe0CWKVElHB8uyiyDA4Rnv+NMsLx1xwxOBdZ6LxyG", + "4uj5nzOuNYmtr8dswzhS5XpNi40jjQ75p9gSw0FMc7TH+qWoq3+kbF34wIa2BxbhUyMoU7OGSa+G9pkz", + "b4xzFbxcYT04sBefqLqZaA0hPMG90zFG+EPJtRRyILFvqwJO4F/aQESAEZiAbZHeeJ92qywMXyGr3mWy", + "7/MYLwtXgT0CL0RLKEQcvPcczMZ5hAMInllbw8qdwHVXddmfkKMEzkHDF0KsSVIhoQkMTOV6V4NYezhy", + "w2ll31o81exgfLgueEMGsOZDPv/QF2W6Hr/qf2Vmp4dl0GD+Y56V6ajPtP8bFWLwUhrcZCTh27kU3nxP", + "SxQ5LJ0Ka+jAdiwyOA/DV8Vu8J2ztBQpFRoScx1XNB0y87O/K/Gx6T5kGnKpDdVYgsJHO+2b54H0k4lm", + "eoChatfexVOs4MZRtglY1uGLyVoqTfxN6H4LmpGl1xMMV3NKCAVCUFoG5EXg94fGAdKxobqwnFaLuiZt", + "EyIwAfqeNrP2Az7wEk3IxwbtG6JqpuhMo0+Vrp+Kt51PUrXpafsEt7171jVNMEge5BlK8pXUElMAEqoN", + "A3V0/W1cZGusWWYy4D9tsYWcrqrc/j/IKKo9yvUnG0X1N3PM2H0MVofZuCsBzhs4zut6ACYsMzqH5h0R", + "Ny3DpMJ9UFv6X8U0ezZ+9sePa99KlMbiKEX6L2USOg4JJCJagb6a9A6GYHAwjZZeUBLNmhdQzYwc7g+n", + "4rXZ9Ja1Z0wkWyb3eEU1OfZFIpV9x+o3GGdBHbbf0Tr75hvin0YAdcg0uUZOw1v5a0zOw4cTL2ETBpdG", + "PZ5APWh4PWEfl8+ruOupwCAFrCdtXzdF1Wwf0DXu6bXBUoUkMO2vbRQxvvxpA8AxZBAGwWNJ8vHV+3cn", + "n66HHXac+QlOH3sq+1rrG1Wqqw4ON7TbNU5XvTuvaRygurIpZGY1BuaIobqwHIMf/kuWuP9kxbJ8UWbE", + "p48Na9qnglUqVnRAgmzI/28aoUZx8/LF2E2jp8+/s78jXs1vEGwR6ufKkiB776QY+Bj8fXvsMKdqNTXW", + "WkYGP5Pz95dXcNkyGY2qOKAX4xdjIEfHg2Q6NSMP/kam7hHLwdUmZxPSlBrTyLVNyZOu44ktG9Hail0Q", + "564Mm3bLv0+j+08V2BomD8ZjOMR40om2y//XUeZprcJPRVFGbNSQ02vTHPug1z/CoGm+VPAn2zSt0uP3", + "Ngx4BIKw057Z+VmZLl153FO3fEgu7GOO4FQYx0uEfO3drHgqZNEl5OsvgHivbC9UFFDG5i8z6E81g8zo", + "h3/86G+Ch+Yw3JuUojIn9oR0llL1fsF+n5EWGFrGxOpw/NB4s7UrJ79GS6a7cs+RpI1po8Bfa9cejv3Z", + "fVzdcMdV2duYvDu5iAmW16jVqPTn7e5EDE+y2pdd3adnpu3f/S0WhIsaNlrRG0auB/zFNVHlYsHv3JGJ", + "vYPAQY76qjyTPfewPZwanGelgpStrbMK7zfsIYi9kdthSY3bu5N1rjcY9NrecAv+jTdTEX6zsnCfSbsr", + "/KrCz24LqCrvwRCnDqv1wodGwHWWdvaj2hIPOGYjdKVjWFcRAga9eh6T12dH7wbmHy+PLq6Q5s7e/v0N", + "6Yh68bfINQch8A2M9W/50L2db+DhEduwQ9+ecaXfulqwf5jCa9Q87xOfyl3K/3f50C/pv+bVxlmXl4bC", + "0NNxvyvrGUPBEWEa1Ct7MILLRlVVgVxWQANvhYXOG9IR48aMTYpnqRAkVaTkWL47O6uVBbUhdOcnF3Br", + "DxOCeXrvehqUU5xGNsrj/cUbCLsJ5111wFqLaJ5W1RZ957P3x6bzmY9T8x3fsVvyX7L4jF0hy5r5bm9P", + "L6HfW64SlmVUMFkqj82WD19dNe/ixbcEkj9//tLWF3vQccZfDKUQdzC948aRw99KSOB5TRNGXp5cXNnL", + "+Paemc51meeukJKuGrE3nOKZAv59XT2k+fBNdOB3dBQ49S4z1GexZ8hbKmk2t7dVTLN2vNwSffWqZJz9", + "UaenrRq9f7K30a762XWO6USCDYP56xTzf4L5HsT1fr397umroV2cfjKG4jblNLAVzPLegC0f0FMVgNOS", + "UBsCXFNI9fr2Th0dY8DXhauk5x9UwKNEGw+2LpWeTMXB0Ac42/FcQT0U2i5qRk3FU+NAQxUoKJDgyu1N", + "xeGQXDKRdqzJ3vcSqsi1Xd81zoMrkjLFl4J8ZixXVR6tppplTCkw5SDzRdkpQ9YHSmmD/6oSYCaXPBn+", + "Zo1Us/E79JE7BcdeM/8BtBLGY9aj+PKCLVjRqdLqoXyP0CQ9hRm9NgEqwaZbSwT6DnZbggPbaasm4lsL", + "6cxCmhDYwGXJU0YwALKKqjIAoFyia01eB+USJ+QdKwuaEcG0VXEFg86NC8CpeC0LT35enzuSB63fjGWM", + "IcVJeyIfKJ6yqbjO+HzkuxodnnyGLJIVT1Y+MLKip4frRzZ1qwF9bmso/lF6NawC+qcr1VoJyA7Zahdv", + "N+gvffo/Q586ZvwN2tSAqFTfplJ6TqHCCcI2jdqol7kttabjAAIcvt+uO2rnHI1h4biLiUSmrABdEZOU", + "wV8DLri2P2EEcNUIEj1Vl6+zw3nJN98Qnxjggs/+7nrUs7b8gaSDaBU/ltYeAAkZ64TyQnlAxaYPiqrK", + "2LrHAkIbApxfWtB8VVDlM7UAey6YkaR8AQkxELSUulwxW/jU5ZpVdVBNL1ebPKNiWcINTkunGm3ShbQK", + "pYi+7ZefW4qZ1qNtnEe3bKJ3QqbleHyYrDL4PyMv2UYaI6/nZ6VpAYddipxoTcl/wGGB8fxoyrOMJnAU", + "WKTqQQcQsPwHRhM3KuD+6TqqXge2L6K48MVa/1JRf93Y7KSjAvHUpVdQUwWVsrZe1DQKZ8Vk6Qt+xWTu", + "i4uhQmhX7hp2XCLrf/hKWn8YezVrpnUg2zYJy2f995xlNy7ctMNdfWbQDIhSQeG97oTso/NTS7o++2lE", + "cx7df7r/vwEAAP//LcZSyv23AAA=", } // GetSwagger returns the content of the embedded swagger specification file diff --git a/pkg/termite/api.go b/pkg/termite/api.go index d455ce5..80eef40 100644 --- a/pkg/termite/api.go +++ b/pkg/termite/api.go @@ -29,9 +29,10 @@ import ( "time" "github.com/antflydb/antfly-go/libaf/ai" - "github.com/antflydb/antfly-go/libaf/embeddings" + libafembed "github.com/antflydb/antfly-go/libaf/embeddings" "github.com/antflydb/antfly-go/libaf/s3" "github.com/antflydb/antfly-go/libaf/scraping" + "github.com/antflydb/termite/pkg/termite/lib/embeddings" "github.com/antflydb/termite/pkg/termite/lib/generation" "github.com/antflydb/termite/pkg/termite/lib/ner" "github.com/bytedance/sonic/decoder" @@ -237,6 +238,49 @@ func (ln *TermiteNode) handleApiEmbed(w http.ResponseWriter, r *http.Request) { return } + // Check if hidden states are requested + if req.ReturnHiddenStates { + // Check if embedder supports hidden states extraction + hsEmbedder, ok := embedder.(embeddings.HiddenStatesEmbedder) + if !ok { + http.Error(w, fmt.Sprintf("model %s does not support hidden states extraction", req.Model), http.StatusBadRequest) + return + } + + // Generate hidden states (skip caching - different output format) + output, err := hsEmbedder.EmbedWithHiddenStates(r.Context(), contents) + if err != nil { + ln.logger.Error("failed to extract hidden states", + zap.String("model", req.Model), + zap.Error(err)) + http.Error(w, fmt.Sprintf("extracting hidden states: %v", err), http.StatusInternalServerError) + return + } + + // Convert int64 attention mask to int for JSON serialization + attentionMask := make([][]int, len(output.AttentionMask)) + for i, mask := range output.AttentionMask { + attentionMask[i] = make([]int, len(mask)) + for j, v := range mask { + attentionMask[i][j] = int(v) + } + } + + // Return hidden states response (always JSON for now) + resp := EmbedResponse{ + Model: req.Model, + HiddenStates: output.HiddenStates, + AttentionMask: attentionMask, + } + w.Header().Set("Content-Type", "application/json") + if err := encoder.NewStreamEncoder(w).Encode(resp); err != nil { + ln.logger.Error("encoding hidden states response", zap.Error(err)) + http.Error(w, err.Error(), http.StatusInternalServerError) + return + } + return + } + // Wrap embedder with caching for deduplicated requests cachedEmbedder := ln.embeddingCache.WrapEmbedder(embedder, req.Model) @@ -343,7 +387,7 @@ func parseEmbedInput( // validateContentTypes checks that all content types in the input are supported // by the embedder's capabilities. -func validateContentTypes(contents [][]ai.ContentPart, caps embeddings.EmbedderCapabilities) error { +func validateContentTypes(contents [][]ai.ContentPart, caps libafembed.EmbedderCapabilities) error { // Build set of supported MIME types supported := make(map[string]bool) for _, m := range caps.SupportedMIMETypes { @@ -372,7 +416,7 @@ func validateContentTypes(contents [][]ai.ContentPart, caps embeddings.EmbedderC } // getMIMETypeList returns a list of supported MIME types for error messages. -func getMIMETypeList(caps embeddings.EmbedderCapabilities) []string { +func getMIMETypeList(caps libafembed.EmbedderCapabilities) []string { types := make([]string, len(caps.SupportedMIMETypes)) for i, m := range caps.SupportedMIMETypes { types[i] = m.MIMEType @@ -1310,6 +1354,9 @@ func (ln *TermiteNode) handleApiRewrite(w http.ResponseWriter, r *http.Request) // Get model from registry model, err := ln.seq2seqRegistry.Get(req.Model) if err != nil { + ln.logger.Error("failed to get rewrite model", + zap.String("model", req.Model), + zap.Error(err)) http.Error(w, fmt.Sprintf("model not found: %s", req.Model), http.StatusNotFound) return } @@ -1343,3 +1390,4 @@ func (ln *TermiteNode) handleApiRewrite(w http.ResponseWriter, r *http.Request) return } } + diff --git a/pkg/termite/embedder_registry.go b/pkg/termite/embedder_registry.go index f461a35..397f142 100644 --- a/pkg/termite/embedder_registry.go +++ b/pkg/termite/embedder_registry.go @@ -200,6 +200,23 @@ func (r *EmbedderRegistry) discoverModels() error { registryFullName := dm.FullName() variants := dm.Variants + // Check if this is a T5Gemma-2 multimodal encoder-decoder model + if isT5Gemma2Model(modelPath) { + r.logger.Info("Discovered T5Gemma-2 embedder model (not loaded)", + zap.String("name", registryFullName), + zap.String("path", modelPath)) + + r.discovered[registryFullName] = &ModelInfo{ + Name: registryFullName, + Path: modelPath, + OnnxFilename: "", // T5Gemma-2 uses multiple files (encoder.onnx, vision_encoder.onnx) + PoolSize: poolSize, + ModelType: "t5gemma2", + Variants: []string{"default"}, + } + continue // Skip standard embedder handling + } + // Check if this is a multimodal (CLIP-style) model hasMultimodalStd, hasMultimodalQt := isMultimodalModel(modelPath) if hasMultimodalStd || hasMultimodalQt { @@ -336,6 +353,14 @@ func (r *EmbedderRegistry) loadModel(info *ModelInfo) (embeddings.Embedder, erro // Handle different model types switch info.ModelType { + case "t5gemma2": + // Load T5Gemma-2 multimodal encoder-decoder model + embedder, backendUsed, err = termembeddings.NewT5Gemma2EmbedderWithSessionManager( + info.Path, + r.sessionManager, + nil, // modelBackends - use default priority (ONNX only) + r.logger.Named(info.Name), + ) case "clip": // Load standard precision CLIP multimodal model embedder, backendUsed, err = termembeddings.NewHugotCLIPEmbedderWithSessionManager( diff --git a/pkg/termite/go.mod b/pkg/termite/go.mod index 62b31a8..3080c0f 100644 --- a/pkg/termite/go.mod +++ b/pkg/termite/go.mod @@ -6,7 +6,7 @@ replace github.com/gomlx/gomlx => github.com/timkaye11/gomlx v0.0.0-202512210449 replace github.com/gomlx/onnx-gomlx => github.com/timkaye11/onnx-gomlx v0.0.0-20251217194823-a871bb8cc687 -replace github.com/knights-analytics/hugot => github.com/ajroetker/hugot v0.0.0-20260107061453-f83ae7b7c356 +replace github.com/knights-analytics/hugot => github.com/timkaye11/hugot v0.0.0-20260108205439-6127b8a4c499 replace github.com/knights-analytics/ortgenai => github.com/ajroetker/ortgenai v0.0.0-20260102002309-90abcfc27670 diff --git a/pkg/termite/go.sum b/pkg/termite/go.sum index 49d7c9c..740421e 100644 --- a/pkg/termite/go.sum +++ b/pkg/termite/go.sum @@ -1,6 +1,4 @@ github.com/RaveNoX/go-jsoncommentstrip v1.0.0/go.mod h1:78ihd09MekBnJnxpICcwzCMzGrKSKYe4AqU6PDYYpjk= -github.com/ajroetker/hugot v0.0.0-20260107061453-f83ae7b7c356 h1:opUqf468UDMKDFEmmQZybIi5s7egT0RZGZX+WeylE8A= -github.com/ajroetker/hugot v0.0.0-20260107061453-f83ae7b7c356/go.mod h1:I3/sobJjSQecs1xBUKnbjBKcbaX03uQ7gfkDW55zAU8= github.com/ajroetker/ortgenai v0.0.0-20260102002309-90abcfc27670 h1:bT8mAdiBPSwQz2O4aPbbfZgj8JQZiqTnBZ+FPqN/TTI= github.com/ajroetker/ortgenai v0.0.0-20260102002309-90abcfc27670/go.mod h1:zolhST8PioaCCsnwHBUesIBKU4CW/ZrF169YjFfg7BY= github.com/antflydb/antfly-go/libaf v0.0.0-20260105234922-f6ed1eb6788d h1:7C9y7i5pb6frXFtsXszDLVhevEQ8/A/ySDwLHrq4dog= @@ -229,6 +227,8 @@ github.com/sugarme/tokenizer v0.3.0 h1:FE8DYbNSz/kSbgEo9l/RjgYHkIJYEdskumitFQBE9 github.com/sugarme/tokenizer v0.3.0/go.mod h1:VJ+DLK5ZEZwzvODOWwY0cw+B1dabTd3nCB5HuFCItCc= github.com/timkaye11/gomlx v0.0.0-20251221044952-ee84bc4bbaf7 h1:9WfUFJSnLpAlSEWUUtucZrDSDZmE60H0xQPHohksQu8= github.com/timkaye11/gomlx v0.0.0-20251221044952-ee84bc4bbaf7/go.mod h1:HKSGc4Btb3VntH/BV/UsJYzgTMbLT/Gk69kv95rjruM= +github.com/timkaye11/hugot v0.0.0-20260108205439-6127b8a4c499 h1:f5+W2JKwNwLLkYzZ+A+Wcv9shdFSffo1L8zasfaIUAw= +github.com/timkaye11/hugot v0.0.0-20260108205439-6127b8a4c499/go.mod h1:F8q/Zn0G8jUFUCp6KAuUvQRRsHZk3VEb0Efo0c6z/N8= github.com/timkaye11/onnx-gomlx v0.0.0-20251217194823-a871bb8cc687 h1:mPN2mhB+q1DwpMnJCdmG79oFtohAgomHfw45pdxurys= github.com/timkaye11/onnx-gomlx v0.0.0-20251217194823-a871bb8cc687/go.mod h1:V0xRbk2eozhH6FCx3RmyjNcfi7AujLP+mmaOHOpr65s= github.com/tinylib/msgp v1.6.3 h1:bCSxiTz386UTgyT1i0MSCvdbWjVW+8sG3PjkGsZQt4s= diff --git a/pkg/termite/lib/embeddings/hidden_states.go b/pkg/termite/lib/embeddings/hidden_states.go new file mode 100644 index 0000000..c64464f --- /dev/null +++ b/pkg/termite/lib/embeddings/hidden_states.go @@ -0,0 +1,50 @@ +// Copyright 2025 Antfly, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package embeddings + +import ( + "context" + + "github.com/antflydb/antfly-go/libaf/ai" + "github.com/antflydb/antfly-go/libaf/embeddings" +) + +// HiddenStatesOutput contains raw encoder hidden states (before pooling). +type HiddenStatesOutput struct { + // HiddenStates are raw encoder outputs [batch_size][seq_len][hidden_size]. + // Each input produces a sequence of hidden state vectors, one per token. + HiddenStates [][][]float32 + + // AttentionMask indicates valid token positions [batch_size][seq_len]. + // 1 = valid token, 0 = padding token. + AttentionMask [][]int64 +} + +// HiddenStatesEmbedder extends Embedder with raw hidden state extraction. +// This is useful for: +// - Passing hidden states to the decode API for text generation +// - Custom pooling strategies (e.g., first token, last token, weighted) +// - Token-level similarity comparisons +type HiddenStatesEmbedder interface { + embeddings.Embedder + + // EmbedWithHiddenStates returns raw encoder hidden states before pooling. + // Unlike Embed() which returns mean-pooled vectors, this returns the full + // sequence of token-level hidden states. + // + // The returned hidden states can be passed to the decode API's embeddings + // field for text generation from custom embeddings. + EmbedWithHiddenStates(ctx context.Context, contents [][]ai.ContentPart) (*HiddenStatesOutput, error) +} diff --git a/pkg/termite/lib/embeddings/t5gemma2.go b/pkg/termite/lib/embeddings/t5gemma2.go new file mode 100644 index 0000000..df62c00 --- /dev/null +++ b/pkg/termite/lib/embeddings/t5gemma2.go @@ -0,0 +1,606 @@ +// Copyright 2025 Antfly, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//go:build onnx && ORT + +package embeddings + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "image" + _ "image/gif" + _ "image/jpeg" + _ "image/png" + "os" + "path/filepath" + "strings" + "sync" + + "github.com/antflydb/antfly-go/libaf/ai" + libafembed "github.com/antflydb/antfly-go/libaf/embeddings" + "github.com/antflydb/termite/pkg/termite/lib/hugot" + khugot "github.com/knights-analytics/hugot" + "github.com/knights-analytics/hugot/backends" + "github.com/knights-analytics/hugot/pipelines" + "github.com/knights-analytics/hugot/util/imageutil" + "go.uber.org/zap" + _ "golang.org/x/image/webp" +) + +// T5Gemma2Embedder implements multimodal embeddings using T5Gemma-2 encoder. +// It uses the encoder.onnx for text embeddings and vision_encoder.onnx for images. +// +// T5Gemma-2 is a multimodal encoder-decoder model from Google's Gemma 3 family, +// trained with UL2 objective. The encoder produces rich embeddings that can be +// used for similarity search and retrieval. +// +// The vision encoder is lazily loaded on first image embedding request to save memory. +// +// Build with: CGO_ENABLED=1 go build -tags="onnx,ORT" +type T5Gemma2Embedder struct { + encoderPipeline *pipelines.FeatureExtractionPipeline + visionPipeline *pipelines.FeatureExtractionPipeline + session *khugot.Session + config *T5Gemma2Config + logger *zap.Logger + caps libafembed.EmbedderCapabilities + modelPath string + sessionShared bool + + // Lazy loading for vision encoder + hasVisionFile bool // true if vision_encoder.onnx exists + visionOnce sync.Once // ensures vision pipeline loaded only once + visionLoadErr error // captures any error during lazy load +} + +// T5Gemma2Config holds the T5Gemma-2 model configuration +type T5Gemma2Config struct { + ModelType string `json:"model_type"` + HiddenSize int `json:"hidden_size"` + VocabSize int `json:"vocab_size"` + ImageSize int `json:"image_size"` + ImageSeqLen int `json:"image_seq_length"` // Tokens per image (typically 256) + Capabilities []string `json:"capabilities"` +} + +// NewT5Gemma2Embedder creates a new T5Gemma-2 embedder using hugot pipelines. +// The directory should contain: +// - encoder.onnx (text + multimodal encoder) +// - vision_encoder.onnx (SigLIP vision encoder) +// - config.json +// - tokenizer.json +// +// Build with -tags="onnx,ORT" to enable this embedder. +func NewT5Gemma2Embedder(modelPath string, logger *zap.Logger) (*T5Gemma2Embedder, error) { + return NewT5Gemma2EmbedderWithSession(modelPath, nil, logger) +} + +// NewT5Gemma2EmbedderWithSessionManager creates a new T5Gemma-2 embedder using a SessionManager. +// The SessionManager handles backend selection and session reuse (required for ONNX Runtime which only allows one session). +// Returns the embedder and the backend type that was used. +func NewT5Gemma2EmbedderWithSessionManager(modelPath string, sessionManager *hugot.SessionManager, modelBackends []string, logger *zap.Logger) (*T5Gemma2Embedder, hugot.BackendType, error) { + if sessionManager == nil { + return nil, "", errors.New("sessionManager is required for T5Gemma-2 embedder (ONNX Runtime only allows one session)") + } + + // T5Gemma-2 requires ONNX Runtime backend (not pure Go or XLA) + if modelBackends == nil { + modelBackends = []string{"onnx"} + } + session, backendUsed, err := sessionManager.GetSessionForModel(modelBackends) + if err != nil { + return nil, "", fmt.Errorf("getting session from manager: %w", err) + } + + embedder, err := NewT5Gemma2EmbedderWithSession(modelPath, session, logger) + if err != nil { + return nil, "", err + } + + // SessionManager owns the session, so mark as shared + embedder.sessionShared = true + + return embedder, backendUsed, nil +} + +// NewT5Gemma2EmbedderWithSession creates a new T5Gemma-2 embedder using an optional shared session. +func NewT5Gemma2EmbedderWithSession(modelPath string, sharedSession *khugot.Session, logger *zap.Logger) (*T5Gemma2Embedder, error) { + if modelPath == "" { + return nil, errors.New("model path is required") + } + + if logger == nil { + logger = zap.NewNop() + } + + logger.Info("Initializing T5Gemma-2 embedder", + zap.String("modelPath", modelPath), + zap.String("backend", hugot.BackendName())) + + // Load configuration + config, err := loadT5Gemma2Config(modelPath) + if err != nil { + return nil, fmt.Errorf("loading T5Gemma-2 config: %w", err) + } + + // Verify required files exist + encoderPath := filepath.Join(modelPath, "encoder.onnx") + if _, err := os.Stat(encoderPath); err != nil { + return nil, fmt.Errorf("encoder model not found: %s", encoderPath) + } + + // Check for optional vision encoder + visionPath := filepath.Join(modelPath, "vision_encoder.onnx") + hasVision := false + if _, err := os.Stat(visionPath); err == nil { + hasVision = true + } + + // Create or reuse session + session, err := hugot.NewSessionOrUseExisting(sharedSession) + if err != nil { + return nil, fmt.Errorf("creating hugot session: %w", err) + } + sessionShared := (sharedSession != nil) + + // Create text encoder pipeline first (always needed) + encoderPipelineName := fmt.Sprintf("%s:encoder:encoder.onnx", modelPath) + encoderConfig := khugot.FeatureExtractionConfig{ + ModelPath: modelPath, + Name: encoderPipelineName, + OnnxFilename: "encoder.onnx", + Options: []backends.PipelineOption[*pipelines.FeatureExtractionPipeline]{ + pipelines.WithNormalization(), + }, + } + + encoderPipeline, err := khugot.NewPipeline(session, encoderConfig) + if err != nil { + if !sessionShared { + _ = session.Destroy() + } + return nil, fmt.Errorf("creating encoder pipeline: %w", err) + } + + // Vision pipeline is loaded lazily on first image embed request to save memory. + // We only check if the file exists here; actual loading happens in loadVisionPipeline(). + if hasVision { + logger.Info("T5Gemma-2 embedder initialized (vision available, will load on first use)", + zap.Int("hiddenSize", config.HiddenSize), + zap.Int("imageSize", config.ImageSize)) + } else { + logger.Info("T5Gemma-2 embedder initialized (text-only, no vision encoder file)", + zap.Int("hiddenSize", config.HiddenSize)) + } + + // Build capabilities based on available features + mimeTypes := []libafembed.MIMETypeSupport{ + {MIMEType: "text/plain"}, + } + if hasVision { + mimeTypes = append(mimeTypes, + libafembed.MIMETypeSupport{MIMEType: "image/png"}, + libafembed.MIMETypeSupport{MIMEType: "image/jpeg"}, + libafembed.MIMETypeSupport{MIMEType: "image/gif"}, + libafembed.MIMETypeSupport{MIMEType: "image/webp"}, + ) + } + + return &T5Gemma2Embedder{ + encoderPipeline: encoderPipeline, + visionPipeline: nil, // Loaded lazily + session: session, + config: config, + logger: logger, + modelPath: modelPath, + sessionShared: sessionShared, + hasVisionFile: hasVision, + caps: libafembed.EmbedderCapabilities{ + SupportedMIMETypes: mimeTypes, + Dimensions: []int{config.HiddenSize}, + DefaultDimension: config.HiddenSize, + SupportsFusion: false, + }, + }, nil +} + +// Capabilities returns the embedder capabilities +func (t *T5Gemma2Embedder) Capabilities() libafembed.EmbedderCapabilities { + return t.caps +} + +// Embed generates embeddings for the given content. +// For text content, uses the text encoder. +// For image content (BinaryContent), uses the vision encoder. +// For mixed content, processes each modality appropriately. +func (t *T5Gemma2Embedder) Embed(ctx context.Context, contents [][]ai.ContentPart) ([][]float32, error) { + if len(contents) == 0 { + return [][]float32{}, nil + } + + embeddings := make([][]float32, len(contents)) + + for i, parts := range contents { + var embedding []float32 + var err error + + for _, part := range parts { + switch p := part.(type) { + case ai.BinaryContent: + if strings.HasPrefix(p.MIMEType, "image/") { + embedding, err = t.embedImage(p.Data) + if err != nil { + return nil, fmt.Errorf("embedding image at index %d: %w", i, err) + } + } + case ai.TextContent: + embedding, err = t.embedText(p.Text) + if err != nil { + return nil, fmt.Errorf("embedding text at index %d: %w", i, err) + } + } + + if embedding != nil { + break + } + } + + if embedding == nil { + return nil, fmt.Errorf("no valid content found at index %d", i) + } + + embeddings[i] = embedding + } + + return embeddings, nil +} + +// loadVisionPipeline lazily loads the vision encoder on first use. +// This saves memory when only text embeddings are needed. +func (t *T5Gemma2Embedder) loadVisionPipeline() error { + t.visionOnce.Do(func() { + if !t.hasVisionFile { + t.visionLoadErr = errors.New("vision encoder not available: vision_encoder.onnx not found") + return + } + + t.logger.Info("Lazily loading vision encoder pipeline", + zap.String("modelPath", t.modelPath)) + + imageSize := t.config.ImageSize + if imageSize == 0 { + imageSize = 896 // Default for T5Gemma-2 (SigLIP) + } + + visionPipelineName := fmt.Sprintf("%s:vision:vision_encoder.onnx", t.modelPath) + visionConfig := khugot.FeatureExtractionConfig{ + ModelPath: t.modelPath, + Name: visionPipelineName, + OnnxFilename: "vision_encoder.onnx", + Options: []backends.PipelineOption[*pipelines.FeatureExtractionPipeline]{ + pipelines.WithImageMode(), + pipelines.WithPreprocessSteps[*pipelines.FeatureExtractionPipeline]( + imageutil.ResizeStep(imageSize), + imageutil.CenterCropStep(imageSize, imageSize), + ), + pipelines.WithNormalizationSteps[*pipelines.FeatureExtractionPipeline]( + imageutil.RescaleStep(), + imageutil.CLIPPixelNormalizationStep(), + ), + pipelines.WithNCHWFormat[*pipelines.FeatureExtractionPipeline](), + pipelines.WithNormalization(), + }, + } + + pipeline, err := khugot.NewPipeline(t.session, visionConfig) + if err != nil { + t.visionLoadErr = fmt.Errorf("loading vision pipeline: %w", err) + t.logger.Error("Failed to load vision encoder", zap.Error(err)) + return + } + + t.visionPipeline = pipeline + t.logger.Info("Vision encoder loaded successfully", + zap.Int("imageSize", imageSize)) + }) + + return t.visionLoadErr +} + +// embedImage processes an image and returns its embedding using the vision pipeline. +// The vision pipeline is lazily loaded on first call to save memory. +func (t *T5Gemma2Embedder) embedImage(imageData []byte) ([]float32, error) { + // Lazy load vision pipeline + if err := t.loadVisionPipeline(); err != nil { + return nil, err + } + + // Decode image + img, _, err := image.Decode(bytes.NewReader(imageData)) + if err != nil { + return nil, fmt.Errorf("decoding image: %w", err) + } + + // Run through vision pipeline + output, err := t.visionPipeline.RunWithImages([]image.Image{img}) + if err != nil { + return nil, fmt.Errorf("running vision pipeline: %w", err) + } + + if len(output.Embeddings) == 0 || len(output.Embeddings[0]) == 0 { + return nil, errors.New("no embedding returned from vision pipeline") + } + + // Pipeline already normalizes if WithNormalization() was used + return output.Embeddings[0], nil +} + +// HasVision returns true if this embedder supports image embeddings. +// Note: This returns true if vision_encoder.onnx exists, even if not yet loaded. +func (t *T5Gemma2Embedder) HasVision() bool { + return t.hasVisionFile +} + +// IsVisionLoaded returns true if the vision pipeline has been loaded. +func (t *T5Gemma2Embedder) IsVisionLoaded() bool { + return t.visionPipeline != nil +} + +// embedText tokenizes text and returns its embedding using the encoder pipeline +func (t *T5Gemma2Embedder) embedText(text string) ([]float32, error) { + // Run through encoder pipeline + output, err := t.encoderPipeline.RunPipeline([]string{text}) + if err != nil { + return nil, fmt.Errorf("running encoder pipeline: %w", err) + } + + if len(output.Embeddings) == 0 || len(output.Embeddings[0]) == 0 { + return nil, errors.New("no embedding returned from encoder pipeline") + } + + // Pipeline already normalizes if WithNormalization() was used + return output.Embeddings[0], nil +} + +// Close releases resources +func (t *T5Gemma2Embedder) Close() error { + if t.session != nil && !t.sessionShared { + t.logger.Info("Destroying Hugot session (owned by this T5Gemma-2 embedder)") + return t.session.Destroy() + } else if t.sessionShared { + t.logger.Debug("Skipping session destruction (shared session)") + } + return nil +} + +// loadT5Gemma2Config loads T5Gemma-2 configuration from model directory +func loadT5Gemma2Config(modelPath string) (*T5Gemma2Config, error) { + // Try t5gemma2_config.json first (our custom config), then config.json + configPaths := []string{ + filepath.Join(modelPath, "t5gemma2_config.json"), + filepath.Join(modelPath, "config.json"), + } + + for _, path := range configPaths { + data, err := os.ReadFile(path) + if err != nil { + continue + } + + // Parse config + var rawConfig struct { + ModelType string `json:"model_type"` + HiddenSize int `json:"hidden_size"` + VocabSize int `json:"vocab_size"` + Capabilities []string `json:"capabilities"` + Encoder struct { + VisionConfig struct { + ImageSize int `json:"image_size"` + } `json:"vision_config"` + MMTokensPerImage int `json:"mm_tokens_per_image"` + } `json:"encoder"` + Decoder struct { + HiddenSize int `json:"hidden_size"` + } `json:"decoder"` + } + + if err := json.Unmarshal(data, &rawConfig); err != nil { + continue + } + + // Extract values with fallbacks + hiddenSize := rawConfig.HiddenSize + if hiddenSize == 0 { + hiddenSize = rawConfig.Decoder.HiddenSize + } + if hiddenSize == 0 { + hiddenSize = 640 // Default for T5Gemma-2 270M + } + + imageSize := rawConfig.Encoder.VisionConfig.ImageSize + if imageSize == 0 { + imageSize = 896 // Default for SigLIP in T5Gemma-2 + } + + imageSeqLen := rawConfig.Encoder.MMTokensPerImage + if imageSeqLen == 0 { + imageSeqLen = 256 // Default + } + + return &T5Gemma2Config{ + ModelType: rawConfig.ModelType, + HiddenSize: hiddenSize, + VocabSize: rawConfig.VocabSize, + ImageSize: imageSize, + ImageSeqLen: imageSeqLen, + Capabilities: rawConfig.Capabilities, + }, nil + } + + // Return default config for T5Gemma-2 270M + return &T5Gemma2Config{ + ModelType: "t5gemma2", + HiddenSize: 640, + VocabSize: 262144, + ImageSize: 896, + ImageSeqLen: 256, + Capabilities: []string{"embeddings", "generation", "decoding", "multimodal"}, + }, nil +} + +// Ensure T5Gemma2Embedder implements HiddenStatesEmbedder +var _ HiddenStatesEmbedder = (*T5Gemma2Embedder)(nil) + +// EmbedWithHiddenStates returns raw encoder hidden states before mean pooling. +// This is useful for passing to the decode API for text generation from custom embeddings. +// +// The returned hidden states have shape [batch_size][seq_len][hidden_size] where: +// - batch_size: number of input texts +// - seq_len: number of tokens in each input (varies per input) +// - hidden_size: 640 for T5Gemma-2 270M +func (t *T5Gemma2Embedder) EmbedWithHiddenStates(ctx context.Context, contents [][]ai.ContentPart) (*HiddenStatesOutput, error) { + if len(contents) == 0 { + return &HiddenStatesOutput{ + HiddenStates: [][][]float32{}, + AttentionMask: [][]int64{}, + }, nil + } + + // Extract text from contents (only text supported for hidden states currently) + texts := make([]string, 0, len(contents)) + for _, parts := range contents { + for _, part := range parts { + if tc, ok := part.(ai.TextContent); ok { + texts = append(texts, tc.Text) + break + } + } + } + + if len(texts) == 0 { + return nil, errors.New("no text content found in inputs") + } + + // Create batch and run preprocessing (tokenization) + batch := backends.NewBatch(len(texts)) + defer func() { + if batch.DestroyInputs != nil { + _ = batch.DestroyInputs() + } + }() + + if err := t.encoderPipeline.Preprocess(batch, texts); err != nil { + return nil, fmt.Errorf("preprocessing inputs: %w", err) + } + + // Run forward pass (encoder inference) + if err := t.encoderPipeline.Forward(batch); err != nil { + return nil, fmt.Errorf("running encoder forward: %w", err) + } + + // Extract raw hidden states from batch.OutputValues + // The output shape is [batch_size, seq_len, hidden_size] + if len(batch.OutputValues) == 0 { + return nil, errors.New("no output from encoder") + } + + hiddenStates, attentionMask, err := t.extractHiddenStates(batch) + if err != nil { + return nil, fmt.Errorf("extracting hidden states: %w", err) + } + + return &HiddenStatesOutput{ + HiddenStates: hiddenStates, + AttentionMask: attentionMask, + }, nil +} + +// extractHiddenStates converts batch.OutputValues to the expected format. +// The raw ONNX output is typically [batch, seq_len, hidden_size] as float32. +func (t *T5Gemma2Embedder) extractHiddenStates(batch *backends.PipelineBatch) ([][][]float32, [][]int64, error) { + // Get the first output (encoder hidden states) + rawOutput := batch.OutputValues[0] + + // The output can be in different formats depending on the backend + var hiddenStates [][][]float32 + + switch v := rawOutput.(type) { + case [][][]float32: + // Already in the expected format [batch, seq_len, hidden_size] + hiddenStates = v + case [][]float32: + // Flattened format [batch, seq_len * hidden_size] - need to reshape + hiddenSize := t.config.HiddenSize + hiddenStates = make([][][]float32, len(v)) + for i, flat := range v { + seqLen := len(flat) / hiddenSize + if seqLen*hiddenSize != len(flat) { + return nil, nil, fmt.Errorf("unexpected embedding dimension: %d (not divisible by hidden_size %d)", len(flat), hiddenSize) + } + hiddenStates[i] = make([][]float32, seqLen) + for j := 0; j < seqLen; j++ { + hiddenStates[i][j] = flat[j*hiddenSize : (j+1)*hiddenSize] + } + } + default: + return nil, nil, fmt.Errorf("unexpected output type: %T", rawOutput) + } + + // Extract attention masks from batch.Input, trimmed to match hidden states length + attentionMask := make([][]int64, len(hiddenStates)) + for i := range hiddenStates { + seqLen := len(hiddenStates[i]) + attentionMask[i] = make([]int64, seqLen) + // Fill with 1s for actual tokens (hidden states only contain non-padded tokens) + for j := 0; j < seqLen; j++ { + attentionMask[i][j] = 1 + } + } + + return hiddenStates, attentionMask, nil +} + +// IsT5Gemma2Model checks if a model directory contains T5Gemma-2 model files. +// Only encoder.onnx is required; vision_encoder.onnx is optional for text-only use. +func IsT5Gemma2Model(modelPath string) bool { + encoderPath := filepath.Join(modelPath, "encoder.onnx") + + // Must have encoder (vision encoder is optional) + if _, err := os.Stat(encoderPath); err != nil { + return false + } + + // Check config for model_type + configPath := filepath.Join(modelPath, "config.json") + if data, err := os.ReadFile(configPath); err == nil { + var config struct { + ModelType string `json:"model_type"` + } + if err := json.Unmarshal(data, &config); err == nil { + return config.ModelType == "t5gemma2" + } + } + + // Also check t5gemma2_config.json + t5gemma2ConfigPath := filepath.Join(modelPath, "t5gemma2_config.json") + if _, err := os.Stat(t5gemma2ConfigPath); err == nil { + return true + } + + return false +} diff --git a/pkg/termite/lib/embeddings/t5gemma2_stub.go b/pkg/termite/lib/embeddings/t5gemma2_stub.go new file mode 100644 index 0000000..669c2aa --- /dev/null +++ b/pkg/termite/lib/embeddings/t5gemma2_stub.go @@ -0,0 +1,97 @@ +// Copyright 2025 Antfly, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//go:build !(onnx && ORT) + +package embeddings + +import ( + "context" + "encoding/json" + "errors" + "os" + "path/filepath" + + "github.com/antflydb/antfly-go/libaf/ai" + libafembed "github.com/antflydb/antfly-go/libaf/embeddings" + "github.com/antflydb/termite/pkg/termite/lib/hugot" + "go.uber.org/zap" +) + +// T5Gemma2Embedder is a stub when built without ONNX support. +// To enable T5Gemma-2 multimodal embeddings, build with: CGO_ENABLED=1 go build -tags="onnx,ORT" +type T5Gemma2Embedder struct{} + +// NewT5Gemma2Embedder returns an error when T5Gemma-2 support is disabled. +func NewT5Gemma2Embedder(modelPath string, logger *zap.Logger) (*T5Gemma2Embedder, error) { + return nil, errors.New("T5Gemma-2 embedder not available: build with -tags=\"onnx,ORT\" to enable") +} + +// NewT5Gemma2EmbedderWithSession returns an error when T5Gemma-2 support is disabled. +func NewT5Gemma2EmbedderWithSession(modelPath string, sharedSession interface{}, logger *zap.Logger) (*T5Gemma2Embedder, error) { + return nil, errors.New("T5Gemma-2 embedder not available: build with -tags=\"onnx,ORT\" to enable") +} + +// NewT5Gemma2EmbedderWithSessionManager returns an error when T5Gemma-2 support is disabled. +func NewT5Gemma2EmbedderWithSessionManager(modelPath string, sessionManager *hugot.SessionManager, modelBackends []string, logger *zap.Logger) (*T5Gemma2Embedder, hugot.BackendType, error) { + return nil, "", errors.New("T5Gemma-2 embedder not available: build with -tags=\"onnx,ORT\" to enable") +} + +// Capabilities returns empty capabilities for the stub. +func (t *T5Gemma2Embedder) Capabilities() libafembed.EmbedderCapabilities { + return libafembed.EmbedderCapabilities{} +} + +// Embed returns an error for the stub since it cannot be used. +func (t *T5Gemma2Embedder) Embed(ctx context.Context, contents [][]ai.ContentPart) ([][]float32, error) { + return nil, errors.New("T5Gemma-2 embedder not available: build with -tags=\"onnx,ORT\" to enable") +} + +// Close is a no-op for the stub. +func (t *T5Gemma2Embedder) Close() error { + return nil +} + +// IsT5Gemma2Model checks if a model directory contains T5Gemma-2 model files +func IsT5Gemma2Model(modelPath string) bool { + encoderPath := filepath.Join(modelPath, "encoder.onnx") + visionPath := filepath.Join(modelPath, "vision_encoder.onnx") + + // Must have both encoder and vision encoder + if _, err := os.Stat(encoderPath); err != nil { + return false + } + if _, err := os.Stat(visionPath); err != nil { + return false + } + + // Check config for model_type + configPath := filepath.Join(modelPath, "config.json") + if data, err := os.ReadFile(configPath); err == nil { + var config struct { + ModelType string `json:"model_type"` + } + if err := json.Unmarshal(data, &config); err == nil { + return config.ModelType == "t5gemma2" + } + } + + // Also check t5gemma2_config.json + t5gemma2ConfigPath := filepath.Join(modelPath, "t5gemma2_config.json") + if _, err := os.Stat(t5gemma2ConfigPath); err == nil { + return true + } + + return false +} diff --git a/pkg/termite/lib/seq2seq/t5gemma2.go b/pkg/termite/lib/seq2seq/t5gemma2.go new file mode 100644 index 0000000..e1bbc59 --- /dev/null +++ b/pkg/termite/lib/seq2seq/t5gemma2.go @@ -0,0 +1,588 @@ +// Copyright 2025 Antfly, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//go:build onnx && ORT + +package seq2seq + +import ( + "bytes" + "context" + "encoding/base64" + "encoding/json" + "errors" + "fmt" + "image" + _ "image/gif" + _ "image/jpeg" + _ "image/png" + "os" + "path/filepath" + "strings" + + "github.com/antflydb/termite/pkg/termite/lib/hugot" + khugot "github.com/knights-analytics/hugot" + "github.com/knights-analytics/hugot/backends" + "github.com/knights-analytics/hugot/pipelines" + "github.com/knights-analytics/hugot/util/imageutil" + "go.uber.org/zap" + _ "golang.org/x/image/webp" +) + +// Ensure T5Gemma2Generator implements the Model interface +var _ Model = (*T5Gemma2Generator)(nil) + +// T5Gemma2Generator implements multimodal seq2seq text generation using T5Gemma-2. +// It supports text-to-text and image+text-to-text generation. +// +// T5Gemma-2 is a multimodal encoder-decoder model from Google's Gemma 3 family, +// trained with UL2 objective. It can generate text conditioned on both text and images. +// +// Build with: CGO_ENABLED=1 go build -tags="onnx,ORT" +type T5Gemma2Generator struct { + session *khugot.Session + seq2seqPipeline *pipelines.Seq2SeqPipeline + visionPipeline *pipelines.FeatureExtractionPipeline + logger *zap.Logger + sessionShared bool + config T5Gemma2GeneratorConfig + modelPath string + pipelineName string // pipeline name for hugot registry cleanup + + hasVisionFile bool // true if vision_encoder.onnx exists +} + +// T5Gemma2GeneratorConfig holds configuration for T5Gemma-2 generation +type T5Gemma2GeneratorConfig struct { + // ModelID is the original HuggingFace model ID + ModelID string `json:"model_id"` + // ModelType should be "t5gemma2" + ModelType string `json:"model_type"` + // Task indicates the model's intended use + Task string `json:"task"` + // MaxNewTokens is the maximum number of tokens to generate + MaxNewTokens int `json:"max_new_tokens"` + // NumBeams is the number of beams for beam search (1 = greedy) + NumBeams int `json:"num_beams"` + // DoSample enables sampling instead of greedy/beam search + DoSample bool `json:"do_sample"` + // Temperature controls randomness (used when DoSample=true) + Temperature float32 `json:"temperature"` + // TopP is the nucleus sampling probability (used when DoSample=true) + TopP float32 `json:"top_p"` + // RepetitionPenalty penalizes repeated tokens (1.0 = no penalty, >1.0 = penalize) + RepetitionPenalty float32 `json:"repetition_penalty"` + // ImageSize for vision encoder preprocessing + ImageSize int `json:"image_size"` + // HiddenSize of the model + HiddenSize int `json:"hidden_size"` +} + +// MultimodalInput represents input that can contain both text and images +type MultimodalInput struct { + // Text is the input text prompt + Text string + // Images contains base64-encoded or raw image data + Images [][]byte +} + +// NewT5Gemma2Generator creates a new T5Gemma-2 generator. +// The directory should contain: +// - encoder.onnx (text + multimodal encoder) +// - decoder.onnx (with past_key_values) +// - decoder-init.onnx (without past_key_values) +// - vision_encoder.onnx (SigLIP vision encoder) +// - config.json or t5gemma2_config.json +// - tokenizer.json +// +// Build with -tags="onnx,ORT" to enable this generator. +func NewT5Gemma2Generator(modelPath string, logger *zap.Logger) (*T5Gemma2Generator, error) { + return NewT5Gemma2GeneratorWithSession(modelPath, nil, logger) +} + +// NewT5Gemma2GeneratorWithSession creates a new T5Gemma-2 generator with an optional shared session. +func NewT5Gemma2GeneratorWithSession(modelPath string, sharedSession *khugot.Session, logger *zap.Logger) (*T5Gemma2Generator, error) { + if modelPath == "" { + return nil, errors.New("model path is required") + } + + if logger == nil { + logger = zap.NewNop() + } + + logger.Info("Initializing T5Gemma-2 generator", + zap.String("modelPath", modelPath), + zap.String("backend", hugot.BackendName())) + + // Load configuration + config, err := loadT5Gemma2GeneratorConfig(modelPath) + if err != nil { + return nil, fmt.Errorf("loading T5Gemma-2 config: %w", err) + } + + // Verify required files exist + requiredFiles := []string{"encoder.onnx", "decoder.onnx", "decoder-init.onnx"} + for _, file := range requiredFiles { + filePath := filepath.Join(modelPath, file) + if _, err := os.Stat(filePath); err != nil { + return nil, fmt.Errorf("required file not found: %s", filePath) + } + } + + // Check for optional vision encoder + visionPath := filepath.Join(modelPath, "vision_encoder.onnx") + hasVision := false + if _, err := os.Stat(visionPath); err == nil { + hasVision = true + } + + // Create or reuse session + session, err := hugot.NewSessionOrUseExisting(sharedSession) + if err != nil { + return nil, fmt.Errorf("creating hugot session: %w", err) + } + sessionShared := (sharedSession != nil) + + // Create Seq2Seq pipeline for text generation + pipelineName := fmt.Sprintf("t5gemma2-seq2seq:%s", filepath.Base(modelPath)) + pipelineOptions := []khugot.Seq2SeqOption{ + pipelines.WithSeq2SeqMaxTokens(config.MaxNewTokens), + // T5Gemma2 models produce a garbage first token due to decoder_start_token_id + // being None in the model config. Skip it in output. + pipelines.WithSkipFirstToken(true), + } + + // Apply repetition penalty if configured (default is 1.2) + if config.RepetitionPenalty > 0 { + pipelineOptions = append(pipelineOptions, + pipelines.WithRepetitionPenalty(config.RepetitionPenalty)) + } + + if config.DoSample && config.TopP > 0 && config.Temperature > 0 { + pipelineOptions = append(pipelineOptions, + pipelines.WithSampling(config.TopP, config.Temperature)) + } + + pipelineConfig := khugot.Seq2SeqConfig{ + ModelPath: modelPath, + Name: pipelineName, + Options: pipelineOptions, + } + + seq2seqPipeline, err := khugot.NewPipeline(session, pipelineConfig) + if err != nil { + if !sessionShared { + session.Destroy() + } + return nil, fmt.Errorf("creating Seq2Seq pipeline: %w", err) + } + + // Load vision pipeline eagerly if available. + // The vision encoder is required for correct text generation behavior, + // even when only processing text (discovered in previous testing). + var visionPipeline *pipelines.FeatureExtractionPipeline + if hasVision { + logger.Info("Loading vision encoder pipeline", + zap.String("modelPath", modelPath)) + + imageSize := config.ImageSize + if imageSize == 0 { + imageSize = 896 // Default for T5Gemma-2 (SigLIP) + } + + visionPipelineName := fmt.Sprintf("t5gemma2-vision:%s", filepath.Base(modelPath)) + visionConfig := khugot.FeatureExtractionConfig{ + ModelPath: modelPath, + Name: visionPipelineName, + OnnxFilename: "vision_encoder.onnx", + Options: []backends.PipelineOption[*pipelines.FeatureExtractionPipeline]{ + pipelines.WithImageMode(), + pipelines.WithPreprocessSteps[*pipelines.FeatureExtractionPipeline]( + imageutil.ResizeStep(imageSize), + imageutil.CenterCropStep(imageSize, imageSize), + ), + pipelines.WithNormalizationSteps[*pipelines.FeatureExtractionPipeline]( + imageutil.RescaleStep(), + imageutil.CLIPPixelNormalizationStep(), + ), + pipelines.WithNCHWFormat[*pipelines.FeatureExtractionPipeline](), + }, + } + + var err error + visionPipeline, err = khugot.NewPipeline(session, visionConfig) + if err != nil { + if !sessionShared { + session.Destroy() + } + return nil, fmt.Errorf("creating vision pipeline: %w", err) + } + + logger.Info("T5Gemma-2 generator initialized (with vision encoder)", + zap.String("task", config.Task), + zap.Int("max_new_tokens", config.MaxNewTokens), + zap.Int("imageSize", imageSize)) + } else { + logger.Info("T5Gemma-2 generator initialized (text-only mode)", + zap.String("task", config.Task), + zap.Int("max_new_tokens", config.MaxNewTokens)) + } + + return &T5Gemma2Generator{ + session: session, + seq2seqPipeline: seq2seqPipeline, + visionPipeline: visionPipeline, + logger: logger, + sessionShared: sessionShared, + config: config, + modelPath: modelPath, + pipelineName: pipelineName, + hasVisionFile: hasVision, + }, nil +} + +// NewT5Gemma2GeneratorWithSessionManager creates a new T5Gemma-2 generator using a SessionManager. +func NewT5Gemma2GeneratorWithSessionManager(modelPath string, sessionManager *hugot.SessionManager, modelBackends []string, logger *zap.Logger) (*T5Gemma2Generator, hugot.BackendType, error) { + if sessionManager == nil { + gen, err := NewT5Gemma2GeneratorWithSession(modelPath, nil, logger) + if err != nil { + return nil, "", err + } + return gen, hugot.BackendType(""), nil + } + + // T5Gemma-2 requires ONNX Runtime + if modelBackends == nil { + modelBackends = []string{"onnx"} + } + session, backendUsed, err := sessionManager.GetSessionForModel(modelBackends) + if err != nil { + return nil, "", fmt.Errorf("getting session from manager: %w", err) + } + + gen, err := NewT5Gemma2GeneratorWithSession(modelPath, session, logger) + if err != nil { + return nil, "", err + } + + gen.sessionShared = true + return gen, backendUsed, nil +} + +// Generate runs text generation on the given text inputs. +func (g *T5Gemma2Generator) Generate(ctx context.Context, inputs []string) (*GeneratedOutput, error) { + if len(inputs) == 0 { + return &GeneratedOutput{ + Texts: [][]string{}, + Tokens: [][][]uint32{}, + }, nil + } + + select { + case <-ctx.Done(): + return nil, ctx.Err() + default: + } + + g.logger.Debug("Starting T5Gemma-2 generation", + zap.Int("num_inputs", len(inputs))) + + output, err := g.seq2seqPipeline.RunPipeline(inputs) + if err != nil { + g.logger.Error("T5Gemma-2 generation failed", zap.Error(err)) + return nil, fmt.Errorf("running T5Gemma-2 pipeline: %w", err) + } + + g.logger.Debug("T5Gemma-2 generation completed", + zap.Int("num_inputs", len(inputs)), + zap.Int("total_outputs", len(output.GeneratedTexts))) + + return &GeneratedOutput{ + Texts: output.GeneratedTexts, + Tokens: output.GeneratedTokens, + }, nil +} + +// GenerateMultimodal generates text from multimodal inputs (text + images). +// +// CURRENT LIMITATION: This method validates and processes images through the vision +// encoder but does not yet inject the image embeddings into the text generation. +// The seq2seq pipeline currently only processes the text with placeholders. +// Full multimodal generation (where image embeddings condition the output) requires +// modifications to the hugot Seq2SeqPipeline to accept external embeddings. +// +// Use ProcessImage() to get image embeddings for custom multimodal workflows. +func (g *T5Gemma2Generator) GenerateMultimodal(ctx context.Context, inputs []MultimodalInput) (*GeneratedOutput, error) { + if len(inputs) == 0 { + return &GeneratedOutput{ + Texts: [][]string{}, + Tokens: [][][]uint32{}, + }, nil + } + + // Load vision pipeline lazily when we have images to process + hasImages := false + for _, input := range inputs { + if len(input.Images) > 0 { + hasImages = true + break + } + } + + if hasImages { + if err := g.loadVisionPipeline(); err != nil { + return nil, fmt.Errorf("multimodal generation: %w", err) + } + } + + select { + case <-ctx.Done(): + return nil, ctx.Err() + default: + } + + // Validate and process images through the vision pipeline + // This ensures images are valid even though embeddings aren't used yet + for i, input := range inputs { + for j, imgData := range input.Images { + if _, err := g.ProcessImage(imgData); err != nil { + return nil, fmt.Errorf("processing image %d for input %d: %w", j, i, err) + } + } + } + + // Process text inputs with image placeholders + // Note: Image embeddings are validated above but not injected into generation yet + textInputs := make([]string, len(inputs)) + for i, input := range inputs { + // Prepend image tokens for each image + prefix := "" + for range input.Images { + prefix += " " + } + textInputs[i] = prefix + input.Text + } + + return g.Generate(ctx, textInputs) +} + +// ProcessImage encodes an image through the vision encoder. +// Returns the image embeddings that can be used for conditioning. +// The vision encoder is loaded lazily on first call. +func (g *T5Gemma2Generator) ProcessImage(imageData []byte) ([]float32, error) { + if err := g.loadVisionPipeline(); err != nil { + return nil, err + } + + // Decode image + img, _, err := image.Decode(bytes.NewReader(imageData)) + if err != nil { + return nil, fmt.Errorf("decoding image: %w", err) + } + + // Run through vision pipeline + output, err := g.visionPipeline.RunWithImages([]image.Image{img}) + if err != nil { + return nil, fmt.Errorf("running vision pipeline: %w", err) + } + + if len(output.Embeddings) == 0 || len(output.Embeddings[0]) == 0 { + return nil, errors.New("no embedding returned from vision pipeline") + } + + return output.Embeddings[0], nil +} + +// HasVision returns true if this generator supports image input. +// This checks if the vision encoder file exists, not if it's currently loaded. +func (g *T5Gemma2Generator) HasVision() bool { + return g.hasVisionFile +} + +// IsVisionLoaded returns true if the vision encoder is currently loaded in memory. +func (g *T5Gemma2Generator) IsVisionLoaded() bool { + return g.visionPipeline != nil +} + +// loadVisionPipeline checks that the vision encoder pipeline is available. +// The vision pipeline is loaded eagerly at initialization time. +func (g *T5Gemma2Generator) loadVisionPipeline() error { + if g.visionPipeline == nil { + return errors.New("vision encoder not available: vision_encoder.onnx not found or failed to load") + } + return nil +} + +// Config returns the generator configuration. +func (g *T5Gemma2Generator) Config() T5Gemma2GeneratorConfig { + return g.config +} + +// Close releases resources. +func (g *T5Gemma2Generator) Close() error { + var errs []error + + // Use ClosePipeline to properly remove from session registry AND destroy resources. + // Just calling Destroy() would leave a stale entry in the registry, causing + // "pipeline already initialised" errors on subsequent loads. + if g.seq2seqPipeline != nil && g.session != nil && g.pipelineName != "" { + if err := khugot.ClosePipeline[*pipelines.Seq2SeqPipeline](g.session, g.pipelineName); err != nil { + errs = append(errs, fmt.Errorf("closing seq2seq pipeline: %w", err)) + } + } + + // Note: FeatureExtractionPipeline doesn't have a Destroy method, + // it's cleaned up when the session is destroyed + + if g.session != nil && !g.sessionShared { + g.logger.Info("Destroying Hugot session (owned by this T5Gemma-2 generator)") + g.session.Destroy() + } + + return errors.Join(errs...) +} + +// loadT5Gemma2GeneratorConfig loads T5Gemma-2 generation config from model directory +func loadT5Gemma2GeneratorConfig(modelPath string) (T5Gemma2GeneratorConfig, error) { + config := T5Gemma2GeneratorConfig{ + MaxNewTokens: 256, + NumBeams: 1, + DoSample: false, + Temperature: 1.0, + TopP: 1.0, + RepetitionPenalty: 1.2, // Default penalty to avoid degenerate repetition + ImageSize: 896, + HiddenSize: 640, + } + + // Try t5gemma2_config.json first, then config.json + configPaths := []string{ + filepath.Join(modelPath, "t5gemma2_config.json"), + filepath.Join(modelPath, "config.json"), + } + + for _, path := range configPaths { + data, err := os.ReadFile(path) + if err != nil { + continue + } + + // Parse config with nested generation_config + var rawConfig struct { + ModelID string `json:"model_id"` + ModelType string `json:"model_type"` + Task string `json:"task"` + HiddenSize int `json:"hidden_size"` + GenerationConfig struct { + MaxNewTokens int `json:"max_new_tokens"` + NumBeams int `json:"num_beams"` + DoSample bool `json:"do_sample"` + Temperature float32 `json:"temperature"` + TopP float32 `json:"top_p"` + RepetitionPenalty float32 `json:"repetition_penalty"` + } `json:"generation_config"` + Encoder struct { + VisionConfig struct { + ImageSize int `json:"image_size"` + } `json:"vision_config"` + } `json:"encoder"` + Decoder struct { + HiddenSize int `json:"hidden_size"` + } `json:"decoder"` + } + + if err := json.Unmarshal(data, &rawConfig); err != nil { + continue + } + + config.ModelID = rawConfig.ModelID + config.ModelType = rawConfig.ModelType + config.Task = rawConfig.Task + + if rawConfig.GenerationConfig.MaxNewTokens > 0 { + config.MaxNewTokens = rawConfig.GenerationConfig.MaxNewTokens + } + if rawConfig.GenerationConfig.NumBeams > 0 { + config.NumBeams = rawConfig.GenerationConfig.NumBeams + } + config.DoSample = rawConfig.GenerationConfig.DoSample + if rawConfig.GenerationConfig.Temperature > 0 { + config.Temperature = rawConfig.GenerationConfig.Temperature + } + if rawConfig.GenerationConfig.TopP > 0 { + config.TopP = rawConfig.GenerationConfig.TopP + } + if rawConfig.GenerationConfig.RepetitionPenalty > 0 { + config.RepetitionPenalty = rawConfig.GenerationConfig.RepetitionPenalty + } + + if rawConfig.Encoder.VisionConfig.ImageSize > 0 { + config.ImageSize = rawConfig.Encoder.VisionConfig.ImageSize + } + if rawConfig.HiddenSize > 0 { + config.HiddenSize = rawConfig.HiddenSize + } else if rawConfig.Decoder.HiddenSize > 0 { + config.HiddenSize = rawConfig.Decoder.HiddenSize + } + + return config, nil + } + + return config, nil +} + +// IsT5Gemma2GeneratorModel checks if a model directory contains T5Gemma-2 generator files. +func IsT5Gemma2GeneratorModel(modelPath string) bool { + // Must have standard seq2seq files + requiredFiles := []string{"encoder.onnx", "decoder.onnx", "decoder-init.onnx"} + for _, file := range requiredFiles { + filePath := filepath.Join(modelPath, file) + if _, err := os.Stat(filePath); err != nil { + return false + } + } + + // Check config for model_type: "t5gemma2" + configPath := filepath.Join(modelPath, "config.json") + if data, err := os.ReadFile(configPath); err == nil { + var config struct { + ModelType string `json:"model_type"` + } + if err := json.Unmarshal(data, &config); err == nil { + return config.ModelType == "t5gemma2" + } + } + + // Also check t5gemma2_config.json + t5gemma2ConfigPath := filepath.Join(modelPath, "t5gemma2_config.json") + if _, err := os.Stat(t5gemma2ConfigPath); err == nil { + return true + } + + return false +} + +// DecodeBase64Image decodes a base64-encoded image (data URI or raw base64). +func DecodeBase64Image(input string) ([]byte, error) { + // Handle data URI format: data:image/png;base64,... + if strings.HasPrefix(input, "data:") { + parts := strings.SplitN(input, ",", 2) + if len(parts) != 2 { + return nil, errors.New("invalid data URI format") + } + input = parts[1] + } + + return base64.StdEncoding.DecodeString(input) +} diff --git a/pkg/termite/lib/seq2seq/t5gemma2_stub.go b/pkg/termite/lib/seq2seq/t5gemma2_stub.go new file mode 100644 index 0000000..38c8a18 --- /dev/null +++ b/pkg/termite/lib/seq2seq/t5gemma2_stub.go @@ -0,0 +1,143 @@ +// Copyright 2025 Antfly, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//go:build !(onnx && ORT) + +package seq2seq + +import ( + "context" + "encoding/base64" + "encoding/json" + "errors" + "os" + "path/filepath" + "strings" + + "github.com/antflydb/termite/pkg/termite/lib/hugot" + "go.uber.org/zap" +) + +// T5Gemma2Generator is a stub when built without ONNX support. +// To enable T5Gemma-2 generation, build with: CGO_ENABLED=1 go build -tags="onnx,ORT" +type T5Gemma2Generator struct{} + +// T5Gemma2GeneratorConfig holds configuration for T5Gemma-2 generation +type T5Gemma2GeneratorConfig struct { + ModelID string `json:"model_id"` + ModelType string `json:"model_type"` + Task string `json:"task"` + MaxNewTokens int `json:"max_new_tokens"` + NumBeams int `json:"num_beams"` + DoSample bool `json:"do_sample"` + Temperature float32 `json:"temperature"` + TopP float32 `json:"top_p"` + ImageSize int `json:"image_size"` + HiddenSize int `json:"hidden_size"` +} + +// MultimodalInput represents input that can contain both text and images +type MultimodalInput struct { + Text string + Images [][]byte +} + +// NewT5Gemma2Generator returns an error when ONNX Runtime is not available. +func NewT5Gemma2Generator(modelPath string, logger *zap.Logger) (*T5Gemma2Generator, error) { + return nil, errors.New("T5Gemma-2 generator requires ONNX Runtime: build with -tags=\"onnx,ORT\"") +} + +// NewT5Gemma2GeneratorWithSession returns an error when ONNX Runtime is not available. +func NewT5Gemma2GeneratorWithSession(modelPath string, sharedSession interface{}, logger *zap.Logger) (*T5Gemma2Generator, error) { + return nil, errors.New("T5Gemma-2 generator requires ONNX Runtime: build with -tags=\"onnx,ORT\"") +} + +// NewT5Gemma2GeneratorWithSessionManager returns an error when ONNX Runtime is not available. +func NewT5Gemma2GeneratorWithSessionManager(modelPath string, sessionManager *hugot.SessionManager, modelBackends []string, logger *zap.Logger) (*T5Gemma2Generator, hugot.BackendType, error) { + return nil, "", errors.New("T5Gemma-2 generator requires ONNX Runtime: build with -tags=\"onnx,ORT\"") +} + +// Generate returns an error for the stub. +func (g *T5Gemma2Generator) Generate(ctx context.Context, inputs []string) (*GeneratedOutput, error) { + return nil, errors.New("T5Gemma-2 generator requires ONNX Runtime: build with -tags=\"onnx,ORT\"") +} + +// GenerateMultimodal returns an error for the stub. +func (g *T5Gemma2Generator) GenerateMultimodal(ctx context.Context, inputs []MultimodalInput) (*GeneratedOutput, error) { + return nil, errors.New("T5Gemma-2 generator requires ONNX Runtime: build with -tags=\"onnx,ORT\"") +} + +// ProcessImage returns an error for the stub. +func (g *T5Gemma2Generator) ProcessImage(imageData []byte) ([]float32, error) { + return nil, errors.New("T5Gemma-2 generator requires ONNX Runtime: build with -tags=\"onnx,ORT\"") +} + +// HasVision returns false for the stub. +func (g *T5Gemma2Generator) HasVision() bool { + return false +} + +// Config returns an empty configuration for the stub. +func (g *T5Gemma2Generator) Config() T5Gemma2GeneratorConfig { + return T5Gemma2GeneratorConfig{} +} + +// Close is a no-op for the stub. +func (g *T5Gemma2Generator) Close() error { + return nil +} + +// IsT5Gemma2GeneratorModel checks if a model directory contains T5Gemma-2 generator files. +func IsT5Gemma2GeneratorModel(modelPath string) bool { + // Must have standard seq2seq files + requiredFiles := []string{"encoder.onnx", "decoder.onnx", "decoder-init.onnx"} + for _, file := range requiredFiles { + filePath := filepath.Join(modelPath, file) + if _, err := os.Stat(filePath); err != nil { + return false + } + } + + // Check config for model_type: "t5gemma2" + configPath := filepath.Join(modelPath, "config.json") + if data, err := os.ReadFile(configPath); err == nil { + var config struct { + ModelType string `json:"model_type"` + } + if err := json.Unmarshal(data, &config); err == nil { + return config.ModelType == "t5gemma2" + } + } + + // Also check t5gemma2_config.json + t5gemma2ConfigPath := filepath.Join(modelPath, "t5gemma2_config.json") + if _, err := os.Stat(t5gemma2ConfigPath); err == nil { + return true + } + + return false +} + +// DecodeBase64Image decodes a base64-encoded image (data URI or raw base64). +func DecodeBase64Image(input string) ([]byte, error) { + if strings.HasPrefix(input, "data:") { + parts := strings.SplitN(input, ",", 2) + if len(parts) != 2 { + return nil, errors.New("invalid data URI format") + } + input = parts[1] + } + + return base64.StdEncoding.DecodeString(input) +} diff --git a/pkg/termite/model_registry.go b/pkg/termite/model_registry.go index 33b14f1..9d6d342 100644 --- a/pkg/termite/model_registry.go +++ b/pkg/termite/model_registry.go @@ -64,12 +64,56 @@ func isMultimodalModel(modelPath string) (hasStandard, hasQuantized bool) { return } +// isT5Gemma2Model checks if a model directory contains T5Gemma-2 model files. +// T5Gemma-2 models have encoder.onnx + vision_encoder.onnx. +func isT5Gemma2Model(modelPath string) bool { + encoderPath := filepath.Join(modelPath, "encoder.onnx") + visionPath := filepath.Join(modelPath, "vision_encoder.onnx") + + // Must have both encoder and vision encoder + if !fileExistsRegistry(encoderPath) || !fileExistsRegistry(visionPath) { + return false + } + + // Check config.json for model_type: "t5gemma2" + configPath := filepath.Join(modelPath, "config.json") + if data, err := os.ReadFile(configPath); err == nil { + var config struct { + ModelType string `json:"model_type"` + } + if err := json.Unmarshal(data, &config); err == nil { + return config.ModelType == "t5gemma2" + } + } + + // Also check t5gemma2_config.json + t5gemma2ConfigPath := filepath.Join(modelPath, "t5gemma2_config.json") + return fileExistsRegistry(t5gemma2ConfigPath) +} + // fileExistsRegistry checks if a file exists func fileExistsRegistry(path string) bool { _, err := os.Stat(path) return err == nil } +// isDirectoryEntry checks if an entry is a directory, following symlinks. +// This is needed because os.DirEntry.IsDir() returns false for symlinks. +func isDirectoryEntry(path string, entry os.DirEntry) bool { + // Fast path: regular directory + if entry.IsDir() { + return true + } + // Check if it's a symlink to a directory + if entry.Type()&os.ModeSymlink != 0 { + info, err := os.Stat(path) // Stat follows symlinks + if err == nil && info.IsDir() { + return true + } + } + return false +} + // DiscoveredModel represents a model found during directory scanning type DiscoveredModel struct { // Ref is the parsed model reference (owner/name) @@ -113,7 +157,8 @@ func discoverModelsInDir(modelsDir string, modelType modelregistry.ModelType, lo } for _, ownerEntry := range ownerEntries { - if !ownerEntry.IsDir() { + // Check if it's a directory (or symlink to directory) + if !isDirectoryEntry(filepath.Join(modelsDir, ownerEntry.Name()), ownerEntry) { continue } @@ -128,7 +173,8 @@ func discoverModelsInDir(modelsDir string, modelType modelregistry.ModelType, lo } for _, modelEntry := range modelEntries { - if !modelEntry.IsDir() { + // Check if it's a directory (or symlink to directory) + if !isDirectoryEntry(filepath.Join(ownerPath, modelEntry.Name()), modelEntry) { continue } modelPath := filepath.Join(ownerPath, modelEntry.Name()) diff --git a/pkg/termite/openapi.yaml b/pkg/termite/openapi.yaml index 4797720..66227bb 100644 --- a/pkg/termite/openapi.yaml +++ b/pkg/termite/openapi.yaml @@ -151,6 +151,12 @@ components: type: boolean default: true description: Truncate input to fit model context length + return_hidden_states: + type: boolean + default: false + description: | + If true, returns raw encoder hidden states [seq_len, hidden_size] per input + instead of mean-pooled embeddings. Useful for passing to the decode API. EmbedResponse: type: object @@ -161,7 +167,6 @@ components: } required: - model - - embeddings properties: model: type: string @@ -174,7 +179,30 @@ components: items: type: number format: float - description: Array of embedding vectors (one per input string) + description: | + Array of mean-pooled embedding vectors [hidden_size] per input. + Present when return_hidden_states is false (default). + hidden_states: + type: array + items: + type: array + items: + type: array + items: + type: number + format: float + description: | + Raw encoder hidden states [seq_len, hidden_size] per input. + Present when return_hidden_states is true. + attention_mask: + type: array + items: + type: array + items: + type: integer + description: | + Attention mask [seq_len] per input indicating valid tokens. + Present when return_hidden_states is true. # Chunking Types - reference existing schemas Chunk: diff --git a/pkg/termite/seq2seq_registry.go b/pkg/termite/seq2seq_registry.go index ddc7d65..ee9d041 100644 --- a/pkg/termite/seq2seq_registry.go +++ b/pkg/termite/seq2seq_registry.go @@ -26,6 +26,7 @@ import ( "github.com/antflydb/termite/pkg/termite/lib/seq2seq" "github.com/jellydator/ttlcache/v3" "go.uber.org/zap" + "golang.org/x/sync/singleflight" ) // Seq2SeqModelInfo holds metadata about a discovered Seq2Seq model (not loaded yet) @@ -47,6 +48,9 @@ type Seq2SeqRegistry struct { // Loaded models with TTL cache cache *ttlcache.Cache[string, seq2seq.Model] + // Singleflight to prevent concurrent loads of the same model + sfGroup singleflight.Group + // Configuration keepAlive time.Duration maxLoadedModels uint64 @@ -196,9 +200,10 @@ func (r *Seq2SeqRegistry) discoverModels() error { return nil } -// Get returns a Seq2Seq model by name, loading it if necessary +// Get returns a Seq2Seq model by name, loading it if necessary. +// Uses singleflight to prevent concurrent loads of the same model. func (r *Seq2SeqRegistry) Get(modelName string) (seq2seq.Model, error) { - // Check cache first + // Check cache first (fast path) if item := r.cache.Get(modelName); item != nil { r.logger.Debug("Seq2Seq cache hit", zap.String("model", modelName)) return item.Value(), nil @@ -213,8 +218,29 @@ func (r *Seq2SeqRegistry) Get(modelName string) (seq2seq.Model, error) { return nil, fmt.Errorf("Seq2Seq model not found: %s", modelName) } - // Load the model - return r.loadModel(info) + // Use singleflight to prevent concurrent loads of the same model + result, err, shared := r.sfGroup.Do(modelName, func() (interface{}, error) { + // Double-check cache in case another goroutine loaded while we waited + if item := r.cache.Get(modelName); item != nil { + r.logger.Debug("Seq2Seq cache hit during singleflight", + zap.String("model", modelName)) + return item.Value(), nil + } + + // Load the model + return r.loadModel(info) + }) + + if err != nil { + return nil, err + } + + if shared { + r.logger.Debug("Seq2Seq singleflight shared result", + zap.String("model", modelName)) + } + + return result.(seq2seq.Model), nil } // GetQuestionGenerator returns a Seq2Seq model as a QuestionGenerator by name @@ -237,22 +263,46 @@ func (r *Seq2SeqRegistry) loadModel(info *Seq2SeqModelInfo) (seq2seq.Model, erro zap.String("model", info.Name), zap.String("path", info.Path)) - // Load the Seq2Seq model - // Restrict to ONNX backend only - CoreML cannot handle dynamic batch sizes > 1 - // and large seq2seq models (like PEGASUS) exceed CoreML's model size limits. - onnxOnly := []string{"onnx"} - model, backendUsed, err := seq2seq.NewHugotSeq2SeqWithSessionManager( - info.Path, r.sessionManager, onnxOnly, r.logger.Named(info.Name)) - if err != nil { - return nil, fmt.Errorf("loading Seq2Seq model %s: %w", info.Name, err) - } + var model seq2seq.Model + var backendUsed string + + // Check if this is a T5Gemma-2 model (multimodal seq2seq) + if seq2seq.IsT5Gemma2GeneratorModel(info.Path) { + r.logger.Info("Detected T5Gemma-2 generator model", + zap.String("model", info.Name)) - config := model.Config() - r.logger.Info("Successfully loaded Seq2Seq model", - zap.String("name", info.Name), - zap.String("task", config.Task), - zap.Int("max_length", config.MaxLength), - zap.String("backend", string(backendUsed))) + t5Model, backend, err := seq2seq.NewT5Gemma2GeneratorWithSessionManager( + info.Path, r.sessionManager, nil, r.logger.Named(info.Name)) + if err != nil { + return nil, fmt.Errorf("loading T5Gemma-2 model %s: %w", info.Name, err) + } + model = t5Model + backendUsed = string(backend) + + config := t5Model.Config() + r.logger.Info("Successfully loaded T5Gemma-2 generator", + zap.String("name", info.Name), + zap.String("task", config.Task), + zap.Int("max_new_tokens", config.MaxNewTokens), + zap.Bool("has_vision", t5Model.HasVision()), + zap.String("backend", backendUsed)) + } else { + // Load as standard HugotSeq2Seq model + hugotModel, backend, err := seq2seq.NewHugotSeq2SeqWithSessionManager( + info.Path, r.sessionManager, nil, r.logger.Named(info.Name)) + if err != nil { + return nil, fmt.Errorf("loading Seq2Seq model %s: %w", info.Name, err) + } + model = hugotModel + backendUsed = string(backend) + + config := hugotModel.Config() + r.logger.Info("Successfully loaded Seq2Seq model", + zap.String("name", info.Name), + zap.String("task", config.Task), + zap.Int("max_length", config.MaxLength), + zap.String("backend", backendUsed)) + } // Add to cache r.cache.Set(info.Name, model, r.keepAlive) @@ -272,6 +322,13 @@ func (r *Seq2SeqRegistry) List() []string { return names } +// GetModelInfo returns the model info for a discovered model (not necessarily loaded). +func (r *Seq2SeqRegistry) GetModelInfo(modelName string) *Seq2SeqModelInfo { + r.mu.RLock() + defer r.mu.RUnlock() + return r.discovered[modelName] +} + // ListLoaded returns only the currently loaded Seq2Seq model names func (r *Seq2SeqRegistry) ListLoaded() []string { return r.cache.Keys() diff --git a/scripts/export_model_to_registry.py b/scripts/export_model_to_registry.py index caf0a4c..24786fe 100755 --- a/scripts/export_model_to_registry.py +++ b/scripts/export_model_to_registry.py @@ -112,7 +112,7 @@ ) logger = logging.getLogger(__name__) -ModelType = Literal["embedder", "reranker", "chunker", "recognizer", "rewriter", "generator"] +ModelType = Literal["embedder", "reranker", "chunker", "recognizer", "rewriter", "generator", "t5gemma2"] # Recognizer capabilities - these describe what extraction tasks the model supports # Used in manifest to advertise model capabilities to Termite @@ -155,6 +155,11 @@ "default_model": "google/gemma-3-1b-it", "dir_name": "generators", }, + "t5gemma2": { + "ort_class": None, # Uses custom export script + "default_model": "google/t5gemma-2-270m-270m", + "dir_name": "t5gemma2", + }, } # Files to include in the manifest (in order of importance) @@ -241,6 +246,24 @@ "generation_config.json", ] +# Files for T5Gemma-2 multimodal encoder-decoder models +T5GEMMA2_MANIFEST_FILES = [ + "encoder.onnx", + "encoder.onnx_data", + "vision_encoder.onnx", + "vision_encoder.onnx_data", + "decoder-init.onnx", + "decoder-init.onnx_data", + "decoder.onnx", + "decoder.onnx_data", + "tokenizer.json", + "config.json", + "tokenizer_config.json", + "special_tokens_map.json", + "t5gemma2_config.json", + "preprocessor_config.json", +] + def detect_recognizer_type(model_id: str) -> tuple[str, list[str]]: """ @@ -1117,6 +1140,78 @@ def detect_and_add_tool_call_format(model_dir: Path) -> None: logger.warning(f"Failed to update genai_config.json: {e}") +def export_t5gemma2_model( + model_id: str, + output_dir: Path, + variants: list[str] | None = None, +) -> Path: + """ + Export a T5Gemma-2 multimodal encoder-decoder model to ONNX format. + + Uses the custom export_t5gemma2.py script which handles: + - Text encoder export (encoder.onnx) + - Vision encoder export (vision_encoder.onnx) if multimodal + - Decoder export (decoder-init.onnx, decoder.onnx) + - Tokenizer and configuration files + + Args: + model_id: HuggingFace model ID (e.g., google/t5gemma-2-270m-270m) + output_dir: Directory to save the model + variants: List of variant types (not used currently, reserved for future) + """ + import subprocess + + output_dir.mkdir(parents=True, exist_ok=True) + + logger.info(f"Exporting T5Gemma-2 model: {model_id}") + logger.info(f"Output: {output_dir}") + + # Get the path to the export script + script_dir = Path(__file__).parent + export_script = script_dir / "export_t5gemma2.py" + + if not export_script.exists(): + raise RuntimeError(f"T5Gemma-2 export script not found: {export_script}") + + # Build the export command + cmd = [ + "python", str(export_script), + "--model", model_id, + "--output", str(output_dir), + ] + + logger.info(f"Running: {' '.join(cmd)}") + + try: + result = subprocess.run( + cmd, + capture_output=True, + text=True, + check=True, + ) + if result.stdout: + for line in result.stdout.strip().split('\n'): + logger.info(f" {line}") + logger.info("T5Gemma-2 export completed successfully") + except subprocess.CalledProcessError as e: + logger.error("T5Gemma-2 export failed") + if e.stdout: + logger.error(f"stdout: {e.stdout}") + if e.stderr: + logger.error(f"stderr: {e.stderr}") + raise RuntimeError(f"T5Gemma-2 export failed: {e.stderr}") + + # Log exported files + logger.info("\nExported files:") + for f in sorted(output_dir.rglob("*")): + if f.is_file(): + size_mb = f.stat().st_size / (1024 * 1024) + rel_path = f.relative_to(output_dir) + logger.info(f" {rel_path}: {size_mb:.2f} MB") + + return output_dir + + def generate_manifest( model_type: ModelType, model_name: str, @@ -1147,10 +1242,13 @@ def generate_manifest( # Use appropriate file list based on model type and capabilities is_multimodal = capabilities and "multimodal" in capabilities is_generator = model_type == "generator" + is_t5gemma2 = model_type == "t5gemma2" is_gliner = recognizer_arch == "gliner" is_rebel = recognizer_arch == "rebel" - if is_multimodal: + if is_t5gemma2: + file_list = T5GEMMA2_MANIFEST_FILES + elif is_multimodal: file_list = MULTIMODAL_MANIFEST_FILES elif model_type == "rewriter": file_list = SEQ2SEQ_MANIFEST_FILES @@ -1866,6 +1964,9 @@ def test_model( logger.info("Testing exported model...") capabilities = capabilities or [] + if model_type == "t5gemma2": + return test_t5gemma2_model(model_dir) + if "multimodal" in capabilities: return test_multimodal_model(model_dir) @@ -2281,6 +2382,103 @@ def test_generator_model(model_dir: Path) -> bool: return False +def test_t5gemma2_model(model_dir: Path) -> bool: + """Test a T5Gemma-2 multimodal encoder-decoder model.""" + import onnxruntime as ort + import numpy as np + + try: + # Check for required files + encoder_path = model_dir / "encoder.onnx" + decoder_init_path = model_dir / "decoder-init.onnx" + + if not encoder_path.exists(): + logger.error(f"encoder.onnx not found in {model_dir}") + return False + if not decoder_init_path.exists(): + logger.error(f"decoder-init.onnx not found in {model_dir}") + return False + + # Check for config + config_path = model_dir / "t5gemma2_config.json" + if config_path.exists(): + logger.info("Loading t5gemma2_config.json...") + with open(config_path) as f: + config = json.load(f) + logger.info(f" Model ID: {config.get('model_id', 'unknown')}") + logger.info(f" Hidden size: {config.get('hidden_size', 'unknown')}") + logger.info(f" Capabilities: {', '.join(config.get('capabilities', []))}") + + # Load encoder + logger.info("Loading encoder ONNX model...") + encoder_session = ort.InferenceSession( + str(encoder_path), + providers=["CPUExecutionProvider"], + ) + + # Get input info + encoder_inputs = encoder_session.get_inputs() + logger.info(f" Encoder inputs: {[i.name for i in encoder_inputs]}") + + # Create dummy inputs + batch_size = 1 + seq_len = 32 + + dummy_input_ids = np.ones((batch_size, seq_len), dtype=np.int64) + dummy_attention_mask = np.ones((batch_size, seq_len), dtype=np.int64) + + # Run encoder + logger.info("Running encoder inference...") + encoder_outputs = encoder_session.run( + None, + { + "input_ids": dummy_input_ids, + "attention_mask": dummy_attention_mask, + }, + ) + encoder_hidden_states = encoder_outputs[0] + logger.info(f" Encoder output shape: {encoder_hidden_states.shape}") + + # Test decoder-init + logger.info("Loading decoder-init ONNX model...") + decoder_init_session = ort.InferenceSession( + str(decoder_init_path), + providers=["CPUExecutionProvider"], + ) + + decoder_inputs = decoder_init_session.get_inputs() + logger.info(f" Decoder inputs: {[i.name for i in decoder_inputs]}") + + # Create dummy decoder inputs + dummy_decoder_input_ids = np.zeros((batch_size, 1), dtype=np.int64) + + logger.info("Running decoder-init inference...") + decoder_outputs = decoder_init_session.run( + None, + { + "input_ids": dummy_decoder_input_ids, + "encoder_hidden_states": encoder_hidden_states, + "encoder_attention_mask": dummy_attention_mask, + }, + ) + logits = decoder_outputs[0] + logger.info(f" Decoder logits shape: {logits.shape}") + + # Check vision encoder if present + vision_path = model_dir / "vision_encoder.onnx" + if vision_path.exists(): + logger.info("Vision encoder found (multimodal model)") + + logger.info("Test passed!") + return True + + except Exception as e: + logger.error(f"Test failed: {e}") + import traceback + traceback.print_exc() + return False + + def cmd_gc(args): """Handle the gc subcommand.""" endpoint = args.r2_endpoint or os.environ.get("AWS_ENDPOINT_URL") @@ -2344,7 +2542,12 @@ def cmd_export(args): # Export model using appropriate function logger.info("\n[1/4] Exporting model to ONNX...") hf_token = getattr(args, "hf_token", None) - if args.model_type == "rewriter": + if args.model_type == "t5gemma2": + # Add default capabilities for T5Gemma-2 + if not capabilities: + capabilities = ["embeddings", "generation", "decoding", "multimodal"] + export_t5gemma2_model(model_id, model_dir, args.variants) + elif args.model_type == "rewriter": export_seq2seq_model(model_id, model_dir, args.variants) elif args.model_type == "generator": export_generator_model(model_id, model_dir, args.variants, hf_token=hf_token) @@ -2524,7 +2727,7 @@ def main(): ) # Export subcommands (one for each model type) - for model_type in ["embedder", "reranker", "chunker", "recognizer", "rewriter", "generator"]: + for model_type in ["embedder", "reranker", "chunker", "recognizer", "rewriter", "generator", "t5gemma2"]: export_parser = subparsers.add_parser( model_type, help=f"Export a {model_type} model to ONNX", diff --git a/scripts/export_t5gemma2.py b/scripts/export_t5gemma2.py new file mode 100755 index 0000000..1fea622 --- /dev/null +++ b/scripts/export_t5gemma2.py @@ -0,0 +1,1143 @@ +#!/usr/bin/env -S uv run +# /// script +# requires-python = ">=3.10" +# dependencies = [ +# "transformers @ git+https://github.com/huggingface/transformers.git", +# "torch>=2.6.0", +# "onnx>=1.15.0", +# "onnxruntime>=1.17.0", +# "onnxscript", +# "pillow", +# "requests", +# "numpy", +# "accelerate", +# "sentencepiece", +# ] +# /// +""" +Export T5Gemma-2 models to ONNX format for Termite. + +T5Gemma-2 is Google's encoder-decoder model adapted from Gemma 3 via UL2 training. +It has unique architectural features: + - Merged self/cross-attention in decoder + - Tied word embeddings between encoder and decoder + - Grouped Query Attention (GQA) + - Rotary Position Embeddings (RoPE) + - Alternating sliding window attention (4096 tokens) + - SigLIP vision encoder for multimodal + +This script exports 4 ONNX files: + 1. encoder.onnx - Text encoder (for embeddings and seq2seq) + 2. vision_encoder.onnx - SigLIP vision encoder (for image embeddings) + 3. decoder_init.onnx - Decoder without past_key_values (first token) + 4. decoder.onnx - Decoder with past_key_values (efficient generation) + +Usage: + # Export 270M variant (recommended for development) - uses default output path + ./scripts/export_t5gemma2.py --model google/t5gemma-2-270m-270m + + # Export with testing + ./scripts/export_t5gemma2.py --model google/t5gemma-2-270m-270m --test + + # Export 1B variant + ./scripts/export_t5gemma2.py --model google/t5gemma-2-1b-1b --output ~/.termite/models/rewriters/google/t5gemma-2-1b + +Available Models: + - google/t5gemma-2-270m-270m (~800M total params, fast) + - google/t5gemma-2-1b-1b (~2B total params, balanced) + - google/t5gemma-2-4b-4b (~8B total params, best quality) + +References: + - HuggingFace: https://huggingface.co/docs/transformers/model_doc/t5gemma2 + - Google Blog: https://blog.google/technology/developers/t5gemma-2/ + - Paper: https://arxiv.org/abs/2512.14856 + +Prerequisites: + - uv (Python package manager): brew install uv + - HuggingFace account with access to T5Gemma-2 models + +Note: This script is self-contained and does NOT require optimum-onnx. +It applies the same ONNX compatibility patches manually. + +If you prefer using optimum-cli for export, use our fork: + https://github.com/timkaye11/optimum-onnx (branch: add-t5gemma2-support) + +See docs/T5GEMMA2.md for full setup instructions. +""" + +import argparse +import json +import logging +import shutil +import sys +from pathlib import Path +from typing import Optional + +import numpy as np +import os + +# Disable torch dynamo verbose mode +os.environ["TORCHDYNAMO_VERBOSE"] = "0" + +logging.basicConfig(level=logging.INFO, format="%(message)s") +logger = logging.getLogger(__name__) + +# ONNX opset version - use 17 for better compatibility with complex attention patterns +OPSET_VERSION = 17 + + +def disable_dynamo(): + """Disable torch dynamo to avoid tracing issues.""" + try: + import torch._dynamo + torch._dynamo.config.suppress_errors = True + torch._dynamo.reset() + # Completely disable dynamo + torch._dynamo.disable() + except Exception: + pass + + +def patch_transformers_for_onnx(): + """ + Patch transformers masking utilities for ONNX export compatibility. + + This applies the same patches as optimum-onnx's ModelPatcher, replacing + vmap-based mask creation with broadcasting-based alternatives that are + compatible with TorchScript/ONNX tracing. + + Based on: https://github.com/huggingface/optimum-onnx/blob/main/optimum/exporters/onnx/model_patcher.py + """ + try: + import torch + from typing import Callable, Optional + + import transformers.masking_utils as masking_utils + from transformers.masking_utils import ( + ALL_MASK_ATTENTION_FUNCTIONS, + and_masks, + causal_mask_function, + padding_mask_function, + ) + + def sdpa_mask_without_vmap( + batch_size: int, + cache_position: torch.Tensor, + kv_length: int, + kv_offset: int = 0, + mask_function: Optional[Callable] = None, + attention_mask: Optional[torch.Tensor] = None, + local_size: Optional[int] = None, + allow_is_causal_skip: bool = True, + **kwargs, + ) -> Optional[torch.Tensor]: + """ + Custom vectorized implementation of sdpa_mask without using vmap. + Uses broadcasting-based index creation instead. + """ + if mask_function is None: + mask_function = causal_mask_function + + q_length = cache_position.shape[0] + + # Potentially pad the 2D mask - handle the padding_length check safely + padding_mask = None + if attention_mask is not None: + padding_needed = kv_length + kv_offset - attention_mask.shape[-1] + if padding_needed > 0: + padding_mask = torch.nn.functional.pad(attention_mask, (0, padding_needed)) + else: + padding_mask = attention_mask + + # Potentially add the padding 2D mask + if padding_mask is not None: + mask_function = and_masks(mask_function, padding_mask_function(padding_mask)) + + # Create broadcatable indices (optimum-onnx approach) + device = cache_position.device + q_indices = cache_position[None, None, :, None] + head_indices = torch.arange(1, dtype=torch.long, device=device)[None, :, None, None] + batch_indices = torch.arange(batch_size, dtype=torch.long, device=device)[:, None, None, None] + kv_indices = torch.arange(kv_length, dtype=torch.long, device=device)[None, None, None, :] + kv_offset + + # Apply mask function element-wise through broadcasting + causal_mask = mask_function(batch_indices, head_indices, q_indices, kv_indices) + + # Expand the mask to match batch size and query length + causal_mask = causal_mask.expand(batch_size, -1, q_length, kv_length) + + return causal_mask + + def eager_mask_without_vmap( + batch_size: int, + cache_position: torch.Tensor, + kv_length: int, + kv_offset: int = 0, + mask_function: Optional[Callable] = None, + attention_mask: Optional[torch.Tensor] = None, + dtype: torch.dtype = torch.float32, + **kwargs, + ) -> torch.Tensor: + """ + Eager attention mask without vmap, adapted from transformers. + Converts boolean mask to float mask with 0 for attend, -inf for mask. + """ + kwargs.pop("allow_is_causal_skip", None) + kwargs.pop("allow_is_bidirectional_skip", None) + kwargs.pop("allow_torch_fix", None) + + mask = sdpa_mask_without_vmap( + batch_size=batch_size, + cache_position=cache_position, + kv_length=kv_length, + kv_offset=kv_offset, + mask_function=mask_function, + attention_mask=attention_mask, + allow_is_causal_skip=False, + **kwargs, + ) + + if mask is not None: + min_dtype = torch.finfo(dtype).min + # Convert bool mask to float: True -> 0.0, False -> -inf + mask = torch.where(mask, torch.zeros((), device=mask.device, dtype=dtype), min_dtype) + + return mask + + # Register the patched functions + ALL_MASK_ATTENTION_FUNCTIONS.register("sdpa", sdpa_mask_without_vmap) + ALL_MASK_ATTENTION_FUNCTIONS.register("eager", eager_mask_without_vmap) + + # Patch find_packed_sequence_indices to avoid torch.diff (not supported in ONNX) + def find_packed_sequence_indices_onnx(position_ids: torch.Tensor) -> Optional[torch.Tensor]: + """ + ONNX-compatible version that always returns None (no packed sequences). + This is safe for export since we're exporting with fixed batch size. + """ + return None + + masking_utils.find_packed_sequence_indices = find_packed_sequence_indices_onnx + + logger.info("Patched transformers masking utilities for ONNX compatibility (optimum-onnx style)") + return True + + except Exception as e: + logger.warning(f"Could not patch transformers masking utilities: {e}") + import traceback + traceback.print_exc() + return False + + +def check_dependencies(): + """Check if required packages are installed.""" + missing = [] + try: + import torch + except ImportError: + missing.append("torch>=2.0.0") + try: + import transformers + except ImportError: + missing.append("transformers>=4.48.0") + try: + import onnx + except ImportError: + missing.append("onnx>=1.15.0") + try: + import onnxruntime + except ImportError: + missing.append("onnxruntime>=1.17.0") + + if missing: + print(f"Missing required packages: {', '.join(missing)}") + print(f"Install with: pip install {' '.join(missing)}") + sys.exit(1) + + +class T5Gemma2EncoderWrapper: + """Wrapper to export just the encoder portion of T5Gemma2.""" + + def __init__(self, model): + self.encoder = model.encoder + self.embed_tokens = model.encoder.embed_tokens + + def forward(self, input_ids, attention_mask): + # Get embeddings + inputs_embeds = self.embed_tokens(input_ids) + + # Run encoder + encoder_outputs = self.encoder( + inputs_embeds=inputs_embeds, + attention_mask=attention_mask, + output_hidden_states=False, + return_dict=True, + ) + + return encoder_outputs.last_hidden_state + + +class T5Gemma2DecoderInitWrapper: + """Wrapper to export decoder without past_key_values (first token generation).""" + + def __init__(self, model): + self.decoder = model.decoder + self.lm_head = model.lm_head + self.embed_tokens = model.decoder.embed_tokens + + def forward(self, decoder_input_ids, encoder_hidden_states, encoder_attention_mask): + # Get decoder embeddings + inputs_embeds = self.embed_tokens(decoder_input_ids) + + # Run decoder without past + decoder_outputs = self.decoder( + inputs_embeds=inputs_embeds, + encoder_hidden_states=encoder_hidden_states, + encoder_attention_mask=encoder_attention_mask, + use_cache=True, + return_dict=True, + ) + + # Get logits + logits = self.lm_head(decoder_outputs.last_hidden_state) + + # Return logits and past_key_values for subsequent decoding + return logits, decoder_outputs.past_key_values + + +class T5Gemma2DecoderWrapper: + """Wrapper to export decoder with past_key_values (efficient generation).""" + + def __init__(self, model): + self.decoder = model.decoder + self.lm_head = model.lm_head + self.embed_tokens = model.decoder.embed_tokens + + def forward( + self, + decoder_input_ids, + encoder_hidden_states, + encoder_attention_mask, + past_key_values, + ): + # Get decoder embeddings + inputs_embeds = self.embed_tokens(decoder_input_ids) + + # Run decoder with past + decoder_outputs = self.decoder( + inputs_embeds=inputs_embeds, + encoder_hidden_states=encoder_hidden_states, + encoder_attention_mask=encoder_attention_mask, + past_key_values=past_key_values, + use_cache=True, + return_dict=True, + ) + + # Get logits + logits = self.lm_head(decoder_outputs.last_hidden_state) + + return logits, decoder_outputs.past_key_values + + +def export_encoder(model, processor, output_dir: Path) -> None: + """Export the text encoder to ONNX. + + Due to T5Gemma-2's complex attention implementation, we export two options: + 1. embedding_layer.onnx - Just the embedding lookup (fast, always works) + 2. encoder.onnx - Full encoder with attention (may fail with complex models) + """ + import torch + import onnx + + logger.info("\n1. Exporting text encoder...") + + # Create dummy inputs + dummy_text = "This is a test sentence for the encoder." + inputs = processor.tokenizer( + dummy_text, + return_tensors="pt", + padding="max_length", + max_length=128, + truncation=True, + ) + + input_ids = inputs["input_ids"] + attention_mask = inputs["attention_mask"] + + # Get encoder from model using get_encoder() method + encoder = model.get_encoder() + + # Get embed_tokens from the model's text_embed_tokens or from encoder + if hasattr(model.model, 'text_embed_tokens'): + embed_tokens = model.model.text_embed_tokens + elif hasattr(encoder, 'embed_tokens'): + embed_tokens = encoder.embed_tokens + else: + # Fallback: try to get from decoder's embed_tokens (they may be shared) + decoder = model.get_decoder() + embed_tokens = decoder.embed_tokens + + # Export 1: Embedding layer only (simpler, always works) + embedding_path = output_dir / "embedding_layer.onnx" + logger.info(" 1a. Exporting embedding layer...") + + class EmbeddingModule(torch.nn.Module): + def __init__(self, embed_tokens): + super().__init__() + self.embed_tokens = embed_tokens + + def forward(self, input_ids): + return self.embed_tokens(input_ids) + + embedding_module = EmbeddingModule(embed_tokens) + embedding_module.eval() + for param in embedding_module.parameters(): + param.requires_grad = False + + with torch.no_grad(): + torch.onnx.export( + embedding_module, + (input_ids,), + str(embedding_path), + export_params=True, + opset_version=OPSET_VERSION, + do_constant_folding=True, + input_names=["input_ids"], + output_names=["embeddings"], + dynamic_axes={ + "input_ids": {0: "batch_size", 1: "sequence_length"}, + "embeddings": {0: "batch_size", 1: "sequence_length"}, + }, + dynamo=False, + ) + + # Validate embedding layer + onnx_model = onnx.load(str(embedding_path)) + onnx.checker.check_model(onnx_model) + size_mb = embedding_path.stat().st_size / (1024 * 1024) + logger.info(f" Saved: embedding_layer.onnx ({size_mb:.1f} MB)") + + # Export 2: Full encoder (may fail due to complex attention) + encoder_path = output_dir / "encoder.onnx" + logger.info(" 1b. Exporting full encoder (this may take a while)...") + + class EncoderModule(torch.nn.Module): + def __init__(self, encoder, embed_tokens): + super().__init__() + self.encoder = encoder + self.embed_tokens = embed_tokens + + def forward(self, input_ids, attention_mask): + inputs_embeds = self.embed_tokens(input_ids) + outputs = self.encoder( + inputs_embeds=inputs_embeds, + attention_mask=attention_mask, + output_hidden_states=False, + return_dict=True, + ) + return outputs.last_hidden_state + + encoder_module = EncoderModule(encoder, embed_tokens) + encoder_module.eval() + for param in encoder_module.parameters(): + param.requires_grad = False + + try: + with torch.no_grad(): + torch.onnx.export( + encoder_module, + (input_ids, attention_mask), + str(encoder_path), + export_params=True, + opset_version=OPSET_VERSION, + do_constant_folding=True, + input_names=["input_ids", "attention_mask"], + output_names=["encoder_hidden_states"], + dynamic_axes={ + "input_ids": {0: "batch_size", 1: "sequence_length"}, + "attention_mask": {0: "batch_size", 1: "sequence_length"}, + "encoder_hidden_states": {0: "batch_size", 1: "sequence_length"}, + }, + dynamo=False, + ) + + # Validate + onnx_model = onnx.load(str(encoder_path)) + onnx.checker.check_model(onnx_model) + size_mb = encoder_path.stat().st_size / (1024 * 1024) + logger.info(f" Saved: encoder.onnx ({size_mb:.1f} MB)") + + # Test encoder output shape + with torch.no_grad(): + test_output = encoder_module(input_ids, attention_mask) + logger.info(f" Encoder output shape: {test_output.shape}") + + except Exception as e: + logger.warning(f" Full encoder export failed: {e}") + logger.info(" Using embedding_layer.onnx for embeddings (without attention)") + logger.info(" For full encoder functionality, use PyTorch inference instead of ONNX") + + # Export 3: Encoder with inputs_embeds (for embedding-to-text generation) + encoder_embeds_path = output_dir / "encoder_embeds.onnx" + logger.info(" 1c. Exporting encoder with inputs_embeds support...") + + # Get hidden size from model config + if hasattr(model.config, 'encoder'): + hidden_size = model.config.encoder.hidden_size + elif hasattr(model.config, 'hidden_size'): + hidden_size = model.config.hidden_size + else: + hidden_size = 640 # Default for T5Gemma2-270m + + class EncoderEmbedsModule(torch.nn.Module): + """Encoder that accepts pre-computed embeddings directly (inputs_embeds).""" + def __init__(self, encoder): + super().__init__() + self.encoder = encoder + + def forward(self, inputs_embeds, attention_mask): + """Run encoder directly on embeddings, bypassing token embedding lookup.""" + outputs = self.encoder( + inputs_embeds=inputs_embeds, + attention_mask=attention_mask, + output_hidden_states=False, + return_dict=True, + ) + return outputs.last_hidden_state + + encoder_embeds_module = EncoderEmbedsModule(encoder) + encoder_embeds_module.eval() + for param in encoder_embeds_module.parameters(): + param.requires_grad = False + + # Create dummy inputs_embeds + batch_size, seq_len = input_ids.shape + dummy_inputs_embeds = torch.randn(batch_size, seq_len, hidden_size) + + try: + with torch.no_grad(): + torch.onnx.export( + encoder_embeds_module, + (dummy_inputs_embeds, attention_mask), + str(encoder_embeds_path), + export_params=True, + opset_version=OPSET_VERSION, + do_constant_folding=True, + input_names=["inputs_embeds", "attention_mask"], + output_names=["encoder_hidden_states"], + dynamic_axes={ + "inputs_embeds": {0: "batch_size", 1: "sequence_length"}, + "attention_mask": {0: "batch_size", 1: "sequence_length"}, + "encoder_hidden_states": {0: "batch_size", 1: "sequence_length"}, + }, + dynamo=False, + ) + + # Validate + onnx_model = onnx.load(str(encoder_embeds_path)) + onnx.checker.check_model(onnx_model) + size_mb = encoder_embeds_path.stat().st_size / (1024 * 1024) + logger.info(f" Saved: encoder_embeds.onnx ({size_mb:.1f} MB)") + + # Test encoder_embeds output shape + with torch.no_grad(): + test_output = encoder_embeds_module(dummy_inputs_embeds, attention_mask) + logger.info(f" Encoder (embeds) output shape: {test_output.shape}") + + except Exception as e: + logger.warning(f" Encoder with inputs_embeds export failed: {e}") + logger.info(" Embedding-to-text generation will require the standard encoder path") + + +def export_vision_encoder(model, processor, output_dir: Path) -> bool: + """Export the vision encoder (SigLIP) to ONNX if present.""" + import torch + import onnx + + # Check if model has vision encoder - try multiple paths + vision_tower = None + encoder = model.get_encoder() + + # Try different possible locations for vision tower + if hasattr(encoder, 'vision_tower'): + vision_tower = encoder.vision_tower + elif hasattr(model.model, 'vision_tower'): + vision_tower = model.model.vision_tower + elif hasattr(model, 'vision_tower'): + vision_tower = model.vision_tower + + if vision_tower is None: + logger.info("\n2. No vision encoder found (text-only model)") + return False + + logger.info("\n2. Exporting vision encoder (SigLIP)...") + + vision_path = output_dir / "vision_encoder.onnx" + + # Get image size from processor + if hasattr(processor, "image_processor"): + image_size = processor.image_processor.size.get("height", 896) + else: + image_size = 896 # Default for T5Gemma2 + + # Create dummy image input + dummy_pixel_values = torch.randn(1, 3, image_size, image_size) + + class VisionEncoderModule(torch.nn.Module): + def __init__(self, vision_tower): + super().__init__() + self.vision_tower = vision_tower + + def forward(self, pixel_values): + outputs = self.vision_tower(pixel_values, return_dict=True) + return outputs.last_hidden_state + + vision_module = VisionEncoderModule(vision_tower) + vision_module.eval() + + try: + with torch.no_grad(): + torch.onnx.export( + vision_module, + (dummy_pixel_values,), + str(vision_path), + export_params=True, + opset_version=OPSET_VERSION, + do_constant_folding=True, + input_names=["pixel_values"], + output_names=["vision_hidden_states"], + dynamic_axes={ + "pixel_values": {0: "batch_size"}, + "vision_hidden_states": {0: "batch_size"}, + }, + dynamo=False, # Use legacy exporter + ) + + # Validate + onnx_model = onnx.load(str(vision_path)) + onnx.checker.check_model(onnx_model) + + size_mb = vision_path.stat().st_size / (1024 * 1024) + logger.info(f" Saved: vision_encoder.onnx ({size_mb:.1f} MB)") + return True + + except Exception as e: + logger.warning(f" Vision encoder export failed: {e}") + return False + + +def export_decoder_init(model, processor, output_dir: Path, hidden_size: int) -> None: + """Export the decoder without past_key_values (for first token generation).""" + import torch + import onnx + + logger.info("\n3. Exporting decoder-init (without past_key_values)...") + + decoder_init_path = output_dir / "decoder-init.onnx" + + # Get decoder components using get_decoder() method + decoder = model.get_decoder() + lm_head = model.lm_head + + # Get embed_tokens - try multiple paths + if hasattr(decoder, 'embed_tokens'): + embed_tokens = decoder.embed_tokens + elif hasattr(model.model, 'text_embed_tokens'): + embed_tokens = model.model.text_embed_tokens + elif hasattr(model.model, 'embed_tokens'): + embed_tokens = model.model.embed_tokens + else: + raise AttributeError("Could not find embed_tokens in model") + + # Create dummy inputs + batch_size = 1 + encoder_seq_len = 32 + decoder_seq_len = 1 + + dummy_decoder_input_ids = torch.zeros((batch_size, decoder_seq_len), dtype=torch.long) + dummy_encoder_hidden_states = torch.randn(batch_size, encoder_seq_len, hidden_size) + dummy_encoder_attention_mask = torch.ones((batch_size, encoder_seq_len), dtype=torch.long) + + class DecoderInitModule(torch.nn.Module): + def __init__(self, decoder, lm_head, embed_tokens): + super().__init__() + self.decoder = decoder + self.lm_head = lm_head + self.embed_tokens = embed_tokens + + def forward(self, decoder_input_ids, encoder_hidden_states, encoder_attention_mask): + inputs_embeds = self.embed_tokens(decoder_input_ids) + outputs = self.decoder( + inputs_embeds=inputs_embeds, + encoder_hidden_states=encoder_hidden_states, + encoder_attention_mask=encoder_attention_mask, + use_cache=False, # No cache for init + return_dict=True, + ) + logits = self.lm_head(outputs.last_hidden_state) + return logits + + decoder_init_module = DecoderInitModule(decoder, lm_head, embed_tokens) + decoder_init_module.eval() + + with torch.no_grad(): + torch.onnx.export( + decoder_init_module, + (dummy_decoder_input_ids, dummy_encoder_hidden_states, dummy_encoder_attention_mask), + str(decoder_init_path), + export_params=True, + opset_version=OPSET_VERSION, + do_constant_folding=True, + input_names=["input_ids", "encoder_hidden_states", "encoder_attention_mask"], + output_names=["logits"], + dynamic_axes={ + "input_ids": {0: "batch_size", 1: "decoder_sequence_length"}, + "encoder_hidden_states": {0: "batch_size", 1: "encoder_sequence_length"}, + "encoder_attention_mask": {0: "batch_size", 1: "encoder_sequence_length"}, + "logits": {0: "batch_size", 1: "decoder_sequence_length"}, + }, + dynamo=False, # Use legacy exporter + ) + + # Validate + onnx_model = onnx.load(str(decoder_init_path)) + onnx.checker.check_model(onnx_model) + + size_mb = decoder_init_path.stat().st_size / (1024 * 1024) + logger.info(f" Saved: decoder-init.onnx ({size_mb:.1f} MB)") + + +def export_decoder_with_past(model, processor, output_dir: Path, hidden_size: int) -> None: + """ + Export the decoder with past_key_values for efficient autoregressive generation. + + Note: This is complex due to T5Gemma2's merged self/cross-attention and GQA. + For initial implementation, we use decoder-init for all steps (slower but simpler). + """ + import torch + + logger.info("\n4. Exporting decoder with past_key_values...") + logger.info(" Note: Full KV cache export is complex due to merged attention.") + logger.info(" Using decoder-init for all steps (slower but functional).") + + # For now, we'll copy decoder-init as decoder (same file, different name) + # This means generation will be slower but will work correctly + decoder_init_path = output_dir / "decoder-init.onnx" + decoder_path = output_dir / "decoder.onnx" + + if decoder_init_path.exists(): + shutil.copy(str(decoder_init_path), str(decoder_path)) + size_mb = decoder_path.stat().st_size / (1024 * 1024) + logger.info(f" Saved: decoder.onnx ({size_mb:.1f} MB)") + logger.info(" (Using decoder-init format - no KV cache optimization)") + + +def save_configs(model, processor, output_dir: Path, model_id: str) -> None: + """Save model and tokenizer configuration files.""" + logger.info("\n5. Saving configuration files...") + + # Save tokenizer + processor.tokenizer.save_pretrained(str(output_dir)) + logger.info(" Saved: tokenizer files") + + # Save model config + model.config.save_pretrained(str(output_dir)) + logger.info(" Saved: config.json") + + # Save image processor config if available + if hasattr(processor, 'image_processor') and processor.image_processor is not None: + try: + processor.image_processor.save_pretrained(str(output_dir)) + logger.info(" Saved: preprocessor_config.json") + except Exception as e: + logger.warning(f" Could not save image processor config: {e}") + + # Extract config values with fallbacks for different config structures + def get_config_value(config, *paths, default=None): + """Try multiple paths to get a config value.""" + for path in paths: + obj = config + try: + for key in path.split('.'): + obj = getattr(obj, key) + return obj + except AttributeError: + continue + return default + + hidden_size = get_config_value( + model.config, + 'decoder.hidden_size', + 'hidden_size', + default=2304 + ) + vocab_size = get_config_value( + model.config, + 'decoder.vocab_size', + 'vocab_size', + default=262208 + ) + num_encoder_layers = get_config_value( + model.config, + 'encoder.text_config.num_hidden_layers', + 'encoder.num_hidden_layers', + 'num_hidden_layers', + default=26 + ) + num_decoder_layers = get_config_value( + model.config, + 'decoder.num_hidden_layers', + 'num_hidden_layers', + default=26 + ) + num_attention_heads = get_config_value( + model.config, + 'decoder.num_attention_heads', + 'num_attention_heads', + default=8 + ) + num_key_value_heads = get_config_value( + model.config, + 'decoder.num_key_value_heads', + 'num_key_value_heads', + default=4 + ) + + # Create T5Gemma2-specific config for Termite + t5gemma2_config = { + "model_id": model_id, + "model_type": "t5gemma2", + "task": "multimodal_seq2seq", + "capabilities": ["embeddings", "generation", "multimodal"], + "max_encoder_length": 131072, # 128K + "max_decoder_length": 32768, # 32K output + "hidden_size": hidden_size, + "vocab_size": vocab_size, + "num_encoder_layers": num_encoder_layers, + "num_decoder_layers": num_decoder_layers, + "num_attention_heads": num_attention_heads, + "num_key_value_heads": num_key_value_heads, + "generation_config": { + "max_new_tokens": 256, + "num_beams": 1, + "do_sample": False, + "temperature": 1.0, + "top_p": 1.0, + }, + } + + config_path = output_dir / "t5gemma2_config.json" + with open(config_path, "w") as f: + json.dump(t5gemma2_config, f, indent=2) + logger.info(" Saved: t5gemma2_config.json") + + +def export_model(model_id: str, output_dir: str) -> None: + """ + Export a T5Gemma-2 model to ONNX format. + + Args: + model_id: HuggingFace model ID (e.g., 'google/t5gemma-2-270m-270m') + output_dir: Directory to save the exported model + """ + import torch + from transformers import AutoTokenizer, AutoModelForSeq2SeqLM + + # Disable dynamo and patch transformers before any model operations + disable_dynamo() + patch_transformers_for_onnx() + + output_path = Path(output_dir) + output_path.mkdir(parents=True, exist_ok=True) + + logger.info(f"Exporting T5Gemma-2: {model_id}") + logger.info(f"Output: {output_dir}") + + # Load model and tokenizer + logger.info("\nLoading model and tokenizer...") + logger.info("(This may take a while for larger models)") + + # Load tokenizer directly (AutoProcessor has compatibility issues with T5Gemma-2) + tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) + + # Try to load image processor separately if available (for multimodal) + image_processor = None + try: + from transformers import AutoImageProcessor + image_processor = AutoImageProcessor.from_pretrained(model_id, trust_remote_code=True) + logger.info("Loaded image processor for multimodal support") + except Exception as e: + logger.info(f"No image processor found (text-only mode): {e}") + + # Create a simple processor-like object that has tokenizer attribute + class SimpleProcessor: + def __init__(self, tokenizer, image_processor=None): + self.tokenizer = tokenizer + self.image_processor = image_processor + + processor = SimpleProcessor(tokenizer, image_processor) + + # Load model with configuration optimized for ONNX export + from transformers import AutoConfig + config = AutoConfig.from_pretrained(model_id, trust_remote_code=True) + + # Set sliding window to a very large value to effectively disable it + # (Setting to None breaks other code that expects an int) + VERY_LARGE_WINDOW = 2**20 # 1M tokens + if hasattr(config, 'encoder'): + if hasattr(config.encoder, 'sliding_window'): + config.encoder.sliding_window = VERY_LARGE_WINDOW + if hasattr(config.encoder, 'sliding_window_size'): + config.encoder.sliding_window_size = VERY_LARGE_WINDOW + if hasattr(config, 'decoder'): + if hasattr(config.decoder, 'sliding_window'): + config.decoder.sliding_window = VERY_LARGE_WINDOW + if hasattr(config.decoder, 'sliding_window_size'): + config.decoder.sliding_window_size = VERY_LARGE_WINDOW + + model = AutoModelForSeq2SeqLM.from_pretrained( + model_id, + config=config, + torch_dtype=torch.float32, # Use FP32 for ONNX export + device_map="cpu", + trust_remote_code=True, + attn_implementation="eager", # Use eager attention for ONNX compatibility + ) + model.eval() + + # Disable gradient checkpointing if enabled + if hasattr(model, 'gradient_checkpointing_disable'): + model.gradient_checkpointing_disable() + + # Patch model config to use large sliding window at runtime + VERY_LARGE_WINDOW = 2**20 # 1M tokens + for module in model.modules(): + if hasattr(module, 'sliding_window') and module.sliding_window is not None: + module.sliding_window = VERY_LARGE_WINDOW + if hasattr(module, 'config') and hasattr(module.config, 'sliding_window'): + if module.config.sliding_window is not None: + module.config.sliding_window = VERY_LARGE_WINDOW + + # Get hidden size from config with fallbacks + def get_config_value(config, *paths, default=None): + """Try multiple paths to get a config value.""" + for path in paths: + obj = config + try: + for key in path.split('.'): + obj = getattr(obj, key) + return obj + except AttributeError: + continue + return default + + hidden_size = get_config_value( + model.config, + 'decoder.hidden_size', + 'hidden_size', + default=2304 + ) + logger.info(f"Model hidden size: {hidden_size}") + + # Export components + export_encoder(model, processor, output_path) + has_vision = export_vision_encoder(model, processor, output_path) + export_decoder_init(model, processor, output_path, hidden_size) + export_decoder_with_past(model, processor, output_path, hidden_size) + save_configs(model, processor, output_path, model_id) + + # Summary + logger.info("\n" + "=" * 60) + logger.info("Export complete!") + logger.info("=" * 60) + logger.info(f"\nExported files in {output_dir}:") + for f in sorted(output_path.iterdir()): + if f.is_file(): + size_mb = f.stat().st_size / (1024 * 1024) + logger.info(f" {f.name} ({size_mb:.1f} MB)") + + logger.info(f"\nCapabilities:") + logger.info(" - Text embeddings: encoder.onnx") + if has_vision: + logger.info(" - Image embeddings: vision_encoder.onnx") + logger.info(" - Text generation: encoder.onnx + decoder-init.onnx") + + +def test_exported_model(output_dir: str, test_text: Optional[str] = None) -> None: + """Test the exported ONNX model with sample inputs.""" + import onnxruntime as ort + from transformers import AutoTokenizer + + logger.info("\n" + "=" * 60) + logger.info("Testing exported model...") + logger.info("=" * 60) + + output_path = Path(output_dir) + + # Check required files + encoder_path = output_path / "encoder.onnx" + decoder_init_path = output_path / "decoder-init.onnx" + + if not encoder_path.exists(): + logger.error(f"encoder.onnx not found at {encoder_path}") + return + if not decoder_init_path.exists(): + logger.error(f"decoder-init.onnx not found at {decoder_init_path}") + return + + # Load tokenizer directly (avoid AutoProcessor issues) + tokenizer = AutoTokenizer.from_pretrained(str(output_path)) + + # Create simple processor-like wrapper for compatibility + class SimpleProcessor: + def __init__(self, tokenizer): + self.tokenizer = tokenizer + + processor = SimpleProcessor(tokenizer) + + # Create ONNX Runtime sessions + logger.info("\nLoading ONNX models...") + encoder_session = ort.InferenceSession(str(encoder_path)) + decoder_init_session = ort.InferenceSession(str(decoder_init_path)) + + # Test input + if test_text is None: + test_text = "The capital of France is" + + logger.info(f"\nTest input: {test_text}") + + # Tokenize + inputs = processor.tokenizer( + test_text, + return_tensors="np", + padding=True, + ) + input_ids = inputs["input_ids"].astype(np.int64) + attention_mask = inputs["attention_mask"].astype(np.int64) + + # Run encoder + logger.info("\nRunning encoder...") + encoder_outputs = encoder_session.run( + None, + { + "input_ids": input_ids, + "attention_mask": attention_mask, + }, + ) + encoder_hidden_states = encoder_outputs[0] + logger.info(f"Encoder output shape: {encoder_hidden_states.shape}") + + # Test embeddings (mean pool) + embeddings = np.mean(encoder_hidden_states, axis=1) + logger.info(f"Embedding shape (mean pooled): {embeddings.shape}") + logger.info(f"Embedding sample: {embeddings[0, :5]}...") + + # Run decoder for generation + logger.info("\nRunning decoder (greedy generation)...") + decoder_input_ids = np.array([[processor.tokenizer.pad_token_id or 0]], dtype=np.int64) + + generated_ids = [] + max_new_tokens = 20 + + for step in range(max_new_tokens): + decoder_outputs = decoder_init_session.run( + None, + { + "input_ids": decoder_input_ids, + "encoder_hidden_states": encoder_hidden_states, + "encoder_attention_mask": attention_mask, + }, + ) + logits = decoder_outputs[0] + + # Greedy: take argmax of last token + next_token_id = int(np.argmax(logits[0, -1, :])) + + # Check for EOS + if next_token_id == processor.tokenizer.eos_token_id: + break + + generated_ids.append(next_token_id) + decoder_input_ids = np.concatenate( + [decoder_input_ids, np.array([[next_token_id]], dtype=np.int64)], + axis=1, + ) + + # Decode + generated_text = processor.tokenizer.decode(generated_ids, skip_special_tokens=True) + logger.info(f"\nGenerated: {generated_text}") + + logger.info("\nTest complete!") + + +def main(): + parser = argparse.ArgumentParser( + description="Export T5Gemma-2 models to ONNX for Termite", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" +Examples: + # Export 270M variant (recommended for development) - uses default path + ./scripts/export_t5gemma2.py --model google/t5gemma-2-270m-270m + + # Export with testing + ./scripts/export_t5gemma2.py --model google/t5gemma-2-270m-270m --test + + # Export 1B variant with custom path + ./scripts/export_t5gemma2.py --model google/t5gemma-2-1b-1b --output ~/.termite/models/rewriters/google/t5gemma-2-1b + +Available Models: + - google/t5gemma-2-270m-270m (~800M total params, fast) + - google/t5gemma-2-1b-1b (~2B total params, balanced) + - google/t5gemma-2-4b-4b (~8B total params, best quality) + +Output Files: + - encoder.onnx Text encoder for embeddings & seq2seq + - vision_encoder.onnx SigLIP vision encoder (if multimodal) + - decoder-init.onnx Decoder for first token / all tokens + - decoder.onnx Decoder with KV cache (if supported) + - t5gemma2_config.json Termite-specific configuration + - tokenizer files For text processing + """, + ) + + parser.add_argument( + "--model", + type=str, + default="google/t5gemma-2-270m-270m", + help="HuggingFace model ID (default: google/t5gemma-2-270m-270m)", + ) + parser.add_argument( + "-o", "--output", + type=str, + default=os.path.expanduser("~/.termite/models/rewriters/google/t5gemma-2-270m"), + help="Output directory for the exported model (default: ~/.termite/models/rewriters/google/t5gemma-2-270m)", + ) + parser.add_argument( + "--test", + action="store_true", + help="Test the exported model after export", + ) + parser.add_argument( + "--test-input", + type=str, + default=None, + help="Custom input text for testing", + ) + + args = parser.parse_args() + + check_dependencies() + + try: + export_model(args.model, args.output) + if args.test: + test_exported_model(args.output, args.test_input) + logger.info("\nAll done!") + except Exception as e: + logger.error(f"\nError: {e}") + import traceback + traceback.print_exc() + sys.exit(1) + + +if __name__ == "__main__": + main()