LangExtract FastChat Provider

A provider plugin for LangExtract that integrates with FastChat to support various large language models served via FastChat's OpenAI-compatible API.

Installation

Install from PyPI:

pip install langextract-fastchat

Or install from source for development:

git clone https://github.com/your-username/langextract-fastchat
cd langextract-fastchat
pip install -e .

Supported Model IDs

This provider handles model IDs that start with fastchat:.

• FastChat models: fastchat:<model_name>

Where <model_name> is the name of the model served by your FastChat instance (e.g., vicuna-7b-v1.5).

You can also specify the FastChat API server's base URL directly in the model_id:

• fastchat:http://your-fastchat-host:port/v1

Usage

Prerequisites

Ensure you have a FastChat controller and model worker running, exposing an OpenAI-compatible API. For example:

# Launch the controller
python3 -m fastchat.serve.controller

# Launch the model worker (replace with your desired model)
python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5

# Launch the OpenAI-compatible API server
python3 -m fastchat.serve.openai_api_server

By default, the API server runs on http://localhost:8000/v1.

Basic Usage

import langextract as lx

# Create model configuration
config = lx.factory.ModelConfig(
    model_id="fastchat:vicuna-7b-v1.5",  # Or "fastchat:http://localhost:8000/v1"
    provider="FastChatLanguageModel",
    provider_kwargs=dict(
        temperature=0.7,
        max_tokens=1024,
        # Additional parameters for the OpenAI client or completion calls
        # For example, if your FastChat server requires an API key:
        # api_key="YOUR_FASTCHAT_API_KEY",
        # base_url="http://localhost:8000/v1", # Only if not specified in model_id
    ),
)
model = lx.factory.create_model(config)

# Extract entities
result = lx.extract(
    text_or_documents="The quick brown fox jumps over the lazy dog.",
    model=model,
    prompt_description="Extract animals and their actions.",
    examples=[
        lx.data.ExampleData(
            text="A cat sat on the mat.",
            extractions=[
                lx.data.Extraction(extraction_class="animal", extraction_text="cat", attributes={"action": "sat"}),
                lx.data.Extraction(extraction_class="object", extraction_text="mat"),
            ],
        ),
    ],
)

print("✅ Extraction successful!")
print(f"Results: {result}")

Configuration Parameters

Provider Arguments (provider_kwargs)

Parameter	Type	Default	Description
api_key	str	"EMPTY"	API key for the FastChat server. Use "EMPTY" if no key is required.
base_url	str	"http://localhost:8000/v1"	Base URL of the FastChat OpenAI-compatible API server. Overridden if specified in model_id.
temperature	float	None	Sampling temperature (0.0-2.0). If None, FastChat's default will be used.
max_tokens	int	None	Maximum tokens to generate. If None, FastChat's default will be used.
top_p	float	None	Top-p sampling parameter. If None, FastChat's default will be used.
max_workers	int	10	Maximum number of parallel API calls for batch processing.

Additional parameters supported by OpenAI's chat.completions.create method can also be passed via provider_kwargs.

Development

1. Install in development mode: pip install -e .
2. Run tests: pytest (after installing pytest in your dev environment)
3. Build package: python -m build
4. Publish to PyPI: twine upload dist/*

Requirements

• langextract
• openai (for API interaction)
• A running FastChat instance (controller, model worker, and OpenAI API server)

License

Apache License 2.0