Semantic Frame Generation using RAG and GenLLMs

Automatically extract semantic frames from text using Retrieval-Augmented Generation (RAG) and generative language models. This project generates frame annotations compatible with FrameNet (Baker, Fillmore, and Lowe, 1998), identifying frame-evoking targets and their frame elements in arbitrary text.

How it works

Highlights

• Frame Extraction: Automatically identify semantic frames and their constituent frame elements in text using FrameNet definitions
• RAG-based Retrieval: Uses FAISS vector embeddings to retrieve relevant candidate frames before generation, improving accuracy and efficiency
• LLM-based Annotation: Leverages GenAI models with LangChain for a model agnostic implementation and structured output parsing for precise frame annotations
• Comprehensive Evaluation: Includes detailed evaluation metrics (precision, recall, F1) at both frame and frame element levels for annotated texts from FrameNet.
• FrameNet Integration: Works with complete FrameNet frame definitions including frame elements, lexical units, and examples

RAG pipeline

1. Embedding: FrameNet frames are embedded using BAAI/bge-m3 and stored in FAISS for efficient retrieval.
2. Retrieval: Input text is embedded, and top-k similar frames are retrieved from FAISS indexing based on Euclidian distance.
3. Prompting: LLM receives task instructions (which may contain annotated examples), retrieved frames, and input text.
4. Generation: LLM outputs structured frame annotations in JSON format (validated via Pydantic schema).

Frame Representation

Each frame embedding encodes the following information (taken from FrameNet):

• Name & Definition: frame name and its definition
• Lexical Units: words that typically evoke the frame
• Core Elements: Essential frame elements (arguments that must be implicitly or explicitly present when a frame is evoked).
• Optional: Example sentences and peripheral frame elements

Getting Started

Prerequisites

• Python 3.11+
• The LLM's API account with credits (or a Nebula's API account)
• FrameNet data files (version 1.7, XML formatted, upon request at FrameNet), or download the already-generated frame embeddings from our repository.

Installation

1. Clone the repository and set up environment

   git clone <this-repo-url>
   cd text2frames
   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate
   

2. Install dependencies

   pip install -r requirements.txt
   

3. Set up environment variables

   Create a .env file in the project root with this variable:

   OPENAI_API_KEY=your_api_key_here
   

Usage

1. Generate frames (and their core frame elements) from input text (with zero-shot prompting, i.e. no example annotated texts provided)

   Generate frames from text in the command line:

   python generate_frames.py --input_text "Apple announced a new iPhone model."

   Generate frames from a text file (with zero-shot prompting, i.e. no example annotated texts provided):

   python generate_frames.py --input_filepath data/inputs/example.txt

   Generate frames from a CSV file:

   python generate_frames.py \
   --input_filepath data/inputs/provo_texts.csv \
   --text_column_name "text" \
   --sep ","

   If frame embeddings have not been generated (or downloaded), generate them by specifying the directory path containing the FrameNet raw XML files:

   python generate_frames.py --frame_dir data/framenet/frames --input_text "Apple announced a new iPhone model."

   To run LLM through Nebula, use the nebula flag and specify the LLM and the model provider.

   python generate_frames.py --input_text "Apple announced a new iPhone model." --nebula --model_name FAST.gpt-oss:120b --model_provider openai

   Output format:

   {
     "responses": [
       {
         "input_text": "Apple announced a new iPhone model.",
         "frames": [
           {
             "name": "Statement",
             "target": "announced",
             "start": 6,
             "end": 14,
             "frame_elements": [
               {
                 "name": "Speaker",
                 "span": "Apple",
                 "start": 0,
                 "end": 5
               },
               {
                 "name": "Message",
                 "span": "a new iPhone model",
                 "start": 19,
                 "end": 36
               }
             ]
           }
         ],
         "candidate_frames": [...]
       }
     ]
   }

2. Evaluate model on FrameNet annotations

   python generate_frames.py \
   --evaluate

   To evaluate pre-generated outputs, specify the file path containing the generated frames for evaluation:

   python generate_frames.py \
   --evaluate \
   --generated_frames_filepath data/output/output_4_eval.json

   To change the matching type from exact to semantic or graph:

       python generate_frames.py \
       --evaluate \
       --match_type semantic # choose between exact, semantic, and graph. Default: exact

   Output includes three evaluation files:

   • eval_scores_<model_name>.json: Contains precision, recall, and F1 scores at both frame and frame element levels for each evaluated text.
   • error_eval_<model_name>.json: Contains more details for each evaluated text, including:
     • correct items: frames or frame elements correctly identified
     • incorrect items: frames or frame elements generated by the model but not matching any annotated frame
     • missed items: frames or frame elements annotated in FrameNet but not matching any generated frame
   • relations_<model_name>.json: Contains the FrameNet relations between predicted and annotated frames.

Command Line Arguments

Frame Configuration

• --frame_dir: Directory containing the frames from FrameNet. Default is 'data/framenet/frames'. If frame embeddings do not exist yet in the current directory, please provide the path to the directory with FrameNet frames (in xml, or json if already parsed) so that the embeddings can be created.
• --frame_output_filepath: Path to save parsed frames. If none is given, the system tries to save parsed frames in the same directory as the raw xml frames.
• --annotated_texts_dir: Directory containing the annotated texts from FrameNet. Default is 'data/framenet/annotated_texts'. Needed for few-shot mode, and evaluation against annotations.

Input Configuration

• --input_text: Input text to generate frames from.
• --input_filepath: File path to input text file.
• --text_column_name: Column name containing text in input CSV file (string, required if input is CSV).
• --sep: Separator used in input CSV file (default: ",").

Output Configuration

• --output_filepath: Path to save output of frame extraction. If none is given, the system tries to save response in current working directory.
• --output_eval_dir: Directory to save evaluation output. If none is given, the system tries to save output files in current working directory.

Retrieval Configuration

• --retrieval_only: Disable generation (LLM prompting). Retrieve candidate frames based on similarity with input text, but do not select and align them to text using an LLM (default:False)
• --top_k: Number of most relevant frames to retrieve from the vector store. Default: 29
• --search_type: FAISS retrieval strategy to select the most relevant frames. Default: similarity
• --embedding_model_name: Name of the HuggingFace embedding model to use for frame retrieval. Default: BAAI/bge-m3

LLM Configuration

• --model_name: Name of OpenAI model to be used for frame extraction (default: "gpt-5.4").
• --api_key_name: Name of user's API key to call model.
• --nebula: Whether to use Nebula models for frame extraction. If not set, nebula models are not used.
• --model_provider: Name of model provider in case of using nebula (LangChain cannot infer the model provider if using nebula),
• --zero_shot: Change mode from few-shot (with examples in LLM prompt) to zero-shot (without examples in LLM prompt).

Evaluation Configuration

• --evaluate: Flag to indicate evaluation mode (boolean).
• --generated_frames_filepath: File path to generated frames JSON file for evaluation (string, required if --evaluate is set)
• --proportion_eval_texts: Proportion of FrameNet texts to use for evaluation (default: 0.1 (=10%)).
• --min_frames_per_eval_text: Minimum number of frames annotated in a text for it to be included in evaluation (default: 2).
• --match_type: Type of matching between model-generated frames and human-annotated frames. Choose between exact, semantic, and graph (default: exact).

Project Architecture

Core Modules

File	Purpose
generate_frames.py	Entry point: orchestrates generation and evaluation
RAG.py	RAG pipeline: generates or loads frame embeddings, retrieves candidate frames, prompts LLM
evaluation.py	Evaluation metris: precision, recall, F1, error analysis
preprocess_FrameNet.py	Parses FrameNet XML to JSON format for frame embedding generation and evaluation
utils.py	Utility functions for file handling, frame filtering, etc.

References

Baker, C. F., Fillmore, C. J., & Lowe, J. B. (1998). The Berkeley FrameNet Project. In COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics.