language-technology-project

Course project for Language Technology Practical 25/26

Installation

1. Clone the repository

git clone https://github.com/RemcoStudent/language-technology-project.git

2. Install the correct dependencies (Python 3.11)

pip install requirements.txt

Running the code

Interactive mode

To run the mode in interactive mode, make sure RUN_EVALUATION is False (default).Then type the following command in the terminal.

python source/main.py

Evaluation mode

To run the mode in evaluation mode, change RUN_EVALUATION in config to True. Then type the following command in the terminal.

python source/main.py

Pre-Commit Hooks

How to Use Pre-Commit

• After installing (pre-commit install), hooks run automatically on git commit.
• To manually run hooks on all files:

  pre-commit run --all-files

• To skip hooks for a commit:

  git commit -m "message" --no-verify

File Structure

language-technology-project/
├── README.md                             # Project overview and installation instructions
├── requirements.txt                      # Dependencies
├── .gitignore
├── .pre-commit-config.yaml
├── data/
│   └── ed3book_aug25-pages-deleted.pdf   # Corpus
├── docs/
│   └── Project_Specification.pdf         # Assignment instructions
├── notebooks/
│   ├── retrieval_notebook_basic.ipynb    # Jupyter notebook demonstrating basic retrieval functionality
│   └── retrieval_notebook_enhancement.ipynb  # Jupyter notebook showcasing enhanced retrieval techniques
├── resources/
│   └── test.json                         # Test dataset with QA
└── source/
    ├── __init__.py
    ├── main.py                           # Runs either Streamlit or evaluates the test set.
    ├── app.py                            # Streamlit interface for interactive question-answering
    ├── config.py                         # Configuration and hyperparameters for models and system settings
    ├── data_processing.py                # Handles PDF loading, text chunking, and QA dataset processing
    ├── retrieval.py                      # Implements hybrid retrieval combining BM25 and dense embeddings with reranking
    ├── generation.py                     # Handles answer generation using LLM with retrieved context
    ├── dialogue.py                       # Manages multi-turn conversation history
    └── evaluation.py                     # Computes evaluation metrics