Skip to content

Added unit tests - #3

Open
AkshatSharma25 wants to merge 1 commit into
AnkitNayak-dev:mainfrom
AkshatSharma25:main
Open

Added unit tests#3
AkshatSharma25 wants to merge 1 commit into
AnkitNayak-dev:mainfrom
AkshatSharma25:main

Conversation

@AkshatSharma25

Copy link
Copy Markdown

Closes #2
The description of the test cases added is as follows:

File 1: test_utils.py

Before data goes anywhere near an AI, it needs a cleanup. These 13 tests make sure our functions aren’t slacking.

Text Cleaning

We don’t want wasted tokens or messy formatting confusing the model.

  • Extra Newlines – Collapses 4+ line breaks into a neat two. No endless scrolling.
  • Space Crusher – Turns Hello World into Hello World.
  • Tab-to-Space – Replaces \t with normal spaces so searches behave.
  • The Trimmer – Cuts sneaky leading/trailing whitespace from copy-paste jobs.
  • “Don’t Panic” Test – Handles empty strings without crashing.

Filename Regex

Makes sure we correctly split CSV rows into filename and content.

  • Standard Case – Parses document.txt,Content cleanly.
  • Special Characters – Supports dashes and underscores in filenames.
  • The Bouncer – Rejects malformed lines instead of breaking the system.

Hash Functions

Prevents us from paying to process the same text twice.

  • Consistency – Same input, same hash. Every time.
  • SHA256 Check – Verifies the hash is the correct 64 characters.
  • Case Insensitivity – Treats “Hello” and “hello” as identical to avoid duplicates.

File 2: test_basic.py

These 9 tests make sure our data structures and settings won’t fall apart when we hit “Go.”

Data Structures

  • Doc Format – Every document must have text and file. No exceptions.
  • Metadata Integrity – Chunks must know their source and index.
  • JSON Round-trip – Convert to JSON and back without losing anything.

Config Guards

  • Overlap Rulechunk_overlap must be smaller than chunk_size.
  • API Safetytemperature stays between 0–2, and max_tokens can’t be negative.

Prompt Templates

  • Placeholder Check – Ensures {context} and {question} still exist.
  • Formatting Test – Runs a mock prompt to confirm it renders exactly right.

Clean data. Solid structure. Safe configs. No surprises.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add: Unit tests

1 participant