[FEATURE] Add Context-Aware Resume Skill Evaluation and Weighted Scoring

[divyankaaaaaa]

Problem

Current resume evaluation may not fully differentiate between contextual usage of skills across projects, internships, research work, and simple keyword mentions.

For example, a skill mentioned under real-world internship experience should carry more significance than the same skill listed only in the skills section.

Proposed Solution

Implement a context-aware skill evaluation system within the existing LLM evaluation pipeline.

Features

• Contextual weighting for skills based on resume sections

• Weighted scoring system for:

  • internships
  • projects
  • research work
  • certifications
  • skills section

• Structured explainable scoring output

• Evidence extraction for recruiter transparency

• JSON schema validation for consistent outputs

Expected Benefits

• More accurate candidate evaluation
• Better recruiter trust and explainability
• Improved ranking quality
• Stronger contextual understanding in resume analysis
• Seamless integration with existing LLM workflow

I’d like to work on this issue under GSSoC'2026.

[sathwikshetty33]

Current resume evaluation may not fully differentiate between contextual usage of skills across projects, internships, research work, and simple keyword mentions.

Any concrete proff for this claim? It has worked good for us!

[Udayan6611]

Hey @sathwikshetty33! To answer your question regarding the need for this: While standard keyword extraction works well for baseline filtering, it struggles with depth. For example, a candidate who simply lists "Python" in a generic skills section currently gets a similar evaluation to a candidate who explicitly writes "Architected a scalable backend microservice using Python" under their Work Experience. By adding a lightweight LLM contextual evaluation step, we can penalize "keyword stuffing" and mathematically boost candidates who demonstrate applied, real-world usage of those skills.

Since I have extensive experience building AI agent evaluation pipelines, working with LangChain, and handling data preprocessing, I would love to build out this context-aware scoring logic and integrate it seamlessly into the existing workflow. Let me know if I can take this on for GSSoC '26!

[KraisuN-1010]

Hi @sathwikshetty33 👋

I'd love to work on this issue as part of GSSoC '26!

I've gone through the codebase and have a clear picture of what's causing the problem — the current ResumeEvaluator receives a flat, unstructured text blob from PyPDF2, and the prompt has no instruction to differentiate skills by resume section. This means a skill listed under an internship is scored the same as one simply mentioned in the Skills section.

My proposed approach:

• Section-aware parser — split extracted text into labeled sections (internship, project, research, certification, skills, etc.) before it reaches the LLM
• Weighted prompt — pass sections with explicit context weights (e.g. internship = 1.0, skills section = 0.3) so the LLM scores skills by evidential depth, not just keyword presence
• Richer JSON output schema — return per-skill breakdown with evidence, found_in_sections, and weighted_score alongside the overall score, enabling recruiter transparency

This keeps changes within the existing app/ai/ and app/utils/ structure and stays compatible with the current TaskIQ pipeline.

Can I be assigned to this? Happy to discuss the approach further before I open a PR. 🙏

[Udayan6611]

Hi @sathwikshetty33,

Just wanted to follow up with a concrete implementation plan for how I intend to execute the Context-Aware Skill Scorer backend to make sure it's fully enterprise-ready:

Strict Output Guardrails: Instead of relying on raw LLM string parsing (which is fragile and prone to breaking), I will implement this using LangChain’s with_structured_output wrapped around a strict Pydantic schema. This guarantees the backend always returns a valid JSON object matching the exact frontend types (skill_name, context_score, evidence_extracted, justification_reasoning).

Deterministic Evaluation Rubric: I will structure the system prompt to explicitly categorize the source context into strict evaluation tiers (e.g., High-Tier for verified Work Experience/Internships, Mid-Tier for Hackathons/Personal Projects, and Low-Tier penalties for uncontextualized generic lists) to completely eradicate keyword stuffing.

Robust Error Handling: I will include safety fallbacks for edge cases, such as when a requested skill is completely missing or when the resume text formatting is heavily corrupted.

I have the LangChain architecture mapped out and can open a clean, tested PR quickly. Would love to be officially assigned so I can get to work!

[sathwikshetty33]

@Udayan6611 We do already have LLM based evalauation and output parsing usibg langchains output parser.

[Udayan6611]

Got it, thanks for the correction @sathwikshetty33! That makes total sense. I'll clone the repo, locate the existing evaluation chain and LangChain output parser setup, and adapt the context-aware scoring rubric (High/Mid/Low tiers) directly into the current prompt engineering framework instead of introducing a new schema structure.

I have the bandwidth to jump into the codebase and execute this right away. Let me know if you can assign it to me!

[Udayan6611]

Hey @sathwikshetty33,

I just spent some time digging through the backend stack (specifically app/models/application.py, app/ai/prompts.py, and the TaskIQ worker flow in the README). I noticed that the resume_evaluator_prompt is already super well-structured for handling complex semantic rules (like the FAANG tiers and role multipliers). Because of that, we can handle the contextual skill weighting natively in the LLM without adding any heavy external text splitters.

Here is my plan to get this done with zero architectural churn:

1. Upgrading the Prompt (app/ai/prompts.py)
   I'll add a "Skill Context Weighting Matrix" directly into the existing evaluation rules. We can explicitly instruct the LLM to:

• Reward High Weights: Only when a skill is embedded in an action narrative (like bullet points under ### Experience or ### Projects).

• Penalize / Assign Low Weights: When skills just appear as a comma-separated list of buzzwords under ### Skills with no context.

2. Recruiter Transparency with Zero DB Migrations
   The issue mentions adding evidence extraction for recruiters. To do this without forcing an Alembic migration or having to rewrite the TaskIQ worker logic, we can just use the existing feedback column (since it's safely typed as unbounded Text in SQLAlchemy).
   I'll update the prompt so the LLM appends a clean "Skill Evidence Validation" breakdown directly into the ResumeEvaluatorResponse.feedback string.

This approach gives recruiters exactly what they need while keeping the current LangChain/LiteLLM pipeline fully intact, no database migrations and no breaking changes.

I've already got the repo cloned and can get a clean PR up for this right away. Let me know if I have the green light to take it on!

[divyankaaaaaa]

Fair point @sathwikshetty33. I went through the existing evaluation flow more carefully after your comment, and I agree the current LLM pipeline already handles semantic understanding quite well.

What I was mainly targeting was improving recruiter-side interpretability and reducing cases where isolated keyword mentions may unintentionally receive similar importance as demonstrated usage inside projects or internships.

Rather than introducing a separate evaluation system, I think this can be approached as a lightweight prompt-engineering enhancement within the current architecture itself, especially by guiding the model to explicitly prioritize evidence-backed skill usage.

Your clarification definitely helped me understand the existing pipeline better.

[sathwikshetty33]

@Udayan6611 , @divyankaaaaaa either of you working on towards improving the prompt, We would want to see the specific result/edgecase that improved prompt does. So your prs won't be considered until we see the actual improvement.

[Udayan6611]

Hey @sathwikshetty33, following up with the concrete proof you requested!

I implemented the Skill Context Weighting Matrix entirely within the existing resume_evaluator_prompt natively using LangChain. To provide recruiter transparency without forcing an Alembic database migration, the LLM is now instructed to append a structured [Skill Evidence Breakdown] directly into the existing unbounded feedback text column.

Here is the edge-case A/B test run through my local TaskIQ worker to show the actual improvement:

1. The "Keyword Stuffer" (Edge-Case)

Input Profile: A candidate who simply lists "FastAPI", "LangChain", and "TaskIQ" as a comma-separated list under their Skills section, but their Experience narrative only mentions "fixing UI bugs."

New Output:

Click to view JSON Output

{
  "score": 2.5,
  "shortlisting_decision": false,
  "feedback": "Basic competencies but significant experience gaps. Company background neutral, but required experience too low to meet minimum standards. [Skill Evidence Breakdown]\n- PENALIZED: All skills listed under Skills section. No contextual evidence."
}

Notice how the model actively penalizes the naked keywords in the Skill Evidence Breakdown.

2. The Contextual Candidate (The Baseline)

Input Profile: A candidate who embeds those exact same skills in an action narrative under their Experience/Projects (e.g., "Architected a context-aware resume scoring backend using FastAPI and TaskIQ...").

New Output:

Click to view JSON Output

{
  "score": 7.2,
  "shortlisting_decision": true,
  "feedback": "Good candidate with 3 years experience. Strong technical skills, but limited company background. [Skill Evidence Breakdown]\n- REWARDED: Machine Learning, FastAPI, TaskIQ, LangChain, Prompt Engineering (Demonstrated applied context in projects).\n- PENALIZED: AI (Only listed as a naked keyword under Skills)\n"
}

Notice how the model now explicitly rewards the skills that are backed by contextual depth.

This proves the prompt engineering successfully solves the depth vs. keyword-stuffing issue natively. Since this is fully tested on the local backend and stays within the existing architecture, let me know if I have the green light to open the PR for GSSOC '26!