fix: improve Cypher query generation accuracy#294
fix: improve Cypher query generation accuracy#294hupeky wants to merge 1 commit intovitali87:mainfrom
Conversation
Summary of ChangesHello @hupeky, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request focuses on enhancing the accuracy of Cypher query generation by the Language Model (LLM). It achieves this by providing more explicit guidance within the LLM prompts regarding schema interpretation, particularly for property usage and relationship types. Additionally, it bolsters the system's ability to parse and extract valid Cypher queries from diverse LLM output formats, making the overall process more reliable. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request effectively improves the accuracy of Cypher query generation by enhancing the prompts with clearer rules and examples, and by making the response cleaning logic more robust. The changes in prompts.py and cypher_queries.py are well-implemented and align with the PR's objectives. I have a few suggestions for the _clean_cypher_response function in codebase_rag/services/llm.py to align it with project standards and further improve its robustness.
| else: | ||
| # Remove markdown bold/headers (e.g., **Cypher Query:**) | ||
| query = re.sub(r"\*\*[^*]+\*\*:?\s*", "", query) | ||
| # Remove single backticks | ||
| query = query.replace(cs.CYPHER_BACKTICK, "") | ||
| # Remove "cypher" prefix if present | ||
| if query.lower().startswith(cs.CYPHER_PREFIX): | ||
| query = query[len(cs.CYPHER_PREFIX):].strip() |
There was a problem hiding this comment.
The current logic in the else block may not correctly handle all cases with leading whitespace, and the cleaning steps could be ordered for better robustness. For example, a response like **Cypher Query:** MATCH (n) would result in a query with a leading space: MATCH (n);, which could cause execution to fail. This refactoring handles whitespace more consistently and correctly identifies and removes the cypher prefix even if it has leading spaces.
| else: | |
| # Remove markdown bold/headers (e.g., **Cypher Query:**) | |
| query = re.sub(r"\*\*[^*]+\*\*:?\s*", "", query) | |
| # Remove single backticks | |
| query = query.replace(cs.CYPHER_BACKTICK, "") | |
| # Remove "cypher" prefix if present | |
| if query.lower().startswith(cs.CYPHER_PREFIX): | |
| query = query[len(cs.CYPHER_PREFIX):].strip() | |
| else: | |
| # Remove markdown bold/headers (e.g., **Cypher Query:**) | |
| query = re.sub(r"\*\*[^*]+\*\*:?\s*", "", query) | |
| # Remove "cypher" prefix if present | |
| if query.lower().strip().startswith(cs.CYPHER_PREFIX): | |
| query = query[query.lower().find(cs.CYPHER_PREFIX) + len(cs.CYPHER_PREFIX):] | |
| # Remove single backticks and strip any remaining whitespace | |
| query = query.replace(cs.CYPHER_BACKTICK, "").strip() |
| """Clean LLM response to extract pure Cypher query. | ||
|
|
||
| Handles markdown formatting that models sometimes output: | ||
| - Triple backticks (```cypher ... ```) | ||
| - Bold text (**Cypher Query:**) | ||
| - Headers and other markdown | ||
| """ |
There was a problem hiding this comment.
According to the project's general rules, docstrings are not allowed. Please remove this docstring to adhere to the project's coding standards.
References
- Docstrings are not allowed in this project, as enforced by a pre-commit hook.
| - Bold text (**Cypher Query:**) | ||
| - Headers and other markdown | ||
| """ | ||
| import re |
There was a problem hiding this comment.
Per PEP 8, imports should be at the top of the file. Please remove this import from here and add import re to the top-level imports section of the file.
References
- PEP 8: Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants. (link)
This PR addresses issues where the LLM generates incorrect Cypher queries due to misunderstanding the graph schema. Changes: - Add CYPHER_EXAMPLE_CLASS_METHODS to demonstrate DEFINES_METHOD pattern - Add VALUE PATTERN RULES to prompts explaining name vs qualified_name usage - Improve _clean_cypher_response() to handle markdown formatting in LLM output The prompt improvements teach the LLM to: - Use `name` property for short class/function names (not qualified_name) - Use correct relationships (DEFINES_METHOD, DEFINES) - Follow proper Cypher patterns for this schema The response cleaner now handles: - Triple backtick code blocks (```cypher ... ```) - Bold markdown headers (**Cypher Query:**) - Mixed formatting in LLM responses Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Greptile OverviewGreptile SummaryThis PR improves Cypher query generation accuracy by teaching the LLM to correctly use the graph schema properties and relationships. Key Changes:
Impact: The enhanced response cleaner now correctly extracts Cypher queries from markdown-formatted LLM responses, improving robustness across different LLM providers and output formats. Confidence Score: 4/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant User
participant CypherGenerator
participant Agent
participant LLM
participant CleanFunction as _clean_cypher_response
User->>CypherGenerator: "generate('What methods does UserService have?')"
CypherGenerator->>Agent: "run(natural_language_query)"
Agent->>LLM: "Send prompt with system prompt and examples"
Note over LLM: Uses VALUE PATTERN RULES<br/>Match by name property<br/>Use DEFINES_METHOD relationship
LLM-->>Agent: "Response with markdown formatting"
Agent-->>CypherGenerator: "result.output"
CypherGenerator->>CleanFunction: "Clean markdown formatting"
Note over CleanFunction: Extract from code blocks<br/>Remove bold headers<br/>Add semicolon
CleanFunction-->>CypherGenerator: "Clean Cypher query"
CypherGenerator-->>User: "Valid Cypher query"
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| - Bold text (**Cypher Query:**) | ||
| - Headers and other markdown | ||
| """ | ||
| import re |
There was a problem hiding this comment.
Move import re to top-level imports (after line 1). Module-level imports belong with stdlib imports at the file top, not inside functions.
| import re | |
| """Clean LLM response to extract pure Cypher query. | |
| Handles markdown formatting that models sometimes output: | |
| - Triple backticks (```cypher ... ```) | |
| - Bold text (**Cypher Query:**) | |
| - Headers and other markdown | |
| """ |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Prompt To Fix With AI
This is a comment left during a code review.
Path: codebase_rag/services/llm.py
Line: 36:36
Comment:
Move `import re` to top-level imports (after line 1). Module-level imports belong with stdlib imports at the file top, not inside functions.
```suggestion
"""Clean LLM response to extract pure Cypher query.
Handles markdown formatting that models sometimes output:
- Triple backticks (```cypher ... ```)
- Bold text (**Cypher Query:**)
- Headers and other markdown
"""
```
<sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub>
How can I resolve this? If you propose a fix, please make it concise.| if query.lower().startswith(cs.CYPHER_PREFIX): | ||
| query = query[len(cs.CYPHER_PREFIX):].strip() |
There was a problem hiding this comment.
Case mismatch: query.lower().startswith() but using original cs.CYPHER_PREFIX length. If cs.CYPHER_PREFIX = "cypher" and query is "CYPHER MATCH...", slicing by len("cypher") (6 chars) from "CYPHER MATCH..." works correctly. However, for safety and clarity, use consistent casing.
| if query.lower().startswith(cs.CYPHER_PREFIX): | |
| query = query[len(cs.CYPHER_PREFIX):].strip() | |
| if query.lower().startswith(cs.CYPHER_PREFIX.lower()): | |
| query = query[len(cs.CYPHER_PREFIX):].strip() |
Prompt To Fix With AI
This is a comment left during a code review.
Path: codebase_rag/services/llm.py
Line: 50:51
Comment:
Case mismatch: `query.lower().startswith()` but using original `cs.CYPHER_PREFIX` length. If `cs.CYPHER_PREFIX = "cypher"` and query is `"CYPHER MATCH..."`, slicing by `len("cypher")` (6 chars) from `"CYPHER MATCH..."` works correctly. However, for safety and clarity, use consistent casing.
```suggestion
if query.lower().startswith(cs.CYPHER_PREFIX.lower()):
query = query[len(cs.CYPHER_PREFIX):].strip()
```
How can I resolve this? If you propose a fix, please make it concise.
hupeky
force-pushed
the
fix/improve-cypher-generation
branch
from
c24af15 to
232fee8
Compare
1 participant
Summary
Problem
The LLM sometimes generates incorrect Cypher queries because it doesn't understand:
namevsqualified_namepropertiesChanges
CYPHER_EXAMPLE_CLASS_METHODSexample_clean_cypher_response()to handle markdown code blocksTesting
Tested with
codellamamodel - queries now generate correctly for class method lookups and other common patterns.🤖 Generated with Claude Code