Track external file references (SQL, templates, configs, assets)

[octo-youcef]

Problem

Code often references non-Python files (SQL queries, templates, config files, static assets) but these dependencies are not tracked. This creates incomplete dependency graphs and makes it hard to assess impact of changes.

Examples of Missing Dependencies

# SQL queries
sql = load_sql("queries/users.sql")
df = pd.read_sql_file("analytics/reports.sql")

# Templates
render_template("emails/welcome.html")
jinja_env.get_template("forms/signup.jinja2")

# Configuration
config = yaml.load(open("config/settings.yaml"))
settings = toml.load("pyproject.toml")

# Static assets
return static_file("assets/logo.png")
url = url_for("static", filename="css/style.css")

# Data files
df = pd.read_csv("data/customers.csv")
with open("fixtures/test_data.json") as f: ...

Impact

• Missing dependencies: File changes not traceable to affected code
• Incomplete graph: Can't see full picture of what code depends on
• Risk of breakage: Deleting/moving files without knowing what uses them
• Documentation gap: External dependencies not visible

Solution

Phase 1: Detect String Literals in File Operations

Track file path string literals in these contexts:

Common patterns:

open(path)
Path(path)
load_sql(path)
read_csv(path)
render_template(path)
static_file(path)

AST detection:

• ast.Call with file operation functions
• Extract string literal arguments
• Resolve relative paths from module location

Phase 2: Create ExternalFile Nodes

CREATE (m:Module {name: "handlers"})
CREATE (f:ExternalFile {path: "queries/users.sql", type: "sql"})
CREATE (m)-[:REFERENCES {context: "load_sql"}]->(f)

Properties:

• path: Relative or absolute path
• type: File extension (sql, html, yaml, json, csv, etc.)
• exists: Boolean - does file exist at analysis time?

Phase 3: Track Reference Context

Store how file is used:

{
  "function": "load_sql",
  "line": 42,
  "context": "SQL query loading"
}

Detection Heuristics

High-confidence patterns:

# File I/O
open(...), Path(...)
read(), write(), load(), save()

# Template engines
render_template(), get_template()
jinja2.*, django.template.*

# Data loading
pd.read_csv(), pd.read_json(), pd.read_sql()
json.load(), yaml.load(), toml.load()

# Static files
static_file(), url_for('static', ...)

Lower-confidence patterns:

# Generic function calls with string args
some_function("path/to/file.ext")
# Only track if:
# - Argument name contains: file, path, template, query
# - String looks like a path (contains / or \, has extension)

Implementation

New AST Extractor Method

def _extract_external_references(self, node: ast.Module) -> list[ExternalFileReference]:
    """Extract references to external files."""
    references = []
    
    for call_node in ast.walk(node):
        if isinstance(call_node, ast.Call):
            # Check if it's a file operation
            if self._is_file_operation(call_node):
                # Extract file path from args
                file_path = self._extract_file_path(call_node)
                if file_path:
                    references.append(ExternalFileReference(
                        path=file_path,
                        line=call_node.lineno,
                        context=self._get_function_name(call_node)
                    ))
    
    return references

Graph Storage

# In GraphLoader
def _create_external_file_node(self, ref: ExternalFileReference) -> str:
    properties = {
        "path": ref.path,
        "type": Path(ref.path).suffix[1:],  # Extension without dot
        "exists": Path(self.root_path / ref.path).exists(),
        "package": self.package_name,
    }
    return self.connection.create_node("ExternalFile", properties)

Queries Enabled

// Find all SQL queries used by a module
MATCH (m:Module)-[:REFERENCES]->(f:ExternalFile {type: 'sql'})
RETURN m.name, f.path

// Find unused files (no references)
MATCH (f:ExternalFile)
WHERE NOT ()-[:REFERENCES]->(f)
RETURN f.path

// Find what depends on a specific file
MATCH (m)-[:REFERENCES]->(f:ExternalFile {path: 'queries/users.sql'})
RETURN m.fqn

// Find missing files (referenced but don't exist)
MATCH (f:ExternalFile {exists: false})
RETURN f.path

Acceptance Criteria

• Detect common file operation patterns
• Create ExternalFile nodes in graph
• Store REFERENCES relationships
• Track file existence at analysis time
• Support relative path resolution
• Tests for various file reference patterns
• Document common query patterns

Related

• Marked as ⭐ High value in ROADMAP.md
• Completes dependency tracking
• Enables impact analysis for file changes

Priority

MEDIUM-HIGH - Valuable for complete dependency mapping, but not blocking core functionality.