-
Notifications
You must be signed in to change notification settings - Fork 5
Fix memory leaks and code quality issues #93
Description
Summary
Several code quality issues affect production stability: a memory leak from unbounded global lists, mutable default arguments that can cause shared-state bugs, unsafe error string parsing, and silent exception swallowing.
Issues
1. Memory Leak: Unbounded global_query_list (High Severity)
Both Query and PoolQuery maintain static class-level lists that grow indefinitely:
# client/query.py
class Query:
global_query_list: List["Query[Any]"] = []
def __init__(self, ...):
Query.global_query_list.append(self) # Never removed
# pool/pool_query.py
class PoolQuery:
global_query_list: List["PoolQuery"] = []
def __init__(self, ...):
PoolQuery.global_query_list.append(self) # Never removedEvery query created during the lifetime of the process is retained in memory forever. In long-running applications or services, this will cause steadily increasing memory consumption.
Fix: Either remove queries on close/cleanup, use weakref.WeakList, or remove the global list entirely if it's not needed externally.
2. Mutable Default Arguments (Medium Severity)
At least 5 instances of opts: Dict[str, Any] = {} across the codebase:
# base_job.py, sql_job.py, etc.
def __init__(self, ..., opts: Dict[str, Any] = {}, **kwargs):
def connect(..., opts: Dict[str, Any] = {}, **kwargs):All calls to these functions share the same dict object. If any code path mutates opts, it affects all subsequent calls.
Fix: Replace with opts: Optional[Dict[str, Any]] = None and if opts is None: opts = {} inside the function body.
3. Unsafe Error Parsing with ast.literal_eval() (Medium Severity)
In core/exceptions.py:
error_dict = ast.literal_eval(error_string)ast.literal_eval() can parse arbitrary Python literals, not just JSON. While it won't execute code, it's the wrong tool for parsing JSON error strings from the server.
Fix: Replace with json.loads(error_string) which is both safer and more appropriate for JSON data.
4. Silent Exception Swallowing (Medium Severity)
In core/cursor.py nextset():
def nextset(self):
try:
if len(self.query_q) > 1:
self.query_q.popleft()
self.query = self.query_q[0]
return True
return None
except Exception:
return None # Silently swallows ALL exceptionsAlso in various query close() paths where AttributeError may be silently ignored.
Fix: Log the exception before returning, or narrow the except clause to specific expected exceptions.
5. Dead Code (Low Severity)
QueryManager.run_query() is defined but never called anywhere in the codebase.
Fix: Remove the dead method, or integrate it into the query execution path if it was intended to be used.
Files to Modify
mapepire_python/client/query.py— Fixglobal_query_listlifecycle, mutable defaultsmapepire_python/pool/pool_query.py— Fixglobal_query_listlifecyclemapepire_python/base_job.py— Fix mutable default argumentsmapepire_python/client/sql_job.py— Fix mutable default argumentsmapepire_python/core/exceptions.py— Replaceast.literal_eval()withjson.loads()mapepire_python/core/cursor.py— Fix silent exception swallowing innextset()mapepire_python/query_manager.py— Remove deadrun_query()method
Acceptance Criteria
-
global_query_listno longer grows unboundedly (queries removed on close or use weak references) - All mutable default arguments replaced with
None+ conditional initialization -
ast.literal_eval()replaced withjson.loads()in error parsing - No bare
except Exception: passorexcept Exception: return None— exceptions are either logged or narrowed - Dead code removed
- All existing tests still pass
- Add a unit test that verifies queries are cleaned up from global list after close