Update

CLAUDE.md

@@ -0,0 +1,117 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+A Python library for automated bot operations on MediaWiki projects (Wikipedia, Wikidata). Provides authenticated API access, page reading/editing/creation, recursive category traversal, Wikidata SPARQL queries, and local database storage via SQLite and MySQL. Used as a dependency by other bot repositories in this ecosystem.
+
+## Commands
+
+### Run Tests
+
+```bash
+pytest                                    # All tests (network disabled by default)
+pytest -m unit                            # Unit tests only
+pytest -m network                         # Network-dependent tests
+pytest -k "test_name"                     # Single test by name
+pytest --cov=newapi --cov-report=term-missing --cov-branch  # With coverage
+```
+
+`pytest.ini` sets `pythonpath = newapi`, `testpaths = tests`, `--maxfail=25`, `--durations=10`, `--strict-markers`. Network tests excluded by default via `-m "not network"`.
+
+The `conftest.py` autouse fixture disables all network sockets via `pytest-socket`.
+
+### Lint & Format
+
+```bash
+ruff check .              # Lint
+ruff check --fix .        # Lint with auto-fix
+ruff format .             # Format
+black .                   # Format (alternative)
+isort .                   # Sort imports
+mypy .                    # Type check
+```
+
+Line length is 120 across all tools. Target Python 3.13.
+
+### Install Dependencies
+
+```bash
+pip install -r requirements.in       # Runtime
+pip install -r requirements-dev.txt  # Testing
+```
+
+Runtime: `PyMySQL`, `Requests`, `sqlite_utils`, `tqdm`, `wikitextparser`, `mwclient`, `ratelimiter`, `SPARQLWrapper`, `colorlog`, `python-dotenv`, `pywikibot`.
+Dev: `pytest`, `pytest-cov`, `pytest-mock`, `pytest-socket`.
+
+## Architecture
+
+Three-layer architecture:
+
+```
+Layer 1: Transport + Auth (newapi/api_client/)
+  RequestsHandler (retry/CSRF/maxlag) -> WikiLoginClient (auth, cookies, pagination)
+  Exceptions: WikiClientError -> LoginError, CSRFError, MaxlagError, MaxRetriesExceeded, CookieError
+
+Layer 2: Wiki Business Logic (newapi/client_wiki/)
+  AllAPIS (main facade) -> MainPage (page ops), CategoryDepth (recursive traversal), NewApi (bulk ops)
+  api_utils/: AskBot, HandleErrors, botEdit, txtlib, wd_sparql, lang_codes
+
+Layer 3: High-Level Operations (newapi/super/S_API/bot_api.py)
+  NewApi class: 30+ methods (search, listing, contributions, upload, move, langlinks, etc.)
+```
+
+### Key Abstractions
+
+- **`AllAPIS`** (`client_wiki/all_apis.py`) -- Main facade. `AllAPIS(lang, family, username, password)` creates authenticated client. Methods: `.MainPage(title)`, `.CatDepth(category)`, `.NewApi()`.
+- **`WikiLoginClient`** (`api_client/client.py`) -- Authenticated HTTP client wrapping `mwclient.Site`. Handles CSRF tokens, maxlag backoff, session recovery, cookie persistence.
+- **`MainPage`** (`client_wiki/pages/super_page.py`) -- Page-level operations: read, edit, create, get categories/templates/links, check editability. Inherits `HandleErrors` and `AskBot`.
+- **`CategoryDepth`** (`client_wiki/categories/category_db.py`) -- Recursive category member retrieval with depth, namespace, template, and language-link filtering.
+- **`NewApi`** (`super/S_API/bot_api.py`) -- 30+ high-level operations delegating to `WikiLoginClient`.
+- **`Settings`** (`config.py`) -- Dataclass singleton loaded from env vars. Sections: `WikidataConfig`, `ApiClientConfig`, `DatabaseConfig`, `DebugConfig`, `BotConfig`, `QueryConfig`, `SiteConfig`.
+
+### Usage Pattern
+
+```python
+from newapi import AllAPIS
+
+api = AllAPIS(lang='en', family='wikipedia', username='user', password='pass')
+page = api.MainPage("Article Title")
+text = page.get_text()
+page.save(newtext, summary="Edit summary")
+```
+
+Legacy convenience (deprecated):
+```python
+from newapi.page import load_main_api
+api = load_main_api("en", "wikipedia")
+```
+
+### Cookie Management
+
+Sessions persisted as Mozilla-format cookie jar files. Auto-expire after 3 days. Path: `{cookies_dir}/{family}_{lang}_{username}.mozilla`. `COOKIES_DIR` env var or defaults to `~/tmp/cookies`.
+
+### Error Handling
+
+Two parallel exception hierarchies:
+1. `api_client/exceptions.py`: `WikiClientError` -> `LoginError`, `CSRFError`, `MaxlagError`, `MaxRetriesExceeded`, `CookieError`
+2. `core/exceptions.py`: `NewApiException` -> `ApiError` -> `AbuseFilterError`, `MaxLagError`, `ArticleExistsError`, `ProtectedPageError`, etc. Includes `parse_api_error()` for mapping API error dicts.
+
+### Patterns to Know
+
+- **`{1: value}` mutable dict pattern** -- used for module-level mutable state.
+- **Module-level side effects** -- importing modules can trigger login, computation, or sys.argv modification.
+- **Mixed naming conventions** -- PascalCase classes, snake_case functions, some camelCase mixed in.
+- **Mixed Arabic/English** in comments and string literals.
+
+## Environment
+
+Credentials from environment variables:
+- `WIKIPEDIA_BOT_USERNAME` / `WIKIPEDIA_BOT_PASSWORD` -- primary bot account
+- `WIKIPEDIA_HIMO_USERNAME` / `WIKIPEDIA_HIMO_PASSWORD` -- alternate account (when `workibrahem` setting active)
+
+## CI/CD
+
+- **Tests**: GitHub Actions (`pytest.yaml`) runs `pytest` on PRs to `main` (Python 3.11 in CI, 3.13 locally).
+- **Publish**: `python-publish.yml` for PyPI releases.

_work_files/tree.py

@@ -14,7 +14,7 @@
     ignoreList=["__pycache__", "old", "app1.py", "example.env", "*.html"],
     onlyFiles=False,
     onlyDirs=False,
-    sortBy=0,
+    sortBy=2,
     raiseException=False,
     printErrorTraceback=False,
 )

newapi/DB_bots/README.md

@@ -0,0 +1,149 @@
+# DB_bots -- Database Abstraction Layer
+
+## Project Overview
+
+The `DB_bots` package provides database access abstractions for both SQLite and MySQL, designed for use by MediaWiki bots for local data storage and querying.
+
+### Main Modules
+
+| Module | Purpose |
+|---|---|
+| `db_bot.py` | SQLite abstraction with two implementations: `LiteDB` (legacy) and `LiteDbRepository` (repository pattern) |
+| `pymysql_bot.py` | MySQL access via PyMySQL with a single `sql_connect_pymysql` function |
+
+### Technologies & Dependencies
+
+- **`sqlite_utils`** -- SQLite database toolkit (used by `LiteDB`)
+- **`pymysql`** -- Pure Python MySQL client
+
+---
+
+## Architecture & Code Quality Review
+
+### Code Organization
+
+**Fair.** Two independent modules with no shared interface or base class. `db_bot.py` contains two unrelated implementations (`LiteDB` and `LiteDbRepository`) with significant API overlap.
+
+### Design Patterns
+
+- **Repository Pattern**: `LiteDbRepository` implements a clean repository interface with `get_by_id`, `find_by`, `insert`, `update`, `delete`.
+- **Active Record-like**: `LiteDB` provides direct table operations with raw SQL access.
+
+### Maintainability
+
+**Low-Moderate.** Having two unrelated SQLite abstractions in the same file is confusing. `pymysql_bot.py` is a single function with no structure.
+
+### Readability
+
+**Fair.** Method names are descriptive but the dual-implementation approach in `db_bot.py` requires careful reading to understand which to use.
+
+### Scalability Considerations
+
+- **MySQL**: No connection pooling -- a new connection is created and destroyed per query.
+- **SQLite**: `LiteDbRepository.count()` fetches all rows into memory to count them instead of using SQL `COUNT`.
+
+---
+
+## Strengths
+
+- **Clean repository interface**: `LiteDbRepository` provides a well-structured CRUD API.
+- **Parameterized queries**: `LiteDB.select` correctly uses `?` placeholders to prevent SQL injection.
+- **Simple MySQL function**: `sql_connect_pymysql` is straightforward for one-off queries.
+- **Type hints**: Both modules use type annotations on method signatures.
+
+---
+
+## Weaknesses
+
+- **Two competing SQLite abstractions**: `LiteDB` and `LiteDbRepository` coexist with overlapping functionality and no shared interface.
+- **Dead code**: `del data` / `del datalist` calls in `LiteDB` are no-ops. The `tracer` function is defined but commented out.
+- **Silent error swallowing**: `LiteDbRepository` catches bare `Exception` and returns `None`/`False`. `pymysql_bot` returns `[]` on all errors.
+- **No connection pooling**: MySQL connections are created fresh for every query call.
+- **Inefficient counting**: `LiteDbRepository.count()` with criteria fetches all rows just to `len()` them.
+
+---
+
+## Critical Issues
+
+### 1. SQL Injection in `LiteDB.query()` and `LiteDB.update()` (HIGH)
+
+```python
+# db_bot.py
+def query(self, sql: str) -> List[tuple]:
+    return self.db.query(sql)  # Raw SQL, no parameterization
+
+def update(self, sql: str) -> None:
+    self.db.executescript(sql)  # Raw SQL, no parameterization
+```
+
+These methods accept raw SQL strings with no sanitization. Any caller passing user-controlled input is fully exposed to SQL injection.
+
+### 2. Mutable Default Arguments in `sql_connect_pymysql()` (HIGH)
+
+```python
+# pymysql_bot.py
+def sql_connect_pymysql(query, return_dict=False, values=None,
+                        main_args={},  # <-- Mutable default!
+                        credentials={},  # <-- Mutable default!
+                        ...):
+```
+
+Modifications to these dicts persist across calls, causing subtle state corruption.
+
+### 3. Silent Error Swallowing (MEDIUM)
+
+`pymysql_bot.py` returns `[]` on all exceptions. `LiteDbRepository` returns `None`/`False`. Callers cannot distinguish "no results" from "database error."
+
+### 4. `LiteDB.insert` Hardcodes Primary Key (LOW)
+
+```python
+# db_bot.py
+def insert(self, table_name, data, check=True):
+    # ... hardcodes pk="id" despite create_table accepting custom pk
+```
+
+Tables with non-`id` primary keys will fail.
+
+---
+
+## Areas That Need Attention
+
+- **Unify SQLite abstractions**: Choose one API (`LiteDbRepository` is cleaner) and deprecate the other.
+- **Remove dead code**: `del data`, `del datalist`, `tracer` function.
+- **Add connection pooling** for MySQL.
+- **Use SQL `COUNT`** instead of fetching all rows for counting.
+- **Add error differentiation**: Raise or return typed errors instead of silent defaults.
+- **Fix mutable default arguments** in `sql_connect_pymysql`.
+
+---
+
+## Improvement Plan
+
+### Quick Wins
+1. Fix mutable default arguments in `sql_connect_pymysql` (`main_args=None`, `credentials=None`).
+2. Remove dead `del` statements and `tracer` function.
+3. Fix `LiteDbRepository.count()` to use SQL `COUNT`.
+
+### Medium-Term Improvements
+1. Deprecate `LiteDB` in favor of `LiteDbRepository`.
+2. Add connection pooling for MySQL (e.g., `DBUtils` or `SQLAlchemy` pool).
+3. Implement proper error handling with typed exceptions.
+4. Add `__all__` to `__init__.py`.
+
+### Long-Term Refactoring
+1. Define a shared abstract interface for both SQLite and MySQL.
+2. Add transaction management for write operations.
+3. Add retry logic for transient database errors.
+4. Add comprehensive unit tests.
+
+---
+
+## Comprehensive Review
+
+| Metric | Score |
+|---|---|
+| **Overall Rating** | **5/10** |
+| **Production Readiness** | Low-Moderate -- SQL injection and mutable defaults are production risks |
+| **Technical Debt** | High -- dual abstractions, dead code, silent errors |
+| **Risk Assessment** | High -- SQL injection vulnerability and mutable default arguments |
+| **Maintainability** | 4/10 -- confusing dual APIs, no tests, silent failures |