Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions .coderabbit.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
language: en-US

reviews:
profile: chill
request_changes_workflow: false
high_level_summary: true
review_status: true
auto_review:
enabled: true
drafts: false
path_filters:
- "!dist/**"
- "!website/build/**"
- "!website/.docusaurus/**"
path_instructions:
- path: "py_sql_cleaner/**/*.py"
instructions: >
Review against the layered architecture in docs/architecture/README.md.
Keep SQLGlot isolated in adapters, keep Typer and Rich in cli, and keep
application code dependent on formatter ports instead of concrete
adapters.
- path: "tests/**/*.py"
instructions: >
Prefer focused behavior tests that cover CLI behavior, formatter
behavior, and safety around skipped f-strings or Jinja-like templates.
- path: "docs/**"
instructions: >
Keep development documentation concise and consistent with
CONTRIBUTING.md and docs/architecture/README.md.
10 changes: 9 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
# Coding Agent Instructions

Follow the package architecture in `docs/architecture/README.md`.
Start here, then open the referenced docs only when they are relevant to the
change.

## Project Map

- Architecture: `docs/architecture/README.md`
- Branch workflow: `docs/development/branching.md`
- Test layout: `tests/README.md`
- Contributor workflow: `CONTRIBUTING.md`

## Dependency Rules

Expand Down
14 changes: 11 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ Check built distributions:
## Contribution Guidelines

- Keep changes focused.
- Use the branching strategy in `docs/development/branching.md`; do not push
directly to `main`.
- Add or update tests for behavior changes.
- Keep package dependencies flowing in the direction documented in
`docs/architecture/README.md`.
Expand All @@ -65,15 +67,21 @@ git config core.hooksPath .githooks

The pre-push hook runs `scripts/check`, matching the local harness and CI.

## Code Review Tooling

CodeRabbit repository settings live in `.coderabbit.yaml`. Install the CodeRabbit
GitHub App on this public repository to enable automated pull request reviews.

## Scope

The current focus is:

- Python files
- triple-quoted SQL strings
- SQLGlot-backed SQL formatting, currently defaulting to the Redshift dialect
- SQLGlot-backed SQL formatting, defaulting to generic SQL with `--dialect`
support for explicitly enabled database-specific formatting
- formatting
- extracting SQL into `.sql` files

Please open an issue before starting large features such as new dialect support,
template handling, or formatter backend changes.
Please open an issue before starting large features such as template handling,
formatter backend changes, or new dialect support.
27 changes: 22 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,9 @@
embedded in Python files.

It is built for Python codebases where long SQL queries are written directly
inside triple-quoted Python strings. The current MVP defaults to the Redshift
dialect through SQLGlot, with room for broader dialect support over time.
inside triple-quoted Python strings. The current MVP uses SQLGlot for formatting,
defaults to SQLGlot's generic dialect, and can format with database-specific
dialects that this project has explicitly enabled via `--dialect`.

```python
query = """
Expand All @@ -20,6 +21,10 @@ file.

`py-sql-cleaner` is an early MVP. It uses SQLGlot internally for best-effort SQL
formatting. It does not connect to databases and does not execute SQL.
Dialect support means SQLGlot parser/formatter mode selection, not exhaustive
database validation. Redshift command-style statements such as `COPY` and
`UNLOAD` are preserved rather than reformatted to avoid changing load/export
options.

> [!NOTE]
> `py-sql-cleaner` is conservative by default: f-strings and Jinja-like templates are
Expand All @@ -33,6 +38,7 @@ formatting. It does not connect to databases and does not execute SQL.
- Detect common SQL variable names such as `sql`, `query`, `*_sql`, and
`*_query`
- Skip unsafe blocks, including f-strings and Jinja-like templates, by default
- Support explicit dialect selection with `--dialect` / `-d`
- Support `check` mode for CI
- Support `dry-run` mode before rewriting files

Expand Down Expand Up @@ -77,13 +83,22 @@ uvx py-sql-cleaner --help
py-sql-cleaner format jobs/load_users.py
```

4. Extract embedded SQL into `.sql` files:
4. Format with a database-specific dialect:

```bash
py-sql-cleaner format jobs/load_users.py -d redshift
```

Currently enabled dialects are `generic`, `mysql`, `postgres`, and
`redshift`.

5. Extract embedded SQL into `.sql` files:

```bash
py-sql-cleaner extract jobs/load_users.py --out-dir sql
```

5. Check formatting for CI:
6. Check formatting for CI:

```bash
py-sql-cleaner check jobs/load_users.py
Expand Down Expand Up @@ -195,6 +210,7 @@ WHERE ds = '{{ ds }}'
| `format` | Format embedded SQL in place | `py-sql-cleaner format jobs/load_users.py` |
| `check` | Check whether embedded SQL is formatted | `py-sql-cleaner check jobs/load_users.py` |
| `extract` | Extract embedded SQL into `.sql` files | `py-sql-cleaner extract jobs/load_users.py --out-dir sql` |
| `dialects` | List accepted dialect values | `py-sql-cleaner dialects` |

## Documentation

Expand All @@ -211,7 +227,8 @@ The current focus is:

- Python files
- triple-quoted SQL strings
- SQLGlot-backed SQL formatting, currently defaulting to the Redshift dialect
- SQLGlot-backed SQL formatting, defaulting to generic SQL with `--dialect`
support for explicitly enabled database-specific formatting
- formatting
- extracting SQL into `.sql` files

Expand Down
97 changes: 97 additions & 0 deletions docs/development/branching.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Branching Strategy

This repository uses a lightweight GitHub Flow. Keep the model simple:
`main` is protected, short-lived branches carry changes, and every change lands
through a pull request.

```mermaid
gitGraph
commit id: "main"s
branch feature/add-postgres-dialect
checkout feature/add-postgres-dialect
commit id: "focused change"
commit id: "tests/docs"
checkout main
merge feature/add-postgres-dialect id: "PR merge"
branch fix/preserve-copy-options
checkout fix/preserve-copy-options
commit id: "bug fix"
checkout main
merge fix/preserve-copy-options id: "PR merge"
```

## Branches

- `main` is the only long-lived branch.
- `main` should always be releasable.
- Do not push commits directly to `main`.
- Do not create a permanent `develop` branch. This project is small enough that
a second integration branch would add delay without improving safety.
- Create short-lived branches from the latest `main`.

Use these branch name prefixes:

- `feature/<short-name>` for user-facing behavior or new capabilities.
- `fix/<short-name>` for bug fixes.
- `docs/<short-name>` for documentation-only changes.
- `chore/<short-name>` for maintenance, tooling, or dependency updates.
- `release/<version>` only when release preparation needs multiple commits.
- `hotfix/<short-name>` for urgent fixes based on `main`.

Examples:

- `feature/add-postgres-dialect`
- `fix/preserve-copy-options`
- `docs/branching-strategy`
- `chore/update-ci-python`
- `release/0.2.0`
- `hotfix/formatter-crash`

## Normal Change Flow

1. Update local `main` from `origin/main`.
2. Create a short-lived branch.
3. Keep commits focused and related to one change.
4. Open a draft pull request early if feedback or CI visibility is useful.
5. Run `scripts/check` locally before marking the pull request ready.
6. Merge only after required checks pass and the review expectation is met.
7. Delete the branch after merge.

Prefer squash merges for small, focused pull requests so `main` stays readable.
Use a regular merge only when preserving multiple commits adds clear value.

## Pull Request Rules

Each pull request should include:

- A short summary of the behavior or documentation change.
- Tests or a note explaining why tests were not needed.
- Any migration, release, or compatibility notes.
- Confirmation that `scripts/check` passed locally, unless CI is the only
practical validation path.

Pull requests should stay small. If a branch grows into unrelated changes, split
it before review.

## Releases

Releases are cut from `main`.

1. Merge release preparation changes through a pull request.
2. Tag the release commit on `main`.
3. Build and publish artifacts from the tagged commit.

Do not release from feature branches.

## Repository Protection

Configure GitHub branch protection for `main`:

- Require pull requests before merging.
- Require status checks to pass before merging.
- Require branches to be up to date before merging when GitHub can enforce it
without creating excessive noise.
- Restrict direct pushes to `main`.

One approval can be optional if required checks are strong, but changes should
still go through a pull request rather than direct pushes.
75 changes: 73 additions & 2 deletions py_sql_cleaner/adapters/sqlglot_formatter.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,27 @@
from __future__ import annotations

from abc import ABC, abstractmethod
from collections.abc import Iterator

import sqlglot
from sqlglot.dialects import Dialect
from sqlglot.tokens import Tokenizer, TokenType

from py_sql_cleaner.domain.config import DEFAULT_BACKEND, DEFAULT_DIALECT
from py_sql_cleaner.domain.errors import FormatterError

SUPPORTED_DIALECTS = ("generic", "mysql", "postgres", "redshift")
REDSHIFT_EXPLICIT_DIALECT_KEYWORDS = (
"COPY",
"UNLOAD",
"IAM_ROLE",
"DISTKEY",
"SORTKEY",
"DISTSTYLE",
"ENCODE",
)
REDSHIFT_PRESERVE_COMMANDS = ("COPY", "UNLOAD")


class FormatterBackend(ABC):
@abstractmethod
Expand All @@ -15,6 +31,13 @@ def format(self, sql: str, dialect: str) -> str:

class SqlglotFormatter(FormatterBackend):
def format(self, sql: str, dialect: str) -> str:
dialect = normalize_dialect(dialect)
if not dialect and _has_redshift_explicit_dialect_keyword(sql):
raise FormatterError("redshift-specific SQL requires --dialect redshift")
if _dialect_name(dialect) == "redshift" and _starts_with_command(
sql, REDSHIFT_PRESERVE_COMMANDS
):
return sql.strip()
try:
expressions = sqlglot.parse(sql, read=dialect)
formatted = ";\n\n".join(expr.sql(dialect=dialect, pretty=True) for expr in expressions)
Expand All @@ -26,11 +49,59 @@ def format(self, sql: str, dialect: str) -> str:
return formatted


def normalize_dialect(dialect: str) -> str:
normalized = dialect.strip().lower()
if normalized == "generic":
return ""
dialect_name = normalized.split(",", maxsplit=1)[0].strip()
if dialect_name not in SUPPORTED_DIALECTS:
supported = ", ".join(SUPPORTED_DIALECTS)
raise FormatterError(f"unsupported SQL dialect: {dialect}. Supported dialects: {supported}")
try:
Dialect.get_or_raise(normalized)
except Exception as exc:
raise FormatterError(f"unsupported SQL dialect: {dialect}. {exc}") from exc
return normalized


def _dialect_name(dialect: str) -> str:
return dialect.split(",", maxsplit=1)[0].strip().lower()


def _has_redshift_explicit_dialect_keyword(sql: str) -> bool:
return any(
token.text.upper() in REDSHIFT_EXPLICIT_DIALECT_KEYWORDS for token in _word_tokens(sql)
)


def _starts_with_command(sql: str, commands: tuple[str, ...]) -> bool:
first_token = next(iter(_word_tokens(sql)), None)
return bool(first_token and first_token.text.upper() in commands)


def _word_tokens(sql: str) -> Iterator:
return (
token
for token in Tokenizer().tokenize(sql)
if token.token_type not in {TokenType.IDENTIFIER, TokenType.STRING, TokenType.UNKNOWN}
and not _is_backtick_quoted_token(sql, token)
)


def _is_backtick_quoted_token(sql: str, token) -> bool:
return (
token.start > 0
and token.end + 1 < len(sql)
and sql[token.start - 1] == "`"
and sql[token.end + 1] == "`"
)

Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

def get_formatter_backend(backend: str) -> FormatterBackend:
if backend != "sqlglot":
if backend != DEFAULT_BACKEND:
raise FormatterError(f"unsupported formatter backend: {backend}")
return SqlglotFormatter()


def format_sql(sql: str, dialect: str = "redshift", backend: str = "sqlglot") -> str:
def format_sql(sql: str, dialect: str = DEFAULT_DIALECT, backend: str = DEFAULT_BACKEND) -> str:
return get_formatter_backend(backend).format(sql.strip(), dialect=dialect)
10 changes: 6 additions & 4 deletions py_sql_cleaner/application/format_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
class FormatSourceResult:
source: str
warnings: list[str]
errors: list[str]


def format_source(
Expand All @@ -24,6 +25,7 @@ def format_source(
formatter: SqlFormatter,
) -> FormatSourceResult:
warnings: list[str] = []
errors: list[str] = []
new_source = source
for block in reversed(blocks):
reason = unsafe_reason(block)
Expand All @@ -35,14 +37,14 @@ def format_source(
continue
try:
formatted = formatter(block.raw_sql, dialect, backend)
except FormatterError:
warnings.append(
except FormatterError as exc:
errors.append(
f"failed to format SQL block {block.file_path}:{block.start_line}-"
f"{block.end_line} with {backend}. Skipping."
f"{block.end_line} with {backend}: {exc}. Skipping."
)
continue
new_source = replace_sql_content(new_source, block, formatted)
return FormatSourceResult(source=new_source, warnings=warnings)
return FormatSourceResult(source=new_source, warnings=warnings, errors=errors)


def unsafe_reason(block: SqlBlock) -> str | None:
Expand Down
Loading
Loading