Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 29 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@

This is the Python implementation. The full spec — all locales, all rules, worked examples in
each — lives in [polytypo/polytypo](https://github.com/polytypo/polytypo). This runtime supports
the `text` and `html` modes fully, and `markdown` for the `commonmark` dialect only — `mdx`
raises `POLYTYPO_INVALID_DIALECT` (no MDX/JSX parser is available for Python; see
the `text`, `html` and `yaml` modes fully, and `markdown` for the `commonmark` dialect only —
`mdx` raises `POLYTYPO_INVALID_DIALECT` (no MDX/JSX parser is available for Python; see
[Supported dialects](#supported-dialects)).

## Install
Expand Down Expand Up @@ -59,21 +59,42 @@ transform('<a title="test... wait">Wait... she said "go on."</a>', locale="en-US
# <a title="test... wait">Wait… she said “go on.”</a>
```

`polytypo.text`, `polytypo.html` and `polytypo.markdown` each exclude the parser dependency the
other modes don't need (importing `polytypo.text` never imports `html.parser`-based span
extraction or tree-sitter). The aggregate `polytypo` module supports every mode via a `mode`
keyword, defaulting to `"text"`. `locale` has no default anywhere and must always be passed
explicitly — there is no silent fallback to English.
`polytypo.text`, `polytypo.html`, `polytypo.markdown` and `polytypo.yaml` each exclude the
parser dependency the other modes don't need (importing `polytypo.text` never imports
`html.parser`-based span extraction or tree-sitter; `polytypo.yaml` needs no parser at all). The
aggregate `polytypo` module supports every mode via a `mode` keyword, defaulting to `"text"`.
`locale` has no default anywhere and must always be passed explicitly — there is no silent
fallback to English.

```python
import polytypo

polytypo.transform("...", locale="fr", mode="markdown", dialect="commonmark")
```

`yaml` mode is the one that asks something of you, and it asks for a reason. YAML is a data
format with prose in some of it, so you name the keys whose values are prose; there is no default
and no guess:

```python
from polytypo.yaml import transform

transform(
"summary: Rates -- all of them...\nrun: git diff -- a--b\n", locale="en-US", keys=["summary"]
)
# summary: Rates—all of them…
# run: git diff -- a--b
```

Nothing in YAML's syntax separates a sentence from a shell script: `description` holds one and
`run` holds the other, spelled identically. Quoting, indentation, anchors and a block scalar's
chomping indicator are never decoded and rewritten — the file is located, not re-emitted — so the
trailing newlines of a `|+` block come back exactly as you wrote them. An empty `keys` list is
legal and processes nothing.

`analyze()` runs the same pipeline and reports what it would do instead of doing it — one record
per edit, each with the rule that made it and code-point offsets into the input you passed (into
the **document**, in `html` and `markdown` mode, not into a span):
the **document**, in `html`, `markdown` and `yaml` mode, not into a span):

```python
import polytypo
Expand Down
32 changes: 25 additions & 7 deletions src/polytypo/__init__.py
Original file line number Diff line number Diff line change
@@ -1,20 +1,23 @@
"""polytypo -- locale-correct quotes, dashes, ellipses and no-break spaces.

The aggregate entry point. Its module graph includes every mode's dependencies (`html.parser`
and tree-sitter): it is the only pipeline module that imports all three mode-specific ones.
`polytypo.text`, `polytypo.html` and `polytypo.markdown` each call their own mode-specific
pipeline directly and never import this module (mirrors polytypo-js's `polytypo`/`polytypo/text`
/`polytypo/html`/`polytypo/markdown` split)."""
and tree-sitter): it is the only pipeline module that imports all four mode-specific ones.
`polytypo.text`, `polytypo.html`, `polytypo.markdown` and `polytypo.yaml` each call their own
mode-specific pipeline directly and never import this module (mirrors polytypo-js's
`polytypo`/`polytypo/text`/`polytypo/html`/`polytypo/markdown`/`polytypo/yaml` split). `yaml`
needs no parser at all (modes.md 3.8.1), so it is the lightest of the four."""

from __future__ import annotations

from polytypo._engine.mode_pipelines import (
analyze_html_pipeline,
analyze_markdown_pipeline,
analyze_text_pipeline,
analyze_yaml_pipeline,
run_html_pipeline,
run_markdown_pipeline,
run_text_pipeline,
run_yaml_pipeline,
)
from polytypo._engine.origin import Change
from polytypo.errors import POLYTYPO_INVALID_MODE, PolytypoError
Expand All @@ -27,10 +30,11 @@
def _resolve_mode(mode: str | None) -> str:
if mode is None or mode == "text":
return "text"
if mode in ("html", "markdown"):
if mode in ("html", "markdown", "yaml"):
return mode
raise PolytypoError(
POLYTYPO_INVALID_MODE, f'Unknown mode "{mode}". Expected "text", "html" or "markdown".'
POLYTYPO_INVALID_MODE,
f'Unknown mode "{mode}". Expected "text", "html", "markdown" or "yaml".',
)


Expand All @@ -40,18 +44,27 @@ def transform(
locale: str,
mode: str | None = None,
dialect: str | None = None,
keys: object = None,
rules: dict[str, bool] | None = None,
narrow_nbsp: str | None = None,
) -> str:
"""`narrow_nbsp="nbsp"` makes the engine emit U+00A0 everywhere it would emit U+202F
(nbsp.md 3.1a). It moves the rule's target rather than post-processing the output, so the
result stays a fixed point. Validation order is mode -> narrow_nbsp -> rules -> locale ->
dialect, and the check runs whether or not `nbsp` is enabled."""
dialect -> keys, and the check runs whether or not `nbsp` is enabled.

`keys` is required when `mode="yaml"` and ignored otherwise (modes.md 3.8.2): YAML is a data
format with islands of prose in it, so the caller names the mapping keys whose values are
prose and the library never guesses. An empty sequence is legal and processes nothing."""
resolved_mode = _resolve_mode(mode)
if resolved_mode == "text":
return run_text_pipeline(input, locale=locale, rules=rules, narrow_nbsp=narrow_nbsp)
if resolved_mode == "html":
return run_html_pipeline(input, locale=locale, rules=rules, narrow_nbsp=narrow_nbsp)
if resolved_mode == "yaml":
return run_yaml_pipeline(
input, locale=locale, keys=keys, rules=rules, narrow_nbsp=narrow_nbsp
)
return run_markdown_pipeline(
input, locale=locale, dialect=dialect, rules=rules, narrow_nbsp=narrow_nbsp
)
Expand All @@ -63,6 +76,7 @@ def analyze(
locale: str,
mode: str | None = None,
dialect: str | None = None,
keys: object = None,
rules: dict[str, bool] | None = None,
narrow_nbsp: str | None = None,
) -> list[Change]:
Expand All @@ -79,6 +93,10 @@ def analyze(
return analyze_text_pipeline(input, locale=locale, rules=rules, narrow_nbsp=narrow_nbsp)
if resolved_mode == "html":
return analyze_html_pipeline(input, locale=locale, rules=rules, narrow_nbsp=narrow_nbsp)
if resolved_mode == "yaml":
return analyze_yaml_pipeline(
input, locale=locale, keys=keys, rules=rules, narrow_nbsp=narrow_nbsp
)
return analyze_markdown_pipeline(
input, locale=locale, dialect=dialect, rules=rules, narrow_nbsp=narrow_nbsp
)
62 changes: 58 additions & 4 deletions src/polytypo/_engine/mode_pipelines.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
"""The three mode-specific top-level pipelines, one per public entry point
(`polytypo`/`polytypo.text`/`polytypo.html`/`polytypo.markdown`). Mirrors polytypo-js's
"""The four mode-specific top-level pipelines, one per public entry point
(`polytypo` and `polytypo.text`/`.html`/`.markdown`/`.yaml`). Mirrors polytypo-js's
text-pipeline.ts/html-pipeline.ts/markdown-pipeline.ts split: each function imports only what its
mode needs, so `polytypo.text` never imports the HTML or Markdown parsers, and `polytypo.html`
never imports tree-sitter-markdown.

Validation order is public, tested behaviour and is identical across all three: `rules` (an
unknown rule id) before `locale` (an unknown locale) before `dialect`/parsing -- see
Validation order is public, tested behaviour and is identical across all four: `rules` (an
unknown rule id) before `locale` (an unknown locale) before `dialect`/`keys`/parsing -- see
`pipeline.prepare`'s own docstring."""

from __future__ import annotations
Expand All @@ -16,6 +16,7 @@
from polytypo._engine.pipeline import prepare, run_rules, run_rules_recording
from polytypo._engine.registry import RuleContext
from polytypo._engine.span_runner import analyze_over_spans, run_over_spans
from polytypo._engine.yaml_keys import resolve_yaml_keys
from polytypo._modes.spans import normalize_spans


Expand Down Expand Up @@ -82,6 +83,34 @@ def run_markdown_pipeline(
return run_over_spans(input_text, spans, plan, locale_data, ctx)


def run_yaml_pipeline(
input_text: str,
*,
locale: object,
keys: object,
rules: dict[str, bool] | None,
narrow_nbsp: str | None = None,
) -> str:
"""`yaml` mode only, and the only mode pipeline with **no parser dependency at all** -- span
selection is the specified scan of modes.md 3.8, not a library. There is likewise no
POLYTYPO_MALFORMED_INPUT counterpart here: with no declared grammar to violate, a file that is
not YAML yields few spans or none and comes back byte for byte (modes.md 3.8.3). The only
throw this mode adds is `keys`, which is about the call and not the input."""
from polytypo._modes.yaml import yaml_spans

narrow_target = resolve_narrow_target(narrow_nbsp)
resolved_locale, locale_data, plan = prepare(locale, rules)
resolved_keys = resolve_yaml_keys(keys)
ctx: RuleContext = {
"mode": "yaml",
"dialect": None,
"locale": resolved_locale,
"narrow_target": narrow_target,
}
spans = normalize_spans(yaml_spans(input_text, resolved_keys))
return run_over_spans(input_text, spans, plan, locale_data, ctx)


def analyze_text_pipeline(
input_text: str,
*,
Expand Down Expand Up @@ -150,3 +179,28 @@ def analyze_markdown_pipeline(
}
spans = normalize_spans(markdown_spans(input_text, dialect))
return analyze_over_spans(input_text, spans, plan, locale_data, ctx)


def analyze_yaml_pipeline(
input_text: str,
*,
locale: object,
keys: object,
rules: dict[str, bool] | None,
narrow_nbsp: str | None = None,
) -> list[Change]:
"""analyze.md section 1, `yaml` mode: offsets are into the document, not into a span
(analyze.md section 6)."""
from polytypo._modes.yaml import yaml_spans

narrow_target = resolve_narrow_target(narrow_nbsp)
resolved_locale, locale_data, plan = prepare(locale, rules)
resolved_keys = resolve_yaml_keys(keys)
ctx: RuleContext = {
"mode": "yaml",
"dialect": None,
"locale": resolved_locale,
"narrow_target": narrow_target,
}
spans = normalize_spans(yaml_spans(input_text, resolved_keys))
return analyze_over_spans(input_text, spans, plan, locale_data, ctx)
32 changes: 32 additions & 0 deletions src/polytypo/_engine/yaml_keys.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
"""modes.md 3.8.2: ``yaml`` mode's ``keys`` option, resolved once at the call boundary.

It has **no default**, for the reason ``dialect`` has none. YAML is a data format with islands of
prose in it, and nothing in its syntax marks them -- ``description`` holds a sentence and ``run``
holds a shell script, spelled identically -- so a default would be a guess about the schema above
the document. An **empty list is legal**: "process nothing" is a choice a caller may make, not an
error.

Checked after ``locale``, last of the option checks, alongside ``dialect`` (ARCHITECTURE.md
section 7). The two never both apply, since each belongs to a different mode."""

from __future__ import annotations

from collections.abc import Sequence

from polytypo.errors import POLYTYPO_INVALID_OPTION, PolytypoError


def resolve_yaml_keys(value: object) -> frozenset[str]:
if isinstance(value, str) or not isinstance(value, Sequence):
raise PolytypoError(
POLYTYPO_INVALID_OPTION,
'"keys" is required when mode is "yaml" and must be a sequence of strings '
f"(modes.md 3.8.2). Received {'nothing' if value is None else repr(value)}.",
)
for key in value:
if not isinstance(key, str):
raise PolytypoError(
POLYTYPO_INVALID_OPTION,
f'"keys" must contain only strings; received {key!r}.',
)
return frozenset(value)
20 changes: 16 additions & 4 deletions src/polytypo/_modes/spans.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@

LINE_TERMINATORS = frozenset({0x0A, 0x0D, 0x0B, 0x0C, 0x85, 0x2028, 0x2029})

#: U+0020, the one emitted code point whose meaning is positional (modes.md 3.4, 5 item 2).
SPACE = 0x20


@dataclass(frozen=True, slots=True)
class Span:
Expand Down Expand Up @@ -99,8 +102,12 @@ def filter_boundary_edits(cp: list[int], edits: list[Edit], ranges: list[SpanRan
1. No edit may contain a marker -- one that does is a bug, discarded rather than
redistributed.
2. The edge-growth rule: an edit is discarded if it would place code points at an extremity
of its span that were not there before (p = s0 and r > d, or q = s1 and r > d, with
d = q - p + 1 the replaced length and r the replacement length).
of its span that were not there before. That sentence is the rule; "r > d" alone is an
incorrect formalisation of it and misses r == d. dashes P3 admits a run of THREE dashes,
so "---" -> U+0020 en-dash U+0020 is 3 -> 3: the length test sees nothing while U+0020
lands on both extremities anyway. The second clause tests the CHARACTER, and only U+0020
needs testing -- it is the one code point any rule emits whose meaning comes from its
position rather than from itself (modes.md 5 item 2).

Deletion at an edge is NOT restricted here -- r > d is always false for a deletion, so this
filter never sees one; that case is spaces.md 3.2 step 4's own edge-as-NONE clause instead.
Expand All @@ -115,8 +122,13 @@ def filter_boundary_edits(cp: list[int], edits: list[Edit], ranges: list[SpanRan
d = edit.end - edit.start
r = len(edit.replacement)
span = _span_containing(ranges, p)
if span is not None and r > d and (p == span.first or q == span.last):
continue
if span is not None and (p == span.first or q == span.last):
if r > d:
continue
if r > 0 and p == span.first and edit.replacement[0] == SPACE and cp[p] != SPACE:
continue
if r > 0 and q == span.last and edit.replacement[r - 1] == SPACE and cp[q] != SPACE:
continue

out.append(edit)
return out
Expand Down
Loading
Loading