Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions docs/conventions/resources.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,8 +133,10 @@ has `get` and `get-by-id`, plain `get` decides), so `ycli … list -o name | xar
works. `-o name` prints it, and a command whose result has no such field is refused before
any request. An item of a listing with no identifier of its own stands in `NO_NAME` of
`tests/architecture/test_names.py` with its reason. The models generated from the DataLens
specification are not marked yet: their resources will name the identifier where the
generator reads it.
specification get the mark from the generator: `_IDENTIFIERS` in
`scripts/gen_datalens_models.py` names the key of each section of the specification (or
`_NO_IDENTIFIER` says why it has none), and `_RETURNS_ANOTHER_KIND` the operations whose
reply holds objects of another kind.

---

Expand Down
2 changes: 1 addition & 1 deletion docs/en/how-to/script-the-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,9 @@ new=$(ycli tracker issues create --queue TEST --summary 'From a script' -o name)

- An object that lies inside another one prints its own identifier: `ycli tracker comments list TEST-1 -o name` prints comment ids, and the issue key is yours to add.
- A command whose result has no identifier (a grant of access, a count) is refused before it sends anything, with exit code 2.
- A DataLens command that starts a long operation (a cloud environment, a Spark or Trino cluster) answers with the operation, so `-o name` prints the operation's id, the one `ycli datalens lakehouseoperations get` takes, not the id of what is being made.
- A reply that leaves the identifier of an item out is an error: nothing is printed, so a script never acts on fewer objects than were listed.
- With `--dry-run` the plan comes back in place of the object, and it is printed as JSON.
- DataLens commands do not print names yet.

## Take a long listing in pieces

Expand Down
2 changes: 1 addition & 1 deletion docs/ru/how-to/script-the-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,9 @@ new=$(ycli tracker issues create --queue TEST --summary 'From a script' -o name)

- Объект, который лежит внутри другого, печатает свой идентификатор: `ycli tracker comments list TEST-1 -o name` печатает id комментариев, а ключ задачи добавляете вы.
- Команде, у результата которой нет идентификатора (выдача доступа, счётчик), отказывают до отправки запроса, код возврата 2.
- Команда DataLens, которая запускает долгую операцию (облачное окружение, кластер Spark или Trino), отвечает операцией, поэтому `-o name` печатает id операции — тот, что принимает `ycli datalens lakehouseoperations get`, а не id создаваемого объекта.
- Ответ, в котором у объекта нет идентификатора, — это ошибка: ничего не печатается, поэтому скрипт не обработает меньше объектов, чем было в списке.
- С `--dry-run` вместо объекта приходит план, и он печатается как JSON.
- Команды DataLens имён пока не печатают.

## Длинный список по частям

Expand Down
42 changes: 21 additions & 21 deletions scripts/datalens_schemas.sha256
Original file line number Diff line number Diff line change
@@ -1,36 +1,36 @@
d4da030189150eaa647c63816c298abd279c661822f5ead6b1ce0086af78ef87 __init__.py
cc78dadc9f3ccb748213d701738c47d1d6243da4a068be7ad9c1b3f52f8d19fe access.py
fa198200babe9e1ba210f6fdd71a2c4a86a62860ceb8ae323dd16e292337d480 audit.py
1204722677ff5b338fe6d4fad9342f9a09782be86fd7b85b05822d7ba9de3574 access.py
f0e012b853d8c0acd2a3706c7fe5a28985c6b8756e05d9b09061d64403fad61b audit.py
e10c5765c9d57f100994680815c7571cb1b4e025623098d13feac39592a3da5c cloud_environment_storage.py
ef89732c72b852ed0834f28446cada5abbe64c8f8816d132bd1a096ef018bfcf cloud_environments.py
00b2f07711b669204a64e6b993ea86856579ad7dff47bc61d101c592ae0f5cf9 collection.py
65a4c8a13d688f8ec8d826da14b8507d90376e7c19365a031aea4d51d87c45d8 connection.py
024afe0197430fc56f770b5cca97b4c2f2841517edd9052e02cc631f86dd8d88 cloud_environments.py
6b954a5659ae499e5c61ba5761fb3f2061b8e61ce5dcc99dd856083e6dab2fcf collection.py
4dbbcb6e4448b96aa429ad4cbbc0229c888028f1c4d86dd032f3350bb5b7c0e1 connection.py
7d8e086b5f1a9160a6267cb1b0565646e172d47fc4c99bb97f6465802a8c638d dashboard.py
8221cb538ba20d8925975e3122d924514b829cbd4069f5c0aee7ed6f245bfb72 data.py
f9efb961a99c1207768c0625821389dcd3595a81bf1691118a94d74dd48beb2f dataset.py
77d5a961a803628ff7c7299b1a4c047b33038a9c4b26c676a4b448ecabe5c9df dataset.py
11d59eae9dab5ebed2873ff1ab328bd0229f28ca59fb6d7429f6e5ca5a5c4bd0 editor.py
e1e8a0118aa1d3cfcd48dfe33a096714c65a7e87114176541bf099da426f96d8 embedding_secrets.py
a5800b847aa019f81329833fe28e3823ce42d9ee607e648b6e99acee1a4bdc10 embeds.py
878dfcfb5d640d231efd8d0993c2064d02b711d1774c7a931d521db99f7fa168 entries.py
6b8f7ec8053181c928c14dc50aeb727e8c82cf8a883a5eae37778150ab304cc6 entry_lock.py
ce3be106f1236b30e9c384d20b73090161ee606f78b954e85bef19e4847b1bad embedding_secrets.py
a9c9f5dee262fbc0db60cfcd3ac7dad9a741361f002737af674a01949f13f59a embeds.py
f2b3d03d70c85db44fd23dc661e4b9ba37973df4546d50c97df6fa8a57152917 entries.py
dcfb8803b0af388376e77369d3d5bdbfe31aedd9bc5ba11c2e37ae59d5747178 entry_lock.py
16a8cd8d0de9e26cb20ca8048e57e61a82cad3edbd2cadc3691f8a295674d22c favorites.py
9b9d22497a187f120dd81712c1a6efb8f5936db8fa52ca5b2824e1661ba889fe folder.py
f4558dffe77955f1a0cf4aaebb5fa351ecb8dd696f3d01d8c3c1268061a3f713 html_pages.py
ff0fc2461a6f0e0c6aa39a7350fce33d0f45f298768fce0d483edf46b62f73dd html_pages.py
a2944d4ad4e3dd23aaeb641801bafb825e1621a38238194226925b4f5a2c99ee lakehouse_operations.py
32679a24c9b1c2bb50d77f6d8903f2a4c4532d84c7b17679ad0f1d86cb109c8d licensing.py
3f28a3426aecd531abf24774b1762a11ca43860fa759185dd6a2571cbcc903e3 navigation.py
bee8ea2dee160e38057d32b7fa0b9a90390888f4f44bb5da8e991da008375064 licensing.py
733300f988cf96804d1397ba0e73f5f51f5cb86df4fcee327d499f0358be04e9 navigation.py
5fb735a13f5958b75052c1f8e486c0987933a424f3f3cc31f9ed9bbfe2a09a46 permissions.py
007e77c5ce13e8a59145112ee40219fcec55973b9f6f095cb48bab7b419b951d ql.py
bf9a01057114a3fda9e1854e9df59f77bf81009b43f49985ed790ebb818637d7 reports.py
660c36e0d9980caaa7ab77b81a7ae91d32b9c9c575be1aa74997edf3f2504c6f rest_catalogs.py
72278dc25dc6ce3a56c0910d8498ed2fa6703bcf1457b8e5622368a953481c20 shared.py
c845b1acfd36711850ca78a649a9ac9352705d9d87d193464c3af06ae5e43526 rest_catalogs.py
a9601ed30129141d23efab390f6e3812a99b8e07e3a059a2673c37dbbe50c8a0 shared.py
186d90efa73a171d65c47a0f31e63fcd0988801e78f029a17659328106160e10 shared_entry.py
c7c5f6dbdb8037594e0526c425d6702026013566cd7d882968a60941bbc30d77 spark_applications.py
c007eaa70d4622c8e4c31863c1d80104e73f9145624dd2781236fc109f4a7de0 spark_clusters.py
a154501fcef69a2c307ef0ee20f4034e6a8e594cb5115f2cf847740f7be23d56 spark_applications.py
9f456027e2ce1b27ef9747a1033875e3f6f692fa7126e2db270fe3c3a4efabbc spark_clusters.py
5fe0117796e50b5bd88b4fa2e1310c9ee9c73a650171a1ae9f6870e917fd0d79 sql_queries.py
5e61dbf756b08efc5dee106611379bf15bde50c1fb6c3725a4137f4a77f21bc9 tenants.py
a691214c05a4267714d4c646580fa23374370a7e85675fb508a1ceca1fa5dcb2 trino_clusters.py
33c3cb4318099d33c18ad4fbe82093583c7d2b801ce610abba98745eb69214ef trino_clusters.py
b60c79744cf6e230401b5ac593df912ce683d1dc074886944411617e16159d15 wizard.py
37d89fa2d64450e2a49e5f9d8dc06239f55d137c2358c22b8a05919422ed2886 workbook.py
b8d351d03d0191744034b9b94c2278600b5513926136b86cd3bdb7ce44df3699 workbook_export.py
5a22a5c3cd2d32cca64c1e5e5e508b822f64120b2a924fbfbdb3b83e45f5d3f8 workbook_import.py
937d3bf38e786643985870d78e7beca1597a206e0ad665762fe4997485aea83a workbook.py
6c8300eabeeeca4cd603ead573f99a5f43361e7e22dd9d84a3c6326c5fefdc03 workbook_export.py
69c0d72090bcc5470155c83e5f06eb5ae3944a69a77878893894c8e6fdce225b workbook_import.py
5 changes: 5 additions & 0 deletions scripts/datalens_templates/pydantic_v2/BaseModel.jinja2
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ class {{ class_name }}({%- if schema_runtime_validation_use_base -%}{{ schema_ru
{#- ycli: a field the API requires as null carries the typed mark (gen_datalens_models) #}
{%- if field.original_name in (no_drop_null or []) %}
{{ field.name }}: Annotated[{{ field.type_hint }}, NoDropNull()] = {{ field.field }}
{#- ycli: the field that names the object carries the typed mark (gen_datalens_models) #}
{%- elif field.original_name in (identity or []) %}
{{ field.name }}: Annotated[{{ field.type_hint }}, Identity()] = {{ field.field }}
{%- else %}
{{ field.name }}: {{ field.type_hint }} = {{ field.field }}
{%- endif %}
Expand All @@ -38,6 +41,8 @@ class {{ class_name }}({%- if schema_runtime_validation_use_base -%}{{ schema_ru
{{ field.name }}: {{ field.annotated }}
{%- elif field.original_name in (no_drop_null or []) %}
{{ field.name }}: Annotated[{{ field.type_hint }}, NoDropNull()]
{%- elif field.original_name in (identity or []) %}
{{ field.name }}: Annotated[{{ field.type_hint }}, Identity()]
{%- else %}
{{ field.name }}: {{ field.type_hint }}
{%- endif %}
Expand Down
164 changes: 158 additions & 6 deletions scripts/gen_datalens_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,9 +74,9 @@
_BASE = "customBasePath"
_CLOSED = "ycli.yandex.models.RequestBody"
# What a generated module may import, inherit from and call (``foreign``).
_MODULES = {"typing", "datetime", "pydantic", "ycli.yandex.models"}
_MODULES = {"typing", "datetime", "pydantic", "ycli.yandex.models", "ycli.yandex.sync.marks"}
_BASES = {"APIModel", "RequestBody", "RootModel"}
_CALLS = {"Field", "ConfigDict", "NoDropNull", "KindByOwnField"}
_CALLS = {"Field", "ConfigDict", "NoDropNull", "KindByOwnField", "Identity"}


def _snake(name: str) -> str:
Expand Down Expand Up @@ -709,6 +709,7 @@ def _mark_union_envelopes(spec: dict[str, Any]) -> int:


_KEEPS_NULL = "x-ycli-no-drop-null"
_NAMES = "x-ycli-identity"
TEMPLATES = ROOT / "scripts" / "datalens_templates"


Expand Down Expand Up @@ -748,6 +749,154 @@ def _mark_nulls_to_keep(spec: dict[str, Any]) -> int:
return marked


# The key of a reply that says which object it is: what the command that reads the object
# takes, so ``-o name`` can print it. One line for each section of the specification. The
# specification itself names it in 9 sections of 34 (the required argument of ``get…`` that
# its reply repeats); elsewhere the reply wraps the object, names the key otherwise than the
# request does, or the section has no ``get``.
_IDENTIFIERS = {
"Access": "sub",
"Audit": "entryId",
"CloudEnvironments": "id",
"Collection": "collectionId",
"Connection": "id",
"Dataset": "id",
"EmbeddingSecrets": "embeddingSecretId",
"Embeds": "embedId",
"Entries": "entryId",
"EntryLock": "entryId",
"HtmlPages": "entryId",
"LakehouseOperations": "id",
"Licensing": "userId",
"Navigation": "entryId",
"RestCatalogs": "id",
"SparkApplications": "id",
"SparkClusters": "id",
"TrinoClusters": "id",
"Workbook": "workbookId",
"WorkbookExport": "exportId",
"WorkbookImport": "importId",
}
# A section whose replies are no object a command reads back, and why.
_NO_IDENTIFIER = {
"CloudEnvironmentStorage": "a file of a bucket is told by its path, which the caller gave",
**dict.fromkeys(
("Dashboard", "Editor", "Reports", "SqlQueries", "Wizard"),
"its replies wrap the object in `entry`; a mark that reaches into a wrapper is not decided",
),
"Data": "rows of data, not objects",
"Favorites": "not wrapped: wrapping the section is not decided",
"Folder": "not wrapped: the old placement model",
"Permissions": "what the caller may do, not an object",
"QL": "its replies are read into a model written by hand",
"SharedEntry": "who has access, told by the subject and the role together",
"Tenants": "the one tenant of the caller",
}
# An operation whose reply holds objects of another kind than its section's, which carry the
# section's key as a reference to their parent: the key of what it returns, or ``None`` where
# it returns several kinds and an identifier alone would not say which command reads it.
_RETURNS_ANOTHER_KIND: dict[str, str | None] = {
"getWorkbookEntries": "entryId",
"getCollectionContent": None,
"getRevisions": "revId",
}


def _rows(schemas: dict[str, Any], reply: dict[str, Any]) -> set[str]:
"""The schemas a command prints as rows: the reply itself, or the items of a list in it.

A reply that is one of several kinds is each of them. What lies deeper (the fields of a
dataset, the object inside an ``entry`` wrapper) is not a row: no command reads it by
the key it holds.
"""
found: set[str] = set()

def take(schema: Any, *, lists: bool) -> None:
if not isinstance(schema, dict):
return
name = schema.get("$ref", "").removeprefix(_REF)
if name and name not in found:
found.add(name)
take(schemas[name], lists=lists)
for member in (*schema.get("oneOf", []), *schema.get("anyOf", [])):
take(member, lists=lists)
take(schema.get("items"), lists=False)
if lists: # the list a reply wraps: ``{entries: [...], nextPageToken}``
for field in (schema.get("properties") or {}).values():
held = schemas.get(field.get("$ref", "").removeprefix(_REF), field)
take(held.get("items"), lists=False)

take(reply, lists=True)
return found


def _mark_identifiers(spec: dict[str, Any]) -> int:
"""Mark the field that names the object in every row of a reply; return how many.

A row (:func:`_rows`) takes the key of the section whose operation returns it, or the key
:data:`_RETURNS_ANOTHER_KIND` gives that operation. The mark becomes
``Annotated[..., Identity()]`` in the generated class (``ycli.yandex.sync.marks``).

Raises:
SystemExit: A section stands in neither table or in both; an operation of
:data:`_RETURNS_ANOTHER_KIND` is not in the specification; a key that no reply
schema it is declared for has; a schema that two keys would name.
"""
schemas = spec["components"]["schemas"]
keys: dict[str, set[str]] = {} # the keys declared for each schema, by what reaches it
marked: dict[str, int] = {} # how many schemas each declaration named
replies = [(name, tag, body) for name, tag, kind, body in _bodies(spec) if kind == "Response"]
sections = {tag for _, tag, _, _ in _bodies(spec)}
undeclared = sections - _IDENTIFIERS.keys() - _NO_IDENTIFIER.keys()
twice = _IDENTIFIERS.keys() & _NO_IDENTIFIER.keys()
gone = _RETURNS_ANOTHER_KIND.keys() - {name for name, _, _ in replies}
stale = (_IDENTIFIERS.keys() | _NO_IDENTIFIER.keys()) - sections
if undeclared or twice or gone or stale:
raise SystemExit(
"the identifier of a section is declared once, in _IDENTIFIERS or _NO_IDENTIFIER: "
f"neither {sorted(undeclared)}, both {sorted(twice)}, no such section "
f"{sorted(stale)}, no such operation {sorted(gone)}"
)
reached: dict[str, set[str]] = {} # the schemas each declaration reaches
for operation, tag, holder in replies:
declared = operation if operation in _RETURNS_ANOTHER_KIND else tag
key = _RETURNS_ANOTHER_KIND.get(operation, _IDENTIFIERS.get(tag))
seen = _rows(schemas, holder["schema"])
reached.setdefault(declared, set()).update(seen)
marked.setdefault(declared, 0)
for name in seen:
keys.setdefault(name, set()).update([key] if key else [])
# What an operation of another kind returns is named by that operation alone.
apart = set().union(*(reached.get(name, set()) for name in _RETURNS_ANOTHER_KIND))
for name in apart:
keys[name] = {
key
for operation in _RETURNS_ANOTHER_KIND
if name in reached[operation]
if (key := _RETURNS_ANOTHER_KIND[operation])
}
total = 0
for name, declared_keys in sorted(keys.items()):
properties = schemas[name].get("properties")
held = sorted(declared_keys & properties.keys()) if isinstance(properties, dict) else []
if len(held) > 1:
raise SystemExit(f"{name} would be named by two keys: {held}")
for key in held:
properties[key][_NAMES] = True
total += 1
for declared, names in reached.items():
own = _RETURNS_ANOTHER_KIND.get(declared, _IDENTIFIERS.get(declared))
marked[declared] += name in names and own == key
unused = sorted(
declared
for declared, count in marked.items()
if not count and _RETURNS_ANOTHER_KIND.get(declared, _IDENTIFIERS.get(declared))
)
if unused:
raise SystemExit(f"no reply schema has the key declared for: {unused}")
return total


def _union_members(spec: dict[str, Any]) -> set[int]:
"""The objects that are members of a union of objects, by ``id``."""
members: set[int] = set()
Expand Down Expand Up @@ -931,6 +1080,7 @@ def prepare(spec: dict[str, Any]) -> dict[str, Any]:
_close_only_requests(spec)
_mark_union_envelopes(spec)
_name_every_object(spec)
_mark_identifiers(spec)
section = _sections(spec)
schemas = {
name: schema for name, schema in spec["components"]["schemas"].items() if name in section
Expand Down Expand Up @@ -1095,15 +1245,15 @@ def foreign(text: str) -> list[str]:
return found


def _taken_marks(schema: dict[str, Any]) -> list[str]:
"""The fields of ``schema`` marked to keep their null, with the mark taken off them."""
def _taken_marks(schema: dict[str, Any], mark: str = _KEEPS_NULL) -> list[str]:
"""The fields of ``schema`` that carry ``mark``, with the mark taken off them."""
properties = schema.get("properties")
if not isinstance(properties, dict):
return []
return [
name
for name, field in properties.items()
if isinstance(field, dict) and field.pop(_KEEPS_NULL, False)
if isinstance(field, dict) and field.pop(mark, False)
]


Expand All @@ -1123,6 +1273,8 @@ def generate(spec: dict[str, Any]) -> dict[str, str]:
data: dict[str, Any] = {}
if kept := _taken_marks(schema):
data["no_drop_null"] = kept
if named := _taken_marks(schema, _NAMES):
data["identity"] = named
if schema.pop(_TOLD_APART, False):
data["kind_by_own_field"] = True
if data:
Expand All @@ -1148,7 +1300,7 @@ def generate(spec: dict[str, Any]) -> dict[str, str]:
*(
"--additional-imports",
"typing.Annotated,ycli.yandex.models.NoDropNull,"
"ycli.yandex.models.KindByOwnField",
"ycli.yandex.models.KindByOwnField,ycli.yandex.sync.marks.Identity",
),
*("--base-class", "ycli.yandex.models.APIModel", "--openapi-scopes", "schemas"),
*("--enum-field-as-literal", "all", "--treat-dot-as-module"),
Expand Down
3 changes: 2 additions & 1 deletion src/ycli/cli/output.py
Original file line number Diff line number Diff line change
Expand Up @@ -439,7 +439,8 @@ def render(self, result: BaseModel, console: Console) -> None:
more = f" (and {len(unnamed) - 1} more)" if len(unnamed) > 1 else ""
key = ".".join(columns[first - 1] or ())
raise YandexUnexpectedReplyError(
f"item {first} of this listing has no {key}{more}: `-o name` cannot name it"
f"item {first} of this listing has no {key}{more}: `-o name` cannot name it. "
"The request was sent: a command that changes something has changed it"
)
console.file.writelines(name + "\n" for name in names)

Expand Down
5 changes: 3 additions & 2 deletions src/ycli/yandex/datalens/schemas/access.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
# Generated from the DataLens specification by scripts/gen_datalens_models.py; do not edit by hand.

from typing import Any, Literal
from typing import Annotated, Any, Literal

from pydantic import Field

from ycli.yandex.models import APIModel, RequestBody
from ycli.yandex.sync.marks import Identity


class AccessExtSubjectClaims(APIModel):
sub: str | None = Field(default=None, description="Subject identifier.")
sub: Annotated[str | None, Identity()] = Field(default=None, description="Subject identifier.")
sub_type: (
Literal[
"SUBJECT_TYPE_UNSPECIFIED",
Expand Down
Loading
Loading