update lookup policy

CHANGELOG.md

@@ -5,6 +5,23 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.24.0]
+
+### Changed
+
+- **Neo4j**: notifications about labels, properties and relation types nothing has written yet are
+  turned off at the driver. Neo4j 5.7 or later is required.
+- **`lookup_candidates`** (breaking): `Candidate.lane` and `.mention` are replaced by `hits`
+  (`LaneHit {lane, matched}`), the best hit of every lane that found the vertex, best first.
+- **`gather_candidates`** (`graflo.link`, breaking):
+  - `EntityCandidate.lane`, `.matched`, `.relation`, `.anchor` and `.role` are replaced by `lanes`
+    (`CandidateHit`, every lane that found the entity; each lane carries only its own fields).
+  - New `attributes` (`AttributeCheck {field, verdict, compare}`) check the record's other stated
+    properties against stored values, which are never returned.
+  - Key comparison is literal, as the field's declared type.
+  - New `comparable=` limits which stated fields are read; reserved fields never are.
+  - `max_candidates` counts entities after the key check.
+
 ## [1.23.0]
 
 ### Added

docs/concepts/operations/finding_vertices_from_text.md

@@ -11,7 +11,7 @@ It uses no model and no vectors. A candidate is found in one of four ways:
 | `normalized` | A run of words, normalised, equals a stored normalised label field |
 | `alias` | A label lane finds an alias vertex, and its alias edge leads to the target |
 
-Each candidate carries the vertex type, its identity fields, the label it was found by, the lane and the words of the text it came from — never the vertex's other fields.
+Each candidate carries the vertex type, its identity fields, its label and `hits`: for every lane that found it, the words of the text it came from. It never carries the vertex's other fields.
 
 ## A first example
 
@@ -28,8 +28,10 @@ spec = LookupSpec(
 candidates = lookup_candidates(
     conn, schema, "the Berlin pump is down, see SUP-42", spec
 )
-# [Candidate(vertex_type="Ticket", key={"number": "SUP-42"}, lane="key", ...),
-#  Candidate(vertex_type="Product", key={"sku": "P-200"}, label="Berlin pump", lane="alias", ...)]
+# [Candidate(vertex_type="Ticket", key={"number": "SUP-42"},
+#            hits=[LaneHit(lane="key", matched="SUP-42")]),
+#  Candidate(vertex_type="Product", key={"sku": "P-200"}, label="Berlin pump",
+#            hits=[LaneHit(lane="alias", matched="Berlin pump")])]
 ```
 
 ## Label fields
@@ -50,8 +52,8 @@ transforms:
 
 ## Results
 
-- One candidate per vertex, from its best lane, in the order of the table above; within a lane, the longer mention first.
-- At most `max_seeds` candidates (default 20). Runs of up to `max_words` words (default 4) are tried as labels.
+- One candidate per vertex. Its `hits` hold the best hit of each lane that found it, in the order of the table above; within a lane, the longer match wins. Vertices are ordered by their best hit.
+- At most `max_seeds` vertices (default 20). Runs of up to `max_words` words (default 4) are tried as labels.
 - `vertex_types` restricts the types returned. Alias types are not returned unless listed there.
 
 ## Candidates for staged records
@@ -79,7 +81,10 @@ for entry in entries:
 ```
 
 - A candidate never closes a record; only keys do. A pick, a rule or a person chooses.
-- A candidate whose stored key or secondary-identity values contradict what the record states is left out.
+- One candidate per entity. `lanes` lists every way it was found, best lane first. A `relation` hit names the relation and the anchor, a `metadata` hit (added by the caller) a role, and every other hit what matched.
+- `attributes` checks each other property the record states against the entity's stored value: `agree`, `contradict`, or `unknown` when nothing is stored. Stored values are never returned. Equality is literal, as the field's declared type, so `"ANN@x.com"` is not `"ann@x.com"`. A field named in the spec's `normalized_fields` compares in `normalize_label` form (`compare: normalized`).
+- A candidate whose stored key or secondary-identity values contradict what the record states is left out. Any other contradiction is reported, not a bar.
+- `comparable(vertex_type, field)` limits which stated properties are probed, looked up by or checked, keys included. Reserved fields, such as access zones, are never compared.
 - A relation between two records states two entities: a record is never offered the entity a related record of the same source resolved to, and a relation whose two records resolve to one entity is not written. `promote` skips it and `resolution.dropped` lists it (`DroppedRelation`, reason `same_entity`); after fixing records by hand, `dropped_relations(resolution)` recomputes the list.
-- Best lane first, at most `max_candidates` per record (default 10); `truncated` says when more were found.
+- Best lane first, at most `max_candidates` entities per record (default 10), counted after lanes are merged and keys checked; `truncated` says when more were found.
 - `guard` limits every read, as in `resolve_records`.

docs/guides/database_connections.md

@@ -38,6 +38,10 @@ PostgreSQL 15 or later, for the unique key on each edge table. An edge table
 created by an earlier version gains that key at schema definition or on its
 first ingest, which fails with a clear error if the table holds duplicate edges.
 
+GraFlo needs Neo4j 5.7 or later. Its queries are built from the schema, so it
+turns off Neo4j's notifications about labels, properties and relation types
+that nothing has written yet.
+
 A URI without a port gets the database's default. For Neo4j the default
 follows the scheme: `bolt_port` when it is set and otherwise 7687 for `bolt`
 and `neo4j` URIs, 7474 for `http` and `https`.

graflo/db/neo4j/conn.py

@@ -27,7 +27,7 @@
 from collections.abc import Sequence
 from typing import Any, LiteralString, cast
 
-from neo4j import GraphDatabase
+from neo4j import GraphDatabase, NotificationDisabledCategory
 from neo4j.exceptions import ClientError
 
 from graflo.architecture.graph_types import EdgeDirection, GraphContainer, Index
@@ -173,7 +173,15 @@ def __init__(self, config: Neo4jConfig):
         auth = None
         if config.password is not None:
             auth = (config.username or DEFAULT_NEO4J_USERNAME, config.password)
-        self._driver = GraphDatabase.driver(uri=config.url, auth=auth)
+        # Queries are built from the schema, so an unrecognized label, property or
+        # relation is one nothing has written yet, never a typo: no notification.
+        self._driver = GraphDatabase.driver(
+            uri=config.url,
+            auth=auth,
+            notifications_disabled_categories=[
+                NotificationDisabledCategory.UNRECOGNIZED
+            ],
+        )
         self.conn = self._driver.session()
 
     def execute(self, query, **kwargs):

graflo/hq/lookup.py

@@ -17,7 +17,7 @@
 from __future__ import annotations
 
 import re
-from typing import Any, Literal
+from typing import Any, Literal, NamedTuple
 
 from pydantic import Field
 
@@ -100,22 +100,64 @@ class LookupSpec(ConfigBaseModel):
     max_words: int = Field(default=4, ge=1)
 
 
+class LaneHit(ConfigBaseModel):
+    """One way a text names a vertex.
+
+    Attributes:
+        lane: How it was found.
+        matched: The words or key value of the text it was found from.
+    """
+
+    lane: Lane
+    matched: str
+
+
 class Candidate(ConfigBaseModel):
     """A vertex a text may name.
 
     Attributes:
         vertex_type: Logical type of the vertex.
         key: Its identity fields and values.
-        label: The label it was found by, if a label lane found it.
-        lane: How it was found.
-        mention: The text it was found from.
+        label: The label of its best label hit, if a label lane found it.
+        hits: The best hit of each lane that found it, best first.
     """
 
     vertex_type: str
     key: dict[str, Any]
     label: str | None = None
-    lane: Lane
-    mention: str
+    hits: list[LaneHit] = Field(min_length=1)
+
+
+class _Found(NamedTuple):
+    vertex_type: str
+    key: dict[str, Any]
+    label: str | None
+    hit: LaneHit
+
+
+def _rank(hit: LaneHit) -> tuple[int, int]:
+    return _LANE_RANK[hit.lane], -len(hit.matched)
+
+
+def _merge(found: list[_Found]) -> list[Candidate]:
+    """One candidate per vertex with its best hit in each lane, best vertex first."""
+    found = sorted(found, key=lambda f: _rank(f.hit))
+    merged: dict[tuple[str, tuple[tuple[str, str], ...]], Candidate] = {}
+    for item in found:
+        identity_key = tuple(sorted((k, str(v)) for k, v in item.key.items()))
+        candidate = merged.get((item.vertex_type, identity_key))
+        if candidate is None:
+            merged[(item.vertex_type, identity_key)] = Candidate(
+                vertex_type=item.vertex_type,
+                key=dict(item.key),
+                label=item.label,
+                hits=[item.hit],
+            )
+        elif all(hit.lane != item.hit.lane for hit in candidate.hits):
+            candidate.hits.append(item.hit)
+            if candidate.label is None:
+                candidate.label = item.label
+    return list(merged.values())
 
 
 def label_fields(vertex: Vertex) -> list[str]:
@@ -212,12 +254,12 @@ def _label_lanes(
     spec: LookupSpec,
     runs: list[str],
     labels: list[str],
-) -> list[Candidate]:
+) -> list[_Found]:
     """The ``exact`` and ``normalized`` lanes for one type."""
     identity = reader.identity(vertex_type)
     if not identity:
         return []
-    found: list[Candidate] = []
+    found: list[_Found] = []
     normal = [normalize_label(run) for run in runs]
     lanes: list[tuple[Lane, list[str], list[str]]] = [
         ("exact", labels, runs),
@@ -236,15 +278,14 @@ def _label_lanes(
             ).items():
                 for doc in docs:
                     found.append(
-                        Candidate(
-                            vertex_type=vertex_type,
-                            key={f: doc.get(f) for f in identity},
-                            label=next(
+                        _Found(
+                            vertex_type,
+                            {f: doc.get(f) for f in identity},
+                            next(
                                 (doc[f] for f in labels if doc.get(f) is not None),
                                 None,
                             ),
-                            lane=lane,
-                            mention=mentions[position],
+                            LaneHit(lane=lane, matched=mentions[position]),
                         )
                     )
     return found
@@ -260,8 +301,9 @@ def lookup_candidates(
 ) -> list[Candidate]:
     """Vertices of *db* that *text* may name, best first, at most ``max_seeds``.
 
-    One candidate per vertex, from its best lane (``exact``, ``key``,
-    ``normalized``, ``alias``); within a lane, the longer mention first.
+    One candidate per vertex, carrying the best hit of every lane that found
+    it, in the order ``exact``, ``key``, ``normalized``, ``alias``; within a
+    lane the longer match wins. Vertices are ordered by their best hit.
 
     Args:
         db: An open graph connection.
@@ -296,7 +338,7 @@ def labels_of(vertex_type: str) -> list[str]:
         vertex = vertex_by_name.get(vertex_type)
         return label_fields(vertex) if vertex is not None else []
 
-    found: list[Candidate] = []
+    found: list[_Found] = []
     for vertex_type in types:
         found.extend(
             _label_lanes(reader, vertex_type, spec, runs, labels_of(vertex_type))
@@ -317,11 +359,11 @@ def labels_of(vertex_type: str) -> list[str]:
             pattern.vertex_type, pattern.field, values, returned
         ).items():
             found.extend(
-                Candidate(
-                    vertex_type=pattern.vertex_type,
-                    key={f: doc.get(f) for f in identity},
-                    lane="key",
-                    mention=values[position],
+                _Found(
+                    pattern.vertex_type,
+                    {f: doc.get(f) for f in identity},
+                    None,
+                    LaneHit(lane="key", matched=values[position]),
                 )
                 for doc in docs
             )
@@ -337,18 +379,12 @@ def labels_of(vertex_type: str) -> list[str]:
                 hop.alias_type, alias.key, hop.relation, hop.target_type
             ):
                 found.append(
-                    Candidate(
-                        vertex_type=hop.target_type,
-                        key={f: doc.get(f) for f in target_identity},
-                        label=alias.label,
-                        lane="alias",
-                        mention=alias.mention,
+                    _Found(
+                        hop.target_type,
+                        {f: doc.get(f) for f in target_identity},
+                        alias.label,
+                        LaneHit(lane="alias", matched=alias.hit.matched),
                     )
                 )
 
-    found.sort(key=lambda c: (_LANE_RANK[c.lane], -len(c.mention)))
-    best: dict[tuple[str, tuple[tuple[str, str], ...]], Candidate] = {}
-    for candidate in found:
-        identity_key = tuple(sorted((k, str(v)) for k, v in candidate.key.items()))
-        best.setdefault((candidate.vertex_type, identity_key), candidate)
-    return list(best.values())[: spec.max_seeds]
+    return _merge(found)[: spec.max_seeds]