GraFlo

Describe a property graph once in YAML; load it from files, SQL, RDF, APIs or Kafka into the database of your choice.

GraFlo is a Python library that turns records from files, SQL databases, RDF, REST APIs or Kafka topics into a labeled property graph. You describe the graph once, in a YAML file called a manifest, and GraFlo creates the schema and writes the vertices and edges into the graph database of your choice, or into a directory on disk.

It is for engineers who build a graph from several sources and want its description in one reviewable file rather than spread across load scripts.

Documentation: growgraph.github.io/graflo

What you can do with it

• Describe a graph once and load data into it. A manifest names the vertex and edge types, says which properties identify a vertex, and says how each kind of record becomes vertices and edges. The same manifest loads into ArangoDB, Neo4j, TigerGraph, FalkorDB, Memgraph, NebulaGraph, PostgreSQL or the file backend, and records with the same identity become one vertex. GraFlo also copies an existing graph from Neo4j, ArangoDB or PostgreSQL into another database (GraphEngine.migrate_graph).
• Change the description over time, with a recorded history. Renaming a type, combining two types or changing a property type is a typed operation. Operations are recorded as commits (graflo commit, log, checkout, verify, revert) that you can replay, check and, for most operations, undo. Two branches of changes to one manifest are reconciled with a three-way merge (graflo merge3), and two manifests written by different teams are combined into one with a union (graflo merge).
• Check and infer descriptions. GraFlo infers a manifest from a PostgreSQL database or an OWL ontology, proposes the properties that identify a record from sample data, and checks a manifest against a conformance profile (graflo check), a set of modeling rules such as "every vertex type declares its identity".

A taste

A manifest has three blocks: schema says what the graph looks like, ingestion_model says how records map onto it, and bindings says where the records come from. This one reads CSV files with the columns person_id, person and department:

schema:
    metadata: {name: hr}
    graph:
        vertex_config:
            vertices:
            -   {name: person, properties: [id, name], identity: [id]}
            -   {name: department, properties: [name], identity: [name]}
        edge_config:
            edges: [{source: person, target: department}]
ingestion_model:
    resources:
    -   name: departments
        pipeline:
        -   {vertex: person, from: {id: person_id, name: person}}
        -   {vertex: department, from: {name: department}}
bindings:
    connectors:
    -   {regex: "^dep.*\\.csv$", sub_path: data, resource_name: departments}

This loads it into ArangoDB:

from graflo import GraphEngine, GraphManifest
from graflo.connections import ArangoConfig

manifest = GraphManifest.from_yaml("manifest.yaml")
manifest.finish_init()
engine = GraphEngine()
engine.define_and_ingest(manifest=manifest, target_db_config=ArangoConfig.from_env())

ArangoConfig.from_env() reads ARANGO_URI, ARANGO_USERNAME, ARANGO_PASSWORD and ARANGO_DATABASE; every database has such a class. See Database connections.

Documentation

Full documentation: growgraph.github.io/graflo

• Quick start: two CSV files into a graph, step by step
• Creating a manifest: the three blocks of a manifest
• Examples: runnable examples, one question each, with their data under examples/
• Concepts: schema, identity, ingestion, connectors, evolution and version control
• Guides: database connections, graph migration, schema inference, API wiring, bulk load
• GraFlo ontology: a manifest as RDF (graflo manifest-to-rdf, graflo rdf-to-manifest)

Installation

GraFlo needs Python 3.11 or newer. The database clients, RDF and Kafka support are part of the default install.

pip install graflo

Optional extras (see the Installation guide):

• dev: pytest and its plugins, hypothesis, ty, pre-commit
• docs: ProperDocs and its plugins, for building the documentation site
• plot: draws graflo plot-manifest and the --plot figures of graflo merge and graflo merge3 (SVG, PDF, PNG); no system Graphviz or fonts needed

pip install "graflo[dev,docs,plot]"

Development

To install from a clone:

git clone git@github.com:growgraph/graflo.git && cd graflo
uv sync --extra dev --extra plot

See the Contributing Guide for the full workflow.

Tests

The database tests need the database containers. Start them from a clone with the scripts under docker/:

cd docker
./start-all.sh    # Start all services
./stop-all.sh     # Stop all services
./cleanup-all.sh  # Remove containers and volumes

Per-engine compose files and ports are documented in the docker README.

To run the tests:

uv run pytest test

TigerGraph, NebulaGraph and Kafka tests are skipped unless you pass --run-tigergraph, --run-nebula or --run-kafka.

Everything outside test/db runs without the containers, and CI runs it on every pull request:

uv run pytest test --ignore=test/db -n auto --dist loadfile

Property tests run a reduced example budget by default; HYPOTHESIS_PROFILE=full runs the full one.

License

Open source under the Apache License 2.0. Copyright and trademark notices are in NOTICE: the license grants no rights in the GraFlo and GrowGraph marks. Releases before the relicensing shipped under the Business Source License 1.1 and keep those terms; see the changelog.

Contributing

Contributions are welcome. See the Contributing Guide. Contributors accept the Contributor License Agreement once, by commenting on their first pull request.