-
Notifications
You must be signed in to change notification settings - Fork 4
ci(shacl): add shape validation for sample data #59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| import json | ||
| import sys | ||
| from pathlib import Path | ||
|
|
||
| from pyshacl import validate | ||
| from rdflib import Graph | ||
|
|
||
| OPENAPI_VER_DIR = Path("openapi/ver") | ||
| SHACL_DIR = Path("data-model/shacl") | ||
|
|
||
|
|
||
| def load_shapes(version): | ||
| shapes_path = SHACL_DIR / version / "shacl.ttl" | ||
| if not shapes_path.exists(): | ||
| return None | ||
| g = Graph() | ||
| g.parse(str(shapes_path), format="turtle") | ||
| return g | ||
|
|
||
|
|
||
| def validate_file(json_path, shapes_graph): | ||
| with open(json_path, encoding="utf-8") as f: | ||
| data = json.load(f) | ||
| data_graph = Graph() | ||
| data_graph.parse(data=json.dumps(data), format="json-ld") | ||
| if len(data_graph) == 0: | ||
| return False, "Parsed RDF graph is empty (context resolution may have failed)" | ||
| conforms, _, results_text = validate( | ||
| data_graph=data_graph, shacl_graph=shapes_graph, debug=False | ||
| ) | ||
| return conforms, results_text | ||
|
|
||
|
|
||
| def main(): | ||
| failures = [] | ||
| versions_found = 0 | ||
|
|
||
| for version_dir in sorted(OPENAPI_VER_DIR.iterdir()): | ||
| sample_data_dir = version_dir / "sample_data" | ||
| json_files = sorted(sample_data_dir.glob("**/*.json")) if sample_data_dir.is_dir() else [] | ||
| if not json_files: | ||
| continue | ||
|
|
||
| shapes_graph = load_shapes(version_dir.name) | ||
|
|
||
| versions_found += 1 | ||
| for json_path in json_files: | ||
| conforms, results_text = validate_file(json_path, shapes_graph) | ||
| if conforms: | ||
| print(f"PASS {json_path}") | ||
| else: | ||
| print(f"FAIL {json_path}\n{results_text}") | ||
| failures.append(json_path) | ||
|
|
||
| print(f"\n{versions_found} versions validated, {len(failures)} failures") | ||
| if failures: | ||
| sys.exit(1) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() |
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. not sure whether the submodule can trigger the action pipeline. Please kindly elaborate. The
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Makes sense. What about filtering on the gitlink path itself, That means validation runs whenever we bump the submodule here, which is the moment we actually want to re-check the sample data against the new shapes. It won't run if the shapes change upstream without a pointer bump. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| [submodule "data-model"] | ||
| path = data-model | ||
| url = https://github.com/skg-if/data-model |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file does not need the api spec (yaml) to run. It checks sample data against
shacl.ttl(which is included as git submodule). Is shacl.ttl derivation of the openapi specs? Am I missing some context?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OpenAPI check and SHACL check are two separate jobs in this workflow. I added the SHACL step leaving the preexisting OpenAPI job untouched