👋 Welcome -- what roar is and why we built it

[christophergeyer]

Welcome!

🦖Thanks for being here! 🦖

Do you know what data when into your ML model? Could you recreate the model if you had do? Weeks later? Do you have a custom data pipeline that changes? And does your boss what's to know what changed? Do you to guarantee that there's no PII in your data pipeline?

We built roar because we wanted a provenance and lineage tool like git, one that didn't require code changes or to use frameworks that locked us in.

Instead of logging provenance, roar captures provenance by tracing all the syscalls that happen when your pipeline runs. It captures every file read, every file written, every package used (deb or pip), the git commit, every environment variable read, everything that went in and everything that went out. And then it hashes what went in and what went out and keeps a record of everything.

But, but...

But you say, "That has got to be slow!" to which I say, read the end-to-end benchmarks: https://glaas.ai/docs/benchmarks-end-to-end. Overhead is dependent on your pipeline but is typically 1-4%.

But you say, "Hashing everything is crazy!" to which I say, read the micro benchmarks: https://glaas.ai/docs/benchmarks. BLAKE3 hashes about 10 GB/s on 8 cores. Furthermore, if you just spent $1,000+ in compute on your job, or if your business relies on your data piepline, spending a tiny bit more to get lineage is usually worth it.

But you say, "My training process is multi-node" to which I say, roar integrates with Ray and automatically propagates roar to workers -- read the Ray integration docs: https://glaas.ai/docs/ray.

But you say, "I have my experiment tracker" to which I say, roar captures the experiment link for you, but your data production pipeline is not an experiment, it's something that we think everyone benefits from visibility into.

GLaaS

roar captures your lineage locally. We've built GLaaS (https://glaas.ai) to help you store your lineage persistently. Roughly: roar : GLaaS ≈ git : GitHub. When you want to share or reproduce across machines you can publish the lineage to GLaaS. And you can use GLaaS to lookup any artifact's lineage by its hash (BLAKE3 hash -> lineage).

Quickstart:

roar run <command>      # e.g., roar run python train.py
roar login              # create account on glaas.ai to save your lineage
roar register @1        # publish your lineage to glaas.ai

roar observes it -- capturing inputs, outputs, code, and environment, and identifying files by content hash so a result always traces back to the exact things that made it. It's local-first (everything lands in .roar/), and

Docs here: https://glaas.ai/docs

No code changes. No frameworks.

While most tools ask you to first restructure your work into a pipeline DSL or a particular framework. We wanted reproducibility to be the default — something you get just by running your normal commands, without changing how you work.

We're listening

We've got our ears to the ground if you've got:

• 💡 Ideas & feature requests — what would make roar a no-brainer in your workflow?
• 🐞 Rough edges — it's early; tell us what's confusing, what breaks, what surprised you.
• 🧪 Your use case — training runs, data pipelines, Ray jobs, something we haven't imagined?
• ❓ Questions — anything goes.

Feel free to drop a comment, start a thread, or just say hi.

Best,
Chris 🦖
https://treqs.ai