Example of pyspark spline atlas

[cometta]

Any example of running pyspark job with spline jars files and push integrate with apache atlas? I can only find example in scala

[wajda]

Atlas integration is out of direct scope for the Spline team. There are a number of examples on the internet, some of them are listed here - https://absaoss.github.io/spline/articles.html
Make sure you are referring to the more recent examples though, the ones that involve Spline version 0.5+
Regarding Scala vs Python, there is no real difference with respect to integration with Atlas. If you can make Spline working for your Python job, the rest steps are pretty much the same. The principle is simple - the Spline agent collects the meta data from your job and sends it over to the destination you like. The destination is specified by the "LineageDispatcher" in use. See https://github.com/AbsaOSS/spline-spark-agent#dispatchers

[cometta]

@wajda May I know is the long term plan is not having Spline integration with Atlas? I read comment on https://community.cloudera.com/t5/Support-Questions/Capture-Spark-Job-Lineage-in-Apache-Atlas/td-p/288337 , starting Spline 0.4, there is no support for Atlas

[wajda]

No, there is no plans for Spline to directly support Atlas. It doesn't mean it cannot be done, it just won't be done by our team. Spline is meant for extensions and integration with all sorts of 3rd-party software, and Atlas is just one of those. If you are interested in Spark agent integration with Atlas, the easiest way is to write a custom lineage dispatcher, e.g. AtlasLineageDispatcher that would be responsible for necessary conversion between Spline and Atlas data model, and sending the data over to Atlas. Make it you own project on GitHub. The example of such extension project can be found here - https://github.com/AbsaOSS/spline-getting-started/tree/main/spark-agent-extension-example

[cometta]

In this article https://medium.com/@reenugrewal/data-lineage-tracking-using-spline-on-atlas-via-event-hub-6816be0fd5c7, it was using Spine 0.3.6 to integrate with Atlas

import za.co.absa.spline.core.SparkLineageInitializer._
spark.enableLineageTracking()

i can't run above 2 lines in pyspark as above code is in scala

[wajda]

You can

Also, you can use a codeless initialization, but I don't remember if it worked in Spline 0.3

[cometta]

I tried codeleess init for 0.3.9 but not working, thus would like to try with python code directly

spark = configure_spark_with_delta_pip(builder).getOrCreate()
spark.sparkContext._jvm.za.co.absa.spline.harvester \
    .SparkLineageInitializer.enableLineageTracking(spark._jsparkSession)

i get spark.sparkContext._jvm.za.co.absa.spline.harvester \ TypeError: 'JavaPackage' object is not callable

[wajda]

In Spline 0.3 that class was located in another package za.co.absa.spline.core.SparkLineageInitializer

[cometta]

nevermind, i can't get this to work.