Spark documentation missing

[goonmeet]

Hi!

I am trying to run the code in step 3. However, I am not sure how the spark service should be set up with respect to your code. Do you have a script to reproduce the setup?

I am also attempting to convert the code to not use spark but I am not sure if that is the right move.

Thanks,
Goonmeet

[marawanokasha]

Hey @goonmeet sorry for the late reply. So for the spark cluster, I used a standalone cluster on 3 nodes as far as I remember. Please follow the readme here https://github.com/marawanokasha/thesis-benchmark for setting up the spark cluster.

Regarding your second question, there is nothing special about my use of Spark, I needed it for easy parallelization of the computation of the different preprocessed formats (binary, TF-IDF, BM25, ...) before doing the classification on a single GPU-powered machine. You can do the preprocessing all on a single machine with a lot of cores, but you'll have to take care of the parallelization with custom code because doing it on a single core would be incredibly slow.

Hope that helped.

Best
Marawan