Synopsis

Tests using Scala and Spark for Big Data projects.

An article was published explaining:

• https://edersoncorbari.github.io/tutorials/scala-spark-graph/
• https://dzone.com/articles/bigdata-developing-a-graph-in-spark-and-scala
• https://dzone.com/articles/hdfs-offline-analysis-of-fsimage-metadata

Provided Mechanisms

• The template of a project using SBT;
• A top-down graph to find a company's employee hierarchy;
• Parsing XML file: FsImage (HDFS).

Emacs

Build and Running

To compile the project, you must have the following tools installed:

• SBT-1.2.0 or greater.

Now run the commands below to compile the project:

$ git clone https://github.com/edersoncorbari/scala-lab.git
$ cd scala-lab
$ sbt update compile run

If you receive an error of type: "Caused by: java.net.UnknownHostException: your-host: localhost: Name or service not known", your hostname is not configured in the /etc/hosts please add and run again.

Testing

To perform all the tests please run the command:

$ sbt test

To run a specific test run the command:

$ sbt

Within the SBT console perform a specific test:

> testOnly io.github.edersoncorbari.graph.HierarchyEmployeeTest
> testOnly io.github.edersoncorbari.hdfs.FsImageTest