Spark by Backwards

Scala module with reusable Spark functionality.

Example usage included while covering various courses.

Table of Contents

• Setup

• SBT

• Release

• Introduction

• Hadoop

• Spark Shell

• SparkConf

• Resilient Distributed Datasets (RDD)

• Architecture

• Pair RDD

• Application JAR

• Courses (module per course/book)

  • Apache Spark Quick Start Guide

  • Learning Spark

  • Spark and Hadoop

NOTE on Intellij - The build has Spark dependencies set to provided. This is fine when running from the sbt console as the build is configured to handle this. However, for every main application run in Intellij, we have to add the following configuration to include any provided dependencies: