I suggest two ways to get started to develop Spark in Scala, both with Eclipse: one is to download (from the site scala-ide.org) the full pre-configured Eclipse which already includes the Scala IDE; another one consists in updating your existing Eclipse adding the Scala plugin (detailed instructions below). This basically will allow you to start Scala projects and run them locally. In each case, at the end of the procedure, in order to start developing in Spark, you have to import inside Eclipse as “existing maven project” a project template (that you can find linked at the bottom of this article).
Now I’ll illustrate how to integrate the Scala plugin in you existing maven installation. In this example I used an Eclipse Kepler EE. From the site http://scala-ide.org/download/current.html copy the latest link version for Kepler, or if not present, follow the link “Older versions” in the page, and choose the right Scala version for you. I copied the link from an older stable version for Scala 2.10.4 (which is the version available in the cluster I’m using at the moment), precisely this: http://download.scala-ide.org/sdk/lithium/e38/scala211/stable/site.
Make sure you have Java JDK 1.7 installed and that Eclipse is pointing at it. Click on [Windows] -> [Preferences] -> (in the left menu) [Java] -> (click on) [Installed JRE] and check if a JDK 1.7 installation is selected. In case, use the search button to automatically find your JDK installed in the file system, and then flag it.
Then with the link you copied before, go to [Help] -> [Install new software]. A new windows will open, click on the [Add] button in the top right side and copy in “Location” the link, and in the “Name” write something, like “Scala plugin”.
Click on [OK] and wait for the windows to load all items. Once done, select all items and press [Next] until you have to confirm for the installation “accepting the agreement”, and the click [Finish].
Wait for the installation to finish and reboot Eclipse when done.
The procedure is exactly the same for Eclipse in Windows, Linux and MacOS. If you are having some problem during the download phase, you can decide either to download a zip with the plugin (to be installed manually) or directly download the full Scala integrated Eclipse solution.
Now you are ready to create any Scala project. To start with Spark projects to create fat jars you can download and import “as existing maven project” this boilerplate maven template from github: https://github.com/H4ml3t/spark-scala-maven-boilerplate-project.
After you’ve finished to modify the code, to produce a jar: [Right-click] on the project -> [Run As..] -> [Maven build…] -> add as “goal”: “clean package” -> click [Run]. The result “jar-with-dependencies” will be found in the “target” folder. Read this other article to learn how to submit a jar within the cluster.
I hope it helps, write a comment for everything you may need.