Setup Eclipse to start developing in Spark Scala and build a fat jar

I suggest two ways to get started to develop Spark in Scala, both with Eclipse: one is to download (from the site scala-ide.org) the full pre-configured Eclipse which already includes the Scala IDE; another one consists in updating your existing Eclipse adding the Scala plugin (detailed instructions below). This basically will allow you to start Scala projects and run them locally. In each case, at the end of the procedure, in order to start developing in Spark, you have to import inside Eclipse as “existing maven project” a project template (that you can find linked at the bottom of this article).

Now I’ll illustrate how to integrate the Scala plugin in you existing maven installation. In this example I used an Eclipse Kepler EE. From the site http://scala-ide.org/download/current.html copy the latest link version for Kepler, or if not present, follow the link “Older versions” in the page, and choose the right Scala version for you. I copied the link from an older stable version for Scala 2.10.4 (which is the version available in the cluster I’m using at the moment), precisely this: http://download.scala-ide.org/sdk/lithium/e38/scala211/stable/site.

scala-ide-site-older-versions

Make sure you have Java JDK 1.7 installed and that Eclipse is pointing at it. Click on [Windows] -> [Preferences] -> (in the left menu) [Java] -> (click on) [Installed JRE] and check if a JDK 1.7 installation is selected. In case, use the search button to automatically find your JDK installed in the file system, and then flag it.

installed jres eclipse window

Then with the link you copied before, go to [Help] -> [Install new software]. A new windows will open, click on the [Add] button in the top right side and copy in “Location” the link, and in the “Name” write something, like “Scala plugin”.

add repository eclipse scala plugin

Click on [OK] and wait for the windows to load all items. Once done, select all items and press [Next] until you have to confirm for the installation “accepting the agreement”, and the click [Finish].

install new software eclipse scala window

Wait for the installation to finish and reboot Eclipse when done.

The procedure is exactly the same for Eclipse in Windows, Linux and MacOS. If you are having some problem during the download phase, you can decide either to download a zip with the plugin (to be installed manually) or directly download the full Scala integrated Eclipse solution.

Now you are ready to create any Scala project. To start with Spark projects to create fat jars you can download and import “as existing maven project” this boilerplate maven template from github: https://github.com/H4ml3t/spark-scala-maven-boilerplate-project.

After you’ve finished to modify the code, to produce a jar: [Right-click] on the project -> [Run As..] -> [Maven build…] -> add as “goal”: “clean package” -> click [Run]. The result “jar-with-dependencies” will be found in the “target” folder. Read this other article to learn how to submit a jar within the cluster.

I hope it helps, write a comment for everything you may need.

meniluca πŸ˜€

Advertisements

4 comments

  1. After getting through all steps, downloading and importing the maven project is showing errors in pom.xml for “scala-maven-plugin:3.1.3:compile” and “scala-maven-plugin:3.1.3:testCompile”. I don’t think I have this “scala maven plugin”, do I need to install this as well?

    Like

    1. Hi Ian,
      what error do you have exactly? Have you tried to take it out? Which maven version are you using (3.0+ is required for that plugin)? Which step did you do? Installing a new Eclipse or installing the Scala plugin? I’m using the maven included in Eclipse Kepler (3.0.4), together with Scala 2.10.4. What about you?
      Let me know πŸ˜€
      Luca

      Like

  2. This may be a suggestion that’s outside of scope, however, could you describe how to setup the project within eclipse? I’m trying to replicate what you did (and KUDOS, this did help me immensely) with a brand new empty project. I’m not sure if the project should be a java or scala project, or what should be in referenced libraries vs in the plugins.
    Again, thank you for these blog posts they are extremely helpful
    Sean

    Like

    1. Thank you very much for your kind feedback, Sean.
      Alas! I’m full of work, I would like so much to find time to improve this blog.
      Anyway once you’ve installed Scala within Eclipse (like described in this article), switch to the Scala prospective (button at the top right corner), right click inside “Package Explorer”, then “import”, “existing maven project” and select the folder of the project linked above in the article.
      It will be treated as a Scala project. However remember that all projects in Scala are Java project and you can mix Scala and Java classes together in the same project.
      Have a good day!
      Luca

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s