おいしいブログ

sbt runの時にprovidedのライブラリをclasspathに追加する

2018-09-02

build.sbtを以下のように定義する。

# build.sbt

libraryDependencies ++= Seq(
  "org.scala-lang" % "scala-library" % "2.11.12",
  "org.scala-lang" % "scala-reflect" % "2.11.12",
  "org.apache.spark" %% "spark-core" % "2.3.1" % Provided,
  "org.apache.spark" %% "spark-sql" % "2.3.1" % Provided
)

それでrunを実行すると、以下のようなエラーが出てぶりぶり怒られてしまう。

[info] Running (fork) com.mylde.pg.spark.WordCountRdd 
[error] Error: A JNI error has occurred, please check your installation and try again
[error] Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
[error]     at java.lang.Class.getDeclaredMethods0(Native Method)
[error]     at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
[error]     at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
[error]     at java.lang.Class.getMethod0(Class.java:3018)
[error]     at java.lang.Class.getMethod(Class.java:1784)
[error]     at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
[error]     at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
[error] Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
[error]     at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
[error]     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
[error]     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
[error]     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
[error]     ... 7 more
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 3 s, completed Sep 1, 2018 6:07:00 PM
sbt:spark-play-ground> 

使用しているsbtは以下。sbt 1.2.1。

sbt:spark-play-ground> about
[info] This is sbt 1.2.1
[info] The current project is ProjectRef(uri("file:/***/spark-play-ground-scala/"), "spark-play-ground-scala") 0.1
[info] The current project is built against Scala 2.11.12
[info] Available Plugins
[info]  - org.scalafmt.sbt.ScalafmtPlugin
[info]  - sbt.ScriptedPlugin
[info]  - sbt.plugins.CorePlugin
[info]  - sbt.plugins.Giter8TemplatePlugin
[info]  - sbt.plugins.IvyPlugin
[info]  - sbt.plugins.JUnitXmlReportPlugin
[info]  - sbt.plugins.JvmPlugin
[info]  - sbt.plugins.SbtPlugin
[info]  - scoverage.ScoverageSbtPlugin
[info]  - wartremover.WartRemover
[info] sbt, sbt plugins, and build definitions are using Scala 2.12.6
sbt:spark-play-ground> 

build.sbtに以下を追加する。

# build.sbt

run in Compile := Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run)).evaluated
runMain in Compile := Defaults.runMainTask(fullClasspath in Compile, runner in(Compile, run)).evaluated

以下を参考。
scala - How to add "provided" dependencies back to run/test tasks' classpath? - Stack Overflow