Fold() and Reduce() are not found at runtime

[Kognefaure]

Hello team!

I just pushed a class ArticleProcessorTest that contains a main() function where I test the ArticleProcessor class using dummy articles.
When running this test, the program crashes when it reaches this line (ArticleProcessor.java:44):

final Tuple2<String, Integer> totalAmount = wordCountRddReduced.fold(...)

The exception thrown is NoSuchMethod, pointing to the function fold.

So the spark running environment cannot recognize the signature of the fold() function I'm using, even though it builds properly inside Eclipse. This could be due to mismatching library versions. However I tried these different ways:

• compiling with Maven, and running on my local spark installation (spark 1.2.1, scala 2.10.4)
• compiling with Maven, and running on HDFS (don't know the config)
• compiling and running with my local spark installation.

Because of this last configuration, I really can't understand why this happens, since it should be the exact same setup both for building and executing.

I tried to replace the fold() with a reduce(), but the same thing happens. The other RDD functions I'm using work fine (even reduceByKey()...)

Has anyone an idea for a fix? I would be grateful if someone could try and run this test as well. The command is (use "local[2]" instead of "yarn-client" if you use your local spark installation):

spark-submit --master yarn-client --class exec.ArticleProcessorTest [Path/to/Your/Jar].jar

[Kognefaure]

Just pushed an ugly fix. I replaced the fold() call by a collect(); this effectively de-parallelizes the RDD, turning it into a normal java list, so it will be inefficient. But it works. So let's keep this version for now.