Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to Spark v3.2.1 #280

Closed
kevinwallimann opened this issue Feb 9, 2022 · 0 comments · Fixed by #281
Closed

Upgrade to Spark v3.2.1 #280

kevinwallimann opened this issue Feb 9, 2022 · 0 comments · Fixed by #281

Comments

@kevinwallimann
Copy link
Collaborator

In Spark 3.2.1, a typo in the method signature for DataSourceUtils.creteDateRebaseFuncInWrite was fixed. apache/spark#34978 Of course, this change is neither backward nor forward compatible. Since the spark-avro jar is a compile-time dependency for ABRiS, other applications that create a fat-jar with the spark-avro v3.2.0 will experience a NoSuchMethodError when running this against Spark 3.2.1 (see below), because the spark-avro jar is usually not part of the Spark distribution.

  Cause: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.DataSourceUtils$.creteDateRebaseFuncInRead(Lscala/Enumeration$Value;Ljava/lang/String;)Lscala/Function1;
  at org.apache.spark.sql.avro.AvroDeserializer.<init>(AvroDeserializer.scala:67)
  at org.apache.spark.sql.avro.AvroDeserializer.<init>(AvroDeserializer.scala:61)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.spark.sql.avro.AbrisAvroDeserializer$$anonfun$1.applyOrElse(AbrisAvroDeserializer.scala:40)
  at org.apache.spark.sql.avro.AbrisAvroDeserializer$$anonfun$1.applyOrElse(AbrisAvroDeserializer.scala:38)
  at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
  at scala.util.Failure.recover(Try.scala:234)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant