Skip to content

Version 6.4.0 failing for Spark 3.5.0 #357

@ravillahari

Description

@ravillahari

Abris Version: 6.4.0
Spark Version: 3.5.0
Databricks Runtime: 14.3

Failing with error:
Cause: java.lang.NoClassDefFoundError: org/apache/spark/sql/internal/SQLConf$LegacyBehaviorPolicy$
at org.apache.spark.sql.avro.AvroDeserializer.(AvroDeserializer.scala:62)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.sql.avro.AbrisAvroDeserializer$$anonfun$1.applyOrElse(AbrisAvroDeserializer.scala:39)
at org.apache.spark.sql.avro.AbrisAvroDeserializer$$anonfun$1.applyOrElse(AbrisAvroDeserializer.scala:37)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
at scala.util.Failure.recover(Try.scala:234)
at org.apache.spark.sql.avro.AbrisAvroDeserializer.(AbrisAvroDeserializer.scala:37)

In Spark 3.4.0 "LegacyBehaviorPolicy" is available in same file as here:
https://github.com/apache/spark/blob/branch-3.4/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

However in Spark 3.5.0 "LegacyBehaviorPolicy" is available and used in package:
https://github.com/apache/spark/blob/branch-3.5/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
But is available new package:
https://github.com/apache/spark/blob/branch-3.5/sql/api/src/main/scala/org/apache/spark/sql/internal/LegacyBehaviorPolicy.scala

Could this be issue for the above error?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions