Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-45631

Broken backward compatibility in PySpark: StreamingQueryListener due to the addition of onQueryIdle

    XMLWordPrintableJSON

Details

    Description

      We observed the issue that existing StreamingQueryListener implementations in PySpark start to fail after upgrading to 3.5.0. We identified that the root cause is https://issues.apache.org/jira/browse/SPARK-43183 since onQueryIdle was defined as `@abstractmethod`, requiring all exiting implementations to implement the method.

      It seems like abstractmethod annotation should be removed to retain backward compatibility.

      Attachments

        Issue Links

          Activity

            People

              kabhwan Jungtaek Lim
              kabhwan Jungtaek Lim
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: