text:
[SPARK-32707] Support File Based spark.authenticate secret in Mesos Backend

What changes were proposed in this pull request?
This PR introduces the functionality of supporting file-based authenticate secret for the Mesos backend by using the available configs spark.authenticate.secret.file, spark.authenticate.secret.driver.file and spark.authenticate.secret.executor.file.
Why are the changes needed?
Passing secrets in a file is more secure than passing secret as plain text.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added a case in unittest.

Can one of the admins verify this patch?

Can one of the admins verify this patch?


text:
[SPARK-32577][TEST] Fix wrong config for shuffled hash join in test in-joins.sql

What changes were proposed in this pull request?
Fix Fix wrong config in test in-joins.sql
Why are the changes needed?
Fix wrong config in test in-joins.sql
Does this PR introduce any user-facing change?
No
How was this patch tested?
UT

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

Test build #130735 has started for PR 30280 at commit a54a3c6.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35344/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35344/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35344/
Test PASSed.

Test build #130735 has finished for PR 30280 at commit a54a3c6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130735/
Test PASSed.

source: gharchive/pull-request

authors: AmplabJenkins SparkQA dongjoon-hyun warrenzhu25

repo: apache/spark

license_type: permissive

license_source: github-api


text:
[SPARK-35769][SQL] Truncate java.time.Period by fields of year-month interval type

What changes were proposed in this pull request?
Support truncate java.time.Period by fields of year-month interval type
Why are the changes needed?
To follow the SQL standard and respect the field restriction of the target year-month type.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added UT

Test build #139910 has started for PR 32945 at commit 22f97c4.

ping @MaxGekk

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44439/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44439/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44439/

Test build #139910 has finished for PR 32945 at commit 22f97c4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139910/

Test build #139922 has started for PR 32945 at commit 8b22f22.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44450/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44450/

Test build #139922 has finished for PR 32945 at commit 8b22f22.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139922/

Test build #139928 has started for PR 32945 at commit 4245764.

Test build #139930 has started for PR 32945 at commit 28cebb2.

Test build #139932 has started for PR 32945 at commit 8408f3e.

Test build #139934 has started for PR 32945 at commit a3a47f4.

Test build #139928 has finished for PR 32945 at commit 4245764.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139928/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44456/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44459/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44461/

Test build #139932 has finished for PR 32945 at commit 8408f3e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44456/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44459/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44461/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139932/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44456/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44459/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44461/

Test build #139930 has finished for PR 32945 at commit 28cebb2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139930/

Test build #139934 has finished for PR 32945 at commit a3a47f4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139934/

@AngersZhuuuu The failed test is related to the changes:
org.scalatest.exceptions.TestFailedException: P-96322443Y-9M did not equal P-96322443Y

Could you fix it, please.

@AngersZhuuuu The failed test is related to the changes:
org.scalatest.exceptions.TestFailedException: P-96322443Y-9M did not equal P-96322443Y

Could you fix it, please.

Done

Test build #139957 has started for PR 32945 at commit e58b0e4.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44484/

Test build #139962 has started for PR 32945 at commit b990432.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44484/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44489/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44484/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44489/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44489/

Test build #139971 has started for PR 32945 at commit a039b61.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44498/

Test build #139973 has started for PR 32945 at commit d7a1d33.

Test build #139957 has finished for PR 32945 at commit e58b0e4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139957/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44498/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44500/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44498/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44500/

Test build #139962 has finished for PR 32945 at commit b990432.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

+1, LGTM. Merging to master.
Thank you, @AngersZhuuuu and @cloud-fan for your review.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44500/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139962/

Test build #139971 has finished for PR 32945 at commit a039b61.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139971/

Test build #139973 has finished for PR 32945 at commit d7a1d33.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139973/

authors: AmplabJenkins AngersZhuuuu MaxGekk SparkQA


text:
Spark 36292

What changes were proposed in this pull request?
Refactor some exceptions in QueryExecutionErrors to use error classes.
cannotGenerateCodeForUnsupportedTypeError
cannotInterpolateClassIntoCodeBlockError
customCollectionClsNotResolvedError
classUnsupportedByMapObjectsError
nullAsMapKeyNotAllowedError
methodNotDeclaredError
constructorNotFoundError
primaryConstructorNotFoundError
unsupportedNaturalJoinTypeError
notExpectedUnresolvedEncoderError
unsupportedEncoderError
notOverrideExpectedMethodsError
failToConvertValueToJsonError
unexpectedOperatorInCorrelatedSubquery
unreachableError
unsupportedRoundingMode
resolveCannotHandleNestedSchema
inputExternalRowCannotBeNullError
fieldCannotBeNullMsg
fieldCannotBeNullError

Why are the changes needed?
There are currently ~350 exceptions in this file; so this PR only focuses on the third set of 20.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Existed UT Testcase

Can one of the admins verify this patch?

Test build #143850 has started for PR 34177 at commit 0870b3a.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48363/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48363/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48363/

Test build #143850 has finished for PR 34177 at commit 0870b3a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143850/

authors: AmplabJenkins SparkQA dchvn


text:
[SPARK-38339][BUILD] Upgrade RoaringBitmap to 0.9.25

What changes were proposed in this pull request?
This pr aims upgrade RoaringBitmap from 0.9.23 to 0.9.25.
Why are the changes needed?
The changes from 0.9.23 to version 0.9.25. are as follows：

change list

ior fission + benchmark to show iand fission is unprofitable  optimizes the performance of RoaringBitmap.or api, and this api is used by Spark code , for example PushBasedFetchHelper.initiateFallbackFetchForPushMergedBlock.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA

Merged to master.

thanks @HyukjinKwon

authors: HyukjinKwon LuciferYang


text:
[SPARK-39553][CORE] Multi-thread unregister shuffle shouldn't throw NPE when using Scala 2.13

What changes were proposed in this pull request?
This pr add a shuffleStatus != null condition to o.a.s.MapOutputTrackerMaster#unregisterShuffle method to avoid throwing NPE when using Scala 2.13.
Why are the changes needed?
Ensure that no NPE is thrown when o.a.s.MapOutputTrackerMaster#unregisterShuffle is called by multiple threads, this pr is only for Scala 2.13.
o.a.s.MapOutputTrackerMaster#unregisterShuffle method will be called concurrently by the following two paths:

BlockManagerStorageEndpoint:

https://github.com/apache/spark/blob/6f1046afa40096f477b29beecca5ca6286dfa7f3/core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala#L56-L62

ContextCleaner:

https://github.com/apache/spark/blob/6f1046afa40096f477b29beecca5ca6286dfa7f3/core/src/main/scala/org/apache/spark/ContextCleaner.scala#L234-L241
When test with Scala 2.13, for example sql/core module,  there are many log as follows，although these did not cause UTs failure:
17:44:09.957 WARN org.apache.spark.storage.BlockManagerMaster: Failed to remove shuffle 87 - null
java.lang.NullPointerException
	at org.apache.spark.MapOutputTrackerMaster.$anonfun$unregisterShuffle$1(MapOutputTracker.scala:882)
	at org.apache.spark.MapOutputTrackerMaster.$anonfun$unregisterShuffle$1$adapted(MapOutputTracker.scala:881)
	at scala.Option.foreach(Option.scala:437)
	at org.apache.spark.MapOutputTrackerMaster.unregisterShuffle(MapOutputTracker.scala:881)
	at org.apache.spark.storage.BlockManagerStorageEndpoint$$anonfun$receiveAndReply$1.$anonfun$applyOrElse$3(BlockManagerStorageEndpoint.scala:59)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.scala:17)
	at org.apache.spark.storage.BlockManagerStorageEndpoint.$anonfun$doAsync$1(BlockManagerStorageEndpoint.scala:89)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:678)
	at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:467)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
17:44:09.958 ERROR org.apache.spark.ContextCleaner: Error cleaning shuffle 94
java.lang.NullPointerException
	at org.apache.spark.MapOutputTrackerMaster.$anonfun$unregisterShuffle$1(MapOutputTracker.scala:882)
	at org.apache.spark.MapOutputTrackerMaster.$anonfun$unregisterShuffle$1$adapted(MapOutputTracker.scala:881)
	at scala.Option.foreach(Option.scala:437)
	at org.apache.spark.MapOutputTrackerMaster.unregisterShuffle(MapOutputTracker.scala:881)
	at org.apache.spark.ContextCleaner.doCleanupShuffle(ContextCleaner.scala:241)
	at org.apache.spark.ContextCleaner.$anonfun$keepCleaning$3(ContextCleaner.scala:202)
	at org.apache.spark.ContextCleaner.$anonfun$keepCleaning$3$adapted(ContextCleaner.scala:195)
	at scala.Option.foreach(Option.scala:437)
	at org.apache.spark.ContextCleaner.$anonfun$keepCleaning$1(ContextCleaner.scala:195)
	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1432)
	at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:189)
	at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:79)

I think this is a bug of Scala 2.13.8 and already submit an issue to https://github.com/scala/bug/issues/12613, this PR is only for protection, we should remove this protection after Scala 2.13(maybe https://github.com/scala/scala/pull/9957) fixes this issue.
Does this PR introduce any user-facing change?
No
How was this patch tested?

Pass GA
Add new test SPARK-39553: Multi-thread unregister shuffle shouldn't throw NPE to MapOutputTrackerSuite, we can test manually as follows:

dev/change-scala-version.sh 2.13
mvn clean install -DskipTests -pl core -am -Pscala-2.13
mvn clean test -pl core -Pscala-2.13 -Dtest=none -DwildcardSuites=org.apache.spark.MapOutputTrackerSuite

Before
- SPARK-39553: Multi-thread unregister shuffle shouldn't throw NPE *** FAILED ***
  3 did not equal 0 (MapOutputTrackerSuite.scala:971)
Run completed in 17 seconds, 505 milliseconds.
Total number of tests run: 25
Suites: completed 2, aborted 0
Tests: succeeded 24, failed 1, canceled 0, ignored 1, pending 0
*** 1 TEST FAILED ***

After
- SPARK-39553: Multi-thread unregister shuffle shouldn't throw NPE
Run completed in 17 seconds, 996 milliseconds.
Total number of tests run: 25
Suites: completed 2, aborted 0
Tests: succeeded 25, failed 0, canceled 0, ignored 1, pending 0
All tests passed.

cc @HyukjinKwon @srowen @dongjoon-hyun

Is it necessary to do this protection work.

Scala 2.13.9 should resolved this issue, but the release time of 2.13.9 is uncertain
https://github.com/scala/scala/milestones

We have a similar pattern in SortShuffleManager - there we wrap result in Option before foreach …

Merged to master/3.3/3.2

We have a similar pattern in SortShuffleManager - but there we wrap result in Option before foreach (given no asScala), and so is fine … did not find any other cases where this is an issue thankfully.

Yes, taskIdMapsForShuffle in SortShuffleManager is safe. Thanks ~

Thanks @srowen @mridulm  ~

+1, late LGTM. Thank you all.

authors: LuciferYang dongjoon-hyun mridulm srowen


text:
[SPARK-4985] [SQL] parquet support for date type

This PR might have some issues with #3732 ,
and this would have merge conflicts with #3820 so the review can be delayed till that 2 were merged.

Test build #27084 has started for   PR 3822 at commit b3ec9b0.

This patch merges cleanly.

Test build #27084 has finished for   PR 3822 at commit b3ec9b0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27084/
Test PASSed.

Test build #27937 has started for   PR 3822 at commit 64cc041.

This patch merges cleanly.

Test build #27937 has finished for   PR 3822 at commit 64cc041.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27937/
Test PASSed.

HIVE-8119 have been merged into hive trunk. I'll test my patch based on that.

I have tested my code with hive trunk, it is compatible.
/cc @marmbrus @rxin @liancheng @yhuai

This looks pretty good to me.  Can you add a test that actually exercises the read/write path?

Test build #28777 has started for   PR 3822 at commit d41fb0c.

This patch merges cleanly.

Test build #28777 has finished for   PR 3822 at commit d41fb0c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28777/
Test PASSed.

Sorry, looks like this has conflicts again. :(  I would love to include it in 1.3.1 if you have time to update soon! :)

rebased

Test build #28966 has started for   PR 3822 at commit 2c5d54d.

This patch merges cleanly.

Test build #28966 has finished for   PR 3822 at commit 2c5d54d.

This patch passes all tests.

This patch merges cleanly.

This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28966/
Test PASSed.

@marmbrus

Thanks! Merged into master and branch-1.3.

authors: AmplabJenkins SparkQA adrian-wang liancheng marmbrus


text:
[SPARK-45137][CONNECT] Support map/array parameters in parameterized sql()

What changes were proposed in this pull request?
In the PR, I propose to extend the proto protocol, and add two more fields to SqlCommand: arguments and pos_arguments. This should allow to construct arrays/maps/structs by the map(), array(), struct(), map_from_arrays() and map_from_entries(), and pass them to sql() as arguments.
Why are the changes needed?
To fix the issue demonstrated by the example:
spark.sql("select element_at(?, 1)", Array(array(lit(1)))).collect()
java.lang.UnsupportedOperationException: literal unresolved_function {
[info]   function_name: "array"
[info]   arguments {
[info]     literal {
[info]       integer: 1
[info]     }
[info]   }
[info] }
[info]  not supported (yet).

Note: The PR https://github.com/apache/spark/pull/42752 supported constructors, and this PR extends the feature on Scala connect client.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
By new checks:
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *.ClientE2ETestSuite"

Was this patch authored or co-authored using generative AI tooling?
No.

Merging to master. Thank you, @HyukjinKwon and @cloud-fan for review.


text:
[SPARK-45616][CORE] Avoid ParVector, which does not propagate ThreadLocals or SparkSession

What changes were proposed in this pull request?
CastSuiteBase and ExpressionInfoSuite use ParVector.foreach() to run Spark SQL queries in parallel. They incorrectly assume that each parallel operation will inherit the main thread’s active SparkSession. This is only true when these parallel operations run in freshly-created threads. However, when other code has already run some parallel operations before Spark was started, then there may be existing threads that do not have an active SparkSession. In that case, these tests fail with NullPointerExceptions when creating SparkPlans or running SQL queries.
The fix is to use the existing method ThreadUtils.parmap(). This method creates fresh threads that inherit the current active SparkSession, and it propagates the Spark ThreadLocals.
This PR also adds a scalastyle warning against use of ParVector.
Why are the changes needed?
This change makes CastSuiteBase and ExpressionInfoSuite less brittle to future changes that may run parallel operations during test startup.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Reproduced the test failures by running a ParVector operation before Spark starts. Verified that this PR fixes the test failures in this condition.
  protected override def beforeAll(): Unit = {
    // Run a ParVector operation before initializing the SparkSession. This starts some Scala
    // execution context threads that have no active SparkSession. These threads will be reused for
    // later ParVector operations, reproducing SPARK-45616.
    new ParVector((0 until 100).toVector).foreach { _ => }

    super.beforeAll()
  }

Was this patch authored or co-authored using generative AI tooling?
No.

cc @cloud-fan @gatorsmile

cc @MaxGekk FYI

The streaming test failure is unrelated, merging to master/3.5, thanks!

authors: HyukjinKwon ankurdave cloud-fan


text:
[SPARK-45837][CONNECT] Improve logging information in handling retries

What changes were proposed in this pull request?
Add suppressed exception when handling retries
Why are the changes needed?
Improves user and debugging experience by showing underlying error.
Does this PR introduce any user-facing change?
Better exceptions.
How was this patch tested?
Hand testing
Was this patch authored or co-authored using generative AI tooling?
NA

@juliuszsompolski you asked for moving this here :).

Could you re-trigger CI, @cdkrot ?

Merged to master.

authors: HyukjinKwon cdkrot dongjoon-hyun


text:
[SPARK-47007][SQL] Add the MapSort expression

What changes were proposed in this pull request?
Added the new MapSort expression in CollationOperations alongside new UTs.
Why are the changes needed?
In order to add the ability to do GROUP BY on map types we first have to be able to sort the maps by their key
Does this PR introduce any user-facing change?
No
How was this patch tested?
New tests in CollectionExpressionSuite
Was this patch authored or co-authored using generative AI tooling?
No

+1, LGTM. Merging to master.
Thank you, @stevomitric and @stefankandic @cloud-fan @markj-db for review.

authors: MaxGekk stevomitric


text:
[SPARK-5494][SQL] SparkSqlSerializer Ignores KryoRegistrators

SparkSqlSerializer completely ignores custom KryoRegistrators (unlike the regular Spark serializer). I've applied a change to call super.newKryo before registering their custom classes, and removed duplicate registrations/settings.

Can one of the admins verify this patch?

Jenkins this is ok to test

Test build #27723 has started for   PR 4693 at commit 8e756cc.

This patch merges cleanly.

Test build #27723 has finished for   PR 4693 at commit 8e756cc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27723/
Test FAILed.

There is no reason to allow custom kryo registration.  The SQL serializer is only ever used to serialize SQL types.  Have you seen some bug with us missing a type?

I believe the bug is due to inconsistency between KryoSerializer and SqlSerializer in the order of registration. I don't know if there is a way to manage multiple Serializers, but mixed applications (sql/raw spark) can run into issues.

@pwoody I don't think that is possible.  As long as the two sides are using copies of kryo with the same order or registration then things should work.  As far as I know we never mix them up.  Please let me know if you have a test case that shows otherwise.
If there are no examples of failures, then I suggest we close this issue.

Talked with @pwoody offline and he's working on a test case, but I'll summarize what I think will break. If you use an SQL Context wrapping a Spark Context where you set spark.serializer, then when you do some things in Spark SQL and try to collect the RDD, the operations in computing the SQL will be fine, but when you collect the RDD it uses the SQL serializer to serialize the results but the driver will kryo-deserialize using spark.serializer.
This was the issue that originally prompted this change.

@mccheah we set the serializer in SQL on a per-shuffle basis, so that would surprise me.  However, if you can show it happening we should certainly fix it.

@pwoody @mccheah any update on it? If you saw an error, what was the exception and error message?

authors: AmplabJenkins SparkQA ash211 hkothari marmbrus mccheah pwoody yhuai


text:
[WIP][SPARK-49746][BUILD] Upgrade Scala to 2.13.15

What changes were proposed in this pull request?
The pr aims to upgrade scala from 2.13.14 to 2.13.15.
Why are the changes needed?
https://contributors.scala-lang.org/t/scala-2-13-15-release-planning/6649

Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA.
Was this patch authored or co-authored using generative AI tooling?
No.

For the record, this is blocked by:

lightbend:genjavadoc
com-lihaoyi:Ammonite

Thank you for tracking this, @panbingkun .

compile warning deprecation
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/TestUtils.scala:424:11: method write in class Files is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.TestUtils.createTempScriptWithExpectedOutput, origin=com.google.common.io.Files.write
[error]     Files.write(script, file, StandardCharsets.UTF_8)
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:170:30: method getAllStatistics in class FileSystem is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.SparkHadoopUtil.getFSBytesReadOnThreadCallback.f.$anonfun, origin=org.apache.hadoop.fs.FileSystem.getAllStatistics
[error]     val f = () => FileSystem.getAllStatistics.asScala.map(_.getThreadStatistics.getBytesRead).sum
[error]                              ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:201:34: method getAllStatistics in class FileSystem is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.SparkHadoopUtil.getFSBytesWrittenOnThreadCallback.threadStats, origin=org.apache.hadoop.fs.FileSystem.getAllStatistics
[error]     val threadStats = FileSystem.getAllStatistics.asScala.map(_.getThreadStatistics)
[error]                                  ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/history/BasicEventFilterBuilder.scala:160:13: class SparkListenerExecutorBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.history.BasicEventFilter._acceptFn, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklisted, version=3.1.0
[error]     case e: SparkListenerExecutorBlacklisted => liveExecutors.contains(e.executorId)
[error]             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/history/BasicEventFilterBuilder.scala:161:13: class SparkListenerExecutorUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.history.BasicEventFilter._acceptFn, origin=org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted, version=3.1.0
[error]     case e: SparkListenerExecutorUnblacklisted => liveExecutors.contains(e.executorId)
[error]             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/history/HistoryAppStatusStore.scala:75:40: value isBlacklisted in class ExecutorSummary is deprecated (since 3.1.0): use isExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.history.HistoryAppStatusStore.replaceExecutorLogs, origin=org.apache.spark.status.api.v1.ExecutorSummary.isBlacklisted, version=3.1.0
[error]       source.totalShuffleWrite, source.isBlacklisted, source.maxMemory, source.addTime,
[error]                                        ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/history/HistoryAppStatusStore.scala:77:14: value blacklistedInStages in class ExecutorSummary is deprecated (since 3.1.0): use excludedInStages instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.history.HistoryAppStatusStore.replaceExecutorLogs, origin=org.apache.spark.status.api.v1.ExecutorSummary.blacklistedInStages, version=3.1.0
[error]       source.blacklistedInStages, source.peakMemoryMetrics, source.attributes, source.resources,
[error]              ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala:219:13: method append in class Files is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.worker.DriverRunner.runDriver.initialize, origin=com.google.common.io.Files.append
[error]       Files.append(header, stderr, StandardCharsets.UTF_8)
[error]             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala:194:13: method write in class Files is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.deploy.worker.ExecutorRunner.fetchAndRunExecutor, origin=com.google.common.io.Files.write
[error]       Files.write(header, stderr, StandardCharsets.UTF_8)
[error]             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/executor/ExecutorSource.scala:36:16: method getAllStatistics in class FileSystem is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.executor.ExecutorSource.fileStats, origin=org.apache.hadoop.fs.FileSystem.getAllStatistics
[error]     FileSystem.getAllStatistics.asScala.find(s => s.getScheme.equals(scheme))
[error]                ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala:121:7: method newTaskTempFile in class FileCommitProtocol is deprecated (since 3.3.0): use newTaskTempFile(..., spec: FileNameSpec) instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.internal.io.FileCommitProtocol.newTaskTempFile, origin=org.apache.spark.internal.io.FileCommitProtocol.newTaskTempFile, version=3.3.0
[error]       newTaskTempFile(taskContext, dir, spec.suffix)
[error]       ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/internal/io/FileCommitProtocol.scala:157:7: method newTaskTempFileAbsPath in class FileCommitProtocol is deprecated (since 3.3.0): use newTaskTempFileAbsPath(..., spec: FileNameSpec) instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.internal.io.FileCommitProtocol.newTaskTempFileAbsPath, origin=org.apache.spark.internal.io.FileCommitProtocol.newTaskTempFileAbsPath, version=3.3.0
[error]       newTaskTempFileAbsPath(taskContext, absoluteDir, spec.suffix)
[error]       ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:193:45: class SparkListenerExecutorBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.EventLoggingListener.onExecutorBlacklisted.event, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklisted, version=3.1.0
[error]   override def onExecutorBlacklisted(event: SparkListenerExecutorBlacklisted): Unit = {
[error]                                             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:202:14: class SparkListenerExecutorBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.EventLoggingListener.onExecutorBlacklistedForStage.event, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklistedForStage, version=3.1.0
[error]       event: SparkListenerExecutorBlacklistedForStage): Unit = {
[error]              ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:211:49: class SparkListenerNodeBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.EventLoggingListener.onNodeBlacklistedForStage.event, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklistedForStage, version=3.1.0
[error]   override def onNodeBlacklistedForStage(event: SparkListenerNodeBlacklistedForStage): Unit = {
[error]                                                 ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:219:47: class SparkListenerExecutorUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.EventLoggingListener.onExecutorUnblacklisted.event, origin=org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted, version=3.1.0
[error]   override def onExecutorUnblacklisted(event: SparkListenerExecutorUnblacklisted): Unit = {
[error]                                               ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:228:41: class SparkListenerNodeBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.EventLoggingListener.onNodeBlacklisted.event, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklisted, version=3.1.0
[error]   override def onNodeBlacklisted(event: SparkListenerNodeBlacklisted): Unit = {
[error]                                         ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:236:43: class SparkListenerNodeUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.EventLoggingListener.onNodeUnblacklisted.event, origin=org.apache.spark.scheduler.SparkListenerNodeUnblacklisted, version=3.1.0
[error]   override def onNodeUnblacklisted(event: SparkListenerNodeUnblacklisted): Unit = {
[error]                                           ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala:120:28: class SparkListenerExecutorUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.HealthTracker.applyExcludeOnFailureTimeout.$anonfun, origin=org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted, version=3.1.0
[error]           listenerBus.post(SparkListenerExecutorUnblacklisted(now, exec))
[error]                            ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala:136:28: class SparkListenerNodeUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.HealthTracker.applyExcludeOnFailureTimeout.$anonfun, origin=org.apache.spark.scheduler.SparkListenerNodeUnblacklisted, version=3.1.0
[error]           listenerBus.post(SparkListenerNodeUnblacklisted(now, node))
[error]                            ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala:240:28: class SparkListenerNodeBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.HealthTracker.updateExcludedForFetchFailure, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklisted, version=3.1.0
[error]           listenerBus.post(SparkListenerNodeBlacklisted(now, host, 1))
[error]                            ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala:253:26: class SparkListenerExecutorBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.HealthTracker.updateExcludedForFetchFailure, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklisted, version=3.1.0
[error]         listenerBus.post(SparkListenerExecutorBlacklisted(now, exec, 1))
[error]                          ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala:290:26: class SparkListenerExecutorBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.HealthTracker.updateExcludedForSuccessfulTaskSet.$anonfun, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklisted, version=3.1.0
[error]         listenerBus.post(SparkListenerExecutorBlacklisted(now, exec, newTotal))
[error]                          ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala:309:28: class SparkListenerNodeBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.HealthTracker.updateExcludedForSuccessfulTaskSet.$anonfun, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklisted, version=3.1.0
[error]           listenerBus.post(SparkListenerNodeBlacklisted(now, node, excludedExecsOnNode.size))
[error]                            ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala:545:28: class SparkListenerExecutorBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListener.onExecutorBlacklisted.executorBlacklisted, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklisted, version=3.1.0
[error]       executorBlacklisted: SparkListenerExecutorBlacklisted): Unit = { }
[error]                            ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala:550:36: class SparkListenerExecutorBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListener.onExecutorBlacklistedForStage.executorBlacklistedForStage, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklistedForStage, version=3.1.0
[error]       executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage): Unit = { }
[error]                                    ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala:555:32: class SparkListenerNodeBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListener.onNodeBlacklistedForStage.nodeBlacklistedForStage, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklistedForStage, version=3.1.0
[error]       nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage): Unit = { }
[error]                                ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala:560:30: class SparkListenerExecutorUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListener.onExecutorUnblacklisted.executorUnblacklisted, origin=org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted, version=3.1.0
[error]       executorUnblacklisted: SparkListenerExecutorUnblacklisted): Unit = { }
[error]                              ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala:565:24: class SparkListenerNodeBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListener.onNodeBlacklisted.nodeBlacklisted, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklisted, version=3.1.0
[error]       nodeBlacklisted: SparkListenerNodeBlacklisted): Unit = { }
[error]                        ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala:570:26: class SparkListenerNodeUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListener.onNodeUnblacklisted.nodeUnblacklisted, origin=org.apache.spark.scheduler.SparkListenerNodeUnblacklisted, version=3.1.0
[error]       nodeUnblacklisted: SparkListenerNodeUnblacklisted): Unit = { }
[error]                          ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:66:41: class SparkListenerExecutorBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklistedForStage, version=3.1.0
[error]       case executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage =>
[error]                                         ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:67:18: method onExecutorBlacklistedForStage in trait SparkListenerInterface is deprecated (since 3.1.0): use onExecutorExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerInterface.onExecutorBlacklistedForStage, version=3.1.0
[error]         listener.onExecutorBlacklistedForStage(executorBlacklistedForStage)
[error]                  ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:68:37: class SparkListenerNodeBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklistedForStage, version=3.1.0
[error]       case nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage =>
[error]                                     ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:69:18: method onNodeBlacklistedForStage in trait SparkListenerInterface is deprecated (since 3.1.0): use onNodeExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerInterface.onNodeBlacklistedForStage, version=3.1.0
[error]         listener.onNodeBlacklistedForStage(nodeBlacklistedForStage)
[error]                  ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:70:33: class SparkListenerExecutorBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklisted, version=3.1.0
[error]       case executorBlacklisted: SparkListenerExecutorBlacklisted =>
[error]                                 ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:71:18: method onExecutorBlacklisted in trait SparkListenerInterface is deprecated (since 3.1.0): use onExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerInterface.onExecutorBlacklisted, version=3.1.0
[error]         listener.onExecutorBlacklisted(executorBlacklisted)
[error]                  ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:72:35: class SparkListenerExecutorUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted, version=3.1.0
[error]       case executorUnblacklisted: SparkListenerExecutorUnblacklisted =>
[error]                                   ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:73:18: method onExecutorUnblacklisted in trait SparkListenerInterface is deprecated (since 3.1.0): use onExecutorUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerInterface.onExecutorUnblacklisted, version=3.1.0
[error]         listener.onExecutorUnblacklisted(executorUnblacklisted)
[error]                  ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:74:29: class SparkListenerNodeBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklisted, version=3.1.0
[error]       case nodeBlacklisted: SparkListenerNodeBlacklisted =>
[error]                             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:75:18: method onNodeBlacklisted in trait SparkListenerInterface is deprecated (since 3.1.0): use onNodeExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerInterface.onNodeBlacklisted, version=3.1.0
[error]         listener.onNodeBlacklisted(nodeBlacklisted)
[error]                  ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:76:31: class SparkListenerNodeUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerNodeUnblacklisted, version=3.1.0
[error]       case nodeUnblacklisted: SparkListenerNodeUnblacklisted =>
[error]                               ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala:77:18: method onNodeUnblacklisted in trait SparkListenerInterface is deprecated (since 3.1.0): use onNodeUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.SparkListenerBus.doPostEvent, origin=org.apache.spark.scheduler.SparkListenerInterface.onNodeUnblacklisted, version=3.1.0
[error]         listener.onNodeUnblacklisted(nodeUnblacklisted)
[error]                  ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/TaskSetExcludeList.scala:150:11: class SparkListenerExecutorBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.TaskSetExcludelist.updateExcludedForFailedTask, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklistedForStage, version=3.1.0
[error]           SparkListenerExecutorBlacklistedForStage(now, exec, numFailures, stageId, stageAttemptId))
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/scheduler/TaskSetExcludeList.scala:161:15: class SparkListenerNodeBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.scheduler.TaskSetExcludelist.updateExcludedForFailedTask, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklistedForStage, version=3.1.0
[error]               SparkListenerNodeBlacklistedForStage(now, host, numFailExec, stageId, stageAttemptId))
[error]               ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:79:6: class AccessController in package security is deprecated (since 17)
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.serializer.SerializationDebugger.enableDebugging, origin=java.security.AccessController, version=17
[error]     !AccessController.doPrivileged(action).booleanValue()
[error]      ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:293:45: class SparkListenerExecutorBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.onExecutorBlacklisted.event, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklisted, version=3.1.0
[error]   override def onExecutorBlacklisted(event: SparkListenerExecutorBlacklisted): Unit = {
[error]                                             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:302:14: class SparkListenerExecutorBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.onExecutorBlacklistedForStage.event, origin=org.apache.spark.scheduler.SparkListenerExecutorBlacklistedForStage, version=3.1.0
[error]       event: SparkListenerExecutorBlacklistedForStage): Unit = {
[error]              ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:311:49: class SparkListenerNodeBlacklistedForStage in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.onNodeBlacklistedForStage.event, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklistedForStage, version=3.1.0
[error]   override def onNodeBlacklistedForStage(event: SparkListenerNodeBlacklistedForStage): Unit = {
[error]                                                 ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:334:47: class SparkListenerExecutorUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerExecutorUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.onExecutorUnblacklisted.event, origin=org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted, version=3.1.0
[error]   override def onExecutorUnblacklisted(event: SparkListenerExecutorUnblacklisted): Unit = {
[error]                                               ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:342:41: class SparkListenerNodeBlacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.onNodeBlacklisted.event, origin=org.apache.spark.scheduler.SparkListenerNodeBlacklisted, version=3.1.0
[error]   override def onNodeBlacklisted(event: SparkListenerNodeBlacklisted): Unit = {
[error]                                         ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:350:43: class SparkListenerNodeUnblacklisted in package scheduler is deprecated (since 3.1.0): use SparkListenerNodeUnexcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.onNodeUnblacklisted.event, origin=org.apache.spark.scheduler.SparkListenerNodeUnblacklisted, version=3.1.0
[error]   override def onNodeUnblacklisted(event: SparkListenerNodeUnblacklisted): Unit = {
[error]                                           ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:400:35: value BLACKLISTED_EXECUTORS in class AppStatusSource is deprecated (since 3.1.0): use excludedExecutors instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.updateExecExclusionStatus.$anonfun, origin=org.apache.spark.status.AppStatusSource.BLACKLISTED_EXECUTORS, version=3.1.0
[error]         appStatusSource.foreach(_.BLACKLISTED_EXECUTORS.inc())
[error]                                   ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:403:35: value UNBLACKLISTED_EXECUTORS in class AppStatusSource is deprecated (since 3.1.0): use unexcludedExecutors instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusListener.updateExecExclusionStatus.$anonfun, origin=org.apache.spark.status.AppStatusSource.UNBLACKLISTED_EXECUTORS, version=3.1.0
[error]         appStatusSource.foreach(_.UNBLACKLISTED_EXECUTORS.inc())
[error]                                   ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala:121:40: value isBlacklisted in class ExecutorSummary is deprecated (since 3.1.0): use isExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusStore.replaceDriverGcTime, origin=org.apache.spark.status.api.v1.ExecutorSummary.isBlacklisted, version=3.1.0
[error]       source.totalShuffleWrite, source.isBlacklisted, source.maxMemory, source.addTime,
[error]                                        ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala:123:14: value blacklistedInStages in class ExecutorSummary is deprecated (since 3.1.0): use excludedInStages instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.AppStatusStore.replaceDriverGcTime, origin=org.apache.spark.status.api.v1.ExecutorSummary.blacklistedInStages, version=3.1.0
[error]       source.blacklistedInStages, source.peakMemoryMetrics, source.attributes, source.resources,
[error]              ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/api/v1/api.scala:529:12: value holdingLocks in class ThreadStackTrace is deprecated (since 4.0.0): using synchronizers and monitors instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.api.v1.ThreadStackTrace.productElement, origin=org.apache.spark.status.api.v1.ThreadStackTrace.holdingLocks, version=4.0.0
[error] case class ThreadStackTrace(
[error]            ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/protobuf/ExecutorStageSummarySerializer.scala:41:39: value isBlacklistedForStage in class ExecutorStageSummary is deprecated (since 3.1.0): use isExcludedForStage instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.protobuf.ExecutorStageSummarySerializer.serialize.builder, origin=org.apache.spark.status.api.v1.ExecutorStageSummary.isBlacklistedForStage, version=3.1.0
[error]       .setIsBlacklistedForStage(input.isBlacklistedForStage)
[error]                                       ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/protobuf/ExecutorSummaryWrapperSerializer.scala:64:31: value isBlacklisted in class ExecutorSummary is deprecated (since 3.1.0): use isExcluded instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.protobuf.ExecutorSummaryWrapperSerializer.serializeExecutorSummary.builder, origin=org.apache.spark.status.api.v1.ExecutorSummary.isBlacklisted, version=3.1.0
[error]       .setIsBlacklisted(input.isBlacklisted)
[error]                               ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/status/protobuf/ExecutorSummaryWrapperSerializer.scala:79:11: value blacklistedInStages in class ExecutorSummary is deprecated (since 3.1.0): use excludedInStages instead
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.status.protobuf.ExecutorSummaryWrapperSerializer.serializeExecutorSummary, origin=org.apache.spark.status.api.v1.ExecutorSummary.blacklistedInStages, version=3.1.0
[error]     input.blacklistedInStages.foreach { stage =>
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/util/collection/AppendOnlyMap.scala:211:45: method murmur3_32 in class Hashing is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.util.collection.AppendOnlyMap.rehash, origin=com.google.common.hash.Hashing.murmur3_32
[error]   private def rehash(h: Int): Int = Hashing.murmur3_32().hashInt(h).asInt()
[error]                                             ^
[error] /Users/panbingkun/Developer/spark/spark-community/core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala:269:47: method murmur3_32 in class Hashing is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.util.collection.OpenHashSet.hashcode, origin=com.google.common.hash.Hashing.murmur3_32
[error]   private def hashcode(h: Int): Int = Hashing.murmur3_32().hashInt(h).asInt()
[error]                                               ^
[error] 61 errors found
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:156:12: method addAllPosArguments in class Builder is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.SparkSession.sql.$anonfun.cmd.$anonfun, origin=org.apache.spark.connect.proto.SqlCommand.Builder.addAllPosArguments
[error]           .addAllPosArguments(args.map(lit(_).expr).toImmutableArraySeq.asJava)))
[error]            ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:155:12: method setSql in class Builder is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.SparkSession.sql.$anonfun.cmd.$anonfun, origin=org.apache.spark.connect.proto.SqlCommand.Builder.setSql
[error]           .setSql(sqlText)
[error]            ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:188:14: method putAllNamedArguments in class Builder is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.SparkSession.sql.$anonfun.cmd.$anonfun, origin=org.apache.spark.connect.proto.SqlCommand.Builder.putAllNamedArguments
[error]             .putAllNamedArguments(args.asScala.map { case (k, v) => (k, lit(v).expr) }.asJava)))
[error]              ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:187:14: method setSql in class Builder is deprecated
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.SparkSession.sql.$anonfun.cmd.$anonfun, origin=org.apache.spark.connect.proto.SqlCommand.Builder.setSql
[error]             .setSql(sqlText)
[error]              ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala:101:11: method createExternalTable in class Catalog is deprecated (since 2.2.0): use createTable instead.
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.catalog.Catalog.createExternalTable, origin=org.apache.spark.sql.api.Catalog.createExternalTable, version=2.2.0
[error]     super.createExternalTable(tableName, path)
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala:105:11: method createExternalTable in class Catalog is deprecated (since 2.2.0): use createTable instead.
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.catalog.Catalog.createExternalTable, origin=org.apache.spark.sql.api.Catalog.createExternalTable, version=2.2.0
[error]     super.createExternalTable(tableName, path, source)
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala:112:11: method createExternalTable in class Catalog is deprecated (since 2.2.0): use createTable instead.
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.catalog.Catalog.createExternalTable, origin=org.apache.spark.sql.api.Catalog.createExternalTable, version=2.2.0
[error]     super.createExternalTable(tableName, source, options)
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala:126:11: method createExternalTable in class Catalog is deprecated (since 2.2.0): use createTable instead.
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.catalog.Catalog.createExternalTable, origin=org.apache.spark.sql.api.Catalog.createExternalTable, version=2.2.0
[error]     super.createExternalTable(tableName, source, options)
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala:134:11: method createExternalTable in class Catalog is deprecated (since 2.2.0): use createTable instead.
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.catalog.Catalog.createExternalTable, origin=org.apache.spark.sql.api.Catalog.createExternalTable, version=2.2.0
[error]     super.createExternalTable(tableName, source, schema, options)
[error]           ^
[error] /Users/panbingkun/Developer/spark/spark-community/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/catalog/Catalog.scala:158:11: method createExternalTable in class Catalog is deprecated (since 2.2.0): use createTable instead.
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.catalog.Catalog.createExternalTable, origin=org.apache.spark.sql.api.Catalog.createExternalTable, version=2.2.0
[error]     super.createExternalTable(tableName, source, schema, options)
[error]           ^
[error] 10 errors found
[error] (core / Compile / compileIncremental) Compilation failed
[error] (connect-client-jvm / Compile / compileIncremental) Compilation failed
[error] Total time: 121 s (02:01), completed Sep 24, 2024, 3:39:40 PM

[error] /home/runner/work/spark/spark/sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/compression/CompressionSchemeBenchmark.scala:94:73: method formatted in class StringFormat is deprecated (since 2.12.16): Use `formatString.format(value)` instead of `value.formatted(formatString)`,
[error] or use the `f""` string interpolator. In Java 15 and later, `formatted` resolves to the new method in String which has reversed parameters.
[error] Applicable -Wconf / @nowarn filters for this fatal warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.execution.columnar.compression.CompressionSchemeBenchmark.runEncodeBenchmark.$anonfun.label, origin=scala.Predef.StringFormat.formatted, version=2.12.16
[error]       val label = s"${getFormattedClassName(scheme)}(${compressionRatio.formatted("%.3f")})"
[error]

cc @dongjoon-hyun @LuciferYang

Let me run Maven's compilation again.

At present, the testing based on sbt has passed: https://github.com/panbingkun/spark/runs/30569167386

authors: dongjoon-hyun panbingkun


text:
[SPARK-7990][SQL] Add methods to facilitate equi-join on multiple joining keys

JIRA: https://issues.apache.org/jira/browse/SPARK-7990

Merged build triggered.

Merged build started.

Test build #34097 has started for   PR 6616 at commit cc90015.

Test build #34097 has finished for   PR 6616 at commit cc90015.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait TypeCheckResult
 case class TypeCheckFailure(message: String) extends TypeCheckResult
abstract class UnaryArithmetic extends UnaryExpression
case class UnaryMinus(child: Expression) extends UnaryArithmetic
case class Sqrt(child: Expression) extends UnaryArithmetic
case class Abs(child: Expression) extends UnaryArithmetic
case class BitwiseNot(child: Expression) extends UnaryArithmetic
case class MaxOf(left: Expression, right: Expression) extends BinaryArithmetic
case class MinOf(left: Expression, right: Expression) extends BinaryArithmetic
case class Atan2(left: Expression, right: Expression)
case class Hypot(left: Expression, right: Expression)
case class EqualTo(left: Expression, right: Expression) extends BinaryComparison

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #34099 has started for   PR 6616 at commit 0400e89.

Test build #34099 has finished for   PR 6616 at commit 0400e89.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Merged build triggered.

Merged build started.

Test build #34185 has started for   PR 6616 at commit c43722c.

Test build #34185 has finished for   PR 6616 at commit c43722c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

retest this please.

Merged build triggered.

Merged build started.

Test build #34189 has started for   PR 6616 at commit c43722c.

Test build #34189 has finished for   PR 6616 at commit c43722c.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #34235 has started for   PR 6616 at commit cd5c888.

Test build #34235 has finished for   PR 6616 at commit cd5c888.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

@rxin Any more comments?

Thanks. I'm merging this.

authors: AmplabJenkins SparkQA rxin viirya


text:
[SPARK-8528] Expose SparkContext.applicationId in PySpark

Use case - we want to log applicationId (YARN in hour case) to request help with troubleshooting from the DevOps

Can one of the admins verify this patch?

The motivation for this change seems fine to me, although I have some nits to pick with the format of the docstrings.

Also, do you mind updating the PR title so that it's not truncated, e.g. "[SPARK-8528] Expose SparkContext.applicationId in PySpark"?

Done updating docstrings. Thank you for suggestions.

Jenkins, this is ok to test.

Merged build triggered.

Merged build started.

Test build #35561 has started for   PR 6936 at commit b4ebfc5.

Test build #35561 timed out     for PR 6936 at commit b4ebfc5     after a configured wait of 175m.

Merged build finished. Test FAILed.

Interesting build results - not tests failed, but whole build timeouted

Jenkins, please test it again

Jenkins, retest this please.

NAVER - http://www.naver.com/
sujkh@naver.com 님께 보내신 메일 <Re: [spark] [SPARK-8528] Expose SparkContext.applicationId in PySpark (#6936)> 이 다음과 같은 이유로 전송 실패했습니다.

받는 사람이 회원님의 메일을 수신차단 하였습니다.

Merged build triggered.

Merged build started.

Test build #35815 has started for   PR 6936 at commit b4ebfc5.

Test build #35815 has finished for   PR 6936 at commit b4ebfc5.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Hm, looks like last test haven't honored ELLIPSIS. Anyway this is doctest is not particularly useful. The contents of the applicationId is set outside of the test - and can be changed anytime, i.e. if tests would run in SparkContext in yarn or remote cluster - the result will be different.
So I'll simplify the test - it will just check that such property exists and returns something
__main__.SparkContext.applicationId
Failed example:
    sc.applicationId  # doctest: +ELLIPSIS
Expected:
    u'local-...'
Got:
    'local-1435285070164'

Please test updated version - I made the doctest stable

Merged build triggered.

Merged build started.

Test build #35861 has started for   PR 6936 at commit 14d77a8.

Test build #35861 has finished for   PR 6936 at commit 14d77a8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

BTW, I think that you can get this to work with doctest.ELLIPSIS; the problem is likely Python 2.x vs 3.x differences.  You can fix this by using the ignore_unicode_prefix decorator defined in rdd.py (see that file or grep the python/pyspark/ subdirectory for examples).

Thanks for hints
Retesting locally with python3 and python2. I will push update in ~30 minutes

Yeah, no rush; sorry for the hassle. At some point we should update the contributing guide to include tips on this stuff, such as the doctest trick.

Merged build triggered.

Merged build started.

Python doctests are rather limited in expressiveness, but efficient in keeping examples bug free in documentation.
Anywhere - here is another patch that works in both py2.6/2.7 and py3.4

Test build #35905 has started for   PR 6936 at commit 870338b.

Test build #35905 has finished for   PR 6936 at commit 870338b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

LGTM, so I'm going to merge this to master.  Thanks!

authors: AmplabJenkins JoshRosen SparkQA smartkiwi sujkh85


text:
[SPARK-10654][MLlib] Add columnSimilarities to IndexedRowMatrix

Add columnSimilarities to IndexedRowMatrix by delegating to functionality already in RowMatrix.
With a test.

Merged build triggered.

Merged build started.

Test build #42570 has started for   PR 8792 at commit 5553271.

Test build #42570 has finished for   PR 8792 at commit 5553271.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42570/
Test PASSed.

Merged build finished. Test PASSed.

LGTM. Merged into master. Thanks!

authors: AmplabJenkins SparkQA mengxr rezazadeh


text:
STORM-2646 NimbusClient Class cast exception when nimbus seeds is not an array of hosts

Can't use ["host"] it is : - > Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.List at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:76)

Hi @ecararus. Could you clarify a bit how the changes fix the issue you were seeing, and maybe also which Storm version you were seeing the issue on?
The default setting for nimbus.seeds is ["localhost"] (https://github.com/apache/storm/blob/master/conf/defaults.yaml#L61), which works fine for me when I try it locally, so I don't think there's a problem specifying a single host. We assume the configuration is a String list here https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/Config.java#L1052.
The change in L118 is checking if the config is a String[], which I would assume doesn't work since it is usually a List<String>? If you need to be able to write nimbus.seeds: "host" instead of nimbus.seeds: ["host"], I think you should update the Config with this annotation https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/Config.java#L1120.
Please also create an issue on JIRA https://issues.apache.org/jira and update your commit message and this PR's title to reflect the issue number (see the other PRs for example of what this should look like). Thanks.

Thank you @srdo for suggestion I've created jira issue also updated PR Title.
Inside my config file i have:
nimbus.seeds= ["somehost.org"]
nimbus.host= somehost.org # temporar fix
nimbus.thrift.port= 66270
more details about project config on jira.
About List i agree with @revans2 about validation and I would like to propose to accept single host also e.g. nimbus.seeds= somehost.org in this way we can support same configuration/behavior like it was with nimbus.host (that might be useful when host is provided by discovery or load distribution service).

@ecararus Thanks. I'm not too familiar with Flux, but going by what you listed in the issue, I'm guessing it's a problem where the ["nb.mycompany.org"] from dev.properties is being substituted into the topology.yaml as a string instead of a list.
Shot in the dark, does your topology.yaml do something like
config:
  nimbus.seeds: "${nimbus.seeds}"

If so, would it work to instead write
config:
  nimbus.seeds: ${nimbus.seeds}

@srdo my topology.yaml has:
config:
  nimbus.seeds: ${nimbus.seeds}

The property from dev.properties is loaded by FluxParser see line 100 there nimbus.seeds is treated as string and when on NimbusClient.java we try to :
 seeds = (List<String>) conf.get(Config.NIMBUS_SEEDS);

then I'm getting class cast ex. and in the exception we can see mesage Can't use ["host"] it is ....
As @revans2 suggested before cast we need to check if this is an array, in the situation described above it is configured as array of hosts but parser istreating as a string with "[host]" .

The patch right now looks fine to me, but I don't think it fixes your issue with flux.  Looking at the flux code inside FluxParser.java the ${NAME} substitution is literately a macro substitution that happens before the YAML parsing.  So if you want it to be an array you need to do like @srdo said and use ["nb.mycompany.org"] for nimbus.seeds in your properties file instead of just "nb.mycompany.org".

I'm a little confused why it doesn't work for you @ecararus. I added a bit to a Flux test to verify that it is possible to pass a list from a properties file into the Flux yaml file via FluxParser, both on master and 1.0.x. Does the diff at https://github.com/apache/storm/pull/2232 accurately reflect what your properties and yaml files contain?

@revans2 This pr will not fully fix my issue however i will not get anymore class cast exception. validation will fail according to annotation and ConfigValidator

@ecararus could you please take a look at https://github.com/revans2/incubator-storm/commit/0978338c7dd7ec23b035abb5f70ce088064bf60b
A lot of it you can ignore, but I added in a test on top of this patch into flux-core that shows that nimbus.seeds works (if you set it properly in your properties file).
Specifically the properties file needs to have in it
nimbus.seeds: ["HOST"]

and not
nimbus.seeds: "HOST"

When I change it to the latter the test fails with an exception of
java.lang.IllegalArgumentException: Field NIMBUS_SEEDS must be an Iterable but was a class java.lang.String
	at org.apache.storm.validation.ConfigValidationUtils$2.validateField(ConfigValidationUtils.java:125)
	at org.apache.storm.validation.ConfigValidationUtils$NestableFieldValidator.validateField(ConfigValidationUtils.java:48)
	at org.apache.storm.validation.ConfigValidation$ListEntryTypeValidator.validateField(ConfigValidation.java:311)
	at org.apache.storm.validation.ConfigValidation$ListEntryTypeValidator.validateField(ConfigValidation.java:306)
	at org.apache.storm.validation.ConfigValidation.validateField(ConfigValidation.java:697)
	at org.apache.storm.validation.ConfigValidation.validateFields(ConfigValidation.java:750)
	at org.apache.storm.validation.ConfigValidation.validateFields(ConfigValidation.java:711)
	at org.apache.storm.flux.parser.FluxParserTest.testCofigParamListParser(FluxParserTest.java:43)

Hello all
Thanks for advice i found the issue and fixed, i would like to apologize for misleading by pointing on FluxParser(however propertied.get(key) is returning string thanks to Yaml.load nimbus.seeds is transformed to a list).  I'm using storm-maven-plugin so the issue was there i'm calling NimbusClient by passing config as:
                in = new FileInputStream(properiesFileLocation);
                Properties prop = new Properties();
                prop.load(in);
                return config.putAll(prop);

I think we can close current issue and apply validation before usage of configuration.
Thanks everybody for help.

authors: ecararus revans2 srdo

repo: apache/storm


text:
[Struts-7] Merge master to 7xx 2024-01-20

Recent merge request of the master branch

@jdyer1 FYI

repo: apache/struts


text:
Hide the 'Edit dashboard' button on standalone

Is your feature request related to a problem? Please describe.
On the project that I'm working on, when we want to display a dashboard we always use the standalone mode, since it removes the navbar from superset that we don't use while analyzing the data.
We are left with the header and the charts. The former has the title, published status, favorite star, the 'Edit dashboard' button and its dropdown.
Currently to hide some elements such as the favorite star we add some css to the dashboard (display: none).
However if we do the same to the 'Edit Dashboard' button, every time we need to edit the dashboard we need to comment the css from the button, edit the dashboard and then uncomment back.
Describe the solution you'd like
If the standalone flag was on the url, the 'Edit Dashboard' wouldn't be displayed/rendered.
Describe alternatives you've considered
Besides the 'Edit Dashboard' button I also thought on not rending the other elements, leaving just the dropdown. Don't know if this is to much though.
I checked the react code and the is no standalone prop on the component rending these elements ('Edit Dashboard' button and so on).
Additional context
:no_mouth:

Did you try with CSS?
.standalone button.slice-header-controls-trigger {
display: none;
}

It does hide the drop down on each chart but the 'Edit dashboard' its still being rendered

Sorry, wrong CSS.
This will hide the entire header, including menu:
.standalone .dashboard-header {
display: none;
}

Hmm, its an option to consider. Thanks :+1:
But maybe I want to keep, at least, the dashboard title

Maybe managing viewer/editor permission for a dashboard can work for you
https://github.com/apache/incubator-superset/issues/10408

@amitNielsen yeah, I think that might be a possible solution.
If you end up creating a PR link this issue. Or I can close it now and say that your SIP solves this.

I already associated between the 2
Basically what you're asking is just another filter type in the dashboard filter component
I'll let you know when you're specific feature can be implemented as part of that SIP

#10408

update: you can now use the following technique
Standalone=1 (hide only navigation bar
Standalone=2 (hide navigation bar and dashboard title)
also there is a specific permission you can choose to assign/deassign from a role and it will show/hide the edit button accordingly

update: you can now use the following technique
Standalone=1 (hide only navigation bar
Standalone=2 (hide navigation bar and dashboard title)
also there is a specific permission you can choose to assign/deassign from a role and it will show/hide the edit button accordingly

@amitmiran137 It seems that "Standalone=2" has a conflict with "Set auto-refresh interval".
For example, the refresh interval is set to 10 seconds. With "standalone=1", the refresh goes well, but when "standalone=2", the refresh is deactivated.
May you please check it?

source: gharchive/issue

repo: apache/superset


text:
Superset Inconsistent with SQL LAB csv downloads

when we are running the query on superset the query gives output ,although our limit is set to 10L, also the table has crores of rows, but only sadly in each iteration I download 10k,12k,15k records come. Also when I hit the API from backend via python its working fine but via superset UI wanted to know why is it so inconsistent

The database is PostgreSQL

Should I try this with 2.0.1 ? or 2.1.0 is stable?

We no longer support 2.x, even. If anyone is still encountering this in current versions (3.x) please open a new Issue or a PR to address the problem.


text:
Oracle as Metadata- Not working

Bug description
superset db upgrade
superset init
This throws a lot of errors from missing columns in metadata tables, and foreign key issues. reserved keyword like row issue.
I know we need to use MySQL and PostgreSQL. My firm wants to use Oracle.
Any update ?
How to reproduce the bug
superset db upgrade
superset init
Screenshots/recordings
No response
Superset version
4.0.1
Python version
3.9
Node version
16
Browser
Chrome
Additional context
No response
Checklist

[ ] I have searched Superset docs and Slack and didn't find a solution to my problem.
[ ] I have searched the GitHub issue tracker and didn't find a similar bug report.
[ ] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

I know we need to use MySQL or PostgreSQL. My firm wants to use Oracle.

You have a good handle on this.  There is no plan to support Oracle as a metadata store DB, so I'm going to close this issue.  If you were to go that route and somehow get it working, there would be no community support available and it's always possible that subsequent changes would break something that previously worked.
I can relate, my org uses MS SQL Server exclusively for application DBs.  We decided that to set up a managed PostgreSQL DB in order to use Superset.  For what it's worth, the amount of knowledge of PostgreSQL needed to maintain a Superset instance is very low.

authors: selvadinesh sfirke


text:
Importing CSV: the csv has a date column but it appears in the database as a string

Bug description
CSV import loads string type every time, instead of date. I check the Columns to be parsed as dates option but it still doesn't work. I tried
Database:File Uploads
Schema: main
I checked all data types in CSV for example YYYY-MM-DD or 2024-07-09 09:15:05
Version: 4.1.0.3
Screenshots/recordings
No response
Superset version
4.1.1
Python version
3.9
Node version
16
Browser
Chrome
Additional context
No response
Checklist

[X] I have searched Superset docs and Slack and didn't find a solution to my problem.
[X] I have searched the GitHub issue tracker and didn't find a similar bug report.
[ ] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

Did the same file + upload choices successfully get turned into a date using Superset version 4.0.2?

I hadn't used the CSV import feature before updating to version 4.1.0.3.
When I uploaded the CSV file to BigQuery and connected it to the database, everything worked correctly.
I am working with Preset.

I can't replicate this in Superset 4.1.1. I created a CSV with a column called "datetime" and a value of 2024-07-09 09:15:05. I then uploaded it into Superset, specifying that the column called "datetime" was a date time column. It loaded into my data warehouse with the correct column type and I was able to make time series charts in Superset with it.
My only guess is that there's some invalid date-time value lurking in that column and causing it to be a string.  I once had this problem and finally figured out that a user had inputted some invalid date, like 2023-02-29 when that year wasn't a leap year for instance. Or an invalid time like 25:00. Once I deleted those few rows, it worked correctly.
If you are able to create a reproducible minimal example that indicates this is indeed a problem with Superset, please share it on this issue.

I created a CSV file with random data, where the date format is similar to yours, but the bug still persists.

health_center_data (1).csv

Hi @Kamilooosx , once the upload has finished, if you're having trouble with the column type, you should be able to go into your dataset under the Columns tab and mark the time column as a temporal column to use it as a time column in charts

Yeah ! It works ! But when i want to filter by Date it's generate error :C

authors: Kamilooosx sfirke yousoph


text:
fix(26338): replace chartsInScope references at import time

SUMMARY
See the description here: https://github.com/apache/superset/issues/26338
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 69.14%. Comparing base (29633e7) to head (c6732c4).
Report is 781 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #26389      +/-   ##
==========================================
- Coverage   69.14%   69.14%   -0.01%
==========================================
  Files        1946     1946
  Lines       75989    75992       +3
  Branches     8479     8479
==========================================
+ Hits        52544    52546       +2
- Misses      21266    21267       +1
  Partials     2179     2179

Flag
Coverage Δ

hive
53.68% <0.00%> (-0.01%)
:arrow_down:

mysql
78.07% <0.00%> (+0.01%)
:arrow_up:

postgres
78.16% <0.00%> (-0.01%)
:arrow_down:

presto
53.63% <0.00%> (-0.01%)
:arrow_down:

python
82.86% <100.00%> (-0.01%)
:arrow_down:

sqlite
76.82% <0.00%> (-0.01%)
:arrow_down:

unit
55.80% <100.00%> (+<0.01%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@eschutho / @betodealmeida still hoping someone with more context on import/export can help review this :)

All, as mentioned in https://github.com/apache/superset/issues/26338, a new commit has been added to the PR to make the charts references fix more complete as I missed a couple of locations where references were not updated at import time.
I guess the PR is re-ready for being reviewed. Thanks in advance.

authors: codecov-commenter rdubois-kh rusackas


text:
THRIFT-4429: Make TThreadPoolServer.executorService_ available in inherited classes and refactor methods to be able customization

Plus some classes cleanup from unused imports

@jeking3 , could you please to look at this PR.
Thanks in advance.

Thanks a lot for review.

repo: apache/thrift


text:
add check in github action for java and kotlin

add check in github action for java and kotlin, and also enable ktmftCheck for kotlin, similar to spotlessCheck
as a follow up on:

#2581

[ ] Did you create an Apache Jira ticket?  (not required for trivial changes)
[ ] If a ticket exists: Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?
[ ] Did you squash your changes to a single commit?  (not required, but preferred)
[ ] Did you do your best to avoid breaking changes?  If one was needed, did you label the Jira ticket with "Breaking-Change"?
[ ] If your change does not involve any code, include [skip ci] anywhere in the commit message to free up build resources.

@ctubbsii this is a follow up so i didn't bother creating a new jira for this

the kotlin code changes in this pull request is done automatically using ktfmtFormat


text:
Update Traffic Portal to account for new response/request structures for /deliveryservices/{{ID}}/servers

I'm submitting a ...

new feature / enhancement request
also kind of a prophecy of a bug that will exist in the future

Description
As soon as #4614 and/or #4615 is/are closed, the Traffic Portal functionality related to servers-to-Delivery Service assignments will be broken. It'll need to be updated to account for the new response/request structures.
This depends on #4614 and #4615 (and therefore #4611)
Traffic Control components affected ...

Traffic Portal

Acceptance Criteria

All Traffic Portal tests are passing
Traffic Portal is capable of displaying, sorting, removing from and assigning to the list of servers assigned to a Delivery Service

See the Additional Server Interfaces blueprint for details.

This was resolved by #4747

repo: apache/trafficcontrol


text:
TPv2: Update required node version to >= 18, eslint-plugin-jsdoc

Closes: #7932

Which Traffic Control components are affected by this PR?

Traffic Portal v2

What is the best way to verify this PR?

If this is a bugfix, which Traffic Control versions contained the bug?

PR submission checklist

[x] This PR has tests
[x] This PR has documentation
[x] This PR has a CHANGELOG.md entry
[x] This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY (see the Apache Software Foundation's security guidelines for details)

I also believe that our current eslint-plugin-jsdoc is not working correctly, when we run: npm run lint the current version