text:
[SPARK-25338][Test] Ensure to call super.beforeAll() and super.afterAll() in test cases

What changes were proposed in this pull request?
This PR ensures to call super.afterAll() in override afterAll() method for test suites.

Some suites did not call super.afterAll()
Some suites may call super.afterAll() only under certain condition
Others never call super.afterAll().

This PR also ensures to call super.beforeAll() in override beforeAll() for test suites.
How was this patch tested?
Existing UTs

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2859/
Test PASSed.

Test build #95698 has started for PR 22337 at commit a429ddb.

Test build #95698 has finished for PR 22337 at commit a429ddb.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95698/
Test FAILed.

retest this please

Test build #95703 has finished for PR 22337 at commit a429ddb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95703/
Test PASSed.

Hi, @kiszk . Oh, did you check them all? Are these all of them?

Could you check the other suites like ExternalAppendOnlyUnsafeRowArraySuite and TakeOrderedAndProjectSuite, too?

Sure, I just focused on files under external. Let me address other files, too.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2895/
Test PASSed.

Test build #95748 has started for PR 22337 at commit 32db2fc.

Test build #95748 has finished for PR 22337 at commit 32db2fc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95748/
Test PASSed.

cc @dongjoon-hyun

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2932/
Test PASSed.

Test build #95805 has started for PR 22337 at commit 45c7a10.

Test build #95805 has finished for PR 22337 at commit 45c7a10.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95805/
Test PASSed.

Merged build finished. Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2942/
Test PASSed.

Test build #95821 has started for PR 22337 at commit 6a4cbcf.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2943/
Test PASSed.

Test build #95822 has started for PR 22337 at commit a314776.

Test build #95822 has finished for PR 22337 at commit a314776.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #95821 has finished for PR 22337 at commit 6a4cbcf.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95822/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95821/
Test FAILed.

retest this please

Test build #95832 has started for PR 22337 at commit a314776.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2947/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2948/
Test PASSed.

Test build #95833 has started for PR 22337 at commit 309e265.

Test build #95832 has finished for PR 22337 at commit a314776.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95832/
Test FAILed.

Test build #95833 has finished for PR 22337 at commit 309e265.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95833/
Test PASSed.

Merged build finished. Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2984/
Test PASSed.

Test build #95890 has started for PR 22337 at commit 2d9e34a.

Test build #95890 has finished for PR 22337 at commit 2d9e34a.

This patch fails from timeout after a configured wait of `400m`.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class FlatMapGroupsWithStateSuite extends StateStoreMetricsTest
class StreamingAggregationSuite extends StateStoreMetricsTest with Assertions
class StreamingDeduplicationSuite extends StateStoreMetricsTest

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95890/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2992/
Test PASSed.

Test build #95909 has started for PR 22337 at commit 2d9e34a.

Test build #95909 has finished for PR 22337 at commit 2d9e34a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class FlatMapGroupsWithStateSuite extends StateStoreMetricsTest
class StreamingAggregationSuite extends StateStoreMetricsTest with Assertions
class StreamingDeduplicationSuite extends StateStoreMetricsTest

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95909/
Test FAILed.

Retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2999/
Test PASSed.

Test build #95917 has started for PR 22337 at commit 2d9e34a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3000/
Test PASSed.

Test build #95917 has finished for PR 22337 at commit 2d9e34a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class FlatMapGroupsWithStateSuite extends StateStoreMetricsTest
class StreamingAggregationSuite extends StateStoreMetricsTest with Assertions
class StreamingDeduplicationSuite extends StateStoreMetricsTest

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95917/
Test FAILed.

retest this please

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3012/
Test PASSed.

Merged build finished. Test PASSed.

Test build #95938 has started for PR 22337 at commit 2d9e34a.

Test build #95938 has finished for PR 22337 at commit 2d9e34a.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class FlatMapGroupsWithStateSuite extends StateStoreMetricsTest
class StreamingAggregationSuite extends StateStoreMetricsTest with Assertions
class StreamingDeduplicationSuite extends StateStoreMetricsTest

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95938/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3024/
Test PASSed.

Test build #95958 has started for PR 22337 at commit 2d9e34a.

Test build #95958 has finished for PR 22337 at commit 2d9e34a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class FlatMapGroupsWithStateSuite extends StateStoreMetricsTest
class StreamingAggregationSuite extends StateStoreMetricsTest with Assertions
class StreamingDeduplicationSuite extends StateStoreMetricsTest

Ur, could you resolve the conflicts, @kiszk ?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3087/
Test PASSed.

Test build #96039 has started for PR 22337 at commit 53d1ba2.

Test build #96039 has finished for PR 22337 at commit 53d1ba2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class FlatMapGroupsWithStateSuite extends StateStoreMetricsTest
class StreamingAggregationSuite extends StateStoreMetricsTest with Assertions
class StreamingDeduplicationSuite extends StateStoreMetricsTest

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96039/
Test PASSed.

Merged to master.

authors: AmplabJenkins SparkQA dongjoon-hyun kiszk


text:
[SPARK-25476][SPARK-25510][TEST] Refactor AggregateBenchmark and add a new trait to better support Dataset and DataFrame API

What changes were proposed in this pull request?
This PR does 2 things:

Add a new trait(SqlBasedBenchmark) to better support Dataset and DataFrame API.
Refactor AggregateBenchmark to use main method. Generate benchmark result:

SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.AggregateBenchmark"

How was this patch tested?
manual tests

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3290/
Test PASSed.

Test build #96338 has started for PR 22484 at commit 649f296.

Test build #96338 has finished for PR 22484 at commit 649f296.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96338/
Test PASSed.

Test build #96473 has started for PR 22484 at commit 42230b6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3380/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96473/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3399/
Test PASSed.

Test build #96500 has started for PR 22484 at commit 2d778a4.

Test build #96500 has finished for PR 22484 at commit 2d778a4.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait SqlBasedBenchmark extends BenchmarkBase

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96500/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test build #96502 has started for PR 22484 at commit 2d778a4.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3401/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3403/
Test PASSed.

Test build #96502 has finished for PR 22484 at commit 2d778a4.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait SqlBasedBenchmark extends BenchmarkBase

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96502/
Test PASSed.

LGTM, cc @dongjoon-hyun for sign-off

@wangyum . Could you make the title and description up-to-date for this PR content? Also, please update JIRA title and description, too.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3416/
Test PASSed.

Test build #96529 has started for PR 22484 at commit 536d4e9.

Thank you for updating, @wangyum .

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3417/
Test PASSed.

Test build #96530 has started for PR 22484 at commit f783099.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3418/
Test PASSed.

Test build #96531 has started for PR 22484 at commit d5fecdc.

@wangyum . If you don't mind, could you review my PR to your branch, https://github.com/wangyum/spark/pull/11, which

deduplicates withSQLConf back
runs on AWS EC2 r3.xlarge?

Also, cc @gatorsmile . Since we need withSQLConf in both Benchmark and TestSuite, I want to introduce SupportWithSQLConf trait .

Thanks @dongjoon-hyun. except withSQLConf. We need withTempTable and withTempPath:
https://github.com/apache/spark/blob/d25f425c9652a3611dd5fea8a37df4abb13e126e/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala#L63-L83

Yep. This PR already is introducing new trait for Benchmark as a part of grand refactoring plan. I think the other new traits also possible at this time. We will see. :)

Test build #96530 has finished for PR 22484 at commit f783099.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96530/
Test PASSed.

Test build #96529 has finished for PR 22484 at commit 536d4e9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96529/
Test PASSed.

Test build #96531 has finished for PR 22484 at commit d5fecdc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96531/
Test PASSed.

I made an official PR for withSQLConf SPARK-25534 which reduces the number of withSQLConf from 6 to 3. The three were the minimal different implementations.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3478/
Test PASSed.

Test build #96616 has started for PR 22484 at commit 0fae54d.

Test build #96616 has finished for PR 22484 at commit 0fae54d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait SqlBasedBenchmark extends BenchmarkBase with SQLHelper

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96616/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3483/
Test PASSed.

Test build #96624 has started for PR 22484 at commit 0fae54d.

Test build #96624 has finished for PR 22484 at commit 0fae54d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait SqlBasedBenchmark extends BenchmarkBase with SQLHelper

@wangyum Could you review and merge https://github.com/wangyum/spark/pull/12 , too?

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3516/
Test PASSed.

Merged build finished. Test PASSed.

Test build #96666 has started for PR 22484 at commit 45add92.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3517/
Test PASSed.

Test build #96667 has started for PR 22484 at commit 3439992.

Test build #96667 has finished for PR 22484 at commit 3439992.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #96666 has finished for PR 22484 at commit 45add92.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96667/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96666/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3534/
Test PASSed.

Test build #96686 has started for PR 22484 at commit 3439992.

Test build #96686 has finished for PR 22484 at commit 3439992.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96686/
Test PASSed.

@dongjoon-hyun Other refactorings are waiting for this commit.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3601/
Test PASSed.

Test build #96811 has started for PR 22484 at commit 6c46ad5.

Test build #96811 has finished for PR 22484 at commit 6c46ad5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96811/
Test PASSed.

Merged to master.

Thank you, @wangyum , @cloud-fan and @gengliangwang !

source: gharchive/pull-request

authors: AmplabJenkins SparkQA cloud-fan dilipbiswal dongjoon-hyun wangyum

repo: apache/spark

license_type: permissive

license_source: github-api


text:
[SPARK-26686][SS] Remove unnecessary KafkaSourceProvider parameter lowercase conversion

What changes were proposed in this pull request?
KafkaSourceProvider receives lowercase parameters but it converts them again which is unnecessary.
In this PR I've removed these conversions.
How was this patch tested?
Existing unit tests.

As the API is not explicitly stating that it provides lowercase parameters better to keep sometimes double conversion for safety reasons.
The bad part is there are code lines which are depending on the assumption that these APIs providing lowercase parameters.


text:
[SPARK-27949][SQL]Support ANSI SQL grammar substring(string_expression from n1 [for n2])

What changes were proposed in this pull request?
Currently, function substr/substring's usage is like substring(string_expression, n1 [,n2]).
But the ANSI SQL defined the pattern for substr/substring is like substring(string_expression from n1 [for n2]). This gap make some inconvenient when we switch to the SparkSQL.
Can we support the ANSI pattern like substring(string_expression from n1 [for n2])?
ANSI SQL-92: http://www.contrib.andrew.cmu.edu/~shadow/sql/sql1992.txt
Below are the mainly DB engines to support the ANSI standard for substring.

Teradata https://docs.teradata.com/reader/756LNiPSFdY~4JcCCcR5Cw/XnePye0Cwexw6Pny_qnxVA
Redshift https://docs.aws.amazon.com/redshift/latest/dg/r_SUBSTRING.html
MySQL https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_substring
Postgresql https://www.postgresql.org/docs/9.1/functions-string.html

Oracle, SQL Server, Hive, Presto don't have this syntax.
How was this patch tested?
Pass the Jenkins with the updated test cases.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11430/
Test PASSed.

Test build #106179 has started for PR 24802 at commit 385f327.

@dongjoon-hyun Just update PR description to list other DB engines' status of substring.

Thanks, @lipzhu !

cc @gatorsmile

Test build #106179 has finished for PR 24802 at commit 385f327.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106179/
Test PASSed.

This PR is not to implement/change the actual execution of substring. The changes made in the parser look fine to me. @lipzhu Please address the minor comments. I would merge it as long as you fix them.
https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/bit.sql also has the test cases to cover this new support. When we port these tests, we will have better test coverage on this.

Thanks. It's nice to have this. I'll leave this to @gatorsmile .

@gatorsmile Thanks for your review and comments, I just update the alias name from field to str.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11583/
Test PASSed.

Test build #106337 has started for PR 24802 at commit 086388f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11585/
Test PASSed.

Test build #106339 has started for PR 24802 at commit 67da08a.

Thank you for updating, @lipzhu . I updated the PR title and description a little bit.

@dongjoon-hyun Thanks for your review, I just did some changes according to your suggestions.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11587/
Test PASSed.

Test build #106341 has started for PR 24802 at commit 62d386d.

Test build #106341 has finished for PR 24802 at commit 62d386d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #106339 has finished for PR 24802 at commit 67da08a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #106337 has finished for PR 24802 at commit 086388f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106341/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106339/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106337/
Test FAILed.

Retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11589/
Test PASSed.

Test build #106343 has started for PR 24802 at commit 62d386d.

Test build #106343 has finished for PR 24802 at commit 62d386d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106343/
Test PASSed.

authors: AmplabJenkins SparkQA dongjoon-hyun gatorsmile lipzhu


text:
[WIP] Move PullOutPythonUDFInJoinCondition rule into 'Extract Python UDFs'

What changes were proposed in this pull request?
This PR targets to move PullOutPythonUDFInJoinCondition to 'Extract Python UDFs' together with other Python UDF related rules.
I will file a JIRA later. I would like to run the tests first.
How was this patch tested?
Existing tests should cover.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13735/
Test PASSed.

Test build #108648 has started for PR 25358 at commit b3c6b1e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13737/
Test PASSed.

Test build #108651 has started for PR 25358 at commit 29c6dc0.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13738/
Test PASSed.

Test build #108652 has started for PR 25358 at commit 7408b3a.

Test build #108648 has finished for PR 25358 at commit b3c6b1e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108648/
Test FAILed.

Test build #108651 has finished for PR 25358 at commit 29c6dc0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108651/
Test PASSed.

Test build #108652 has finished for PR 25358 at commit 7408b3a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108652/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13751/
Test PASSed.

Test build #108666 has started for PR 25358 at commit 4bc07bb.

Test build #108673 has started for PR 25358 at commit e6d22c8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13759/
Test PASSed.

Test build #108666 has finished for PR 25358 at commit 4bc07bb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108666/
Test PASSed.

Test build #108673 has finished for PR 25358 at commit e6d22c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108673/
Test PASSed.

Merged build finished. Test PASSed.

LGTM
Merged to master.

authors: AmplabJenkins HyukjinKwon SparkQA gatorsmile


text:
[SPARK-31293][DSTREAMS][Kinesis][DOC] Fix wrong examples and help messages for Kinesis integration

What changes were proposed in this pull request?

This PR fixes wrong command examples, parameter descriptions and help message format for Amazon Kinesis integration with Spark Streaming.
Why are the changes needed?

To improve usability of those commands.
Does this PR introduce any user-facing change?

No
How was this patch tested?

I ran the fixed commands manually and confirmed they worked as expected.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

Test build #120537 has started for PR 28063 at commit 20efadd.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25243/
Test PASSed.

Test build #120537 has finished for PR 28063 at commit 20efadd.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120537/
Test FAILed.

Test build #120549 has started for PR 28063 at commit 742c00a.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25255/
Test PASSed.

Merged build finished. Test PASSed.

Test build #120549 has finished for PR 28063 at commit 742c00a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120549/
Test PASSed.

+1, LGTM. Thank you, @sekikn , @srowen , and @kiszk .
Merged to master/3.0/2.4.

authors: AmplabJenkins SparkQA dongjoon-hyun kiszk sekikn


text:
[SPARK-30660][ML][PYSPARK] LinearRegression blockify input vectors

What changes were proposed in this pull request?
1, add new param blockSize;
2, add a new class InstanceBlock;
3, if blockSize==1, keep original behavior; if blockSize>1, stack input vectors to blocks (like ALS/MLP);
4, if blockSize>1, standardize the input outside of optimization procedure;
Why are the changes needed?
it will obtain performance gain on dense datasets
1, reduce RAM to persist traing dataset;
2, use Level-2 BLAS routines;
Does this PR introduce any user-facing change?
Yes, a new param is added
How was this patch tested?
existing and added testsuites

This is a update of https://github.com/apache/spark/pull/27396, this PR avoids performance regression on sparse datasets with blockSize=1 by default.

Test build #122386 has started for PR 28471 at commit 643470c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27051/
Test PASSed.

Test build #122386 has finished for PR 28471 at commit 643470c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122386/
Test PASSed.

testCode:

import org.apache.spark.ml.regression._
import org.apache.spark.storage.StorageLevel

val df = spark.read.option("numFeatures", "2000").format("libsvm").load("/data1/Datasets/epsilon/epsilon_normalized.t")
df.persist(StorageLevel.MEMORY_AND_DISK)
df.count

// Default SquaredError
val lir = new LinearRegression().setMaxIter(50).setSolver("l-bfgs")
lr.fit(df)

val results = Seq(1, 4, 16, 64, 256, 1024, 4096).map { size => val start = System.currentTimeMillis; val model = lir.setBlockSize(size).fit(df); val end = System.currentTimeMillis; (size, model, end - start) }

// Huber
val lir = new LinearRegression().setMaxIter(50).setSolver("l-bfgs").setLoss("huber")

val results = Seq(1, 4, 16, 64, 256, 1024, 4096).map { size => val start = System.currentTimeMillis; val model = lir.setBlockSize(size).fit(df); val end = System.currentTimeMillis; (size, model, end - start) }

Result:
Huber
scala> results.foreach(t => println(t._2.coefficients.toString.take(100)))
[0.6424729506929868,-1.2292544634895406,1.7757284244387601,0.2974549255492886,0.935593736145218,-0.7
[0.6425694454021473,-1.22927248948364,1.7757268011427252,0.2974675136619897,0.9357357899375097,-0.77
[0.6424853073405433,-1.2292568932224641,1.775728104037945,0.29745659508182537,0.9356121907344618,-0.
[0.6429959531002037,-1.229354196708174,1.7757175326389232,0.29752415047198205,0.9363681130113419,-0.
[0.6424842544336862,-1.2292566837366543,1.775728142782258,0.29745645263402737,0.9356106116206417,-0.
[0.6430623639968113,-1.2293667308116527,1.7757163246117786,0.29753287650971944,0.9364661504709426,-0
[0.6421067597074761,-1.229184819802838,1.7757358482626238,0.2974067609553463,0.9350521980706877,-0.7

scala>

scala> results.map(_._2.summary.totalIterations)
res22: Seq[Int] = List(51, 51, 51, 51, 51, 51, 51)

scala> results.map(_._3)
res23: Seq[Long] = List(135189, 12046, 11783, 14307, 14399, 14026, 14329)

Up to 11X speedup
SquaredError

scala> results.foreach(t => println(t._2.coefficients.toString.take(100)))
[-0.2652587613623121,-0.0707048016667831,0.420750805149307,0.09194452205365045,0.05059855709172461,0
[-0.26525865677193483,-0.07070480422610806,0.42075057109293873,0.09194450660409814,0.050598582385999
[-0.26526635878841553,-0.07070459501522909,0.42076764197826116,0.09194564264821924,0.050596757952969
[-0.26525612879310023,-0.07070486362689087,0.42074489424648864,0.09194413298339,0.050599198284442995
[-0.2652611612036013,-0.07070474076806575,0.42075615827182555,0.09194487637000792,0.0505979807182646
[-0.26526190169346425,-0.07070472114668785,0.42075780340712315,0.0919449856303737,0.0505978044045689
[-0.265262317030222,-0.07070470997227678,0.42075872481615506,0.09194504690071631,0.05059770582212817

scala> results.map(_._2.summary.totalIterations)
res26: Seq[Int] = List(51, 51, 51, 51, 51, 51, 51)

scala> results.map(_._3)
res27: Seq[Long] = List(71269, 11828, 12254, 15331, 14963, 14420, 14022)

Up to 6X speedup

merged to master

Let's don't merge without review or approval, @zhengruifeng

authors: AmplabJenkins HyukjinKwon SparkQA zhengruifeng


text:
[SPARK-35289][SQL] Make CatalogString contains nullable information

What changes were proposed in this pull request?
This patch proposes to make CatalogString contains nullable information to help understand Casting failure message.
Why are the changes needed?
When trying to cast ArrayType/MapType/StructType from containing NULL to not contain NULL, we will get a confusing failure message.
Here's a example:
test("array casting") {
 val arrayContainsNull = Literal.create(Seq("contains", "null", null),
 ArrayType(StringType, containsNull = true))

 val ret = cast(arrayContainsNull, ArrayType(StringType, containsNull = false))
 assert(ret.resolved === false)
 println(ret.typeCheckFailureMessage)
}

Failure massage:
cannot cast array<string> to array<string>

From the above error message, it is not clear why the error is reported, because the data type before and after is the same.
Therefore, we need to specify nullable info for ArrayType/MapType?/StructType in the Catalog String to identify the reason for Casting Failures.
Failure massage after this patch:
cannot cast array<string(nullable=true)> to array<string(nullable=false)>

Does this PR introduce any user-facing change?
No
How was this patch tested?
New UT

Can one of the admins verify this patch?

I plan to fix these failed unit tests later. Any comments on this PR? I think it needs more discussion to help it complete.

Could you also give the current unclear error messages and the proposed catalog strings for MapType and StructType? Then we can discuss the ideal format for catalog strings based on that.

authors: AmplabJenkins hezuojiao wzhfy


text:
[SPARK-39758][SQL][3.2] Fix NPE from the regexp functions on invalid patterns

What changes were proposed in this pull request?
In the PR, I propose to catch PatternSyntaxException while compiling the regexp pattern by the regexp_extract, regexp_extract_all and regexp_instr, and substitute the exception by Spark's exception w/ the error class INVALID_PARAMETER_VALUE. In this way, Spark SQL will output the error in the form:
org.apache.spark.SparkRuntimeException
The value of parameter(s) 'regexp' in `regexp_instr` is invalid: ) ?

instead of (on Spark 3.3.0):
java.lang.NullPointerException: null

Also I propose to set lastRegex only after the compilation of the regexp pattern completes successfully.
This is a backport of https://github.com/apache/spark/pull/37171.
Why are the changes needed?
The changes fix NPE portrayed by the code on Spark 3.3.0:
spark-sql> SELECT regexp_extract('1a 2b 14m', '(?l)');
22/07/12 19:07:21 ERROR SparkSQLDriver: Failed in [SELECT regexp_extract('1a 2b 14m', '(?l)')]
java.lang.NullPointerException: null
	at org.apache.spark.sql.catalyst.expressions.RegExpExtractBase.getLastMatcher(regexpExpressions.scala:768) ~[spark-catalyst_2.12-3.3.0.jar:3.3.0]

This should improve user experience with Spark SQL.
Does this PR introduce any user-facing change?
No. In regular cases, the behavior is the same but users will observe different exceptions (error messages) after the changes.
How was this patch tested?
By running new tests:
$ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z regexp-functions.sql"
$ build/sbt "test:testOnly *.RegexpExpressionsSuite"
$ build/sbt "sql/test:testOnly org.apache.spark.sql.expressions.ExpressionInfoSuite"

Authored-by: Max Gekk max.gekk@gmail.com
Signed-off-by: Max Gekk max.gekk@gmail.com
(cherry picked from commit 5b96bd5cf8f44eee7a16cd027d37dec552ed5a6a)
Signed-off-by: Max Gekk max.gekk@gmail.com

@cloud-fan Tests passed in fact. Could you look at the backport, please.

Merging to 3.2. Thank you, @srowen @cloud-fan for review.


text:
[Minor] Fix the value represented by spark.executor.id for the driver of local mode.

When we run application in local mode, the property spark.executor.id represents driver for the driver.
While we run in anything else mode, the property represents <driver for the driver.
It's inconsistent.
This issue is minor so I didn't file this in JIRA.

Test build #25234 has started for   PR 3812 at commit d885498.

This patch merges cleanly.

Test build #25234 has finished for   PR 3812 at commit d885498.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25234/
Test FAILed.

retest this please.

Test build #25243 has started for   PR 3812 at commit d885498.

This patch merges cleanly.

LGTM I will merge this once the tests pass. Quick question though this seems to affect not only local mode right? If so, could you remove that from the title and the description?

@andrewor14 Yeah, this can affects to any other modes so I changed the description and title.

Test build #25243 has finished for   PR 3812 at commit d885498.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25243/
Test PASSed.

@srowen @andrewor14 This change can affects for users monitoring driver's metrics. The property is used for metrics value.
eg: <EXTRA_ELEMENT NAME="DESC" VAL="local-1422350105844.<driver>.jvm.pools.Eden-Space.usage"/>,  parser error : Unescaped '<' not allowed in attributes values.

@sarutak I still don't get the purpose of this change, and as you say it not only potentially broke metrics, but seems to have. Although maybe you can fix it forward, is this a good change?

Sorry I didn't notice this change affects for specific metrics sinks.
I still think this change is needed. The identifier is used for metrics name so without this change, the metrics name for the driver is different between local mode and any other mode. It confuses monitoring.
Also, regardless of this change, the parse issue could occur because the identifier <driver> has been used for spark.executor.id in local mode even before this change.
I'll fix this issue by renaming the identifier from <driver> to driver.

OK. Can this be fixed by just escaping the value in the XML output? because if it's <driver> elsewhere then it seems like it would also break the output elsewhere.

So, how about escaping the specific characters like < or > in each sink instead of changing the identifier?

Maybe, we can resolve the parse issue by updating com.codahale.metrics library from 3.0.0 to 3.1.0. 3.1.0 sanitizes < and >.

Already proposed at https://issues.apache.org/jira/browse/SPARK-5413. Perhaps we can close this issue and PRs then if that's the way forward.

O.K, let's wait for being merged the PR.

authors: AmplabJenkins SparkQA andrewor14 sarutak srowen wangxiaojing


text:
[SPARK-41365][UI][Backport] Stages UI page fails to load for proxy in specific yarn environment

backport https://github.com/apache/spark/pull/38882
What changes were proposed in this pull request?
Stages UI page fails to load for proxy in some specific yarn environment.
Why are the changes needed?
My environment CDH 5.8 , click to enter the spark UI from the yarn Resource Manager page
when visit the stage URI, it fails to load,  URI is
http://:8088/proxy/application_1669877165233_0021/stages/stage/?id=0&attempt=0
The issue is similar to, the final phenomenon of the issue is the same, because the parameter encode twice
SPARK-32467
SPARK-33611
The two issues solve two scenarios to avoid encode twice:

https redirect proxy
set reverse proxy enabled (spark.ui.reverseProxy)  in Nginx

But if encode twice due to other reasons, such as this issue (yarn proxy), it will also fail when visit stage page.
It is better to decode parameter twice here.
Just like fix here SPARK-12708 codes
Does this PR introduce any user-facing change?
No
How was this patch tested?
new added UT

[SPARK-41365][UI][3.3]

It seems that Jenkins is stuck...

retest this please

I added you to the Apache Spark contributor group and assign SPARK-41365 to you.
Welcome to the Apache Spark community, @yabola .

@dongjoon-hyun Thank you very much!

authors: LuciferYang dongjoon-hyun yabola


text:
[SPARK-42244][PYTHON][FOLLOWUP] Fix error messages to keep the consistency

What changes were proposed in this pull request?
This PR follow-ups for https://github.com/apache/spark/pull/39815 to fix error messages that's not meet the consistency with other error messages.
Why are the changes needed?

To keep the consistency across all error messages.
Does this PR introduce any user-facing change?

No, error message changes.
How was this patch tested?

The existing CI should pass.

Merged to master and branch-3.4.

authors: HyukjinKwon itholic


text:
[SPARK-46620][PS][CONNECT] Introduce a basic fallback mechanism for frame methods

What changes were proposed in this pull request?
1, Introduce a basic fallback mechanism for frame methods, with a new option compute.pandas_fallback default false;
2, implement Frame.asfreq and Frame.asof
Why are the changes needed?
for pandas parity
Does this PR introduce any user-facing change?
yes
In [1]: import pyspark.pandas as ps
   ...: import pandas as pd
   ...:
   ...: index = pd.date_range('1/1/2000', periods=4, freq='min')
   ...: series = pd.Series([0.0, None, 2.0, 3.0], index=index)
   ...: pdf = pd.DataFrame({'s': series})
   ...: psdf = ps.from_pandas(pdf)

In [2]: psdf.asfreq(freq='30s')
---------------------------------------------------------------------------
PandasNotImplementedError                 Traceback (most recent call last)
Cell In[2], line 1
----> 1 psdf.asfreq(freq='30s')

File ~/Dev/spark/python/pyspark/pandas/missing/__init__.py:23, in unsupported_function.<locals>.unsupported_function(*args, **kwargs)
     22 def unsupported_function(*args, **kwargs):
---> 23     raise PandasNotImplementedError(
     24         class_name=class_name, method_name=method_name, reason=reason
     25     )

PandasNotImplementedError: The method `pd.DataFrame.asfreq()` is not implemented yet.

In [3]: ps.set_option("compute.pandas_fallback", True)

In [4]: psdf.asfreq(freq='30s')
/Users/ruifeng.zheng/Dev/spark/python/pyspark/pandas/utils.py:1015: PandasAPIOnSparkAdviceWarning: `asfreq` is executed in fallback mode. It loads partial data into the driver's memory to infer the schema, and loads all data into one executor's memory to compute. It should only be used if the pandas DataFrame is expected to be small.
  warnings.warn(message, PandasAPIOnSparkAdviceWarning)
/Users/ruifeng.zheng/Dev/spark/python/pyspark/pandas/utils.py:1015: PandasAPIOnSparkAdviceWarning: If the type hints is not specified for `groupby.apply`, it is expensive to infer the data type internally.
  warnings.warn(message, PandasAPIOnSparkAdviceWarning)
Out[4]:
                       s
2000-01-01 00:00:00  0.0
2000-01-01 00:00:30  NaN
2000-01-01 00:01:00  NaN
2000-01-01 00:01:30  NaN
2000-01-01 00:02:00  2.0
2000-01-01 00:02:30  NaN
2000-01-01 00:03:00  3.0

How was this patch tested?
added ut
Was this patch authored or co-authored using generative AI tooling?
no

cc @HyukjinKwon @itholic @xinrong-meng @ueshin

Nice!
Just for my personal clarity, so now only DataFrame.asfreq and DataFrame.asof can be fallback to Pandas when the option is True, right?

yes, I think we need to check other missing methods by adding UTs.
and some cases can not be simply supported in this way, e.g. io functions like to_hdf, plotting functions, multiple frame operations like compare

thanks for reviews. The last commit only updates the todo item.
merged to master


text:
[SPARK-47873][SQL] Write collated strings to Hive metastore using the regular string type

What changes were proposed in this pull request?

When writing table schema to hive stop writing collated strings as string COLLATE name but instead just write them as regular string type as hive doesn't support collations.
Since we write the original schema as json to table properties in hive so we will able to read the collation back.
Also when reading back the table from the catalog, aside from ignoring case and nullability we should now also ignore any differences in string types as well.
Why are the changes needed?

In order to not break hive compatibility with external engines using hive that would otherwise fail to parse this new type.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Unit tests.
Was this patch authored or co-authored using generative AI tooling?

No.

@cloud-fan all tests passing after scalastyle fix

thanks, merging to master!

authors: cloud-fan stefankandic


text:
[SPARK-48309][YARN]Stop am retry, in situations where some errors and retries may not be successful

… retries may not be successful

What changes were proposed in this pull request?
In yarn cluster mode, spark.yarn.maxAppAttempts will be configured. In our production environment, it is configured as 2 If the first execution fails, AM will retry. However, in some scenarios, even attempting a second task may fail.
For example:
org. apache. park. SQL AnalysisException: Table or view not found: test.testxxxx_xxxxx; Line 1 pos 14;
Project
+-Unresolved Relationship [bigdata_qa, testxxxxx_xxxxx], [], false
Other example:
Caused by: org. apache. hadoop. hdfs. protocol NSQuotaExceededException: The NameSpace quota (directories and files) of directory/tmp/xxx_file/xxxx is exceeded: quota=1000000 file count=1000001
Would it be more appropriate to try capturing these exceptions and stopping retry?
Why are the changes needed?
In some scenarios, even attempting a second task may fail.
Does this PR introduce any user-facing change?
The user can throw a SparkStopAMRetryException, and the Application Master will catch the exception and stop retry
set spark.yarn.maxAppAttempts=2
For examle
`
val spark = SparkSession
  .builder()

  .appName("Spark SQL basic example")

  .enableHiveSupport()

  .getOrCreate()

try {

  spark.sql("select * from test.testxxxx_xxxxx;").show

} catch {

  case e:AnalysisException => throw new SparkStopAMRetryException("this is a test", e)

} finally {

  spark.stop()

}

`
How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

While this PR does not include it, in order to leverage the change introduced for SparkStopAMRetryException - existing exception handling will need to be changed (to throw SparkStopAMRetryException instead of whatever is being thrown now) - which will be a backwardly incompatible change.
I am wondering if we can leverage SparkException.errorClass instead - since SparkException is thrown by Spark ? Return EXIT_STOP_AM_RETRY for some specific subset of error classes ?
+CC @MaxGekk in case this idea makes sense !

I am wondering if we can leverage SparkException.errorClass instead - since SparkException is thrown by Spark ? Return EXIT_STOP_AM_RETRY for some specific subset of error classes ?

Agree, this can be used to handle existing exception, Maybe it's a good idea to include several highest frequencies error classes in your production environment
While SparkStopAMRetryException can be used to handle error scenarios afterwards this pr
+CC @LuciferYang

I am wondering if we can leverage SparkException.errorClass instead - since SparkException is thrown by Spark ? Return EXIT_STOP_AM_RETRY for some specific subset of error classes ?

Agree, this can be used to handle existing exception, Maybe it's a good idea to include several highest frequencies error classes in your production environment While SparkStopAMRetryException can be used to handle error scenarios afterwards this pr +CC @LuciferYang

@mridulm  @summaryzb
Thank you both for helping me review this pr.
In fact, I was initially thinking about whether I could use Spark's existing exception classes to achieve reuse.
But if I don't use new exception information, it may not work for me to do this place. Because in the yarn-cluster mode, the application master determines whether a retry is needed based on the current exception information, such as:
e.getCause match {
case _: InterruptedException =>
case e: SparkUserAppException(exitCode) =>
##     e.getMessage container message ("Table or view not found:") or highest frequencies error message
this code is in ApplicationMaster.
But if the user throws their own defined exception information，such as：
throw new MyTestExecption("this is a test exception, I want to stop am retry.")
In ApplicationMaster, unable to capture user-defined exception information.

In the example code, if it throws SparkUserAppException(18) instead of SparkStopAMRetryException , will it also trigger a retry with this pr?

In the example code, if it throws SparkUserAppException(18) instead of SparkStopAMRetryException , will it also trigger a retry with this pr?

@LuciferYang Lu Thank you for review. I

In the example code, if it throws SparkUserAppException(18) instead of SparkStopAMRetryException , will it also trigger a retry with this pr?

@LuciferYang  Thank you for your review. I made the modifications as you said.

Hi, @guixiaowen I need some time to think about it as it might break some existing workloads.
Meantime, you can

Update the PR desc for better readability
Update the PR desc according to the latest change
Revise the doc of spark.yarn.maxAppAttempts

Hi, @guixiaowen I need some time to think about it as it might break some existing workloads.
Meantime, you can

Update the PR desc for better readability
Update the PR desc according to the latest change
Revise the doc of spark.yarn.maxAppAttempts

@yaooqinn  Ok,  Thank you.

authors: LuciferYang guixiaowen mridulm summaryzb yaooqinn


text:
[SPARK-8903] Fix bug in cherry-pick of SPARK-8803

This fixes a bug introduced in the cherry-pick of #7201 which led to a NullPointerException when cross-tabulating a data set that contains null values.

/cc @brkyvz

Merged build triggered.

Merged build started.

LGTM. Thanks @JoshRosen for your swift response!

Test build #36813 has started for   PR 7295 at commit 5489948.

Test build #36813 has finished for   PR 7295 at commit 5489948.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

[error] /home/jenkins/workspace/SparkPullRequestBuilder/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:342: File line length exceeds 100 characters
[error] (yarn/compile:scalastyle) errors exist
[error] Total time: 9 s, completed Jul 8, 2015 12:03:30 PM
[error] Got a return code of 1 on line 133 of the run-tests script.

Looks like branch-1.4 is failing its Scala style tests.

Yeah I think the cherry-pick broke the style checks. Lets hot-fix that first ?

@JoshRosen @shivaram this is probably my error which I believe I just fixed in branch 1.4 from an unrelated cherry pick. Let me double check

Jenkins, retest this please.

LGTM

Merged build triggered.

Merged build started.

Test build #36824 has started for   PR 7295 at commit 5489948.

Test build #36824 has finished for   PR 7295 at commit 5489948.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Jenkins, retest this please.

Merged build triggered.

Merged build started.

Test build #36828 has started for   PR 7295 at commit 5489948.

Alright, this passed the relevant tests so I'm speculatively merging it so that I can kick off the time-consuming release packaging build.

Test build #36828 has finished for   PR 7295 at commit 5489948.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

close this please

authors: AmplabJenkins JoshRosen SparkQA andrewor14 brkyvz shivaram srowen


text:
[SPARK-8925] [MLlib] Add @since tags to mllib.util

Went thru the history of changes the file MLUtils.scala and picked up the version that the change went in.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@sthota2014 You don't need to tag private or package private method. We only need @since on public methods.

@mengxr Removed the @since from the code for non-public methods.

ok to test

Merged build triggered.

Merged build started.

Test build #37597 has started for   PR 7436 at commit 201e0fd.

Test build #37597 has finished for   PR 7436 at commit 201e0fd.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
   * @deprecated("use method without multiclass argument, which no longer has effect", "1.1.0")
   * @deprecated("use method without multiclass argument, which no longer has effect", "1.1.0")
   * @deprecated("use method without multiclass argument, which no longer has effect", "1.1.0")

Merged build finished. Test PASSed.

@mengxr Any thoughts on this?

Merged build triggered.

Merged build started.

Test build #38273 has started for   PR 7436 at commit 902d49b.

Test build #38273 has finished for   PR 7436 at commit 902d49b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
   * @deprecated("use method without multiclass argument, which no longer has effect", "1.1.0")
   * @deprecated("use method without multiclass argument, which no longer has effect", "1.1.0")

Merged build finished. Test PASSed.

@mengxr Any thoughts on this ?

@mengxr can you please help me complete the cycle.

Merged build triggered.

Merged build started.

Test build #39905 has started for   PR 7436 at commit 865149c.

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Test build #39905 has finished for   PR 7436 at commit 865149c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

@mengxr Any thoughts on this ?

LGTM. Merged into master and branch-1.5. Thanks!

How come the Jenkins run here didn't trigger test failures?

authors: AmplabJenkins SparkQA mengxr rxin sthota2014


text:
[SPARK-3533][Core] Add saveAsTextFileByKey() method to RDDs

This adds the functionality of saving a RDD[(K, V)] to multiple text files split by key. It covers Scala/Java/Python API.
This is based on the Stackoverflow answer linked in the original JIRA and  https://github.com/apache/spark/pull/4895.
Furthermore, I have fixed an issue where part filenames were not included in the paths causing executors to override each other. Tests verify written contents.
I'm very intrigued by @silasdavis's approach to use MultipleOutputs providing a more generic interface to write arbitrary outputs by key: https://gist.github.com/silasdavis/d1d1f1f7ab78249af462
However I have been unsuccessful porting it to work with Hadoop 1 mapred API.
Therefore I'm put forth this PR that achieves the simpler but highly requested goal of writing text files only.

Can one of the admins verify this patch?

Can't wait for that to be released :+1:

Let's not make the best the enemy of the good. I can base the more general approach off this work when I have time.

@silasdavis What's the progress on your side?

@saurfang Do you plan to merge this or you are waiting for @silasdavis's changes?

I myself definitely want to get this merged. That being said, this will require a spark committer to drive the process. If you feel strongly about this feature, voice your support on the JIRA. I can rebase this PR if there are interests in merging.

Test build #54310 has started for   PR 8375 at commit 43ad218.

Test build #54310 has finished for   PR 8375 at commit 43ad218.

This patch fails Python style tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54310/
Test FAILed.

@saurfang This could be done easily with DataFrame APIs:
val df = rdd.map(t => Row(gen_key(t), t)).toDF("key", "text")
df.write.partitionBy("key").text(path)

DataFrame APIs also support many other data type and file formats.
Would you mind close this PR and put this workaround in the JIRA?

Thanks @davies. This looks to be a very reasonable workaround.

authors: AmplabJenkins SparkQA davies n-my saurfang silasdavis utkarsh2012


text:
[SPARK-10824] [ SparkR] show(df) method should show first N number of rows similar to R

We will need this also for the columns.
The actual structure of dataframe can be retrieved by str() function, which Oscar has implemented and will check-in soon.

Can one of the admins verify this patch?

@NarineK Could you close this PR for now ? As we discussed on the JIRA we can get back to this after SPARK-9317 has been resolved.
(Also FYI because of the way our project is setup only the PR creator has permission to close a PR)

Closing this PR for now.

authors: AmplabJenkins NarineK shivaram


text:
[SPARK-9026] Modifications to JobWaiter, FutureAction, and AsyncRDDActions to support non-blocking operation

These changes rework the implementations of SimpleFutureAction, ComplexFutureAction, JobWaiter, and AsyncRDDActions such that asynchronous callbacks on the generated Futures NEVER block waiting for a job to complete. A small amount of mutex synchronization is necessary to protect the internal fields that manage cancellation, but these locks are only held very briefly and in practice should almost never cause any blocking to occur. The existing blocking APIs of these classes are retained, but they simply delegate to the underlying non-blocking API and Await the results with indefinite timeouts.
Associated JIRA ticket: https://issues.apache.org/jira/browse/SPARK-11296
This pull request contains all my own original work, which I release to the Spark project under its open source license.

Can one of the admins verify this patch?

Test build #1952 has started for PR 9264 at commit 1ad1abd.

Test build #1952 has finished for PR 9264 at commit 1ad1abd.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

This seems like it overlaps with #7385, an old PR of mine which was put on hold when I became too busy with other tickets (and when the original motivation for it was worked around via another means). This would still be good to fix, though, and I'm happy to work with you to try to do this.

@JoshRosen Agreed that this is largely a duplicate.  One of SPARK-9026 and SPARK-11296 should be marked as a dup and closed.  And fwiw, at least the test failures on the jobFailed race that you were dealing with in that PR have since been dealt with in a separate PR, recently merged.

I agree that the JIRA issue is a duplicate. Odd that the old one didn't turn up for it when I searched for existing issues. I've linked the issues in JIRA.

It looks like our approaches are very similar, though my changes to JobWaiter and SimpleFutureAction are a bit smaller, and I also included fixes for ComplexFutureAction and AsyncRDDFunctions.takeAsync. I haven't been able to fully test mine yet, though, because doing a full build of Spark causes my junky old laptop to go into seizures.

I've added some unit tests that verify that my changes, do, indeed, stop callbacks from consuming threads before the jobs have completed.

What else needs to be done to get this PR merged?

Hi @reggert , I'm not the best person to review this, so I can't definitely say what needs to be done to get it merged.  I pointed out a few minor style issues just to keep things moving along.  You also need to resolve conflicts with master.

Jenkins, ok to test

Build triggered.

Build started.

Test build #45395 has started for PR 9264 at commit 93b3065.

Test build #45395 has finished for PR 9264 at commit 93b3065.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

@reggert I've closed your issue as a duplicate. Can you remove it from the title and use only SPARK-9026 to avoid potential confusion?

@squito I merged with master 2 days ago, and more conflicts were introduced on master less than 24 hours later. I'll merge one more time in order to fix the issue with the accidentally-deleted Utils class, but I won't do it again after that until it's the only thing left to do.

Merged build triggered.

Merged build started.

Test build #45491 has started for PR 9264 at commit dcd2883.

Merged build finished. Test FAILed.

Test build #45491 has finished for PR 9264 at commit dcd2883.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45491/
Test FAILed.

Back up to date with master. All style issues should be resolved.

Merged build triggered.

Merged build started.

Test build #45494 has started for PR 9264 at commit 31de51b.

Test build #45494 has finished for PR 9264 at commit 31de51b.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45494/
Test FAILed.

"[info] spark-mllib: found 0 potential binary incompatibilities (filtered 64)
java.lang.RuntimeException: spark-core: Binary compatibility check failed!" .... uh.... wat?!

yeah its confusing to go through the mima msgs -- the one log line you quoted is for spark-mllib.  the issues are in spark-core:
[error]  * synthetic method org$apache$spark$ComplexFutureAction$$thread_=(java.lang.Thread)Unit in class org.apache.spark.ComplexFutureAction does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.ComplexFutureAction.org$apache$spark$ComplexFutureAction$$thread_=")
[error]  * method runJob(org.apache.spark.rdd.RDD,scala.Function1,scala.collection.Seq,scala.Function2,scala.Function0)Unit in class org.apache.spark.ComplexFutureAction does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.ComplexFutureAction.runJob")

Why do we care about binary compatibility for an internal API that is only used in one place?
In any case, the implicit ExecutionContext being passed to ComplexFutureAction.runJob apparently wasn't being used. I don't really know what to do about the removed Thread field, since it's not needed and it's not a public API. SimpleFutureAction and ComplexFutureAction really ought to be marked private[spark] anyway.

Merged build triggered.

Merged build started.

Test build #45584 has started for PR 9264 at commit 0a6c614.

Test build #45584 has finished for PR 9264 at commit 0a6c614.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45584/
Test FAILed.

I've added the @DeveloperApi annotation to SimpleFutureAction and ComplexFutureAction, which should resolve the MiMa errors.

Merged build triggered.

Merged build started.

Test build #45587 has started for PR 9264 at commit 2c2da36.

Test build #45587 has finished for PR 9264 at commit 2c2da36.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45587/
Test FAILed.

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45654/
Test FAILed.

Merged build triggered.

Merged build started.

Test build #45660 has started for PR 9264 at commit 0dd1128.

Merged build triggered.

Merged build started.

It seems like Jenkins is not in a happy place:
 > git fetch --tags --progress https://github.com/apache/spark.git +refs/pull/9264/*:refs/remotes/origin/pr/9264/* # timeout=15
ERROR: Timeout after 15 minutes
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/apache/spark.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:763)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1012)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1043)
	at hudson.scm.SCM.checkout(SCM.java:485)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1275)
	at

Test build #45663 has started for PR 9264 at commit 50cd13e.

Merged build triggered.

Merged build started.

Test build #45665 has started for PR 9264 at commit 7f4244d.

@reggert I took a pass and it looks like this simplifies the code a bit by pushing some logic into scala itself. However, there are a lot of style inconsistencies and it's distracting for the reviewers. Can you fix them according to https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide and the suggestions I left?

@andrewor14  Thanks. I've fixed the issues you pointed out and anything else I could find. See my question above regarding resultFunc, though.

Merged build triggered.

Merged build started.

Test build #45670 has started for PR 9264 at commit 19ef962.

Test build #45670 has finished for PR 9264 at commit 19ef962.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45670/
Test FAILed.

Merged build finished. Test FAILed.

Test build #45660 has finished for PR 9264 at commit 0dd1128.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45660/
Test FAILed.

Test build #45663 has finished for PR 9264 at commit 50cd13e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45663/
Test FAILed.

Merged build finished. Test FAILed.

Test build #45665 has finished for PR 9264 at commit 7f4244d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45665/
Test FAILed.

Merged build triggered.

Merged build started.

Test build #45675 has started for PR 9264 at commit d1a905d.

The Jenkins build has been failing due to ForkErrors being thrown by random tests that have nothing to do with anything I changed.  The error itself has no message attached,  so I have no idea what its problem is.

Test build #45675 has finished for PR 9264 at commit d1a905d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45675/
Test PASSed.

Well, the tests that randomly fail passed this time. Yay?

Merged build triggered.

Merged build started.

Test build #45695 has started for PR 9264 at commit c63ac17.

Test build #45695 has finished for PR 9264 at commit c63ac17.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45695/
Test FAILed.

Merged build finished. Test FAILed.

Build failed again, in a test that has nothing to do with this PR. Those tests seem extremely flaky.

Is there anything left to be done before merging this?

@srowen @JoshRosen can you have a look?

Test build #45938 has started for PR 9264 at commit 489aabc.

Test build #45939 has started for PR 9264 at commit c19b3c0.

Test build #45938 has finished for PR 9264 at commit 489aabc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45938/
Test PASSed.

Test build #45939 has finished for PR 9264 at commit c19b3c0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45939/
Test FAILed.

It sure would be nice if Spark's unit tests were consistent. I add whitespace and suddenly the build fails?

Unrelated to this PR (except that it may be partially responsible for the most recent test failure), there appears to be a race condition here:
if (!executionContext.isShutdown) {
      val f = Future { deleteFiles() }

(from FileBasedWriteAheadLog.clean in spark-streaming)
If the ExecutionContext shuts down after isShutdown is called but before the task underlying the Future is enqueued on it, an exception will be thrown, which appears to be what's happening in CommonWriteAheadLogTests.logCleanUpTest. I'm not certain why the ExecutionContext would be getting shut down, but it does seem like the test that's failing has an awfully short timeout, and according to the stack trace, the thread pool is very busy at the time of the failure.

@reggert yeah, with thousands of tests, some of them integration-style tests, flakiness is a pretty regular occurrence. You can see a number of PRs to improve indivudal ones. We can just retest. Don't worry about fixing tests here.

Test build #2060 has started for PR 9264 at commit c19b3c0.

Test build #2060 has finished for PR 9264 at commit c19b3c0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

@srowen Okay. I was just worried I was going to get blamed for breaking something. ;-)
What's left to do here?

I don't know the logic well enough to asses the change, but the style looks OK and the tests pass, and it does appear to be a simplification, which is always good.

I'm not particularly happy about the (internal) API exposed by ComplexFutureAction. Having callers instantiate the action and then mutate it by calling the run and submitJob methods just seems sloppy and error-prone.
I think I would prefer that instead of having a run method that takes a closure returning Future[T], ComplexFutureAction should accept a constructor parameter of type JobSubmitter => Future[T], where JobSubmitter would be a (Spark-private) trait providing the submitJob method. This would make ComplexFutureAction more or less immutable after construction (except for cancellation) and prevent someone from calling run or submitJob from outside Spark and making a mess of things. This is a fairly major changes introducing a new trait, so I will hold off implementing it until I get some positive feedback about it.
Additionally, it seems like certain common aspects of SimpleFutureAction and ComplexFutureAction  (such as the _cancellation field and the base implementation of cancel) could be pulled out into an implementation trait (i.e., FutureActionLike) to avoid duplicating code.
Thoughts?

I took a stab at implementing the above refactoring, and it came out looking pretty nice, so I went ahead and committed it. Please take a look and let me know if you have any problems with it.

Test build #46048 has started for PR 9264 at commit 601bb95.

Test build #46049 has started for PR 9264 at commit 9da3533.

Test build #46049 has finished for PR 9264 at commit 9da3533.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * trait JobSubmitter \n  * class ComplexFutureAction[T](run : JobSubmitter => Future[T])\n  * class BitSet(numBits: Int) extends Serializable \n  * class Pipeline(override val uid: String) extends Estimator[PipelineModel] with Writable \n  * class StreamingListener(object):\n  * abstract class Aggregator[-A, B, C] extends Serializable \n

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46049/
Test FAILed.

Test build #46048 has finished for PR 9264 at commit 601bb95.

This patch fails PySpark unit tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):\n  * trait JobSubmitter \n  * class ComplexFutureAction[T](run : JobSubmitter => Future[T])\n

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46048/
Test FAILed.

I most certainly did not add all those classes (other than JobSubmitter)! @SparkQA is confused.

@reggert yeah the "experimental" feature generally throws up lots of false positives, but they're easy to dismiss if you know you didn't add a certain class or method. The PySpark test failure is again probably spurious, so just retest.

Test build #2075 has started for PR 9264 at commit 9da3533.

Any thoughts on the latest changes (besides the merge)?

Can someone re-run the build?

I've come up with a reusable way to make use of semaphores to control timing of tasks during unit tests. Please see the Smuggle class and let me know what you think.

Test build #46483 has started for PR 9264 at commit a8ba899.

Test build #46483 has finished for PR 9264 at commit a8ba899.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * trait JobSubmitter \n  * class ComplexFutureAction[T](run : JobSubmitter => Future[T])\n

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46483/
Test PASSed.

Merged build finished. Test PASSed.

SparkQA seems to be happy now.
@JoshRosen What else needs to be done to move forward on this? The current implementation on master requires a dedicated thread pool for all callbacks on FutureActions to avoid depleting the global ExecutionContext (and this fact is entirely undocumented), so it would be good to have these changes in 1.6.

It appears that this PR may also fix SPARK-4514.

Test build #46498 has started for PR 9264 at commit 5816489.

Correction: This PR only fixes SPARK-4514 as long as the {{ComplexFutureAction}} only spawns a single job. The job group information is lost in subsequent jobs.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46497/
Test FAILed.

Merged build finished. Test FAILed.

I've implemented a two-line fix for SPARK-4514.

Test build #46499 has started for PR 9264 at commit 8fe8000.

Test build #46498 has finished for PR 9264 at commit 5816489.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * trait JobSubmitter \n  * class ComplexFutureAction[T](run : JobSubmitter => Future[T])\n

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46498/
Test PASSed.

Merged build finished. Test PASSed.

Test build #46499 has finished for PR 9264 at commit 8fe8000.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * trait JobSubmitter \n  * class ComplexFutureAction[T](run : JobSubmitter => Future[T])\n

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46499/
Test PASSed.

@zsxwing, could you take a look at this patch? I think you'd be a good reviewer for this given your experience with similar patches in streaming.

Any feedback?

@zsxwing @JoshRosen I just want to make sure that you guys haven't forgotten about this. I haven't heard anything in a week and a half.

All I want for Christmas is a code review. :-)

@reggert sorry for the delay. LGTM except a nit for test. Thanks very much.

Test build #47629 has started for PR 9264 at commit 539ac43.

Test build #47629 has finished for PR 9264 at commit 539ac43.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * trait JobSubmitter \n  * class ComplexFutureAction[T](run : JobSubmitter => Future[T])\n

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47629/
Test PASSed.

Merged build finished. Test PASSed.

LGTM

Thanks, merging into master.

By the way, @reggert how is this related to SPARK-4514?

@andrewor14 I included a couple lines to propagate the local properties (from the thread that created the ComplexFutureAction) to each of the spawned jobs (all but the first of which may be launched from other threads, losing the local properties unless setLocalProperties is called). AFAIK this resolves SPARK-4514.

authors: AmplabJenkins JoshRosen SparkQA andrewor14 markhamstra reggert squito srowen zsxwing


text:
[SPARK-11973] [SQL] push filter through aggregation with alias and literals

Currently, filter can't be pushed through aggregation with alias or literals, this patch fix that.
After this patch, the time of TPC-DS query 4 go down to 34 seconds from 141 seconds (4x improvements).
cc @nongli  @yhuai

Test build #46668 has started for PR 9959 at commit 2fb7a1c.

cc @cloud-fan  too

Test build #46668 has finished for PR 9959 at commit 2fb7a1c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46668/
Test FAILed.

Test build #46699 has started for PR 9959 at commit 0f5d7ba.

Test build #46701 has started for PR 9959 at commit 951fe7a.

Test build #46701 has finished for PR 9959 at commit 951fe7a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46701/
Test PASSed.

Test build #46699 has finished for PR 9959 at commit 0f5d7ba.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46699/
Test FAILed.

Test build #2116 has started for PR 9959 at commit 951fe7a.

Test build #46749 has started for PR 9959 at commit 37b3088.

Test build #2116 has finished for PR 9959 at commit 951fe7a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

LGTM

Test build #46749 has finished for PR 9959 at commit 37b3088.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46749/
Test PASSed.

Merging into master and 1.6 branch.

authors: AmplabJenkins SparkQA cloud-fan davies rxin


text:
Update storm-kafka-client.md

fix a word fault

+1

authors: cutd hmcl

repo: apache/storm


text:
fix HBaseSecurityUtil    double_checked,

update HBaseSecurityUtil singleton to fix double_checked,  http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html

thanks for review

Nice find.
Before we can merge you should create an issue for this at https://issues.apache.org/jira and then rename the PR and commit message to contain the issue number.


text:
STORM-3437: Automate license checks more

Based on https://github.com/apache/storm/pull/3051.
The idea here is to add license checks to CI, so PRs fail unless they keep the LICENSE-binary and DEPENDENCY-LICENSES files up to date.

Thanks for reviewing.


text:
Recently viewed Dashboards do not appear in the Recents section

How to reproduce the bug
With administrator privileges, I go sequentially to several dashboards
At the same time, 2 specific dashboards are constantly displayed in the Recents section and new dashboards that I just went to do not appear
Screenshots

the initial state

2.sequentially open several dashboards

In the Recents section, nothing changes
Environment
SuperSet 1.4.1
Python 3.8.13

node.js version: v16.14.2
Google Chrome 100.0.4896.127

This is likely fixed by now, and is pretty out of date if not. If people are still encountering this in current versions (3.x) please open a new Issue or a PR to address the problem.

source: gharchive/issue

authors: VosmirkoSergey rusackas

repo: apache/superset


text:
ROW_LIMIT not working

Bug description
Hello there. my project is last year september october version but i have row limit not working how to fix it. I don't know how to fix it. Help me pleace

How to reproduce the bug
1
Screenshots/recordings
@michael-s-molina @betodealmeida @rusackas @jon
Superset version
master / latest-dev
Python version
3.9
Node version
16
Browser
Chrome
Additional context
Hi
Checklist

[ ] I have searched Superset docs and Slack and didn't find a solution to my problem.
[ ] I have searched the GitHub issue tracker and didn't find a similar bug report.
[ ] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

@michael-s-molina @betodealmeida @rusackas @jon

“ROW_LIMIT” sozlamasining to‘g‘ri qo‘llanilmasligi bilan bog‘liq muammo bir necha omillarga, jumladan, Superset kod bazasidagi so‘nggi o‘zgarishlar, backend konfiguratsiyalari yoki frontend boshqaruv mantig‘iga bog‘liq bo‘lishi mumkin. Ushbu muammoni hal qilishda sizga yordam beradigan ba'zi qadamlar va fikrlar:
Backend konfiguratsiyalari
Backend konfiguratsiyasi config.pysizning “ROW_LIMIT” sozlamangizni bekor qilmasligiga ishonch hosil qiling. Tekshirish uchun asosiy sozlamalar:

SQL_MAX_ROW : Har qanday analitik ma'lumotlar bazasi so'rovi uchun qaytarilgan satrlarning maksimal sonini belgilaydi.
DISPLAY_MAX_ROW : SQL Lab UI da ko'rsatiladigan qatorlar sonini cheklaydi.
DEFAULT_SQLLAB_LIMIT : SQL Lab soʻrovlari uchun standart qator chegarasi.
SUPERSET_META_DB_LIMIT : Xususiyat bayrog'i ENABLE_SUPERSET_META_DByoqilganda cheklash.
SQLLAB_CTAS_NO_LIMIT : SQL Lab-ga CTAS funksiyasidan foydalanganda qator cheklovisiz so'rovlarni bajarishga ruxsat beradi.

Frontend boshqaruv mantiqi
Siz oʻrnatgan qiymat (20.000.000) ruxsat etilgan parametrlar ichida ekanligiga yoki erkin shakldagi kiritish sifatida toʻgʻri tasdiqlanganligiga ishonch hosil qiling. Tekshirish row_limitmaxsus variantlar va tekshirish mantig'i bilan belgilanadi.
Superset kod bazasidagi so'nggi o'zgarishlar
"ROW_LIMIT" sozlamasiga ta'sir qilishi mumkin bo'lgan Superset kod bazasida yaqinda o'zgarishlar bo'ldi. Ta'kidlash joizki, №27700 va №26151 so'rovlari qator chegaralari bilan bog'liq muammolarni hal qiladi.
Jurnal va disk raskadrovka
Supersetda “ROW_LIMIT” sozlamasi qanday qoʻllanilishi yoki soʻrovni bajarish vaqtida eʼtiborga olinmasligini kuzatishga yordam beradigan jurnalga yozish mexanizmlari mavjud. Kalit ro'yxatga olish bayonotlari faylda mavjud superset/commands/sql_lab/execute.py.
row_limitNazoratning ko'rinishi
Boshqaruv yoqilmaganligiga ishonch hosil qiling server_pagination, chunki bu "ROW_LIMIT" sozlamasining e'tiborsiz qolishiga olib kelishi mumkin.
Muammo davom etsa, "ROW_LIMIT" sozlamasining ishlashiga ta'sir qilishi mumkin bo'lgan Superset kod bazasidagi so'nggi o'zgarishlarni ko'rib chiqing.
Bu masala chuqurroq ko'rib chiqishni talab qiladi shekilli,@rusackasIltimos, o'z tajribangizni taqdim eta olasizmi?
Tafsilotlar
Sources

Limit issue
superset/migrations/versions/2019-06-04_10-12_d7c1a0d6f2da_remove_limit_used_from_query_model.py
superset-frontend/plugins/plugin-chart-handlebars/src/plugin/controls/limits.ts
superset-frontend/src/explore/controls.jsx
superset/config.py
docker/pythonpath_dev/superset_config.py
superset/sql_lab.py
superset/commands/sql_lab/execute.py
CHANGELOG/3.1.0.md
CHANGELOG/4.0.1.md
superset-frontend/packages/superset-ui-chart-controls/src/shared-controls/sharedControls.tsx
superset-frontend/src/SqlLab/components/ResultSet/index.tsx

About Dosu This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
Suhbatni davom ettirish uchun zikr qiling@dosu.

I have run my superset project version for the first time since September October last year, I tried the above solutions and it didn't work

“ROW_LIMIT” sozlamasining to‘g‘ri qo‘llanilmasligi bilan bog‘liq muammo bir necha omillarga, jumladan, Superset kod bazasidagi so‘nggi o‘zgarishlar, backend konfiguratsiyalari yoki frontend boshqaruv mantig‘iga bog‘liq bo‘lishi mumkin. Ushbu muammoni hal qilishda sizga yordam beradigan ba'zi qadamlar va fikrlar:
Backend konfiguratsiyalari
Backend konfiguratsiyasi config.pysizning “ROW_LIMIT” sozlamangizni bekor qilmasligiga ishonch hosil qiling. Tekshirish uchun asosiy sozlamalar:

SQL_MAX_ROW : Har qanday analitik ma'lumotlar bazasi so'rovi uchun qaytarilgan satrlarning maksimal sonini belgilaydi.
DISPLAY_MAX_ROW : SQL Lab UI da ko'rsatiladigan qatorlar sonini cheklaydi.
DEFAULT_SQLLAB_LIMIT : SQL Lab soʻrovlari uchun standart qator chegarasi.
SUPERSET_META_DB_LIMIT : Xususiyat bayrog'i ENABLE_SUPERSET_META_DByoqilganda cheklash.
SQLLAB_CTAS_NO_LIMIT : SQL Lab-ga CTAS funksiyasidan foydalanganda qator cheklovisiz so'rovlarni bajarishga ruxsat beradi.

Frontend boshqaruv mantiqi
Siz oʻrnatgan qiymat (20.000.000) ruxsat etilgan parametrlar ichida ekanligiga yoki erkin shakldagi kiritish sifatida toʻgʻri tasdiqlanganligiga ishonch hosil qiling. Tekshirish row_limitmaxsus variantlar va tekshirish mantig'i bilan belgilanadi.
Superset kod bazasidagi so'nggi o'zgarishlar
"ROW_LIMIT" sozlamasiga ta'sir qilishi mumkin bo'lgan Superset kod bazasida yaqinda o'zgarishlar bo'ldi. Ta'kidlash joizki, №27700 va №26151 so'rovlari qator chegaralari bilan bog'liq muammolarni hal qiladi.
Jurnal va disk raskadrovka
Supersetda “ROW_LIMIT” sozlamasi qanday qoʻllanilishi yoki soʻrovni bajarish vaqtida eʼtiborga olinmasligini kuzatishga yordam beradigan jurnalga yozish mexanizmlari mavjud. Kalit ro'yxatga olish bayonotlari faylda mavjud superset/commands/sql_lab/execute.py.
row_limitNazoratning ko'rinishi
Boshqaruv yoqilmaganligiga ishonch hosil qiling server_pagination, chunki bu "ROW_LIMIT" sozlamasining e'tiborsiz qolishiga olib kelishi mumkin.
Muammo davom etsa, "ROW_LIMIT" sozlamasining ishlashiga ta'sir qilishi mumkin bo'lgan Superset kod bazasidagi so'nggi o'zgarishlarni ko'rib chiqing.
Bu masala chuqurroq ko'rib chiqishni talab qiladi shekilli,@rusackasIltimos, o'z tajribangizni taqdim eta olasizmi?
Tafsilotlar
Sources

Limit issue
superset/migrations/versions/2019-06-04_10-12_d7c1a0d6f2da_remove_limit_used_from_query_model.py
superset-frontend/plugins/plugin-chart-handlebars/src/plugin/controls/limits.ts
superset-frontend/src/explore/controls.jsx
superset/config.py
docker/pythonpath_dev/superset_config.py
superset/sql_lab.py
superset/commands/sql_lab/execute.py
CHANGELOG/3.1.0.md
CHANGELOG/4.0.1.md
superset-frontend/packages/superset-ui-chart-controls/src/shared-controls/sharedControls.tsx
superset-frontend/src/SqlLab/components/ResultSet/index.tsx

About Dosu This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
Suhbatni davom ettirish uchun zikr qiling@dosu.

I have run my superset project version for the first time since September October last year, I tried the above solutions and it didn't work

@dosu._

I have run my superset project version for the first time since September October last year, I tried the above solutions and it didn't work
@dosu._

@mistercrunch

@rusackas @michael-s-molina

Hello, you mentionned a september / october version. I recalled we faced the same issue around the same period, it was solved for us since 3.1.3.

Ho @mujohiddin. This bug was fixed by https://github.com/apache/superset/pull/26807 which is available in 3.1.3 and 4.0.1 so you just need to use one of these versions. You can always check in which version a fix is available using the CHANGELOG.
FYI: It's not a good practice to tag multiple people multiple times. Keep in mind that we receive many messages every day and might be busy resolving other issues.

authors: gaful michael-s-molina mujohiddin


text:
fix(window): unavailable localStorage and sessionStorage

SUMMARY
If you tweak your web browser settings a little bit and embed Superset into an iframe, you could encounter an issue with unavailable local storage and/or session storage.
Using the web browser API must be surrounded by try-catch blocks.
More info in this great article

https://blog.tomayac.com/2022/08/30/things-not-available-when-someone-blocks-all-cookies/

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

Go on Edge, for example
Block 3rd-party cookies

And try to visit an embedded Superset
💥

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

@eschutho any chance to trigger cy tests again from your side?
I think it's not matching the branch protection rules?

Oops, sorry @justinpark / @john-bodley - I got a little trigger-happy there. Please do review, and let us know if this needs course correction.

LGTM

it's all good.. thanks @rusackas!

authors: eschutho frassinier justinpark rusackas