text:
Add finagle plugin

Please answer these questions before submitting pull request

Why submit this pull request?

[ ] Bug fix

[x] New feature provided

[ ] Improve performance

Related issues
https://github.com/apache/skywalking/issues/4433

Bug fix

Bug description.

How to fix?

New feature or improvement

Add finagle plugin, version from 6.25.0 to 20.1.0

Please follow these four first

Make codes well formatted, including comments of the codes
Add unit tests to your plugin codes.
Make CI and plugin integration test work
supported doc update.

OK!  I will follow these steps to improve.

Please follow these four first

Make codes well formatted, including comments of the codes
Add unit tests to your plugin codes.
Make CI and plugin integration test work
supported doc update.

Additional:

add it to component-libraries.yaml
add ICON to SkyWalking-UI repo

add ICON to SkyWalking-UI repo
Hi, I think i need some help.  After checking the skywalking-ui directory,  I could't find where the icon should be added?  Also are there any requirements for icons, such as size.

You need to add your test into the CI pass by changing this file,  https://github.com/apache/skywalking/blob/master/.github/workflows/plugins-test.yaml . Please use the fast group.

After check the git history of the file, I find the last merged plugin is avro, does the fast group is Spring31x_ES5_Gateway_Avro_Struts25?  Please tolerate me with this kind of questions, I did not use github actions before.

After check the git history of the file, I find the last merged plugin is avro, does the fast group is Spring31x_ES5_Gateway_Avro_Struts25? Please tolerate me with this kind of questions, I did not use github actions before.

Open the actions tab, https://github.com/apache/skywalking/pull/4441/checks. Open the PluginsTest tab on the left side, then you could check every sub group execution time. Or check the list above

That is easy.

Open the actions tab, https://github.com/apache/skywalking/pull/4441/checks. Open the PluginsTest tab on the left side, then you could check every sub group execution time. Or check the list above

I see. thanks!

The comments of your codes are required. All the review are based on my guess, please don't let me do that.
Comments are important for other contributors and reviewers.

You are right, sorry for that. I will add comments this weekend.

Codecov Report

Merging #4441 into master will increase coverage by 0.39%.
The diff coverage is 62.11%.

@@            Coverage Diff             @@
##           master    #4441      +/-   ##
==========================================
+ Coverage   25.14%   25.53%   +0.39%
==========================================
  Files        1223     1235      +12
  Lines       28293    28615     +322
  Branches     3890     3926      +36
==========================================
+ Hits         7115     7308     +193
- Misses      20521    20631     +110
- Partials      657      676      +19

Impacted Files
Coverage Δ

.../define/ClientDestTracingFilterInstrumetation.java
0% <0%> (ø)

.../apm/network/trace/component/ComponentsDefine.java
0% <0%> (ø)
:arrow_up:

...plugin/finagle/ClientTracingFilterInterceptor.java
100% <100%> (ø)

...lking/apm/plugin/finagle/ContextHolderFactory.java
34.82% <34.82%> (ø)

...alking/apm/plugin/finagle/AbstractInterceptor.java
43.75% <43.75%> (ø)

...in/finagle/ClientDestTracingFilterInterceptor.java
52.38% <52.38%> (ø)

...ywalking/apm/plugin/finagle/SWContextCarrier.scala
66.66% <66.66%> (ø)

...lking/apm/plugin/finagle/ContextCarrierHelper.java
72.72% <72.72%> (ø)

...lking/apm/plugin/finagle/CompatibilityChecker.java
73.33% <73.33%> (ø)

...che/skywalking/apm/plugin/finagle/FinagleCtxs.java
76.47% <76.47%> (ø)

... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0df2d0a...4ed9a08. Read the comment docs.

The comments have been added.  The implementation of this plugin is described in the AbstractAbstractceptor class, and other classes have added their own comments.  I hope these comments can be described clearly, if there is any ambiguity, please point out, i will modify again.

Have you followed no threadlocal and using the witness class? That should be followed.

Have you followed no threadlocal and using the witness class? That should be followed.

https://github.com/apache/skywalking/pull/4441#discussion_r389245836

I think you update the query-protocol submodule by a mistake.

This has been corrected.

As #4458 merged, the expected YAML file has fields been renamed. Take a look the PR and updated plugin test doc.

As #4458 merged, the expected YAML file has fields been renamed. Take a look the PR and updated plugin test doc.

OK.

I noticed, this PR fails every time on AppVeyor. Strange.

I noticed, this PR fails every time on AppVeyor. Strange.

I will check this issue.

Most concerns have been eased. Just CodecUtils and ContextHolderFactory may leak comments.
To be honest, I can't get too much details of this plugin, as I am not familiar with the Finagle :). All these requirements are making another contributor or further bug fix one could understand your logic. Feel free to add a .md in your plugin rood folder to do better explanation.

I will take your suggesiton, and to summarize what i will do next:

Fix ContextHolderFactory
For the current implentation of ContextHolderFactory,   if server calls another rpc, for example  A ->B(serverB) ->C(serverC),  there will have context pollution in serverB. I will test this and fix.
Fix the failure of appveyor
Looks likedto it caused by scala codes.  First, I will try to replace these codes with java, if not, I will try to find the compatible version for related components.
Improve comments.
Add a implementation.md in root folder

Hope to finish above in next week!

Thanks for the update. If all these provided in next week, we could add this as a part of 7.0.0 release.

Looks like it caused by scala codes. First, I will try to replace these codes with java, if not, I will try to find the compatible version for related components.

appveyor.yml is controlling the CI process in the appveyor. We used to use appveyor for Windows compiling, but this has been rechecked in GitHub action too, here is the control file, https://github.com/apache/skywalking/blob/master/.github/workflows/ci-it.yaml#L53-L72.
I don't know what is the difference.

@aderm @arugal @dmsolr @kezhenxu94 Any of you could recheck?

Merging. Thank you for your contributions.

I'm doing additional tests for scenario:  A -> ServerB -> ServerC and meet above exception.  I haven't had time to find out why, it better to revert this pr first.

@huangyoje I think this is my fault. I was not expecting the tracing context could be finished in the main thread when we do lazy injection. I will submit a fix PR soon.

@wu-sheng OK.

@huangyoje Ping you at the PR.

Merging. Thank you for your contributions.

🍺🍺

authors: codecov-io dmsolr huangyoje wu-sheng


text:
SOLR-16158: Remove server jars from solr-core

https://issues.apache.org/jira/browse/SOLR-16158

Move SolrJettyRunner to test framework. Solr-core now no longer has jetty server deps.
This is a slight back-compat concern, but I'm not sure the SolrJettyRunner is used widely enough for it to warrant keeping this from 9x
server/lib/ext now contains the following jars:

logging
metrics
Shared server & core dependencies

The web-app is now setup so that "server" jars can be used by the Solr webapp. We no longer have to duplicate these jetty "server" jars between server/lib and WEB-INF/lib.
This is also a slight back-compat concern, but we are already changing our jetty xml files for the jetty 10 upgrade, so this change is likely ok too.
The solr-core & solr-server dependencies now just work, and shouldn't require special setup.

Most of the files are just changing the location of SolrJettyRunner. If you want to see the actual changes here, then just look at the */build.gradle and solr/server/contexts/solr-jetty-context.xml files.

Note: the jetty-server jar is in lib/ext even though Solr core/SolrJ should not need it. We can hunt that down in a separate issue though if need be.

@dsmiley @risdenk can I get a quick check on my upgrade notes? https://github.com/apache/solr/pull/1179/commits/1fea79bc61f61a6fde5e0f4fb06ed9e91aac4aac

The CHANGES.txt text speaks of duplicated JARs but does not mention any.  If this was actually a problem in 9.1 (I forget?) then okay -- could you list at least one there?  If that was not a released problem (e.g. happened after 9.1) then don't mention it; keep it in the commit notes, though.

Ok listed 2, and expanded a bit in both sections.

source: gharchive/pull-request

authors: HoustonPutman dsmiley

repo: apache/solr

license_type: permissive

license_source: bigquery


text:
[SPARK-13096] [TEST] Fix flaky verifyPeakExecutionMemorySet

Previously we would assert things before all events are guaranteed to have been processed. To fix this, just block until all events are actually processed, i.e. until the listener queue is empty.
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7/79/testReport/junit/org.apache.spark.util.collection/ExternalAppendOnlyMapSuite/spilling/

@zsxwing

LGTM

Test build #50414 has started for PR 10990 at commit d67a529.

Test build #50414 has finished for PR 10990 at commit d67a529.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50414/
Test PASSed.

Merged into master.

authors: AmplabJenkins SparkQA andrewor14 zsxwing

repo: apache/spark

license_source: github-api


text:
[SPARK-13601] [TESTS] use 1 partition in tests to avoid race conditions

What changes were proposed in this pull request?
Fix race conditions when cleanup files.
How was this patch tested?
Existing tests.

cc @zsxwing

Test build #52429 has started for PR 11507 at commit 499bb8f.

LGTM. Tested 20 times locally and all passed.

Thanks, I'm going to merge this to hotfix master (since we only touch the flaky tests here).

Test build #52429 has finished for PR 11507 at commit 499bb8f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52429/
Test PASSed.

Merged build finished. Test PASSed.

authors: AmplabJenkins SparkQA davies zsxwing


text:
[SPARK-16621][SQL][WIP] Use a stable ID generation method for attributes in SQLBuilder

What changes were proposed in this pull request?
Currently, the generated SQL has stable IDs for subquery, but not-stable IDs for gen_attr. This issue provides stable IDs for gen_attr, too.
The stable generated SQL will give more benefit for understanding or testing the queries.
Before
scala> new org.apache.spark.sql.catalyst.SQLBuilder(sql("select 1")).toSQL
res0: String = SELECT `gen_attr_0` AS `1` FROM (SELECT 1 AS `gen_attr_0`) AS gen_subquery_0
scala> new org.apache.spark.sql.catalyst.SQLBuilder(sql("select 1")).toSQL
res1: String = SELECT `gen_attr_4` AS `1` FROM (SELECT 1 AS `gen_attr_4`) AS gen_subquery_0

After
scala> new org.apache.spark.sql.catalyst.SQLBuilder(sql("select 1")).toSQL
res1: String = SELECT `gen_attr_0` AS `1` FROM (SELECT 1 AS `gen_attr_0`) AS gen_subquery_0
scala> new org.apache.spark.sql.catalyst.SQLBuilder(sql("select 1")).toSQL
res2: String = SELECT `gen_attr_0` AS `1` FROM (SELECT 1 AS `gen_attr_0`) AS gen_subquery_0

TODO:

[x] Make gen_subquery_xxx unique
[x] Provide stable gen_attr_xxx
[x] Provide internal pretty SQL formatter for generated SQLs
[ ] Pass the HiveCompatibilitySuite.

How was this patch tested?
Pass the existing Jenkins tests.

Test build #62517 has started for PR 14257 at commit 2699f9e.

Test build #62517 has finished for PR 14257 at commit 2699f9e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62517/
Test FAILed.

Test build #62761 has started for PR 14257 at commit 12180db.

Test build #62761 has finished for PR 14257 at commit 12180db.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62761/
Test FAILed.

Test build #62772 has started for PR 14257 at commit a1776ef.

Test build #62772 has finished for PR 14257 at commit a1776ef.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62772/
Test FAILed.

Merged build finished. Test FAILed.

Test build #62796 has started for PR 14257 at commit 06fb99c.

Test build #62796 has finished for PR 14257 at commit 06fb99c.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class SQLBuilder(

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62796/
Test PASSed.

Merged build finished. Test PASSed.

Hi, @rxin .
Could you review this stable SQL generation PR?

I'm less sure about the new format -- it leads to tons of whitespaces for very long SQL queries.

No problem. We can simply remove object SQLBuilder and modify two lines.
May I remove those?

It's just for helping review processes.

I removed the new format stuff. I'll update this PR, soon.

Now, the PR is updated with new golden SQL files.

Test build #62829 has started for PR 14257 at commit 6b3adbf.

cc @liancheng

Test build #62829 has finished for PR 14257 at commit 6b3adbf.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class SQLBuilder(

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62829/
Test PASSed.

Test build #62862 has started for PR 14257 at commit 16b0f49.

Test build #62862 has finished for PR 14257 at commit 16b0f49.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62862/
Test PASSed.

LGTM except for one minor comment. Thanks!

Test build #62893 has started for PR 14257 at commit 29e953a.

Test build #62893 has finished for PR 14257 at commit 29e953a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62893/
Test FAILed.

It was Java internal error.
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (sharedRuntime.cpp:834), pid=49858, tid=140477527770880

Retest this please.

Test build #62896 has started for PR 14257 at commit 29e953a.

Test build #62896 has finished for PR 14257 at commit 29e953a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62896/
Test PASSed.

Hi, @liancheng and @rxin .
It's not ready for review again.
Thank you always!

LGTM, merging to master. Thanks!

I'm also going to merge this in branch-2.0 to avoid conflicts in fixing bugs in this area.

Thank you, @liancheng and @rxin !

authors: AmplabJenkins SparkQA dongjoon-hyun liancheng rxin


text:
[SPARK-19309][SQL] disable common subexpression elimination for conditional expressions

What changes were proposed in this pull request?
As I pointed out in https://github.com/apache/spark/pull/15807#issuecomment-259143655 , the current subexpression elimination framework has a problem, it always evaluates all common subexpressions at the beginning, even they are inside conditional expressions and may not be accessed.
Ideally we should implement it like scala lazy val, so we only evaluate it when it gets accessed at lease once. https://github.com/apache/spark/issues/15837 tries this approach, but it seems too complicated and may introduce performance regression.
This PR simply stops common subexpression elimination for conditional expressions, with some cleanup.
How was this patch tested?
regression test

Test build #71720 has started for PR 16659 at commit 45608b1.

cc @viirya @kiszk @hvanhovell

This looks good to me. Just few comments.

Test build #71720 has finished for PR 16659 at commit 45608b1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71720/
Test FAILed.

Test build #71752 has started for PR 16659 at commit cda9723.

Test build #71752 has finished for PR 16659 at commit cda9723.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71752/
Test FAILed.

The child expression in Sum is wrapped in Coalesce. So making org.apache.spark.sql.SQLQuerySuite.Common subexpression elimination test failed.

Test build #71759 has started for PR 16659 at commit 9d50048.

LGTM

I reran the DatasetBenchmark, there is no performance regression.

Test build #71759 has finished for PR 16659 at commit 9d50048.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71759/
Test FAILed.

Test build #71763 has started for PR 16659 at commit e7d928c.

Test build #71763 has finished for PR 16659 at commit e7d928c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71763/
Test PASSed.

LGTM except a few comments.

Test build #71818 has started for PR 16659 at commit 0753ee6.

LGTM pending test.

Test build #71818 has finished for PR 16659 at commit 0753ee6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71818/
Test PASSed.

thanks, merging to master!

authors: AmplabJenkins SparkQA cloud-fan gatorsmile viirya


text:
[SPARK-20665][SQL][FOLLOW-UP]Move test case to MathExpressionsSuite

What changes were proposed in this pull request?
add test case to MathExpressionsSuite as #17906
How was this patch tested?
unit test cases

Can one of the admins verify this patch?

Done. Should i delete the unit test case from MathFunctionsSuite.scala?
@gatorsmile

Hm I'm not sure if it is a good idea to run so many "unit test" style tests for expressions in the end to end suites. It takes a lot of time than just running unit tests.

@10110346 Could you move them to MathExpressionsSuite, based on @rxin 's comments? Thanks!

@gatorsmile OK,i will do,thanks

Can one of the admins verify this patch?

@gatorsmile  Please review it again,thanks

ok to test

Test build #77647 has started for PR 18082 at commit 5b747ef.

Test build #77647 has finished for PR 18082 at commit 5b747ef.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77647/
Test FAILed.

Test build #77663 has started for PR 18082 at commit d9d245a.

Test build #77663 has finished for PR 18082 at commit d9d245a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77663/
Test PASSed.

Test passed, thanks.  @gatorsmile

Test build #77834 has started for PR 18082 at commit 0ca63be.

Test build #77834 has finished for PR 18082 at commit 0ca63be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77834/
Test PASSed.

@gatorsmile  ok,i will modify and  resubmit

Test build #77904 has started for PR 18082 at commit 4e20839.

Test build #77904 has finished for PR 18082 at commit 4e20839.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77904/
Test PASSed.

Thanks! Merging to master.

authors: 10110346 AmplabJenkins SparkQA gatorsmile rxin


text:
[SPARK-22349]In on-heap mode, when allocating memory from pool,we should fill memory with MEMORY_DEBUG_FILL_CLEAN_VALUE

What changes were proposed in this pull request?
In on-heap mode, when allocating memory from pool,we should fill memory with MEMORY_DEBUG_FILL_CLEAN_VALUE
How was this patch tested?
added unit tests

Test build #83035 has started for PR 19572 at commit 95ae9c3.

cc @sameeragarwal @ericl

Test build #83035 has finished for PR 19572 at commit 95ae9c3.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83035/
Test FAILed.

retest this please

Test build #83036 has started for PR 19572 at commit 95ae9c3.

LGTM

Test build #83036 has finished for PR 19572 at commit 95ae9c3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83036/
Test PASSed.

authors: 10110346 AmplabJenkins SparkQA kiszk


text:
[SPARK-22614] Dataset API: repartitionByRange(...)

What changes were proposed in this pull request?
This PR introduces a way to explicitly range-partition a Dataset. So far, only round-robin and hash partitioning were possible via df.repartition(...), but sometimes range partitioning might be desirable: e.g. when writing to disk, for better compression without the cost of global sort.
The current implementation piggybacks on the existing RepartitionByExpression LogicalPlan and simply adds the following logic: If its expressions are of type SortOrder, then it will do RangePartitioning; otherwise HashPartitioning. This was by far the least intrusive solution I could come up with.
How was this patch tested?
Simple end-to-end test in SQLQuerySuite.

Can one of the admins verify this patch?

ok to test

Test build #84222 has started for PR 19828 at commit 950b3dc.

add to whitelist

Test build #84223 has started for PR 19828 at commit 950b3dc.

@adrian-ionescu this looks pretty good. I left a few small comments.

Test build #84225 has started for PR 19828 at commit 671e9e4.

Test build #84223 has finished for PR 19828 at commit 950b3dc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84223/
Test PASSed.

Test build #84222 has finished for PR 19828 at commit 950b3dc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84222/
Test PASSed.

Test build #84225 has finished for PR 19828 at commit 671e9e4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84225/
Test PASSed.

Merged build finished. Test PASSed.

Test build #84247 has started for PR 19828 at commit fe690fc.

LGTM.
Like what @hvanhovell suggested, we can fix it in the follow-up PR. Thanks!

Test build #84247 has finished for PR 19828 at commit fe690fc.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84247/
Test FAILed.

[error] running /home/jenkins/workspace/SparkPullRequestBuilder/build/sbt -Phadoop-2.6 -Phive-thriftserver -Phive -Dtest.exclude.tags=org.apache.spark.tags.ExtendedHiveTest,org.apache.spark.tags.ExtendedYarnTest hive-thriftserver/test mllib/test hive/test repl/test examples/test sql/test sql-kafka-0-10/test catalyst/test ; process was terminated by signal 9

Does not seem to be related.. can we retrigger the tests?

jenkins retest this please

Test build #84252 has started for PR 19828 at commit fe690fc.

Test build #84252 has finished for PR 19828 at commit fe690fc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84252/
Test PASSed.

Test build #84258 has started for PR 19828 at commit 66b192d.

Test build #84258 has finished for PR 19828 at commit 66b192d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84258/
Test PASSed.

Test build #84296 has started for PR 19828 at commit 012d617.

Test build #84296 has finished for PR 19828 at commit 012d617.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84296/
Test PASSed.

Test build #84308 has started for PR 19828 at commit 60ec0e3.

Test build #84308 has finished for PR 19828 at commit 60ec0e3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84308/
Test PASSed.

Merged build finished. Test PASSed.

Spoke to @rxin offline and we decided not to merge this as is, so please don't merge yet. I'll update this soon.

Test build #84338 has started for PR 19828 at commit 67ca139.

Test build #84338 has finished for PR 19828 at commit 67ca139.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84338/
Test PASSed.

Test build #84347 has started for PR 19828 at commit 012baa0.

Test build #84347 has finished for PR 19828 at commit 012baa0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84347/
Test PASSed.

Merged build finished. Test PASSed.

LGTM
Thanks! Merged to master.

authors: AmplabJenkins SparkQA adrian-ionescu gatorsmile hvanhovell


text:
[SPARK-26966][ML] Update to JPMML 1.4.8

What changes were proposed in this pull request?
JPMML apparently only supports Java 9 in 1.4.2+. We are seeing text failures from JPMML relating to JAXB when running on Java 11. It's shaded and not a big change, so should be safe.
How was this patch tested?
Existing tests.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8167/
Test PASSed.

Test build #102616 has started for PR 23868 at commit 3ff554d.

JPMML-Model development branches 1.3.X and 1.4.X are generating PMML schema version 4.3 markup by default; the 1.5.X development branch will be started when the PMML schema version 4.4 is released. Perhaps this information will be useful for mapping PMML schema versions with Apache Spark development branches (2.X vs 3.X?).
Your "native PMML integration" probably doesn't cover decision tree models yet. Anyway, there have been a couple of breaking JPMML-Model API changes in the latest 1.4.X versions (such as making the org.dmg.pmml.tree.Node class polymorphic), which are all aimed at reducing the memory consumption of the PMML class model object. There's a JPMML-Model version 1.4.9 coming up in a couple weeks time, which keeps moving the needle in this area.

Test build #102616 has finished for PR 23868 at commit 3ff554d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/102616/
Test FAILed.

Merged build finished. Test FAILed.

Test build #4564 has started for PR 23868 at commit 3ff554d.

Test build #4567 has started for PR 23868 at commit 3ff554d.

Test build #4567 has finished for PR 23868 at commit 3ff554d.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test build #4568 has started for PR 23868 at commit 3ff554d.

Test build #4568 has finished for PR 23868 at commit 3ff554d.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #4569 has started for PR 23868 at commit 3ff554d.

Test build #4569 has finished for PR 23868 at commit 3ff554d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #4570 has started for PR 23868 at commit 3ff554d.

Test build #4572 has started for PR 23868 at commit 3ff554d.

Test build #4572 has finished for PR 23868 at commit 3ff554d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged to master

authors: AmplabJenkins SparkQA srowen vruusmann


text:
Add root degree to PolynomialExpansion

To PolynomialExpansion I added expansion for root degrees.
I created a pull request for the same problem in the file PolynomialExpansionGeneralized, I shall close it to avoid confusions.
For example, for root degree 3, and basis vector (x,y) we have : (x, x^(1/2), x^(1/3), y, y^(1/2), y^(1/3)
We can merge simple polynomial expansion with root polynomial expansion: for simple degree 2 and root degree 2:
x, y, xy, x^2, y^2, x^(1/2), y^(1/2)
I used java.lang.Math.pow, and there is a case in it :
If the first argument is finite and less than zero and if the second argument is finite and not an integer,then the result is NaN
For this case I raised a IllegalArgumentException; the user should normalize the features in order to have only positive features.
The case of root degree polynomial expansion is mentioned in the Stanford Machine Learning course on Coursera.
It helped myself to minimize error when performing Linear Regression.
What changes were proposed in this pull request?
To PolynomialExpansion I added expansion for root degrees.
How was this patch tested?
I did manual test on different numeric data (Integer, Double, positive, negative, zero, very large numbers, very small numbers). I tested also that the value x^1 appears once and only once in the features, whatever the setting is (setDegree only, setDegreeRoot only, both setDegree and setDegreeRoot).
Please review https://spark.apache.org/contributing.html before opening a pull request.

PolynomialExpansion with root degrees

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

I think the main purpose of the polynomial expansion support is to generate interaction terms like "x * y". This only generates fractional powers of an input. I doubt that anything but square root would be useful in almost all cases? so I doubt this is worth adding to the API.

We use sqrt for engineered features quite often, for some of the variables with a larger scale
of values. Haven't seen our DS folks use anything else by 0.5 (sqrt) for the pow .. it doesn't mean
somebody else doesn't have use cases for this. Thanks!

I used 3rd degree root for a polynomial curve, for temporal linear regression. The 3rd degree root allowed to obtain  particular shape for several exceptional elements. Now, one may say that such a curve favorises high variance, yet it described well those elements as well as the others, and helped minimizing the error for the test set and future data.

Sure, but none of that seems to call for a general function to generate the nth roots of an input. It seems harder in the common case that you want just the square or cube root. I would not add this.

authors: AmplabJenkins Catadanna Tagar srowen


text:
[SPARK-25202][SQL][FOLLOW-UP] Keep the old parameter name 'pattern' at split in Scala API

What changes were proposed in this pull request?
To address the concern pointed out in https://github.com/apache/spark/pull/22227. This will make split source-compatible by removing minimal cosmetic changes.
Why are the changes needed?
For source compatibility.
Does this PR introduce any user-facing change?
No (it will prevent potential user-facing change from the original PR)
How was this patch tested?
Unittest was changed (in order for us to detect that source compatibility easily).

Test build #119151 has started for PR 27756 at commit 4d1a86f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23893/
Test PASSed.

Test build #119151 has finished for PR 27756 at commit 4d1a86f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119151/
Test FAILed.

retest this please

Test build #119165 has started for PR 27756 at commit 4d1a86f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23908/
Test PASSed.

Test build #119165 has finished for PR 27756 at commit 4d1a86f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119165/
Test PASSed.

Oops, mistake. I was removing my old branches.

Thanks guys. I am going to merge this because the last change was comment-only. I locally ran the Scala linter to make sure.

Merged to master and branch-3.0.

authors: AmplabJenkins HyukjinKwon SparkQA maropu


text:
[SPARK-32312][SQL][PYTHON][test-java11] Upgrade Apache Arrow to version 1.0.1

What changes were proposed in this pull request?

Upgrade Apache Arrow to version 1.0.1 for the Java dependency and increase minimum version of PyArrow to 1.0.0.
Version 0.15.1 to 1.0.1 includes the following selected fixes/improvements relevant to Spark users:

Why are the changes needed?

Upgrade brings fixes, improvements and stability guarantees.
Does this PR introduce any user-facing change?

No
How was this patch tested?

Existing tests with pyarrow 1.0.0 and 1.0.1

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33047/
Test PASSed.

CC @HyukjinKwon @viirya @kiszk

Test build #128423 has started for PR 29686 at commit b4d01f8.

@HyukjinKwon so github workflow has been installing the latest pyarrow and testing against that?

Test build #128423 has finished for PR 29686 at commit b4d01f8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128423/
Test PASSed.

Version 0.15.1 to 1.0.1 includes the following selected fixes/improvements relevant to Spark users:

Is it going to update in the PR description but forgotten?

Retest this please.

Test build #128427 has started for PR 29686 at commit b4d01f8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33051/
Test PASSed.

Test build #128427 has finished for PR 29686 at commit b4d01f8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128427/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33055/
Test PASSed.

Test build #128431 has started for PR 29686 at commit b4d01f8.

Is it going to update in the PR description but forgotten?

Yes, I meant to go back and update the PR desc. I'll do that in the morning.

Test build #128431 has finished for PR 29686 at commit b4d01f8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128431/
Test PASSed.

One question. Do we still need ARROW_PRE_0_15_IPC_FORMAT ?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33086/
Test PASSed.

Test build #128462 has started for PR 29686 at commit 1cf7db1.

One question. Do we still need the environment variable ARROW_PRE_0_15_IPC_FORMAT ?

This is only an env variable for pyarrow, it's not used by Spark directly. It's if the user has an older version of Spark with Arrow Java < 0.15.0 but a later version of pyarrow >= 0.15.0.

Test build #128462 has finished for PR 29686 at commit 1cf7db1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128462/
Test PASSed.

Merged to master.

Thanks all for reviewing!

authors: AmplabJenkins BryanCutler HyukjinKwon SparkQA dongjoon-hyun kiszk viirya


text:
[MINOR][INFRA] Update a broken link in .asf.yml.

What changes were proposed in this pull request?
This PR aims to update a broken link in asf.yml to the correct one.
Why are the changes needed?
The previous link shows a page not found error.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
N/A

Test build #134149 has started for PR 31210 at commit 5f86261.

Test build #134149 has started for PR 31210 at commit 5f86261.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38732/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38732/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38732/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38732/

Merged to master. Thanks!

Merged to master. Thanks!

Thank you @dongjoon-hyun!

Thank you @dongjoon-hyun!

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38732/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38732/

Test build #134149 has finished for PR 31210 at commit 5f86261.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134149 has finished for PR 31210 at commit 5f86261.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134149/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134149/

authors: AmplabJenkins SparkQA dongjoon-hyun williamhyun


text:
[SPARK-34611][DOCS] Fix grammar issues in sql-migration-guide

What changes were proposed in this pull request?

Fix some spelling in sql-migration-guide
Why are the changes needed?

Fix typo
Does this PR introduce any user-facing change?

No
How was this patch tested?

Built the docs locally and check the doc

Can one of the admins verify this patch?

authors: AmplabJenkins pingsutw


text:
[MINOR][SQL] Modify the example of rand and randn.

What changes were proposed in this pull request?

This PR fixes the examples of rand and randn.
Why are the changes needed?

SPARK-23643 (#20793) fixes an issue which is related to the seed and it causes the result of rand and randn.
Now the results of SELECT rand(0) and SELECT randn((null) are 0.7604953758285915 and 1.6034991609278433 respectively, and they should be deterministic because the number of partitions are always 1 (the leaf node is OneRowRelation).
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Built the doc and confirmed it.

Test build #139578 has started for PR 32844 at commit db48cfc.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44104/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44104/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44104/

Merged to master, brnach-3.1 and branch-3.0.

authors: AmplabJenkins HyukjinKwon SparkQA sarutak


text:
[SPARK-36824][R] Add sec and csc as R functions

What changes were proposed in this pull request?

Add secant and cosecant as R functions.
Why are the changes needed?

SEC and CSC have been added thus these functions need R support.
Does this PR introduce any user-facing change?

Yes, users can now call those functions as R functions.
How was this patch tested?

unit tests added.

Test build #143499 has started for PR 34067 at commit 1464972.

Test build #143499 has finished for PR 34067 at commit 1464972.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48010/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143499/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48010/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48010/

Merged to master.

authors: AmplabJenkins HyukjinKwon SparkQA yutoacts


text:
[SPARK-38679][CORE] Expose the number partitions in a stage to TaskContext

What changes were proposed in this pull request?

Add a new api to expose total partition count in the stage belonging to the task in TaskContext,
Why are the changes needed?

Add a new api to expose total partition count in the stage belonging to the task in TaskContext, so that the task knows what fraction of the computation is doing.
With this extra information, users can generate 32bit unique int ids as below rather than using monotonically_increasing_id which generates 64bit long ids.
rdd.mapPartitions { rowsIter =>
  val partitionId = TaskContext.get().partitionId()
  val numPartitions = TaskContext.get().numPartitions()
  var i = 0
  rowsIter.map { row =>
    val rowId = partitionId + i * numPartitions
    i += 1
    (rowId, row)
  }
}

Does this PR introduce any user-facing change?

No
How was this patch tested?

Added new unit tests to verify the number of partitions retrieved from TaskContext is expected.

It looks like barrier execution can also use this api to simplify the implementation of numTasks : )

Thank you @jiangxb1987 for reviewing. Could you please take another look at the change? Had to exclude the new method from binary compatibility test.

@cloud-fan @Ngone51 Although this one adds a new API, it's a pretty straightforward change. It looks pretty safe to me to backport into 3.3. What do you think? Also cc @MaxGekk

What do you think? Also cc @MaxGekk

Since this is either not a bug fix nor in the allow list https://lists.apache.org/thread/zrd7lcm5f5f3md7wffjy7x6w2pdmxxp7, we cannot just silently merge to branch-3.3. @zsxwing @vkorukanti Could you write an email to the thread in the dev list, and explain why we need this in 3.3 and cannot postpone to 3.4.

@MaxGekk Thanks for the feedback. We will not merge this to 3.3.

authors: MaxGekk jiangxb1987 vkorukanti zsxwing


text:
[SPARK-38796][SQL] Update to_number and try_to_number functions to restrict MI sequence to start or end only

What changes were proposed in this pull request?
Update to_number and try_to_number functions to restrict MI sequence to start or end only.
This satisfies the following specification:
to_number(expr, fmt)
fmt
  { ' [ MI ] [ S ] [ L | $ ]
      [ 0 | 9 | G | , ] [...]
      [ . | D ]
      [ 0 | 9 ] [...]
      [ L | $ ] [ PR | MI | S ] ' }

Why are the changes needed?
After reviewing the specification, this behavior makes the most sense.
Does this PR introduce any user-facing change?
Yes, a slight change in the behavior of the format string.
How was this patch tested?
Existing and updated unit test coverage.

LGTM, can we update the function doc as well?

Can one of the admins verify this patch?

LGTM, can we update the function doc as well?

@cloud-fan this is done. It looks like the On pull requests / Build and test (pull_request_target) check is failing due to the same bug in the Spark Github REST API that is affecting many other PRs. Once that is fixed, the test should pass.

The test build: https://github.com/dtenedor/spark/actions/runs/2185310585

thanks, merging to master/3.3!

authors: AmplabJenkins cloud-fan dtenedor


text:
[SPARK-40440][PS][DOCS] Fix wrong reference and content in PS windows related doc

What changes were proposed in this pull request?
Fix wrong reference and content in PS windows related doc:

Add  pyspark.pandas. for window function doc
Change pandas_on_spark.DataFrame to pyspark.pandas.DataFrame to make sure link generate correctly.
Fix Returns and See Also for Rolling.count
Add ewm doc for Dataframe and series

Why are the changes needed?
Does this PR introduce any user-facing change?

How was this patch tested?
cd ~/spark/python/docs
make html

cc @HyukjinKwon @zhengruifeng

Merged into master, thanks all

authors: Yikun zhengruifeng


text:
[MINOR][DOCS] Reviews and updates the doc links for running-on-yarn

What changes were proposed in this pull request?
The pr aim to reviews and updates the doc links for  running-on-yarn.
Why are the changes needed?
Improve docs.
After SPARK-39863, hadoop has been upgraded to version 3.3.4, but docs still retains the old link.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Manually verified.

Can one of the admins verify this patch?

Merged to master

authors: AmplabJenkins panbingkun srowen


text:
[SPARK-40718][BUILD][CONNECT][FOLLOWUP] Explicitly add Netty related dependencies for the connect module

What changes were proposed in this pull request?
After https://github.com/apache/spark/pull/38179, there are 2 netty version in Spark:

4.1.72 for connect module
4.1.80 for other modules

So this pr explicitly add netty related dependencies for the connect module to ensure Spark use unified netty version.
Why are the changes needed?
Ensure Spark use unified netty version.
Does this PR introduce any user-facing change?
No
How was this patch tested?

Pass GitHub Actions
Manual test：

run mvn dependency:tree -pl connector/connect | grep netty
Before
[INFO] |  +- io.netty:netty-all:jar:4.1.80.Final:provided
[INFO] |  |  +- io.netty:netty-buffer:jar:4.1.80.Final:compile
[INFO] |  |  +- io.netty:netty-codec:jar:4.1.80.Final:compile
[INFO] |  |  +- io.netty:netty-common:jar:4.1.80.Final:compile
[INFO] |  |  +- io.netty:netty-handler:jar:4.1.80.Final:compile
[INFO] |  |  +- io.netty:netty-resolver:jar:4.1.80.Final:provided
[INFO] |  |  +- io.netty:netty-transport:jar:4.1.80.Final:compile
[INFO] |  |  +- io.netty:netty-transport-classes-epoll:jar:4.1.80.Final:provided
[INFO] |  |  \- io.netty:netty-transport-classes-kqueue:jar:4.1.80.Final:provided
[INFO] |  +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.80.Final:provided
[INFO] |  +- io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.80.Final:provided
[INFO] |  +- io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.1.80.Final:provided
[INFO] |  +- io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.80.Final:provided
[INFO] |  +- io.netty:netty-tcnative-classes:jar:2.0.54.Final:provided
[INFO] |  \- org.apache.arrow:arrow-memory-netty:jar:9.0.0:provided
[INFO] +- io.grpc:grpc-netty:jar:1.47.0:compile
[INFO] |  +- io.netty:netty-codec-http2:jar:4.1.72.Final:compile
[INFO] |  |  \- io.netty:netty-codec-http:jar:4.1.72.Final:compile
[INFO] |  +- io.netty:netty-handler-proxy:jar:4.1.72.Final:runtime
[INFO] |  |  \- io.netty:netty-codec-socks:jar:4.1.72.Final:runtime
[INFO] |  \- io.netty:netty-transport-native-unix-common:jar:4.1.72.Final:runtime

After
[INFO] |  +- io.netty:netty-all:jar:4.1.80.Final:provided
[INFO] |  |  +- io.netty:netty-resolver:jar:4.1.80.Final:provided
[INFO] |  |  +- io.netty:netty-transport-classes-epoll:jar:4.1.80.Final:provided
[INFO] |  |  \- io.netty:netty-transport-classes-kqueue:jar:4.1.80.Final:provided
[INFO] |  +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.80.Final:provided
[INFO] |  +- io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.80.Final:provided
[INFO] |  +- io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.1.80.Final:provided
[INFO] |  +- io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.80.Final:provided
[INFO] |  +- io.netty:netty-tcnative-classes:jar:2.0.54.Final:provided
[INFO] |  \- org.apache.arrow:arrow-memory-netty:jar:9.0.0:provided
[INFO] +- io.grpc:grpc-netty:jar:1.47.0:compile
[INFO] +- io.netty:netty-codec-http2:jar:4.1.80.Final:provided
[INFO] |  +- io.netty:netty-common:jar:4.1.80.Final:provided
[INFO] |  +- io.netty:netty-buffer:jar:4.1.80.Final:provided
[INFO] |  +- io.netty:netty-transport:jar:4.1.80.Final:provided
[INFO] |  +- io.netty:netty-codec:jar:4.1.80.Final:provided
[INFO] |  +- io.netty:netty-handler:jar:4.1.80.Final:provided
[INFO] |  \- io.netty:netty-codec-http:jar:4.1.80.Final:provided
[INFO] +- io.netty:netty-handler-proxy:jar:4.1.80.Final:provided
[INFO] |  \- io.netty:netty-codec-socks:jar:4.1.80.Final:provided
[INFO] +- io.netty:netty-transport-native-unix-common:jar:4.1.80.Final:provided

cc @dongjoon-hyun @HyukjinKwon @grundprinzip

Merged to master.

+1, very late LGTM. Thank you. :)

authors: HyukjinKwon LuciferYang dongjoon-hyun


text:
[SPARK-43026][SQL] Apply AQE with non-exchange table cache

What changes were proposed in this pull request?

Apply AQE with non-exchange table cache at InsertAdaptiveSparkPlan
Why are the changes needed?

TableCacheQueryStageExec supports to report runtime statistics, so it's possible that AQE  plans a better executed during re-optimization. Then it has benefits to apply AQE even without shuffle.
Does this PR introduce any user-facing change?

no
How was this patch tested?

add test

thanks, merging to master!

authors: cloud-fan ulysses-you


text:
[SPARK-44618][INFRA] Free up disk space for non-container jobs

What changes were proposed in this pull request?
uninstall unneeded packages, referring to https://github.com/apache/flink/blob/master/tools/azure-pipelines/free_disk_space.sh
Why are the changes needed?
increase the available disk space
before:

after:

Unfortunately, this won't work for container jobs (pyspark, sparkr, lint), since those packages were not installed at all
Does this PR introduce any user-facing change?
no, test-only
How was this patch tested?
updated CI

thanks, merged to master


text:
[SPARK-45967][BUILD] Upgrade jackson to 2.16.0

What changes were proposed in this pull request?
The pr aims to upgrade FasterXML jackson from 2.15.2 to 2.16.0.
Why are the changes needed?
New version that fix some bugs, release notes as follows:

2.1.6.0 https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.16, eg:
Databind #1770: Incorrect deserialization for BigDecimal numbers
2.15.3 https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.15.3, eg:
Databind #3968: Records with additional constructors failed to deserialize

The last upgrade occurred 6 months ago, https://github.com/apache/spark/pull/41414
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA.
Was this patch authored or co-authored using generative AI tooling?
No.

cc @pjfanning ~ Are there any areas to focus on when upgrading to use Jackson 2.16.0? Thanks ~

@LuciferYang it appears you are hitting some issues with tests but other than that, I don't expect significant issues. The changes are less significant than in 2.15.x.

https://issues.apache.org/jira/browse/SPARK-42854

Thanks for your explain @pjfanning

late LGTM

authors: LuciferYang panbingkun pjfanning


text:
[SPARK-49387][PYTHON] Fix type hint for accuracy in percentile_approx and approx_percentile

What changes were proposed in this pull request?
Fix type hint for accuracy in percentile_approx and approx_percentile
Why are the changes needed?
float accuracy is not supported:
In [9]: df.select(approx_percentile("value", [0.25, 0.5, 0.75], 1.1).alias("quantiles")).show()

...

AnalysisException: [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve "approx_percentile(value, array(0.25, 0.5, 0.75), 1.1)" due to data type mismatch: The third parameter requires the "INTEGRAL" type, however "1.1" has the type "DOUBLE". SQLSTATE: 42K09;

Does this PR introduce any user-facing change?
yes, minor doc change
How was this patch tested?
CI
Was this patch authored or co-authored using generative AI tooling?
No

Merged to master.

authors: HyukjinKwon zhengruifeng


text:
[SPARK-6492][CORE] SparkContext.stop() can deadlock when DAGSchedulerEventProcessLoop dies

I've added a timeout and retry loop around the SparkContext shutdown code that should fix this deadlock. If a SparkContext shutdown is in progress when another thread comes knocking, it will wait for 10 seconds for the lock, then fall through where the outer loop will re-submit the request.

Test build #29430 has started for   PR 5277 at commit 7fb795b.

(Cross-posted from JIRA):
Timeouts are one way to fix this, but I wonder if we could also try to remove the circular wait condition by modifying EventLoop so that stopped is set before we call onError. This would prevent calls to EventLoop.stop() from blocking while the event loop is in the process of shutting down, which should prevent this race.
/cc @zsxwing, who wrote EventLoop.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29430/
Test FAILed.

whoops, my bad.  didn't mean to kill this job.

jenkins, test this please

Test build #29431 has started for   PR 5277 at commit 7fb795b.

@JoshRosen  - there's a number of places throughout the code (SparkContext.stop(), ContextCleaner.stop(), SparkEnv.stop() that I've found so far where the stopped boolean is set before the stop sequence is complete. As I mention in the intro for this PR, this can introduce a host of problems with incomplete shutdowns. I'd like to clean this up throughout the code base and would prefer to avoid adding more instances of this practice.

Test build #29432 has started for   PR 5277 at commit cdf7073.

retest this please

Hmm, good points.
In the case of the race reported in the original JIRA, I guess the main thread's SparkContext.stop() call wins, so the EventLoop thread would lose and throw the timeout exception, unblocking the main thread and allowing it to finish cleanup.  If the locks were acquired in the other order, then I guess that the EventLoop would wind up trying to join() on its own thread.  Based on http://stackoverflow.com/questions/5999595/thread-join-on-itself, it sounds like this is not safe.  Do you think that we should guard against this by checking the current thread id in EventLoop.stop() before calling join()?

Test build #29433 has started for   PR 5277 at commit 6e8a7f7.

Test build #29431 timed out     for PR 5277 at commit 7fb795b     after a configured wait of 120m.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29431/
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29432/
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29433/
Test FAILed.

From the comments, only  activeContext, contextBeingConstructed are protected by SPARK_CONTEXT_CONSTRUCTOR_LOCK. So we can move most of codes out of the lock. E.g.,
  def stop() {
    SparkContext.SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
      if (!stopped) {
        stopped = true
      } else {
        logInfo("SparkContext already stopped")
        return
      }
    }
    postApplicationEnd()
    ui.foreach(_.stop())
    env.metricsSystem.report()
    metadataCleaner.cancel()
    cleaner.foreach(_.stop())
    dagScheduler.stop()
    dagScheduler = null
    listenerBus.stop()
    eventLogger.foreach(_.stop())
    env.rpcEnv.stop(heartbeatReceiver)
    progressBar.foreach(_.stop())
    taskScheduler = null
    // TODO: Cache.stop()?
    env.stop()
    SparkEnv.set(null)
    logInfo("Successfully stopped SparkContext")
    SparkContext.clearActiveContext()
  }

@JoshRosen I assume these stops should be thread-safe. Do you know which is not?

Timeouts are one way to fix this, but I wonder if we could also try to remove the circular wait condition by modifying EventLoop so that stopped is set before we call onError. This would prevent calls to EventLoop.stop() from blocking while the event loop is in the process of shutting down, which should prevent this race.

Do you mean calling EventLoop.stop in dagScheduler.onError? I just found that calling EventLoop.stop in EventLoop.onReceive and EventLoop.onError won't call onStop. I will fix it later.

Fix the EventLoop.stop issue in #5280

Do you mean calling EventLoop.stop in dagScheduler.onError?

I was referring to the race that I reported in SPARK-6492 (referred to by this PR), where the EventLoop.onError() calls SparkContext.stop(), which blocks on the SPARK_CONTEXT_CONSTRUCTOR_LOCK lock, while another thread has simultaneously called SparkContext.stop(), acquired the SPARK_CONTEXT_CONSTRUCTOR_LOCK, and called DAGScheduler.stop(), which calls EventLoop.stop().  My comment was referring to this EventLoop.stop() call blocking indefinitely while waiting to join() on the event processing thread, which is blocked on acquiring the SPARK_CONTEXT_CONSTRUCTOR_LOCK.
Based on the comments upthread from @ilganeli, I don't think that we should adopt my earlier suggestion of having EventLoop.stop() being a no-op as long as some other thread is in the process of stopping the EventLoop, since this could result in a scenario where in-progress cleanup is still happening after a user's call to SparkContext.stop() has returned, which could lead to cleanup being skipped were the JVM to exit at that point.
I am worried about the case where calling EventLoop.stop() from the event loop thread itself (which can happen transitively here) leads to a one-threaded deadlock, but I guess your other patch addresses this?

I am worried about the case where calling EventLoop.stop() from the event loop thread itself (which can happen transitively here) leads to a one-threaded deadlock, but I guess your other patch addresses this?

Yes.  Calling EventLoop.stop() from the event loop thread itself won't lead a one-threaded deadlock.
@JoshRosen Your thoughts about moving these stops out of the lock?

From the comments and codes, only activeContext, contextBeingConstructed and stopped are protected by SPARK_CONTEXT_CONSTRUCTOR_LOCK.

@zsxwing, I took a closer look and I'm not even sure that we need the synchronization for guarding stopped, since it's volatile:
  @volatile private var stopped: Boolean = false

I suppose that the SPARK_CONTEXT_CONSTRUCTOR_LOCK was preventing multiple threads from being in stop() at the same time, but I don't know that we need to enforce that requirement; it's not enforced in pre-1.2 versions of the code (such as https://github.com/apache/spark/blob/39761f515d65afff377873ee4701b9313c317a60/core/src/main/scala/org/apache/spark/SparkContext.scala#L1024).  Maybe we can resolve this deadlock by removing this synchronized call?

In fact, it looks like the synchronized call in stop() was added in a very early iteration on that patch but became unnecessary after the code was refactored during code review.

Maybe we can resolve this deadlock by removing this synchronized call?

Removing synchronized is not enough. stopped also need to be AtomicBoolean to make the reading and writing stopped atomic.

stopped also need to be AtomicBoolean to make the reading and writing stopped atomic.

Ah, right, since we're doing a compare-and-swap here (I got mixed up thinking of a different patch; sorry).

Ok so it looks like we have a simple fix for the time being, in the long-term, so we aren't assigning to the stopped before the entire process is complete, I'd prefer a slightly different solution. I would add a second AtomicBoolean stopping and make the if check to be if(!stopping && !stopped). Stopping would be set true in the beginning of the process and stopped would be set true at the end. This would guard against incomplete stops.

Test build #29455 has started for   PR 5277 at commit 76fc825.

Do we really need both stopping and stopped? From the usage of assertNotStopped(), I think if sc.stop() is called, calling other methods of sc should be rejected.

Test build #29455 has finished for   PR 5277 at commit 76fc825.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29455/
Test FAILed.

Woah, very weird MiMa error:
[error]  * synthetic method org$apache$spark$SparkContext$$SPARK_CONTEXT_CONSTRUCTOR_LOCK()java.lang.Object in object org.apache.spark.SparkContext does not have a correspondent in new version
[error]    filter with: ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.SparkContext.org$apache$spark$SparkContext$$SPARK_CONTEXT_CONSTRUCTOR_LOCK")

@zsxwing My reasoning for having both is not to allow other methods of sc to be called but to address the scenario where the context doesn't shut down completely during the shutdown step. stopping provides the synchronization and stopped is only set once all is complete.

Test build #29460 has started for   PR 5277 at commit a0e2c70.

LGTM

Test build #29460 has finished for   PR 5277 at commit a0e2c70.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29460/
Test FAILed.

Any notion of what's going on with this MiMa failure?

@ilganeli Could you add
ProblemFilters.exclude[MissingMethodProblem]("org.apache.spark.SparkContext.org$apache$spark$SparkContext$$SPARK_CONTEXT_CONSTRUCTOR_LOCK")

to MimaExcludes.scala

Certainly - just trying to understand what it's complaining about.

Test build #29462 has started for   PR 5277 at commit 2fbab66.

Before your change, the java interface of SparkContext$ contains:
public final class org.apache.spark.SparkContext$ extends java.lang.Object implements org.apache.spark.Logging{
    public java.lang.Object org$apache$spark$SparkContext$$SPARK_CONTEXT_CONSTRUCTOR_LOCK();
}

But after your change, SPARK_CONTEXT_CONSTRUCTOR_LOCK isn't used out of SparkContext$, so the compiler does not generate
public java.lang.Object org$apache$spark$SparkContext$$SPARK_CONTEXT_CONSTRUCTOR_LOCK();

I think it's OK since this is not a Scala public API.

Test build #29462 has finished for   PR 5277 at commit 2fbab66.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29462/
Test PASSed.

LGTM. I'll wait a bit for more comments.

Any other comments? I'd like to merge today.

:+1:

Ack, sorry @ilganeli but looks like this needs a little rebase. It's ready to go then.

Fixed. Thanks.

retest this please

Test build #29679 has started for   PR 5277 at commit 8617a7e.

Test build #29679 has finished for   PR 5277 at commit 8617a7e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29679/
Test PASSed.

authors: AmplabJenkins JoshRosen SparkQA ilganeli shaneknapp srowen zsxwing


text:
[SPARK-5371][SQL] Propagate types after function conversion, before futher resolution

Before it was possible for a query to flip back and forth from a resolved state, allowing resolution to propagate up before coercion had stabilized.  The issue was that ResolvedReferences would run after FunctionArgumentConversion, but before PropagateTypes had run.  This PR ensures we PropagateTypes after any coercion has applied.

Test build #29440 has started for   PR 5278 at commit dc3581a.

Test build #29440 has finished for   PR 5278 at commit dc3581a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29440/
Test PASSed.

authors: AmplabJenkins SparkQA marmbrus


text:
[SPARK-1855] Local checkpointing

Certain use cases of Spark involve RDDs with long lineages that must be truncated periodically (e.g. GraphX). The existing way of doing it is through rdd.checkpoint(), which is expensive because it writes to HDFS. This patch provides an alternative to truncate lineages cheaply without providing the same level of fault tolerance.
Local checkpointing writes checkpointed data to the local file system through the block manager. It is much faster than replicating to a reliable storage and provides the same semantics as long as executors do not fail. It is accessible through a new operator rdd.localCheckpoint() and leaves the old one unchanged. Users may even decide to combine the two and call the reliable one less frequently.
The bulk of this patch involves refactoring the checkpointing interface to accept custom implementations of checkpointing. Design doc.

Merged build triggered.

Merged build started.

Test build #36749 has started for   PR 7279 at commit 1324d25.

Test build #36749 has finished for   PR 7279 at commit 1324d25.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

This is very cool PR.

Thank you @witgo. :)

Merged build triggered.

Merged build started.

Test build #36756 has started for   PR 7279 at commit 6602052.

Test build #36756 has finished for   PR 7279 at commit 6602052.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #36761 has started for   PR 7279 at commit d980757.

Test build #36761 has finished for   PR 7279 at commit d980757.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #36815 has started for   PR 7279 at commit ee8e85e.

Test build #36815 has finished for   PR 7279 at commit ee8e85e.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #36820 has started for   PR 7279 at commit 125af6f.

Merged build triggered.

Merged build started.

Test build #36826 has started for   PR 7279 at commit 03f3126.

Test build #36826 has finished for   PR 7279 at commit 03f3126.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

retest this please

Merged build triggered.

Merged build started.

Test build #36836 has started for   PR 7279 at commit 3d4a717.

Test build #36836 has finished for   PR 7279 at commit 3d4a717.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test build #36820 has finished for   PR 7279 at commit 125af6f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #36882 has started for   PR 7279 at commit 5da18c7.

Test build #36882 has finished for   PR 7279 at commit 5da18c7.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Merged build triggered.

Merged build started.

Test build #36885 has started for   PR 7279 at commit c449b38.

Merged build finished. Test FAILed.

Test build #1017 has started for   PR 7279 at commit c449b38.

Test build #1018 has started for   PR 7279 at commit c449b38.

Test build #1017 has finished for   PR 7279 at commit c449b38.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #36885 has finished for   PR 7279 at commit c449b38.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test build #1018 has finished for   PR 7279 at commit c449b38.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build triggered.

Merged build started.

Test build #37169 has started for   PR 7279 at commit db70dc2.

Test build #37169 timed out     for PR 7279 at commit db70dc2     after a configured wait of 175m.

Merged build finished. Test FAILed.

retest this please

Merged build triggered.

Merged build started.

Test build #37204 has started for   PR 7279 at commit db70dc2.

Test build #37204 timed out     for PR 7279 at commit db70dc2     after a configured wait of 175m.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #37306 has started for   PR 7279 at commit 4eb6eb1.

Test build #37306 has finished for   PR 7279 at commit 4eb6eb1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Exchange(newPartitioning: Partitioning, child: SparkPlan) extends UnaryNode

Merged build finished. Test PASSed.

@tdas can you start looking at this?

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Test build #1079 has started for   PR 7279 at commit a92657d.

Test build #1080 has started for   PR 7279 at commit a92657d.

Test build #1081 has started for   PR 7279 at commit a92657d.

Test build #1080 has finished for   PR 7279 at commit a92657d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #1081 has finished for   PR 7279 at commit a92657d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #1079 has finished for   PR 7279 at commit a92657d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #1089 has started for   PR 7279 at commit a92657d.

Test build #1091 has started for   PR 7279 at commit a92657d.

Test build #1091 has finished for   PR 7279 at commit a92657d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #1089 has finished for   PR 7279 at commit a92657d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build triggered.

Merged build started.

Test build #37549 has started for   PR 7279 at commit f5be0f3.

Test build #37549 has finished for   PR 7279 at commit f5be0f3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #37568 has started for   PR 7279 at commit e908a42.

Test build #37568 has finished for   PR 7279 at commit e908a42.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Hi all, I spent some time benchmarking the performance improvements of this feature. I ran the following on a 4-node m1.xlarge cluster with 6 executors, each with 4g of memory. The numbers reported here are the median over 10 runs.
Micro-benchmark. Checkpoint a ~700MB input RDD

normal checkpoint: 9337ms
local checkpoint: 1599ms
improvement: 82.9%

Macro-benchmark. Run EM LDA on a ~235MB file on HDFS

--maxIterations = 20
--checkpointInterval = 2
--k = 10
normal checkpoint: 96078ms
local checkpoint: 79592ms
improvement: 17.2%

The performance improvement in LDA is not as dramatic as the micro benchmark. This is because the majority of time is spent on actual computation rather than checkpointing.

This is a very good performance for ML.    I will test this feature on cloudml/zen

Merged build triggered.

@witgo super cool! I'd be interested to hear back from your benchmark results.
@tdas I believe I addressed all of your comments. Please have another look.

Merged build started.

Test build #38622 has started for   PR 7279 at commit c2e111b.

Test build #38622 has finished for   PR 7279 at commit c2e111b.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class RequestExecutors(
class RFormula(override val uid: String) extends Estimator[RFormulaModel] with RFormulaBase

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

retest this please

Merged build triggered.

Merged build started.

Test build #38652 has started for   PR 7279 at commit 3b7dc6a.

Test build #38652 has finished for   PR 7279 at commit 3b7dc6a.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #38722 has started for   PR 7279 at commit ab003a3.

Test build #38722 has finished for   PR 7279 at commit ab003a3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Ping @tdas can you do a final sign-off?

I did a pass. And there is one critical question that is not clear to me -- how does the LocalCheckpointRDDData.doCheckpoint() work in a distributed manner.

OK, I fixed the concern with the local doCheckpoint(). That was a good catch.

Merged build triggered.

Merged build started.

Test build #39406 has started for   PR 7279 at commit 3be5aea.

Test build #39406 has finished for   PR 7279 at commit 3be5aea.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
      class SpecificSafeProjection extends $
case class FromUTCTimestamp(left: Expression, right: Expression)
case class ToUTCTimestamp(left: Expression, right: Expression)
case class DateDiff(endDate: Expression, startDate: Expression)
case class InitCap(child: Expression) extends UnaryExpression with ImplicitCastInputTypes

Merged build finished. Test FAILed.

retest this please

Test build #1281 has started for   PR 7279 at commit 3be5aea.

Test build #1282 has started for   PR 7279 at commit 3be5aea.

Merged build triggered.

Merged build started.

Test build #39442 has started for   PR 7279 at commit 3be5aea.

Test build #1282 has finished for   PR 7279 at commit 3be5aea.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #1281 has finished for   PR 7279 at commit 3be5aea.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #39442 has finished for   PR 7279 at commit 3be5aea.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Merged build triggered.

Merged build started.

Test build #39466 has started for   PR 7279 at commit 34bc059.

Test build #1284 has started for   PR 7279 at commit 34bc059.

Test build #1285 has started for   PR 7279 at commit 34bc059.

Test build #1286 has started for   PR 7279 at commit 34bc059.

Test build #1284 has finished for   PR 7279 at commit 34bc059.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class JavaKMeansExample
 class FreqSequence[Item](val sequence: Array[Array[Item]], val freq: Long) extends Serializable
class PrefixSpanModel[Item](val freqSequences: RDD[PrefixSpan.FreqSequence[Item]])
      class SpecificSafeProjection extends $
case class FromUTCTimestamp(left: Expression, right: Expression)
case class ToUTCTimestamp(left: Expression, right: Expression)
case class DateDiff(endDate: Expression, startDate: Expression)
case class InitCap(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
public final class UnsafeKVExternalSorter

Test build #1285 has finished for   PR 7279 at commit 34bc059.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 class FreqSequence[Item](val sequence: Array[Array[Item]], val freq: Long) extends Serializable
class PrefixSpanModel[Item](val freqSequences: RDD[PrefixSpan.FreqSequence[Item]])
public final class UnsafeKVExternalSorter

Test build #39466 has finished for   PR 7279 at commit 34bc059.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test build #1286 has finished for   PR 7279 at commit 34bc059.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build triggered.

Merged build started.

Test build #1300 has started for   PR 7279 at commit 34bc059.

Test build #39495 has started for   PR 7279 at commit 729600f.

Test build #1301 has started for   PR 7279 at commit 729600f.

Test build #1302 has started for   PR 7279 at commit 729600f.

Test build #39495 has finished for   PR 7279 at commit 729600f.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class RequestExecutors(appId: String, requestedTotal: Int)
  case class KillExecutors(appId: String, executorIds: Seq[String])
public class JavaKMeansExample
 class FreqSequence[Item](val sequence: Array[Array[Item]], val freq: Long) extends Serializable
class PrefixSpanModel[Item](val freqSequences: RDD[PrefixSpan.FreqSequence[Item]])
      class SpecificSafeProjection extends $
case class FromUTCTimestamp(left: Expression, right: Expression)
case class ToUTCTimestamp(left: Expression, right: Expression)
case class DateDiff(endDate: Expression, startDate: Expression)
case class InitCap(child: Expression) extends UnaryExpression with ImplicitCastInputTypes
public final class UnsafeKVExternalSorter

Merged build finished. Test FAILed.

Test build #1301 has finished for   PR 7279 at commit 729600f.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #1300 has finished for   PR 7279 at commit 34bc059.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #1302 timed out     for PR 7279 at commit 729600f     after a configured wait of 175m.

retest this please

Merged build triggered.

Merged build started.

Test build #39536 has started for   PR 7279 at commit 729600f.

Test build #39536 has finished for   PR 7279 at commit 729600f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test build #1304 has started for   PR 7279 at commit 729600f.

Test build #1304 has finished for   PR 7279 at commit 729600f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

@tdas good to go?

Yep, LGTM. Merging this to master. Great patch!! Thanks!

authors: AmplabJenkins SparkQA andrewor14 tdas witgo


text:
[SPARK-8888][SQL] Use java.util.HashMap in DynamicPartitionWriterContainer.

Just a baby step towards making it more efficient.

Merged build triggered.

Merged build started.

Test build #36763 has started for   PR 7282 at commit 3da51ae.

Test build #36763 has finished for   PR 7282 at commit 3da51ae.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

LGTM

authors: AmplabJenkins SparkQA liancheng rxin


text:
Quick Fix: fix storm-cassandra to not exceed current max violation count

This broke the Travis CI builds for other pull requests.
Sure this is the temporary solution, and I filed issues for reducing check style violation count for each module.

+1

+1

Since this patch just adds lines and builds against master are all failing, I'd rather just merge this.

authors: Ethanlm HeartSaVioR srdo

repo: apache/storm


text:
MINOR: correcting markdown format

A really trivial fix

Thanks @Ethanlm Merged into master.

@Ethanlm can you please "Quick Fix" with "MINOR: " Thanks.
+1 after fixing the commit message.

@hmcl  This is merged already. Is it possible to change the commit message?

@Ethanlm yeah, I noticed it right after my comment. Somehow I had not refreshed my PRs view. If it is easy I will change the commit message since I am about to merge something. Otherwise we will just leave it.

@hmcl  Thanks. Will use "MINOR:" next time

authors: Ethanlm hmcl vesense


text:
Datasource access on saved queries

We have created several custom roles like DB_ACCESS_XXX so that each team cannot see and interact with datasets from an other team.
So each user have the following roles :

Gamma
sql_lab : [can list on UserDBModelView, can sql json on Superset, can sqllab on Superset, can search queries on Superset, can csv on Superset, can sqllab viz on Superset, menu access on SQL Lab, menu access on SQL Editor, menu access on Saved Queries, menu access on Query Search, can list on UserLDAPModelView, can sqllab table viz on Superset]
DB_ACCESS_XXX : [database access on [Timescaledb XXX].(id:3)]
tables_modify : [can edit on TableModelView, can add on TableModelView, can list on TableModelView, can show on TableModelView, can tables on Superset, refresh on TableModelView, can save on Datasource, can get on Datasource, can datasources on Superset, menu access on Tables, can list on TableColumnInlineView, can show on TableColumnInlineView, can fetch datasource metadata on Superset]
Annotations : [can list on AnnotationLayerModelView, can show on AnnotationLayerModelView, can delete on AnnotationLayerModelView, can edit on AnnotationLayerModelView, can add on AnnotationLayerModelView, muldelete on AnnotationLayerModelView, can list on AnnotationModelView, can show on AnnotationModelView, can delete on AnnotationModelView, can edit on AnnotationModelView, can add on AnnotationModelView, muldelete on AnnotationModelView, can annotation json on Superset, menu access on Annotation Layers, menu access on Annotations, menu access on Manage]

Expected results
When users save queries, we are expecting users to only see queries that are in relation to the database XXX to which they have access, and not see the savec queries from other databases.
Example :
user1 has access to DB ABC, user 2 has access to DB EDF.
user1 saves a query qurying a table from ABC.
user1 can see their saved query, but user2 cannot.
Actual results
Taking the same example, what actually happens is that user2 can see the query user1 saved, even though they don't have access to DB ABC.
They can not run it because they don't have database access on ABC, but they can see saved results, and they can also modify the saved request. So this is posing quite a problem regarding security.
Environment

superset version: 0.38.0

[ EDIT ]
This is causing a security issue on our cluster, is this the expected behaviour? How can we change our custom roles to correct this?

Tagging @dpgaspar as he's the most familiar with the permissions system.

Tagging @dpgaspar as he's the most familiar with the permissions system.

This may also have some relationship with issue #12263.

This may also have some relationship with issue #12263.

The problem seems to have been fixed in version 1.0.1

The problem seems to have been fixed in version 1.0.1
Users now only see their own saved queries.
Now the problem is that, I share a saved query link to another admin, the admin does not have access.

source: gharchive/issue

authors: Bill0412 akelai helo-ch willbarrett

repo: apache/superset


text:
Track dashboard's RBAC role in CLI export/import

Problem
I build dashboards on a “dev” instance, export them, and import the archive on the “prod” one; all via CLI.
I grant access to these dashboards to certain roles using RBAC — i.e. I edit dashboard properties and add one (or more) user group(s) to the “Role” selector (e.g. guest in the example below):

However, the dashboard's YAML file in the export does not include any mention of the role(s) I grant access to.
So I then need to grant access to these roles again, after the exported archive get imported on “prod”.
Wished solution
I would like that the imported dashboards grant the same RBAC authorisations as their exported counterpart.
Practically, this means that I would like to:

be able to track RBAC roles on the YAML file of an exported dashboard,
be able to set RBAC roles of an imported dashboard, based on the imported YAML file data.

Alternative
Manual work… but I'm afraid I'm lazy :wink:
Additional context
Both dev and prod are running Superset 1.3.2, alongside the following feature flags:
FEATURE_FLAGS = {
    "VERSIONED_EXPORT": True,
    "DASHBOARD_RBAC": True,
}

I understand that this could have unwanted security implications, or may not always be wanted (e.g. if the roles are not the same in dev and in prod)… so this feature could be activated as an optional flag in the export and/or import CLI.

Duplicates #17564

authors: EBoisseauSierra geido


text:
Getting Error while running Queries on SQL LAB for MSSQL

A clear and concise description of what the bug is.
How to reproduce the bug

Enable MSSQL DataSource.
Run Query on SQL LAB
Create Charts

Expected results
Should able to run query
what you expected to happen.
Should able to run query via SQL Lab
Actual results
For Point 2 : Run Query on SQL LAB
I am getting this error : '_AppCtxGlobals' object has no attribute 'user'
For Point 3: If we create charts directly via Charts tab it's working fine
Screenshots
If applicable, add screenshots to help explain your problem.
Environment
(please complete the following information):

browser type and version:
superset version:  1.1.0
python version: python --version 3.7.9
node.js version: node -v
any feature flags active:

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[ ] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context
Add any other context about the problem here.

Does this issue persist for you?  I am able to run queries in SQL Lab on a SQL Server data source, in versions 2.0.0 and 2.0.1.

Closing as resolved, but if you still face this in the versions mentioned by @sfirke, we can revisit the discussion. Thanks!

authors: Asturias-sam rusackas sfirke


text:
fix: 500s uniqueness issue on updates

SUMMARY

Users would intermittently get Fatal Error 500 errors when trying to update datasets from SQLLab. The issue is happening
with if history.has_changes() coming back as true and DatasetDAO.validate_uniqueness failing. Users should be able to overwrite
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

#16859


text:
feat: Add Apache Doris support

SUMMARY

The Apache Doris database is based on the MySQL protocol and introduces the concept of Multi Catalog. Therefore, the engine specification of mysqldb will be incompatible with the catalog, and some field types are not the same as those in mysql. To fix these issues, I added the Apache Doris engine specification.
Main difference:

Supports the connection of Doris's Multi Catalog

Fixed the problem that only "BASE TABLE" was displayed in information_schema

Fixed data types for JSONB, DATEV2, DATETIMEV2, DECIMALV3, MAP, STRUCT, LARGEINT, QUANTILE_STATE and AGG_STATE

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[x] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[x] Introduces new feature or API
[ ] Removes existing feature or API

LGTM

@villebro Hello, can you help me review it?

LGTM

@villebro
Can you help trigger workflows?

@ betodealmeida Hello, excuse me for a few minutes, can you help me review it?

@liujiwen-up I've triggered the workflows for you.

Thank you for adding this! We'll try to get it into Superset. 3.1 officially. Happy to continue supporting these integrations in the Apache ecosystem :)

authors: john-bodley liujiwen-up rusackas


text:
TAJO-2071: Supporting DATE type in Parquet format

Tested to read in Hive with Parquet file created from Tajo

+1 LGTM!

authors: eminency hyunsik

repo: apache/tajo


text:
TAJO-1138 TestKillQuery.testKillQueryFromInitState fails occasionally

Simple fix ignoring exception thrown when starting scheduler in stage of KILL_WAIT or KILLED state.

TAJO-1138 was fixed by TAJO-1279, please close this PR

authors: jinossy navis


text:
TAJO-1638 Remove offset parameter from rest api result/{cacheId}

in result/{cacheId}
rest api.
offset parameter can make user confused.
because offset can't go back.
It's only set forward(not backward)
and add 3 new header for result response.
X-Tajo-Offset : from start offset
X-Tajo-Count: fetch count
X-Tajo-EOS: if there is end stream. it sets EOS true

@charsyam
Thanks for your contribution.
But I failed to merge this patch as following
Auto-merging tajo-core/src/test/java/org/apache/tajo/ws/rs/resources/TestQueryResultResource.java
Auto-merging tajo-core/src/main/java/org/apache/tajo/ws/rs/resources/QueryResultResource.java
CONFLICT (content): Merge conflict in tajo-core/src/main/java/org/apache/tajo/ws/rs/resources/QueryResultResource.java
Automatic merge failed; fix conflicts and then commit the result.

Could you check it ?

@blrunner
HI, Thanks. I rebased it :)

+1
Thank you for your contribution.

authors: blrunner charsyam


text:
THRIFT-4578 Move TAsyncProtocolProcessor into main thrift library

Updated CMake and automake files for C++.

The build failures are unrelated to the change.

repo: apache/thrift


text:
List duplicate locations when failing a deploy due to duplicated fragment names

For complex deployments with many dependencies, it's easy to lose track of exactly which ones are causing trouble. Currently, you need to look inside each of your dependency jars in order to figure out what's wrong - I think this change is a harmless quality-of-life improvement, and that it should apply cleanly to all three release branches.
Caused by: java.lang.IllegalArgumentException: More than one fragment with the name [some-popular-library] was found.
This is not legal with relative ordering. See section 8.2.2 2c of the Servlet specification for details.
Consider using absolute ordering.
(Duplicates found in: [[file:/home/mads/dev/tomcat/output/build/webapps/madswar/WEB-INF/lib/popular-lib-duplicate.jar, file:/home/mads/dev/tomcat/output/build/webapps/madswar/WEB-INF/lib/popular-lib.jar]]).

Thanks for the PR. I like it. I'm probably being over cautious but I'd like to retain backwards compatibility for the existing methods on WebXml - at least up to 10.0.x anyway. Therefore, I'm going to apply this manually with a few tweaks.

AS per previous comment, merged manually. Thanks very much for the PR.

authors: markt-asf rolsdorph

repo: apache/tomcat