text:
[SPARK-27276][PYTHON][SQL] Increase minimum version of pyarrow to 0.12.1 and remove prior workarounds

What changes were proposed in this pull request?
This increases the minimum support version of pyarrow to 0.12.1 and removes workarounds in pyspark to remain compatible with prior versions. This means that users will need to have at least pyarrow 0.12.1 installed and available in the cluster or an ImportError will be raised to indicate an upgrade is needed.
How was this patch tested?
Existing tests using:
Python 2.7.15, pyarrow 0.12.1, pandas 0.24.2
Python 3.6.7, pyarrow 0.12.1, pandas 0.24.0

Need to check about Pandas / Numpy requirements that might go along with this.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9639/
Test PASSed.

I think we go with 0.12.1 because of https://issues.apache.org/jira/browse/ARROW-4582, might cause a problem

Test build #104301 has started for PR 24298 at commit 5f91d98.

@shaneknapp this passes locally for me so lets give it a shot with the new environment when you have time, thanks!  cc @HyukjinKwon @ueshin

Test build #104301 has finished for PR 24298 at commit 5f91d98.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104301/
Test PASSed.

@shaneknapp are you going to leave Pandas at version 0.19.2 or is that being upgraded as well for the python 3.6 env?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9664/
Test PASSed.

Test build #104331 has started for PR 24298 at commit 057f505.

Test build #104331 has finished for PR 24298 at commit 057f505.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104331/
Test FAILed.

Test build #104402 has started for PR 24298 at commit cc9a305.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9733/
Test PASSed.

Test build #104402 has finished for PR 24298 at commit cc9a305.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104402/
Test FAILed.

Test build #104459 has started for PR 24298 at commit 852dc0f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/9789/
Test PASSed.

Also to note that Arrow v0.13.0 has recently been released. There were no breaking changes so it is still compatible and increasing the minimum to that version wouldn't help clean up this code anymore. Version 0.12.1 has been pretty stable so far and I still think it's the best choice for a minimum supported version right now.

Test build #104459 has finished for PR 24298 at commit 852dc0f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104459/
Test PASSed.

test this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10071/
Test PASSed.

Test build #104756 has started for PR 24298 at commit 852dc0f.

Test build #104756 has finished for PR 24298 at commit 852dc0f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104756/
Test PASSed.

From the test output, it passed all pyarrow tests on Python 3.6 with pyarrow 0.12.1, so I think this is good to go! It looks like pyarrow 0.8.0 is installed in the Python 2.7 environment, which shows tests skipped messages. This isn't a problem since we do Arrow testing with Python 3, so the old version of pyarrow can be uninstalled sometime later to clean up the test output. Let's test this one more time and I'll merge later if no more comments.

test this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10076/
Test PASSed.

Test build #104761 has started for PR 24298 at commit 852dc0f.

From the test output, it passed all pyarrow tests on Python 3.6 with pyarrow 0.12.1, so I think this is good to go! It looks like pyarrow 0.8.0 is installed in the Python 2.7 environment, which shows tests skipped messages. This isn't a problem since we do Arrow testing with Python 3, so the old version of pyarrow can be uninstalled sometime later to clean up the test output. Let's test this one more time and I'll merge later if no more comments.

if i'm not mistaken, you're looking at the Running PySpark packaging tests section and seeing the 2.7/arrow tests being skipped.
could you copy/pasta the skipped message you're seeing so that we can ascertain if we're indeed looking at the same thing?

Test build #104761 has finished for PR 24298 at commit 852dc0f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/104761/
Test PASSed.

Update: @shaneknapp is planning to upgrade the Python 2.7 env to use Pandas 0.24.2 and Pyarrow 0.12.1 also, which will be good to verify Arrow tests pass with both Python 2 & 3. This will hopefully be done on Monday and then this PR can be merged after.

Update: @shaneknapp is planning to upgrade the Python 2.7 env to use Pandas 0.24.2 and Pyarrow 0.12.1 also, which will be good to verify Arrow tests pass with both Python 2 & 3. This will hopefully be done on Monday and then this PR can be merged after.

actually, i just had a thought:
all spark branches (master, 2.3, 2.4) use the same python2.7 env...  will this impact the older branches negatively?
i can (and will) test to confirm.

all spark branches (master, 2.3, 2.4) use the same python2.7 env

I forgot about the branches sharing the same env, in that case we definitely don't want to upgrade pyarrow for the branches. I think it's ok if we leave as is, then master will just skip arrow tests for Python 2 and we are still running the same tests with Python 3.

I forgot about the branches sharing the same env, in that case we definitely don't want to upgrade pyarrow for the branches. I think it's ok if we leave as is, then master will just skip arrow tests for Python 2 and we are still running the same tests with Python 3.

sgtm++

Merged to master.

Thanks all! I'll keep an eye out on Jenkins and make sure this is running ok.

this build was triggered by the merge:
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7/5764
all the python tests passed, but holy CRAP the "skipped tests" output is super verbose and could use a for-serious refactor.

authors: AmplabJenkins BryanCutler HyukjinKwon SparkQA shaneknapp


text:
[WIP][SPARK-28292][SQL] Enable Injection of User-defined Hint

What changes were proposed in this pull request?
Move the rule RemoveAllHints after the batch Resolution.
Why are the changes needed?
User-defined hints can be resolved by the rules injected via extendedResolutionRules or postHocResolutionRules.
Does this PR introduce any user-facing change?
No
How was this patch tested?
WIP

Test build #110428 has started for PR 25746 at commit 5ea0e89.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15415/
Test PASSed.

Test build #110428 has finished for PR 25746 at commit 5ea0e89.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110428/
Test PASSed.

This way will be more flexible. Thanks .

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15620/
Test PASSed.

LGTM

Hi, @gatorsmile . Shall we remove [WIP]?

Retest this please.

Test build #110841 has started for PR 25746 at commit 5ea0e89.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15975/
Test PASSed.

Test build #110842 has started for PR 25746 at commit 5ea0e89.

Test build #110841 has finished for PR 25746 at commit 5ea0e89.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110841/
Test PASSed.

Test build #110842 has finished for PR 25746 at commit 5ea0e89.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110842/
Test PASSed.

We need to add a test case to ensure the other user-added hints can be added and recognized.

Got it. Thank you for making this change, @gatorsmile !

Test build #111203 has started for PR 25746 at commit 15d3958.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16281/
Test PASSed.

Test build #111203 has finished for PR 25746 at commit 15d3958.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111203/
Test FAILed.

retest this please

Test build #111226 has started for PR 25746 at commit 15d3958.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16301/
Test PASSed.

nit: How was this patch tested? should be also updated now.

Test build #111226 has finished for PR 25746 at commit 15d3958.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111226/
Test PASSed.

thanks, merging to master!

source: gharchive/pull-request

authors: AmplabJenkins AngersZhuuuu SparkQA cloud-fan dongjoon-hyun gatorsmile kiszk

repo: apache/spark

license_type: permissive

license_source: github-api


text:
[SPARK-29674][CORE] Update dropwizard metrics to 4.1.x for JDK 9+

What changes were proposed in this pull request?
Update the version of dropwizard metrics that Spark uses for metrics to 4.1.x, from 3.2.x.
Why are the changes needed?
This helps JDK 9+ support, per for example https://github.com/dropwizard/metrics/pull/1236
Does this PR introduce any user-facing change?
No, although downstream users with custom metrics may be affected.
How was this patch tested?
Existing tests.

Test build #112974 has started for PR 26332 at commit aefde48.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17897/
Test PASSed.

Wow. Nice!

BTW, what happens to ganglia? Do we still have Ganglia support after this PR?

Test build #112974 has finished for PR 26332 at commit aefde48.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112974/
Test FAILed.

I don't know of a reason it would change Ganglia support. metrics-ganglia is still included, updated, and built here.

Oh, wait no I see the issue. Ganglia support is in a different module that I didn't enable. Indeed it isn't present in 4.x: https://search.maven.org/search?q=g:io.dropwizard.metrics AND a:metrics-ganglia&core=gav
Hm, OK, that's a problem, as we have historically maintained some Ganglia support.
But there are actually no tests for this integration, so we wouldn't have noticed any issues with it anyway.
Maybe I need to take this to dev@ ... if the support doesn't necessarily work on JDK 11 (unproven) then do we need to consider removing it?

Since we dropped kafka-0.8 at 2.4.x (Scala-2.12), can can disable Ganglia support in JDK11 first? It's because the main distribution of Apache Spark 3.0.0 is still JDK8.

Oh.. It's difficult to keep separately for JDK8/11.

Adding a link to a similar discussion in #26212
Probably getting rid of the Ganglia reporter is a good solution. Just a thought is that if some people still need it in Spark 3.x, maybe the Ganglia reporter can be supported by Apache Spark? It does not seem like a lot of code. There is the annoying issue of LGPL licensing of course though, that's why Dropwizard metrics library dropped it, I understand.

Yep, I am next going to try that, just inlining the simple single file metrics-ganglia contained. Dropwizard itself is ALv2. Ganglia is not, but, Spark doesn't publish this module directly, so that doesn't change.
Honestly most of the work is updating the license files!

Yep, I am next going to try that, just inlining the simple single file metrics-ganglia contained. Dropwizard itself is ALv2. Ganglia is not, but, Spark doesn't publish this module directly, so that doesn't change.
Honestly most of the work is updating the license files!
If this file gets hard to maintain, we can consider removal. If it's pretty easy to keep it for now, we can just keep it.
Honestly the only problem here is lack of tests.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17957/
Test PASSed.

Test build #113036 has started for PR 26332 at commit b842ad4.

Test build #113036 has finished for PR 26332 at commit b842ad4.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class GangliaReporter extends ScheduledReporter
   public static class Builder

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113036/
Test FAILed.

Test build #113037 has started for PR 26332 at commit 79bd378.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17958/
Test PASSed.

Test build #113037 has finished for PR 26332 at commit 79bd378.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113037/
Test PASSed.

IIUC, we are publishing our Ganglia module to Maven directly from Apache Spark 1.2.0 to 2.4.4 technically.

Spark doesn't publish this module directly, so that doesn't change.

https://repo1.maven.org/maven2/org/apache/spark/spark-ganglia-lgpl_2.11/

Ah sorry, you are correct. What I really mean is that it isn't bundled in the binary release, because that would entail distributing the ganglia dependency. At least this was true last time I checked!

Retest this please.

Test build #113178 has started for PR 26332 at commit 79bd378.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18092/
Test PASSed.

Test build #113178 has finished for PR 26332 at commit 79bd378.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113178/
Test FAILed.

Retest this please.

Test build #113182 has started for PR 26332 at commit 79bd378.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18096/
Test PASSed.

Test build #113182 has finished for PR 26332 at commit 79bd378.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113182/
Test PASSed.

Thank you so much, @srowen . Merged to master.
Also, thank you, @LucaCanali .

@LucaCanali @jerryshao maybe I can get your advice on something related here.
I was investigating back-porting this to 2.4.x. I find that the tests run out of memory with this change. A heap dump suggests that a lot of memory in consumed by MetricRegistry objects, with transitive references to all sorts of big objects. (Let's put aside why those references exist for now.)
As part of my hacking, I found that manually clearing all the Metrics in the registry when Spark's MetricSystem stops seems to resolve it. As in, in MetricSystem:
def stop(): Unit = {
    ...
      sinks.foreach(_.stop)
      registry.removeMatching(new MetricFilter {
        override def matches(name: String, metric: Metric): Boolean = true
      })
    ...
  }

Is this crazy? like, is clearing out all of the metrics going to break something? or is this a sensible cleanup step?
I'm still investigating, but wanted to reach out to anyone who may be familiar with this code.

What you write sounds reasonable to me. As an additional comment I'd say that while stopping the metrics system makes sense for a testing suite, it is not something that happens often in real usage: for most cases I can think of, metrics are started/registered at the component instance startup and they stay up for all its life.

Thanks @LucaCanali . Yeah, that's my worry. This means that after the Spark cluster is stopped, the metrics all disappear. Is that expected, or a problem? I am not sure how metrics are used. Of course at that point the components' lifecycle has ended anyway. Would external services rely on 'pulling' metrics after it's done but before the JVM stops, or do metrics implementations typically 'push' as they go? sorry for the dumb question.

Metrics are written to sinks at regular time intervals (configurable, let’s say 10 seconds). I mostly use the Graphite sink to write to an InfluxDB instance. Metrics are persisted there and consumed from InfluxDB by a Grafana dashboard.
The metrics system also has the MetricsServlet and more recently PrometheusServlet sink, which publish metrics to the Spark UI, + also I believe the JmxSink is quite popular. In these cases the idea is for the metrics to be consumed by the final user by pulling. It would be the end user/consumer's job to keep a history of the gathered metrics in these cases.

OK it's sounding reasonable to explicitly remove the registry then, as it's going away pretty immediately anyway, and the sinks are shutdown. I will propose it.
That helps the tests, which do start/stop contexts, and I think we otherwise somehow 'leak' all the previous metrics registries in this case. However it's kind of specific to the tests.
The other question remains: why is this an issue only in 2.4 (as far as I can tell), and only with metrics 4.x? Since you've looked at a lot of metrics related changes @LucaCanali , do you recall anything we fixed in 3.0 that might help avoid using a lot of memory in the metrics system? I don't, and didn't see anything in skimming JIRAs.
I am still not clear whether the big closures captured by gauges that Spark registers are necessary or an accident that will otherwise cause long-running apps to 'leak' somehow. I don't think so; what it retains a reference to are all the current Spark machinery like BlockManager. Not a problem as long as there is only one live BlockManager anyway. Still, need to keep an eye on it in 3.0.

Most of the PRs for the metrics system targeting 3.0 so far have been about adding new metrics.
It's good to know that the memory usage issue does not seem to affect 3.0 though.
BTW we could work on adding a few configuration parameters to allow turning on/off some of the more "chatty metrics"? For example: #26320

authors: AmplabJenkins LucaCanali SparkQA dongjoon-hyun srowen


text:
[SPARK-36900][CORE][TEST] Refactor SPARK-36464: size returns correct positive number even with over 2GB data to pass with Java 8, 11 and 17

What changes were proposed in this pull request?
Refactor SPARK-36464: size returns correct positive number even with over 2GB data in ChunkedByteBufferOutputStreamSuite to reduce the total use of memory for this test case, then this case can pass with Java 8, Java 11 and Java 17 use -Xmx4g.
Why are the changes needed?
SPARK-36464: size returns correct positive number even with over 2GB data pass with Java 8 but OOM with Java 11 and Java 17.
Does this PR introduce any user-facing change?
No.
How was this patch tested?

Pass the Jenkins or GitHub Action
Manual test

mvn clean install -pl core -am -Dtest=none -DwildcardSuites=org.apache.spark.util.io.ChunkedByteBufferOutputStreamSuite

with Java 8, Java 11 and Java 17, all tests passed.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144247/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48766/

Test build #144287 has finished for PR 34284 at commit 5cfc955.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48784/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48784/

Test build #144305 has finished for PR 34284 at commit 5cfc955.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144305/

Hi, All.
I'll cherry-pick this test PR to branch-3.2 to improve branch-3.2 stability.

authors: AmplabJenkins LuciferYang SparkQA dongjoon-hyun


text:
[SPARK-39571][K8S] Add net-tools to Spark docker files

What changes were proposed in this pull request?
This PR aims to add net-tools to Spark docker files.
Why are the changes needed?
This will help to check the pod network status.
Does this PR introduce any user-facing change?
No. This is just additional installation on the base images.
How was this patch tested?
Pass the CIs.

Could you review this, @viirya ? This is helpful to check Pod network status to check IPv6.

Thank you so much, @viirya . Merged to master.


text:
[SPARK-39887][SQL][3.2] RemoveRedundantAliases should keep aliases that make the output of projection nodes unique

What changes were proposed in this pull request?
Keep the output attributes of a Union node's first child in the RemoveRedundantAliases rule to avoid correctness issues.
Why are the changes needed?
To fix the result of the following query:
SELECT a, b AS a FROM (
  SELECT a, a AS b FROM (SELECT a FROM VALUES (1) AS t(a))
  UNION ALL
  SELECT a, b FROM (SELECT a, b FROM VALUES (1, 2) AS t(a, b))
)

Before this PR the query returns the incorrect result:
+---+---+
|  a|  a|
+---+---+
|  1|  1|
|  2|  2|
+---+---+

After this PR it returns the expected result:
+---+---+
|  a|  a|
+---+---+
|  1|  1|
|  1|  2|
+---+---+

Does this PR introduce any user-facing change?
Yes, fixes a correctness issue.
How was this patch tested?
Added new UTs.

cc @cloud-fan

Could you re-trigger CI, @peter-toth ? Linter failure is okay, but there exists SQL failure.

Could you re-trigger CI, @peter-toth ? Linter failure is okay, but there exists SQL failure.

Thanks @dongjoon-hyun for checking the results. I've rerun the tests and only [Linters, licenses, dependencies and documentation generation failed this time.

Thank you. I confirmed that, @peter-toth .
Merged to branch-3.2.

Thank you, @peter-toth and @cloud-fan

authors: dongjoon-hyun peter-toth


text:
[SPARK-5115] Bump default hadoop.version to 2.4.0 in pom.xml.

Fixes intellij's inability to resolve many hadoop/yarn dependencies.

Test build #25121 has started for   PR 3917 at commit c78b16e.

This patch merges cleanly.

OK, I don't think the issue in SPARK-5115 should be the motivation here, even if all else equal, I'd prefer to use Hadoop 2+ as a default. Really, for most purposes, the Hadoop version does not matter, and where it matters, you have to set this version to the desired version anyway.

It sounds like you're confirming my suspicion that the default is not 1.0.4 for any good reason.
Meanwhile, SPARK-5115 outlines a pretty basic developer-environment-sanity issue that 1.0.4 is causing.
If you want me to rewrite this PR to have a different stated motivation, let me know what I should say.

Test build #25121 has finished for   PR 3917 at commit c78b16e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25121/
Test PASSed.

changed this PR to target SPARK-5134

Hm. I'd like to merge this since Hadoop 2.x seems like a much more sensible default these days.
This doesn't change whether Spark works with Hadoop 1.x and doesn't change what builds are released, etc.
The thing I think it will change is what version of Hadoop the Maven artifacts appear to depend on. As we've discussed a lot, this shouldn't matter in theory. Still, a 2.x dependency strikes me as less likely to cause problems since people are much more on 2.x these days.
It will also mean a default "mvn compile" will show some warnings about deprecated Hadoop APIs, but hey.
Any other thoughts on this? I would like to get at least one more confirmation that this seems sensible before I push the button.
@vanzin @pwendell

The one negative side-effect I can see is that it will make it easier for people to inadvertently break the hadoop 1.x build. I know because I've done that (since all my build scripts hardcode the hadoop version at 2.something).
Other than that, I'd like to meet that guy who still uses hadoop 1, and talk him out of it. :-)

@vanzin I hear you, although the flip side is that now it's a little easier to do wrong by Hadoop 2.x. Of course, it is generally backwards compatible, but we all know it's not 100%. And this is what Jenkins is for too. And yes who is still using 1.x...
I'm inclined to make this change but will wait a bit more for other thoughts.

@pwendell I'd love to hear your thoughts one way or the other on this one. My feeling is we should do this, but want a third opinion.

The purpose for this is to make intellij easier. But can't you just enable the hadoop profiles when important and get the same effects? Is it different?
I would have concerns with this (as I said on a few other occasions too) on the basis of changing people's dependency graphs substantially for downstream projects that link against our published pom's. There are embedded applications that use Spark and don't really care about being on newer Hadoop API's - for instance applications that primarily interact with S3, or they use their own storage. This gives them a major build change since Hadoop itself has a lot of transitive dependencies.
I guess the path for such applications would be that they would exclude Spark's Hadoop and add their own Hadoop dependency with the older version?

Yeah I don't think IJ is the issue. Good point about the transitive deps. My concern is that at some point, the transitive deps of a really old Hadoop version cause more surprise and harm than the change does.
Those apps shouldn't be relying on transitive deps to begin with. They have to harmonize with the version they deploy on, and so should be fixing the version / profiles they build against even now. In an ideal world, of course.
Is this something best reserved for, say, 1.4 or 2.x?

I think we can close this for now as it's probably not going to happen until 2.x. SPARK-5134 remains open.

k, thanks for following up @srowen

yeah - sorry to be a wet blanket. Basically my feeling is that we shouldn't optimize for our own convenience as developers in turn for making life more difficult (in a regressing way) for people building downstream projects based on Spark. We can make sure to clearly document how to import into intelliJ and help people enable profiles on import, etc.

@pwendell To be clear this didn't end up being anything to do with IntelliJ. It's a question of whether building vs Hadoop 1.0.4 or Hadoop 2.x by default is better. I think it reduces surprise both for developers and downstream consumers, but, at this stage it's a subtle publishing change even if in theory it doesn't affect people that use Spark artifacts "correctly". I think this should be updated for Spark 2.x at the latest.

np @pwendell, I understand there's some complicated stability concerns here.

Hey I commented on the JIRA, but some recent changes in the way we publish artifacts actually makes this more tenable of a change.
https://issues.apache.org/jira/browse/SPARK-5134

authors: AmplabJenkins SparkQA pwendell ryan-williams srowen vanzin


text:
[SPARK-41703][CONNECT][PYTHON] Combine NullType and typed_null in Literal

What changes were proposed in this pull request?
Combine NullType and typed_null in Literal
Why are the changes needed?
we can use typed_null to express NullType-d null
Does this PR introduce any user-facing change?
no
How was this patch tested?
updated ut

merged into master, thanks @HyukjinKwon for the reivews


text:
[SPARK-42437][PySpark][Connect] PySpark catalog.cacheTable will allow to specify storage level

Currently PySpark version of catalog.cacheTable function does not support to specify storage level. This is to add that.
After changes:
Spark Connect
bin/pyspark --remote "local[*]"
Python 3.9.5 (default, Nov 23 2021, 15:27:38)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/02/17 20:41:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to

      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.5.0.dev0
      /_/

Using Python version 3.9.5 (default, Nov 23 2021 15:27:38)
Client connected to the Spark Connect server at localhost
SparkSession available as 'spark'.
>>> spark.range(1).write.saveAsTable("tab1", format="csv", mode="overwrite")
>>> spark.catalog.listTables()
[Table(name='tab1', catalog='spark_catalog', namespace=['default'], description=None, tableType='MANAGED', isTemporary=False)]
>>> spark.catalog.cacheTable("tab1")
>>> spark.catalog.isCached("tab1")
True
>>> spark.catalog.clearCache()
>>> spark.catalog.isCached("tab1")
False
>>> from pyspark.storagelevel import StorageLevel
>>> spark.catalog.cacheTable("tab1", StorageLevel.OFF_HEAP)
>>> spark.catalog.isCached("tab1")
True

PySpark
/home/spark# bin/pyspark
Python 3.9.5 (default, Nov 23 2021, 15:27:38)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/02/17 20:43:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.5.0-SNAPSHOT
      /_/

Using Python version 3.9.5 (default, Nov 23 2021 15:27:38)
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1676666626670).
SparkSession available as 'spark'.

>>> spark.range(1).write.saveAsTable("tab2", format="csv", mode="overwrite")
>>> spark.catalog.listTables()
[Table(name='tab2', catalog='spark_catalog', namespace=['default'], description=None, tableType='MANAGED', isTemporary=False)]
>>> spark.catalog.cacheTable("tab2")
>>>
>>> spark.catalog.isCached("tab2")
True
>>> spark.catalog.clearCache()
>>> spark.catalog.isCached("tab2")
False
>>> from pyspark.storagelevel import StorageLevel
>>> spark.catalog.cacheTable("tab2", StorageLevel.OFF_HEAP)
>>> spark.catalog.isCached("tab2")
True

What changes were proposed in this pull request?
Add extra parameter to catalog.cacheTable
Why are the changes needed?
To allow users specify which storage level to use in cache in PySpark/Connect code
Does this PR introduce any user-facing change?
Yes
How was this patch tested?
Updated existing test cases

cc @zhengruifeng IIRC there were some open questions to support storage level in Connect?

@HyukjinKwon can you please check if it looks Ok?

@HyukjinKwon @amaliujia can you please review?
If you think implementation of StorageLevel is not right I will close and can open a new one for the param only (when there's alternative implementation is available). The problem is you cant just add something to PySpark without touching Connect. I guess it's intentional.

Closing as no traction

Sorry for late responses. We should better have this feature parity. But I need to check w/ the concern about storage level (raised internally). I will ping the guy to comment here.

Thanks @HyukjinKwon

Hi @khalidmammadov, now that Catalog in Scala client including protobuf definition has been implemented, do you want to continue working on this?
Otherwise, I can take this over.
Thanks.

Hi, thanks for letting me know. I will look at it
On Mon, 3 Apr 2023, 21:15 Takuya UESHIN, @.***> wrote:

Hi @khalidmammadov https://github.com/khalidmammadov, now that Catalog
in Scala client including protobuf definition has been implemented, do you
want to continue working on this?
Otherwise, I can take this over.
Thanks.
—
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/40015#issuecomment-1494921787, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ACYJ3NHUQOOWBY4SFT7D3XDW7MVVDANCNFSM6AAAAAAU3TDHCU
.
You are receiving this because you were mentioned.Message ID:
@.***>

The build was failing due to https://github.com/apache/spark/pull/40674 and now fixed by https://github.com/apache/spark/pull/40681
Rebased.

Thanks! merging to master.

Thanks @ueshin!

authors: HyukjinKwon amaliujia khalidmammadov ueshin


text:
[SPARK-42708][DOCS] Improve doc about protobuf java file can't be indexed.

What changes were proposed in this pull request?
Improve README doc for developers about protobuf java file can't be indexed.

Why are the changes needed?
To make sure developers to start spark easier.

Does this PR introduce any user-facing change?
NO

How was this patch tested?
No

It seems that this PR change is not necessary, I will close it, thank @LuciferYang @HyukjinKwon  for your review


text:
[SPARK-44179][CORE]Fix the number of executors is calculated incorrctly when the task fails and it is speculated that the task is still executing

What changes were proposed in this pull request?
The dynamic scheduler listener keeps track of the number of running speculative tasks. When a task fails and is not a speculative task, and its corresponding speculative task is still running, the stageAttemptToTaskIndices will not remove the task index.
Why are the changes needed?
Assuming a stage has Task 1, with Task 1.0 and a speculative task Task 1.1 running concurrently, the dynamic scheduler calculates the number of executors as 2 (pendingTask: 0, pendingSpeculative: 0, running: 2).
At this point, Task 1.0 fails, and the dynamic scheduler recalculates the number of executors as 2 (pendingTask: 1, pendingSpeculative: 0, running: 1).
Due to the failure of Task 1.0, copyRunning(1) becomes 1. As a result, Task 1 is speculated again and a SparkListenerSpeculativeTaskSubmitted event is triggered. However, the dynamic scheduler's calculation for the number of executors becomes 3 (pendingTask: 1, pendingSpeculative: 1, running: 1), which is obviously not as expected.
Then, Task 1.2 starts, and it is marked as a speculative task. However, the dynamic scheduler still calculates the number of executors as 3 (pendingTask: 1, pendingSpeculative: 1, running: 1), which again is not as expected.
Does this PR introduce any user-facing change?
No
How was this patch tested?
add ut

cc @cloud-fan @LuciferYang thanks

Thanks for contributing this patch @lyy-pineapple, I will take a look at it later tomorrow.

The issue here is in TaskSetManager (tsm), not ExecutorAllocationManager.
The scenario you laid out is correct, namely:
Initial state:

For partition P, original task T1 = 1.0  and speculative task T2 = 1.1
speculatableTasks is empty, since 1.1 has started already.

T1 fails [1]:

tsm.handleFailedTask adds the task again (addPendingTask at the end).
This in turn adds P to all the task lists

Assuming [1] has not yet been scheduled and we have checkSpeculatableTasks invoked.
The condition in checkAndSubmitSpeculatableTasks will be true, namely : !successful(index) && copiesRunning(index) == 1 && !speculatableTasks.contains(index).
Hence we add a speculative task for schedule.
This is the reason why we have 3 pending tasks - the running speculated task, the re-added task from failure, and the new speculative task.
So what ExecutorAllocationManager is doing is IMO correct - what is incorrect is addition of the second speculative task itself.
The fix would be to change the condition in checkAndSubmitSpeculatableTasks to something like:
  !successful(index) && copiesRunning(index) == 1 && !speculatableTasks.contains(index) &&
  // make sure the currently running copy is not a speculative task
  !info.speculative

+CC @Ngone51 as well, since this is a bit more involved change (though a 1 line change :-) )

The issue here is in TaskSetManager (tsm), not ExecutorAllocationManager. The scenario you laid out is correct, namely:
Initial state:

For partition P, original task T1 = 1.0  and speculative task T2 = 1.1
speculatableTasks is empty, since 1.1 has started already.

T1 fails [1]:

tsm.handleFailedTask adds the task again (addPendingTask at the end).
This in turn adds P to all the task lists

Assuming [1] has not yet been scheduled [1] and we have checkSpeculatableTasks invoked, we will have:
The condition in checkAndSubmitSpeculatableTasks will be true, namely : !successful(index) && copiesRunning(index) == 1 && !speculatableTasks.contains(index) Hence we add a speculative task for schedule. Now we actually have the running speculative task, the re-added failed task replacement, and yet another speculative task added. This is the reason why we have 3 pending tasks.
So what is incorrect is addition of the second speculative task itself. The fix would be to change the condition in checkAndSubmitSpeculatableTasks to something like:
  !successful(index) && copiesRunning(index) == 1 && !speculatableTasks.contains(index) &&
  // make sure the currently running copy is not a speculative task
  !info.speculative

+CC @Ngone51 as well, since this is a bit more involved change (though a 1 line change :-) )
[1] If scheduled, copiesRunning > 1, so no speculative task will be scheduled again in checkAndSubmitSpeculatableTasks

if add condition  !info.speculative ,the speculative task would not be speculated.I thought there may be some issues with this

We already have a speculative task running, and have scheduled the retry for the task when it failed ... so I dont see an issue with adding the condition ... having said that, given how sensitive this code is, will wait for @Ngone51 to also take a look.
Also, +CC @JoshRosen, @jiangxb1987 in case you have bandwidth to look into this :-) Would love an extra pair of eyes on this.

if we already have a speculative task running, and failed task would not retry because dequeueTaskFromList  need copiesRunning(index) == 0:
 if (!successful(index)) {             if (copiesRunning(index) == 0) {               return Some(index)             } else if (speculative && copiesRunning(index) == 1) {               return Some(index)             }           }

Yes, that condition would need to be relooked into as well, good point.

authors: lyy-pineapple mridulm


text:
[SPARK-44418][PYTHON][CONNECT] Upgrade protobuf from 3.19.5 to 3.20.3

What changes were proposed in this pull request?
This PR proposes to upgrade protobuf from 3.19.5 to 3.20.3.
Why are the changes needed?
To use the latest version in protobuf 3.x
Does this PR introduce any user-facing change?
No.
How was this patch tested?
CI int his PR should verify them.

All related tests passed.
Merged to master.


text:
[SPARK-46525][DOCKER][TESTS] Fix docker-integration-tests on Apple Sillicon

What changes were proposed in this pull request?

com.spotify.docker.client is not going to support Apple Silicons as it has already been archived and the jnr-unixsocket 0.18 it uses is not compatible with Apple Silicons.
If we run our docker IT tests on Apple Silicons, it will fail like
[info] org.apache.spark.sql.jdbc.MariaDBKrbIntegrationSuite *** ABORTED *** (2 seconds, 264 milliseconds)
[info]   com.spotify.docker.client.exceptions.DockerException: java.util.concurrent.ExecutionException: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException: java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider
[info]   at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:2828)
[info]   at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2692)
[info]   at com.spotify.docker.client.DefaultDockerClient.ping(DefaultDockerClient.java:574)
[info]   at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.$anonfun$beforeAll$1(DockerJDBCIntegrationSuite.scala:124)
[info]   at org.apache.spark.sql.jdbc.DockerIntegrationFunSuite.runIfTestsEnabled(DockerIntegrationFunSuite.scala:49)
[info]   at org.apache.spark.sql.jdbc.DockerIntegrationFunSuite.runIfTestsEnabled$(DockerIntegrationFunSuite.scala:47)
[info]   at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.runIfTestsEnabled(DockerJDBCIntegrationSuite.scala:95)
[info]   at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.beforeAll(DockerJDBCIntegrationSuite.scala:118)
[info]   at org.apache.spark.sql.jdbc.DockerKrbJDBCIntegrationSuite.super$beforeAll(DockerKrbJDBCIntegrationSuite.scala:65)
[info]   at org.apache.spark.sql.jdbc.DockerKrbJDBCIntegrationSuite.$anonfun$beforeAll$1(DockerKrbJDBCIntegrationSuite.scala:65)
[info]   at org.apache.spark.sql.jdbc.DockerIntegrationFunSuite.runIfTestsEnabled(DockerIntegrationFunSuite.scala:49)
[info]   at org.apache.spark.sql.jdbc.DockerIntegrationFunSuite.runIfTestsEnabled$(DockerIntegrationFunSuite.scala:47)
[info]   at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.runIfTestsEnabled(DockerJDBCIntegrationSuite.scala:95)
[info]   at org.apache.spark.sql.jdbc.DockerKrbJDBCIntegrationSuite.beforeAll(DockerKrbJDBCIntegrationSuite.scala:44)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info]   at java.base/java.lang.Thread.run(Thread.java:840)
[info]   Cause: java.util.concurrent.ExecutionException: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException: java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider
[info]   at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
[info]   at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
[info]   at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
[info]   at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2690)
[info]   at com.spotify.docker.client.DefaultDockerClient.ping(DefaultDockerClient.java:574)
[info]   at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.$anonfun$beforeAll$1(DockerJDBCIntegrationSuite.scala:124)
[info]   at org.apache.spark.sql.jdbc.DockerIntegrationFunSuite.runIfTestsEnabled(DockerIntegrationFunSuite.scala:49)
[info]   at org.apache.spark.sql.jdbc.DockerIntegrationFunSuite.runIfTestsEnabled$(DockerIntegrationFunSuite.scala:47)
[info]   at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.runIfTestsEnabled(DockerJDBCIntegrationSuite.scala:95)
[info]   at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.beforeAll(DockerJDBCIntegrationSuite.scala:118)
[info]   at org.apache.spark.sql.jdbc.DockerKrbJDBCIntegrationSuite.super$beforeAll(DockerKrbJDBCIntegrationSuite.scala:65)
[info]   at org.apache.spark.sql.jdbc.DockerKrbJDBCIntegrationSuite.$anonfun$beforeAll$1(DockerKrbJDBCIntegrationSuite.scala:65)
[info]   at org.apache.spark.sql.jdbc.DockerIntegrationFunSuite.runIfTestsEnabled(DockerIntegrationFunSuite.scala:49)
[info]   at org.apache.spark.sql.jdbc.DockerIntegrationFunSuite.runIfTestsEnabled$(DockerIntegrationFunSuite.scala:47)
[info]   at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.runIfTestsEnabled(DockerJDBCIntegrationSuite.scala:95)
[info]   at org.apache.spark.sql.jdbc.DockerKrbJDBCIntegrationSuite.beforeAll(DockerKrbJDBCIntegrationSuite.scala:44)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:69)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info]   at java.base/java.lang.Thread.run(Thread.java:840)
[info]   Cause: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException: java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:202)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.client.ClientRuntime.access$400(ClientRuntime.java:79)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:182)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:315)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:297)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:267)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:210)
[info]   at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info]   at java.base/java.lang.Thread.run(Thread.java:840)
[info]   Cause: java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider
[info]   at jnr.ffi.provider.InvalidProvider$1.loadLibrary(InvalidProvider.java:48)
[info]   at jnr.ffi.LibraryLoader.load(LibraryLoader.java:325)
[info]   at jnr.unixsocket.Native.<clinit>(Native.java:80)
[info]   at jnr.unixsocket.UnixSocketChannel.<init>(UnixSocketChannel.java:101)
[info]   at jnr.unixsocket.UnixSocketChannel.open(UnixSocketChannel.java:60)
[info]   at com.spotify.docker.client.UnixConnectionSocketFactory.createSocket(UnixConnectionSocketFactory.java:69)
[info]   at com.spotify.docker.client.UnixConnectionSocketFactory.createSocket(UnixConnectionSocketFactory.java:44)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:118)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.apache.connector.ApacheConnector$1.run(ApacheConnector.java:491)
[info]   at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at jersey.repackaged.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
[info]   at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
[info]   at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50)
[info]   at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:487)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:178)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:315)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:297)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:267)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:210)
[info]   at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info]   at java.base/java.lang.Thread.run(Thread.java:840)
[info]   Cause: java.lang.ExceptionInInitializerError:
[info]   at jnr.ffi.provider.jffi.NativeRuntime.getInstance(NativeRuntime.java:58)
[info]   at jnr.ffi.provider.jffi.Provider.<init>(Provider.java:29)
[info]   at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[info]   at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
[info]   at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[info]   at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
[info]   at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
[info]   at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:347)
[info]   at java.base/java.lang.Class.newInstance(Class.java:645)
[info]   at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:68)
[info]   at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.<clinit>(FFIProvider.java:57)
[info]   at jnr.ffi.provider.FFIProvider.getSystemProvider(FFIProvider.java:35)
[info]   at jnr.ffi.LibraryLoader.create(LibraryLoader.java:73)
[info]   at jnr.unixsocket.Native.<clinit>(Native.java:76)
[info]   at jnr.unixsocket.UnixSocketChannel.<init>(UnixSocketChannel.java:101)
[info]   at jnr.unixsocket.UnixSocketChannel.open(UnixSocketChannel.java:60)
[info]   at com.spotify.docker.client.UnixConnectionSocketFactory.createSocket(UnixConnectionSocketFactory.java:69)
[info]   at com.spotify.docker.client.UnixConnectionSocketFactory.createSocket(UnixConnectionSocketFactory.java:44)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:118)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
[info]   at com.spotify.docker.client.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.apache.connector.ApacheConnector$1.run(ApacheConnector.java:491)
[info]   at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at jersey.repackaged.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
[info]   at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123)
[info]   at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50)
[info]   at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:487)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:178)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:315)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:297)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:267)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
[info]   at com.spotify.docker.client.shaded.org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:210)
[info]   at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[info]   at java.base/java.lang.Thread.run(Thread.java:840)
[info]   Cause: java.lang.IllegalStateException: Can't overwrite cause with java.lang.UnsatisfiedLinkError: java.lang.UnsatisfiedLinkError: /Users/hzyaoqin/spark/target/tmp/jffi15403099445119552969.dylib: dlopen(/Users/hzyaoqin/spark/target/tmp/jffi15403099445119552969.dylib, 0x0001): tried: '/Users/hzyaoqin/spark/target/tmp/jffi15403099445119552969.dylib' (fat file, but missing compatible architecture (have 'i386,x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/hzyaoqin/spark/target/tmp/jffi15403099445119552969.dylib' (no such file), '/Users/hzyaoqin/spark/target/tmp/jffi15403099445119552969.dylib' (fat file, but missing compatible architecture (have 'i386,x86_64', need 'arm64'))

In this PR, we use its alternative to enable docker-related tests on Apple Chips
    <dependency>
      <groupId>com.github.docker-java</groupId>
      <artifactId>docker-java</artifactId>
      <scope>test</scope>
    </dependency>

Why are the changes needed?

For developers who use Apple Silicons, w/ this patch, they can test JDBC/Docker Integration test locally instead of suffering slowness from GitHub actions.
Does this PR introduce any user-facing change?

dev only
How was this patch tested?

docker-integration-tests
Note that, [Oracle|DB2】IntegrationSuite is still unavailable even with this patch because the upstream image‘s support for arm64 remains a blocker.
FYI, https://github.com/gvenzl/oci-oracle-free?tab=readme-ov-file#oracle-database-free-on-apple-m-chips.
Was this patch authored or co-authored using generative AI tooling?

PTAL when you have some time, and Happy New Year. @dongjoon-hyun

Happy New Year, too. @yaooqinn . 😄

authors: dongjoon-hyun yaooqinn


text:
[SPARK-46646][SQL][TESTS] Improve TPCDSQueryBenchmark to support other file formats

What changes were proposed in this pull request?
This PR aims to improve TPCDSQueryBenchmark to support other file formats.
Why are the changes needed?
Currently, parquet is a hard-coded because it's the default value of spark.sql.sources.default.
https://github.com/apache/spark/blob/48d22e9f876f070d35ff3dd011bfbd1b6bccb4ac/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/TPCDSQueryBenchmark.scala#L77
Does this PR introduce any user-facing change?
No
How was this patch tested?
Manual.
BEFORE
$ build/sbt "sql/Test/runMain org.apache.spark.sql.execution.benchmark.TPCDSQueryBenchmark --data-location /tmp/tpcds-sf-1-orc-snappy/"
...
[info] 18:36:39.698 ERROR org.apache.spark.executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0)
[info] java.lang.RuntimeException: file:/tmp/tpcds-sf-1-orc-snappy/catalog_page/part-00000-40446d2a-f814-4e26-b3e1-664b833bf041-c000.snappy.orc is not a Parquet file. Expected magic number at tail, but found [79, 82, 67, 25]
[info] 	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:565)
...

AFTER
$ JDK_JAVA_OPTIONS='-Dspark.sql.sources.default=orc' \
build/sbt "sql/Test/runMain org.apache.spark.sql.execution.benchmark.TPCDSQueryBenchmark --data-location /tmp/tpcds-sf-1-orc-snappy/"
...
[info] Running benchmark: TPCDS Snappy
[info]   Running case: q1
[info]   Stopped after 6 iterations, 2028 ms
[info] OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Mac OS X 14.3
[info] Apple M1 Max
[info] TPCDS Snappy:                             Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] q1                                                  305            338          24          1.5         660.4       1.0X

Was this patch authored or co-authored using generative AI tooling?

Could you review this PR, @LuciferYang ?

Thank you so much, @LuciferYang .

Since this is irrelevant from the CI result and I verified this manually, let me merge this~

Late LGTM


text:
[SPARK-5507] Added documentation for BlockMatrix

Docs for BlockMatrix. @mengxr

Test build #27662 has started for   PR 4664 at commit 4db30b0.

This patch merges cleanly.

Test build #27662 has finished for   PR 4664 at commit 4db30b0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class SparkJobInfo(namedtuple("SparkJobInfo", "jobId stageIds status")):
class SparkStageInfo(namedtuple("SparkStageInfo",
class StatusTracker(object):

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27662/
Test PASSed.

LGTM. Merged into master and branch-1.3. Thanks!

authors: AmplabJenkins SparkQA brkyvz mengxr


text:
[Spark-7496][MLlib] Update Programming guide with Online LDA

jira: https://issues.apache.org/jira/browse/SPARK-7496
Update LDA subsection of clustering section of MLlib programming guide to include OnlineLDA.

Merged build triggered.

Merged build started.

Test build #32358 has started for   PR 6046 at commit fd4c983.

Test build #32358 has finished for   PR 6046 at commit fd4c983.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class PCA(val k: Int)
class Evaluator(Params):

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32358/
Test PASSed.

Merged build finished. Test PASSed.

@hhbyyh Thanks for sending the PR.  Could you please update it to explain when users might choose EM vs. online?  You can keep it simplistic.  Can you also please link the Online LDA paper [https://www.cs.princeton.edu/~blei/papers/HoffmanBleiBach2010b.pdf]?

Merged build triggered.

Merged build started.

Test build #32482 has started for   PR 6046 at commit 4b6fbfa.

@jkbradley Thanks  for taking the time review.
I added something and didn't mention stream since we don't support it publicly in release 1.4. Maybe it's still obscure.

Test build #32482 has finished for   PR 6046 at commit 4b6fbfa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32482/
Test PASSed.

Merged build finished. Test PASSed.

@hhbyyh I agree we should not mention Streaming yet.  Thanks for the update!  LGTM, merging into master and branch-1.4

authors: AmplabJenkins SparkQA hhbyyh jkbradley


text:
[SPARK-8049] [MLLIB] drop tmp col from OneVsRest output

The temporary column should be dropped after we get the prediction column. @harsha2010

Merged build triggered.

Merged build started.

Test build #34018 has started for   PR 6592 at commit 6ee70de.

Test build #34018 has finished for   PR 6592 at commit 6ee70de.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

+1 looks good. thanks @mengxr
I do have one question: does it make sense to promote this column which actually stores the predictions of the underlying classifiers for debugging purposes. For a test example, if my classification error is high, it would be nice to be able to inspect this column for debugging.

We could add a debug flag (or debugLevel) in the future to control extra columns. By default, users shouldn't see this column.

Merged build triggered.

Merged build started.

Test build #34024 has started for   PR 6592 at commit 1d89107.

@mengxr , yes, that works. thanks!

Test build #34024 has finished for   PR 6592 at commit 1d89107.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

authors: AmplabJenkins SparkQA harsha2010 mengxr


text:
[SPARK-8861][SQL]Add accumulators to SparkPlan operations

Prototype implementations of metrics accumulator update included in a LeafNode (LocalTableScan) and a UnaryNode (basicOperators#Project). Will extend to other SparkPlan ops after initial review.
Notes for reviewers:

Accumulator updates are currently achieved by prematurely calling actions, which is undesirable. Perhaps it may be better to instrument everything except for LeafNodes?

Merged build triggered.

Merged build started.

Test build #38050 has started for   PR 7590 at commit 1a4c9de.

Test build #38050 has finished for   PR 7590 at commit 1a4c9de.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class Accumulator[T](

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Merged build triggered.

Merged build started.

Test build #38095 has started for   PR 7590 at commit 9629ce0.

Merged build finished. Test FAILed.

Test build #38095 has finished for   PR 7590 at commit 9629ce0.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class Accumulator[T](

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #38099 has started for   PR 7590 at commit 1849fe3.

Test build #38099 has finished for   PR 7590 at commit 1849fe3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #38120 has started for   PR 7590 at commit 1d5fb86.

Test build #38120 has finished for   PR 7590 at commit 1d5fb86.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #38732 has started for   PR 7590 at commit 8b9a36c.

@andrewor14 Test was failing because cacheTable was creating SparkPlans and I'm not sure that my fix is appropriate. I know you're working on something larger involving this instrumentation so do you mind checking to make sure this PR is useful? If not, I can close.

Merged build triggered.

Merged build started.

Test build #38745 has started for   PR 7590 at commit d5b070e.

Test build #38745 has finished for   PR 7590 at commit d5b070e.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test build #38732 has finished for   PR 7590 at commit 8b9a36c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Abs(child: Expression) extends UnaryExpression with ExpectsInputTypes

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #38764 has started for   PR 7590 at commit a5455b1.

Test build #38764 has finished for   PR 7590 at commit a5455b1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

@feynmanliang I think the changes here are useful. We do want to thread the memory statistics to the SQL operators as well. However, my parallel changes may conflict with this patch.

@andrewor14 Ok, I have some other things to do for QA so I will hold off on
this until you merge.
On Tue, Jul 28, 2015 at 3:58 PM andrewor14 notifications@github.com wrote:

@feynmanliang https://github.com/feynmanliang I think the changes here
are useful. We do want to thread the memory statistics to the SQL operators
as well. However, my parallel changes may conflict with this patch.
—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/7590#issuecomment-125776599.

@feynmanliang Thanks for your PR. However, I resolved SPARK-8861 in #7774 in a similar way. Could you close this one?

authors: AmplabJenkins SparkQA andrewor14 feynmanliang zsxwing


text:
[SPARK-9432][SQL] Audit expression unit tests to make sure we pass the proper numeric ranges

JIRA: https://issues.apache.org/jira/browse/SPARK-9432

Merged build started.

Test build #39723 has started for   PR 7933 at commit e719f78.

Test build #39723 has finished for   PR 7933 at commit e719f78.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

jenkins, retest this please.

Merged build triggered.

Merged build started.

Test build #216 has started for   PR 7933 at commit e719f78.

Merged build triggered.

Merged build started.

Test build #39783 has started for   PR 7933 at commit e719f78.

Test build #216 has finished for   PR 7933 at commit e719f78.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test build #39783 has finished for   PR 7933 at commit e719f78.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

authors: AmplabJenkins SparkQA yjshen


text:
WW-5406 Ensure Action excluded patterns are reinjected

WW-5406

Ignore SonarCloud results, they are incorrect again

Ignore SonarCloud results, they are incorrect again

Probably because the last scan was for Struts 7 branch, I will fix that

Ah wait, just realised I made a breaking change in PrepareOperations.java, let me fix that

Done

authors: kusalk lukaszlenart

repo: apache/struts


text:
Jinja template is not working when async query is enabled

Following up from #16412
I tried using jinja template with global async queries(GAQ) enabled in version 1.3.0. I got the same error as stated in here (#14786).
In recent release (v1.3.1rc1), the issue was addressed and it should be fixed. However, I tried to install the new release and enable jinja template as well as GAQ to simulate the previous error. It doesn't throw any error, but now it shows wrong values.
Expected results
GAQ should work with jinja templating and shows correct values.
Actual results
As you can see from the screenshot, the chart returns 0. However, if you look at the bottom data, it returns 4625 (which is the correct one) instead of 0.
Screenshots

Looking at the logs it seems the first query (for the chart) pass through the jinja template, but has no context/variable that should be appended, hence it put the value None.
superset_app             | 172.21.0.1 - - [09/Sep/2021 10:47:39] "GET /api/v1/async_event/?last_id=1631183590197-0 HTTP/1.1" 200 -
superset_app             | 2021-09-09 10:47:39,394:INFO:werkzeug:172.21.0.1 - - [09/Sep/2021 10:47:39] "GET /api/v1/async_event/?last_id=1631183590197-0 HTTP/1.1" 200 -
superset_app             | 2021-09-09 10:47:39,443:DEBUG:superset.sql_parse:Parsing with sqlparse statement: SELECT *
superset_app             | FROM "Flights"
superset_app             | WHERE "AIRLINE" = 'None'
superset_app             | 2021-09-09 10:47:39,456:INFO:superset.common.utils:Cache key: 165593a0806194e5c6b40923f7e946b2
superset_app             | 2021-09-09 10:47:39,456:DEBUG:superset.stats_logger:[stats_logger] (incr) loading_from_cache
superset_app             | 2021-09-09 10:47:39,456:DEBUG:superset.stats_logger:[stats_logger] (incr) loaded_from_cache
superset_app             | 2021-09-09 10:47:39,456:INFO:superset.common.utils:Serving from cache
superset_app             | 2021-09-09 10:47:39,461:DEBUG:superset.stats_logger:[stats_logger] (incr) ChartRestApi.data_from_cache.success
superset_app             | 2021-09-09 10:47:39,461:DEBUG:superset.stats_logger:[stats_logger] (timing) ChartRestApi.data_from_cache.time | 45.62420000002021
superset_app             | 172.21.0.1 - - [09/Sep/2021 10:47:39] "GET /api/v1/chart/data/qc-3420381c3553925b0c069dc140aaca1a HTTP/1.1" 200 -
superset_app             | 2021-09-09 10:47:39,467:INFO:werkzeug:172.21.0.1 - - [09/Sep/2021 10:47:39] "GET /api/v1/chart/data/qc-3420381c3553925b0c069dc140aaca1a HTTP/1.1" 200 -
superset_app             | 2021-09-09 10:47:39,592:DEBUG:superset.sql_parse:Parsing with sqlparse statement: SELECT *
superset_app             | FROM "Flights"
superset_app             | WHERE "AIRLINE" = 'UA'

Just for reference, I used this query for this test.
SELECT *
FROM "Flights"
WHERE "AIRLINE" = '{{ url_param('test') }}'

How to reproduce the bug

Enable GAQ.
Use url_param function in jinja template.
Run any dataset with jinja templating.

Environment
(please complete the following information):

superset version: 1.3.1rc1
python version: 3.7.9
node.js version: 14

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

+1. Noticing same behaviour.
The second time the chart is reloaded, the url_param gets passed in properly.

Validated on latest master.

@zhaoyongjie I am also encountering this 422 Error. Is your response also {"message":"Error loading data from cache"}? It seems to be related to the chart data cache.

@zhaoyongjie I am also encountering this 422 Error. Is your response also {"message":"Error loading data from cache"}? It seems to be related to the chart data cache.

I am working on this. this is because the cache-key was incorrect calculation when it applied url_param.

Hi, I am facing the same kind of issue. I am using the Jinja template variable {{ current_aw_user_id() }} in my queries.
my queries is very simple:
SELECT * FROM attendance where id ='{{current_aw_user_id()}}';
SELECT * FROM user where reporting_to_id ='{{current_aw_user_id()}}';
I am facing the same "Unexpected error" issue.

what could be the issue, I went through multiple posts but didn't find any solution.

I believe I'm having the same issue in my deployment.
The dashboard/chart loads just fine after I refresh the page.

Hi all... is this still an issue in 3.x that anyone can validate?

Closing as stale, but happy to revisit/reopen if folks are facing this in 4.x

source: gharchive/issue

authors: SanjaySharma a-cid anthony6401 kekwan rusackas zhaoyongjie

repo: apache/superset


text:
Possible hide default native filter section in standalone mode /  embedded in iframe

Screenshot

Description
Could you let me know how I can collapse Dashboard Native Filter section when I embedded dashboard in iframe ?
Possible solution
It will be nice if it will be additional parameters as it is eg. for Power BI (https://community.powerbi.com/t5/Service/Hiding-Report-Level-filters-in-the-Iframe/m-p/64756)
filterPaneEnabled=false
http://localhost:8088/superset/dashboard/12/?standalone=2&filterPaneEnabled=false

Pinging @suddjian @lilykuang @amitmiran137 to see if/how they've considered this need in particular.

IIRC it can be hidden/collapsed with a URL param. Let me see if I can find it and add a note in the docs.

I specifically use the show_native_filters property under the dashboard metadata to decide per dashboard whether to show and use the native filter left side panel

authors: amitmiran137 rafalkasa rusackas villebro


text:
Incomplete UI

Hello
I have tried the production docker-compose installation, but when logged on the browser, the UI is not shown correctly. Assets such as the icon and the style of the page is not available. I have tested on both Google Chrome and Firefox browsers.
How to reproduce the bug

Follow the instructions at: https://superset.apache.org/docs/installation/installing-superset-using-docker-compose to install superset
Go to the browser and set: localhost:8088
See the page similar to the one attach here

Expected results
I was expecting to see something similar to: https://git.cloudron.io/cloudron/superset-app/-/blob/main/screenshots/apache_superset_00.jpg?ref_type=heads
Actual results
See attached screenshot.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment

browser type and version: Google Chrome Version 117.0.5938.132 (Official Build) (64-bit)
superset version: 2.0
python version: Python 3.9.18
node.js version: node -v

The app container does not seen to have npm installed. It gives a "Command not found" error when executing:
$docker exec -it <app container> /bin/bash
#node -v
node: Command not found

The app container does not seen to have npm installed. It gives a "Command not found" error when executing: docker exec 24e node -v

You say you are using Superset 2.0.0, how are you specifying that to Docker?  Can you confirm that when you click Settings, in the bottom of the dropdown bar it says 2.0.0 and not 0.0.0-dev?

I am having the same issue with python3.10 on Superset 3.0.0

The problem is due to the content-security-policy header.
This is the error message:
Content-Security-Policy: The page’s settings blocked the loading of a resource at inline (“style-src”).

I'm not very experienced with CSP, so I don't know but perhaps it has to do with this commit: https://github.com/apache/superset/commit/1716b9f8f68c7abe4c1a082e11ccdb26dbe6a3db

cc @dpgaspar , who wrote that pull request -- any thoughts?

You say you are using Superset 2.0.0, how are you specifying that to Docker? Can you confirm that when you click Settings, in the bottom of the dropdown bar it says 2.0.0 and not 0.0.0-dev?

You are correct, it says 0.0.0-dev

I can also confirm this issue reproduced in 3.0.0 with a Content Security Policy error in the console when attempting to use a remotely hosted image for the app icon. Just upgraded from 2.1.0 to 3.0.0 today and this was working fine previously on 2.1.0.

I believe this has been closed by https://github.com/apache/superset/pull/25468.  Please reply if that is not the case.

I can confirm that for 3.0.0 it is now fixed, not using the docker-compose without tag.

authors: MathenyCode SaviorXTanren mookie- newtester805 sfirke


text:
Traffic Portal ANY_MAP Delivery Service form contains Traffic Router configurations

ANY_MAP delivery services are unknown to the Traffic Router. There are some TR configuration options present in the form that should be removed:

TR Request Headers
TR DNS TTL
Geo Limit fields (RGB, Geolimit Countries, Geolimit Redirect)

so sounds like all of these fields should be removed from the any_map ds form:

repo: apache/trafficcontrol


text:
Coverity: Fix for CID-1523671

Fix for CID-1523671 Using invalid iterator

Looks good. thanks.

repo: apache/trafficserver


text:
Ts 2790 deprecated vc doio

With clang-format complete.
Also added .vscode/ to the .gitignore

This looks like it's seriously behind on the current master. Can you rebase and push again please?

We're working on it.

I created a new pull request for this. I'm a git noob.

authors: SolidWallOfCode a-canary zwoop


text:
Doc: Add java/plantuml dependency to building the documentation.

I asked about this on the mailing list, received no replies. The primary point is to be able to use plantuml natively in the documents. This is a UML text based diagram builder that is in turn based on GraphViz (dot). It makes very nice class and sequence diagrams and I have been using it heavily in my personal Traffic Server documentation (see here for an example).
The changes here cause

configuration to check for a runtime java iff doc building is enabled.
build the SVG files currently in use from the plantuml source instead of having them checked in.
require sphinxcontrib.plantuml to support inline UML diagrams.

[approve ci rat]


text:
More HTTP/2 local and server renaming

This continues the work of #9084 with more renaming of HTTP/2 connection and stream variables and functions like so:

client -> peer
server -> local

Thanks for the review.

I feel like local / remote is a more natural combination, but I'm not a best person to choose English words. I believe your choice.

That's true. Often local and remote do go together. But I think in this case peer is better than remote for what these variables reference. The HTTP/2 RFC generally uses the term "peer" when referring to the other host:
https://www.rfc-editor.org/rfc/rfc9113.html
The term "remote" is used in the RFC, but generally to qualify peer (i.e., it says "remote peer" sometimes), or in reference in a few places to the name of a state. But in general, when discussing the other host, it calls it a "peer".
Also, #9084 already started the trend of renaming the client to peer rather than remote. So if we did choose remote instead of peer, we would have to change those places too.

I approve this PR for now, but we have remote_hpack_handle in Http2ConnectionState. Please rename it as well if you think renaming it makes sense.

Good catch. Let's rename that too.

Okay, let's stick with the terms that are used in RFC.
The new commit didn't stale my review somehow. Please go ahead and merge this PR.

authors: bneradt maskit


text:
[Bug] Check failed: (reporter->AssertEQ(data->shape[data->shape.size() - 1], weight->shape[1])) is false: DenseRel: input dimension doesn’t match, data shape=[1, 512], weight shape=[512, 1000]

Convert torch model error

Excuse me, how do you solve this problem. I meet the problem too.

repo: apache/tvm


text:
[FIX] fix bug when normalize iter with different lower bounds

If an iter has been normalized with a lower bound, and then try to normalize with a new lower bound, the iter_min need to be updated only when the new lower bound is smaller than the original one.

thanks @Karbit

authors: Karbit tqchen


text:
Fix GraphRuntime with -link-params over RPC

NDArray handle is passed incorrectly. Unit test didn't properly exercise this case--fixed test and bug.

@tqchen


text:
[microTVM][Arduino] Fix Arduino Versions in RVM Build

Currently it downloads the latest cli version which causes breaks in RVM build. This PR fixes the version at 0.18.3 which is the version that we have tested the hardware.
Also the SPRESENSE release version was generic which we should fix to 2.2.1 based on the current hack.
cc @areusch @guberti

This version is released here: https://app.vagrantup.com/tlcpack/boxes/microtvm-arduino-0.18.3/versions/0.0.2

thanks @mehrdadh , the PR is merged!

Thanks for the PR @mehrdadh! Recent changes to the Spresense SDK have rendered some of our bugfixes unnecessary, so I'm working on a follow up PR to remove those.

@guberti sounds great!

authors: areusch guberti mehrdadh


text:
The object datamodel implementation

Previous pull request has conflicts with the master so I have made another pull request

The changes are coming together.

I merged all but the last commit. The last commit was unrelated to the object builder and pointable. Make sure to keep each branch clean with only the changes for that topic/issue.
Please close.

repo: apache/vxquery

license_source: bigquery


text:
WICKET-6287 Switch from json.org to open-json - 8.x

Repository: https://github.com/tdunning/open-json

Not sure why, but I see 1.2-SNAPSHOT in the comments coming through.
That is going to point at open-json 1.3 when you actually move forward,
right?
(just call me "better safe than sorry")
On Mon, Dec 5, 2016 at 9:13 PM, Andrea Del Bene notifications@github.com
wrote:

@bitstorm approved this pull request.
Green light for me. I've also prepared a pr for Wicketstuff to work with
this new library:
wicketstuff/core#566 https://github.com/wicketstuff/core/pull/566
In pom.xml
https://github.com/apache/wicket/pull/193#pullrequestreview-10368392:

@@ -447,6 +447,12 @@
jar

    	<groupId>com.tdunning</groupId>

    	<artifactId>json</artifactId>

		<version>1.2-SNAPSHOT</version>

I've already opened an issue. Just in case...
tdunning/open-json#2 https://github.com/tdunning/open-json/issues/2
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/apache/wicket/pull/193#pullrequestreview-10368392,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPSeqjqTjPZGaOQmErVakoBjXmRza0Gks5rE__-gaJpZM4K74MM
.

@tdunning @klopfdreh
yes, the 1.2-SNAPSHOT must turn into 1.3

Hi @bitstorm - I switched it to version 1.3.

Summary before integration:

json.org is replaced with open-json, now
Because Apache Wicket 8 is at milestone level API changes are allowed and because of this all JSON library classes are removed and the external dependency of open-json has been added. This way it is easier to keep the library updated. Users have to organize imports.

@klopfdreh @martin-g
Shall we go ahead merging or do you prefer to port the pr to the other two branches first?

@bitstorm - there are two different PRs for 6.x and 7.x to preserve the API integrity.

Sorry I wasn't clear. I asked if this PR is complete and if we should proceed merging it and wicketstuff/core#566.

I vote to merge the 8.x PR. For 6.x / 7.x @martin-g already did build tests and pointed out some issues and I would suggest to check out the PRs and test them even with wicketstuff to ensure not to break something.

+1 to merge. I've tested with WicketStuff and with @sebfz1  wicket-jquery-ui. Two PR have been created for this two project but everything seems to work fine with them.

I have pushed a few additional versions of open-json that might help (or
hinder) wicket.
The basic issue was that in the bean constructor, objects were not
converted all the way down. This led to really bad behavior because the
native to-string was used on things like lists that led to ugly and
non-functional results.
This was resolved in open-json 1.6 in a way that I don't think would hurt
wicket, but it might be good for you guys to let me know if it does and if
there is a middle ground that does not.
On Mon, Dec 12, 2016 at 7:35 PM, Andrea Del Bene notifications@github.com
wrote:

+1 to merge. I've tested with WicketStuff and with @sebfz1
https://github.com/sebfz1 wicket-jquery-ui. Two PR have been created
for this two project but everything seems to work fine with them.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/wicket/pull/193#issuecomment-266395023, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAPSeqb2rypinCugy7pJriZdbrJzLbB7ks5rHSONgaJpZM4K74MM
.

Hi @tdunning - thanks for keeping us up to date with the changes! 👍
I am going to set the version to 1.6 so that we get the latest version.

Closed due to:

https://github.com/apache/wicket/commit/4e3194466c4df50c9d38ea1ed50800dea592c72b
https://github.com/apache/wicket/commit/cc98cb0675ffbe9be956e33d6b827f4d61f9f3d7
https://github.com/apache/wicket/commit/49a486ac9fb379426b36ee29a8c9101fa818e669

authors: bitstorm klopfdreh tdunning

repo: apache/wicket


text:
Broken header Downloads>Mirrors link

Clicking on the Downloads dropdown in the header and then the Mirrors option results in a 404.
It links to https://www.staged.apache.org/dyn/closer.lua whereas the production site links to https://apache.org/dyn/closer.cgi
@dave2wave  - does this have to the with the ezmd file? Should I update the base.html file to use ".cgi" instead of ".lua"?

No. Infra is overlaying the support for mirrors and lua and it is not in staged. @Humbedooh  can you clear out the extra files in mirrors and confirm the status?

repo: apache/www-site


text:
Login warning message appears on initial login

I have included the following in my report:

[4.3-SNAPSHOT] CAS server version
[Tomcat8] Container/environment type/version (Tomcat, Jetty, Java, Linux/Windows, etc)
[see below] Steps to duplicate the problem
[n/a] Stacktraces and CAS logs where appropriate.
[n/a] Configuration/Overlay pushed to https://github.com/cas-projects/cas-issues/

When the user previously elected to have warnings when logging into other services and their TGC has expired or been removed, they will be warned they are already logged in after they authenticate.

I will verify this in 5.0.0.RC1-SNAPSHOT once I get it working.

Please do. There is no such thing as 4.3 anymore, and the SNAPSHOT if it exists, is super old.

I suspect logout does not clear that warning cookie.

Even if it did, there are other ways to be forced to authenticate. I suspect, but haven't tried, reauth will trigger this as well. I know if the TGC expires, the user is warned. This warning should only trigger if the login flow was bypassed due to a valid TGC.

Agreed.

Not relevant for 5 because that feature has been removed.

I am wrong. I take that back. It's just not on by default in the UI.

This appears fixed in master.

authors: mmoayyed tduehr

repo: apereo/cas


text:
Shibcas Decode entity from request parameter

This PR pertains to the cas-server-support-shibboleth for resolving services from an external Shibboleth IdP using the Shibcas pluglin.

Adds a URL decoder step when extracting the entityId parameter from the request query params.
Removes cas-server-support-saml and cas-server-support-trusted as dependencies
The SAML module was included in the build to only provide SAML_PROTOCOL_CONSTANTS_ENTITY_ID.  Use of the shibboleth module does not require that SAML be included in the build for the CAS Server.  This protocol is stable enough that defining it locally should not be a risk, and saves compiling the saml module into the build.
Looks for the existence of the entityId as service registry entry
This may be more specific to our use of the registry than how others use it.  We don't require SPs coming from the Shibcas Plugin to be also registered in CAS, besides our IDP.  We register SP entity ids only when they need to trigger some advanced feature like MFA in CAS.  We disable the catch all wildcard and run a closed registry.  If an entity is not found in the registry, we want to use the Shib call back url in the service parameter of request.

Very nice. One question: you are actually removing implementation project(":support:cas-server-support-saml-core"), which seems like does nothing to activate SAML support. The reference is only there to provide that constant (which is likely not good). Could you clarify?

The reference is only there to provide that constant (which is likely not good). Could you clarify?

That is exactly why I removed it. We can put it back after the saml core refactor and remove the static final from here if that would be preferred.

That is exactly why I removed it. We can put it back after the saml core refactor and remove the static final from here if that would be preferred.

No it's fine. I think in master, this sort of thing is broken down so when you do a PR for master please do take this into account.

authors: mmoayyed tsschmidt


text:
Turning off unknown tooltip part

I tried turning off the y/x axis tooltip and marker but the tooltip part with the arrow pointing does not turn off. I can't seem to find what this would be called to turn off.

What is its name?

In your css just add the following
.pexcharts-xaxistooltip {   display: none; }

You can do
xaxis: {
  tooltip: {
    enabled: false
  }
}

@junedchhipa this is what I was looking for. Didn't think to look in the xaxis options. Thanks

@junedchhipa Thanks. It just happened to me and I was trying

The x->show:false option didn't work at all. Maybe might be necessary some update to the documentation (?)

authors: AmberWilliams jota12x junedchhipa

license: MIT


text:
Fetch gas price natively

See: https://api3workspace.slack.com/archives/C06R2JCFSS1/p1715585624116669?thread_ts=1715333508.842309&cid=C06R2JCFSS1

We discussed on the call to implement this now, but there was an idea to make sure sanitized gas price is higher than the base fee. Tracked in https://github.com/api3dao/airseeker-v2/issues/306

We discussed on call to only implement https://github.com/api3dao/airseeker-v2/issues/306

Although the issue https://github.com/api3dao/airseeker-v2/issues/306 requires obtaining the base fee for blocks, my investigation of the underlying getFeeData function in ethers.js revealed that it does not directly return the block's base fee. Instead, it performs heuristic calculations using both the base fee and the maximum priority fee (eth_maxPriorityFeePerGas). Specifically, it returns a value computed as 2 * base fee + max priority fee. You can see the implementation here: ethers.js source code.
While it is theoretically possible to reverse this calculation to derive the base fee, it introduces several potential issues, such as handling cases where the maximum priority fee is null, and increases the risk of errors during the reverse calculation. Addition to that, we will still end up with making 3 calls (eth_gasPrice, eth_getBlockByNumber, eth_maxPriorityFeePerGas).
Therefore, it would be more efficient to fetch gas price using low-level RPC call.

Thanks for that. I agree.


text:
apiaryio/client in Docker Hub not up to date

Was looking to upgrade our usage of the apiaryio/client docker image to the latest but there does not seem to be anything >= 0.15.0 (latest seems to be 0.14.1 at https://hub.docker.com/r/apiaryio/client/tags?page=1&ordering=last_updated).
Could you please advise if there are plans to get the Docker Hub images up to date? Thanks.

hmm, process is automated for release gem and docker. We will look into why missing that image there.

Issue should be resolved.

authors: abtris bchew

repo: apiaryio/apiary-client

license: mit