text:
[WIP][SPARK-22096][ML] use aggregateByKeyLocally in feature frequency calc…

What changes were proposed in this pull request?
NaiveBayes currently takes aggreateByKey followed by a collect to calculate frequency for each feature/label. We can implement a new function 'aggregateByKeyLocally' in RDD that merges locally on each mapper before sending results to a reducer to save one stage.
We tested on NaiveBayes and see ~16% performance gain on training with these changes.
Signed-off-by: Vincent Xie vincent.xie@intel.com
How was this patch tested?
existing test

Test build #82059 has started for PR 19318 at commit efb0fe9.

Test build #82059 has finished for PR 19318 at commit efb0fe9.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82059/
Test FAILed.

The aggregateByKeyLocally haven't been implemented. Please wait until #19317 finished or merge that PR contents into this, otherwise this PR cannot use. We should not open a PR which cannot build.

@VinceShieh can you please mark this PR's title as "[WIP]"?

thanks :)

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4201/
Test PASSed.

Test build #97740 has started for PR 19318 at commit efb0fe9.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4223/
Test PASSed.

Test build #97740 has finished for PR 19318 at commit efb0fe9.

This patch fails due to an unknown error code, -9.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97740/
Test FAILed.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4239/
Test PASSed.

Test build #97752 has started for PR 19318 at commit efb0fe9.

Test build #97752 has finished for PR 19318 at commit efb0fe9.

This patch fails to build.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97752/
Test FAILed.

Test build #97768 has started for PR 19318 at commit efb0fe9.

Test build #97768 has finished for PR 19318 at commit efb0fe9.

This patch fails to build.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97768/
Test FAILed.

Test build #97818 has started for PR 19318 at commit efb0fe9.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4273/
Test PASSed.

Test build #97818 has finished for PR 19318 at commit efb0fe9.

This patch fails build dependency tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97818/
Test FAILed.

@VinceShieh if you'd rebase this, we can consider it now that aggregateByKeyLocally is implemented

OK. Thanks. @srowen

it seems 'aggregateByKeyLocally' is not implemented

Oops yeah https://github.com/apache/spark/pull/19317 was not merged. I think this can't continue.

authors: AmplabJenkins SparkQA VinceShieh WeichenXu123 chenghao-intel srowen


text:
[SPARK-25088][CORE][MESOS][DOCS] Update Rest Server docs & defaults.

What changes were proposed in this pull request?
(a) disabled rest submission server by default in standalone mode
(b) fails the standalone master if rest server enabled & authentication secret set
(c) fails the mesos cluster dispatcher if authentication secret set
(d) doc updates
(e) when submitting a standalone app, only try the rest submission first if spark.master.rest.enabled=true
otherwise you'd see a 10 second pause like
18/08/09 08:13:22 INFO RestSubmissionClient: Submitting a request to launch an application in spark://...
18/08/09 08:13:33 WARN RestSubmissionClient: Unable to connect to server spark://...
I also made sure the mesos cluster dispatcher failed with the secret enabled, though I had to do that on slightly different code as I don't have mesos native libs around.
How was this patch tested?
I ran the tests in the mesos module & in core for org.apache.spark.deploy.*
I ran a test on a cluster with standalone master to make sure I could still start with the right configs, and would fail the right way too.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2049/
Test PASSed.

Test build #94572 has started for PR 22071 at commit b4ca224.

Test build #94572 has finished for PR 22071 at commit b4ca224.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94572/
Test FAILed.

Test build #4241 has started for PR 22071 at commit b4ca224.

@tnachen

Test build #4244 has started for PR 22071 at commit b4ca224.

Test build #4244 has finished for PR 22071 at commit b4ca224.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test build #4245 has started for PR 22071 at commit b4ca224.

Test build #4245 has finished for PR 22071 at commit b4ca224.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #94706 has started for PR 22071 at commit 897b587.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2149/
Test PASSed.

Test build #94706 has finished for PR 22071 at commit 897b587.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94706/
Test PASSed.

LGTM as well

+1

Merged to master

any objections about putting this in prior branches as well?

in this case maybe ok. perhaps just rel note this iff there's another 2.2.x or 2.1.x releases?

Hi @squito, if I may, I have a few comments to share regarding this change and probably you can help me to understand it in better details.
From  Spark Security Doc page it looks like spark.master.rest.enabled property belongs to Spark Standalone Master only. IMO, It makes sense, because it's the server side who is responsible for enabling and disabling REST endpoint according to the source code.
Now, from the [comment in the code] (https://github.com/apache/spark/blob/v2.4.0/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L176-L180) of SparkSubmit.scala it looks like REST-based gateway introduced in Spark 1.3 is the default way of communication with Spark Standalone Master with a fallback to legacy RPC.
And here's the problem: now spark.master.rest.enabled became a client-side property, not Spark Standalone Master property. So in order to communicate with Master, a user have to provide spark.master.rest.enabled with each job she submits because without it, the submission will fail. This is a pretty breaking change.
What is the reason to fail the start of MesosClusterDispatcher if spark.authenticate.secret is set? Mesos uses different mechanisms for authentication and there's no need to fail the Dispatcher - this property can be just ignored or logged at most.
TL;DR It looks like spark.master.rest.enabled should reside only in Spark Standalone Master code and shouldn't leak into the client side of SparkSubmit. Also, it can be completely ignored with Mesos Dispatcher, because REST interface is the only endpoint it exposes.
Please let me know what you think. Thanks.

source: gharchive/pull-request

authors: AmplabJenkins SparkQA akirillov felixcheung squito srowen tgravescs tnachen

repo: apache/spark

license_type: permissive

license_source: github-api


text:
[SPARK-22590][SQL] Copy sparkContext.localproperties to child thread in BroadcastExchangeExec.executionContext

What changes were proposed in this pull request?
In org.apache.spark.sql.execution.exchange.BroadcastExchangeExec#relationFuture make a copy of org.apache.spark.SparkContext#localProperties and pass it to the broadcast execution thread in org.apache.spark.sql.execution.exchange.BroadcastExchangeExec#executionContext
Why are the changes needed?
When executing BroadcastExchangeExec, the relationFuture is evaluated via a separate thread. The threads inherit the localProperties from sparkContext as they are the child threads.
These threads are created in the executionContext (thread pools). Each Thread pool has a default keepAliveSeconds of 60 seconds for idle threads.
Scenarios where the thread pool has threads which are idle and reused for a subsequent new query, the thread local properties will not be inherited from spark context (thread properties are inherited only on thread creation) hence end up having old or no properties set. This will cause taskset properties to be missing when properties are transferred by child thread via sparkContext.runJob/submitJob
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added UT

@ajithme are you still working on it? I'd like to get this in before 3.0 release and take over this one if you don't have time to complete it.

@ajithme are you still working on it? I'd like to get this in before 3.0 release and take over this one if you don't have time to complete it.

Yes. I will update the PR shortly. Sorry for delay

Test build #118581 has started for PR 27266 at commit 0f39043.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23335/
Test PASSed.

@cloud-fan @hvanhovell @HyukjinKwon Updated as per all the comments, please review

Test build #118582 has started for PR 27266 at commit 742d322.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23336/
Test PASSed.

Test build #118583 has started for PR 27266 at commit 11fffca.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23338/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23339/
Test PASSed.

Test build #118585 has started for PR 27266 at commit 763d1bc.

Test build #118586 has started for PR 27266 at commit 2ed76c3.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23340/
Test PASSed.

Test build #118583 has finished for PR 27266 at commit 11fffca.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118583/
Test FAILed.

Test build #118581 has finished for PR 27266 at commit 0f39043.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118581/
Test FAILed.

Test build #118585 has finished for PR 27266 at commit 763d1bc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118585/
Test FAILed.

Test build #118586 has finished for PR 27266 at commit 2ed76c3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118586/
Test PASSed.

thanks, merging to master/3.0!

Test build #118582 has finished for PR 27266 at commit 742d322.

This patch fails from timeout after a configured wait of 400m.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118582/
Test FAILed.

+1 from me too. Thanks @ajithme.

authors: AmplabJenkins HyukjinKwon SparkQA ajithme cloud-fan


text:
[SPARK-24266][k8s] Restart the watcher when we receive a version changed from k8s

What changes were proposed in this pull request?

Restart the watcher when it failed with a HTTP_GONE code from the kubernetes api. Which means a resource version has changed.
For more relevant information see here: https://github.com/fabric8io/kubernetes-client/issues/1075
Why are the changes needed?

Does this PR introduce any user-facing change?

No
How was this patch tested?

Running spark-submit to a k8s cluster.
Not sure how to make an automated test for this. If someone can help me out that would be great.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

Thank you for your contribution, @stijndehaes .

Test build #122141 has started for PR 28423 at commit a9ce548.

Test build #122141 has finished for PR 28423 at commit a9ce548.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122141/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/26813/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/26813/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26813/
Test PASSed.

How do we feel about backporting this to Spark 2.4.6?

How do we feel about backporting this to Spark 2.4.6?

I would very much like that, we ran into this using spark 2.4.x.

Do you think we can have a unit test case for this, @stijndehaes ?

Test build #122248 has started for PR 28423 at commit 2a4cbb6.

Test build #122248 has finished for PR 28423 at commit 2a4cbb6.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122248/
Test FAILed.

Test build #122249 has started for PR 28423 at commit f832acf.

Test build #122249 has finished for PR 28423 at commit f832acf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122249/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/26919/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/26920/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/26919/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26919/
Test PASSed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/26920/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26920/
Test PASSed.

Do you think we can have a unit test case for this, @stijndehaes ?

The current tests completely mock out this behavior, see org.apache.spark.deploy.k8s.submit.ClientSuite, I think writing a test for this where we manipulate and fake the HTTP gone with a mock is not that useful. Maybe I can look into an integration test, but then I have to be able to trigger that a resource version changes. Not sure if I will be able to

@holdenk Maybe we should refactor this behavior using the sharedinformers.
See the comment made here: https://github.com/fabric8io/kubernetes-client/issues/1075#issuecomment-623354623
I can make an example implementation of this, maybe best to do that in another PR. What do you think?

Ok I have tested this in production, there is something wrong with the code, went ahead and tried the sharedinformers approach. Will try that in production today. You can see the code here: https://github.com/stijndehaes/spark/tree/test/shared-informers

Ok reverting back to the old approach found the missing piece I think testing that out.
Shared informers have the problem that you have to watch every pod in the whole cluster atm.

Test build #122321 has started for PR 28423 at commit dad7ea2.

Test build #122321 has finished for PR 28423 at commit dad7ea2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122321/
Test FAILed.

Test build #122322 has started for PR 28423 at commit f05db8f.

Test build #122322 has finished for PR 28423 at commit f05db8f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122322/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/26991/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/26991/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26991/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/26992/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/26992/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26992/
Test PASSed.

@holdenk @dongjoon-hyun I have tested this code in production and it works. I have a couple of jobs that take roughly 4 hours to finish, these all failed without the fix and are now succeeding.
Could you take the time to review the code again?

+1 for this.  Hit this in GKE today.

Retest this please.

Hi, @stijndehaes . Could you describe how to manually verify this PR?

Test build #124237 has started for PR 28423 at commit f05db8f.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28856/
Test FAILed.

Test build #124237 has finished for PR 28423 at commit f05db8f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124237/
Test PASSed.

Test build #124283 has started for PR 28423 at commit 6ef9e12.

Test build #124283 has finished for PR 28423 at commit 6ef9e12.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124283/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28902/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28902/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28902/
Test PASSed.

Test build #124323 has started for PR 28423 at commit 581b6f9.

Test build #124323 has finished for PR 28423 at commit 581b6f9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124323/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28942/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28942/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28942/
Test FAILed.

Test build #124636 has started for PR 28423 at commit 9da57c9.

Test build #124636 has finished for PR 28423 at commit 9da57c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29254/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124636/
Test PASSed.

Retest this please

Test build #125529 has started for PR 28423 at commit 9da57c9.

Test build #125529 has finished for PR 28423 at commit 9da57c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125529/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30144/
Test FAILed.

Hi Folks @dongjoon-hyun and @holdenk, do we plan to include this patch in 2.4.7 release?

@ScrapCodes the code in the 2.4.x is significantly different from the code here. But we can reuse the same idea as here. I guess it has to be a new PR

BTW, when do we receive a version changed from K8s?

Test build #125828 has started for PR 28423 at commit 51dc743.

@ScrapCodes the code in the 2.4.x is significantly different from the code here. But we can reuse the same idea as here. I guess it has to be a new PR

Currently the JIRA has target version set as "2.4.7", so I wanted to clarify.

BTW, when do we receive a version changed from K8s?

It happens when etcd compaction kicks in for example. On aws EKS I never saw this happening on EKS 1.14, but it happens roughly every hour on EKS 1.15. Meaning that longer spark application continued working but the watch failed.

Test build #125828 has finished for PR 28423 at commit 51dc743.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125828/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30439/
Test FAILed.

BTW, when do we receive a version changed from K8s?

It happens when etcd compaction kicks in for example. On aws EKS I never saw this happening on EKS 1.14, but it happens roughly every hour on EKS 1.15. Meaning that longer spark application continued working but the watch failed.

Okay makes sense, thank you for explaining. I am wondering how useful is the watch for longer running jobs, for example all it reports is that jobs is running, completed or errored. Probably, spark ui is more informative.

BTW, when do we receive a version changed from K8s?

It happens when etcd compaction kicks in for example. On aws EKS I never saw this happening on EKS 1.14, but it happens roughly every hour on EKS 1.15. Meaning that longer spark application continued working but the watch failed.

Okay makes sense, thank you for explaining. I am wondering how useful is the watch for longer running jobs, for example all it reports is that jobs is running, completed or errored. Probably, spark ui is more informative.

Our use case is that airflow does a spark submit. And thus airflow follows it up that way :)

Will there be a backport of this to branch-2.4?

test this please

(btw i'm testing the freshly upgraded R installation on the minikube workers)

Test build #126271 has started for PR 28423 at commit 51dc743.

Test build #126271 has finished for PR 28423 at commit 51dc743.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126271/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/30887/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/30887/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30887/
Test PASSed.

Lets focus on 3.1 and then explore backporting after.

LGTM pending Jenkins

Oh wait it has passed Jenkins, excellent. If @dongjoon-hyun is ok with this PR I'll merge it by the end of the week.

@holdenk . Thank you for pinging me. Feel free to merge if you think it's okay. I don't want to be a blocker for the community PR. ;)

Whats your JIRA username @stijndehaes  ?

@holdenk my JIRA username if sdehaes

I took the commits from master and made a partial attempt to rebase this onto branch-2.4 [1].
However, the k8s api has evolved from 2.4 quite a bit so the watchOrStop function needs to be backported [2].
You can see the error message in this gitlab build [3].
Would it be useful to make a WIP pull request from [1] ?
[1] https://github.com/jkleckner/spark/tree/SPARK-24266-on-branch2.4
[2] https://github.com/jkleckner/spark/blob/SPARK-24266-on-branch2.4/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala#L193
[3] https://gitlab.com/jkleckner/spark/-/jobs/651515950

@stijndehaes In private discussions about the hang we are seeing, there appears to be another watcher [1] for the driver watching executors that also may lose notifications.
Have you run into any situations like this?
[1]  https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsWatchSnapshotSource.scala

@jkleckner I have never had a problem with the driver watching the executors. I think there was already a fallback mechanism there, but I never looked into the code for that one.

@liyinan926 Do you think there is an adequate existing fallback mechanism or do you still believe that there is a need to create a similar patch for ExecutorPodsWatchSnapshotSource ?

I see this error a lot in the batch jobs:
{"level":"WARN","timestamp":"2020-08-10 19:17:35,985","thread":"OkHttp https://kubernetes.default.svc/...","source":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager", "line":"209","message":"Exec Failure"}
java.io.EOFException
	at okio.RealBufferedSource.require(RealBufferedSource.java:61)
	at okio.RealBufferedSource.readByte(RealBufferedSource.java:74)
	at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:117)
	at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101)
	at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
^C

I do think its related to the above issue. The batch job starts, Driver is able to spin up new executors, communicate with them and get the job done, but cannot clean them up.
This is with Spark 2.4.5 and Kubernetes Version: 1.15 and 1.16 with Multi Kubernetes Masters.
The above message repeats every 10 seconds. Let me know if its not related

It looks a bit different from what I see. For me, it appears to get stuck at the very end of writing data to Bigtable in the very last task of a job. Our partner is working to back port the fix I mentioned and I will let you know if that addresses the hang.

authors: AmplabJenkins ScrapCodes SparkQA dongjoon-hyun holdenk jkleckner puneetloya shaneknapp stijndehaes


text:
[do not review][testing master PR build][discard] comment change

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Test build #126344 has started for PR 29190 at commit 4b88a16.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30960/
Test PASSed.

Test build #126344 has finished for PR 29190 at commit 4b88a16.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126344/
Test PASSed.

Test build #126360 has started for PR 29190 at commit 13951c8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30976/
Test PASSed.

Test build #126360 has finished for PR 29190 at commit 13951c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126360/
Test PASSed.

Merged build finished. Test PASSed.

Test build #126387 has started for PR 29190 at commit fa41817.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31002/
Test PASSed.

Test build #126387 has finished for PR 29190 at commit fa41817.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126387/
Test FAILed.

Test build #126455 has started for PR 29190 at commit 54f8574.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31069/
Test PASSed.

Test build #126455 has finished for PR 29190 at commit 54f8574.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126455/
Test FAILed.

Test build #126508 has started for PR 29190 at commit 2f6264d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31127/
Test PASSed.

Test build #126508 has finished for PR 29190 at commit 2f6264d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126508/
Test PASSed.

Superceded by https://github.com/apache/spark/pull/29226

authors: AmplabJenkins SparkQA agrawaldevesh


text:
[SPARK-33841][CORE] Fix issue with jobs disappearing intermittently from the SHS under high load

What changes were proposed in this pull request?

Mark SHS event log entries that were processing at the beginning of the checkForLogs run as not stale and check for this mark before deleting an event log. This fixes the issue when a particular job was displayed in the SHS and disappeared after some time, but then, in several minutes showed up again.
Why are the changes needed?

The issue is caused by SPARK-29043, which is designated to improve the concurrent performance of the History Server. The change breaks the "app deletion" logic because of missing proper synchronization for processing event log entries. Since SHS now filters out all processing event log entries, such entries do not have a chance to be updated with the new lastProcessed time and thus any entity that completes processing right after filtering and before the check for stale entities will be identified as stale and will be deleted from the UI until the next checkForLogs run. This is because updated lastProcessed time is used as criteria, and event log entries that missed to be updated with a new time, will match that criteria.
The issue can be reproduced by generating a big number of event logs and uploading them to the SHS event log directory on S3. Essentially, around 236(26.7 MB) copies of an event log directory were created using shs-monitor script. Strange behavior of SHS counting the total number of applications was noticed - at first, the number was increasing as expected, but with the next page refresh, the total number of applications decreased. No errors were logged by SHS.
58 entities are displayed at 17:35:35:

25 entities are displayed at 17:36:40:

Does this PR introduce any user-facing change?

Yes, SHS users won't face the behavior when the number of displayed applications decreases periodically.
How was this patch tested?

Tested using shs-monitor script:

Build SHS with the proposed change
Download Hadoop AWS and AWS Java SDK
Prepare S3 bucket and user for programmatic access, grant required roles to the user. Get access key and secret key
Configure SHS to read event logs from S3
Start monitor script to query SHS API
Run 5 producers for ~5 mins, create 125(14.2 MB) event log directory copies
Wait for SHS to load all the applications
Verify that the number of loaded applications increases continuously over time

For more details, please refer to the shs-monitor repository.

This version of the reproduction uses event log directories instead of single files, since recent optimization
SPARK-33790 makes it hard to reproduce the issue with single event log files.

Can one of the admins verify this patch?

cc @HeartSaVioR

The PR has been rebased.
Thank you, @dongjoon-hyun!

add to whitelist

retest this, please

I don't think this PR has an issue, but we need to ping @tgravescs to get his approval technically. Ping, @tgravescs .

+1 and I'd consider @tgravescs approved this PR as only target branch is different.

Test build #133038 has started for PR 30845 at commit e9f13a5.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37637/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37637/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37637/

Thank you for your first contribution, @vladhlinsky .
I added you to the Apache Spark contributor group and assigned SPARK-33841 to you.
Welcome to the Apache Spark community.

Thank you, @dongjoon-hyun!

Test build #133038 has finished for PR 30845 at commit e9f13a5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133038/

authors: AmplabJenkins HeartSaVioR SparkQA dongjoon-hyun vladhlinsky


text:
[MINOR][DOCS] Update pandas_pyspark.rst

What changes were proposed in this pull request?

'to' --> 'the'
Why are the changes needed?

Grammar fix
Does this PR introduce any user-facing change?

Documentation fix
How was this patch tested?

No test needed, same as yesterday's PR

Can one of the admins verify this patch?

@kamelCased while we're here, can we find all these instances, and fix them in one PR?

@HyukjinKwon I checked it's the last one in this file. I will check the rest of the PySpark docs.

The test fails because your bunch is not synced to the latest matster branch. do you mind rebasing to the latest master branch?

Also please update PR description to reflect the changes.

ok to test

@HyukjinKwon fetched upstream and PR description updated

@HyukjinKwon fetched upstream and PR description updated

@HyukjinKwon fetched upstream and PR description updated

Merged to master

authors: AmplabJenkins HyukjinKwon kamelCased srowen


text:
[SPARK-39249][SQL] Improve subexpression elimination for conditional expressions

What changes were proposed in this pull request?
Currently we can do subexpression elimination for conditional expressions when the subexpression is common across all branchGroups. In fact, we can farther improve this when there are common expressions between alwaysEvaluatedInputs and branchGroups.
Why are the changes needed?
Take the following case as an example
IF(IsNull(a), b, KnowNotNull(a))

a may miss subexpression elimination chances since it is not the common expression between all branchGroups, but it's safe to evaluate a as common subexpression and eagerly execute it since it's part of the prediction, which will always be executed. If a is a time-expensive expression, we may waste time on running it.
This kind of expressions are common when we do sum on decimal type because of https://github.com/apache/spark/pull/29026
https://github.com/apache/spark/blob/291d155b3c514f8b590a6b078f7efd42a30e67f0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Sum.scala#L125
Performance result on TPC-DS 10T

Query
With this PR
Without this PR
Speed up

4
310.862
635.299
104.37%

80
36.723
46.006
25.28%

Does this PR introduce any user-facing change?
No
How was this patch tested?
add more UT.

@viirya @cloud-fan Could you please help review this?

Can one of the admins verify this patch?

FYI I created https://github.com/apache/spark/pull/32987 a while ago to address this in a much more general way. I've tried to keep it up to date, but there seemed to be concerns about creating a subexpression for something that might only execute once I guess? Even though that's already happening in certain cases

@viirya @cloud-fan Could you please help review this?

authors: AmplabJenkins Kimahriman WangGuangxin


text:
[SPARK-4874] [CORE] Collect record count metrics

Collects record counts for both Input/Output and Shuffle Metrics. For the input/output metrics, it just appends the counter every time the iterators get accessed.
For shuffle on the write side, we count the metrics post aggregation (after a map side combine) and on the read side we count the metrics pre aggregation. This allows both the bytes read/written metrics and the records read/written to line up.
For backwards compatibility, if we deserialize an older event that doesn't have record metrics, we set the metric to -1.

Test build #25628 has started for   PR 4067 at commit 571cb69.

This patch merges cleanly.

Test build #25628 has finished for   PR 4067 at commit 571cb69.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25628/
Test PASSed.

Can you also paste some screenshots on what the UI changes look like? Thanks.

Test build #25638 has started for   PR 4067 at commit 1572054.

This patch does not merge cleanly.

This change was dependent on https://github.com/apache/spark/pull/3120, that just got merged and now there are some merge conflicts. I need to fix those first and will update the pr.

Hey @ksakellis - Thanks for working on this.
I took a very quick look at the patch. Overall I feel the patch should be fairly straightforward, but the specific implementations might've gone a bit over board with Scala language features (a lot of Options, orElse, closures, etc) and design (too many new classes introduced). If we can reduce those, the pr would be a lot easier to understand.

@rxin I updated the PR after doing a rebase and also incorporated some of your feedback. You made two general comments:

specific implementations might've gone a bit over board with Scala language features
too many new classes introduced
Can you please be more specific here pointing to the specific code. These comments are not very actionable as is.

Hi again - can't find my previous comment since the line is no longer in the diff due to the other pr being merged. Can you still add comment for that one (the part with Option and orElse and set ...)? Want to make sure if we read that code one year from now, we can still understand what's going on.

Test build #25641 has started for   PR 4067 at commit 3c2d021.

This patch merges cleanly.

The Scala stuff was mostly about the previous PR that got merged (and now no longer showing up as part of this diff).

So is this code you were referring to in HadoopRDD?
      // Find a function that will return the FileSystem bytes read by this thread. Do this before
      // creating RecordReader, because RecordReader's constructor might read some bytes
      val bytesReadCallback = inputMetrics.bytesReadCallback.orElse(
        split.inputSplit.value match {
          case split: FileSplit =>
            SparkHadoopUtil.get.getFSBytesReadOnThreadCallback(split.getPath, jobConf)
          case _ => None
        }
      )

Shows a stage that has Input Metrics (reading from a file) and writes data for next stage.

Shows a stage that has both shuffle reading and writing - no input or output metrics.

Shows a stage that has outputting to a file.

Test build #25638 timed out     for PR 4067 at commit 1572054     after a configured wait of 120m.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25638/
Test FAILed.

Test build #25641 has finished for   PR 4067 at commit 3c2d021.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AfterNextInterceptingIterator[A](sub: Iterator[A]) extends Iterator[A]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25641/
Test PASSed.

What about combining the input size and records in the same column. Overall this will help with the expansion in the number of columns. The title could be "Input Size / Records"

If we do that we wouldn't be able to sort on num records and bytes independently.

Yes - you'd only be able to sort on bytes. Wouldn't that be okay? These would likely track closely in most cases.

A big motivation to add recordsRead/Written was to detect data skew. In these cases bytes and records might not track very closely.
Thinking more about this, I suspect that having an Avg. record Size column (bytesRead/recordsRead) would be what you'd want to sort on. We could add this metric to the UI, make it sortable and then combine the bytesRead and recordsRead metrics into one column. Thoughts?

New Screenshot that correspond to CR feedback.

Shows a stage that has Input Metrics (reading from a file) and writes data for next stage.

Shows a stage that has both shuffle reading and writing - no input or output metrics.

Shows a stage that has outputting to a file.

@pwendell can you please re-review this? I'd like to get it in to 1.3. Some of our customers have been asking for metrics to help them determine data skew. Thx.

Test build #26441 has started for   PR 4067 at commit fe3f715.

This patch merges cleanly.

Test build #26441 has finished for   PR 4067 at commit fe3f715.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class AfterNextInterceptingIterator[A](sub: Iterator[A]) extends Iterator[A]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26441/
Test PASSed.

Having a count does seem like a good idea, but I think the current approach of adding function calls for each iteration is too expensive. What about using a counting iterator and just asking it at the end?

BTW we can also just add counting directly to InterruptibleIterator

@rxin yeah that seems good.
@ksakellis one other thing I realized which is a little confusing, right now we report the bytes as "read" from a cached RDD as soon as they are fetched/present on the executor even if they have not been consumed by the task. Tracking consumption incrementally (in bytes) will be really hard, so maybe this is the best answer for now. It's a bit weird though - I think this logic was written before we sent incremental updates back. In terms of getting the total records read, we might need to make the assumption that the iterator is consumed in its entirety.

@pwendell I'm not sure how we can do what you propose without having an O(n) loop through all the records before passing the InterruptableIterator? We could do something fancy like counting incrementally and when we finish the task, if there are more records left, then do the loop to count the rest of the unread records. I don't think the complication is worth it. Also, reporting the accurate records read i think is better.
Alternatively, we can fix the bytesRead to be more accurate. Right now they are computed in ShuffleBlockFetcherIterator and calculated based on the blocks fetched. Since we do the flatMap on that iterator in BlockStoreShuffleFetcher we report that we read all the bytes even if we didn't. We can move the bytesRead collection out of ShuffleBlockFetcherIterator and move it into the same iterator that computes the # records read. So they line up and are more accurate.

If it would save on perf, I would be in favor of adding the counting to InterruptibleIterator.  In core loops like these, I think efficiency is more important than elegance.

Test build #26530 has started for   PR 4067 at commit 1aaa980.

This patch merges cleanly.

Test build #26530 has finished for   PR 4067 at commit 1aaa980.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ChiSqSelectorModel (val selectedFeatures: Array[Int]) extends VectorTransformer
class ChiSqSelector (val numTopFeatures: Int)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26530/
Test PASSed.

Test build #26678 has started for   PR 4067 at commit 9ecf912.

This patch merges cleanly.

Test build #26678 has finished for   PR 4067 at commit 9ecf912.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class SimpleFunctionRegistry(val caseSensitive: Boolean) extends FunctionRegistry
class StringKeyHashMap[T](normalizer: (String) => String)
case class MultiAlias(child: Expression, names: Seq[String])

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26678/
Test FAILed.

Hey @ksakellis - I did a pretty thorough review here, any chance you could take a look? Most of the things were small. The only major thing is that I think if we are going to always show the output metrics for Hadoop (even for older versions) we need to make sure that we at least do a "best effort" presentation of the output size for older versions. I think it would be good if we could sneak that in, since otherwise that output metrics thing is not so useful for many people running with older versions. However, I'd also be okay just not showing either bytes or count in that case (which I think was the behavior before).
A second issue I realized (let's open a new JIRA for this) is that we never track output metrics (size or bytes) when data is written to cache, which is confusing because we track input metrics when data is read from cache.

New screenshots with the irrelevant columns invisible:

Shows a stage that has Input Metrics (reading from a file) and writes data for next stage.

Shows a stage that has both shuffle reading and writing - no input or output metrics.

Shows a stage outputting to an HDFS file.

Test build #26876 has started for   PR 4067 at commit e156560.

This patch merges cleanly.

Test build #26876 has finished for   PR 4067 at commit e156560.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26876/
Test FAILed.

Jenkins, retest this please

Test build #26896 has started for   PR 4067 at commit e156560.

This patch merges cleanly.

Test build #26896 has finished for   PR 4067 at commit e156560.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26896/
Test PASSed.

Test build #26904 has started for   PR 4067 at commit dad4d57.

This patch merges cleanly.

Test build #26906 has started for   PR 4067 at commit bd919be.

This patch merges cleanly.

Test build #26904 has finished for   PR 4067 at commit dad4d57.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26904/
Test PASSed.

Test build #26906 has finished for   PR 4067 at commit bd919be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26906/
Test PASSed.

Jenkins, test this please. This LGTM pending tests.

Test build #26936 has started for   PR 4067 at commit bd919be.

This patch merges cleanly.

Test build #26936 has finished for   PR 4067 at commit bd919be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26936/
Test PASSed.

Merging this, thanks Kos.

It looks like the "InputOutputMetricsSuite input metrics with mixed read methods" and "InputOutputMetricsSuite input metrics with interleaved reads" test may have started failing in the hadoop-2.2 build since this patch:
https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-1.3-SBT/AMPLAB_JENKINS_BUILD_PROFILE=hadoop2.2,label=centos/29/testReport/

Yikes, @JoshRosen i'm looking into this.

@ksakellis How do I collect these metrics on a console right after the task is done.

authors: AmplabJenkins JoshRosen SparkQA aguyyala ksakellis pwendell rxin sryza


text:
[SPARK-45655][SQL][SS] Allow non-deterministic expressions inside AggregateFunctions in CollectMetrics

What changes were proposed in this pull request?

This PR allows non-deterministic expressions wrapped inside an AggregateFunction such as count inside CollectMetrics node. CollectMetrics is used to collect arbitrary metrics from the query, in certain scenarios user would like to collect metrics for filtering based on non-deterministic expressions (see query example below).
Currently, Analyzer does not allow non-deterministic expressions inside a AggregateFunction for CollectMetrics. This constraint is relaxed to allow collection of such metrics. Note that the metrics are relevant for a completed batch, and can change if the batch is replayed (because non-deterministic expression can behave differently for different runs).
While working on this feature, I found a issue with checkMetric logic to validate non-deterministic expressions inside an AggregateExpression. An expression is determined as non-deterministic if any of its children is non-deterministic, hence we need to match the case for !e.deterministic && !seenAggregate after we have matched if the current expression is a AggregateExpression. If the current expression is a AggregateExpression, we should validate further down in the tree recursively - otherwise we will fail for any non-deterministic expression.

val inputData = MemoryStream[Timestamp]

inputData.toDF()
      .filter("value < current_date()")
      .observe("metrics", count(expr("value >= current_date()")).alias("dropped"))
      .writeStream
      .queryName("ts_metrics_test")
      .format("memory")
      .outputMode("append")
      .start()

Why are the changes needed?

Added a testcase to calculate dropped rows (by CurrentBatchTimestamp) and ensure the query is successful.

As an example, the query below fails (without this change) due to observe call on the DataFrame.

val inputData = MemoryStream[Timestamp]

inputData.toDF()
      .filter("value < current_date()")
      .observe("metrics", count(expr("value >= current_date()")).alias("dropped"))
      .writeStream
      .queryName("ts_metrics_test")
      .format("memory")
      .outputMode("append")
      .start()

Added testing in AnalysisSuite for non-deterministic expressions inside a AggregateFunction.

Does this PR introduce any user-facing change?

No
How was this patch tested?

Unit test cases added.

[warn] 20 warnings found
WARNING: Using incubator modules: jdk.incubator.vector, jdk.incubator.foreign
[info] StreamingQueryStatusAndProgressSuite:
09:14:39.684 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0
[info] No tests to run for hive / Test / testOnly
[info] - StreamingQueryProgress - prettyJson (436 milliseconds)
[info] - StreamingQueryProgress - json (3 milliseconds)
[info] - StreamingQueryProgress - toString (5 milliseconds)
[info] - StreamingQueryProgress - jsonString and fromJson (163 milliseconds)
[info] - StreamingQueryStatus - prettyJson (1 millisecond)
[info] - StreamingQueryStatus - json (1 millisecond)
[info] - StreamingQueryStatus - toString (2 milliseconds)
09:14:41.674 WARN org.apache.spark.sql.execution.streaming.ResolveWriteToStream: Temporary checkpoint location created which is deleted normally when the query didn't fail: /Users/bhuwan.sahni/workspace/spark/target/tmp/temporary-34d2749f-f4d0
-46d8-bc51-29da6411e1c5. If it's required to delete it under any circumstances, please set spark.sql.streaming.forceDeleteTempCheckpointLocation to true. Important to know deleting temp checkpoint folder is best effort.
09:14:41.710 WARN org.apache.spark.sql.execution.streaming.ResolveWriteToStream: spark.sql.adaptive.enabled is not supported in streaming DataFrames/Datasets and will be disabled.
[info] - progress classes should be Serializable (5 seconds, 552 milliseconds)
09:14:46.345 WARN org.apache.spark.sql.execution.streaming.ResolveWriteToStream: Temporary checkpoint location created which is deleted normally when the query didn't fail: /Users/bhuwan.sahni/workspace/spark/target/tmp/temporary-3a41d397-c3c1
-490b-9cc7-d775b0c42208. If it's required to delete it under any circumstances, please set spark.sql.streaming.forceDeleteTempCheckpointLocation to true. Important to know deleting temp checkpoint folder is best effort.
09:14:46.345 WARN org.apache.spark.sql.execution.streaming.ResolveWriteToStream: spark.sql.adaptive.enabled is not supported in streaming DataFrames/Datasets and will be disabled.
[info] - SPARK-19378: Continue reporting stateOp metrics even if there is no active trigger (1 second, 337 milliseconds)
09:14:47.677 WARN org.apache.spark.sql.execution.streaming.ResolveWriteToStream: spark.sql.adaptive.enabled is not supported in streaming DataFrames/Datasets and will be disabled.
[info] - SPARK-29973: Make `processedRowsPerSecond` calculated more accurately and meaningfully (455 milliseconds)
09:14:48.174 WARN org.apache.spark.sql.execution.streaming.ResolveWriteToStream: Temporary checkpoint location created which is deleted normally when the query didn't fail: /Users/bhuwan.sahni/workspace/spark/target/tmp/temporary-360fc3b9-a2c5
-430c-a892-c9869f1f8339. If it's required to delete it under any circumstances, please set spark.sql.streaming.forceDeleteTempCheckpointLocation to true. Important to know deleting temp checkpoint folder is best effort.
09:14:48.174 WARN org.apache.spark.sql.execution.streaming.ResolveWriteToStream: spark.sql.adaptive.enabled is not supported in streaming DataFrames/Datasets and will be disabled.
[info] - SPARK-45655: Use current batch timestamp in observe API (587 milliseconds)
09:14:48.768 WARN org.apache.spark.sql.streaming.StreamingQueryStatusAndProgressSuite:

Was this patch authored or co-authored using generative AI tooling?

No

FYI, the previous test failure was relevant to this change, and @sahnib is looking into it.

(I'm sorry to miss leaving a note in prior.)

Please check SPARK-45655: Use current batch timestamp in observe API *** FAILED *** (241 milliseconds)

Please check SPARK-45655: Use current batch timestamp in observe API *** FAILED *** (241 milliseconds)

This test was failing because CI machine has milliseconds precision of 8 decimal points, I have fixed the testcase.

Thanks! Merging to master.

@sahnib Thanks for your first contribution to Apache Spark! I merged this to master.

authors: HeartSaVioR beliefer sahnib


text:
[MINOR] Fix CodeStyle Error for ExpressionImplUtilsSuite

What changes were proposed in this pull request?
The pr is following up https://github.com/apache/spark/pull/46845, to fix  dev/scalastyle check error.
Why are the changes needed?
Make sh dev/scalastyle hadppy.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Manually test
sh dev/scakastyle

Before:
(base) ➜  spark-community git:(master) ✗ sh dev/scalastyle
-e Scalastyle checks failed at following occurrences:
[error] /Users/panbingkun/Developer/spark/spark-community/sql/catalyst/src/test/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtilsSuite.scala:394:28: nonascii.message
[error] /Users/panbingkun/Developer/spark/spark-community/sql/catalyst/src/test/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtilsSuite.scala:396:28: nonascii.message
[error] Total time: 23 s, completed Jun 26, 2024, 9:49:06 AM

After:
(base) ➜  spark-community git:(fix_scalastyle) ✗ sh dev/scalastyle
-e Scalastyle checks passed.

Was this patch authored or co-authored using generative AI tooling?
No.

cc @HyukjinKwon @cloud-fan @uros-db

let me merge it now to unblock the CI.

authors: cloud-fan panbingkun


text:
[SQL] Faster Scala row conversion

This is a follow-up of #5279 and #5398. ScalaReflection.convertRowToScala is on a critical path, but was implemented in a rather inefficient way.

Test build #29856 has started for   PR 5419 at commit d698d03.

Test build #29856 has finished for   PR 5419 at commit d698d03.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29856/
Test PASSed.

Actually the convertRowToScala part is already handled in #5279, and the lazy val schema part is handled in #5398, so I'm closing this.

authors: AmplabJenkins SparkQA liancheng


text:
[SPARK-11221] [SPARKR] fix R doc for lit and add examples

Currently the documentation for lit is inconsistent with doc format, references "Scala symbol" and has no example. Fixing that.
@shivaram

Can one of the admins verify this patch?

Jenkins, ok to test

Jenkins, add to whitelist

Merged build triggered.

Merged build started.

Test build #44013 has started for PR 9187 at commit 1ca6612.

Test build #44013 has finished for PR 9187 at commit 1ca6612.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * class BinaryClassificationEvaluator @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n  * class MulticlassClassificationEvaluator @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)\n  * final class RegressionEvaluator @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n  * abstract class ColumnarIterator extends Iterator[InternalRow] \n  *       class SpecificColumnarIterator extends $\n

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44013/
Test PASSed.

LGTM. Thanks @felixcheung  Merging this

authors: AmplabJenkins SparkQA felixcheung shivaram


text:
WIP: STORM-2648/STORM-2357: Add storm-kafka-client support for at-most-onc…

…e processing and a toggle for whether messages should be emitted with a message id when not using at-least-once
See https://issues.apache.org/jira/browse/STORM-2357 and https://issues.apache.org/jira/browse/STORM-2648.
I'd like to get some opinions on whether this approach is a good idea, or whether I've overlooked a better option, before finishing this up with some tests. I don't love that we'll end up with 3 different committing behaviors.
In 2357 it was noted that the spout doesn't currently support true at-most-once, because using Kafka's auto commit option leaves the possibility that the spout receives a tuple, emits it to the topology, crashes and recovers, and then receives and emits the same tuple. The linked issue suggests solving this by committing polled offsets before emitting them to the topology, which is an option added here.
2648 notes that there is currently no way to make Storm track messages when using auto commit with this spout. This prevents Storm UI from showing the complete latency for the spout, and I would assume also prevents max spout pending from having an effect. I've added a toggle to KafkaSpoutConfig to force the spout to emit messages with message ids, even when using auto commit or the at-most-once option. The spout does nothing on ack or fail when not doing at-least-once.
I'd like to keep the spout config simple for the user, so I've added a processing guarantee setting corresponding to the standard at-least-once code path, the path that uses auto commit, and the path that commits offsets before emitting any tuples.

@srdo can you please assign JIRA's to you and mark them as in progress as you work on them and/or submit a pull request. Thanks.

Yes, I forgot.

Conceptually the changes look good to me.  I have not dug into it in great detail yet, but I do like the direction of the change.
I would also like to see the documentation and examples updated to reflect the new change.

+1

@srdo reviewing it

Added some tests and updated the docs.

@hmcl Are you reviewing this, or are you satisfied with it?

Still +1

@hmcl I don't mean to try to rush you, but please let me know if you're still reviewing. If not I'll probably merge in the next few days.

@srdo apologies for the delay. I will finish today.

Thanks :)

@hmcl Are you still reviewing this? It's fine if you don't have time to look at this, but please say so. I'd like to not keep holding this up.

This PR has been waiting for about 2 months, and once it gets +1 and no -1, it can be merged.
I'm +1 and will just merge.
@hmcl Please vote -1 later and rollback the merge if you have concern about the patch and would want to vote -1.

Merged via https://github.com/apache/storm/commit/48f6969027e7b02a5b9220577189d3911aa2226d
@srdo
Sorry I forgot to add auto close message while squashing commits. Could you close this?
And please craft the patch for 1.x branch since it doesn't looks like a clean cherry-pick.
Thanks in advance!

Thanks for reviews. Opened the 1.x version here https://github.com/apache/storm/pull/2353.

authors: HeartSaVioR hmcl revans2 srdo

repo: apache/storm


text:
STORM-3071: change checkstyle plugin setting logViolationsToConsole t…

…o true

Logging checkstyle violations to console which is now possible since all violations have been fixed
Remove unnecessary maxAllowedViolations from configurations of projects

Note: The removal of maxAllowedViolations in the server project is covered in #3050 which needs to be merged before.

I took a look at #3050 . We can come back to this once the former's changes are merged.

It looks nice. Will come back to this once #3050 is merged

#3050 is now merged. Could you please rebase this so we can get this in to? Thanks again for the efforts

The failed test doesn't seem related
[INFO] Running org.apache.storm.daemon.drpc.DRPCTest
[ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.038 s <<< FAILURE! - in org.apache.storm.daemon.drpc.DRPCTest
[ERROR] testGoodBlocking  Time elapsed: 0.012 s  <<< ERROR!
java.util.concurrent.TimeoutException


text:
SUBMARINE-487. JNA does not work with maven-shade-plugin

What is this PR for?
submarine-spark-security fails with java.lang.UnsatisfiedLinkError
What type of PR is it?
[Bug Fix]
Todos

[ ] - Task

What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-487
How should this be tested?
Screenshots (if appropriate)
Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

@gumartinm Thank you for your interest in the submarine project. :-)
@yaooqinn Please help review this PR, Thanks!

Thanks for the fix, LGTM

@gumartinm feel free to send follow-ups with your no-uber-jar ideas, I use uber-jar here because of jar conflict between spark and ranger project.
@liuxunorg I am ok to merge this.

@gumartinm feel free to send follow-ups with your no-uber-jar ideas, I use uber-jar here because of jar conflict between spark and ranger project.
@liuxunorg I am ok to merge this.

@yaooqinn I just wanted to say that you are doing a great job with this project. I've been following it since it was called spark-authorizer. Hopefully, one day, it will be integrated in the Spark framework.

authors: gumartinm liuxunorg yaooqinn

repo: apache/submarine


text:
Determination of the select queries in BaseEngineSpec

Hello!
We are now developing a new SQLAlchemy dialect for Kusto (Azure Data Explorer) and want add support for Kusto database in Superset in nearly future. (This also covers https://github.com/apache/superset/issues/10646.)
Kusto language has two dialects: SQL and KQL. While there is no problem with SQL dialect, we also want to integrate KQL in Superset.
Problem
KQL select queries do not contain the word "SELECT" so they look like
MyTable | take 10 (this is equivalent of Select * from MyTable Limit 10).
The absence of the "SELECT" keyword breaks the limit behavior in SQLLab because it relies on the sqlparse package and we can't add a limit to queries from UI.
Proposed solution
The solution is to make the determination of the select queries in BaseEngineSpec so that you can override it in your own spec. This is the same way how base dialect determines read-only queries with is_readonly_query.
We will provide PR to implement this small feature.
Thank you!

Closed by https://github.com/apache/superset/pull/15013

source: gharchive/issue

repo: apache/superset


text:
Edit CSS Live can not @import google fonts?

Bug description
I try to @import google font in Edit CSS Live section. It not effect ???
@import url('https://fonts.googleapis.com/css2?family=Prompt:ital,wght@0,100;0,200;0,300;0,400;0,500;0,600;0,700;0,800;0,900;1,100;1,200;1,300;1,400;1,500;1,600;1,700;1,800;1,900&display=swap');
body{
font-family:'Prompt';
}
Screenshots/recordings
No response
Superset version
4.1.1
Python version
3.9
Node version
16
Browser
Chrome
Additional context
No response
Checklist

[ ] I have searched Superset docs and Slack and didn't find a solution to my problem.
[X] I have searched the GitHub issue tracker and didn't find a similar bug report.
[ ] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

You can, but dosubot is rigjt... you will have to punch a hole in TALISMAN_CONFIG for it.

Closing this since it's not a bug, but a question that's now been answered. Holler if you have any further troubles with it :)

authors: psongyoo rusackas


text:
refactor: Move is_user_admin() to security_manager

SUMMARY
Having is_user_admin is a design flaw, but in any
case should always be handled within the security
manager so it can be overridden.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS
Test admin user works as befre
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

@bolkedebruin are you still hoping to get this PR merged? There are some review comments that would need to be addressed to get this merged (and a rebase).

Yes, I'll take a look at it again.

authors: bolkedebruin villebro


text:
fix(install): set SUPERSET_VERSION_RC at the right time

fixes #21082
SUMMARY
Fixes problem as described in the issue #21082
TESTING INSTRUCTIONS

execute command below in a environment which doesn't contain any of the superset env variables.

[YOUR_SUPERSET_PATH]/RELEASING/make_tarball.sh [VERSION] [RC] [PGP_KEY_NAME]

this should run without failures.

ADDITIONAL INFORMATION
fixes #21082

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

@Joel-Haeberli  — Sorry this slipped though the cracks for so long! Thanks for reporting both the issue and opening the PR. Please feel free to tag committers here or ping us on Superset Slack if things get stuck like this.
@eschutho this seems like a straightforward fix to a problem I haven't encountered... curious your take :)

authors: Joel-Haeberli rusackas


text:
fix: tests on database, dataset apis

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 64.13%. Comparing base (2e5f3ed) to head (2a94fab).
Report is 57 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #28397      +/-   ##
==========================================
+ Coverage   60.49%   64.13%   +3.64%
==========================================
  Files        1931      521    -1410
  Lines       76241    37255   -38986
  Branches     8566        0    -8566
==========================================
- Hits        46122    23895   -22227
+ Misses      28015    13360   -14655
+ Partials     2104        0    -2104

Flag
Coverage Δ

hive
49.08% <ø> (-0.09%)
:arrow_down:

javascript
?

presto
53.66% <ø> (-0.14%)
:arrow_down:

python
64.13% <ø> (+0.65%)
:arrow_up:

unit
58.35% <ø> (+0.72%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter dpgaspar


text:
TOMEE-3871 - TomEE Plume is missing BatchEE / JCS Cache

batchee and jcs added in Plume to reflect Plus flavor
removed xml api from from Plume to reflect Plus flavor
remains some jar refs in jar scan exclusions (xerces, xml-apis)

AFAIK plume is a webprofile based distro so batchee and jcs should belong to a new "plume-all" distro probably?

AFAIK plume is a webprofile based distro so batchee and jcs should belong to a new "plume-all" distro probably?

This originates from this list thread and @dblevins comment there, that: "Those distributions are supposed to be the same minus the JPA and JSF providers.". There is also a blog post about the differences but I am too new to judge the original thoughts regarding "plume" vs "plus".

AFAIK plume is a webprofile based distro

if this is correct then Plume should not contains jars for MicroProfile nor JMS ActiveMQ

Yep, think they shouldnt but as Im not a big user of that I dont care much of what it becomes while webpro and plus stays aligned on their goal ;)

webpro and plus have a clear focus to me as well, even though "Plus" branding could be changed to better tell its "Full EE" 🤔 or i can just describe that in the flavors page.
You could drop Plume but you would lose the benefits from its "ready to try out" capabilities.
i am thinking about the AutoConfiguration for Jakarta Persistence (JPA) which facilitates learning Tomee.

I think it is good to have this discussion to clarify what the intended distributions should look like in 2022 considering the history of Plus / Plume and the incosistent current status of the distros / website. So thanks for the additional thoughts @rmannibucau and @sultan

does this need a mailing list vote from committers ?

Here's the list message on the name and motivation for Plume:

https://lists.apache.org/thread/tkpfjlzv70y8kzz5f905qj9vs95b34rw

"a version of TomEE Plus with Mojarra and EclipseLink. Tentatively called it "TomEE Plume" which stands for TomEE PLUs, Mojarra, Eclipselink (PLU.M.E)."

nice to read !

i learned that a setup installer for TomEE "like eclipse oomph" (cherry pick modules) is not possible, as it breaks TCK rules. unless we want a parallel non TCK certified TomEE (meh)
i thought like @rmannibucau at first with "per functionalities" LIB folders (like others EE servers) but this would make TomEE harder to configure / extend. (where do user put their jars !?)
there could be "per functionalities" uber or shaded jars to ease remove and replace implementations by others. but i wonder if i did read that current project policy is to avoid shading/uber.
there could be a common POM between plus and plume to help fill the gaps/ease maintenance.
there could be an external tool to modify a dezipped certified TomEE to change its implementation jars.

the only one i think i can currently help on is n°4 (common plus-plume pom)

there could be an external tool to modify a dezipped certified TomEE to change its implementation jars.

If you look tomee-maven-plugin it enables to do any distro from plain tomcat to advanced customized tomee and you can use it to build a tomee plume from tomee webprofile or any distro ;). Installer is legacy since years :)

@sultan Can you check your rebase with tomee-8.x. It does not look sane to me (at least judging from the changeset now).

i think i was able to clean up my mess by reverting to my previous draft. commits seems legit now. can someone launch checks ?

authors: dblevins rmannibucau rzo1 sultan

repo: apache/tomee


text:
Revert Traffic Monitor renaming to rascal

Which Traffic Control components are affected by this PR?

Documentation
Traffic Control Cache Config (t3c, formerly ORT)
Traffic Control Health Client (tc-health-client)
Traffic Control Client
Traffic Monitor
Traffic Ops
Traffic Portal
Traffic Router
Traffic Stats
Grove
CDN in a Box
Automation

What is the best way to verify this PR?

Run integration, unit test.
Ensure CIAB build correctly
TO API endpoints work as expected.
Clients (eg: T3C) work as expected.
If this is a bugfix, which Traffic Control versions contained the bug?

PR submission checklist

[x] This PR has tests
[x] This PR has documentation
[x] This PR has a CHANGELOG.md entry
[x] This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY (see the Apache Software Foundation's security guidelines for details)

This PR reverts #7068

repo: apache/trafficcontrol


text:
Still load legacy config if it exists instead of failing

Everything is broken and I'm not prepared for records.yaml

Thanks, @cmcfarlen - swapping from a format that's been in use for years to yaml necessitates a change of a ton of tooling.  The prior change would have been breaking for just about every org that uses ATS, with a significant investment in trying to change every part of build, validation, and config management.  Even if you believe that swapping to yaml would be better, you cannot expect that to be a minor change.

We have already talked about this on slack but just for the record:
As discussed(dev and user mailing list) before, we agreed to have single parsing and fail if the legacy file is found. As also requested,  a tool is provided for the translation with a guide on how to move on.
I have no issues with having it dual but just making sure that we are all on the same page.
If there is anything I can do to make the process easy just let me know and I'll do it.

I've created an issue which I will be implementing to track a solution for this.
https://github.com/apache/trafficserver/issues/9384

We talked about this in the PR/issue scrub and would like to have a discussion at next weeks PR/issue scrub.  A final decision should go to the mailing list.   Marking it as a draft, so it doesn't get merged in before we discuss.

I'm going to close this and we can work on the yaml support around tooling instead.

authors: DoingTheNeedful brbzull0 bryancall cmcfarlen

repo: apache/trafficserver


text:
[skip ci] Skip test_conv2d in Hexagon

https://github.com/apache/tvm/pull/10666 marked this xfail but it still runs the test causing segfault and CI failure in PR jobs and main.
@driazati @areusch @junrushao1994

But this test already has
https://github.com/apache/tvm/blob/871a83f9719d74d2eb15786d8b8878229b194c53/tests/python/contrib/test_hexagon/test_conv2d_conv2d.py#L165-L167
So I wonder why i386 test is still failing...

Existing
@pytest.mark.skipif(
         platform.processor() == "i686", reason="Test known to be flaky on i386 machines"
     )

doesn't seem to help actually skipping test. I'm trying if platform.processor() == "i386" does make things skip.

As the log https://ci.tlcpack.ai/blue/rest/organizations/jenkins/pipelines/tvm/branches/PR-10687/runs/7/nodes/315/steps/583/log/?start=0 shows, platform.machine() prints x86_64 despite being supposed to run on the i386 image. I don't know have a good way to conditionally skip this test on i386, so I'm going to skip it completely.

repo: apache/tvm


text:
[Fix][TIR] Fix tvm::arith::UnionLowerBound

The UnionLowerBound function does not take into account the condition that  the empty set has a special representation [+inf, -inf].
cc @wrongtest-intellif

cc @wrongtest-intellif

Could you please add a regression test?

We could add a testcase to tests/python/unittest/test_arith_intset.py::test_union_lower_bound

Could you please add a regression test?

Already added, thanks for your suggestion.

We could add a testcase to tests/python/unittest/test_arith_intset.py::test_union_lower_bound

Already added, thanks for your suggestion

@tvm-bot rerun

Please make the CI green and we can get it in :-)

Hi, CI is finished.

authors: Biubiubiu12 Hzfengsy wrongtest-intellif


text:
[YUNIKORN-2587] Core: Convert AllocationID to AllocationKey

What is this PR for?
Having both AllocationID and AllocationKey in the core is redundant. Replace all references to AllocationID with AllocationKey and remove the duplicate variable in Allocation.
What type of PR is it?

[ ] - Bug Fix
[ ] - Improvement
[ ] - Feature
[ ] - Documentation
[ ] - Hot Fix
[ ] - Refactoring

Todos

[x] - Task

What is the Jira issue?
https://issues.apache.org/jira/browse/YUNIKORN-2587
How should this be tested?
Unit tests updated as necessary.
Screenshots (if appropriate)
Questions:

[ ] - The licenses files need update.
[ ] - There is breaking changes for older versions.
[ ] - It needs documentation.

Codecov Report
Attention: Patch coverage is 79.16667% with 15 lines in your changes are missing coverage. Please review.

:exclamation: No coverage uploaded for pull request base (master@e17eafa). Click here to learn what that means.

Files
Patch %
Lines

pkg/scheduler/partition.go
75.75%
7 Missing and 1 partial :warning:

pkg/examples/simple_example.go
0.00%
2 Missing :warning:

pkg/scheduler/objects/application.go
83.33%
2 Missing :warning:

pkg/scheduler/objects/node.go
86.66%
2 Missing :warning:

pkg/scheduler/context.go
0.00%
1 Missing :warning:

Additional details and impacted files
@@            Coverage Diff            @@
##             master     #856   +/-   ##
=========================================
  Coverage          ?   77.02%
=========================================
  Files             ?       97
  Lines             ?    12002
  Branches          ?        0
=========================================
  Hits              ?     9244
  Misses            ?     2421
  Partials          ?      337

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter craigcondit

repo: apache/yunikorn-core


text:
[ZEPPELIN-1196] Fix for bug ZEPPELIN-1196

What is this PR for?
Fixed issue related to connecting to remote running interpreter process with multiple interpreters in interpreter group throws illegal thread state exception
What type of PR is it?
Bug Fix
Todos
What is the Jira issue?
[ZEPPELIN-1196] https://issues.apache.org/jira/browse/ZEPPELIN-1196
Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

One of the travis check is failing but its not related to my changes

Can someone please review this PR

@SachinJanani Hi, can you add a test case for it? It would help understand and test this PR.

@jongyoul Thanks for reviewing.As suggested I have added a test case for this PR

@SachinJanani Thanks for the quick fix, but if it fails to launch RemoteInterpreterServer with 10 seconds, this test always fails, isn't it? how to deal with that case? I think we need to make it more concrete. How about you?

@jongyoul  Yes you are right if RemoteInterpreterServer fails to launch within 10 seconds then this test will fail,but we need to have some threshold for that because if say there is some bug in RemoteInterpreterServer which causes it to hang then our test will never complete.Should I increase it to 20 seconds.What you think?

@jongyoul  One more thing I found that as per ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT  property default timeout that we consider is 30 seconds.So I think setting it to 30 seconds will make more sense.

@SachinJanani 30 seconds would be realistic. And could you please leave memo that it may occurs a potential bug when it fails launching it?

Thanks @jongyoul. Sure will make the change also will add a comment in the code about about the 30 seconds timeout

LGTM. Merging if there's no more discussion.

Can someone please merge this PR as I don't have permission

I'll merge it after 24 hours.

@SachinJanani I tried to merge it but this is based on branch-0.6. Could you please rebase from current master? I've tried to do it but I occurs error about merge conflict. Can you handle it? And for the next time, you'd better contribute the codes to avoid this situation.

@jongyoul This changes are only for branch-0.6 as master does not have this files (even this code) anymore due to helium changes.Also i think issue might not occur for zeppelin-0.7.So this is fix will only be for zeppelin-0.6.1.I think it should be only be merged in branch-0.6.

@jongyoul @SachinJanani I managed it to be merged only to branch-0.6. Will close this PR manually.

Thanks @minahlee

authors: SachinJanani jongyoul minahlee

repo: apache/zeppelin

license_source: bigquery


text:
Finished genReadOp() description. #20

Answered all the questions of part 1 of the assignment.
Should fix #20.

Codecov Report

Merging #43 into lab3 will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             lab3      #43   +/-   ##
=======================================
  Coverage   68.31%   68.31%
=======================================
  Files         107      107
  Lines        7334     7334
  Branches     1388     1388
=======================================
  Hits         5010     5010
  Misses       1868     1868
  Partials      456      456

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8c3c77...82b8b44. Read the comment docs.

Codecov Report

Merging #43 into lab3 will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             lab3      #43   +/-   ##
=======================================
  Coverage   68.31%   68.31%
=======================================
  Files         107      107
  Lines        7334     7334
  Branches     1388     1388
=======================================
  Hits         5010     5010
  Misses       1868     1868
  Partials      456      456

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8c3c77...82b8b44. Read the comment docs.

authors: codecov-io emilelmarsson

repo: apeinot/java

license: MIT


text:
Fix saving uuid as item title

This bug happens in some cases when hitting Save button the XML of an item is still not available .

Not sure this is the correct fix.  I mean, it may work, but I think the problem comes from the XML doc being out of date because the client script getState promise only runs once.
There is also still the problem of validation hooks not being unloaded, but that's a different issue :)

Not sure this is the correct fix. I mean, it may work, but I think the problem comes from the XML doc being out of date because the client script getState promise only runs once.
There is also still the problem of validation hooks not being unloaded, but that's a different issue :)

Yes, getState promise only runs once. So I try to get it running again inside the "presubmit". And then I found the real reason is the item's XML is not really ready when calling getState.
If you go to line 106 of WizardApi, you would see 'itemPack.getXml ' actually does not have the XML we need yet.

Something I would test is:

Put a repeater on the first page and add some repeater items.
Now remove a repeater item.
Now save + publish.
Check the item you removed is not in the XML

And:

Put some optional controls on page 2
Go to page 2 and fill in some values
Go back to page 1
Go back to page 2 and put in some new values
Now save + publish.
Check that the first set of values you put on page 2 are not in the XML.

Something I would test is:

Put a repeater on the first page and add some repeater items.
Now remove a repeater item.
Now save + publish.
Check the item you removed is not in the XML

And:

Put some optional controls on page 2
Go to page 2 and fill in some values
Go back to page 1
Go back to page 2 and put in some new values
Now save + publish.
Check that the first set of values you put on page 2 are not in the XML.

Thanks Aaron. Let me test these two cases now.

For the second test, looks good.

For the Repeater test :

authors: PenghaiZhang abidingotter

repo: apereo/openEQUELLA


text:
hysteria2 ERROR info in openwrt

1.system:
run hysteria2 in OpenWrt with transparent proxy
2.client config
{
"tls": {
"insecure": false,
"sni": "mysite.com"
},
"lazy": true,
"tcpTProxy": {
"listen": "0.0.0.0:1234"
},
"socks5":{
"listen": "0.0.0.0:1080"
},
"fast_open": true,
"bandwidth": {
"down": "300 mbps",
"up": "100 mbps"
},
"disable_mtu_discovery": false,
"auth": "Strong_password",
"server": "mysite.com:443"
}
3.ERROR info
2023-09-03T14:11:20Z    INFO    TCP transparent proxy listening {"addr": "0.0.0.0:1234"}
2023-09-03T14:11:23Z    INFO    connected to server     {"count": 1}
2023-09-03T14:11:23Z    ERROR   TCP transparent proxy error     {"addr": "123.96.20.227:40526", "reqAddr": "127.0.0.1:1234", "error":                                    "dial error: dial tcp4 127.0.0.1: 1234: connect: connection refused"}
2023-09-03T14:11:23Z    ERROR   TCP transparent proxy error     {"addr": "123.96.20.227:40534", "reqAddr": "127.0.0.1:1234", "error":                                    "dial error: dial tcp4 127.0.0.1: 1234: connect: connection refused"}
2023-09-03T14:11:23Z    ERROR   TCP transparent proxy error     {"addr": "123.96.20.227:40542", "reqAddr": "127.0.0.1:1234", "error":                                    "dial error: dial tcp4 127.0.0.1: 1234: connect: connection refused"}
2023-09-03T14:11:24Z    ERROR   TCP transparent proxy error     {"addr": "123.96.20.227:40558", "reqAddr": "127.0.0.1:1234", "error":                                    "dial error: dial tcp4 127.0.0.1: 1234: connect: connection refused"}

server's ACL causes the issue.solved

服务器的 ACL 导致该问题。已解决

你是怎么解决的？我也有同样的错误～

authors: AmberisMyShiba sjkhsl

repo: apernet/hysteria


text:
Not able to use toggleSeries with next js application

I want to toggleSeries data of a chart in my next js application. I am trying to use that method in 2 different ways, but none of them is working. I will show you the error I am getting with both these methods differently. Please help me with a solution.
Method 1:
import ApexCharts from 'apexcharts';

toggleSeries = (seriesName) => {
  ApexCharts.exec('myChartId', 'toggleSeries', 'seriesName');
}

In method 1 there are 2 problems
Most of the time I get window is not defined for ApexCharts
Second, sometimes I get: apexcharts__WEBPACK_IMPORTED_MODULE_9___default.a.exec is not a function
In order to resolve error: window is not defined I use dynamic import as I use dynamic import for react-apexcharts see below, even checking from typeof window !== undefined is not working here
Method 2:
const ApexCharts = dynamic(() => import('apexcharts'), { ssr: false });

toggleSeries = (seriesName) => {
  ApexCharts.exec('myChartId', 'toggleSeries', 'seriesName');
}

In method 2 it says ApexCharts.exec is not a function.
Please help me with the solution for this

Basically exec function is coming undefined
Can anybody help ??


text:
Does apexcharts support this kind of chart, see image in the link?

Does apexcharts support this kind of chart, see image in the link, I'm more interested on the top green label (+$6.31) (need to show the difference in amount between this year and last year in numbers)
https://imgur.com/a/moLi1Q4

No, ApexCharts directly doesn't support this functionality.
Although, you can use annotations and do the calculations to put such custom label anywhere on the chart.
https://apexcharts.com/javascript-chart-demos/line-charts/line-chart-annotations/
Docs

authors: junedchhipa visaru


text:
'sys' is not defined

In linux ubuntu I got the error showed in the image below:

Solution: I fixed it adding the following import line in the beginning of the ETL_coletar_dados_e_gravar_BD.py file:
import sys

Thank you @jeff-pal ! Solved by this commit: https://github.com/aphonsoar/Receita_Federal_do_Brasil_-_Dados_Publicos_CNPJ/commit/95fbb3bbb2d474053793caaa08dc4629d7c28871

authors: aphonsoar jeff-pal


text:
Remove stopDataFetcher

Closes https://github.com/api3dao/airseeker-v2/issues/129

If you remove the stop function you probably don't need the interval stored in state either, but again, super minor

Yeah, I removed that in some later PR but I could have done it here.


text:
Request headers not rendered

This is my code:
### Set/Update customer info [PUT /xxx/customer/{id}]
Update the personal info of the customer. None of the fields are mandatory. The fields not specified will not be overwritten in the database.

+ Parameters
    + id (string) - Id of the customer

+ Request

    + Headers

            Authentication: Bearer SomeTokenValue

    + Attributes (object)
        + email (string, optional) - email address of the customer
        + password (string, optional) - email address of the customer
        + firstName (string, optional) - first name
        + lastName (string, optional) - last name
        + phoneNumber (string, optional) - phone number
        + receiveOffers (number, optional) - 1 if the customer wants to receive offers, 0 otherwise

    + Body

            {
                "email":"example@domain.com",
                "firstName":"Jos",
                "lastName":"Bos",
                "phoneNumber":"+32488697059",
                "receiveOffers":1
            }

This is what the result looks like

Why is the request header not shown?

The request header is shown inside your code example (on line 7). If you don't want to view a code example you can switch to the default (Raw) to see the raw request information (headers and request body).

But why is it separate from the parameters and the attributes?
That makes no sense to me.
You can also say that the request parameters are shown to inside the code example, but doesn't it make more sense to have all variables grouped together? Why put url parameters at the top, but http headers in another section?

authors: ir-fuel kylef

repo: apiaryio/api-blueprint

license: mit


text:
Updated NPM_VERSION for nodejs 4 and 6

Please do not merge it yet. I want to properly test it in core app and update readme in coreapp.

@freaz fine by me; if you want to have both, you can add an extra folder to create a new tag (e.g. nodejs/6-npm4-4 or something less ugly...)

No I want have just one version, but the npm update was side effect and I wanted to test it at least locally.

authors: freaz nadade

repo: apiaryio/docker-base-images


text:
Report possible keywords

For example in
# Resource [/]
+ Model
    + Body (application/json)

            { ... }

The + Body (application/json) is not recognized as a body (content type should be defined on the model level) and thus the whole list item is considered to be model's description.

Migrated to apiaryio/drafter#645

authors: abtris zdne

repo: apiaryio/snowcrash