text:
how to add broker smoothly in a running dledger cluster?

hello guys,
I want to add a broker to my running dledger cluster,  and the cluster can't restart.  I try
to use mqadmin with updateBrokerConfig to update dLegerPeers, I can see dLegerPeers is already update to every broker,  but the new broker can't work, the logs show me  "WARN NettyServerPublicExecutor_1 - [BUG] [HandleVote] remoteId=n3 is an unknown member".
could anybody help me?

currently, it is not able to add broker to a dledger group dynamically.  You need to change the dledger configuration and restart all the brokers in the group.

@qq958905365 Could you describe the detailed use case? Why do need to add a broker to a running cluster?
In fact, I used to think this op is not practical.
It is better to fix the replica number during the deployment.

maybe i can try to do it

authors: dongeforever qq958905365 yiduwangkai

repo: apache/rocketmq


text:
[ISSUE #5605]Introduce tag estimation for lag calculation

Close #5605

Codecov Report

Merging #5606 (6f2bc17) into develop (08dbe7e) will increase coverage by 0.09%.
The diff coverage is 61.87%.

:exclamation: Current head 6f2bc17 differs from pull request most recent head c30e31b. Consider uploading reports for the commit c30e31b to get more accurate results

@@              Coverage Diff              @@
##             develop    #5606      +/-   ##
=============================================
+ Coverage      42.37%   42.47%   +0.09%
- Complexity      7929     7966      +37
=============================================
  Files           1022     1023       +1
  Lines          71254    71443     +189
  Branches        9399     9438      +39
=============================================
+ Hits           30196    30344     +148
- Misses         37230    37264      +34
- Partials        3828     3835       +7

Impacted Files
Coverage Δ

.../java/org/apache/rocketmq/common/BrokerConfig.java
28.01% <0.00%> (-0.14%)
:arrow_down:

...org/apache/rocketmq/store/DefaultMessageStore.java
52.76% <0.00%> (-0.66%)
:arrow_down:

...n/java/org/apache/rocketmq/store/MessageStore.java
0.00% <ø> (ø)

...test/listener/rmq/concurrent/RMQBlockListener.java
0.00% <0.00%> (ø)

...rg/apache/rocketmq/test/util/MQAdminTestUtils.java
0.00% <0.00%> (ø)

...apache/rocketmq/tools/admin/DefaultMQAdminExt.java
30.28% <0.00%> (-0.18%)
:arrow_down:

...he/rocketmq/tools/admin/DefaultMQAdminExtImpl.java
25.39% <0.00%> (-0.03%)
:arrow_down:

...n/java/org/apache/rocketmq/store/ConsumeQueue.java
69.50% <78.84%> (+0.94%)
:arrow_up:

...apache/rocketmq/store/queue/BatchConsumeQueue.java
70.21% <85.00%> (+1.79%)
:arrow_up:

...ava/org/apache/rocketmq/store/MappedFileQueue.java
62.25% <100.00%> (+1.14%)
:arrow_up:

... and 23 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

source: gharchive/pull-request

authors: ShadowySpirits codecov-commenter

license_type: permissive

license_source: github-api


text:
spark Panel not being recognized as an ILayoutParent

I'm seeing this stack trace when clicking on the OpenButton.
LayoutBase.js:241 Uncaught TypeError: Cannot read property 'getLayoutHost' of null
at spark.layouts.supportClasses.SparkLayoutBead.org.apache.royale.core.LayoutBase.performLayout (LayoutBase.js:241)
at spark.layouts.supportClasses.SparkLayoutBead.org.apache.royale.core.LayoutBase.handleInitComplete (LayoutBase.js:167)
at spark.components.Panel.org.apache.royale.events.EventDispatcher.fireListeners (EventDispatcher.js:97)
at Function.goog.events.EventTarget.dispatchEventInternal_ (eventtarget.js:381)
at spark.components.Panel.org.apache.royale.events.EventDispatcher.dispatchEvent (EventDispatcher.js:72)
at spark.components.Panel.org.apache.royale.core.ElementWrapper.dispatchEvent (ElementWrapper.js:229)
at spark.components.Panel.spark.components.SkinnableContainer.addedToParent (SkinnableContainer.js:138)
at mx.containers.beads.PanelInternalContainer.org.apache.royale.core.UIBase.addElement (UIBase.js:397)
at mx.containers.beads.PanelInternalContainer.mx.core.UIComponent.addElement (UIComponent.js:2424)
at mx.containers.beads.PanelInternalContainer.mx.core.Container.addElement (Container.js:301)
Originally posted by @yishayw in https://github.com/apache/royale-asjs/issues/1140#issuecomment-886236424

That's surprising because Panel.as extends SkinnableContainer
and
public class SkinnableContainer extends SkinnableContainerBase implements IContainer, IContainerBaseStrandChildrenHost, ILayoutParent
When debugging Language.is() I'm seeing leftOperand.constructor.superClass_  evaluate as an spark.components.supportClasses.SkinnableContainerBase (not a SkinnableContainer) I don't know if this is right.

That's another issue I only need blur background nothing else

The exception (assuming all source is up-to-date) was caught by my change in PR #1129 that was done to catch things like this on purpose.  The question is, what is the host in your performLayout()?
Note that on develop, SkinnableContainer is an ILayoutParent.  In PR #1135, I moved ILayoutParent to SkinnableContainerBase.
Also, I'm not getting any exception with develop + PR #1135 when I click on the button.

@estanglerbm PR #1135 looks great, but I don't have time to review it right now. The host is a spark panel. For now  (as I need to move on to other bugs) I think I'll explicitly mark Panel as an ILayoutParent implementer. We can analyze why Panel which extends SkinnableContainer isn't detected to implicitly be one later.

This was an issue in my local environment. Sorry for the noise.

source: gharchive/issue

authors: estanglerbm pashminakazi yishayw

repo: apache/royale-asjs


text:
[Improvement] jdbc split support customize function

Purpose of this pull request
close https://github.com/apache/seatunnel/issues/7293

Does this PR introduce any user-facing change?

How was this patch tested?

add new test
Check list

[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update change log that in connector document. For more details you can refer to connector-v2
Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist

[ ] Update the release-note.

Why not separate the dialect from the driver？
Jdbc {
    driver = com.mysql.cj.jdbc.Driver
    url = "jdbc:mysql://e2e_starRocksdb:9030"

    dialect = "starrocks"
  }

Why not separate the dialect from the driver？
cc @Hisoka-X
Jdbc {
    driver = com.mysql.cj.jdbc.Driver
    url = "jdbc:mysql://e2e_starRocksdb:9030"

    dialect = "starrocks"
}

Now the dialect is parse from jdbc url, mysql and starrocks both use mysql driver jdbc:mysql, so will use same mysql dialect.

Why not separate the dialect from the driver？
cc @Hisoka-X
Jdbc {
    driver = com.mysql.cj.jdbc.Driver
    url = "jdbc:mysql://e2e_starRocksdb:9030"

    dialect = "starrocks"
}

I prefer adding the split.hash-function parameter because it gives users more options.

We often use a certain client to access a variety of different database servers, which have many differences. Dialects can cover more differences.

We often use a certain client to access a variety of different database servers, which have many differences. Dialects can cover more differences.
e.g: mysql client -> doris,starrocks,ocenbase,... pg client -> ...

I get it. Make sense to me.

Why not separate the dialect from the driver？
cc @Hisoka-X
Jdbc {
    driver = com.mysql.cj.jdbc.Driver
    url = "jdbc:mysql://e2e_starRocksdb:9030"

    dialect = "starrocks"
}

updated, has add a parameter dialect.name, when choose dialect, will both check jdbc url and name.

authors: Hisoka-X hailin0 liunaijie

repo: apache/seatunnel


text:
org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
[✅ ] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?

Bug

Which version of SkyWalking, OS and JRE?
6.0.0GA for JDK1.8、CentOS7 64bit

Which company or project?

What happen?

I've 4 Server Application config the anget,but the Webpage is 0。
show me the Skywaking server log.
2019-01-30 16:00:42,699 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch.cache.ServiceInventoryCacheEsDAO - 86 [grpc-default-executor-0] ERROR [] - null java.lang.NullPointerException: null 2019-01-30 16:00:42,699 - org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2 - 111 [grpc-default-executor-0] ERROR [] - null java.lang.NullPointerException: null 	at org.apache.skywalking.oap.server.receiver.trace.provider.parser.standardization.SpanIdExchanger.exchange(SpanIdExchanger.java:119) ~[skywalking-trace-receiver-plugin-6.0.0-GA.jar:6.0.0-GA] 	at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2.preBuild(SegmentParseV2.java:146) ~[skywalking-trace-receiver-plugin-6.0.0-GA.jar:6.0.0-GA] 	at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2.parse(SegmentParseV2.java:89) [skywalking-trace-receiver-plugin-6.0.0-GA.jar:6.0.0-GA] 	at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2$Producer.send(SegmentParseV2.java:265) [skywalking-trace-receiver-plugin-6.0.0-GA.jar:6.0.0-GA] 	at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler$1.onNext(TraceSegmentReportServiceHandler.java:58) [skywalking-trace-receiver-plugin-6.0.0-GA.jar:6.0.0-GA] 	at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler$1.onNext(TraceSegmentReportServiceHandler.java:50) [skywalking-trace-receiver-plugin-6.0.0-GA.jar:6.0.0-GA] 	at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248) [grpc-stub-1.15.1.jar:1.15.1] 	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:263) [grpc-core-1.15.1.jar:1.15.1] 	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:683) [grpc-core-1.15.1.jar:1.15.1] 	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.15.1.jar:1.15.1] 	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.15.1.jar:1.15.1] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144] 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

Requirement or improvement

Please describe about your requirements or improvement suggestions.

I'm using 6.1.0, and still have this issue. :)
2019-06-26 21:08:58,569 - org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2 - 114 [grpc-default-executor-0] ERROR [] - null
java.lang.NullPointerException: null
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.standardization.SpanIdExchanger.exchange(SpanIdExchanger.java:119) ~[skywalking-trace-receiver-plugin-6.1.0.jar:6.1.0]
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2.preBuild(SegmentParseV2.java:149) ~[skywalking-trace-receiver-plugin-6.1.0.jar:6.1.0]
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2.parse(SegmentParseV2.java:92) [skywalking-trace-receiver-plugin-6.1.0.jar:6.1.0]
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2$Producer.send(SegmentParseV2.java:270) [skywalking-trace-receiver-plugin-6.1.0.jar:6.1.0]
at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler$1.onNext(TraceSegmentReportServiceHandler.java:58) [skywalking-trace-receiver-plugin-6.1.0.jar:6.1.0]
at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler$1.onNext(TraceSegmentReportServiceHandler.java:50) [skywalking-trace-receiver-plugin-6.1.0.jar:6.1.0]
at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248) [grpc-stub-1.15.1.jar:1.15.1]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:263) [grpc-core-1.15.1.jar:1.15.1]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:683) [grpc-core-1.15.1.jar:1.15.1]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.15.1.jar:1.15.1]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.15.1.jar:1.15.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

Again not a bug at all.

Again not a bug at all.

OK,  I will try to restart all agents. Thank you!

I've had this problem too, but I don't know which clients are using it. What should I do

I've had this problem too, but I don't know which clients are using it. What should I do

I think it is impossible to know, unless to open the logs(will be tons of logs, be careful). That is why the latest releases have agent hot reboot.

authors: stalary wu-sheng xiaour yijiuping

repo: apache/skywalking

license_source: bigquery


text:
Is there a health check interface for grpc server?

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
in kubernetes, liveness is a way to promise service normal,is there a health check interface to do it ?

Check the doc and CLI tool, it is there recently.

authors: EvanLjp wu-sheng


text:
[SPARK-13874][Doc]Remove docs of streaming-akka, streaming-zeromq, streaming-mqtt, streaming-flume and streaming-twitter

What changes were proposed in this pull request?
This PR removes all docs about the old streaming-akka, streaming-zeromq, streaming-mqtt, streaming-flume and streaming-twitter projects since I have already copied them to https://github.com/spark-packages
Also remove flume_wordcount.py and mqtt_wordcount.py that I forgot to remove previously.
How was this patch tested?
Jenkins PR Build.

Test build #53545 has started for PR 11824 at commit ebf35a3.

Test build #53545 has finished for PR 11824 at commit ebf35a3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53545/
Test PASSed.

cc @rxin

This will need to add back in the Flume docs, but otherwise OK.
We should also remove references in LICENSE, NOTICE, and licenses/. It looks like the only instance, however, that is relevant, is this in NOTICE:
     (Eclipse Public License - Version 1.0) mqtt-client (org.eclipse.paho:mqtt-client:0.4.0 - http://www.eclipse.org/paho/mqtt-client)

Bonus: while you're editing those, if you like, you can remove licenses/LICENSE-jblas.txt. I removed the last usages of it in master and looks like I forgot to remove this.
I wondered if we also needed to update our dependency files that enumerate the transitivie dependencies for various builds, but doesn't look like it.
python/docs/pyspark.streaming.rst still has a reference to MQTT docs.
Finally: at the very end of MimaBuild.scala you can remove the MQTT resolver, I think.

Test build #53980 has started for PR 11824 at commit dcdd3cb.

Test build #53980 has finished for PR 11824 at commit dcdd3cb.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53980/
Test FAILed.

Retest this please

Test build #54004 has started for PR 11824 at commit dcdd3cb.

Test build #54004 has finished for PR 11824 at commit dcdd3cb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54004/
Test FAILed.

LGTM pending tests. Let's try again

Jenkins retest this please

Test build #54027 has started for PR 11824 at commit dcdd3cb.

Test build #54027 has finished for PR 11824 at commit dcdd3cb.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54027/
Test FAILed.

retest this please

Test build #54056 has started for PR 11824 at commit dcdd3cb.

retest this please

Test build #54079 has started for PR 11824 at commit dcdd3cb.

Test build #54056 has finished for PR 11824 at commit dcdd3cb.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54056/
Test FAILed.

Test build #54079 has finished for PR 11824 at commit dcdd3cb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54079/
Test PASSed.

Merged build finished. Test PASSed.

retest this please

Test build #54244 has started for PR 11824 at commit dcdd3cb.

Test build #54244 has finished for PR 11824 at commit dcdd3cb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54244/
Test FAILed.

retest this please

Test build #54249 has started for PR 11824 at commit dcdd3cb.

Test build #54249 has finished for PR 11824 at commit dcdd3cb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54249/
Test FAILed.

retest this please

Test build #54255 has started for PR 11824 at commit dcdd3cb.

Test build #54255 has finished for PR 11824 at commit dcdd3cb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54255/
Test PASSed.

Merged build finished. Test PASSed.

Thanks - merging in master.

authors: AmplabJenkins SparkQA rxin srowen zsxwing

repo: apache/spark


text:
[SPARK-15665] [CORE] spark-submit --kill and --status are not working

What changes were proposed in this pull request?
--kill and --status were not considered while handling in OptionParser and due to that it was failing. Now handling the --kill and --status options as part of OptionParser.handle.
How was this patch tested?
I have verified these manually by running --kill and --status commands.

Can one of the admins verify this patch?

CC @vanzin

Jenkins test this please

Test build #59647 has started for PR 13407 at commit 62ebc7a.

Test build #59647 has finished for PR 13407 at commit 62ebc7a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59647/
Test PASSed.

LGTM but can you add unit tests so that these aren't inadvertently broken again?

Actually, looking at it again, I think the fix makes the code a little confusing. Why is appResource being used to store the argument to an option? I know it all aligns in the end, but it looks weird, and it might be possible to break this.
I think it would be better to keep another flag saying whether appResource is required or not, and only checking that (around L159) depending on the value of the flag. Then the existing cases (no args and help) could be changed to set that flag too.

or just add a special appResource like PYSPARK_SHELL or SPARK_SHELL

or just add a special appResource like PYSPARK_SHELL or SPARK_SHELL

If SparkSubmit can still process --kill and --status with those, then that's fine too (just use SparkLauncher.NO_RESOURCE).

@devaraj-kavali did you have a chance to look at the feedback?

Thanks @vanzin and @andrewor14 for looking into this, sorry for the delay.

If SparkSubmit can still process --kill and --status with those, then that's fine too (just use SparkLauncher.NO_RESOURCE).

I tried this but it doesn't work with the below error
[devaraj@server2 spark-master]$ ./bin/spark-submit --kill driver-20160531171222-0000
Error: Cannot load main class from JAR spark-internal with URI null. Please specify a class through --class.
Run with --help for usage help or --verbose for debug output

I have renamed the printInfo flag to isAppResourceReq and used the same for kill and status cases also.
Please review and let me know your feedback.

A couple of nits, but looks ok pending tests, which I have no idea why they haven't run. retest this please

Test build #59862 has started for PR 13407 at commit 30dfb0f.

Test build #59862 has finished for PR 13407 at commit 30dfb0f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59862/
Test FAILed.

I think these tests are flaky, but... retest this please

Test build #59875 has started for PR 13407 at commit 30dfb0f.

Test build #59875 has finished for PR 13407 at commit 30dfb0f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59875/
Test PASSed.

Merging to master / 2.0. If the names become an issue we can change later.

Thanks @vanzin for review and merging.

authors: AmplabJenkins SparkQA andrewor14 devaraj-kavali srowen vanzin


text:
[SPARK-20239][CORE][2.1-backport] Improve HistoryServer's ACL mechanism

Current SHS (Spark History Server) has two different ACLs:

ACL of base URL, it is controlled by "spark.acls.enabled" or "spark.ui.acls.enabled", and with this enabled, only user configured with "spark.admin.acls" (or group) or "spark.ui.view.acls" (or group), or the user who started SHS could list all the applications, otherwise none of them can be listed. This will also affect REST APIs which listing the summary of all apps and one app.
Per application ACL. This is controlled by "spark.history.ui.acls.enabled". With this enabled only history admin user and user/group who ran this app can access the details of this app.

With this two ACLs, we may encounter several unexpected behaviors:

if base URL's ACL (spark.acls.enable) is enabled but user A has no view permission. User "A" cannot see the app list but could still access details of it's own app.
if ACLs of base URL (spark.acls.enable) is disabled, then user "A" could download any application's event log, even it is not run by user "A".
The changes of Live UI's ACL will affect History UI's ACL which share the same conf file.

The unexpected behaviors is mainly because we have two different ACLs, ideally we should have only one to manage all.
So to improve SHS's ACL mechanism, here in this PR proposed to:

Disable "spark.acls.enable" and only use "spark.history.ui.acls.enable" for history server.
Check permission for event-log download REST API.

With this PR:

Admin user could see/download the list of all applications, as well as application details.
Normal user could see the list of all applications, but can only download and check the details of applications accessible to him.

New UTs are added, also verified in real cluster.
CC tgravescs vanzin please help to review, this PR changes the semantics you did previously. Thanks a lot.

Test build #76127 has started for PR 17755 at commit 2fc1525.

Test build #76127 has finished for PR 17755 at commit 2fc1525.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76127/
Test PASSed.

CC @vanzin , this backport can be merged to branch 2.0 cleanly.

Merging to 2.1 / 2.0. Please close the PR manually.

Thanks @vanzin .

authors: AmplabJenkins SparkQA jerryshao vanzin


text:
[SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to …

…base directory
What changes were proposed in this pull request?

run ./build/mvn -DskipTests clean package and we get many files 'dependency-reduced-pom.xml'
under the source directory

run ./build/mvn clean, the files dependency-reduced-pom.xml can not be cleaned.

So i suggest to specify dependencyReducedPomLocation to 'target/' to store these files.
How was this patch tested?
manual tests

Can one of the admins verify this patch?

I like the idea, but see this note: https://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html#dependencyReducedPomLocation . I'm not clear whether this has side-effects.

@srowen Thanks for your time. I am not sure the open issue, which is mentioned in the note,  whether has bean resolved.  as the note said

setting a value for this parameter with a directory other than ${basedir} will change the value of ${basedir}
,but after running mvn checkstyle:check , i get the right path of ${basedir}/target/checkstyle-output.xml

That's not the problem this would create -- it changes basedir after execution of the shade plugin, so it might affect things like the release build. Is it really a problem? gitignore already ignores these files, even though they're a little annoying.

I just  use mvn checkstyle:check ,which contains output files with basedir, to check whether the basedir is correct with this PR.
Could you please specify the affect exactly as you said? what is your mean about release build? or does it the output package is not right?

Yes this PR is a trivial Improvement for me. i will closed it if this has side-effects exactly. Thanks.

Meaning I think you'd have to run the release process to verify the artifacts still work, because it's the process that uses shading

authors: AmplabJenkins srowen zuotingbing


text:
[MINOR][ML]Updated english.txt word ordering

Ordered alphabetically, for better readability.
What changes were proposed in this pull request?
Alphabetical ordering of the stop words.

Can one of the admins verify this patch?

@animenon can you please fix the PR title like what other PR did. Also is this only for better readability or do you fix any other issue? IMO, I found that previous txt is more readable than your change, since they're ordered by different kind.

@jerryshao done. And browsing through the list is made easier with this ordering.
@srowen I wanted to search if few of the words were on the list and found it to be rather tough to find. It would be much easier to browse through this way.

Just ctrl-F and find?

Right, but if you want to check a few things together like "who", "who's", "why", "when", etc you will have to do many finds; but in an ordered document it would be much easier.

authors: AmplabJenkins animenon jerryshao srowen


text:
[WIP][SPARK-20629] Copy shuffle data when nodes are being shutdown

Why are the changes needed?
Recomputting shuffle blocks can be expensive, we should take advantage of our decommissioning time to migrate these blocks.
Does this PR introduce any user-facing change?
This PR introduces two new configs parameters, spark.storage.decommission.shuffle_blocks & spark.storage.decommission.rdd_blocks that control which blocks should be migrated during storage decommissioning.
How was this patch tested?
New unit test & expansion of the Spark on K8s decom test to assert that decommisioning with shuffle block migration means that the results are not recomputed even when the original executor is terminated.

Test build #121782 has started for PR 28331 at commit 4126c1b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26457/
Test PASSed.

Test build #121782 has finished for PR 28331 at commit 4126c1b.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class RDDBlockSavedOnDecommissionedBlockManagerException(blockId: RDDBlockId)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121782/
Test FAILed.

If you've got some cycles to take a look at the WIP PR @prakharjain09 I'd appreciate it. This builds on top of the cache block migration.
Why this PR is still a work in progress:

Only works with indexed shuffle files
I'm not sure if this is the best way to copy shuffle files between executors

Future work (not planned in this PR but in the future):

Supporting write back to some type of DFS
Updating the listener that tracks location of shuffle blocks for scale down to understand block migrations
Integrate decommissioning into our planned scale down, not just cluster manager/cloud triggered scale downs.

Test build #121784 has started for PR 28331 at commit 8ee8949.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26459/
Test PASSed.

Test build #121784 has finished for PR 28331 at commit 8ee8949.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121784/
Test FAILed.

Test build #121785 has started for PR 28331 at commit 4071ae2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26460/
Test PASSed.

Test build #121785 has finished for PR 28331 at commit 4071ae2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121785/
Test FAILed.

Jenkins retest this please
On Fri, Apr 24, 2020 at 6:11 PM UCB AMPLab notifications@github.com wrote:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121785/
Test FAILed.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/28331#issuecomment-619296594, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAOT5JBB2W37B3JVTA5KLDROI2DNANCNFSM4MQOCCXA
.

--
Cell : 425-233-8271

cc @dbtsai

Test build #122153 has started for PR 28331 at commit ff620ba.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26824/
Test PASSed.

Test build #122153 has finished for PR 28331 at commit ff620ba.

This patch fails from timeout after a configured wait of 400m.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122153/
Test FAILed.

Jenkins retest this please

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26850/
Test PASSed.

The original cache block migration is reverted, isn't it?

Test build #122178 has started for PR 28331 at commit ff620ba.

Oh yeah that's the conflicts, but I'm curious about the 400 minute timeout issue in Jenkins since I'm getting different results with the executor exiting locally.

Test build #122178 has finished for PR 28331 at commit ff620ba.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122178/
Test FAILed.

Test build #122181 has started for PR 28331 at commit adb03db.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26853/
Test PASSed.

Test build #122183 has started for PR 28331 at commit be2a5e7.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26855/
Test PASSed.

Test build #122181 has finished for PR 28331 at commit adb03db.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122181/
Test FAILed.

Test build #122183 has finished for PR 28331 at commit be2a5e7.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122183/
Test PASSed.

Test build #122183 has finished for PR 28331 at commit be2a5e7.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

@holdenk All tests started passing after this commit of yours. This includes all my changes also. Any specific change you know of in your branch which has fixed all the tests?

Jenkins retest this please.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26947/
Test PASSed.

Test build #122279 has started for PR 28331 at commit be2a5e7.

Test build #122279 has finished for PR 28331 at commit be2a5e7.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/122279/
Test FAILed.

Test build #123152 has started for PR 28331 at commit dbe2418.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27783/
Test PASSed.

Test build #123152 has finished for PR 28331 at commit dbe2418.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123152/
Test FAILed.

Test build #123154 has started for PR 28331 at commit a240f98.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27784/
Test PASSed.

Test build #123154 has finished for PR 28331 at commit a240f98.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123154/
Test FAILed.

There are two issues I can see.

Locally (at my side) the verify that an already running task which is going to cache data succeeds on a decommissioned executor always fails.

It fails with sbt and with maven too. Last time I tried with:
$ ./build/mvn clean install -DskipTests
$ ./build/mvn -pl :spark-core_2.12 test -Dtest=none '-Dsuites=*.BlockManagerDecommissionSuite'

The failure is:
Run starting. Expected test count is: 2
BlockManagerDecommissionSuite:
- verify that an already running task which is going to cache data succeeds on a decommissioned executor *** FAILED ***
  Vector(rdd_1_0) did not equal List() (BlockManagerDecommissionSuite.scala:141)
- verify that shuffle blocks are migrated.
Run completed in 26 seconds, 757 milliseconds.
Total number of tests run: 2
Suites: completed 1, aborted 0
Tests: succeeded 1, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***

The relevant log from core/target/unit-tests.log:
20/05/27 00:06:11.432 ScalaTest-main-running-BlockManagerDecommissionSuite INFO StandaloneSchedulerBackend: Propagating executor decommission to driver.
20/05/27 00:06:11.433 dispatcher-CoarseGrainedScheduler ERROR CoarseGrainedSchedulerBackend$DriverEndpoint: Received decommission executor message 1.
20/05/27 00:06:11.433 dispatcher-CoarseGrainedScheduler INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Starting decommissioning executor 1.
20/05/27 00:06:11.434 dispatcher-CoarseGrainedScheduler INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Finished decommissioning executor 1.
20/05/27 00:06:11.434 dispatcher-CoarseGrainedScheduler INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Starting decommissioning block manager corresponding to executor 1.
20/05/27 00:06:11.435 dispatcher-CoarseGrainedScheduler INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Acknowledged decommissioning block manager corresponding to 1.
20/05/27 00:06:11.922 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_0 in memory on 192.168.1.210:57044 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:11.927 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_1 in memory on 192.168.1.210:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:11.933 task-result-getter-2 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 10) in 610 ms on 192.168.1.210 (executor 1) (1/10)
20/05/27 00:06:11.940 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 12, 192.168.1.210, executor 0, partition 2, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:11.940 task-result-getter-3 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 11) in 616 ms on 192.168.1.210 (executor 0) (2/10)
20/05/27 00:06:12.454 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_2 in memory on 192.168.1.210:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:12.459 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 13, 192.168.1.210, executor 0, partition 3, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:12.460 task-result-getter-0 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 12) in 521 ms on 192.168.1.210 (executor 0) (3/10)
20/05/27 00:06:12.971 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_3 in memory on 192.168.1.210:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:12.977 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 4.0 in stage 1.0 (TID 14, 192.168.1.210, executor 0, partition 4, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:12.977 task-result-getter-1 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 13) in 518 ms on 192.168.1.210 (executor 0) (4/10)
20/05/27 00:06:13.489 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_4 in memory on 192.168.1.210:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:13.494 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 5.0 in stage 1.0 (TID 15, 192.168.1.210, executor 0, partition 5, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:13.495 task-result-getter-2 INFO TaskSetManager: Finished task 4.0 in stage 1.0 (TID 14) in 518 ms on 192.168.1.210 (executor 0) (5/10)
20/05/27 00:06:14.007 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_5 in memory on 192.168.1.210:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:14.012 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 6.0 in stage 1.0 (TID 16, 192.168.1.210, executor 0, partition 6, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:14.012 task-result-getter-3 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID 15) in 519 ms on 192.168.1.210 (executor 0) (6/10)
20/05/27 00:06:14.523 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_6 in memory on 192.168.1.210:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:14.527 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 7.0 in stage 1.0 (TID 17, 192.168.1.210, executor 0, partition 7, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:14.528 task-result-getter-0 INFO TaskSetManager: Finished task 6.0 in stage 1.0 (TID 16) in 517 ms on 192.168.1.210 (executor 0) (7/10)
20/05/27 00:06:15.040 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_7 in memory on 192.168.1.210:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:15.044 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 8.0 in stage 1.0 (TID 18, 192.168.1.210, executor 0, partition 8, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:15.045 task-result-getter-1 INFO TaskSetManager: Finished task 7.0 in stage 1.0 (TID 17) in 518 ms on 192.168.1.210 (executor 0) (8/10)
20/05/27 00:06:15.559 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_8 in memory on 192.168.1.210:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:15.564 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 9.0 in stage 1.0 (TID 19, 192.168.1.210, executor 0, partition 9, PROCESS_LOCAL, 7454 bytes) taskResourceAssignments Map()
20/05/27 00:06:15.564 task-result-getter-2 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID 18) in 520 ms on 192.168.1.210 (executor 0) (9/10)
20/05/27 00:06:16.079 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added rdd_1_9 in memory on 192.168.1.210:57048 (size: 56.0 B, free: 546.3 MiB)
20/05/27 00:06:16.084 task-result-getter-3 INFO TaskSetManager: Finished task 9.0 in stage 1.0 (TID 19) in 520 ms on 192.168.1.210 (executor 0) (10/10)
20/05/27 00:06:16.084 task-result-getter-3 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
20/05/27 00:06:16.085 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 1 (countAsync at BlockManagerDecommissionSuite.scala:92) finished in 4.771 s
20/05/27 00:06:16.085 dag-scheduler-event-loop INFO DAGScheduler: Job 1 is finished. Cancelling potential speculative or zombie tasks for this job
20/05/27 00:06:16.085 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 1: Stage finished
20/05/27 00:06:17.105 ScalaTest-main-running-BlockManagerDecommissionSuite INFO SparkContext: Starting job: count at BlockManagerDecommissionSuite.scala:134
20/05/27 00:06:17.106 dag-scheduler-event-loop INFO DAGScheduler: Got job 2 (count at BlockManagerDecommissionSuite.scala:134) with 10 output partitions
20/05/27 00:06:17.106 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 2 (count at BlockManagerDecommissionSuite.scala:134)
20/05/27 00:06:17.106 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List()
20/05/27 00:06:17.107 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List()
20/05/27 00:06:17.107 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[1] at mapPartitions at BlockManagerDecommissionSuite.scala:63), which has no missing parents
20/05/27 00:06:17.110 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.2 KiB, free 2.1 GiB)
20/05/27 00:06:17.111 dag-scheduler-event-loop INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 2.1 GiB)
20/05/27 00:06:17.112 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on apiros-mbp16.lan:57037 (size: 2.4 KiB, free: 2.1 GiB)
20/05/27 00:06:17.112 dag-scheduler-event-loop INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1293
20/05/27 00:06:17.113 dag-scheduler-event-loop INFO DAGScheduler: Submitting 10 missing tasks from ResultStage 2 (MapPartitionsRDD[1] at mapPartitions at BlockManagerDecommissionSuite.scala:63) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
20/05/27 00:06:17.113 dag-scheduler-event-loop INFO TaskSchedulerImpl: Adding task set 2.0 with 10 tasks resource profile 0
20/05/27 00:06:17.117 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 20, 192.168.1.210, executor 0, partition 1, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.129 dispatcher-BlockManagerMaster INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.1.210:57048 (size: 2.4 KiB, free: 546.3 MiB)
20/05/27 00:06:17.135 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID 21, 192.168.1.210, executor 0, partition 2, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.136 task-result-getter-0 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID 20) in 20 ms on 192.168.1.210 (executor 0) (1/10)
20/05/27 00:06:17.141 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 3.0 in stage 2.0 (TID 22, 192.168.1.210, executor 0, partition 3, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.142 task-result-getter-1 INFO TaskSetManager: Finished task 2.0 in stage 2.0 (TID 21) in 7 ms on 192.168.1.210 (executor 0) (2/10)
20/05/27 00:06:17.147 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 4.0 in stage 2.0 (TID 23, 192.168.1.210, executor 0, partition 4, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.147 task-result-getter-2 INFO TaskSetManager: Finished task 3.0 in stage 2.0 (TID 22) in 6 ms on 192.168.1.210 (executor 0) (3/10)
20/05/27 00:06:17.153 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 5.0 in stage 2.0 (TID 24, 192.168.1.210, executor 0, partition 5, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.153 task-result-getter-3 INFO TaskSetManager: Finished task 4.0 in stage 2.0 (TID 23) in 6 ms on 192.168.1.210 (executor 0) (4/10)
20/05/27 00:06:17.159 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 6.0 in stage 2.0 (TID 25, 192.168.1.210, executor 0, partition 6, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.159 task-result-getter-0 INFO TaskSetManager: Finished task 5.0 in stage 2.0 (TID 24) in 6 ms on 192.168.1.210 (executor 0) (5/10)
20/05/27 00:06:17.166 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 7.0 in stage 2.0 (TID 26, 192.168.1.210, executor 0, partition 7, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.166 task-result-getter-1 INFO TaskSetManager: Finished task 6.0 in stage 2.0 (TID 25) in 8 ms on 192.168.1.210 (executor 0) (6/10)
20/05/27 00:06:17.171 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 8.0 in stage 2.0 (TID 27, 192.168.1.210, executor 0, partition 8, PROCESS_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.171 task-result-getter-2 INFO TaskSetManager: Finished task 7.0 in stage 2.0 (TID 26) in 6 ms on 192.168.1.210 (executor 0) (7/10)
20/05/27 00:06:17.177 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 9.0 in stage 2.0 (TID 28, 192.168.1.210, executor 0, partition 9, PROCESS_LOCAL, 7454 bytes) taskResourceAssignments Map()
20/05/27 00:06:17.177 task-result-getter-3 INFO TaskSetManager: Finished task 8.0 in stage 2.0 (TID 27) in 6 ms on 192.168.1.210 (executor 0) (8/10)
20/05/27 00:06:17.183 task-result-getter-0 INFO TaskSetManager: Finished task 9.0 in stage 2.0 (TID 28) in 7 ms on 192.168.1.210 (executor 0) (9/10)
20/05/27 00:06:20.661 dispatcher-CoarseGrainedScheduler INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 29, 192.168.1.210, executor 0, partition 0, NODE_LOCAL, 7397 bytes) taskResourceAssignments Map()
20/05/27 00:06:20.679 task-result-getter-1 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 29) in 18 ms on 192.168.1.210 (executor 0) (10/10)
20/05/27 00:06:20.679 task-result-getter-1 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
20/05/27 00:06:20.680 dag-scheduler-event-loop INFO DAGScheduler: ResultStage 2 (count at BlockManagerDecommissionSuite.scala:134) finished in 3.572 s
20/05/27 00:06:20.680 dag-scheduler-event-loop INFO DAGScheduler: Job 2 is finished. Cancelling potential speculative or zombie tasks for this job
20/05/27 00:06:20.680 dag-scheduler-event-loop INFO TaskSchedulerImpl: Killing all running tasks in stage 2: Stage finished
20/05/27 00:06:20.680 ScalaTest-main-running-BlockManagerDecommissionSuite INFO DAGScheduler: Job 2 finished: count at BlockManagerDecommissionSuite.scala:134, took 3.575365 s
20/05/27 00:06:20.699 ScalaTest-main-running-BlockManagerDecommissionSuite INFO BlockManagerDecommissionSuite:

===== FINISHED o.a.s.storage.BlockManagerDecommissionSuite: 'verify that an already running task which is going to cache data succeeds on a decommissioned executor' =====

As I cannot see  a "Starting block manager decommissioning process..."  somewhere after the "Starting decommissioning block manager corresponding to executor 1" line I suspect it must be the asynchronous execution.
Here https://github.com/apache/spark/blob/a240f9857dcf53631b269422b4652a05304691be/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L320 we return with a Future  which might not be completed, see:
https://github.com/apache/spark/blob/a240f9857dcf53631b269422b4652a05304691be/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L160-L163

The 2nd problem is the test "verify that shuffle blocks are migrated" runs successfully even as ShuffleBlockBatchId  blocks are targeted for migration and not correct ShuffleDataBlockId blocks.
One reason could be the recompute kicks in. Although the completed task are counted and checked whether we have the expected number of successfully finished tasks. I need some time to find out what's going on here.

So it is not recalculation but the timing: the decommissioned executor is still giving back those shuffle data files:
You can check it by adding  a temporary log:
diff --git a/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala b/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
index b959e83599..ca45dc0905 100644
--- a/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
+++ b/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
@@ -85,9 +85,11 @@ private[spark] class IndexShuffleBlockResolver(
    */
    def getDataFile(shuffleId: Int, mapId: Long, dirs: Option[Array[String]]): File = {
     val blockId = ShuffleDataBlockId(shuffleId, mapId, NOOP_REDUCE_ID)
-    dirs
+    val file = dirs
       .map(ExecutorDiskUtils.getFile(_, blockManager.subDirsPerLocalDir, blockId.name))
       .getOrElse(blockManager.diskBlockManager.getFile(blockId))
+    logInfo(s"Attila: shuffle file: ${file.getAbsolutePath()}", new Exception())
+    file
   }

   /**

Then please remove the work directory content and execute this test only i.e. by sbt:
> testOnly *.BlockManagerDecommissionSuite -- -z "verify that shuffle blocks are migrated"
[warn] Multiple main classes detected.  Run 'show discoveredMainClasses' to see the list
[info] Packaging /Users/attilazsoltpiros/git/attilapiros/reviewing/core/target/scala-2.12/spark-core_2.12-3.1.0-SNAPSHOT.jar ...
[info] Done packaging.
[info] BlockManagerDecommissionSuite:
[info] - verify that shuffle blocks are migrated. (12 seconds, 968 milliseconds)
[info] ScalaTest
[info] Run completed in 15 seconds, 145 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 19 s, completed May 28, 2020 7:17:21 PM

You can check the logs in the work directory:
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/reviewing/work ‹pr/28331*›
╰─$ ag Attila
app-20200528101709-0000/0/target/unit-tests.log
69:20/05/28 19:17:14.044 Executor task launch worker for task 10 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/33/shuffle_0_10_0.data
86:20/05/28 19:17:14.363 Executor task launch worker for task 10 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/33/shuffle_0_10_0.data
106:20/05/28 19:17:14.886 Executor task launch worker for task 12 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/31/shuffle_0_12_0.data
123:20/05/28 19:17:14.894 Executor task launch worker for task 12 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/31/shuffle_0_12_0.data
143:20/05/28 19:17:15.409 Executor task launch worker for task 13 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/30/shuffle_0_13_0.data
160:20/05/28 19:17:15.416 Executor task launch worker for task 13 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/30/shuffle_0_13_0.data
180:20/05/28 19:17:15.933 Executor task launch worker for task 14 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/2f/shuffle_0_14_0.data
197:20/05/28 19:17:15.940 Executor task launch worker for task 14 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/2f/shuffle_0_14_0.data
217:20/05/28 19:17:16.457 Executor task launch worker for task 15 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0e/shuffle_0_15_0.data
234:20/05/28 19:17:16.463 Executor task launch worker for task 15 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0e/shuffle_0_15_0.data
254:20/05/28 19:17:16.975 Executor task launch worker for task 16 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/13/shuffle_0_16_0.data
271:20/05/28 19:17:16.981 Executor task launch worker for task 16 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/13/shuffle_0_16_0.data
291:20/05/28 19:17:17.497 Executor task launch worker for task 17 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0c/shuffle_0_17_0.data
308:20/05/28 19:17:17.502 Executor task launch worker for task 17 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0c/shuffle_0_17_0.data
328:20/05/28 19:17:18.017 Executor task launch worker for task 18 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/15/shuffle_0_18_0.data
345:20/05/28 19:17:18.022 Executor task launch worker for task 18 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/15/shuffle_0_18_0.data
365:20/05/28 19:17:18.538 Executor task launch worker for task 19 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0a/shuffle_0_19_0.data
382:20/05/28 19:17:18.543 Executor task launch worker for task 19 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0a/shuffle_0_19_0.data
412:20/05/28 19:17:19.177 Executor task launch worker for task 20 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0a/shuffle_0_19_0.data
437:20/05/28 19:17:19.204 Executor task launch worker for task 21 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/33/shuffle_0_10_0.data
467:20/05/28 19:17:19.248 Executor task launch worker for task 23 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/31/shuffle_0_12_0.data
492:20/05/28 19:17:19.259 Executor task launch worker for task 24 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/30/shuffle_0_13_0.data
517:20/05/28 19:17:19.271 Executor task launch worker for task 25 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/2f/shuffle_0_14_0.data
542:20/05/28 19:17:19.282 Executor task launch worker for task 26 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0e/shuffle_0_15_0.data
567:20/05/28 19:17:19.295 Executor task launch worker for task 27 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/13/shuffle_0_16_0.data
592:20/05/28 19:17:19.305 Executor task launch worker for task 28 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0c/shuffle_0_17_0.data
617:20/05/28 19:17:19.316 Executor task launch worker for task 29 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/15/shuffle_0_18_0.data
646:20/05/28 19:17:20.375 Executor task launch worker for task 30 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0a/shuffle_0_19_0.data
671:20/05/28 19:17:20.386 Executor task launch worker for task 31 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/33/shuffle_0_10_0.data
701:20/05/28 19:17:20.411 Executor task launch worker for task 33 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/31/shuffle_0_12_0.data
726:20/05/28 19:17:20.422 Executor task launch worker for task 34 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/30/shuffle_0_13_0.data
751:20/05/28 19:17:20.433 Executor task launch worker for task 35 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/2f/shuffle_0_14_0.data
776:20/05/28 19:17:20.443 Executor task launch worker for task 36 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0e/shuffle_0_15_0.data
801:20/05/28 19:17:20.454 Executor task launch worker for task 37 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/13/shuffle_0_16_0.data
826:20/05/28 19:17:20.466 Executor task launch worker for task 38 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/0c/shuffle_0_17_0.data
851:20/05/28 19:17:20.476 Executor task launch worker for task 39 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-4803c415-97f1-4429-87d7-537603fcce3b/blockmgr-add08797-7db9-4b18-9fff-76fb9a92cc72/15/shuffle_0_18_0.data

app-20200528101709-0000/1/target/unit-tests.log
97:20/05/28 19:17:14.044 Executor task launch worker for task 11 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
114:20/05/28 19:17:14.363 Executor task launch worker for task 11 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
132:20/05/28 19:17:19.231 shuffle-server-6-1 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
189:20/05/28 19:17:20.397 shuffle-server-6-1 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data

So from this you can see shuffle_0_11_0.data is only accessed in one of the executor. You can see this more easily by this:
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/reviewing/work ‹pr/28331*›
╰─$ ag shuffle_0_11_0.data
app-20200528101709-0000/1/target/unit-tests.log
97:20/05/28 19:17:14.044 Executor task launch worker for task 11 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
114:20/05/28 19:17:14.363 Executor task launch worker for task 11 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
132:20/05/28 19:17:19.231 shuffle-server-6-1 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
189:20/05/28 19:17:20.397 shuffle-server-6-1 INFO IndexShuffleBlockResolver: Attila: shuffle file: /Users/attilazsoltpiros/git/attilapiros/reviewing/target/tmp/spark-31f83c40-e25a-4247-8eee-d5fb7f2ec5f8/executor-f1e44135-db75-4b91-88c1-4f8819398e51/blockmgr-bb40b0c5-fd6e-45fd-8a2f-d3f3d38fd441/2e/shuffle_0_11_0.data
╭─attilazsoltpiros@apiros-MBP16 ~/git/attilapiros/reviewing/work ‹pr/28331*›

And you can check the call stack too for these accesses.
We should harden this test to produce failure when this happens. The test should fail even when the decommission is not triggered.
And I believe we need some tests at deeper level (this one might be too high: way above unit testing).
And It came to my mind how good it would be to have some metrics for the number of migrated blocks too (but is not a priority but something to think about).

Probably the easist thing to do is delete the executor in the K8s test after it's had a chance to migrate the blocks.

Test build #123244 has started for PR 28331 at commit 838a346.

Oh wait we already do that in the K8s integration test. So I think that's tested but I'll see what I can do to improve the unit tests as well.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27867/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27870/
Test PASSed.

Test build #123248 has started for PR 28331 at commit 2da0f2d.

Test build #123248 has finished for PR 28331 at commit 2da0f2d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123248/
Test FAILed.

Test build #123252 has started for PR 28331 at commit a92025c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27874/
Test PASSed.

I've verified the updated unit test fails when I manually set migrate shuffle blocks to false, thanks @attilapiros :)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27876/
Test PASSed.

Test build #123254 has started for PR 28331 at commit fe265d7.

Test build #123244 has finished for PR 28331 at commit 838a346.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123244/
Test PASSed.

Test build #123252 has finished for PR 28331 at commit a92025c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123252/
Test FAILed.

Test build #123254 has finished for PR 28331 at commit fe265d7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123254/
Test FAILed.

Test build #123259 has started for PR 28331 at commit 70c3871.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27881/
Test PASSed.

Test build #123259 has finished for PR 28331 at commit 70c3871.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123259/
Test PASSed.

Test build #123297 has started for PR 28331 at commit 069dd3b.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27919/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123297/
Test PASSed.

Test build #123301 has started for PR 28331 at commit 6340f9b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/27923/
Test PASSed.

Test build #123301 has finished for PR 28331 at commit 6340f9b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123301/
Test FAILed.

Test build #123394 has started for PR 28331 at commit 4cfeb8e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28013/
Test PASSed.

Test build #123396 has started for PR 28331 at commit e81aa5a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28014/
Test PASSed.

Test build #123394 has finished for PR 28331 at commit 4cfeb8e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123394/
Test FAILed.

Test build #123400 has started for PR 28331 at commit 155aeb2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28018/
Test PASSed.

Test build #123396 has finished for PR 28331 at commit e81aa5a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123396/
Test FAILed.

Test build #123407 has started for PR 28331 at commit 7f93df6.

If this passes the new tests I'm going to make a new non-WIP PR. Thank you for all your help reviewing @attilapiros the suggestions have really sped up my work here :)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28024/
Test PASSed.

Merged build finished. Test PASSed.

Test build #123409 has started for PR 28331 at commit 7e32341.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28025/
Test PASSed.

Test build #123400 has finished for PR 28331 at commit 155aeb2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123400/
Test PASSed.

Test build #123409 has finished for PR 28331 at commit 7e32341.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123409/
Test FAILed.

Test build #123407 has finished for PR 28331 at commit 7f93df6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123407/
Test PASSed.

Test build #123446 has started for PR 28331 at commit a3aa8eb.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28064/
Test PASSed.

I've made a non-WIP PR based on this, so closing this on for now.

Test build #123446 has finished for PR 28331 at commit a3aa8eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123446/
Test PASSed.

authors: AmplabJenkins SparkQA attilapiros dongjoon-hyun holdenk prakharjain09


text:
[WIP][SPARK-33600] Group exception messages in execution/datasources/v2

What changes were proposed in this pull request?
This PR groups exception messages in execution/datasources/v2.
Why are the changes needed?
It will largely help with standardization of error messages and its maintenance.
Does this PR introduce any user-facing change?
No. Error messages remain unchanged.
How was this patch tested?
No new tests - pass all original tests to make sure it doesn't break any existing behavior.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39948/

Test build #135368 has started for PR 31619 at commit edb529c.

Test build #135368 has finished for PR 31619 at commit edb529c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135368/

Test build #135370 has started for PR 31619 at commit d004f5c.

Test build #135370 has finished for PR 31619 at commit d004f5c.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135370/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39950/

Test build #135611 has started for PR 31619 at commit 21bfce8.

Test build #135611 has finished for PR 31619 at commit 21bfce8.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135611/

Test build #135613 has started for PR 31619 at commit 3c0463a.

Test build #135613 has finished for PR 31619 at commit 3c0463a.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40192/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135613/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40192/

Test build #135614 has started for PR 31619 at commit 862ce67.

Test build #135614 has finished for PR 31619 at commit 862ce67.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135614/

Test build #135615 has started for PR 31619 at commit 847ee74.

Test build #135615 has finished for PR 31619 at commit 847ee74.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135615/

Test build #135767 has started for PR 31619 at commit 36c0ada.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40350/

Test build #135767 has finished for PR 31619 at commit 36c0ada.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135767/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40350/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135770/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40352/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135786/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40368/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40368/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40373/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40368/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135791/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40373/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40373/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135951/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40536/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40536/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40562/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135978/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40562/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40562/

@cloud-fan, can you take another look?

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136178/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40760/

test this please

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136180/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40762/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40762/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40762/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136221/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40803/

@cloud-fan, can you take a look?

thanks, merging to master!

authors: AmplabJenkins SparkQA cloud-fan karenfeng shaneknapp


text:
[WIP][SPARK-36994][BUILD] Update Thrift to 0.15.0

What changes were proposed in this pull request?
This PR proposes to upgrade Thrift to 0.15.0.
Why are the changes needed?
To addresses CVEs:

Component Name
Component Version Name
Vulnerability
Fixed version

Apache Thrift
0.11.0-4.
CVE-2019-0205
0.13.0

Apache Thrift
0.11.0-4.
CVE-2019-0210
0.13.0

Apache Thrift
0.11.0-4.
CVE-2020-13949
0.14.1

Does this PR introduce any user-facing change?
I think no. CVE-2019-0205 and CVE-2019-0210 are more a bug fix.
CVE-2020-13949 might be valid.
How was this patch tested?
Existing tests should cover.

Test build #144228 has started for PR 34280 at commit 01d95b2.

Test build #144228 has finished for PR 34280 at commit 01d95b2.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144228/

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48708/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48708/

Test build #144347 has started for PR 34280 at commit de3f92f.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144347/

@wangyum, I just noticed that we have some diff in Thriftserver. Just to confirm, have you made some modifications to match with Spark's Thriftversion before?
e.g.) https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java#L241-L283 vs https://github.com/apache/hive/blob/rel/release-2.3.7/service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java#L240-L254

Test build #144351 has started for PR 34280 at commit a796548.

Test build #144351 has finished for PR 34280 at commit a796548.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48829/

Thanks @wangyum. Yeah, probably should better do that. Would you min taking a look when you find some time? I can give a shot too but that would take a while 😢

Thanks @wangyum. feel free to take this over!

authors: AmplabJenkins HyukjinKwon SparkQA


text:
[SPARK-37285] [ML] Add Weight of Evidence and Information value to ml.feature

JIRA Issue: https://issues.apache.org/jira/browse/SPARK-37285
The following shows the differences with this [PR-10803] :

More accurate algorithm to reduce the variance when the classification data skewed (all 1 or 0 in the bin of a feature) .
Programming based on RDD and better performance.
Separate IV and WOE modules to avoid coupling.

Can one of the admins verify this patch?


text:
[MINOR] Remove unused import

What changes were proposed in this pull request?
Remove unused import in numerics.
Why are the changes needed?
Cleanup
Does this PR introduce any user-facing change?
No
How was this patch tested?
N/A

@MaxGekk

Can one of the admins verify this patch?

+1, LGTM. Merged to master.
Thank you, @panbingkun.

authors: AmplabJenkins MaxGekk panbingkun


text:
[SPARK-39687][PYTHON][DOCS] Make sure new catalog methods listed in API reference

What changes were proposed in this pull request?
1, add new methods to catalog.rst;
2, follow sphinx synctax AnalysisException -> :class:`AnalysisException
Why are the changes needed?
Make sure new catalog methods listed in API reference
Does this PR introduce any user-facing change?
No, docs only
How was this patch tested?
manually doc build and check, such as

Merged to master.

Thank you @HyukjinKwon  for remindering me to check the doc and helping view

authors: HyukjinKwon zhengruifeng


text:
[SPARK-4973][CORE] Local directory in the driver of client-mode continues remaining even if application finished when external shuffle is enabled

When we enables external shuffle service, local directories in the driver of client-mode continue remaining even if application has finished.
I think local directories for drivers should be deleted.

Test build #25233 has started for   PR 3811 at commit 88feecd.

This patch merges cleanly.

Test build #25233 has finished for   PR 3811 at commit 88feecd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25233/
Test FAILed.

Why does it matter whether it's client mode or cluster mode? I'd say it should always clean it if it's the driver (which doesn't get killed).

@andrewor14 Ah, exactly. The driver is not killed until the end of the application. I'll fix my previous changes.

Test build #25253 has started for   PR 3811 at commit ad944ab.

This patch merges cleanly.

LGTM merging into master when tests pass

Test build #25253 has finished for   PR 3811 at commit ad944ab.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25253/
Test FAILed.

retest this please

Test build #25257 has started for   PR 3811 at commit ad944ab.

This patch merges cleanly.

Test build #25257 has finished for   PR 3811 at commit ad944ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25257/
Test PASSed.

authors: AmplabJenkins SparkQA andrewor14 sarutak


text:
[SPARK-44957][PYTHON][SQL][TESTS] Make PySpark (pyspark-sql module) tests passing without any dependency

What changes were proposed in this pull request?
This PR proposes to fix the tests to properly run or skip when there aren't optional dependencies installed.
Why are the changes needed?
Currently, it fails as below:
Running PySpark tests. Output is in /.../spark/python/unit-tests.log
Will test against the following Python executables: ['python3']
Will test the following Python modules: ['pyspark-sql']
python3 python_implementation is CPython
python3 version is: Python 3.10.12
Starting test(python3): pyspark.sql.tests.pandas.test_pandas_grouped_map_with_state (temp output: /.../spark/python/target/8e530108-4d5e-46e4-88fb-8f0dfb7b47e2/python3__pyspark.sql.tests.pandas.test_pandas_grouped_map_with_state__jggatex7.log)
Starting test(python3): pyspark.sql.tests.pandas.test_pandas_grouped_map (temp output: /.../spark/python/target/3b6e9e5a-c479-408c-9365-8286330e8e7c/python3__pyspark.sql.tests.pandas.test_pandas_grouped_map__1lrovmur.log)
Starting test(python3): pyspark.sql.tests.pandas.test_pandas_cogrouped_map (temp output: /.../spark/python/target/68c7cf56-ed7a-453e-8d6d-3a0eb519d997/python3__pyspark.sql.tests.pandas.test_pandas_cogrouped_map__sw2875dr.log)
Starting test(python3): pyspark.sql.tests.pandas.test_pandas_map (temp output: /.../spark/python/target/90712186-a104-4491-ae0d-2b5ab973991b/python3__pyspark.sql.tests.pandas.test_pandas_map__ysp4911q.log)
Traceback (most recent call last):
  File "/.../miniconda3/envs/vanilla-3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/.../miniconda3/envs/vanilla-3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/.../workspace/forked/spark/python/pyspark/sql/tests/pandas/test_pandas_map.py", line 27, in <module>
    from pyspark.testing.sqlutils import (
  File "/.../workspace/forked/spark/python/pyspark/testing/__init__.py", line 19, in <module>
    from pyspark.testing.pandasutils import assertPandasOnSparkEqual
  File "/.../workspace/forked/spark/python/pyspark/testing/pandasutils.py", line 22, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'

PySpark tests should pass without optional dependencies.
Does this PR introduce any user-facing change?
No, test-only.
How was this patch tested?
Manually ran as described above.
Was this patch authored or co-authored using generative AI tooling?
No.

Merged to master and branch-3.5 (as it fixes testing framework too)


text:
[SPARK-46698][CORE] Replace Timer with single thread scheduled executor for ConsoleProgressBar.

What changes were proposed in this pull request?
This PR propose to replace Timer with single thread scheduled executor for ConsoleProgressBar.
Why are the changes needed?
The javadoc recommends ScheduledThreadPoolExecutor instead of Timer.

This change based on the following two points.
System time sensitivity
Timer scheduling is based on the absolute time of the operating system and is sensitive to the operating system's time. Once the operating system's time changes, Timer scheduling is no longer precise.
The scheduled Thread Pool Executor scheduling is based on relative time and is not affected by changes in operating system time.
Are anomalies captured
Timer does not capture exceptions thrown by Timer Tasks, and in addition, Timer is single threaded. Once a scheduling task encounters an exception, the entire thread will terminate and other tasks that need to be scheduled will no longer be executed.
The scheduled Thread Pool Executor implements scheduling functions based on a thread pool. After a task throws an exception, other tasks can still execute normally.
Does this PR introduce any user-facing change?
'No'.
How was this patch tested?
Manual tests.
Was this patch authored or co-authored using generative AI tooling?
'No'.

ping @srowen @dongjoon-hyun @LuciferYang

Looks OK, just rerun tests

Merged to master. I verified manually the console progress bar. It looks working without any issue as expected.

@dongjoon-hyun @srowen @LuciferYang Thank you!

authors: beliefer dongjoon-hyun srowen


text:
[SPARK-7964][SQL] remove unnecessary type coercion rule

We have defined these logics in Cast already, I think we should remove this rule.

Merged build triggered.

Merged build started.

Test build #33796 has started for   PR 6516 at commit d5035a4.

LGTM.

Can you create a JIRA ticket for this?

Test build #33796 has finished for   PR 6516 at commit d5035a4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

OK I'm merging this. If @marmbrus comments later, we can revert it.

If I remember correctly, the original idea here was that allowing Cast from numeric types to boolean was kind of weird so we made it a rule instead of part of the Cast operator itself.  I guess subsequent refactorings have moved this logic into Cast and that seems okay to me (It looks like there is a test for this, right?).
Thank you for creating a JIRA though.  We should reserve [minor] for doc changes or things that definitely can't cause regressions.  This seems safe but is not super easy to reason about.

authors: AmplabJenkins SparkQA cloud-fan marmbrus rxin