text:
The TransactionMQProducer When check set state ROLLBACK_MESSAGE ，the execute should not COMMIT_MESSAGE again！

When I  execute this code，I can see the halfmq on RocketMQ-Console，actually，the check was priority set ‘LocalTransactionState.ROLLBACK_MESSAGE’ ，then the executeLocalTransaction back ‘LocalTransactionState.COMMIT_MESSAGE’。Strictly speaking，this mq message was rollback， but was not the case！！
`public static void main(String[] args) throws MQClientException {
String topic = "topic_test";
TransactionMQProducer producer = new TransactionMQProducer("test_group");
producer.setNamesrvAddr("localhost:9876");
producer.setTransactionListener(new TransactionListener() {
@Override
public LocalTransactionState executeLocalTransaction(Message msg, Object arg) {
System.out.println(" execute local trans: begin ");
doSomeThing(65);
System.out.println(" execute local trans: over ");
return LocalTransactionState.COMMIT_MESSAGE;
}
        @Override
        public LocalTransactionState checkLocalTransaction(MessageExt msg) {
            System.out.println(" check: begin ");
            doSomeThing(30);
            System.out.println(" check: over ");
            return LocalTransactionState.ROLLBACK_MESSAGE;
        }

        private void doSomeThing(long second) {
            try {
                TimeUnit.SECONDS.sleep(second);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    });
    producer.start();

    try {
        String msgbody = "mq-half-message";
        Message message = new Message(topic, msgbody.getBytes());
        message.setKeys("mq-key");
        TransactionSendResult send = producer.sendMessageInTransaction(message, null);
        System.out.println(send);
    } catch (Exception e) {
        e.printStackTrace();
    }
}`

The transaction message review first executes executeLocalTransaction, then there is no commit, and then checkLocalTransaction will be executed all the time

The transaction message review first executes executeLocalTransaction, then there is no commit, and then checkLocalTransaction will be executed all the time

The half message was send ，when executes executeLocalTransaction blocked，the checkLocalTransaction begin to execute and  set the half message would be rollback，how can executeLocalTransaction change the state? I can not understand it。
It's like a  variable happens ABA problems;

when the method executeLocalTransaction return COMMIT, method checkLocalTransaction will be not executed.
when the method executeLocalTransaction return UNKNOW, method checkLocalTransaction will be executed until the result is returned。
when the method executeLocalTransaction blocking, from the perspective of Broker, the status of this Half status message is unknown. Therefore, the Broker will periodically ask the sender to check the Half status message and report its final status.

The problem still exists
version：4.7.1
config：transientStorePoolEnable=false

authors: ZhangHLong fengyiluo hellodake panzhi33

repo: apache/rocketmq


text:
使用acl功能，关于版本的选择

从这里了解到https://github.com/apache/rocketmq/issues/3922 同时自己也做了测试，
使用最新稳定版4.9.3的acl功能，每次编辑plain_acl.yml文件后需要重启broker，
这样的话，是否就不能使用该版本，或者有其他规避方法吗？

@lsw21 https://github.com/apache/rocketmq/pull/3927 这个PR 针对这个问题进行了一些修复，帮忙一起再重新review一下，新版本即将发布，可以先用develop分支本地打包试下，欢迎反馈

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
Build failure: 0.96

The following problem was fixed by removing -SNAPSHOT from royale.framework.version in examples¥Pom.xml.
—————————————————
I run "mvn clean install" at C:\apache-royale-0.9.6-bin-js\royale-asjs\examples on Windows 10 with Japanese.  Build failed with the following console.
This is the first time build of Apache Royale examples on this machine.  I cannot fine the subject WAR file in the designated link.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  19:06 min
[INFO] Finished at: 2019-10-27T14:01:01+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:2.10:unpack (unpack-fonts) on project DataBindingExample_Flat: Unable to find artifact.: Could not find artifact org.apache.royale.framework:Fonts:war:0.9.6-SNAPSHOT in apache-snapshots (https://repository.apache.org/content/repositories/snapshots)
[ERROR]
[ERROR] Try downloading the file manually from the project website.
[ERROR]
[ERROR] Then, install it using the command:
[ERROR]     mvn install:install-file -DgroupId=org.apache.royale.framework -DartifactId=Fonts -Dversion=0.9.6-SNAPSHOT -Dpackaging=war -Dfile=/path/to/file

Hi,
to build examples, first you need to build in your local machine the three repos of Royale in order:
1.- compiler
2.- typedefs
3.- framework (royale-asjs)
have you perform that three steps?
then you'll have in your .m2 the latest snapshot (currently 0.9.7-SNAPSHOT) and you can go to examples dir an run mvn clean install to build all examples.

Carlos, that’s mvn build. What I’m asking now is how to compile application
tutorial’s mxml with project.json using mxmlc.  Tutorial does not mention
where to put project.json.  Then, I’m asking it.
2019年10月28日(月) 17:56 Carlos Rovira notifications@github.com:

Hi,
to build examples, first you need to build in your local machine the three
repos of Royale in order:
1.- compiler
2.- typedefs
3.- framework (royale-asjs)
have you perform that three steps?
then you'll have in your .m2 the latest snapshot (currently
0.9.7-SNAPSHOT) and you can go to examples dir an run mvn clean install
to build all examples.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/royale-asjs/issues/530?email_source=notifications&email_token=AESPMHW3NBDOLNRWTSXU5J3QQ2SK7A5CNFSM4JFUOBRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECMETII#issuecomment-546851233,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AESPMHXDCAYRYUBWNVPPMSLQQ2SK7ANCNFSM4JFUOBRA
.

--
Shoichiro Takeshita
武下 祥一郎

Hi Takeshita, but that refers to the other issue I just responded right?
Anyway for this maven issue, I just read you conversation with @aharui . I'm just building all, to test nothing is broken, but I'm not on windows, I'm on Mac, so maybe there's still an issue in Windows for what I read of that conversation.
thanks

Carols, this is different than Maven issue.  It is solved.  This is a new
question re Application Tutorial build explained in the Royale homepage.
2019年10月28日(月) 18:03 Carlos Rovira notifications@github.com:

Hi Takeshita, but that refers to the other issue I just responded right?
Anyway for this maven issue, I just read you conversation with @aharui
https://github.com/aharui . I'm just building all, to test nothing is
broken, but I'm not on windows, I'm on Mac, so maybe there's still an issue
in Windows for what I read of that conversation.
thanks
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/royale-asjs/issues/530?email_source=notifications&email_token=AESPMHT5BRPPSUZ5A3GRWZDQQ2TG7A5CNFSM4JFUOBRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECMFHQA#issuecomment-546853824,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AESPMHW37PE5B3NJCDS7JZTQQ2TG7ANCNFSM4JFUOBRA
.

--
Shoichiro Takeshita
武下 祥一郎

Carlos,
It is not needed. All stuff is available online on main Apache, so those
example should be buildable without building anything else by using -s
settings-template.xml or even without that.
Thanks,
Piotr
pon., 28 paź 2019 o 09:56 Carlos Rovira notifications@github.com
napisał(a):

Hi,
to build examples, first you need to build in your local machine the three
repos of Royale in order:
1.- compiler
2.- typedefs
3.- framework (royale-asjs)
have you perform that three steps?
then you'll have in your .m2 the latest snapshot (currently
0.9.7-SNAPSHOT) and you can go to examples dir an run mvn clean install
to build all examples.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/royale-asjs/issues/530?email_source=notifications&email_token=AF3K2K4BJJTAGT74P5GJZ7TQQ2SK5A5CNFSM4JFUOBRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECMETII#issuecomment-546851233,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AF3K2K6A2ZOM2ONL5274OR3QQ2SK5ANCNFSM4JFUOBRA
.

--
Piotr Zarzycki
Patreon: https://www.patreon.com/piotrzarzycki
https://www.patreon.com/piotrzarzycki

Hi,
to build examples, first you need to build in your local machine the three repos of Royale in order:
1.- compiler
2.- typedefs
3.- framework (royale-asjs)
have you perform that three steps?
then you'll have in your .m2 the latest snapshot (currently 0.9.7-SNAPSHOT) and you can go to examples dir an run mvn clean install to build all examples.

@carlosrovira
It is not needed. All stuff is available online on main Apache, so those example should be buildable without building anything else by using -s settings-template.xml or even without that.

Hi,
project.json is explained here.
https://apache.github.io/royale-docs/create-an-application/application-tutorial/data-model
Compile command is here, but no project.json specified.
https://apache.github.io/royale-docs/create-an-application/application-tutorial/build
On Mon, Oct 28, 2019 at 8:17 PM Piotr Zarzycki notifications@github.com
wrote:

Hi,
to build examples, first you need to build in your local machine the three
repos of Royale in order:
1.- compiler
2.- typedefs
3.- framework (royale-asjs)
have you perform that three steps?
then you'll have in your .m2 the latest snapshot (currently
0.9.7-SNAPSHOT) and you can go to examples dir an run mvn clean install
to build all examples.
@carlosrovira https://github.com/carlosrovira
It is not needed. All stuff is available online on main Apache, so those
example should be buildable without building anything else by using -s
settings-template.xml or even without that.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/royale-asjs/issues/530?email_source=notifications&email_token=AESPMHXDUTAPX55YKXWLZ4TQQ3C33A5CNFSM4JFUOBRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECMQZ3Y#issuecomment-546901231,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AESPMHQTI4GJFC6EWVV3NHDQQ3C33ANCNFSM4JFUOBRA
.

--
Shoichiro Takeshita
武下 祥一郎

The instructions are not very clear on how to build. I think the easiest path for the tutorial is using asconfig.json to configure the project. @cottage14 thoughts?

I will take a look over the next day or so. We are much involved with
closing up the house of a dear friend who died just at Christmas.
On Wed, Dec 29, 2021 at 6:56 PM Harbs @.***> wrote:

The instructions are not very clear on how to build. I think the easiest
path for the tutorial is using asconfig.json to configure the project.
@cottage14 https://github.com/cottage14 thoughts?
—
Reply to this email directly, view it on GitHub
https://github.com/apache/royale-asjs/issues/530#issuecomment-1002804668,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AC3RFXOLHP3RNX7UB5VBRS3UTOGZJANCNFSM4JFUOBRA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID:
@.***>

--
Andrew Wetmore
Editor, Moose House Publications https://moosehousepress.com
Editor-Writer, The Apache Software Foundation https://apache.org

No worries. So sorry to hear! 😢

repo: apache/royale-asjs


text:
Misc. Util cleanup

Major changes:

Broke up 'Util' class into multiple classes: 'FileUtil', 'HttpUtil', 'CoordinatorStreamUtil'.
Consolidated some Util classes: MathUtil, ScalaJavaUtil
Removed redundant Util classes: ClassloaderUtil, ScalaToJavaUtil
Renamed some Util classes for consistency: TimerUtils -> TimerUtil.
Inlined some util classes and methods to where they're used: LexicographicComparator to RocksDBKeyValueStore, defaultSerdeFactoryFromSerdeName to SerdeManager, etc.

Rest of the changes are updates to use the new classes and methods.
Testing: Local build and test works. Tested with a locally deployed Samza job.

@vjagadish1989 @jmakes

FYI @xinyuiscool @sborya @nickpan47.

@prateekm thanks for the patch. Just noticed that there is a conflict with master now. Could you help to update this patch against the master? Thanks!

@nickpan47 rebased from master. Will commit by EOD if everything looks good.

source: gharchive/pull-request

repo: apache/samza

license_source: bigquery


text:
[Feature][JDBC Sink] Add DM upset support

Purpose of this pull request

Check list

[ ] Code changed are covered with tests, or it does not need tests for reason:
[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update change log that in connector document. For more details you can refer to connector-v2
Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist

[ ] Update the release-note.

Please add e2e for this pr.

done

repo: apache/seatunnel


text:
[type:fix] fix e2e sync time

Make sure that:

[x] You have read the contribution guidelines.
[x] You submit test cases (unit or integration tests) that back your changes.
[ ] Your local test passed ./mvnw clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #5040 (8474afb) into master (b612264) will decrease coverage by 0.02%.
Report is 1 commits behind head on master.
The diff coverage is n/a.

:exclamation: Current head 8474afb differs from pull request most recent head 86134af. Consider uploading reports for the commit 86134af to get more accurate results

@@             Coverage Diff              @@
##             master    #5040      +/-   ##
============================================
- Coverage     64.23%   64.21%   -0.02%
  Complexity     8575     8575
============================================
  Files          1209     1209
  Lines         35909    35909
  Branches       3221     3221
============================================
- Hits          23066    23059       -7
- Misses        10984    10994      +10
+ Partials       1859     1856       -3

see 5 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter moremind

repo: apache/shenyu


text:
[bug]batchnorm running mean var not updated

Hi team,
Shouldn't autograd batchnorm  class update its running_mean and running_var from c++ backend? I could not find anywhere running_mean/var are updated.

I think this is pass by reference (the running mean and variance singa tensors' blocks are fed to the function cudnnBatchNormalizationForwardTraining, so there is no need to return the running mean and variance singa tensors (because the _BatchNorm2d already have them)?

Oh I see. The running mean and var are updated using NOT in place assignment.
running_mean = running_mean * bnh.factor + mean * (1 - bnh.factor);
If we change it to in place operation, there is no need to return, i.e.
running_mean*=bnh.factor;
Axpy(1-bnh.factor,mean,running_mean)

Thank you @chrishkchris
I tested and currently in python the running mean is not updated after GpuBatchNormForwardTraining returns.
Another thing to note is that, SWIG has this apply OUTPUT which might solve this. http://www.swig.org/Doc1.3/Arguments.html
I looking into this.

Yes, I see why the running mean is not updated. This is because
running_mean = running_mean * bnh.factor + mean * (1 - bnh.factor);
It generates a new tensor to be stored in the variable running mean, since it is NOT in place operation. this is why the  original running mean is not updated
So if we use in place operation the Original running mean tensor can be updated.

Hi @chrishkchris thank you for the help. This is false alarm. the code is working as @chrishkchris described.

authors: chrishkchris dcslin

repo: apache/singa


text:
hotfix: cpp softmax not impl

moved softmax(..., axis) implementation one level up, from cuda to tensor. After the change, softmax(..., axis) implemenation could serve for both cuda and cpp. Tested is added accordingly.

Hi @chrishkchris , thank you for pointing this out. could you please help on testing at your side?

thanks. I will test soon

@dcslin  I tested the examples/autograd/mlp.py (multilayer perception) and there is such error:
ubuntu@ip-172-31-26-47:~/singa/examples/autograd$ python3 mlp.py
train_data_shape: (400, 2)
train_label_shape: (400, 2)
training loss =  6.682101
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0113 09:20:05.180658 12032 tensor_math_cpp.h:357] Check failed: a > 0.f (-nan vs. 0)
*** Check failure stack trace: ***
Aborted (core dumped)

However if I uses softmax_cross_entropy instead of softmax + cross_entropy it will be totally okay, i.e.
loss = autograd.softmax_cross_entropy(x, target)
The good output will be:
ubuntu@ip-172-31-26-47:~/singa/examples/autograd$ python3 mlp.py
train_data_shape: (400, 2)
train_label_shape: (400, 2)
training loss =  0.6908062
training loss =  0.5960194
training loss =  0.57797414
training loss =  0.55334115
training loss =  0.48568404
training loss =  0.38458923
training loss =  0.30776194
training loss =  0.24188559
training loss =  0.18657134
training loss =  0.15864176
training loss =  0.13929243

@dcslin  Another observation: in mnist_cnn.py, if I change softmax_cross_entropy instead of softmax + cross_entropy, it fails no matter what the axis is selected for SoftMax
ubuntu@ip-172-31-26-47:~/singa/examples/autograd$ python3 mnist_cnn.py
Starting Epoch 0:
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0113 09:57:47.470569 14016 tensor_math_cpp.h:357] Check failed: a > 0.f (0 vs. 0)
*** Check failure stack trace: ***
Aborted (core dumped)

@dcslin
I used a simple test code to compare with pytorch when the input vector is of the size [1, 5].
The output is the same as pytorch. I suppose the difference maybe from cross_entropy.

singa
ubuntu@ip-172-31-26-47:~$ python3 softmaxtest.py
x = [1. 3. 5. 7. 9.]
dy = [1. 3. 5. 7. 9.]
y = [[2.9007587e-04 2.1433870e-03 1.5837606e-02 1.1702495e-01 8.6470395e-01]]
dx = [[-0.00222993 -0.01219034 -0.05839987 -0.19747007  0.27029037]]

pytorch
(pytorch_p36) ubuntu@ip-172-31-16-63:~/pytorch_ops_test$ python softmaxtest.py
x = [[1. 3. 5. 7. 9.]]
dy = [[1. 3. 5. 7. 9.]]
y = [2.9007587e-04 2.1433870e-03 1.5837606e-02 1.1702495e-01 8.6470395e-01]
dx = [-0.00222993 -0.01219034 -0.05839989 -0.19747011  0.27029008]

To sum up, this PR fixed the cpp SoftMax not implemented error. After checking the result with pytorch, it is okay.
So this fix is ready for merge.
Some future work maybe to check cross_entropy

Hi @chrishkchris , thank you the fix on mlp.py


text:
Got an  Exception while fetching data (/globalPercentile)

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
[x] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?

Bug

Which version of SkyWalking, OS and JRE?
skywalking 7.0.0   elasticsearch  6.8.6

Which company or project?

What happened?
{"errors":[{"message":"Exception while fetching data (/globalPercentile) : Metrics:all_percentile doesn't have value column definition"}]}
oapServer log:
2020-08-25 18:40:47,453 - org.elasticsearch.client.RestClient - 65 [I/O dispatcher 1] WARN  [] - request [HEAD http://10.82.26.123:9200/_template/elasticsearch_instance_clr_gen0_collect_count] returned 1 warnings: [299 Elasticsearch-6.8.6-3d9f765 "[types removal] The parameter include_type_name should be explicitly specified in get template requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
2020-08-25 18:40:47,453 - org.apache.skywalking.oap.server.core.storage.model.ModelInstaller - 66 [main] INFO  [] - table: instance_clr_gen0_collect_count_hour does not exist
2020-08-25 18:40:47,453 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.StorageEsInstaller - 71 [main] INFO  [] - index elasticsearch_instance_clr_gen0_collect_count's columnTypeEsMapping builder str: {type={properties={service_id={type=integer}, time_bucket={type=long}, entity_id={type=keyword}, value={type=long}}}}
2020-08-25 18:40:47,457 - org.elasticsearch.client.RestClient - 65 [I/O dispatcher 2] WARN  [] - request [HEAD http://10.82.26.130:9200/_template/elasticsearch_instance_clr_gen0_collect_count] returned 1 warnings: [299 Elasticsearch-6.8.6-3d9f765 "[types removal] The parameter include_type_name should be explicitly specified in get template requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]

Requirement or improvement

Please describe your requirements or improvement suggestions.

This seems your OAP doesn't start up correctly.

This seems your OAP doesn't start up correctly.

need i use the  script 'oapServiceNoInit.sh'  to start server?

authors: kanouken wu-sheng

repo: apache/skywalking


text:
call statistics of interface

Please answer these questions before submitting your issue.

Why do you submit this issue?
[x] Question or discussion
[ ] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?
display the call statistics of interface

Which company or project?
spring-mvc

What happened?
in sykwalking-ui cannot display the call statistics of interface , just only display interface trace.

Requirement or improvement

Please describe your requirements or improvement suggestions.

Again, this kind of description means nothing.
This is open source community, not commercial customer service.
You said nothing showed, so?


text:
[Feature]  Support export to a file（eg: csv） for log query

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
Current  on the  log query page, query results cannot be exported to a file.  the user experience is not very good. Considering save the search results to a CSV file for downloading.
Use case
No response
Related issues
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

We are welcome you to contribute, but the feature request is usually self assign, meaning someone will submit PR later.
Close for now, as you are not going to do.


text:
Fix #2546

Please answer these questions before submitting pull request

Why submit this pull request?

[x] Bug fix

[ ] New feature provided

[ ] Improve performance

Related issues
#2546

Coverage remained the same at 16.011% when pulling 12b6da9c51fedafd5a3ff4098b298a107c17ba1e on ascrutae:fix/spring-plugin-issue into f02300a945bd4670f36db00ca7e15ff7442d277e on apache:master.

authors: ascrutae coveralls


text:
Add missing test for component-libraries.yml

Motivation:
Add test to guarantee that the components defined in class ComponentsDefine are in sync with those in file component-libraries.yml.
Modifications:

Add missing component in component-libraries.yml.

Add test to verify the class and the yml are in sync.

Fix wrong component id of Undertow.

Result:
Components definitions in class ComponentsDefine and yaml are in sync, and will be verified in every pull request.

Related issues are: #3859 #3241 #2679 #2122 , so many

Codecov Report

Merging #4398 into master will decrease coverage by 0.09%.
The diff coverage is 0%.

@@            Coverage Diff            @@
##           master    #4398     +/-   ##
=========================================
- Coverage   25.65%   25.55%   -0.1%
=========================================
  Files        1209     1209
  Lines       27914    27914
  Branches     3851     3851
=========================================
- Hits         7160     7134     -26
- Misses      20103    20129     +26
  Partials      651      651

Impacted Files
Coverage Δ

.../apm/network/trace/component/ComponentsDefine.java
0% <0%> (ø)
:arrow_up:

...apm/agent/core/remote/GRPCStreamServiceStatus.java
41.17% <0%> (-41.18%)
:arrow_down:

...ache/skywalking/apm/agent/core/jvm/JVMService.java
60% <0%> (-20%)
:arrow_down:

...alking/apm/agent/core/remote/AgentIDDecorator.java
66.66% <0%> (-18.52%)
:arrow_down:

...m/agent/core/remote/TraceSegmentServiceClient.java
78.08% <0%> (-1.37%)
:arrow_down:

...king/apm/agent/core/remote/GRPCChannelManager.java
67.94% <0%> (-1.29%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 67b7f22...82af5a9. Read the comment docs.

Well, I just forgot to update the component id in plugin test 😢


text:
[SPARK-15632][SQL]Typed Filter should NOT change the Dataset schema

What changes were proposed in this pull request?
This PR makes sure the typed Filter doesn't change the Dataset schema.
Before the change:
scala> val df = spark.range(0,9)
scala> df.schema
res12: org.apache.spark.sql.types.StructType = StructType(StructField(id,LongType,false))
scala> val afterFilter = df.filter(_=>true)
scala> afterFilter.schema   // !!! schema is CHANGED!!! Column name is changed from id to value, nullable is changed from false to true.
res13: org.apache.spark.sql.types.StructType = StructType(StructField(value,LongType,true))

SerializeFromObject and DeserializeToObject are inserted to wrap the Filter, and these two can possibly change the schema of Dataset.
After the change:
scala> afterFilter.schema   // schema is NOT changed.
res47: org.apache.spark.sql.types.StructType = StructType(StructField(id,LongType,false))

How was this patch tested?
Unit test.

Test build #60062 has started for PR 13529 at commit 00a873e.

Test build #60062 has finished for PR 13529 at commit 00a873e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60062/
Test FAILed.

we can also remove the EmbedSerializerInFilter and its related tests

maybe ignore them for now? useful as reference when we add those optimizations.

@rxin By "ignore" did you mean remove the rule from the batch but keep the code lines of the rule?

I was referring to the tests.

Test build #60085 has started for PR 13529 at commit 9bebd69.

Test build #60087 has started for PR 13529 at commit 5e899f6.

LGTM except for a minor comment, and pending Jenkins.

@cloud-fan I think we can also disable EmbedSerializerInFilter now?

I think we should just remove EmbedSerializerInFilter, as we need a totally different optimization strategy for the new typed filter.

@liancheng @cloud-fan
About EmbedSerializerInFilter, I have noted in source code:
TODO: Remove this after we completely fixes SPARK-15632 by adding optimization rules For typed filters.
We can remove EmbedSerializerInFilter in follow up PR when we finished the optimizers for typed filter.

Test build #60091 has started for PR 13529 at commit 1281131.

Test build #60085 has finished for PR 13529 at commit 9bebd69.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60085/
Test FAILed.

Test build #60087 has finished for PR 13529 at commit 5e899f6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60087/
Test PASSed.

Test build #60091 has finished for PR 13529 at commit 1281131.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60091/
Test PASSed.

Merging to master and branch-2.0. Thanks!

authors: AmplabJenkins SparkQA clockfly cloud-fan liancheng rxin

repo: apache/spark


text:
[SPARK-16052][SQL] Add CollapseRepartitionBy optimizer

What changes were proposed in this pull request?
This issue adds a new optimizer, CollapseRepartitionBy, which is similar with CollapseRepartition. Also, this PR adds a testsuite for both CollapseRepartition and CollapseRepartitionBy.
Before
scala> spark.range(10).repartition(1, $"id").repartition(1, $"id").explain
== Physical Plan ==
Exchange hashpartitioning(id#0L, 1)
+- Exchange hashpartitioning(id#0L, 1)
   +- *Range (0, 10, splits=8)

After
scala> spark.range(10).repartition(1, $"id").repartition(1, $"id").explain
== Physical Plan ==
Exchange hashpartitioning(id#0L, 1)
+- *Range (0, 10, splits=8)

How was this patch tested?
Pass the Jenkins tests (including a new testsuite).

Test build #60790 has started for PR 13765 at commit caeb08f.

Test build #60790 has finished for PR 13765 at commit caeb08f.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60790/
Test FAILed.

Test build #60793 has started for PR 13765 at commit fd53a5a.

Test build #60793 has finished for PR 13765 at commit fd53a5a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60793/
Test PASSed.

Hi, @cloud-fan .
Could you review this optimizer?

Sorry we are very close to 2.0 release (and very busy to get it done) and this is too late to get into 2.0.  How about we revisit it after 2.0 release? thanks!

Thank you for comments!
I see. No problem! :)

Hi, @cloud-fan .
Could you review this optimizer?

Rebased.

Test build #61182 has started for PR 13765 at commit 80c6c52.

Test build #61182 has finished for PR 13765 at commit 80c6c52.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61182/
Test PASSed.

Merged build finished. Test PASSed.

Test build #61255 has started for PR 13765 at commit c3016f3.

Test build #61255 has finished for PR 13765 at commit c3016f3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61255/
Test PASSed.

Hi, @cloud-fan .
Could you review this CollapseRepartitionBy optimizer when you have some time?

Test build #61309 has started for PR 13765 at commit 4bb9047.

Hi, @cloud-fan .
Now, this PR can handle all combinations of Repartition and RepartitionBy.
I updated the description of PR and JIRA, too.
Thank you so much for making this PR much useful.

Test build #61309 has finished for PR 13765 at commit 4bb9047.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Hi, @cloud-fan and @yhuai .
I added more description about the logic. I hope that makes the intention of this PR more clear.

Test build #61390 has started for PR 13765 at commit f1ee771.

Retest this please.

Test build #61398 has started for PR 13765 at commit f1ee771.

Test build #61398 has finished for PR 13765 at commit f1ee771.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61398/
Test PASSed.

@rxin .
For this CollapseRepartition optimizer, may I split into another file?

Test build #61564 has started for PR 13765 at commit c75b542.

Test build #61564 has finished for PR 13765 at commit c75b542.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61564/
Test PASSed.

Hi, @cloud-fan .
Could you review CollapseRepartition optimizer again when you have some time?

Rebased to the master.

Test build #61663 has started for PR 13765 at commit e26e956.

Test build #61663 has finished for PR 13765 at commit e26e956.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61663/
Test PASSed.

Merged build finished. Test PASSed.

LGTM, cc @yhuai to take another look

Hi, @yhuai .
Could you review this PR when you have some time?

Test build #61834 has started for PR 13765 at commit 71d0e38.

Test build #61834 has finished for PR 13765 at commit 71d0e38.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61834/
Test PASSed.

Merged build finished. Test PASSed.

Ping @cloud-fan and @yhuai .  :)

Under what circumstances will a user use 2 or more adjacent re-partitioning operators?

There are three possibilities.

User mistakes. (Rarely)
Intermediate results of optimization. (More frequently.)
View (or pre-designed Dataset).

Can you show us an example query?

Thank you for review, @gatorsmile . You work at night, too! :)
BTW, as you know, I don't have a real query. May I make some sample queries from my mind?

Hi, @cloud-fan , @liancheng , @gatorsmile .
We can imagine some data sciences environment having many predefined system-wide tables or datasets. But, here, I simplify that into an extremely two line example.
dsView1 is one of the system-wide predefined table. The number of partition of dsView is optimized for the common usage for the performance (or for preventing skewness.)
Data scientist can not modify that Dataset because it's a shared and predefined one.
However, Data scientist want to save that data into a single file.  (You can see this kind of pattern in current Spark MLlib code, too.)
scala> val dsView1 = spark.range(8).repartition(8)
scala> dsView1.repartition(1).write.json("/tmp/jsonsinglefile")

You can see that the obvious repetition of repartitions.
How do you think about this virtual scenario?

By the way, this PR is about improving the existing CollapseRepartition, not about introducing new one. :)

Hi, All. Here is a better example.
I'll update the PR description with this.
Thank you, all.
Target Scenario
scala> val dsView1 = spark.range(8).repartition(8, $"id")
scala> dsView1.createOrReplaceTempView("dsView1")
scala> sql("select id from dsView1 distribute by id").explain(true)

Before
scala> sql("select id from dsView1 distribute by id").explain(true)
== Parsed Logical Plan ==
'RepartitionByExpression ['id]
+- 'Project ['id]
   +- 'UnresolvedRelation `dsView1`

== Analyzed Logical Plan ==
id: bigint
RepartitionByExpression [id#0L]
+- Project [id#0L]
   +- SubqueryAlias dsview1
      +- RepartitionByExpression [id#0L], 8
         +- Range (0, 8, splits=8)

== Optimized Logical Plan ==
RepartitionByExpression [id#0L]
+- RepartitionByExpression [id#0L], 8
   +- Range (0, 8, splits=8)

== Physical Plan ==
Exchange hashpartitioning(id#0L, 200)
+- Exchange hashpartitioning(id#0L, 8)
   +- *Range (0, 8, splits=8)

After
scala> sql("select id from dsView1 distribute by id").explain(true)
== Parsed Logical Plan ==
'RepartitionByExpression ['id]
+- 'Project ['id]
   +- 'UnresolvedRelation `dsView1`

== Analyzed Logical Plan ==
id: bigint
RepartitionByExpression [id#0L]
+- Project [id#0L]
   +- SubqueryAlias dsview1
      +- RepartitionByExpression [id#0L], 8
         +- Range (0, 8, splits=8)

== Optimized Logical Plan ==
RepartitionByExpression [id#0L]
+- Range (0, 8, splits=8)

== Physical Plan ==
Exchange hashpartitioning(id#0L, 200)
+- *Range (0, 8, splits=8)

@gatorsmile Does it make sense to you? I inserted SQL case for you. :)

Thanks for the examples. This makes sense and LGTM now. Merging into master.

Definitely, it's my bad. I should spend more time to make a reasonable scenario from the first.
Thank you, @liancheng , @cloud-fan , @gatorsmile .

Unfortunately I'm having network issue here and failed to fetch the branch from GitHub :(
@cloud-fan Could you please help merge this one? Thanks.

thanks, merging to master!

authors: AmplabJenkins SparkQA cloud-fan dongjoon-hyun gatorsmile liancheng


text:
[SPARK-16202] [SQL] [DOC] Correct The Description of CreatableRelationProvider's createRelation

What changes were proposed in this pull request?
The API description of createRelation in CreatableRelationProvider is misleading. The current description only expects users to return the relation.
trait CreatableRelationProvider {
  def createRelation(
      sqlContext: SQLContext,
      mode: SaveMode,
      parameters: Map[String, String],
      data: DataFrame): BaseRelation
}

However, the major goal of this API should also include saving the DataFrame.
Since this API is critical for Data Source API developers, this PR is to correct the description.
How was this patch tested?
N/A

cc @rxin @yhuai Please correct me if my understanding is not right. Thanks!

Test build #61223 has started for PR 13903 at commit 37c4817.

Test build #61223 has finished for PR 13903 at commit 37c4817.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61223/
Test PASSed.

Merged build finished. Test PASSed.

Merging in master.

authors: AmplabJenkins SparkQA gatorsmile rxin


text:
[SPARK-16280][SQL] Implement histogram_numeric SQL function

What changes were proposed in this pull request?
This PR implements histogram_numeric according to the Algorithm I & II on the paper: Yael Ben-Haim and Elad Tom-Tov, "A streaming parallel decision tree algorithm", J. Machine Learning Research 11 (2010), pp. 849--872.
How was this patch tested?
I add a test "SPARK-16280: test histogram_numeric." and "SPARK-16280: skip null when computing histogram_numeric." in DataFrameAggregateSuite.

Can one of the admins verify this patch?

@tilumi could show some benchmarks for this? I think that this will have some performance problems. Using an array in a DeclarativeAggregate is not the most efficient.

Test build #3178 has started for PR 14129 at commit 08065d8.

Test build #3178 has finished for PR 14129 at commit 08065d8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class NumericHistogram(child: Expression, nb: Expression) extends DeclarativeAggregate
case class MergeHistograms(left: Expression,

Ok, I'll change the implementation to non-array version and run benchmark on both versions.

May I ask why an array in DeclarativeAggregate is not the most efficient??

I think the cause may be it'll use sortAggregation instead of hashaggregation if I use array as an buffer

I Implement 3 kinds of histogram_numeric and the result is
  (100000, 100)).map((pair) => {

According to the result, I figured out the performance  of the  codegen version is bounded by the number of bins since the large generated code for array creation.

I'll improve it

I finally come out a optimal solution. Since the number of attributes in aggregation buffer is linear to generated code size & code generation time, I instead use just one attribute with array type.
The benchmark result is:

(rows, numOfBins)
codegen_with_arrayAggBuffer_histogram_numeric
codegen_histogram_numeric
declarative_histogram_numeric
imperative_histogram_numeric

(10,1)
346
166
186
336

(10,10)
132
209
136
203

(10,100)
138
1967
226
261

(1000,1)
128
140
159
222

(1000,10)
122
154
181
265

(1000,100)
131
1690
300
494

(100000,1)
169
188
561
729

(100000,10)
219
506
1609
1901

(100000,100)
331
3926
6371
8394

I'll polish the PR for reviewing

I optimized the 'codeGenWithArrayAggBufferNumericHistogram' algorithm and the benchmark result is:

(rows, numOfBins)
codegen_with_array_agg_buffer_histogram_numeric
codegen_histogram_numeric
declarative_histogram_numeric
imperative_histogram_numeric

(10,1)
472
248
263
708

(10,10)
180
250
166
328

(10,100)
260
2023
170
226

(1000,1)
101
135
183
208

(1000,10)
122
164
162
230

(1000,100)
137
1902
367
546

(100000,1)
219
311
860
864

(100000,10)
231
447
1417
1543

(100000,100)
354
3345
8739
8155

It's ready for review.

would love to have this.
also can we also have the same functionality not only in sql, but for RDD/Dataset too?

ping @tilumi someone can review this?

Is this pr available?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

When is this going to be available?

@hvanhovell @cloud-fan @wzhfy can you verify this patch that it can get merged? Sad to see that such a useful implementation gets orphaned caused by missing feedback by maintainers

authors: AmplabJenkins SparkQA cenyuhai hvanhovell iceted rolandjohann tilumi windpiger yangyangyyy


text:
[SPARK-13747][SQL]Fix concurrent executions in ForkJoinPool for SQL

What changes were proposed in this pull request?
Calling Await.result will allow other tasks to be run on the same thread when using ForkJoinPool. However, SQL uses a ThreadLocal execution id to trace Spark jobs launched by a query, which doesn't work perfectly in ForkJoinPool.
This PR just uses Awaitable.result instead to  prevent ForkJoinPool from running other tasks in the current waiting thread.
How was this patch tested?
Jenkins

Test build #67088 has started for PR 15520 at commit 54078ce.

I'm not sure I completely understand why you have to use Awaitable instead of Await. Is it to avoid the blocking() call in Await?

Is it to avoid the blocking() call in Await?

Yep.

That makes sense, thanks for explaining. Would installing a custom BlockingContext on threads that run tasks also work? I'm not very familiar with the internals of the Awaitable api so I'm just throwing this out there in the hopes you can tell me more :)

Test build #67088 has finished for PR 15520 at commit 54078ce.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67088/
Test FAILed.

Would installing a custom BlockingContext on threads that run tasks also work?

Yes, I think so. It just needs more lines :)

retest this please

Test build #67094 has started for PR 15520 at commit 54078ce.

Yes, I think so. It just needs more lines :)

I think the solution you propose is sound. Do you think that a quick mention about avoiding the BlockingContext in a comment would help in case someone sees this code in the future and is puzzled as to what it does?

Test build #67094 has finished for PR 15520 at commit 54078ce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67094/
Test PASSed.

cc @andrewor14

Test build #67112 has started for PR 15520 at commit 6aa9e2f.

Test build #67112 has finished for PR 15520 at commit 6aa9e2f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67112/
Test PASSed.

Test build #67461 has started for PR 15520 at commit e9753ce.

Test build #67461 has finished for PR 15520 at commit e9753ce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67461/
Test PASSed.

Merged build finished. Test PASSed.

lgtm

Thanks! Merging to master and 2.0.

There are conflicts with 2.0. #15646 is the backport for 2.0.

authors: AmplabJenkins SparkQA jodersky yhuai zsxwing


text:
[SPARK-21472][SQL] Introduce ArrowColumnVector as a reader for Arrow vectors.

What changes were proposed in this pull request?
Introducing ArrowColumnVector as a reader for Arrow vectors.
It extends ColumnVector, so we will be able to use it with ColumnarBatch and its functionalities.
Currently it supports primitive types and StringType, ArrayType and StructType.
How was this patch tested?
Added tests for ArrowColumnVector and existing tests.

cc @BryanCutler @kiszk @cloud-fan

Test build #79752 has started for PR 18680 at commit 73899b2.

Test build #79752 has finished for PR 18680 at commit 73899b2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public final class ArrowColumnVector extends ColumnVector

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79752/
Test PASSed.

Test build #79763 has started for PR 18680 at commit ddfcf36.

Test build #79763 has finished for PR 18680 at commit ddfcf36.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79763/
Test FAILed.

Test build #79787 has started for PR 18680 at commit 91b94ef.

@BryanCutler Thank you for reviewing!
As for scope, yes, I'd like these APIs to be public. Do you have any concerns about it?

Test build #79787 has finished for PR 18680 at commit 91b94ef.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79787/
Test PASSed.

Merged build finished. Test PASSed.

@BryanCutler all classes under the execution package are meant to be private, in the future we will move them to a new package if we are ready to public them.

Test build #79793 has started for PR 18680 at commit 2d1dad9.

LGTM, pending jenkins

Test build #79793 has finished for PR 18680 at commit 2d1dad9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79793/
Test PASSed.

thanks, merging to master!

Have you guys checked the performance of this change? It changes the number of concrete implementations for column vector from 2 to 3 (and potentially 1 to 2 at runtime). This might (or might not) have huge performance implications because it might disable inlining, or force virtual dispatches. (It depends on how we can column vector).

cc @ueshin http://apache-spark-developers-list.1001551.n3.nabble.com/Fwd-spark-git-commit-SPARK-21472-SQL-Introduce-ArrowColumnVector-as-a-reader-for-Arrow-vectors-tc22003.html

@viirya and the original reporter, thank you for reporting it!
I submitted a follow-up pr #18701.

authors: AmplabJenkins SparkQA cloud-fan rxin ueshin viirya


text:
[SPARK-24181][SQL] Better error message for writing sorted data

What changes were proposed in this pull request?
The exception message should clearly distinguish sorting and bucketing in save and jdbc write.
When a user tries to write a sorted data using save or insertInto, it will throw an exception with message that s"'$operation' does not support bucketing right now"".
We should throw s"'$operation' does not support sorting right now"" instead.
How was this patch tested?
More tests in DataFrameReaderWriterSuite.scala

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2919/
Test PASSed.

Test build #90206 has finished for PR 21235 at commit 72efec1.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90206/
Test FAILed.

Merged build finished. Test FAILed.

Test build #90208 has started for PR 21235 at commit 1a57a84.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2920/
Test PASSed.

Test build #90208 has finished for PR 21235 at commit 1a57a84.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90208/
Test PASSed.

Test build #90395 has started for PR 21235 at commit 7b64a97.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3061/
Test PASSed.

LGTM

Test build #90395 has finished for PR 21235 at commit 7b64a97.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90395/
Test FAILed.

Merged build finished. Test FAILed.

retest this please.

test it again

Test build #90402 has started for PR 21235 at commit 7b64a97.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3067/
Test PASSed.

Test build #90402 has finished for PR 21235 at commit 7b64a97.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90402/
Test PASSed.

Merged build finished. Test PASSed.

Merged into master.

authors: AmplabJenkins SparkQA dbtsai viirya


text:
[SPARK-25673][BUILD] Remove Travis CI which enables Java lint check

What changes were proposed in this pull request?
https://github.com/apache/spark/pull/12980 added Travis CI file mainly for linter because we disabled Java lint check in Jenkins.
It's enabled as of https://github.com/apache/spark/pull/21399 and now SBT runs it. Looks we can now remove the file added before.
How was this patch tested?
N/A
Closes #22665

cc @srowen and @dongjoon-hyun

Test build #97092 has started for PR 22667 at commit 7a535db.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3780/
Test PASSed.

Test build #97092 has finished for PR 22667 at commit 7a535db.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97092/
Test PASSed.

Merged to master and branch-2.4.
Thanks @srowen and @felixcheung.

Thank you for taking care of this, @HyukjinKwon .

authors: AmplabJenkins HyukjinKwon SparkQA dongjoon-hyun


text:
[SPARK-26977][CORE] Fix warn against subclassing scala.App

What changes were proposed in this pull request?
Fix warn against subclassing scala.App
How was this patch tested?
Manual test

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@srowen I added more comments based on my learnings from https://www.scala-lang.org/api/current/scala/App.html. As a non-native speaker, I'm not sure they can be easily understood  so feel free to rephrase it. Thanks.

I just tried compiling object MyApp extends scala.App with Scala 2.12 and I get in the bytecode...
public final class MyApp$ implements scala/App.
class MyApp extends scala.App yields public class MyApp implements scala/App. I don't see any childMainClass, etc.
I am actually not sure if this ever worked with the class version... it tries to invoke a static main() method, but these classes don't get a static main(); they get a non-static main() though.
I find that the current check works if you check whether "MyApp$" extends scala.App.
I'd suggest we just add that to the condition?

I find that the current check works if you check whether "MyApp$" extends scala.App

Could you please provide a spark-submit example ?
Both spark-submit --class MyApp (for class MyApp extends scala.App) and spark-submit --class MyApp$ (for object MyApp extends scala.App) will fail with
Exception in thread "main" java.lang.IllegalStateException: The main method in the given main class must be static
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:44)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Heh, it's possible this just doesn't work at all. I don't think we ever envisioned people writing a scala.App nor is it tested. If so, it's fine to detect these cases correctly and then just fail rather than warn. Or, just remove this check; it'll fail anyway with a good error. (Or, add a note about scala.App to the exception you get)

Yes, the warning is actually buried in the massive driver logs. Do you mean we should block all scala.App applications ? They will still work if no uninitialized fields are used.

Wait, aren't you saying above that they never work? what version does?

In summary,

spark-submit --class MyApp my-app.jar works fine

object MyApp extends scala.App {

  val spark = SparkSession.builder
    .getOrCreate()

  spark.createDataFrame(Seq("a" -> 1, "b" -> 2, "c" -> 3)).foreachPartition { rowIter =>
    println(rowIter.mkString(","))
  }

spark-submit --class MyApp my-app.jar foo will fail since foo is uninitialized (as in SPARK-4170

object MyApp extends scala.App {
  val foo = args(0)
  val spark = SparkSession.builder
    .getOrCreate()

  spark.createDataFrame(Seq("a" -> 1, "b" -> 2, "c" -> 3)).foreachPartition { rowIter =>
      assert(foo != null)
      println(rowIter.mkString(","))
  }

spark-submit --class MyApp$ my-app.jar fail since main method must be static

object MyApp extends scala.App {
   // whatever
}

4. `spark-submit --class MyApp my-app.jar` fail since `main method must be static`

```scala
class MyApp extends scala.App {
    // whatever
}

I get it, I misread your comment. The warning can stay as-is. Now we're back to the original issue. I'm still not clear why you don't just want to check whether MyApp$ extends scala.App then, if the user tries to run MyApp. in the object case that works, that is the class that extends scala.App. I don't see the synthetic fields you are checking.

When user tries to run MyApp, the passed in main class is MyApp instead of MyApp$. The fix is to check whether MyApp$ extends scala.App.
childMainClass == "MyApp"
s"$childMainClass$$" == "MyApp$"

Test build #4578 has started for PR 23903 at commit faeff46.

Test build #4581 has started for PR 23903 at commit faeff46.

Test build #4582 has started for PR 23903 at commit faeff46.

Test build #4582 has finished for PR 23903 at commit faeff46.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged to master. Thanks for your patience -- almost all of the delay here was me not reading your comments correctly!

Thanks for review. 😄

authors: AmplabJenkins SparkQA manuzhang srowen


text:
[SPARK-28654][SQL] Move "Extract Python UDFs" to the last in optimizer

What changes were proposed in this pull request?
Plans after "Extract Python UDFs" are very flaky and error-prone to other rules.
For instance, if we add some rules, for instance, PushDownPredicates in postHocOptimizationBatches, the test in BatchEvalPythonExecSuite fails:
test("Python UDF refers to the attributes from more than one child") {
  val df = Seq(("Hello", 4)).toDF("a", "b")
  val df2 = Seq(("Hello", 4)).toDF("c", "d")
  val joinDF = df.crossJoin(df2).where("dummyPythonUDF(a, c) == dummyPythonUDF(d, c)")
  val qualifiedPlanNodes = joinDF.queryExecution.executedPlan.collect {
    case b: BatchEvalPythonExec => b
  }
  assert(qualifiedPlanNodes.size == 1)
}

Invalid PythonUDF dummyUDF(a#63, c#74), requires attributes from more than one child.

This is because Python UDF extraction optimization is rolled back as below:
=== Applying Rule org.apache.spark.sql.catalyst.optimizer.PushDownPredicates ===
!Filter (dummyUDF(a#7, c#18) = dummyUDF(d#19, c#18))   Join Cross, (dummyUDF(a#7, c#18) = dummyUDF(d#19, c#18))
!+- Join Cross                                         :- Project [_1#2 AS a#7, _2#3 AS b#8]
!   :- Project [_1#2 AS a#7, _2#3 AS b#8]              :  +- LocalRelation [_1#2, _2#3]
!   :  +- LocalRelation [_1#2, _2#3]                   +- Project [_1#13 AS c#18, _2#14 AS d#19]
!   +- Project [_1#13 AS c#18, _2#14 AS d#19]             +- LocalRelation [_1#13, _2#14]
!      +- LocalRelation [_1#13, _2#14]

Seems we should do Python UDFs cases at the last even after post hoc rules.
Note that this actually rather follows the way in previous versions when those were in physical plans (see SPARK-24721 and SPARK-12981). Those optimization rules were supposed to be placed at the end.
Note that I intentionally didn't move ExperimentalMethods (spark.experimental.extraStrategies). This is an explicit experimental API and I wanted to just-in-case workaround after this change for now.
How was this patch tested?
Existing tests should cover.

cc @cloud-fan

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13886/
Test PASSed.

Test build #108805 has started for PR 25386 at commit ba8dcb7.

Test build #108805 has finished for PR 25386 at commit ba8dcb7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108805/
Test PASSed.

makes sense, merging to master!

authors: AmplabJenkins HyukjinKwon SparkQA cloud-fan


text:
[SPARK-26598] Fix HiveThriftServer2 cannot be modified hiveconf/hivevar variables

What changes were proposed in this pull request?
The intent to use the --hiveconf/--hivevar parameter is just an initialization value, so setting it once in SparkSQLSessionManager#openSession is sufficient, and each time the SparkExecuteStatementOperation setting causes the variable to not be modified.
Why are the changes needed?
It is wrong to set the --hivevar/--hiveconf variable in every SparkExecuteStatementOperation, which prevents variable updates.
Does this PR introduce any user-facing change?
cat <<EOF > test.sql
select '\${a}', '\${b}';
set b=bvalue_MOD_VALUE;
set b;
EOF

beeline -u jdbc:hive2://localhost:10000 --hiveconf a=avalue --hivevar b=bvalue -f test.sql

current result:
+-----------------+-----------------+--+
|     avalue      |     bvalue      |
+-----------------+-----------------+--+
| avalue          | bvalue          |
+-----------------+-----------------+--+
+-----------------+-----------------+--+
|       key       |      value      |
+-----------------+-----------------+--+
| b               | bvalue          |
+-----------------+-----------------+--+
1 row selected (0.022 seconds)

after modification:
+-----------------+-----------------+--+
|     avalue      |     bvalue      |
+-----------------+-----------------+--+
| avalue          | bvalue          |
+-----------------+-----------------+--+
+-----------------+-----------------+--+
|       key       |      value      |
+-----------------+-----------------+--+
| b               | bvalue_MOD_VALUE|
+-----------------+-----------------+--+
1 row selected (0.022 seconds)

How was this patch tested?
modified the existing unit test

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Because the developer who reported SPARK-26598 had not responded for some time, I now take over the issue.
Adding test on the basis of https://github.com/apache/spark/pull/23516
@wangyum

ok to test

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15304/
Test PASSed.

Test build #110309 has started for PR 25722 at commit 2716281.

Test build #110309 has finished for PR 25722 at commit 2716281.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110309/
Test PASSed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15305/
Test PASSed.

Test build #110310 has started for PR 25722 at commit 2716281.

Test build #110310 has finished for PR 25722 at commit 2716281.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110310/
Test PASSed.

cc @HyukjinKwon

Thank you @cxzl25

Merged to master

cc @juliuszsompolski

authors: AmplabJenkins SparkQA cxzl25 gatorsmile wangyum


text:
[SPARK-30107][SQL] Expose nested schema pruning to all V2 sources

What changes were proposed in this pull request?
This PR exposes the existing logic for nested schema pruning to all sources, which is in line with the description of SupportsPushDownRequiredColumns .
Right now, SchemaPruning (rule, not helper utility) is applied in the optimizer directly on certain instances of Table ignoring SupportsPushDownRequiredColumns that is part of ScanBuilder. I think it would be cleaner to perform schema pruning and filter push-down in one place. Therefore, this PR moves all the logic into V2ScanRelationPushDown.
Why are the changes needed?
This change allows all V2 data sources to benefit from nested column pruning (if they support it).
Does this PR introduce any user-facing change?
No.
How was this patch tested?
This PR mostly relies on existing tests. On top, it adds one test to verify that top-level schema pruning works as well as one test for predicates with subqueries.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19620/
Test PASSed.

Test build #114799 has started for PR 26751 at commit 638c2af.

@dongjoon-hyun @cloud-fan @dbtsai @viirya @gengliangwang  @gatorsmile could you take a look whenever you have time?

Test build #114799 has finished for PR 26751 at commit 638c2af.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114799/
Test FAILed.

Thank you for pinging me, @aokolnychyi !

The 5 failures look relevant. Could you fix the UT failure first?

Test build #114856 has started for PR 26751 at commit ee977c1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19679/
Test PASSed.

Hi, @aokolnychyi .
Is there a merge-conflict? If you do force-pushed, it hides the last change from the reviewers . In other words, the reviews need to restart from the bottom .

Test build #114856 has finished for PR 26751 at commit ee977c1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114856/
Test PASSed.

Sorry, @dongjoon-hyun. I wanted to have a clean base that passes all tests. I know the review usually doesn't start before that.

That's true, but I wanted to know what breaks the UT at the initial commit.

I know the review usually doesn't start before that.

@aokolnychyi Thanks for the work!
I'm +1 with the proposal.

@dongjoon-hyun, the original commit did not take into account two edge cases:

overlapping data and partition columns (partially mentioned in this comment)
projections without references such as input_file_name that might result in empty rootFields

Thank you for summarizing, @aokolnychyi .

Hi, @rdblue and @cloud-fan . Could you give some directional advice for the following?

https://github.com/apache/spark/pull/26751#discussion_r354102149

Test build #114944 has started for PR 26751 at commit 0c128fd.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19765/
Test PASSed.

Test build #114945 has started for PR 26751 at commit 4ce0a70.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19766/
Test PASSed.

Test build #114945 has finished for PR 26751 at commit 4ce0a70.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #114944 has finished for PR 26751 at commit 0c128fd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114945/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114944/
Test PASSed.

Hi, @aokolnychyi . #26830 is merged to support this PR.
Could you rebase this PR?

Test build #115117 has started for PR 26751 at commit 2bdaa10.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19930/
Test PASSed.

Thanks, @dongjoon-hyun! I've rebased the PR.

Test build #115117 has finished for PR 26751 at commit 2bdaa10.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115117/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20001/
Test PASSed.

Test build #115190 has started for PR 26751 at commit 4d732b2.

Test build #115190 has finished for PR 26751 at commit 4d732b2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115190/
Test PASSed.

Thanks everyone for the feedback. Let me know if there are any other comments.

+1
Thanks for working on this! Maybe we can get it into the next 3.0 preview.

Thanks for this work! LGTM

Thanks, merging to master

authors: AmplabJenkins SparkQA aokolnychyi dongjoon-hyun gengliangwang rdblue viirya


text:
[SPARK-30504][PYTHON][ML]  Set weightCol in OneVsRest(Model) _to_java and _from_java

What changes were proposed in this pull request?

This PR adjusts _to_java and _from_java of OneVsRest and OneVsRestModel to preserve weightCol.
Why are the changes needed?

Currently both Params don't preserve weightCol Params when data is saved / loaded:
from pyspark.ml.classification import LogisticRegression, OneVsRest, OneVsRestModel
from pyspark.ml.linalg import DenseVector

df = spark.createDataFrame([(0, 1, DenseVector([1.0, 0.0])), (0, 1, DenseVector([1.0, 0.0]))], ("label", "w", "features"))

ovr = OneVsRest(classifier=LogisticRegression()).setWeightCol("w")
ovrm = ovr.fit(df)
ovr.getWeightCol()
## 'w'
ovrm.getWeightCol()
## 'w'

ovr.write().overwrite().save("/tmp/ovr")
ovr_ = OneVsRest.load("/tmp/ovr")
ovr_.getWeightCol()
## KeyError
## ...
## KeyError: Param(parent='OneVsRest_5145d56b6bd1', name='weightCol', doc='weight column name. ...)

ovrm.write().overwrite().save("/tmp/ovrm")
ovrm_ = OneVsRestModel.load("/tmp/ovrm")
ovrm_ .getWeightCol()
## KeyError
## ...
## KeyError: Param(parent='OneVsRestModel_598c6d900fad', name='weightCol', doc='weight column name ...

Does this PR introduce any user-facing change?

After this PR is merged, loaded objects will have weightCol Param set.
How was this patch tested?

Manual testing.
Extension of existing persistence tests.

Test build #116636 has started for PR 27190 at commit c9ccbb2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21417/
Test PASSed.

Test build #116636 has finished for PR 27190 at commit c9ccbb2.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116636/
Test FAILed.

Test build #116646 has started for PR 27190 at commit 88298ba.

Test build #116646 has finished for PR 27190 at commit 88298ba.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116646/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21425/
Test PASSed.

Test build #116648 has started for PR 27190 at commit 83b9565.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21427/
Test PASSed.

Test build #116648 has finished for PR 27190 at commit 83b9565.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116648/
Test PASSed.

CC @huaxingao @zhengruifeng as you've been looking at handling weight cols recently

Test build #116726 has started for PR 27190 at commit aae798d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21501/
Test PASSed.

Test build #116726 has finished for PR 27190 at commit aae798d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116726/
Test PASSed.

authors: AmplabJenkins SparkQA srowen zero323


text:
[SPARK-30519][CORE] Use spark.executorEnv.HADOOP_USER_NAME to set executor's hadoop user

What changes were proposed in this pull request?

Let executor can specify hadoop user by using spark.executorEnv.HADOOP_USER_NAME
Why are the changes needed?

We can use HADOOP_USER_NAME to specify hadoop user while submitting a job on driver, but invalid to set executor's hadoop user even by using spark.executorEnv.HADOOP_USER_NAME.
Does this PR introduce any user-facing change?

No
How was this patch tested?

This is a behavior change, and I don't think we should introduce this change by the approach proposed this way.
If both SPARK_USER and HADOOP_USER_NAME are specified in the env, we should prioritize SPARK_USER over HADOOP_USER_NAME. Also, I'm hesitate whether we should support HADOOP_USER_NAME at all.

This is a behavior change, and I don't think we should introduce this change by the approach proposed this way.
If both SPARK_USER and HADOOP_USER_NAME are specified in the env, we should prioritize SPARK_USER over HADOOP_USER_NAME. Also, I'm hesitate whether we should support HADOOP_USER_NAME at all.

But SPARK_USER is used for scheduler such as Mesos, Kubernetes and HADOOP_USER_NAME is used for Hadoop cluster such HDFS. At Spark on Kubernetes, SPARK_USER is not over HADOOP_USER_NAME


text:
[SPARK-32175][CORE] Fix the order between initialization for ExecutorPlugin and starting heartbeat thread

What changes were proposed in this pull request?

This PR changes the order between initialization for ExecutorPlugin and starting heartbeat thread in Executor.
Why are the changes needed?

In the current master, heartbeat thread in a executor starts after plugin initialization so if the initialization takes long time, heartbeat is not sent to driver and the executor will be removed from cluster.
Does this PR introduce any user-facing change?

Yes. Plugins for executors will be allowed to take long time for initialization.
How was this patch tested?

New testcase.

Test build #124964 has started for PR 29002 at commit d724179.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29570/
Test PASSed.

Test build #124964 has finished for PR 29002 at commit d724179.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124964/
Test FAILed.

retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29599/
Test PASSed.

Test build #124993 has started for PR 29002 at commit d724179.

Test build #124993 has finished for PR 29002 at commit d724179.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124993/
Test FAILed.

retest this please.

Test build #124994 has started for PR 29002 at commit d724179.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29601/
Test PASSed.

Test build #124994 has finished for PR 29002 at commit d724179.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124994/
Test FAILed.

retest this please.

Test build #124999 has started for PR 29002 at commit d724179.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29606/
Test PASSed.

Test build #124999 has finished for PR 29002 at commit d724179.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124999/
Test FAILed.

retest this please.

Test build #125110 has started for PR 29002 at commit d724179.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29716/
Test PASSed.

Test build #125110 has finished for PR 29002 at commit d724179.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125110/
Test FAILed.

Test build #125316 has started for PR 29002 at commit 1ccb1b7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29928/
Test PASSed.

Test build #125316 has finished for PR 29002 at commit 1ccb1b7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125316/
Test FAILed.

As soon as we heartbeat to driver, we will start getting task assignment, etc - how does the interact with a plugin which has not yet completed initialization ?
Instead of changing order, why not document that inordinate delays in plugin initialization should be avoided and deferred to seperate thread/managed by plugin.

@Ngone51

Is this from a real issue?
Shall we do the same thing for DriverPlugin or we already did?
It happens not in the production but an environment for experiment.
I confirmed we don't need to reorder the initialization for DriverPlugin because executors send hearbeat to the driver but the driver doesn't send it to self.

@mridulm

As soon as we heartbeat to driver, we will start getting task assignment, etc - how does the interact with a plugin which has not yet completed initialization ?

Yeah that's right.

Instead of changing order, why not document that inordinate delays in plugin initialization should be avoided and deferred to >seperate thread/managed by plugin.

This can be a reasonable compromise. Another solution can be to perform initialization before registering executors and then it's guaranteed that plugins are initialized when tasks start running.
What do you think?

This can be a reasonable compromise. Another solution can be to perform initialization before registering executors and then it's guaranteed that plugins are initialized when tasks start running.
What do you think?

Isn't that not what is implicitly handling now ? And causing issues if plugin takes a long time to initialize ?

As soon as we heartbeat to driver, we will start getting task assignment, etc - how does the interact with a plugin which has not yet completed initialization ?

I wonder if this is correct.  Executor only gets task assignment after it sends LaunchedExecutor to driver:
https://github.com/apache/spark/blob/3946b243284fbd3bd98b456115ae194ad49fe8fe/core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala#L156-L158
After heartbeater started, the executor can send heartbeat to the driver asynchronically and the initialization of the plugin can keep going after heartbeater starting. It doesn't matter if it takes a long time because we'd send heartbeat now but no task assignment since LaunchedExecutor hasn't been sent.

ExecutorRegistered is regarded as the first heartbeat message although it's not the real Heartbeat. So If we reorder the initialization, it should be placed before the registration, not after heartbeater started.

You are right @Ngone51, I misread the proposed change.

So If we reorder the initialization, it should be placed before the registration, not after heartbeater started right?

Ideally, that way should work as well. However, it seems plugin initialization relies on some internal instances of Executor. So, if we want to initialize plugin out of the executor, I'm afraid we need more changes.
BTW, I think your current implementation can already work as I mentioned above.

Ideally, that way should work as well. However, it seems plugin initialization relies on some internal instances of Executor. So, if > we want to initialize plugin out of the executor, I'm afraid we need more changes.

Yes, if we reorder the initialization with another way, the initialization would be put on executor backends like CoarseGrainedExecutorBackend and we might need more changes as you are afraid.

BTW, I think your current implementation can already work as I mentioned above.
Yeah, I misunderstood. LaunchExecutor is sent after Executor is instansiated so the current implementation works.

retest this please.

Test build #125591 has started for PR 29002 at commit 1ccb1b7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30206/
Test PASSed.

Test build #125591 has finished for PR 29002 at commit 1ccb1b7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125591/
Test PASSed.

@Ngone51 Thanks for the feedback. I'll fix them.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30400/
Test PASSed.

Test build #125789 has started for PR 29002 at commit dba4c91.

Test build #125789 has finished for PR 29002 at commit dba4c91.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
        |public class $className $extendsText $implementsText

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125789/
Test FAILed.

According to the log for the previous unit test, the reason why the new test failed seems not to related to heartbeat.
It seems to have taken about 30 secs until Executor started.
20/07/13 13:49:15.501 redirect stderr for command /home/jenkins/workspace/SparkPullRequestBuilder/bin/spark-submit INFO Utils: SLF4J: Class path contains multiple SLF4J bindings.

20/07/13 13:49:43.912 dispatcher-Executor INFO Executor: Starting executor ID 0 on host 192.168.10.25

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125789/artifact/core/target/unit-tests.log
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/125789/artifact/work/app-20200713134929-0000/0/target/unit-tests.log
So, I'll extend the timeout.

Test build #125798 has started for PR 29002 at commit d768385.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30409/
Test PASSed.

Test build #125798 has finished for PR 29002 at commit d768385.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125798/
Test FAILed.

retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30442/
Test PASSed.

Test build #125831 has started for PR 29002 at commit d768385.

Test build #125831 has finished for PR 29002 at commit d768385.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125831/
Test FAILed.

retest this please.

retest this please.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30503/
Test PASSed.

Test build #125895 has started for PR 29002 at commit d768385.

Test build #125895 has finished for PR 29002 at commit d768385.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125895/
Test FAILed.

retest this please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30542/
Test PASSed.

Test build #125932 has started for PR 29002 at commit d768385.

Test build #125932 has finished for PR 29002 at commit d768385.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/125932/
Test FAILed.

retest this please.

Test build #125997 has started for PR 29002 at commit d768385.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30611/
Test PASSed.

Test build #125997 has finished for PR 29002 at commit d768385.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

retest this please.

Test build #126105 has started for PR 29002 at commit d768385.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30717/
Test PASSed.

Seems like all the tests passed in GitHub Actions?

Test build #126105 has finished for PR 29002 at commit d768385.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126105/
Test PASSed.

cc: @vanzin @tgravescs

Yeah I think this is ok, especially since normally if your plugin is less then heartbeat time there shouldn't be any difference. I would like to test this with a couple scenarios though.
When we first put this in, I brought up that the plugin could block other executor functionality by running a long time, one thought was we could put plugin into another thread so it didn't block the executor. I think the thought at the time was if you have a long running executor plugin, the user should handle that by doing it in another thread or something.   Now you could make the argument you have something long running in the plugin and you want to block the executor for other initialization.  The heartbeat is 10seconds by default and I thought the timeout was the network timeout by default (120s), which seems like a really long time.  I guess if you change those timeouts, but even then seems like a long time.

sorry had stuff come up internally, I'll look more today.

ok looked at this a bit more and tested a few scenarios, I think this is fine, I would like it put right after heartbeater unless we have reason and would be nice to get test to run faster.

Test build #126520 has started for PR 29002 at commit 0b8da96.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31139/
Test PASSed.

Test build #126520 has finished for PR 29002 at commit 0b8da96.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126520/
Test FAILed.

retest this please.

Test build #126533 has started for PR 29002 at commit 0b8da96.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31152/
Test PASSed.

Test build #126533 has finished for PR 29002 at commit 0b8da96.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126533/
Test PASSed.

it looks like arrow tests failing which shouldn't be related, kicked once more

It's strange that the reason of the R test failure implies that deprecated function is used. But it's resolved in #29252 .
I'll rebase this change.
2020-07-26T03:10:50.2710895Z test_sparkSQL_arrow.R:39: error: createDataFrame/collect Arrow optimization
2020-07-26T03:10:50.2711900Z (converted from warning) Use 'read_ipc_stream' or 'read_feather' instead.
2020-07-26T03:10:50.2712135Z Backtrace:
2020-07-26T03:10:50.2712459Z   1. base::tryCatch(...) tests/fulltests/test_sparkSQL_arrow.R:39:2
2020-07-26T03:10:50.2712724Z   7. SparkR::collect(createDataFrame(mtcars))
2020-07-26T03:10:50.2712988Z   8. SparkR:::.local(x, ...)
2020-07-26T03:10:50.2713221Z  11. arrow::read_arrow(readRaw(conn))
2020-07-26T03:10:50.2713946Z  12. base::.Deprecated(msg = "Use 'read_ipc_stream' or 'read_feather' instead.")
2020-07-26T03:10:50.2714178Z  13. base::warning(...)
2020-07-26T03:10:50.2714462Z  14. base::withRestarts(...)
2020-07-26T03:10:50.2714735Z  15. base:::withOneRestart(expr, restarts[[1L]])
2020-07-26T03:10:50.2714999Z  16. base:::doWithOneRestart(return(expr), restart)

Test build #126742 has started for PR 29002 at commit f106fe9.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31365/
Test PASSed.

Test build #126742 has finished for PR 29002 at commit f106fe9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126742/
Test PASSed.

LGTM

finally, tests pass, merged to master and branch-3.0.  thanks @sarutak

Would you guys mind taking a look for the flaky tests added here? It seems failing in GitHub Actions as below:
[info] - SPARK-32175: Plugin initialization should start after heartbeater started *** FAILED *** (32 seconds, 514 milliseconds)
[info]   The code passed to failAfter did not complete within 30000000000 nanoseconds. (SparkSubmitSuite.scala:1438)
[info]   org.scalatest.exceptions.TestFailedDueToTimeoutException:
[info]   at java.lang.Thread.getStackTrace(Thread.java:1559)
[info]   at org.scalatest.concurrent.TimeLimits.failAfterImpl(TimeLimits.scala:234)
[info]   at org.scalatest.concurrent.TimeLimits.failAfterImpl$(TimeLimits.scala:233)
[info]   at org.apache.spark.deploy.SparkSubmitSuite$.failAfterImpl(SparkSubmitSuite.scala:1419)
[info]   at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:230)
[info]   at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:229)
[info]   at org.apache.spark.deploy.SparkSubmitSuite$.failAfter(SparkSubmitSuite.scala:1419)
[info]   at org.apache.spark.deploy.SparkSubmitSuite$.runSparkSubmit(SparkSubmitSuite.scala:1438)
[info]   at org.apache.spark.executor.ExecutorSuite.$anonfun$new$17(ExecutorSuite.scala:469)
[info]   at org.apache.spark.executor.ExecutorSuite.$anonfun$new$17$adapted(ExecutorSuite.scala:407)
[info]   at org.apache.spark.SparkFunSuite.withTempDir(SparkFunSuite.scala:170)
[info]   at org.apache.spark.executor.ExecutorSuite.$anonfun$new$16(ExecutorSuite.scala:407)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:189)
[info]   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:158)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:187)

https://github.com/apache/spark/runs/928145740
https://github.com/apache/spark/runs/928147695
https://github.com/apache/spark/runs/928059788
https://github.com/apache/spark/runs/927624834
https://github.com/apache/spark/runs/927483940
https://github.com/apache/spark/runs/925705459
https://github.com/apache/spark/runs/925169167
https://github.com/apache/spark/runs/925406406
https://github.com/apache/spark/runs/923441051
The failure seems pretty frequent.

@HyukjinKwon I'll look into that. Should we ignore the test first?

Personally I would be fine with just removing the test.  I'm not sure how much benefit it really adds for the time it takes to run the test and if its being flaky I'm guessing you are going to have to increase the times.

Yeah, I wonder we need to increase the times. The testcase is to detect regressions which can happen in the future but I've left the comment about the order so removing the test can be fine.
@HyukjinKwon , what do you think?

Sure, removing sounds fine and simpler to me.

O.K, I'll make a PR for removing it.

authors: AmplabJenkins HyukjinKwon Ngone51 SparkQA maropu mridulm sarutak tgravescs