text:
TrustManagerProxy will not load a file with multiple certificates correctly

I'm using a TLS-enabled admin client configured with a cert/key pair file and a trusted certificate file, the latter of which is a bundle with multiple certificates. From what I can tell, the client is created with an instance of TrustManagerProxy, which uses CertificateFactory#generateCertificate (singular) instead of CertificateFactory#generateCertificates (plural) and thus doesn't load all certificates in the bundle.

Could you provide your config file?

I don't have one, the client is created using org.apache.pulsar.client.admin.PulsarAdminBuilder and setting tlsTrustCertsFilePath. Note that the client is used by the Flink-Pulsar connector, and it seems that it also uses org.apache.pulsar.client.impl.AuthenticationUtil#create under the hood to instantiate org.apache.pulsar.client.impl.auth.AuthenticationTls.

I submitted https://github.com/apache/pulsar/pull/14798 to support this, could you review this PR?

source: gharchive/issue

authors: asardaes nodece

repo: apache/pulsar

license_type: permissive

license_source: github-api


text:
[fix][txn] Transaction metadata client cnx is null

Motivation
fix transactionMetadataStoreHandle cnx is null.
https://github.com/apache/pulsar/blob/b13d15c4d6d795f33c6ed23f920fabbb3eeaf745/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TransactionMetaStoreHandler.java#L134-L154
because
https://github.com/apache/pulsar/blob/b13d15c4d6d795f33c6ed23f920fabbb3eeaf745/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TransactionMetaStoreHandler.java#L142
is the async method, set cnx, and change to ready state is not an atomic operation.

set cnx into this handle
reconnect will set the cnx to null https://github.com/apache/pulsar/blob/29308e48b78a01959ee8067dbcac67d0e903e8a4/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConnectionHandler.java#L101
then send connect request and get a response that will change state to ready state.
the client will not reconnect because the state is ready
the cnx also is null. transactionMetadataStoreHandle will not do txn op.

Modifications
fix race condition
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation

Does this pull requestintroducese a new feature? (yes)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create afollow-upp issue for adding the documentation

/pulsarbot run-failure-checks

source: gharchive/pull-request


text:
Allow non-pesistent topics to be retrieved along with persistent ones from the "GetTopicsOfNamespace" method

Motivation
Please see issue #2009 for a detailed bug report.
In our use case we require using java client with pattern subscription to read from a set of non-persistent topics. Unfortunately, right now this feature doesn't work. After researching the cause for this I have found out that under the hood the client is requesting a list of topics by namespace from the server and then filters them out by pattern and subscribes to them. The method in Pulsar broker NamespaceService class that is responsible for searching for required topics only uses ledgers, thus returning only persistent topics to the client. The goal of this pull request is to provide a solution for that problem.
Modifications
This pull request updates getListOfTopics method of NamespaceService class to also include active non-pesistent topics from local broker cache inside the multiLayerTopicsMap collection of BrokerService in the result.
Result
As a result, requesting a list of topics by namespace using the HTTP API or binary API (and thus via the clients) will add non-persistent topics to search result, allowing pattern subscription to be used with non-persistent topics.
Considerations

Since this method pulls non-persistent topics from local broker cache, this probably means that this solution will only work for Pulsar installations with a single broker. And if there are multiple brokers, results might be inconsistent. Unfortunately I don't really know if non-persistent themselves work in multi-broker setups. I have recently asked on Slack if non-persistent topics are being replicated in any way and @merlimat's response was that they don't. Also it seems to be that some other methods that are working with non-persistent topics are using this very same collection.

It seems to me that unit tests have made sure that Java client can work with this setup, but this might still be a breaking change for other clients or if applications working with this API are not expecting non-persistent topics in result.

I have made sure that old unit tests inside the pulsar-broker subproject are still working and updated some old tests for this particular use case. Are there any more tests that I can add.

Overall, we really need this and I would appreciate if maintainers could share their opinion. Thanks in advance.

@jiazhai please review this when you have time.

Just noticed integration tests are failing. Is this my fault? Should I fix that?

retest this please

Unfortunately I don't really know if non-persistent themselves work in multi-broker setups. I have recently asked on Slack if non-persistent topics are being replicated in any way and @merlimat's response was that they don't. Also it seems to be that some other methods that are working with non-persistent topics are using this very same collection.

A single topic (or partition of a topic) is assigned to a single broker, at a given point in time.
Topics belonging to a namespace can be assigned to different brokers (that is true for both persistent and non-persistent topics)
Persistent topics use BookKeeper to persist data and maintain state --> When a broker crashes, the topic can move immediately to a different broker and restart with the same exact state.
Non-persistent topics do not store any kind of state durably. The messages are kept for a very short amount of time in memory in the broker and dropped if they cannot be delivered immediately. The data for a single non-persistent topic is not stored in multiple machines. If the broker crashes, some messages will be lost and the topic will continue on the new broker.

Thanks for reviewing this everyone.
@merlimat we expected this feature to work through Pulsar Java client. I guess to use that API we will have to write our own client logic that queries for non-persistent topics first. I will take a look.
@jiazhai I must have forgotten my IDE is configured to use 2 whitespaces.

@merlimat we expected this feature to work through Pulsar Java client. I guess to use that API we will have to write our own client logic that queries for non-persistent topics first. I will take a look.

Yes, I was meaning that in broker side we should use the client API (there are already several cases that already do that). Since a single broker will not have all the topics, and that non-persistent topics don't have additional metadata, the client API call is the only option to get the correct list of non-persistent topics.
An additional improvement could be to only make this call if we know the client is interested in non-persistent topics and skip it otherwise.

@merlimat @jiazhai I have updated this feature as you suggested.
I haven't yet added tests for this or fixed any old ones, just made sure it compiles. I will test this ASAP, just wanted to make sure this functionality is OK before I go any further.
This is what was done in the latest commits:

Additional property "Mode" was added to "CommandGetTopicsOfNamespace" entity of Pulsar binary API via Protobuf. This is a enum that has 3 values: "Persistent", "Non-Persistent", or "All". Value "Persistent" was set to be the default one.

Broker binary protocol server code was updated to retrieve topics from lookup based on this enum. More importantly, broker will attempt to retrieve a list of non-persistent topics from another cluster if necessary. A pool of Pulsar Clients was made for that purpose inside NamespaceService.

I have attempted to do the same for HTTP APIs, but I have only done so for API V1. I am not sure what to do about API V2, as it seems to use a somewhat more complicated functionality.

Client code was updated to support this. When building a consumer it is now possible to specify to which kind of topics you want to subscribe when using pattern subscription.

Will be glad to receive your feedback.

@merlimat @jiazhai can you review this PR and give @gordeevbr some feedback?

ping @merlimat and @jiazhai . lets give some feedback to @gordeevbr , so that we can move forward with the change here.

retest this please

@jiazhai @merlimat @sijie
I have fixed old tests that were failing, added some new tests to test suites where it was possible, updated features to be consistent with HTTP Lookup, fixed all found issues, and updated with master branch.
There's not much new in these commits, mostly fixes and tests.
I think it should be ready now. I am, of course, ready to update this PR if deemed necessary.

@gordeevbr cool, thanks. I removed "[wip]". so other people know this PR is ready to review.

Would you mind reruning these tests? I'm quite sure some of these were working just now.

run java8 tests

retest this please

run java8 tests

Great. Thank you very much!

This is not working for me in Pulsar 2.11.1. I am able to receive messages from non-persistent topics belonging to single broker only.

authors: VijayRohra gordeevbr jiazhai merlimat sijie srkukarni


text:
[improve] [proxy] Not close the socket if lookup failed caused by too many requests

Motivation
Background: The Pulsar client will close the socket if it receives a ServiceNotReady error when doing a lookup.
see https://github.com/apache/pulsar/blob/af20a8a8623055bab665e566b3f54f6f884d121d/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ClientCnx.java#L1193-L1200
The Broker will respond to the client with a TooManyRequests error if there are too many lookup requests in progress, but the Pulsar Proxy responds to the client with a ServiceNotReady error in the same scenario.
Modifications
Make Pulsar Proxy respond to the client with a TooManyRequests error if there are too many lookup requests in progress.
Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository: x

Codecov Report

Merging #21216 (addd907) into master (eefc517) will increase coverage by 36.09%.
Report is 3 commits behind head on master.
The diff coverage is 100.00%.

@@              Coverage Diff              @@
##             master   #21216       +/-   ##
=============================================
+ Coverage     36.75%   72.85%   +36.09%
+ Complexity    12193     3684     -8509
=============================================
  Files          1698     1887      +189
  Lines        130430   140118     +9688
  Branches      14250    15426     +1176
=============================================
+ Hits          47940   102081    +54141
+ Misses        76167    29931    -46236
- Partials       6323     8106     +1783

Flag
Coverage Δ

inttests
24.21% <0.00%> (+0.07%)
:arrow_up:

systests
24.79% <0.00%> (+0.02%)
:arrow_up:

unittests
72.11% <100.00%> (+40.22%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed
Coverage Δ

...apache/pulsar/proxy/server/LookupProxyHandler.java
60.16% <100.00%> (+17.37%)
:arrow_up:

... and 1456 files with indirect coverage changes

LGTM


text:
fix some typos in the comments

fix some typos in the comments

run java8 tests

run java8 tests

run java8 tests

run java8 tests

authors: codelipenghui jiazhai sijie tuteng


text:
RocketMQ Document Typesetting Errors Issue

BUG REPORT

Please describe the issue you observed:

What did you do (The steps to reproduce)?
I will fix RocketMQ Document Typesetting Errors Issue.

What did you expect to see?
I will fix RocketMQ Document Typesetting Errors Issue.

What did you see instead?
I have found there are some RocketMQ Document Typesetting Errors Issues.

Please tell us about your environment:

Other information (e.g. detailed explanation, logs, related issues, suggestions how to fix, etc):

sub-task1-issue-number: sub-task1 description here,
sub-task2-issue-number: sub-task2 description here,
...

@zongtanghu Could you help to polish the distribution package rocketmq-all to a more user-friendly name in the 4.5.2 version?

authors: vongosling zongtanghu

repo: apache/rocketmq


text:
[ISSUE #4921] Fix wrong log output in TimerMessageStore

Make sure set the target branch to develop
What is the purpose of the change
Fix wrong log output in TimerMessageStore
Close #4921

Brief changelog
XX
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Codecov Report

Merging #4922 (a18342e) into develop (c1c960b) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@              Coverage Diff              @@
##             develop    #4922      +/-   ##
=============================================
+ Coverage      43.09%   43.10%   +0.01%
+ Complexity      7675     7674       -1
=============================================
  Files            994      994
  Lines          68939    68939
  Branches        9132     9132
=============================================
+ Hits           29706    29713       +7
+ Misses         35494    35491       -3
+ Partials        3739     3735       -4

Impacted Files
Coverage Δ

...apache/rocketmq/store/timer/TimerMessageStore.java
54.18% <100.00%> (ø)

...apache/rocketmq/broker/longpolling/PopRequest.java
31.03% <0.00%> (-13.80%)
:arrow_down:

...ache/rocketmq/common/stats/MomentStatsItemSet.java
39.13% <0.00%> (-4.35%)
:arrow_down:

.../apache/rocketmq/controller/ControllerManager.java
73.11% <0.00%> (-2.16%)
:arrow_down:

...mq/client/impl/consumer/RebalanceLitePullImpl.java
69.86% <0.00%> (-1.37%)
:arrow_down:

...he/rocketmq/client/impl/consumer/ProcessQueue.java
61.92% <0.00%> (-0.46%)
:arrow_down:

...ent/impl/consumer/DefaultLitePullConsumerImpl.java
69.37% <0.00%> (ø)

...mq/store/ha/autoswitch/AutoSwitchHAConnection.java
71.38% <0.00%> (+0.27%)
:arrow_up:

.../apache/rocketmq/logging/inner/LoggingBuilder.java
63.92% <0.00%> (+0.47%)
:arrow_up:

...client/consumer/store/RemoteBrokerOffsetStore.java
69.29% <0.00%> (+0.87%)
:arrow_up:

... and 5 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: RongtongJin codecov-commenter


text:
SAMZA-2648: Add simple placeholder FaultDomainManager for when YARN is not used

Issues: There is currently only one concrete FaultDomainManager implementation, and it is for YARN usage. It is also specified as the default in code. However, for non-YARN environments (such as Kubernetes), we shouldn't include YARN dependencies, so we need another concrete implementation. For now, we can just add a very simple implementation as a placeholder.
Changes: Added SingleFaultDomainManager which is a simple impl of FaultDomainManager which only has a single fault domain. This isn't wired in anywhere, but a job can configure it as the FaultDomainManager if desirable.
Tests: Ran a simple test job on Kubernetes using this class (also excluded all code from samza-yarn). Any features which depend on fault domain awareness won't work (e.g. using standby containers with cluster-manager.fault-domain-aware.standby.enabled = true won't work), but it can still be a useful placeholder when standby containers are not needed.
API changes and usage/upgrade instructions:
Set the config cluster-manager.fault-domain-manager.factory to org.apache.samza.clustermanager. SingleFaultDomainManagerFactory to use this new fault domain manager.

What do you mean by this line in your description: This won't work if standby containers are enabled, but it can still be a useful placeholder when standby containers are not needed.?
For jobs to be fault-domain aware, they need to turn on this config specifically: cluster-manager.fault-domain-aware.standby.enabled. Not all jobs with standby containers are fault domain aware.

What do you mean by this line in your description: This won't work if standby containers are enabled, but it can still be a useful placeholder when standby containers are not needed.?
For jobs to be fault-domain aware, they need to turn on this config specifically: cluster-manager.fault-domain-aware.standby.enabled. Not all jobs with standby containers are fault domain aware.

Sorry, my comment wasn't correct. Thanks for pointing out that config. I updated it to Any features which depend on fault domain awareness won't work (e.g. using standby containers with cluster-manager.fault-domain-aware.standby.enabled = true won't work), but it can still be a useful placeholder when standby containers are not needed.

repo: apache/samza

license_source: bigquery


text:
[Bug] [Seatunnel-web] HiveJDBC should not be used as a sink datasource

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
HiveJDBC is available sink datasource. It should not be available as sink datasource.
SeaTunnel Version
2.3.7
SeaTunnel Config
NA

Running Command
NA

Error Exception
NA

Zeta or Flink or Spark Version
No response
Java or Scala Version
No response
Screenshots
No response
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

PR is merged
https://github.com/apache/seatunnel-web/commit/2c09444ce3255bcaf5abf06916ab779fac131a82

repo: apache/seatunnel


text:
[Feature][Connectors-v2-Paimon] Adaptation Paimon 0.6 Version

Purpose of this pull request
Implement Feature https://github.com/apache/seatunnel/issues/6058

Does this PR introduce any user-facing change?

No.
How was this patch tested?

Re-run Paimon e2e Test. Perfect Pass
Check list

[x] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update change log that in connector document. For more details you can refer to connector-v2
Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist

[ ] Update the release-note.

cc @TyrantLucifer


text:
[Task] ShenYu e2e refactor

Description
ShenYu e2e module needs to refactor at present, refactor the code support shenyu check style.
Task List

[ ] refactor shenyu e2e engine code support shenyu check style
[ ] refactor shenyu e2e common code support shenyu check style @runqi-zhao
[ ] refactor shenyu e2e client code support shenyu check style @Albertsirius
[ ] refactor shenyu case engine code support shenyu check style @omernaci

Hi @moremind, I would happy to refactor shenyu-e2e-engine with shenyu-check-style.

Hi @moremind, I would happy to refactor shenyu-e2e-engine with shenyu-check-style rules.

Hi, @omernaci , I have assigned this task to you.

Hi @moremind, I would happy to refactor shenyu-e2e-common with shenyu-check-style rules.

@runqi-zhao I have assigned this task to you.

Hi, I would like to try "refactor shenyu e2e client code support shenyu check style" task. please assign to me

Hi, I would like to try "refactor shenyu e2e client code support shenyu check style" task. please assign to me

ok

Hi @moremind ,I would like to try the remaining task，but I don't know witch one remains. @omernaci said he wanted to refactor shenyu-e2e-engine but you assigned shenyu-case-engine to him.

Hi, @moremind pls assign task 1 to me. Thank u.

pls assign 2 me.

@omernaci @runqi-zhao @Albertsirius Hi, this task is overtime, I will reassign it to @damonxue

authors: Albertsirius arenick001 damonxue moremind omernaci runqi-zhao

repo: apache/shenyu


text:
Add benchmarks for topological queries

The specification of the test environment is:
OS: Ubuntu 22.04.3 LTS x86_64
CPU: AMD EPYC 7282 (4) @ 2.794GHz
Memory: 7951MiB
Topological query benchmark results before series index optimization:
Metric Name, Min, Max, Mean, Median, P95
topology-0, 0.038543, 0.362319, 0.099407, 0.084507, 0.234849
Topological query benchmark results after series index optimization:
Metric Name, Min, Max, Mean, Median, P95
topology-0, 0.038886, 0.236007, 0.082450, 0.075451, 0.133140
Index query trace before optimization:

Index query trace after optimization:

As we can see, the topological query performance has improved, especially for p95. The index optimization proves effective, as expected.

@ButterBright Could you share the query tracing result between them? I want to ensure the scanning strategy is improved as we expected.

Updated.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 51.27%. Comparing base (70557d3) to head (c1d3bdf).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #530      +/-   ##
==========================================
+ Coverage   51.25%   51.27%   +0.01%
==========================================
  Files         165      165
  Lines       20299    20299
==========================================
+ Hits        10405    10409       +4
+ Misses       9105     9102       -3
+ Partials      789      788       -1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

repo: apache/skywalking-banyandb


text:
bug about prepareBath in MetricsPersistentWorker

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
[ X] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?

https://github.com/apache/skywalking/blob/c5df0760f257e523902b641755b1cf7e6cee4476/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/worker/MetricsPersistentWorker.java#L133
Math.max make maxBatchGetSize meaningless. And from the context , should it be Math.min ?

Bug

Which version of SkyWalking, OS and JRE?
8.1.X

Which company or project?

What happened?
If possible, provide a way to reproduce the error. e.g. demo application, component version.

Requirement or improvement

Please describe your requirements or improvement suggestions.

What do you mean meaningless? Could you explain?

https://github.com/apache/skywalking/blob/c5df0760f257e523902b641755b1cf7e6cee4476/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/worker/MetricsPersistentWorker.java#L133-L147
If use Math.max, bathSize will always be astCollection.size() when astCollection.size() is grather than 2000. And then flushDataToStorage  all lastCollection。
If use Math.min, bathSize will  be 2000 when astCollection.size() is grather than 2000. And then flushDataToStorage  2000 size of
lastCollection every time.

Make sense, could you send the pull request to fix this?

Of course.

repo: apache/skywalking


text:
[Bug] Broken links on download page

Search before asking

[X] I had searched in the issues and found no similar issues.

Apache SkyWalking Component
Nginx Lua Agent (apache/skywalking-nginx-lua)
What happened
The download page links for the 0.6.0 lua code are broken.
Also the source for a current release MUST be served from the mirror system, i.e. it must be present under
https://downloads.apache.org/skywalking/nginx-lua/0.6.0/
It looks like the wrong names have been used here.
Please fix urgently
What you expected to happen
As above
How to reproduce
As above
Anything else
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hi, I checked the links and files, the file names were not in the correct format by the release manager, and kept latest in the name.
I fixed them in https://dist.apache.org/repos/dist/release/skywalking/nginx-lua/0.6.0/, but https://archive.apache.org/dist/skywalking/nginx-lua/0.6.0/ is not updated.
Is there a period task running in the background? Should we wait or do something?

This download link works now, https://www.apache.org/dyn/closer.cgi/skywalking/nginx-lua/0.6.0/skywalking-nginx-lua-0.6.0-src.tgz

Thanks, the download page is good now.
The archive may take up to a day to catch up.

authors: sebbASF wu-sheng


text:
fix  skywalking refister-receiver artifactId name

refs: https://github.com/apache/skywalking/blob/master/oap-server/server-receiver-plugin/pom.xml#L35

@JaredTan95 The label no update is for closing PR having no activity. The chore is for min/typo/no-important fix.

@JaredTan95 The label no update is for closing PR having no activity. The chore is for min/typo/no-important fix.

okay, got it.

Seems you mischange something?

Codecov Report

Merging #4697 into master will decrease coverage by 51.31%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master   #4697       +/-   ##
============================================
- Coverage     51.31%   0.00%   -51.32%
============================================
  Files          1276     146     -1130
  Lines         27692    3650    -24042
  Branches       3004     439     -2565
============================================
- Hits          14211       0    -14211
+ Misses        12838    3650     -9188
+ Partials        643       0      -643

Impacted Files
Coverage Δ
Complexity Δ

...king/apm/util/RunnableWithExceptionProtection.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (-2.00%)

...ywalking/apm/agent/core/logging/core/LogLevel.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (-1.00%)

...ywalking/apm/agent/core/plugin/EnhanceContext.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (-5.00%)

...ywalking/apm/agent/core/profile/ProfileStatus.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (-1.00%)

...walking/apm/agent/core/context/SW8CarrierItem.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (-2.00%)

...walking/apm/agent/core/jvm/cpu/SunCpuAccessor.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (-2.00%)

...walking/apm/agent/core/logging/core/LogOutput.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (-1.00%)

...walking/apm/agent/core/plugin/match/NameMatch.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (-3.00%)

...alking/apm/agent/core/context/CarrierItemHead.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (-1.00%)

...alking/apm/agent/core/jvm/gc/ParallelGCModule.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (-3.00%)

... and 1254 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cf7482a...266758d. Read the comment docs.

authors: JaredTan95 codecov-io wu-sheng


text:
Support PagerDuty Alarm Hook

[x] If this is non-trivial feature, paste the links/URLs to the design doc.

[x] Update the documentation to include this new feature.

[x] Tests(including UT, IT, E2E) are added to verify the new feature.

[x] If it's UI related, attach the screenshots below.

[x] Closes #7572.

[ ] Update the CHANGES log.

Example of triggered incidents on PagerDuty

@wu-sheng The changelog link in the PR template is no longer valid.
However, I found this one: https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md. Should I update it?

I can see template is updated, what do you mean?
https://github.com/apache/skywalking/blob/master/.github/PULL_REQUEST_TEMPLATE#L33

And you should make CI and e2e passed locally.
Please attach a screenshot to verify how this works

@wu-sheng OK, I'll fix CI / E2E first.

@wu-sheng Just fixed a checkStyle error. But I don't know how to run all the CI / E2E workflows locally, could you please show me how?

This is your first PR to this repository, so I(or another committer) have to approve for every commit you made. Those tasks would run automatically once this first PR gets merged.

Where are the values of these two fields from?

@wu-sheng The Title field is just the alarm message we're sending. In this case, the text template is set to "Apache SkyWalking Alarm: \n %s.", and the %s is replaced with the alarm message anotherAlarmMessage.setAlarmMessage("anotherAlarmMessage with [DefaultScopeDefine.Endpoint]"); , so the formatted message becomes the one you see.
The Service field has nothing to do with us. It's the name that the PagerDuty user gives to their service. Basically:

The user creates a service on PagerDuty and gives it a name (like in this case, "test service");
The user creates an "integration" for the service and copies the integration key;
The user configures the integration key(s) in our YAML file. So when an alarm is triggered, the message is sent to the service(s) that corresponds to the integration key(s).

@wu-sheng The changelog link in the PR template is no longer valid.
However, I found this one: https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md. Should I update it?

Oh, I noticed what is breaking here. If you want, feel free to fix this through another PR. Or I could do this fix too. Wait for your reply.

And don't forget to update changelog for your own PR. :)

@wu-sheng Yeah, I'd like to create another PR to fix the changelog link in the PR template :)
And just updated the changelog for this PR here: https://github.com/apache/skywalking/pull/9247/commits/1a48697e3ea32acefc22daba68780d4799f8a286

authors: chen-ni wu-sheng


text:
SLING-4734 - Sightly should try to instanciate Sling Models first via…

Updated JavaUseProvider to check Sling request for adaptability before current resource.

@lykorian  - the Jira issue has been resolved, can you please close the pull request?

authors: lykorian rombert

repo: apache/sling


text:
TestExportWriter fix, loosen order constraint

This test was asserting the docs from /export came in the same order as an equivalent from /select.  But that's too strong an expectation; only the sorted values should be equivalent.
It's remarkable to me how much code was written to test it as it was before... I wonder if I'm missing something by replacing it with something fairly trivial?

The following test was failing with this seed:

./gradlew :solr:core:test --tests "org.apache.solr.handler.export.TestExportWriter.testMultipleSorts"   -Ptests.seed=D7C851B5C905B099

I debugged it for a while; didn't fail due to any recent changes; isn't concurrency related.  Even optimizing to one segment still had the same test failure.

I'll merge this in a couple days.

repo: apache/solr


text:
[SPARK-15587] [ML] ML 2.0 QA: Scala APIs audit for ml.feature

What changes were proposed in this pull request?
ML 2.0 QA: Scala APIs audit for ml.feature. Mainly include:

Remove seed for QuantileDiscretizer, since we use approxQuantile to produce bins and seed is useless.
Scala API docs update.
Sync Scala and Python API docs for these changes.

How was this patch tested?
Exist tests.

Test build #59651 has started for PR 13410 at commit 570c333.

Test build #59651 has finished for PR 13410 at commit 570c333.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59651/
Test PASSed.

LGTM

Merged to master/branch-2.0

authors: AmplabJenkins MLnick SparkQA yanboliang

repo: apache/spark


text:
[SPARK-15950][SQL] Eliminate unreachable code at projection for complex types

What changes were proposed in this pull request?
This PR eliminates unreachable code at projection for complex types (e.g. array, map, and struct)  to reduce the total size of Java byte code in a method generated by whole stage codegen.
This PR eliminates the following code blocks in the following cases:

project_value is null
each element of project_values is null if it never occurs

Examples for complex types
    val df = sparkContext.parallelize(Seq(0.1, 0.2), 1).toDF("v")
    df.selectExpr("Array(v + 2.2, v + 3.3)").collect
    df.selectExpr("struct(v + 3, v + 4)").collect
    df.selectExpr("map(v + 3, v + 4)").collect

Before applying this PR into Array
/* 028 */   protected void processNext() throws java.io.IOException {
/* 029 */     while (inputadapter_input.hasNext()) {
/* 030 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();
/* 031 */       final boolean project_isNull = false;
/* 032 */       this.project_values = new Object[2];
/* 033 */       double inputadapter_value = inputadapter_row.getDouble(0);
/* 034 */
/* 035 */       double project_value1 = -1.0;
/* 036 */       project_value1 = inputadapter_value + 2.2D;
/* 037 */       if (false) {
/* 038 */         project_values[0] = null;
/* 039 */       } else {
/* 040 */         project_values[0] = project_value1;
/* 041 */       }
/* 042 */
/* 043 */       double inputadapter_value1 = inputadapter_row.getDouble(1);
/* 044 */
/* 045 */       double project_value4 = -1.0;
/* 046 */       project_value4 = inputadapter_value1 + 3.3D;
/* 047 */       if (false) {
/* 048 */         project_values[1] = null;
/* 049 */       } else {
/* 050 */         project_values[1] = project_value4;
/* 051 */       }
/* 052 */
/* 053 */       final ArrayData project_value = new org.apache.spark.sql.catalyst.util.GenericArrayData(project_values);
/* 054 */       this.project_values = null;
/* 055 */       project_holder.reset();
/* 056 */
/* 057 */       project_rowWriter.zeroOutNullBytes();
/* 058 */
/* 059 */       if (project_isNull) {
/* 060 */         project_rowWriter.setNullAt(0);
/* 061 */       } else {
/* 062 */         // Remember the current cursor so that we can calculate how many bytes are
/* 063 */         // written later.
/* 064 */         final int project_tmpCursor = project_holder.cursor;
/* 065 */
/* 066 */         if (project_value instanceof UnsafeArrayData) {
/* 067 */           final int project_sizeInBytes = ((UnsafeArrayData) project_value).getSizeInBytes();
/* 068 */           // grow the global buffer before writing data.
/* 069 */           project_holder.grow(project_sizeInBytes);
/* 070 */           ((UnsafeArrayData) project_value).writeToMemory(project_holder.buffer, project_holder.cursor);
/* 071 */           project_holder.cursor += project_sizeInBytes;
/* 072 */
/* 073 */         } else {
/* 074 */           final int project_numElements = project_value.numElements();
/* 075 */           project_arrayWriter.initialize(project_holder, project_numElements, 8);
/* 076 */
/* 077 */           for (int project_index = 0; project_index < project_numElements; project_index++) {
/* 078 */             if (project_value.isNullAt(project_index)) {
/* 079 */               project_arrayWriter.setNullAt(project_index);
/* 080 */             } else {
/* 081 */               final double project_element = project_value.getDouble(project_index);
/* 082 */               project_arrayWriter.write(project_index, project_element);
/* 083 */             }
/* 084 */           }
/* 085 */         }
/* 086 */
/* 087 */         project_rowWriter.setOffsetAndSize(0, project_tmpCursor, project_holder.cursor - project_tmpCursor);
/* 088 */         project_rowWriter.alignToWords(project_holder.cursor - project_tmpCursor);
/* 089 */       }
/* 090 */       project_result.setTotalSize(project_holder.totalSize());
/* 091 */       append(project_result);
/* 092 */       if (shouldStop()) return;
/* 093 */     }
/* 094 */   }

After applying this PR into Array
/* 028 */   protected void processNext() throws java.io.IOException {
/* 029 */     while (inputadapter_input.hasNext()) {
/* 030 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();
/* 031 */       final boolean project_isNull = false;
/* 032 */       this.project_values = new Object[2];
/* 033 */       double inputadapter_value = inputadapter_row.getDouble(0);
/* 034 */
/* 035 */       double project_value1 = -1.0;
/* 036 */       project_value1 = inputadapter_value + 2.2D;
/* 037 */       project_values[0] = project_value1;
/* 038 */       double inputadapter_value1 = inputadapter_row.getDouble(1);
/* 039 */
/* 040 */       double project_value4 = -1.0;
/* 041 */       project_value4 = inputadapter_value1 + 3.3D;
/* 042 */       project_values[1] = project_value4;
/* 043 */       final ArrayData project_value = new org.apache.spark.sql.catalyst.util.GenericArrayData(project_values);
/* 044 */       this.project_values = null;
/* 045 */       project_holder.reset();
/* 046 */
/* 047 */       // Remember the current cursor so that we can calculate how many bytes are
/* 048 */       // written later.
/* 049 */       final int project_tmpCursor = project_holder.cursor;
/* 050 */
/* 051 */       if (project_value instanceof UnsafeArrayData) {
/* 052 */         final int project_sizeInBytes = ((UnsafeArrayData) project_value).getSizeInBytes();
/* 053 */         // grow the global buffer before writing data.
/* 054 */         project_holder.grow(project_sizeInBytes);
/* 055 */         ((UnsafeArrayData) project_value).writeToMemory(project_holder.buffer, project_holder.cursor);
/* 056 */         project_holder.cursor += project_sizeInBytes;
/* 057 */
/* 058 */       } else {
/* 059 */         final int project_numElements = project_value.numElements();
/* 060 */         project_arrayWriter.initialize(project_holder, project_numElements, 8);
/* 061 */
/* 062 */         for (int project_index = 0; project_index < project_numElements; project_index++) {
/* 063 */           if (project_value.isNullAt(project_index)) {
/* 064 */             project_arrayWriter.setNullAt(project_index);
/* 065 */           } else {
/* 066 */             final double project_element = project_value.getDouble(project_index);
/* 067 */             project_arrayWriter.write(project_index, project_element);
/* 068 */           }
/* 069 */         }
/* 070 */       }
/* 071 */
/* 072 */       project_rowWriter.setOffsetAndSize(0, project_tmpCursor, project_holder.cursor - project_tmpCursor);
/* 073 */       project_rowWriter.alignToWords(project_holder.cursor - project_tmpCursor);
/* 074 */       project_result.setTotalSize(project_holder.totalSize());
/* 075 */       append(project_result);
/* 076 */       if (shouldStop()) return;
/* 077 */     }
/* 078 */   }

How was this patch tested?
Added some unit tests into DataFrameComplexTypeSuite

Test build #60495 has started for PR 13663 at commit 9e7b91d.

Test build #60495 has finished for PR 13663 at commit 9e7b91d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60495/
Test PASSed.

IIUC, this PR adds a global variable to eliminate some if branches that are very easy to be optimized by javac or JIT.
I think this is not a good tradeoff, the readability of the scala code is much more important than the readability(or code size) of the generated java code.
cc @davies

+1 for @cloud-fan said.

@cloud-fan and @davies , thank you for your comments.
A global variable is used for 1. I will address 1. by using another approach without using a global variable in another PR. This PR will focus on 2. and 3. that are addressed without using a global variable.

Test build #60569 has started for PR 13663 at commit 4458f4f.

Test build #60569 has finished for PR 13663 at commit 4458f4f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60569/
Test PASSed.

Test build #60582 has started for PR 13663 at commit a193007.

Test build #60582 has finished for PR 13663 at commit a193007.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60582/
Test PASSed.

Test build #60639 has started for PR 13663 at commit 8d7d311.

Test build #60639 has finished for PR 13663 at commit 8d7d311.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60639/
Test PASSed.

@cloud-fan , @davies , I would appreciate it if you would look at this again. I think that I had addressed all of your comments.

Test build #60939 has started for PR 13663 at commit 7ad542d.

Test build #60939 has finished for PR 13663 at commit 7ad542d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60939/
Test PASSed.

@cloud-fan , @davies , I would appreciate it if you would look at this again.

https://github.com/apache/spark/pull/16008 implemented the same solution.

authors: AmplabJenkins SparkQA cloud-fan davies kiszk


text:
[SPARK-18124] Observed delay based Event Time Watermarks

This PR adds a new method withWatermark to the Dataset API, which can be used specify an event time watermark.  An event time watermark allows the streaming engine to reason about the point in time after which we no longer expect to see late data.  This PR also has augmented StreamExecution to use this watermark for several purposes:

To know when a given time window aggregation is finalized and thus results can be emitted when using output modes that do not allow updates (e.g. Append mode).
To minimize the amount of state that we need to keep for on-going aggregations, by evicting state for groups that are no longer expected to change.  Although, we do still maintain all state if the query requires (i.e. if the event time is not present in the groupBy or when running in Complete mode).

An example that emits windowed counts of records, waiting up to 5 minutes for late data to arrive.
df.withWatermark("eventTime", "5 minutes")
  .groupBy(window($"eventTime", "1 minute") as 'window)
  .count()
  .writeStream
  .format("console")
  .mode("append") // In append mode, we only output finalized aggregations.
  .start()

Calculating the watermark.
The current event time is computed by looking at the MAX(eventTime) seen this epoch across all of the partitions in the query minus some user defined delayThreshold.  An additional constraint is that the watermark must increase monotonically.
Note that since we must coordinate this value across partitions occasionally, the actual watermark used is only guaranteed to be at least delay behind the actual event time.  In some cases we may still process records that arrive more than delay late.
This mechanism was chosen for the initial implementation over processing time for two reasons:

it is robust to downtime that could affect processing delay
it does not require syncing of time or timezones between the producer and the processing engine.

Other notable implementation details

A new trigger metric eventTimeWatermark outputs the current value of the watermark.
We mark the event time column in the Attribute metadata using the key spark.watermarkDelay.  This allows downstream operations to know which column holds the event time.  Operations like window propagate this metadata.
explain() marks the watermark with a suffix of -T${delayMs} to ease debugging of how this information is propagated.
Currently, we don't filter out late records, but instead rely on the state store to avoid emitting records that are both added and filtered in the same epoch.

Remaining in this PR

[ ] The test for recovery is currently failing as we don't record the watermark used in the offset log.  We will need to do so to ensure determinism, but this is deferred until #15626 is merged.

Other follow-ups
There are some natural additional features that we should consider for future work:

Ability to write records that arrive too late to some external store in case any out-of-band remediation is required.
Update mode so you can get partial results before a group is evicted.
Other mechanisms for calculating the watermark.  In particular a watermark based on quantiles would be more robust to outliers.

Test build #67839 has started for PR 15702 at commit 5b92132.

Test build #67839 has finished for PR 15702 at commit 5b92132.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67839/
Test FAILed.

@ekl - flaky test...  Should we turn it off for now?
retest this please

I'm still trying to find a failure that includes https://github.com/apache/spark/pull/15701/files. Until then it's hard to debug.

Test build #67866 has started for PR 15702 at commit 14a728e.

Test build #67866 has finished for PR 15702 at commit 14a728e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67866/
Test PASSed.

A very dumb question (I apologize), there is nothing stopping a user to actually use processing time as watermarks with this API either. One can easily do df.withColumn("timestamp", current_timestamp()).withWatermark("timestamp"). I like that we're suggesting users to use eventTime for stability, but we're not actually constraining them, right?
My biggest confusion here, that I couldn't find documented was the Type of the watermark column. Does it need to be timestamp type or can it be LongType?

Not a dumb question!  You can certainly use processing time if those are the semantics you require.  I do think there is a little bit of work we need to do to ensure determinism for these functions.  Specifically, now() in SQL is supposed to return the moment query execution began.  For streaming we'll need to record this and make sure that we give the same time even in the case of failures (based on the batch id with the current execution model).
Good point on the documentation.  The thing you are missing is that it must be used in a window function, which does require TimestampType.  I can see how to make this more clear.

Test build #67939 has started for PR 15702 at commit 311e7c0.

Test build #67939 has finished for PR 15702 at commit 311e7c0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67939/
Test PASSed.

Given the concerns Ofir raised about a single far future event screwing up monotonic event time, do you want to document that problem even if there isn't an enforced filter for it?

Test build #67998 has started for PR 15702 at commit 2685771.

Test build #67998 has finished for PR 15702 at commit 2685771.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67998/
Test FAILed.

Test build #3397 has started for PR 15702 at commit 2685771.

Test build #3398 has started for PR 15702 at commit 2685771.

Test build #3397 has finished for PR 15702 at commit 2685771.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test build #3398 has finished for PR 15702 at commit 2685771.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test build #68023 has started for PR 15702 at commit 379255d.

Test build #68025 has started for PR 15702 at commit 7a9b6dd.

Test build #68023 has finished for PR 15702 at commit 379255d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68023/
Test FAILed.

Test build #68025 has finished for PR 15702 at commit 7a9b6dd.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68025/
Test FAILed.

Merged build finished. Test FAILed.

Test build #68029 has started for PR 15702 at commit 1d4784f.

Test build #68029 has finished for PR 15702 at commit 1d4784f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68029/
Test PASSed.

Test build #68496 has started for PR 15702 at commit de601bb.

LGTM, pending tests.

Test build #68496 has finished for PR 15702 at commit de601bb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68496/
Test FAILed.

jenkins, test this please

Test build #68504 has started for PR 15702 at commit de601bb.

Test build #68504 has finished for PR 15702 at commit de601bb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68504/
Test PASSed.

Test build #68631 has started for PR 15702 at commit 87d8618.

Test build #68631 has finished for PR 15702 at commit 87d8618.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68631/
Test FAILed.

jenkins test this please

Test build #68637 has started for PR 15702 at commit 87d8618.

Test build #68637 has finished for PR 15702 at commit 87d8618.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68637/
Test PASSed.

I am merging this to master and 2.1

authors: AmplabJenkins SparkQA brkyvz ericl koeninger marmbrus tdas


text:
[SPARK-18808][ML][MLLIB] ml.KMeansModel.transform is very inefficient

What changes were proposed in this pull request?
mllib.KMeansModel.clusterCentersWithNorm is a method than ends up being called every time predict is called on a single vector, which is bad news for now the ml.KMeansModel Transformer works, which necessarily transforms one vector at a time.
This causes the model to just store the vectors with norms upfront. The extra norm should be small compared to the vectors. This would avoid this form of overhead on this and other code paths.
How was this patch tested?
Existing tests.

Test build #70318 has started for PR 16328 at commit cddf553.

Test build #70318 has finished for PR 16328 at commit cddf553.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70318/
Test FAILed.

Merged build finished. Test FAILed.

Test build #70320 has started for PR 16328 at commit b68b230.

Test build #70320 has finished for PR 16328 at commit b68b230.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70320/
Test FAILed.

Test build #70323 has started for PR 16328 at commit b04f083.

I finished this approach for completeness, but I see that it requires 'fixing' StreamingKMeansModel to return a new model instead of strictly updating its vectors in place. It's not actually a big deal but would cause a problem if anyone called StreamingKMeansModel.update without using its return value. Spark doesn't and I'm not sure if it's valid.
There's another way to approach this by passing around a broadcast variable. It's narrower and a bit more complex, so it'd be nice if we could make this change, but it's not the only solution. Leaving it open for comments, esp. from Jenkins.
CC @freeman-lab and @mengxr

Test build #70323 has finished for PR 16328 at commit b04f083.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70323/
Test PASSed.

Ping @jkbradley or mabye @MLnick for an opinion on whether this is an OK solution. Certainly it's a nicer fix than the alternative but has a slight behavior change.

Going once going twice -- I favor making this change but want to see if anyone feels at all strongly about it.

Merged to master

authors: AmplabJenkins SparkQA srowen


text:
[SPARK-19724][SQL]create a managed table with an existed default table should throw an exception

What changes were proposed in this pull request?
This JIRA is a follow up work after SPARK-19583
As we discussed in that PR
The following DDL for a managed table with an existed default location should throw an exception:
CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ...
CREATE TABLE ... (PARTITIONED BY ...)

Currently there are some situations which are not consist with above logic:

CREATE TABLE ... (PARTITIONED BY ...) succeed with an existed default location
situation: for both hive/datasource(with HiveExternalCatalog/InMemoryCatalog)

CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ...
situation: hive table succeed with an existed default location

This PR is going to make above two situations consist with the logic that it should throw an exception
with an existed default location.
How was this patch tested?
unit test added

Test build #74436 has started for PR 17272 at commit a0ba419.

Test build #74436 has finished for PR 17272 at commit a0ba419.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74436/
Test FAILed.

Test build #74437 has started for PR 17272 at commit 65d7ea9.

Test build #74441 has started for PR 17272 at commit 0e753cc.

Test build #74437 has finished for PR 17272 at commit 65d7ea9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74437/
Test FAILed.

Test build #74441 has finished for PR 17272 at commit 0e753cc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74441/
Test FAILed.

retest this please

Test build #74445 has started for PR 17272 at commit c8d9b77.

Test build #74445 has finished for PR 17272 at commit c8d9b77.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74445/
Test FAILed.

retest this please

Test build #74468 has started for PR 17272 at commit c8d9b77.

Test build #74472 has started for PR 17272 at commit cd4a091.

Test build #74468 has finished for PR 17272 at commit c8d9b77.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74468/
Test FAILed.

Test build #74472 has finished for PR 17272 at commit cd4a091.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74472/
Test FAILed.

Merged build finished. Test FAILed.

Test build #74493 has started for PR 17272 at commit 2ac70b4.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74493/
Test FAILed.

retest this please

Test build #74499 has started for PR 17272 at commit 2ac70b4.

Test build #74499 has finished for PR 17272 at commit 2ac70b4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74499/
Test FAILed.

Test build #74516 has started for PR 17272 at commit 516c4e4.

Test build #74516 has finished for PR 17272 at commit 516c4e4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74516/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #74524 has started for PR 17272 at commit 516c4e4.

Test build #74524 has finished for PR 17272 at commit 516c4e4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74524/
Test FAILed.

Merged build finished. Test FAILed.

Test build #74535 has started for PR 17272 at commit 739f207.

Test build #74535 has finished for PR 17272 at commit 739f207.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74535/
Test FAILed.

Test build #74604 has started for PR 17272 at commit 4351cd7.

Test build #74604 has finished for PR 17272 at commit 4351cd7.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74604/
Test FAILed.

Merged build finished. Test FAILed.

@windpiger is this still active?

cc @ltnwgl Could you please take this over? Thanks!

cc @dongjoon-hyun Are you interested in this PR? Just take it over?

@gatorsmile I will take it over :)

authors: AmplabJenkins HyukjinKwon SparkQA gatorsmile gengliangwang windpiger


text:
[SPARK-20498][PYSPARK][ML] Expose getMaxDepth for ensemble tree model in PySpark

What changes were proposed in this pull request?
add getMaxDepth method for ensemble tree models:

RandomForestClassifierModel
RandomForestRegressionModel
GBTClassificationModel
GBTRegressionModel

How was this patch tested?

[ ] pass all unit tests.
[ ] add new doctest.

@keypointt Hi, could you help check the pr is consistent with your #17207 ? Thanks.

Can one of the admins verify this patch?

Hi Facai, you exposed api getMaxDepth() in TreeEnsembleModel, and the rest changes are in comment and not being run.
I'd suggest you put the tests in test module and run them to verify

Hi, @keypointt . It's the feature of Python. The doctest is both document and unit test.

Oh sorry i missed it, thanks for the heads up.
On Sat, May 27, 2017 at 2:16 PM Yan Facai (颜发才) notifications@github.com
wrote:

Hi, @keypointt https://github.com/keypointt . It's the feature of
Python. The doctest is both document and unit test.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/18120#issuecomment-304477003, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADvmieVs45e2uCq-g3NV2ViMyc8AbiDEks5r-JJ4gaJpZM4NnVv4
.

cc @BryanCutler.
Bryan did some work on https://github.com/apache/spark/pull/17849. It seems even with that patch, we still need to add methods like these, hoping Bryan can confirm. If we're going to add some param accessors to the models, best to do them all at once yes? E.g. we can also add getMaxBins and others.

Thanks @facaiy for the PR.  This might be enough to simply retrieve the value from the Java model, but I think the Python model also needs to "own" the param.  For example, if we have a DecisionTreeRegressor called dt and a DecisionTreeRegressionModel called model then
In [8]: dt.hasParam("maxDepth")
Out[8]: True

In [9]: model.hasParam("maxDepth")
Out[9]: False

This is because the Python object does not have an instance of the param, its only getting a value from the Java model.  Additionally, many of the methods you would expect to work from class Params would raise an error like
In [4]: dt.explainParam("maxDepth")
Out[4]: 'maxDepth: Maximum depth of the tree. (>= 0) E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes. (default: 5, current: 2)

In [5]: model.explainParam("maxDepth")
...
AttributeError: 'DecisionTreeRegressionModel' object has no attribute 'maxDepth'

As @sethah pointed out #17849 has the fix so that the Python models would have an instance of each param, so that should go in first.  Then, the accessor could be written like this:
def getMaxDepth(self):
    return self.getOrDefault(self.maxDepth)

I'm not sure what the best approach for adding these accessors, all at once or one by one as needed, like with maxDepth?
cc @holdenk @jkbradley for your input

Thanks, @BryanCutler.
It seems that #17849 copys Params from Estimator to Model automatically, which is pretty useful. However, getter method is still missing and need to be added manually as the pr.
If only model could inherit both params and its getter method, perhaps it will be better in my opinion.
Anyway, it is OK whether the pr will be merged or not, as it is trivial.

authors: AmplabJenkins BryanCutler facaiy keypointt sethah


text:
[SPARK-23146][WIP] Support client mode for Kubernetes in Out-Cluster mode

What changes were proposed in this pull request?
The changes allow to support Kubernetes resource manager in client mode (upon the existing cluster mode)
How was this patch tested?
The initial changes were done on the latest commits in the spark-k8s fork (https://github.com/apache-spark-on-k8s/spark) and have been tested on AWS with real data processing.
In an effort to merge back the latests features to apache master, I open here untested changes subject to feedback and discussion.
Documentation will be updated when code will be discussed, but in the meantime there is a indigest design document that can be read to know more about the changes. In- and Out- K8s Cluster considerations, as deps and hdfs access is discussed there.
Upon the current design and implementation constructs, an open point I have is about the way we wanna configure the path of the k8s config in case of OutCluster mode. Options are:

Force use to specify the path and fail if this property is not given
In case of absence of /var/run/secrets/kubernetes.io/serviceaccount/token (which is there for InCluster), fall back automatically to the given property, or if no property has been given, fallback to the $HOME/.kube/config (in this latter case, there is no separate cacert nor keyfile, those details are all bundled in the single $HOME/.kube/config file).

The tests so far have been done with separated config, cacert and key files (I guess the single config file should not give any issue).
A last important point is how we move forward with this for the merge. To have a client mode better coverage, it would be interesting to have also downstream https://github.com/apache-spark-on-k8s/spark/pull/540 which is not only Kerberos, but also the Hadoop steps much needed to mount Hadoop conf to connect HDFS from Driver/Executors.
Also to avoid mess in future merge, I list here the changes I had to deal with applying the patch on the apache master repo:

submitsteps package is steps
no OptionRequirements class (used in SparkKubernetesClientFactory)
no ExecutorLocalDirVolumeProvider in ExecutorPodFactory
no APISERVER_AUTH_DRIVER_MOUNTED_CONF_PREFIX in config.scala

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1755/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1756/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1757/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/1735/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/1735/

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1779/
Test PASSed.

Merged build finished. Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/1756/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/1756/

Current state works for out-cluster client mode.
For in-cluster, we miss the DriverServiceBootstrapStep of the fork which "Allows the driver to be reachable by executor pods through a headless service. The service's ports should correspond to the ports that the executor will reach the pod at for RPC."
Running in cluster gives exception:
Caused by: java.io.IOException: Failed to connect to spark-pod:43957
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)
	at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198)
Caused by: java.net.UnknownHostException: spark-pod
...

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1784/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/1761/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/1761/

May I ask, what's the release plan for this?

Now that #20910  has been merged, I will update this PR to take account the refactoring. @inetfuture Once these changes are pushed, there is the review process which needs to occur, so difficult to give you a plan.

I have updated this branch and tested successfully client mode OutCluster.
Happy to get confirmation of this from anyone here (cc/ @foxish)
For InCluster (was working fine before ##20910), worker starts but fails due to UnknownHostException: The worker can not resolve the Driver hostname): @mccheah was there anything changed in terms of HeadlessDriver step during the #20910 refactoring?
k logs -f spark-exec-1
++ id -u
+ myuid=0
++ id -g
+ mygid=0
+ set +e
++ getent passwd 0
+ uidentry=root:x:0:0:root:/root:/bin/ash
+ set -e
+ '[' -z root:x:0:0:root:/root:/bin/ash ']'
+ SPARK_K8S_CMD=executor
+ '[' -z executor ']'
+ shift 1
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sort -t_ -k4 -n
+ sed 's/[^=]*=\(.*\)/\1/g'
+ readarray -t SPARK_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -n '' ']'
+ case "$SPARK_K8S_CMD" in
+ CMD=(${JAVA_HOME}/bin/java "${SPARK_JAVA_OPTS[@]}" -Xms$SPARK_EXECUTOR_MEMORY -Xmx$SPARK_EXECUTOR_MEMORY -cp "$SPARK_CLASSPATH" org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url $SPARK_DRIVER_URL --executor-id $SPARK_EXECUTOR_ID --cores $SPARK_EXECUTOR_CORES --app-id $SPARK_APPLICATION_ID --hostname $SPARK_EXECUTOR_POD_IP)
+ exec /sbin/tini -s -- /usr/lib/jvm/java-1.8-openjdk/bin/java -Xms1g -Xmx1g -cp ':/opt/spark/jars/*' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@spark-pod:33913 --executor-id 1 --cores 1 --app-id spark-application-1525426093472 --hostname 172.17.0.9
2018-05-04 09:28:16 INFO  CoarseGrainedExecutorBackend:2502 - Started daemon with process name: 15@spark-exec-1
2018-05-04 09:28:16 INFO  SignalUtils:54 - Registered signal handler for TERM
2018-05-04 09:28:16 INFO  SignalUtils:54 - Registered signal handler for HUP
2018-05-04 09:28:16 INFO  SignalUtils:54 - Registered signal handler for INT
2018-05-04 09:28:17 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-05-04 09:28:17 INFO  SecurityManager:54 - Changing view acls to: root
2018-05-04 09:28:17 INFO  SecurityManager:54 - Changing modify acls to: root
2018-05-04 09:28:17 INFO  SecurityManager:54 - Changing view acls groups to:
2018-05-04 09:28:17 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-05-04 09:28:17 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1904)
	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:65)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:281)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
	at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
	at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:201)
	at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:66)
	at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:65)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1886)
	... 4 more
Caused by: java.io.IOException: Failed to connect to spark-pod:33913
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)
	at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
	at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198)
	at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
	at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: spark-pod
	at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at java.net.InetAddress.getByName(InetAddress.java:1076)
	at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:146)
	at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:143)
	at java.security.AccessController.doPrivileged(Native Method)
	at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:143)
	at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:43)
	at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:63)
	at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:55)
	at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:57)
	at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:32)
	at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:108)
	at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:208)
	at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:49)
	at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:188)
	at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:174)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:978)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:512)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	... 1 more

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2904/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2828/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2828/

@echarles I'm wondering do you have time to sync this with the master and make it to a shape that's ready for review? Thanks!

Just wanted to say this PR is really great, with this it's possible to for example use Jupyter with Spark on K8s:

@rayburgemeestre This is cool!
Apache Toree configuration for K8S is on my todo list but would be happy to copycat your conf...
Any gist?
IMHO both client and cluster mode should be fine with Toree. Did you try cluster mode?

Yes, absolutely, but I made those changes in a hardcoded way though, just to try it out. So I'm not sure how helpful it is.
Near the end of this file: /usr/local/share/jupyter/kernels/apache_toree_pyspark/bin/run.sh (which is invoked by the pyspark kernel)
Added these options:
    --master k8s://https://rb-spark:6443 \
    --deploy-mode cluster \
    --conf spark.executor.instances=3 \
    --conf spark.kubernetes.container.image=node001:5000/brightcomputing/spark-py:v2.3.0 \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \

So the exec call looks like this:
eval exec \
     "${SPARK_HOME}/bin/spark-submit" \
     --name "'Apache Toree'" \
     "${SPARK_OPTS}" \
     --master k8s://https://rb-spark:6443 \
     --conf spark.executor.instances=3 \
     --conf spark.kubernetes.container.image=node001:5000/brightcomputing/spark-py:v2.3.0 \
     --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
     --class org.apache.toree.Main \
     "${TOREE_ASSEMBLY}" \
     "${TOREE_OPTS}" \
     "$@"

A few notes:

node001:5000 is my docker registry
I merged some changes from master into your PR before manually building the 2.4.0-SNAPSHOT jar file. Changes that make an additional "spark-py" image available) The PR for that was: https://github.com/apache/spark/pull/21092
The spark image in my shell script says "spark-py:v2.3.0" but it's actually a newer version v2.4.0-SNAPSHOT. I just called the docker_image_tool.sh script with the wrong version number in this case.

@liyinan926 I pushed the merge with master and tested OutCluster (I didn't test InCluster but as mentioned previously, we would need something similar to the DriverServiceBootstrapStep of the fork).
@squito @tnachen Thx for your reviews and comments which make sense. My approach is to change what is needed for the client-mode. I prefer leaving constructs, code removal and enhancement considerations in other PRs.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/820/

@rayburgemeestre Thx for sharing you Toree conf. Are you sure you are running in client mode? (I don't see something similar to --conf spark.submit.deployMode=client). Maybe it just works in cluster mode (with the extra cost of 1 driver pod)?

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/820/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/820/
Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/820/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/821/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/821/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/821/
Test PASSed.

@mccheah who mentioned a plan to create a separate change for client mode support.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/855/
Test FAILed.

@liyinan926 Thx for you comments (see my answers). I have merged the last changes, added a tiny doc section and pushed again.

@echarles I'm planning to push a different change that takes a different approach from this.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/856/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/856/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/856/
Test PASSed.

See https://github.com/apache/spark/pull/21748

See #21748

authors: AmplabJenkins SparkQA echarles inetfuture liyinan926 mccheah rayburgemeestre


text:
[MINOR][CORE][TEST] Fix afterEach() in TastSetManagerSuite and TaskSchedulerImplSuite

What changes were proposed in this pull request?
In the afterEach() method of both TastSetManagerSuite and TaskSchedulerImplSuite, super.afterEach() shall be called at the end, because it shall stop the SparkContext.
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93706/testReport/org.apache.spark.scheduler/TaskSchedulerImplSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/
The test failure is caused by the above reason, the newly added barrierCoordinator required rpcEnv which has been stopped before TaskSchedulerImpl doing cleanup.
How was this patch tested?
Existing tests.

@jiangxb1987, thanks! I am a bot who has found some folks who might be able to help with the review:@squito, @kayousterhout and @mateiz

Test build #93727 has finished for PR 21908 at commit 0bb6a5b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93727/
Test PASSed.

good catch, LGTM

Merged to master, branch-2.3 and branch-2.2.

late lgtm from me too

authors: AmplabJenkins HyukjinKwon SparkQA holdensmagicalunicorn jiangxb1987 kiszk squito


text:
[SPARK-28140][MLLIB][PYTHON] Accept DataFrames in RowMatrix and IndexedRowMatrix constructors

What changes were proposed in this pull request?
In both cases, the input DataFrame schema must contain only the information that's required for the matrix object, so a vector column in the case of RowMatrix and long and vector columns for IndexedRowMatrix.
How was this patch tested?
Unit tests that verify:

RowMatrix and IndexedRowMatrix can be created from DataFrames
If the schema does not match expectations, we throw an IllegalArgumentException

Please review https://spark.apache.org/contributing.html before opening a pull request.

Can one of the admins verify this patch?

add to whitelist

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Thanks for the initial look @srowen. I fixed the accidental import changes.
@jkbradley Looks like the incantation to enable tests didn't work

Test build #4815 has started for PR 24953 at commit d620b43.

Test build #4815 has finished for PR 24953 at commit d620b43.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #4818 has started for PR 24953 at commit 4a40143.

Test build #4818 has finished for PR 24953 at commit 4a40143.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged to master

Thanks @srowen !

authors: AmplabJenkins SparkQA henrydavidge jkbradley srowen


text:
[SPARK-29201][INFRA][2.4] Add Hadoop 2.6 combination to GitHub Action

What changes were proposed in this pull request?
This PR aims to add Hadoop 2.6 combination to branch-2.4 GitHub Action.
Why are the changes needed?
This will help branch-2.4 maintenance to prevent Hadoop-2.6 related issue at PR stages.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
This PR's GitHub Action should show both Hadoop 2.6/2.7.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16208/
Test PASSed.

Merged build finished. Test PASSed.

Test build #111129 has started for PR 25886 at commit d2e6f48.

Test build #111129 has finished for PR 25886 at commit d2e6f48.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111129/
Test PASSed.

Merged to branch-2.4.

Thank you for review and merging, @HyukjinKwon !

authors: AmplabJenkins HyukjinKwon SparkQA dongjoon-hyun


text:
[SPARK-31988][SQL] Schema pruning may discard attribute metadata

What changes were proposed in this pull request?
Fixing the getRootFields function to preserve attribute metadata
Why are the changes needed?
This can lead to a potential loss of metadata on an attribute in some code paths.
For example - when reading a parquet file with a schema that has metadata and writing it back the parquet footer it will not contain the metadata that was originally there.
code snippet to reproduce (assuming datasource v2):
    // create custom dataset
    val data = Seq(
      Row("a", "b")
    )
    val schema = List(
      StructField("col_a", StringType, true,
        new sql.types.MetadataBuilder().putString("key", "value").build()),
      StructField("col_b", StringType, true)
    )
    val df = spark.createDataFrame(
      spark.sparkContext.parallelize(data),
      StructType(schema)
    )
    // write
    df.write.parquet("/tmp/check")
    // read and verify the metadata exists
    val readDF = spark.read.parquet("/tmp/check")
    readDF.schema.foreach(s => println(s.metadata))
    // write again
    readDF.write.parquet("/tmp/check2")
    // read again and verify no metadata
    val readDF2 = spark.read.parquet("/tmp/check2")
    readDF2.schema.foreach(s => println(s.metadata))

Note that this doesn't happen in datasource v1.
Does this PR introduce any user-facing change?
No
How was this patch tested?
No tests were added as this is a minor change to a private function and no tests exist to check this function so far.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Could you add some tests first? Thanks!

ok to test

Test build #124019 has started for PR 28826 at commit 0c46105.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28638/
Test PASSed.

Test build #124019 has finished for PR 28826 at commit 0c46105.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124019/
Test FAILed.

retest this please

cc @viirya FYI

Test build #124044 has started for PR 28826 at commit 0c46105.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28663/
Test PASSed.

@maropu which test do you suggest to add?
This is private function which is not tested anywhere also not all file formats are able to save the metadata (for example csv) so seems to me that adding a test to SchemaPruningSuite is not the right thing.
I can add a test similar to the above code snippet, if that seems ok to you where would you suggest to put it?

Test build #124044 has finished for PR 28826 at commit 0c46105.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124044/
Test PASSed.

Basically, I think we need the two types of tests: end-2-end (coarse-grained) tets and fine-grained tests. For the former tests, how about adding some tests in DataFrameSuite?

+1, for @maropu 's suggestion. You can use your example in the PR description, @guykhazma .

BTW, @guykhazma . Is the example in the PR description enough? If I follow the direction, the metadata is not lost.
scala> sc.version
res1: String = 3.0.0
...
{"key":"value"}
{}
{"key":"value"}
{}

@dongjoon-hyun did you make sure to run it with datasource v2?
I will add tests for both and a better explanation later today.

Test build #124340 has started for PR 28826 at commit 2664737.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28959/
Test PASSed.

@HyukjinKwon @maropu @dongjoon-hyun @viirya added a test and also another fix that was needed - the function sortLeftFieldsByRight also ignored the metadata field.
As for why it happens only for v2 is because V2ScanRelationPushDown calls pruneColumns which trigger this code path (see here)

Test build #124340 has finished for PR 28826 at commit 2664737.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124340/
Test PASSed.

Thanks for updating, @guykhazma .

While v1 calls for prunePhysicalColumns
https://github.com/apache/spark/blob/d2a656c81ef784657a02e7347bfe87e4331fd2c9/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SchemaPruning.scala#L63
and the property root.derivedFromAtt is set to true so the structType from requestedRootFields is not used.

Thanks for the update, @guykhazma. So, does it mean if we have derivedFromAtt is false on any requested root field, v1 path will also discard attribute metadata. It sounds like we can also have a test case for that?

@viirya derivedFromAtt is set to false when the expression is requesting a nested field .
The metadata for a nested column is not preserved (also in Spark 2.4) so I am not sure what the expected behaviour should be here.
Note that the metadata for nested field also not preserved when using:
df.select("col_a.name").schema

when df is the dataframe that was created locally with the specified schema.
If this is considered a bug then we can resolve this as well (will require some more changes).
For example this test triggers a code path where derivedFromAtt is false but it currently passes since metadata is not preserved for nested columns:
  test("SPARK-31988 - make sure schema metadata is preserved - nested schema") {
    withSQLConf((SQLConf.USE_V1_SOURCE_LIST.key, "avro,csv,json,kafka,orc,text,parquet")) {
      withTempPath{ f =>
        // create custom dataset with schema metadata
        val data = Seq(
          Row(Row("a", 45), "b")
        )
        val schema = List(
          StructField("col_a", StructType(
            List(
              StructField("name", StringType, true,
                new MetadataBuilder().putString("check", "b").build()),
              StructField("age", IntegerType, true)
            )
          ), true,
            new MetadataBuilder().putString("key", "value").build()),
          StructField("col_b", StringType, true)
        )

        val df = spark.createDataFrame(
          spark.sparkContext.parallelize(data),
          StructType(schema)
        )
        df.write.parquet(f.getAbsolutePath)

        // read from storage
        val readDF = spark.read.parquet(f.getAbsolutePath)
        // write again
        withTempPath { f =>
          readDF.select("col_a.name").write.parquet(f.getAbsolutePath)
          // read again and verify the schema is equal (including the metadata)
          val readDF2 = spark.read.parquet(f.getAbsolutePath)
          assert(readDF.select("col_a.name").schema == readDF2.schema)
        }
      }
    }
  }

@viirya @maropu any comments?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@viirya @maropu any comments?

Can one of the admins verify this patch?

authors: AmplabJenkins HyukjinKwon SparkQA dongjoon-hyun guykhazma maropu viirya