text:
Flaky-test: TxnLogBufferedWriterTest.testFlushThresholds

TxnLogBufferedWriterTest.testFlushThresholds is flaky. It fails sporadically.
example failure
  Error:  Tests run: 6, Failures: 2, Errors: 0, Skipped: 4, Time elapsed: 7.931 s <<< FAILURE! - in org.apache.pulsar.transaction.coordinator.impl.TxnLogBufferedWriterTest
  Error:  testFlushThresholds(org.apache.pulsar.transaction.coordinator.impl.TxnLogBufferedWriterTest)  Time elapsed: 0.326 s  <<< FAILURE!
  java.lang.AssertionError: expected [100] but found [101]
  	at org.testng.Assert.fail(Assert.java:99)
  	at org.testng.Assert.failNotEquals(Assert.java:1037)
  	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
  	at org.testng.Assert.assertEquals(Assert.java:122)
  	at org.testng.Assert.assertEquals(Assert.java:907)
  	at org.testng.Assert.assertEquals(Assert.java:917)
  	at org.apache.pulsar.transaction.coordinator.impl.TxnLogBufferedWriterTest.testFlushThresholds(TxnLogBufferedWriterTest.java:400)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
  	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
  	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
  	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  	at java.base/java.lang.Thread.run(Thread.java:833)

assign to me

authors: poorbarcode shibd


text:
Retrieving data from Tiered Storage (S3) fails initially then succeeds after 15 second retry (Pulsar 2.6.1)

Describe the bug
Retrieving data from Tiered Storage (S3) fails initially according to Broker logs ("unknown exception") then succeeds after 15 second retry resulting in a 15 second response time minimum.
I can reproduce this consistently.
To Reproduce
Steps to reproduce the behavior:

Configure Tiered storage offload to S3
Upload data to a topic
Wait for Managed Ledger Rollver
Offload Topic
Attempt to retrieve data

Expected behavior
I expect some delay but not an exception followed by a 15 second retry
Screenshots
Logs
 17:50:18.735 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO  org.apache.bookkeeper.mledger.impl.MetaStoreImpl - [cogito-dialog/wav/persistent/57f5ea00-a7a2-49c3-95d3-c8237d893d31] [PulsarService-31955749-c92b-4bfe-a06a-2985d66b69c7] Updating cursor info ledgerId=1108490 mark-delete=1107259:-1
17:50:18.739 [bookkeeper-ml-workers-OrderedExecutor-5-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [cogito-dialog/wav/persistent/57f5ea00-a7a2-49c3-95d3-c8237d893d31] Updated cursor PulsarService-31955749-c92b-4bfe-a06a-2985d66b69c7 with ledger id 1108490 md-position=1107259:-1 rd-position=1107259:0
17:50:18.739 [bookkeeper-ml-workers-OrderedExecutor-5-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [cogito-dialog/wav/persistent/57f5ea00-a7a2-49c3-95d3-c8237d893d31] Opened new cursor: ManagedCursorImpl{ledger=cogito-dialog/wav/persistent/57f5ea00-a7a2-49c3-95d3-c8237d893d31, name=PulsarService-31955749-c92b-4bfe-a06a-2985d66b69c7, ackPos=1107259:-1, readPos=1107259:0}
17:50:18.739 [bookkeeper-ml-workers-OrderedExecutor-5-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [cogito-dialog/wav/persistent/57f5ea00-a7a2-49c3-95d3-c8237d893d31-PulsarService-31955749-c92b-4bfe-a06a-2985d66b69c7] Rewind from 1047055:0 to 1047055:0
17:50:18.739 [bookkeeper-ml-workers-OrderedExecutor-5-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://cogito-dialog/wav/57f5ea00-a7a2-49c3-95d3-c8237d893d31] There are no replicated subscriptions on the topic
17:50:18.739 [bookkeeper-ml-workers-OrderedExecutor-5-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://cogito-dialog/wav/57f5ea00-a7a2-49c3-95d3-c8237d893d31][PulsarService-31955749-c92b-4bfe-a06a-2985d66b69c7] Created new subscription for 0
17:50:18.739 [bookkeeper-ml-workers-OrderedExecutor-5-0] INFO  org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:43862] Created subscription on topic persistent://cogito-dialog/wav/57f5ea00-a7a2-49c3-95d3-c8237d893d31 / PulsarService-31955749-c92b-4bfe-a06a-2985d66b69c7
17:50:18.771 [bookkeeper-ml-workers-OrderedExecutor-5-0] WARN  org.apache.bookkeeper.mledger.impl.OpReadEntry - [cogito-dialog/wav/persistent/57f5ea00-a7a2-49c3-95d3-c8237d893d31][PulsarService-31955749-c92b-4bfe-a06a-2985d66b69c7] read failed from ledger at position:1047055:0 : Unknown exception
17:50:18.771 [broker-topic-workers-OrderedScheduler-4-0] ERROR org.apache.pulsar.broker.service.persistent.PersistentDispatcherSingleActiveConsumer - [persistent://cogito-dialog/wav/57f5ea00-a7a2-49c3-95d3-c8237d893d31 / PulsarService-31955749-c92b-4bfe-a06a-2985d66b69c7-Consumer{subscription=PersistentSubscription{topic=persistent://cogito-dialog/wav/57f5ea00-a7a2-49c3-95d3-c8237d893d31, name=PulsarService-31955749-c92b-4bfe-a06a-2985d66b69c7}, consumerId=0, consumerName=8f423, address=/127.0.0.1:43862}] Error reading entries at 1047055:0 : Unknown exception - Retrying to read in 15.0 seconds

There was no stack trace for that error
Desktop (please complete the following information):

OS: Linux
Pulsar: 2.6.1

Additional context
Tiered Offload Settings
$ ./apache-pulsar-2.6.1/bin/pulsar-admin --admin-url http://platform-pulsar-broker:8080 namespaces get-offload-threshold cogito-dialog/wav
0
$ ./apache-pulsar-2.6.1/bin/pulsar-admin --admin-url http://platform-pulsar-broker:8080 namespaces get-offload-deletion-lag cogito-dialog/wav
15 minute(s)

Broker Overrides to defaults
        managedLedgerMaxLedgerRolloverTimeMinutes: "90"

Other Broker settings
# grep ffload broker.conf  | grep -v "#"
managedLedgerOffloadDeletionLagMs=14400000
managedLedgerOffloadAutoTriggerSizeThresholdBytes=-1
offloadersDirectory=./offloaders
managedLedgerOffloadDriver=aws-s3
managedLedgerOffloadMaxThreads=2
managedLedgerOffloadPrefetchRounds=1
s3ManagedLedgerOffloadRegion=us-east-1
s3ManagedLedgerOffloadBucket=eks-saas-pulsar-455275ef9748510192359f2f92675b2e
s3ManagedLedgerOffloadServiceEndpoint=
s3ManagedLedgerOffloadMaxBlockSizeInBytes=67108864
s3ManagedLedgerOffloadReadBufferSizeInBytes=1048576
gcsManagedLedgerOffloadRegion=
gcsManagedLedgerOffloadBucket=
gcsManagedLedgerOffloadMaxBlockSizeInBytes=67108864
gcsManagedLedgerOffloadReadBufferSizeInBytes=1048576
gcsManagedLedgerOffloadServiceAccountKeyFile=
fileSystemProfilePath=../conf/filesystem_offload_core_site.xml

Is it possible to reduce the retry time (15s) as a workaround?

Is it possible to reduce the retry time (15s) as a workaround?

Good idea - I looked through the broker conf files and could not find where this was specified

Good idea - I looked through the broker conf files and could not find where this was specified

I have submitted a pull request to show more info about the "Unknown Exception", https://github.com/apache/pulsar/pull/9323
It's better to decide how to deal with it(retry or do something extra) after we actually know what it is.

I have submitted a pull request to show more info about the "Unknown Exception", https://github.com/apache/pulsar/pull/9323
It's better to decide how to deal with it(retry or do something extra) after we actually know what it is.

Thanks @Renkai

Thanks @Renkai

Thanks indeed - we are using Pulsar 2.6.1 - is there any chance this could be back ported to the next maintenance release (2.6.4?)

Thanks indeed - we are using Pulsar 2.6.1 - is there any chance this could be back ported to the next maintenance release (2.6.4?)

Hi, @frankjkelly. Sorry, the next release version is 2.7.1 and we do not have a plan to release 2.6.x. If you want to apply this patch, we can help to cherry-pick this fix to the branch-2.6 and you can build it by yourself.
And the streamnative.io have a weekly release at https://github.com/streamnative/pulsar/releases, https://hub.docker.com/r/streamnative/pulsar/tags?page=1&ordering=last_updated. This release is based on the official branch. We can start a quick release for 2.6.3.x with this path well.

Hi, @frankjkelly. Sorry, the next release version is 2.7.1 and we do not have a plan to release 2.6.x. If you want to apply this patch, we can help to cherry-pick this fix to the branch-2.6 and you can build it by yourself.
And the streamnative.io have a weekly release at https://github.com/streamnative/pulsar/releases, https://hub.docker.com/r/streamnative/pulsar/tags?page=1&ordering=last_updated. This release is based on the official branch. We can start a quick release for 2.6.3.x with this path well.

@codelipenghui it appears now that there might be a plan for a 2.6.4 release (I noticed the release label on https://github.com/apache/pulsar/pull/7356). Any chance https://github.com/apache/pulsar/pull/9323 could be back-ported?
Thanks in advance.

FYI could be a dupe of https://github.com/apache/pulsar/issues/8282

The issue had no activity for 30 days, mark with Stale label.

No longer an issue for us - we wrote our own S3 offloader for various reasons.

source: gharchive/issue

authors: Renkai apapia codelipenghui frankjkelly

repo: apache/pulsar

license_type: permissive

license_source: github-api


text:
[broker] Delete TopicPoliciesService#clean, not used anymore.

Motivation
TopicPoliciesService#clean was introduced to solve listener leaking problem in #10466.
And this is already addressed in #12904 using unregisterListener.
So this method is no use any more, and I don't see any situations that we may use this again.
Modifications
Delete TopicPoliciesService#clean
Verifying this change

[ ] Make sure that the change passes the CI checks.

This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?

[x] no-need-doc

Code clean up.

/pulsarbot run-failure-checks

source: gharchive/pull-request


text:
Fix log problem in ThreadRuntime, make it's member final

Modifications

Fix a log not correct in ThreadRuntime
Make some their fields final

Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?

[x] no-need-doc
simple optmize

/pulsarbot run-failure-checks

@codelipenghui @hangc0276 @315157973 @michaeljmarshall  @lhotari PTAL

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks


text:
[fix][jdk17] Enable Netty and BookKeeper IO optimizations on jdk17

Motivation
Follow up of https://github.com/apache/pulsar/pull/14999.
Based on https://github.com/netty/netty/issues/12265 and https://github.com/apache/bookkeeper/pull/3234 it is better to open other modules.
Modifications

Open java.base/java.util.zip module for BookKeeper
Open java.base/jdk.internal.misc module for Netty

Note that the functions worker will spawn the process without these optimizations. It could be useful in case the connector uses Netty. Very unlikely that the function uses BookKeeper. I can follow up the Netty ones in another pull (with a lower priority).

[x] no-need-doc

Very unlikely that the function uses BookKeeper.

@nicoloboschi - a function instance connects to bookkeeper when running stateful functions. I don't know any helpful client internals to determine if we'd need to look closer, though.

/pulsarbot rerun-failure-checks

/pulsarbot rerun-failure-checks

/pulsarbot rerun-failure-checks

authors: lhotari michaeljmarshall nicoloboschi


text:
[fix][broker]Ensure that the ledgerId updated by currentLedger is monotonically increasing

Motivation
When updating the currentLedger, it is necessary to ensure that the updated ledgerId is greater than the ledgerId of the current currentLedger.
Documentation
Check the box below or label this PR directly.
Need to update docs?

[ ] doc-required
(Your PR needs to update docs and you will update later)

[x] doc-not-needed
(Please explain why)

[ ] doc
(Your PR contains doc changes)

[ ] doc-complete
(Docs have been already added)

/pulsarbot run-failure-checks

@lordcheng10 Looks like we have the lock to ensure concurrency problem, could you describe the detail?

When there is a serious Full GC, there may be multiple ledgers to update the metadata information on zookeeper at the same time. Suppose we create ledger in the order of ledger1, ledger2, ledger3, ledger4, ledger5, and assume that a comparison occurs. In severe Full GC, ledger5 may grab the metadataMutex lock before ledger4, so ledger5 may trigger the zookeeper callback method operationComplete before ledger4, so that ledger4 may cover ledger5

@codelipenghui @hangc0276 PTAL,thanks!


text:
[fix][py] Correct configuration of pattern discovery period

Motivation
Currently, the pattern_auto_discovery_period for topic pattern consumers is fixed to 60 seconds, no matter what is passed into the subscribe function. This is a fix for that bug.
Modifications
Pattern discovery period configuration is now passed through correctly from the subscribe function.
New test to exercise pattern discovery.
Removed misleading configuration in other test which does not use automated pattern discovery.
Made log level configurable in test file with environment variable.
If the changes are ok, I can backport these changes to earlier branches - we use v2.9 currently so I have a specific requirement for that version. I can also open a PR against the new v3 repository if required.
Verifying this change

[x] Make sure that the change passes the CI checks.

A new integration test for the Python client library has been added
An existing test was slightly changed to remove misleading comments

Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes

[ ] Dependencies (add or upgrade a dependency)
[ ] The public API
[ ] The schema
[ ] The default values of configurations
[ ] The binary protocol
[ ] The REST endpoints
[ ] The admin CLI options
[ ] Anything that affects deployment

Documentation

[ ] doc-required
(Your PR needs to update docs and you will update later)

[x] doc-not-needed
Documentation now matches behaviour described for the pattern_auto_discovery_period option.

[ ] doc
(Your PR contains doc changes)

[ ] doc-complete
(Docs have been already added)

Matching PR in forked repository
PR in forked repository:
Successful manual run of CI job in forked repository

The Python client has been moved out of the main repo now, please open a PR here: https://github.com/apache/pulsar-client-python

@BewareMyPower how do I get this fix into any of the v2 versions of the client if I open it against the new repo? My understanding is that repo is for v3+ only. According to the PEP, bug fixes are still allowed for existing branches.

@sbreatnach Yes. But currently the commits to existing branches cannot trigger the CI for Python tests. Though currently the CI in the new repo is not ready at the moment. IMO,
I reopened this PR just now, but I think the correct process is:

Open a PR in https://github.com/apache/pulsar-client-python, which should trigger the CI tests.
If the PR is a bug fix, cherry-pick it back into the existing branch-2.x branches.

WDYT? @merlimat

Keeping this PR open might lead some committers to click the "Squash and merge" button to merge this PR.

So I closed this PR again until getting the response from @merlimat

Yes, @sbreatnach please open the PR in the new repo and once it gets merged, the committers will port it back to the older branches.

authors: BewareMyPower merlimat sbreatnach


text:
[improve][broker] allow client skips the non-continued deleted ranges stats

Fixes #18240

Master Issue: #18240
Motivation

Modifications

add a flag getTotalNonContiguousDeletedMessagesRange to allow users can skip  the non-continued deleted ranges stats
Verifying this change

[ ] Make sure that the change passes the CI checks.

https://github.com/leizhiyuan/pulsar/pull/5
(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes

[ ] Dependencies (add or upgrade a dependency)
[ ] The public API
[ ] The schema
[ ] The default values of configurations
[ ] The threading model
[ ] The binary protocol
[x] The REST endpoints
[ ] The admin CLI options
[ ] Anything that affects deployment

Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository:

maybe we should  use GetStatsOptions next pr, not original name like getPreciseBacklog,subscriptionBacklogSize etc.

There is a failed broker test in your https://github.com/leizhiyuan/pulsar/pull/4 :
https://github.com/leizhiyuan/pulsar/actions/runs/3373856231/jobs/5598944551#step:10:1209
if it is caused by your change please fix it.

There is a failed broker test in your leizhiyuan#4 : https://github.com/leizhiyuan/pulsar/actions/runs/3373856231/jobs/5598944551#step:10:1209
if it is caused by your change please fix it.

https://github.com/leizhiyuan/pulsar/pull/5
I didn't sync the master branch ,now I opened a new pr

There is a failed broker test in your leizhiyuan#4 : https://github.com/leizhiyuan/pulsar/actions/runs/3373856231/jobs/5598944551#step:10:1209
if it is caused by your change please fix it.

leizhiyuan#5
I didn't sync the master branch ,now I opened a new pr

Add some case  seems be flaky, It is not related with the pr，  I ran ok in my machine。

/pulsarbot run-failure-checks

Codecov Report

Merging #18263 (8460be2) into master (46c0438) will decrease coverage by 3.94%.
The diff coverage is 47.16%.

@@             Coverage Diff              @@
##             master   #18263      +/-   ##
============================================
- Coverage     38.84%   34.90%   -3.95%
+ Complexity     8286     4709    -3577
============================================
  Files           683      400     -283
  Lines         67323    43651   -23672
  Branches       7215     4477    -2738
============================================
- Hits          26152    15235   -10917
+ Misses        38210    26257   -11953
+ Partials       2961     2159     -802

Flag
Coverage Δ

unittests
34.90% <47.16%> (-3.95%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...he/pulsar/broker/admin/v2/NonPersistentTopics.java
22.68% <ø> (-39.36%)
:arrow_down:

...rg/apache/pulsar/broker/service/BrokerService.java
48.46% <0.00%> (-9.46%)
:arrow_down:

...n/java/org/apache/pulsar/broker/service/Topic.java
18.18% <ø> (ø)

...oker/service/nonpersistent/NonPersistentTopic.java
24.69% <0.00%> (-30.19%)
:arrow_down:

...ker/stats/prometheus/NamespaceStatsAggregator.java
0.00% <0.00%> (ø)

...pulsar/broker/admin/impl/PersistentTopicsBase.java
17.76% <9.09%> (-34.81%)
:arrow_down:

...pache/pulsar/broker/admin/v2/PersistentTopics.java
13.16% <45.45%> (-57.56%)
:arrow_down:

...pache/pulsar/broker/admin/v1/PersistentTopics.java
11.40% <57.14%> (-34.30%)
:arrow_down:

...sar/broker/service/persistent/PersistentTopic.java
46.86% <83.33%> (-10.06%)
:arrow_down:

...sar/broker/resourcegroup/ResourceGroupService.java
32.97% <100.00%> (-4.80%)
:arrow_down:

... and 423 more

could you please merge the apache/master, then rerun the test

@leizhiyuan  could you please rebase master and take a look at the failed tests?

related to #23380

It's been too long (2 years) since the pull request. I can't even remember some of the code at that time. I will close this PR.

authors: codecov-commenter codelipenghui congbobo184 labuladong leizhiyuan lhotari poorbarcode


text:
[fix][broker] fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker

Fixes #21016
Motivation
fix UniformLoadShedder select the wrong overloadbroker and underloadbroker
Modifications

correct the procedure of selecting overload and underload broker
separating msgRate and msgThrought, and msgRate has higher priority(same as before)

Documentation

[x] doc-not-needed

Matching PR in forked repository
PR in forked repository: https://github.com/aloyszhang/pulsar/pull/19

Test with three
- 3 brokers
- 6 topics, each one has 11 partitions, topics have different msgRates and msgSizes
At 16:52 start load-balance with this fix and the following parameters:

loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold=1.3
loadBalancerMsgRateDifferenceShedderThreshold=0 (disable)

We can see all brokers are balanced well based on throughput.
At 17:04 modify loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold from 1.3 to 1.05
All brokers load are more balanced.
Then fallback server to origin UniformLoadShedder(without this fix) and restart broker at 17:15, all configurations are not modified:

loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold=1.05
loadBalancerMsgRateDifferenceShedderThreshold=0 (disable)
Brokers load  balanced at 17:24, but it's not quite balanced, maxThroughput is 128MB/s and minThroughput is 88 MB/s, the finnaly loadDifferent is 128/88 = 1.45 which is bigger than loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold(1.05)

This is great - does it need to be backported to prior versions?

Backport pull request links：
branch-3.1 : https://github.com/apache/pulsar/pull/21180
branch-3.0: https://github.com/apache/pulsar/pull/21179
branch-2.11: https://github.com/apache/pulsar/pull/21178
branch-2.10: https://github.com/apache/pulsar/pull/21182

This is great - does it need to be backported to prior versions?

@frankjkelly I'll backport this change to versions after 2.10, including 2.10, 2.11, 3.0 and 3.1

That would be wonderful - I am using ThresholdShedder in 2.10.3 and am not thrilled with how complex it is to manage. I'd love to use UniformShedder in 2.11.X with this fix. Thanks so much!

@aloyszhang Could you create a new PR to cherry-pick this PR into branch-3.0?

authors: aloyszhang frankjkelly poorbarcode


text:
Update the pulsar admin cli java doc to support set/get/remove deduplication on topic level

Fixes #7920
Motivation
set/get/remove deduplication policy is on topic level. But the pulsar admin cli java doc is not supported accordingly.
Modifications
add methods to CmdTopics.java
Verifying this change
unit test
org.apache.pulsar.admin.cli.PulsarAdminToolTest#topics

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks


text:
[Doc] Add docs for ProtobufNative Schema

Add docs based on https://github.com/apache/pulsar/pull/8372 and update the green-highlighted parts.

@congbobo184 could u pls help review? Thank you.

@Jennifer88huang thanks for your suggestions. I've incorporated your comments, PTAL.


text:
Delete unused data members

Delete unused data members

it can be close

repo: apache/rocketmq-mqtt


text:
[ISSUE #56] add exportJvmInfo to judge prometheus whether export jvm info or not

fix #56 add exportJvmInfo to judge prometheus whether export jvm info or not

Codecov Report

Merging #57 (f5a2779) into main (0177b54) will decrease coverage by 0.03%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main      #57      +/-   ##
==========================================
- Coverage   23.77%   23.74%   -0.04%
==========================================
  Files          96       96
  Lines        4176     4182       +6
  Branches      662      663       +1
==========================================
  Hits          993      993
- Misses       2960     2966       +6
  Partials      223      223

Impacted Files
Coverage Δ

...rg/apache/rocketmq/mqtt/cs/config/ConnectConf.java
0.00% <0.00%> (ø)

...pache/rocketmq/mqtt/cs/starter/ExporterServer.java
0.00% <0.00%> (ø)

...rg/apache/rocketmq/mqtt/exporter/MqttExporter.java
0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0177b54...f5a2779. Read the comment docs.

authors: codecov-commenter tianliuliu


text:
CONSUME_FROM_TIMESTAMP doesn't work

BUG REPORT
1. Please describe the issue you observed:
【需求】：
使用PushConsumer订阅了一个topic，并且想只订阅从这个consumer订阅的时间点起往后的数据（不需要历史数据）。我的想法是设置consumer的ConsumeFromWhere.CONSUME_FROM_TIMESTAMP和当前时间作为timestamp，如下：
    consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_TIMESTAMP);
    String timestamp = UtilAll.timeMillisToHumanString3(System.currentTimeMillis());
    consumer.setConsumeTimestamp(timestamp);

为了保证consumer启动的时候所在的consumergroup每次都是新的，我将consumergroup设置为时间戳：
Date date = new Date();
String strDateFormat = "yyyyMMddHHmmss";
SimpleDateFormat sdf = new SimpleDateFormat(strDateFormat);
String now = sdf.format(date);
// 实例化消费者
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(now);
【操作】：
在启动这个consumer之前，我先写了一个producer，往这个topic中写数据（写了10条）。确保数据写完之后，我启动了consumer。
【实际结果】：
这个consumer依然读取到了这10条数据。
ConsumeFromWhere.CONSUME_FROM_TIMESTAMP没有起作用。
【预期结果】：
由于这个consumer启动晚于producer，所以应该读取不到这些数据。
2. Please tell us about your environment:
rocketmq 4.7.0 单broker；单namesrv
3. Other information (e.g. detailed explanation, logs, related issues, suggestions how to fix, etc):
完整代码如下：
producer:
public class SyncProducer {
public static String TOPIC_NAME = "newManualTopic";
public static void main(String[] args) throws Exception {
// 实例化消息生产者Producer
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
// 设置NameServer的地址
producer.setNamesrvAddr("localhost:9876");
// 启动Producer实例
producer.start();
for (int i = 0; i < 10; i++) {
// 创建消息，并指定Topic，Tag和消息体
Message msg = new Message(TOPIC_NAME /* Topic /,
"TagA" / Tag /,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET) / Message body */
);
// 发送消息到一个Broker
SendResult sendResult = producer.send(msg);
// 通过sendResult返回消息是否成功送达
System.out.printf("%s%n", sendResult);
}
// 如果不再发送消息，关闭Producer实例。
producer.shutdown();
}
}
Consumer:
import static producer.SyncProducer.TOPIC_NAME;
public class Consumer {
public static void main(String[] args) throws InterruptedException, MQClientException {
Date date = new Date();
String strDateFormat = "yyyyMMddHHmmss";
SimpleDateFormat sdf = new SimpleDateFormat(strDateFormat);
String now = sdf.format(date);
// 实例化消费者
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(now);
// 设置NameServer的地址
consumer.setNamesrvAddr("localhost:9876");
// 订阅一个或者多个Topic，以及Tag来过滤需要消费的消息
consumer.subscribe(TOPIC_NAME, "*");
// 注册回调实现类来处理从broker拉取回来的消息
consumer.registerMessageListener(new MessageListenerConcurrently() {
public ConsumeConcurrentlyStatus consumeMessage(List msgs, ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
// 标记该消息已经被成功消费
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_TIMESTAMP);//ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET
String timestamp = UtilAll.timeMillisToHumanString3(System.currentTimeMillis());
consumer.setConsumeTimestamp(timestamp);
System.out.println("now:"+now);
System.out.println("timestamp:"+timestamp);
// 启动消费者实例
consumer.start();
System.out.printf("Consumer Started.%n");
}
}

Maybe this article can solve your doubts https://blog.csdn.net/prestigeding/article/details/96576932

Issues are generally used for bug reports and feature requests, so if you have any questions about     Apache  RocketMQ, it is recommended to discuss it on the mailing list first, the mailing list [address]    (https://rocketmq.apache.org/about/contact). I will close the Issue first, but please feel free to reopen it if you have any other issues.

感谢回复。根据您给的链接，我仔细研究了一下。我理解CONSUM_FROM_TIMESTAMP的逻辑是：
1.先找当前consumergroup在这个queue上的最新offset
2.如果没找到就去找这个queue当前的最早offset(这个offset就是consumerqueue文件组中最早的那个文件的第一个offset，也就是在磁盘上这个queue中最早的那个offset)
3.要是以上都没有再去按照timestamp去定位到哪个consumequeue文件，再在这个文件中通过折半查找去定位到匹配的timestamp的offset。
但是这个逻辑似乎是不是永远都走不到3？因为2的consumequeue文件应该永远都会有的吧？另外，要是没consumequeue文件的话，3也没有意义呀？请问什么情况才能走到3呢？
另外，我看链接中的解决方案也是通过cloneoffset方法，让新的consumergroup从其他consumergroup中把offset拷贝过来。让它最终能走1.
请问什么时候才能真正走到根据timestamp去查找offset的第3步呢？
谢谢！

authors: RongtongJin yidaimi

repo: apache/rocketmq


text:
[ISSUE #6244]Fix issue template feature request order incorrect

Make sure set the target branch to develop
What is the purpose of the change

fix #6244
Brief changelog

Fix issue template feature request order incorrect

Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Codecov Report

Merging #6245 (250fed0) into develop (d5d5a63) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             develop    #6245      +/-   ##
=============================================
- Coverage      43.19%   43.18%   -0.01%
  Complexity      8848     8848
=============================================
  Files           1094     1094
  Lines          77193    77193
  Branches       10070    10070
=============================================
- Hits           33342    33336       -6
- Misses         39678    39689      +11
+ Partials        4173     4168       -5

Impacted Files
Coverage Δ

...apache/rocketmq/broker/longpolling/PopRequest.java
31.03% <0.00%> (-13.80%)
:arrow_down:

...a/org/apache/rocketmq/filter/util/BloomFilter.java
60.43% <0.00%> (-2.20%)
:arrow_down:

...e/rocketmq/store/ha/autoswitch/EpochFileCache.java
77.08% <0.00%> (-1.39%)
:arrow_down:

...e/rocketmq/remoting/netty/NettyRemotingServer.java
57.83% <0.00%> (-1.05%)
:arrow_down:

...che/rocketmq/test/lmq/benchmark/BenchLmqStore.java
50.24% <0.00%> (-0.50%)
:arrow_down:

...ent/impl/consumer/DefaultLitePullConsumerImpl.java
67.53% <0.00%> (-0.47%)
:arrow_down:

...a/org/apache/rocketmq/store/StoreStatsService.java
34.53% <0.00%> (ø)

...cketmq/broker/processor/PopBufferMergeService.java
33.33% <0.00%> (ø)

...main/java/org/apache/rocketmq/store/CommitLog.java
65.13% <0.00%> (+0.10%)
:arrow_up:

...cketmq/store/ha/autoswitch/AutoSwitchHAClient.java
76.42% <0.00%> (+0.35%)
:arrow_up:

... and 3 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter mxsm


text:
[ISSUE #8049] Fix tiered storage delete topic NPE in default metadata store

Which Issue(s) This PR Fixes

Fixes #8049
Brief Description

xxx
How Did You Test This Change?

Create a topic and delete it without production.

Rebase master to solve the CI problem

Rebase master to solve the CI problem

Done.

authors: bxfjb lizhimins


text:
[Bug]clickhouse cdc sink   ClickhouseConnectorException

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
source mysq-cdc
sink clickhouse
发生ClickhouseConnectorException
SeaTunnel Version
2.3.3
SeaTunnel Config
env {
  execution.parallelism = 4
  job.mode = "STREAMING"
}

Clickhouse {
        host="xxxxx:8123"
        database="test"
        table="test"
        username="default"
        password="123456"
        }

Running Command
./../bin/seatunnel.sh --config cdc.config

Error Exception
Caused by: org.apache.seatunnel.connectors.seatunnel.clickhouse.exception.ClickhouseConnectorException: ErrorCode:[COMMON-10], ErrorDescription:[Flush data operation that in sink connector failed] - Clickhouse execute batch statement error
	at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.ClickhouseSinkWriter.flush(ClickhouseSinkWriter.java:119)
	at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.ClickhouseSinkWriter.flush(ClickhouseSinkWriter.java:134)
	at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.ClickhouseSinkWriter.prepareCommit(ClickhouseSinkWriter.java:93)
	at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:166)
	... 15 more
	Suppressed: java.sql.SQLException: Cannot operate on a closed statement
		at com.clickhouse.jdbc.SqlExceptionUtils.clientError(SqlExceptionUtils.java:73)
		at com.clickhouse.jdbc.internal.ClickHouseStatementImpl.ensureOpen(ClickHouseStatementImpl.java:114)
		at com.clickhouse.jdbc.internal.InputBasedPreparedStatement.setInt(InputBasedPreparedStatement.java:217)
		at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.FieldNamedPreparedStatement.setInt(FieldNamedPreparedStatement.java:88)
		at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.inject.IntInjectFunction.injectFields(IntInjectFunction.java:34)
		at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.JdbcRowConverter.toExternal(JdbcRowConverter.java:79)
		at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.SimpleBatchStatementExecutor.addToBatch(SimpleBatchStatementExecutor.java:42)
		at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.BufferedBatchStatementExecutor.executeBatch(BufferedBatchStatementExecutor.java:51)
		at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.BufferedBatchStatementExecutor.closeStatements(BufferedBatchStatementExecutor.java:61)
		at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.JdbcBatchStatementExecutor.close(JdbcBatchStatementExecutor.java:37)
		at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.ClickhouseSinkWriter.flush(ClickhouseSinkWriter.java:137)
		... 17 more
Caused by: java.sql.SQLException: Cannot operate on a closed statement
	at com.clickhouse.jdbc.SqlExceptionUtils.clientError(SqlExceptionUtils.java:73)
	at com.clickhouse.jdbc.internal.ClickHouseStatementImpl.ensureOpen(ClickHouseStatementImpl.java:114)
	at com.clickhouse.jdbc.internal.InputBasedPreparedStatement.setInt(InputBasedPreparedStatement.java:217)
	at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.FieldNamedPreparedStatement.setInt(FieldNamedPreparedStatement.java:88)
	at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.inject.IntInjectFunction.injectFields(IntInjectFunction.java:34)
	at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.JdbcRowConverter.toExternal(JdbcRowConverter.java:79)
	at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.SimpleBatchStatementExecutor.addToBatch(SimpleBatchStatementExecutor.java:42)
	at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.executor.BufferedBatchStatementExecutor.executeBatch(BufferedBatchStatementExecutor.java:51)
	at org.apache.seatunnel.connectors.seatunnel.clickhouse.sink.client.ClickhouseSinkWriter.flush(ClickhouseSinkWriter.java:117)
	... 18 more

	at org.apache.seatunnel.engine.client.job.ClientJobProxy.waitForJobComplete(ClientJobProxy.java:122)
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:184)
	... 2 more
2023-11-09 11:03:21,074 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - run shutdown hook because get close signal

Zeta or Flink or Spark Version
zeta
Java or Scala Version
1.8
Screenshots

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

This is fixed by https://github.com/apache/seatunnel/pull/5448

repo: apache/seatunnel


text:
if i deploy cluster nodes with explicit role，how to set SW_AGENT_COLLECTOR_BACKEND_SERVICES on agent side?

OS：CentOS Linux release 7.7.1908 (Core)
Skywalking version：apache-skywalking-apm-es7-8.1.0
I have a three node cluster with the following roles:
node01：SW_ CORE_ ROLE:Receiver
node02：SW_ CORE_ ROLE:Receiver
node03：SW_ CORE_ ROLE:Aggregator
In addition to the role settings of the three nodes, do I need other settings?
How should I set SW_ AGENT_ COLLECTOR_ BACKEND_ SERVICES on agent side？
Please forgive my poor English.

We would recommend you to set a proxy for node01 and node02 as LB. Or you could add both node01 and node02 at the agent config.

authors: shuxiaxia wu-sheng

repo: apache/skywalking

license_source: bigquery


text:
Add dashboard configuration for on demand log

[ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
[ ] Update the CHANGES log.


text:
Allow sessionAffinity for the SolrCloud CommonService

Kubernetes ClusterIP Services allow for sessionAffinity, and customization of the max session stickiness timeout.
More information can be found here: https://kubernetes.io/docs/reference/networking/virtual-ips/#session-affinity
All we would need is these options to be given under SolrCloud.Spec.customSolrKubeOptions.commonServiceOptions, and then passed to the service. This doesn't make sense for the headless service or node services.

Took a pass at this in https://github.com/apache/solr-operator/pull/571
Just started learning Go this week, but imagine I'm missing a step or two here. LMK!

authors: HoustonPutman trevorpburke

repo: apache/solr-operator


text:
SOLR-10452: setQueryParams should be deprecated in favor of SolrClientBuilder methods

https://issues.apache.org/jira/browse/SOLR-10452
Description
setQueryParams should be deprecated in favor of SolrClientBuilder methods.
Solution
I moved the the setQueryParams over for the Http2SolrClient, it already had been done for HttpSolrClient.   I noticed we have a addQueryParams method, which I marked deprecated and we shouldn't use, as that goes against the idea of a Solr Client being immutable.
One area I tried to fix and gave up on was the DelegationTokenHttpSolrClient, I couldn't quite figure out what to do there, would love a suggestion or a fix ;-)
Tests
Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.
Checklist
Please review the following and check all that apply:

[ ] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
[ ] I have created a Jira issue and added the issue ID to my pull request title.
[ ] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
[ ] I have developed this patch against the main branch.
[ ] I have run ./gradlew check.
[ ] I have added tests for my changes.
[ ] I have added documentation for the Reference Guide

Hey @stillalex would love a review.

tests all pass!

I find LBHttp2SolrClient's use of queryParams a bit confusing (it seems to be on LBSolrClient level), apologies if this is just my incorrect understanding. Why does the LBSolrClient need to maintain a separate set of query params? LBHttp2SolrClient already receives a Http2SolrClient which ideally would have been init with correct query params. any (now deprecated) method to modify query params would just delegate to Http2SolrClient instance. I would cautiously suggest to remove the extra set of query params, which would also simplify a bit the code.

proposal for DelegationTokenHttpSolrClient:

move constructor 'setQueryParams(' call to Builder itself. the call comes from the builder already, so no added complexity https://github.com/apache/solr/blob/0f912f6b84c6b0a42079df2479ac6bbecc3c2c67/solr/solrj/src/java/org/apache/solr/client/solrj/impl/DelegationTokenHttpSolrClient.java#L40
mark setQueryParams method deprecated
I think this should be it. in the future we can remove the DelegationTokenHttpSolrClient#setQueryParams method and the query params will already have the expected param because the builder injected it at build time.

LBHttp2SolrClient

Would you be interested in pushing up a commit to make this change to remove the method?  I'd lvoe to see it!

setQueryParams

Likewise @stillalex I think I get what you are suggesting, you want to push up a change for this?

Okay, I think I've dealt with the small changes.   @stillalex I'd love you to push up changes for the two suggestions you had, they sound great.   @dsmiley maybe you can weigh in on the name of the field, and I'll make that change, and then this should be ready to merge!   Thanks for the feedback.

setQueryParams

Likewise @stillalex I think I get what you are suggesting, you want to push up a change for this?

sure. how should we do this? push a proposed change to your branch directly for review?

setQueryParams

Likewise @stillalex I think I get what you are suggesting, you want to push up a change for this?

sure. how should we do this? push a proposed change to your branch directly for review?

Yeah, I think you can checkout my PR, or maybe my epugh/solr repo?  Honestly, my Git Fu isn't great!   I am always very  happy to have other folks directly contribute to my branches ;-)   Casue generally I am bumbling along!

setQueryParams

Likewise @stillalex I think I get what you are suggesting, you want to push up a change for this?

sure. how should we do this? push a proposed change to your branch directly for review?

Yeah, I think you can checkout my PR, or maybe my epugh/solr repo? Honestly, my Git Fu isn't great! I am always very happy to have other folks directly contribute to my branches ;-) Casue generally I am bumbling along!

With small concrete commits, I can always revert something ;-)

ok. I was not able to push to your git, not sure why, I don't think I have the rights. I pushed to my repo based on your branch. could you take a look and let me know if it looks ok? https://github.com/stillalex/solr/commit/0490ab420ff1d90536f3572fc7b17b194f0724c1

odd, cause I think you have permissiosn to apache/solr repo?   Can you check out the PR from apache/solr instead of my branch from epugh/solr, and then push up changes????

odd, cause I think you have permissiosn to apache/solr repo? Can you check out the PR from apache/solr instead of my branch from epugh/solr, and then push up changes????

I don't, I'm not a Solr committer :)

suggested fix for the DelegationTokenHttpSolrClient https://github.com/stillalex/solr/commit/5086e0fe099d5c2fa5942c6b3f18b7d803370136 .
The constructor code looks a bit iffy to me. does it really overwrite any existing param from the builder to set this single param? it can't be correct.
setQueryParams(new TreeSet<>(Arrays.asList(DELEGATION_TOKEN_PARAM)));

@stillalex I learned somethign new!  added you as another repo, and was able to cherry-pick your commit!

I've seen you only partially took changes from https://github.com/stillalex/solr/commit/0490ab420ff1d90536f3572fc7b17b194f0724c1 was there anything failing? or was the change too drastic?

I've seen you only partially took changes from stillalex@0490ab4 was there anything failing? or was the change too drastic?

Huh...   I don't know why my cherry pick of that commit didn't bring it all?   I am going to manually review and copy over now ;-(

Okay, are we done?

yep, looks good to me!

authors: epugh stillalex

repo: apache/solr


text:
[SPARK-12625][SPARKR][SQL] replace R usage of Spark SQL deprecated API

@rxin @davies @shivaram
Took save mode from my PR #10480, and move everything to writer methods. This is related to PR #10559

[x] it seems jsonRDD() is broken, need to investigate - this is not a public API though; will look into some more tonight. (fixed)

Test build #48708 has started for PR 10584 at commit e0e2a9c.

Test build #48708 has finished for PR 10584 at commit e0e2a9c.

This patch fails R style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48708/
Test FAILed.

Merged build finished. Test FAILed.

Thanks for submitting this.
FYI https://github.com/apache/spark/pull/10559 was merged, and R tests were disabled there so you should re-enable them.

Test build #48722 has started for PR 10584 at commit 203a52f.

Test build #48724 has started for PR 10584 at commit d8d9b21.

Thanks @felixcheung - LGTM if tests pass. Also @davies will be good if you can take a look.

I found another one after rebasing on #10559
also fixed jsonRDD

Test build #2320 has started for PR 10584 at commit dc59f84.

Test build #48722 has finished for PR 10584 at commit 203a52f.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48722/
Test FAILed.

Test build #48732 has started for PR 10584 at commit dc59f84.

Test build #48724 has finished for PR 10584 at commit d8d9b21.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48724/
Test FAILed.

Test build #2320 has finished for PR 10584 at commit dc59f84.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Thanks - merging this in master.

Test build #48732 has finished for PR 10584 at commit dc59f84.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48732/
Test FAILed.

authors: AmplabJenkins SparkQA felixcheung rxin shivaram

repo: apache/spark


text:
[SPARK-13609] [SQL] Support Column Pruning for MapPartitions

What changes were proposed in this pull request?
This PR is to prune unnecessary columns when the operator is  MapPartitions. The solution is to add an extra Project in the child node.
For the other two operators AppendColumns and MapGroups, it sounds doable. More discussions are required. The major reason is the current implementation of the inputPlan of groupBy is based on the child of AppendColumns. It might be a bug? Thus, will submit a separate PR.
How was this patch tested?
Added a test case in ColumnPruningSuite to verify the rule. Added another test case in DatasetSuite.scala to verify the data.

Test build #52292 has started for PR 11460 at commit bd55678.

Test build #52292 has finished for PR 11460 at commit bd55678.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52292/
Test PASSed.

Merged build finished. Test PASSed.

@davies @marmbrus @rxin Could you please review it? Thanks!

LGTM

Merging this into master, thanks!

authors: AmplabJenkins SparkQA cloud-fan davies gatorsmile


text:
[SPARK-15230] [SQL] distinct() does not handle column name with dot properly

What changes were proposed in this pull request?
When table is created with column name containing dot, distinct() will fail to run. For example,
val rowRDD = sparkContext.parallelize(Seq(Row(1), Row(1), Row(2)))
val schema = StructType(Array(StructField("column.with.dot", IntegerType, nullable = false)))
val df = spark.createDataFrame(rowRDD, schema)

running the following will have no problem:
df.select(new Column("`column.with.dot`"))

but running the query with additional distinct() will cause exception:
df.select(new Column("`column.with.dot`")).distinct()

The issue is that distinct() will try to resolve the column name, but the column name in the schema does not have backtick with it. So the solution is to add the backtick before passing the column name to resolve().
How was this patch tested?
Added a new test case.

Test build #58656 has started for PR 13140 at commit 2f7ffbd.

Test build #58656 has finished for PR 13140 at commit 2f7ffbd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58656/
Test PASSed.

Test build #60429 has started for PR 13140 at commit 2f7ffbd.

Test build #60429 has finished for PR 13140 at commit 2f7ffbd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60429/
Test PASSed.

Test build #60880 has started for PR 13140 at commit 0c72506.

Test build #60880 has finished for PR 13140 at commit 0c72506.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60880/
Test PASSed.

Test build #60959 has started for PR 13140 at commit 48a6f50.

Test build #60959 has finished for PR 13140 at commit 48a6f50.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
final class Binarizer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)
final class Bucketizer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)
final class ChiSqSelector @Since(\"1.6.0\") (@Since(\"1.6.0\") override val uid: String)
class CountVectorizer @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)
class CountVectorizerModel(
class DCT @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)
class ElementwiseProduct @Since(\"2.0.0\") (@Since(\"2.0.0\") override val uid: String)
class HashingTF @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)
final class IDF @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)
class Interaction @Since(\"1.6.0\") (@Since(\"1.6.0\") override val uid: String) extends Transformer
class MaxAbsScaler @Since(\"2.0.0\") (@Since(\"2.0.0\") override val uid: String)
class MinMaxScaler @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)
class NGram @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)
class Normalizer @Since(\"2.0.0\") (@Since(\"2.0.0\") override val uid: String)
class OneHotEncoder @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String) extends Transformer
class PCA @Since(\"1.5.0\") (
class PolynomialExpansion @Since(\"2.0.0\") (@Since(\"2.0.0\") override val uid: String)
final class QuantileDiscretizer @Since(\"1.6.0\") (@Since(\"1.6.0\") override val uid: String)
class RFormula @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)
class SQLTransformer @Since(\"1.6.0\") (@Since(\"1.6.0\") override val uid: String) extends Transformer
class StandardScaler @Since(\"1.4.0\") (
class StopWordsRemover @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)
class StringIndexer @Since(\"1.4.0\") (
class Tokenizer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)
class RegexTokenizer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)
class VectorAssembler @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)
class VectorIndexer @Since(\"1.4.0\") (
final class VectorSlicer @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)
final class Word2Vec @Since(\"1.4.0\") (

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60959/
Test PASSed.

Merged build finished. Test PASSed.

LGTM except a minor comment about test, thanks for working on it!

@cloud-fan thanks for your concise codes!

Test build #61045 has started for PR 13140 at commit 0e565db.

Test build #61045 has finished for PR 13140 at commit 0e565db.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #61062 has started for PR 13140 at commit 23a3a50.

Test build #61062 has finished for PR 13140 at commit 23a3a50.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ElementwiseProduct @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)
class Normalizer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)
class PolynomialExpansion @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61062/
Test FAILed.

i do not know what happened to jenkin, looks the failure is irrelevant.

retest this please

Test build #61084 has started for PR 13140 at commit 23a3a50.

Test build #61084 has finished for PR 13140 at commit 23a3a50.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ElementwiseProduct @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)
class Normalizer @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)
class PolynomialExpansion @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61084/
Test PASSed.

thanks, merging to master and 2.0!

authors: AmplabJenkins SparkQA bomeng cloud-fan


text:
[SPARK-17320] add build_profile_flags entry to mesos build module

What changes were proposed in this pull request?
add build_profile_flags entry to mesos build module
How was this patch tested?
unit tests

Can you attack this in the title to the original JIRA to make the Mesos profile? LGTM though

Test build #64672 has started for PR 14885 at commit 4b2ee7c.

Damn, you're quick.  I added a new JIRA before I saw this message.  Hope that's alright too.

OK, no big deal.

Test build #64672 has finished for PR 14885 at commit 4b2ee7c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64672/
Test FAILed.

Merged build finished. Test FAILed.

Test build #3239 has started for PR 14885 at commit 4b2ee7c.

Test build #3239 has finished for PR 14885 at commit 4b2ee7c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cc @srowen for merge

Merging to master.

authors: AmplabJenkins SparkQA mgummelt srowen vanzin


text:
[SPARK-18353][CORE] spark.rpc.askTimeout defalut value is not 120s

What changes were proposed in this pull request?
Avoid hard-coding spark.rpc.askTimeout to non-default in Client; fix doc about spark.rpc.askTimeout default
How was this patch tested?
Existing tests

Test build #68415 has started for PR 15833 at commit ae2fb2f.

Test build #68415 has finished for PR 15833 at commit ae2fb2f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68415/
Test FAILed.

Test build #68465 has started for PR 15833 at commit 55cd875.

Test build #68465 has finished for PR 15833 at commit 55cd875.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68465/
Test PASSed.

Test build #68771 has started for PR 15833 at commit e272739.

Test build #68771 has finished for PR 15833 at commit e272739.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68771/
Test PASSed.

Going to merge this because it doesn't change any default behavior, just appears to let this configuration be set by the user in some contexts where not possible before.

Merged to master/2.1

authors: AmplabJenkins SparkQA srowen


text:
[SPARK-18447][DOCS] Fix the markdown for Note:/NOTE:/Note that across Python API documentation

What changes were proposed in this pull request?
It seems in Python, there are

Note:
NOTE:
Note that
.. note::

This PR proposes to fix those to .. note:: to be consistent.
Before

After

How was this patch tested?
The notes were found via
grep -r "Note: " .
grep -r "NOTE: " .
grep -r "Note that " .

And then fixed one by one comparing with API documentation.
After that, manually tested via make html under ./python/docs.

Test build #68903 has started for PR 15947 at commit 46173a3.

cc @srowen and @aditya1702 who was interested in this issue.

Test build #68903 has finished for PR 15947 at commit 46173a3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68903/
Test PASSed.

LGTM

Merged to master/2.1

authors: AmplabJenkins HyukjinKwon SparkQA jagadeesanas2 srowen


text:
[SPARK-19750][UI][branch-2.1] Fix redirect issue from http to https

What changes were proposed in this pull request?
If spark ui port (4040) is not set, it will choose port number 0, this will make https port to also choose 0. And in Spark 2.1 code, it will use this https port (0) to do redirect, so when redirect triggered, it will point to a wrong url:
like:
/tmp/temp$ wget http://172.27.25.134:55015
--2017-02-23 12:13:54--  http://172.27.25.134:55015/
Connecting to 172.27.25.134:55015... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://172.27.25.134:0/ [following]
--2017-02-23 12:13:54--  https://172.27.25.134:0/
Connecting to 172.27.25.134:0... failed: Can't assign requested address.
Retrying.

--2017-02-23 12:13:55--  (try: 2)  https://172.27.25.134:0/
Connecting to 172.27.25.134:0... failed: Can't assign requested address.
Retrying.

--2017-02-23 12:13:57--  (try: 3)  https://172.27.25.134:0/
Connecting to 172.27.25.134:0... failed: Can't assign requested address.
Retrying.

--2017-02-23 12:14:00--  (try: 4)  https://172.27.25.134:0/
Connecting to 172.27.25.134:0... failed: Can't assign requested address.
Retrying.

So instead of using 0 to do redirect, we should pick a bound port instead.
This issue only exists in Spark 2.1-, and can be reproduced in yarn cluster mode.
How was this patch tested?
Current redirect UT doesn't verify this issue, so extend current UT to do correct verification.

Was this fixed otherwise in master, or did some other change make it obsolete? just trying to link this to whatever reason it's only a problem in 2.1, for the record.

Due to the change of (https://github.com/apache/spark/pull/16625), the issue is obsolete. So it effects spark 2.1 and 2.0.

Not sure why Jenkins test cannot be started automatically.

Test build #73498 has started for PR 17083 at commit 5408005.

Test build #73498 has finished for PR 17083 at commit 5408005.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73498/
Test PASSed.

Test build #73551 has started for PR 17083 at commit 9ee6a09.

Test build #73551 has finished for PR 17083 at commit 9ee6a09.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73551/
Test PASSed.

Ping @vanzin , do you have any further comments? Thanks a lot.

Merging to 2.1 / 2.0.

@jerryshao I think you'll need to manually close the PR.

OK, Thanks @vanzin .

authors: AmplabJenkins SparkQA jerryshao srowen vanzin


text:
[SPARK-20017][SQL] change the nullability of function 'StringToMap' from 'false' to 'true'

What changes were proposed in this pull request?
Change the nullability of function StringToMap from false to true.

Can one of the admins verify this patch?

Could you add tests?

@maropu The test case is added. Please check it, thanks~

@hvanhovell could you check?

May I ask why you want to change the nullability of StringToMap?
also cc @gatorsmile

Also, add an end-to-end test case in string-function.sql. Update the function description to explain when we return null.

@wzhfy why wouldn't we change the nullability? This is obviously a bug in the current master: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala#L438

@gatorsmile  Is it OK? I don't know how to add test case in string-function.sql. Ｗould you give me some guidance? Thanks~

@hvanhovell Got it. Thanks!

Add the queries to the following file: string-function.sql
Run the following command
SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/test-only *SQLQueryTestSuite"

Thank you for updating the example, but you forgot to update the function description.

ok to test

Test build #74986 has started for PR 17350 at commit 1260ef7.

Test build #74986 has finished for PR 17350 at commit 1260ef7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74986/
Test PASSed.

LGTM
Since it is very close to code freeze, let me merge it to master and 2.1 at first. You can submit the PR to address the issues as a follow-up PR. Thanks!

@zhaorongsheng Not sure whether you can help us check whether all the functions have an issue in nullability setting?

@gatorsmile OK, I will do it and I will give you feedback as soon as possible.

Thanks!

@gatorsmile Sorry for the late reply.
I have checked all the functions' nullability setting and I didn't found any issue.
Thanks~

Thanks!

authors: AmplabJenkins SparkQA gatorsmile hvanhovell maropu wzhfy zhaorongsheng


text:
[SPARK-24556][SQL] ReusedExchange should rewrite output partitioning also when child's partitioning is RangePartitioning

What changes were proposed in this pull request?
Currently, ReusedExchange would rewrite output partitioning if child's partitioning is HashPartitioning, but it does not do the same when child's partitioning is RangePartitioning, sometimes, it could introduce extra shuffle, see:
val df = Seq(1 -> "a", 3 -> "c", 2 -> "b").toDF("i", "j")
val df1 = df.as("t1")
val df2 = df.as("t2")
val t = df1.orderBy("j").join(df2.orderBy("j"), $"t1.i" === $"t2.i", "right")
t.cache.orderBy($"t2.j").explain

Before:
== Physical Plan ==
*(1) Sort [j#14 ASC NULLS FIRST], true, 0
+- Exchange rangepartitioning(j#14 ASC NULLS FIRST, 200)
   +- InMemoryTableScan [i#5, j#6, i#13, j#14]
         +- InMemoryRelation [i#5, j#6, i#13, j#14], CachedRDDBuilder...
               +- *(2) BroadcastHashJoin [i#5], [i#13], RightOuter, BuildLeft
                  :- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as...
                  :  +- *(1) Sort [j#6 ASC NULLS FIRST], true, 0
                  :     +- Exchange rangepartitioning(j#6 ASC NULLS FIRST, 200)
                  :        +- LocalTableScan [i#5, j#6]
                  +- *(2) Sort [j#14 ASC NULLS FIRST], true, 0
                     +- ReusedExchange [i#13, j#14], Exchange rangepartitioning(j#6 ASC NULLS FIRST, 200)

After this PR, we can avoid Exchange rangepartitioning(j#14 ASC NULLS FIRST, 200):
== Physical Plan ==
*(1) Sort [j#14 ASC NULLS FIRST], true, 0
+- InMemoryTableScan [i#5, j#6, i#13, j#14]
      +- InMemoryRelation [i#5, j#6, i#13, j#14], CachedRDDBuilder...
            +- *(2) BroadcastHashJoin [i#5], [i#13], RightOuter, BuildLeft
               :- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)))
               :  +- *(1) Sort [j#6 ASC NULLS FIRST], true, 0
               :     +- Exchange rangepartitioning(j#6 ASC NULLS FIRST, 200)
               :        +- LocalTableScan [i#5, j#6]
               +- *(2) Sort [j#14 ASC NULLS FIRST], true, 0
                  +- ReusedExchange [i#13, j#14], Exchange rangepartitioning(j#6 ASC NULLS FIRST, 200)

How was this patch tested?
Add new tests.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Test build #91829 has started for PR 21564 at commit f37139b.

@cloud-fan @viirya @gatorsmile , could you help review this?

Test build #91829 has finished for PR 21564 at commit f37139b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91829/
Test PASSed.

Test build #91855 has started for PR 21564 at commit 0ef99cc.

@viirya I thinkPartitioningCollection should be considered. Like below case:
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
spark.conf.set("spark.sql.codegen.wholeStage", false)
val df1 = Seq(1 -> "a", 3 -> "c", 2 -> "b").toDF("i", "j").as("t1")
val df2 = Seq(1 -> "a", 3 -> "c", 2 -> "b").toDF("m", "n").as("t2")
val d = df1.join(df2, $"t1.i" === $"t2.m")
d.cache
val d1 = d.as("t3")
val d2 = d.as("t4")
d1.join(d2, $"t3.i" === $"t4.i").explain

SortMergeJoin [i#5], [i#54], Inner
:- InMemoryTableScan [i#5, j#6, m#15, n#16]
:     +- InMemoryRelation [i#5, j#6, m#15, n#16], CachedRDDBuilder
:           +- SortMergeJoin [i#5], [m#15], Inner
:              :- Sort [i#5 ASC NULLS FIRST], false, 0
:              :  +- Exchange hashpartitioning(i#5, 10)
:              :     +- LocalTableScan [i#5, j#6]
:              +- Sort [m#15 ASC NULLS FIRST], false, 0
:                 +- Exchange hashpartitioning(m#15, 10)
:                    +- LocalTableScan [m#15, n#16]
+- Sort [i#54 ASC NULLS FIRST], false, 0
   +- Exchange hashpartitioning(i#54, 10)
      +- InMemoryTableScan [i#54, j#55, m#58, n#59]
            +- InMemoryRelation [i#54, j#55, m#58, n#59], CachedRDDBuilder
                  +- SortMergeJoin [i#5], [m#15], Inner
                     :- Sort [i#5 ASC NULLS FIRST], false, 0
                     :  +- Exchange hashpartitioning(i#5, 10)
                     :     +- LocalTableScan [i#5, j#6]
                     +- Sort [m#15 ASC NULLS FIRST], false, 0
                        +- Exchange hashpartitioning(m#15, 10)
                           +- LocalTableScan [m#15, n#16]

Exchange hashpartitioning(i#54, 10) is extra shuffle.

@mgaido91 I update the codes as per your suggestion, thanks!

Test build #91856 has started for PR 21564 at commit 405ba94.

@yucai thanks, can you please also add more UTs in order to cover all the possible cases? Thanks.

Test build #91855 has finished for PR 21564 at commit 0ef99cc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91855/
Test PASSed.

Test build #91856 has finished for PR 21564 at commit 405ba94.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91856/
Test PASSed.

Merged build finished. Test PASSed.

LGTM except the test

Test build #91968 has started for PR 21564 at commit 9da85e0.

Test build #91970 has started for PR 21564 at commit 85dc1bc.

Test build #91968 has finished for PR 21564 at commit 9da85e0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91968/
Test PASSed.

Test build #91972 has started for PR 21564 at commit 7d1c2f2.

LGTM

Test build #91970 has finished for PR 21564 at commit 85dc1bc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91970/
Test PASSed.

Test build #91972 has finished for PR 21564 at commit 7d1c2f2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91972/
Test PASSed.

@cloud-fan thanks for reviewing, tests have been updated.

Test build #92068 has started for PR 21564 at commit 6744a9e.

Test build #92068 has finished for PR 21564 at commit 6744a9e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92068/
Test FAILed.

retest this please.

Test build #92074 has started for PR 21564 at commit 6744a9e.

Test build #92080 has started for PR 21564 at commit dcd0ce9.

Test build #92074 has finished for PR 21564 at commit 6744a9e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92074/
Test FAILed.

retest this please.

Test build #92082 has started for PR 21564 at commit dcd0ce9.

Test build #92080 has finished for PR 21564 at commit dcd0ce9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92080/
Test PASSed.

Test build #92082 has finished for PR 21564 at commit dcd0ce9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92082/
Test PASSed.

thanks, merging to master!

authors: AmplabJenkins SparkQA cloud-fan mgaido91 viirya yucai


text:
[SPARK-29191][TESTS] Add tag ExtendedSQLTest for SQLQueryTestSuite

What changes were proposed in this pull request?
This PR aims to add tag ExtendedSQLTest for SQLQueryTestSuite.
This doesn't affect our Jenkins test coverage.
Instead, this tag gives us an ability to parallelize them by splitting this test suite and the other suites.
Why are the changes needed?
SQLQueryTestSuite takes 45 mins alone because it has many SQL scripts to run.

Does this PR introduce any user-facing change?
No.
How was this patch tested?
build/sbt "sql/test-only *.SQLQueryTestSuite" -Dtest.exclude.tags=org.apache.spark.tags.ExtendedSQLTest
...
[info] SQLQueryTestSuite:
[info] ScalaTest
[info] Run completed in 3 seconds, 147 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0
[success] Total time: 22 s, completed Sep 20, 2019 12:23:13 PM

Test build #111087 has started for PR 25872 at commit 3d85e88.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/16172/
Test PASSed.

cc @gengliangwang , @gatorsmile , @wangyum , @HyukjinKwon

Is it not possible to exclude a single suite? OK if this is necessary to achieve that, it's just a tag.

Right. So, we use this way.

Test build #111087 has finished for PR 25872 at commit 3d85e88.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111087/
Test PASSed.

What does Extended mean? This tag is only used for SQLQueryTestSuite?
Also, we need [SQL] in the title?

Tag can be reused later. For example, ThriftServerQueryTestSuite extends SQLQueryTestSuite.
Historically, see ExtendedHiveTest tag case. We added it once and reuse it later other places.

Ah, I see.

Hi, @srowen and @maropu .
Could you review this PR once more? Thanks!

Thank you for approval, @srowen .
Merged to master.

Late LGTM

Thank you, @maropu !

authors: AmplabJenkins SparkQA dongjoon-hyun maropu srowen


text:
[SPARK-30346][CORE]Improve logging when events dropped

What changes were proposed in this pull request?
Make logging events dropping every 60s works fine, the orignal implementaion some times not working due to susequent events comming and updating the DroppedEventCounter
Why are the changes needed?
Currenly, the logging may be skipped and delayed a long time under high concurrency, that make debugging hard. So This PR will try to fix it.
Does this PR introduce any user-facing change?
No
How was this patch tested?
NA

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@seanli-rallyhealth Thanks for reply, I've updated as commented.

cc. @vanzin

adding cc. @cloud-fan and @squito for expanding visibility

ok to test

Test build #116192 has started for PR 27002 at commit 5bf09c2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20986/
Test PASSed.

Thanks for reply @vanzin, I think this log can still do much help even after it was logged at first, in real cluster, the logging always configured as rolling by size, and the log is rolling fast, expecially for SparkSQL thriftserver case. With this log we can still know what happened and probably find out what make it slow with the threaddump even after the log was rolled.

The thing is that the current event dispatcher is processing now may not be the culprit of slowdown, and the log could give wrong information which points out innocent one which would lead us wasting time to investigate. What's worse, we cannot determine whether the information is correct or not. Please correct me if I'm missing here.

Test build #116192 has finished for PR 27002 at commit 5bf09c2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116192/
Test PASSed.

@HeartSaVioR I agree that this log can not definitely tell us where the slow must be, but at least can provide some clues, it's no worse than no clues. Moreover, I think if there are some hot points are really slow, then there are larger probability that it will appear in the periodic log.

Quoting my comment again:

Spark will introduce a new configuration to log the event and listener name if the execution takes more than configured threshold; it would be more helpful. The commit is 0346afa

@liupc Any update here?

@liupc Any update here?

Sorry for late reply, already fixed the indent.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23276/
Test PASSed.

Test build #118521 has started for PR 27002 at commit a216f58.

Test build #118521 has finished for PR 27002 at commit a216f58.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118521/
Test PASSed.

thanks, merging to master!

also merged to branch-3.0.

authors: AmplabJenkins HeartSaVioR HyukjinKwon SparkQA cloud-fan liupc vanzin


text:
[SPARK-4017] show progress bar in console

The progress bar will look like this:

In the right corner, the numbers are: finished tasks, running tasks, total tasks.
After the stage has finished, it will disappear.
The progress bar is only showed if logging level is WARN or higher (but progress in title is still showed), it can be turned off by spark.driver.showConsoleProgress.

How to turn it off on earth? Below does not work.
log4j.rootCategory=ERROR, console
spark.driver.showConsoleProgress=false
spark.ui.showConsoleProgress=false

spark.ui.showConsoleProgress=false works for me. I pass it via --conf to spark-submit. Try that if you haven't already.

I run it in eclispe. Every interval I see 200+ lines of "===>".

authors: davies leonsoft nchammas


text:
[SPARK-34141][SQL] Remove side effect from ExtractGenerator

What changes were proposed in this pull request?

Rewrote one ExtractGenerator case such that it would not rely on a side effect of the flatmap function.
Why are the changes needed?

With the dataframe api it is possible to have a lazy sequence as the output of a LogicalPlan. When exploding a column on this dataframe using the withColumn("newName", explode(col("name"))) method, the ExtractGenerator does not extract the generator and CheckAnalysis would throw an exception.
Does this PR introduce any user-facing change?

Bugfix
Before this, the work around was to put .select("*") before the explode.
How was this patch tested?

UT

@HyukjinKwon , any chance that this bugfix could get to 3.1.1?

@HyukjinKwon , any chance that this bugfix could get to 3.1.1?

Test build #134154 has started for PR 31213 at commit 08e367c.

Test build #134154 has started for PR 31213 at commit 08e367c.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38737/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38737/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38737/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38737/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38737/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38737/

Test build #134154 has finished for PR 31213 at commit 08e367c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134154 has finished for PR 31213 at commit 08e367c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134154/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134154/

Test build #134158 has started for PR 31213 at commit cefc79f.

Test build #134158 has started for PR 31213 at commit cefc79f.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38741/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38741/

@LuciferYang, you have been working with 2.13.
Any recommendations on how to get a lazy Seq on both 2.12 and 2.13.
On 2.13 the Seq.view does not return a Seq anymore.

@LuciferYang, you have been working with 2.13.
Any recommendations on how to get a lazy Seq on both 2.12 and 2.13.
On 2.13 the Seq.view does not return a Seq anymore.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38741/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38741/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38741/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38741/

Test build #134158 has finished for PR 31213 at commit cefc79f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134158 has finished for PR 31213 at commit cefc79f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134158/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134158/

@tanelk In Scala 2.13, Views  are collections whose transformation operations are non strict and they no longer extend their corresponding collection type, for example, an IndexedSeqView no longer extends IndexedSeq.
Why do we need a non strict collection here? Can it be a strict collection?
If it must be a non strict collection, may need to define the type of output as scala.collection.SeqView for version compatibility.

@tanelk In Scala 2.13, Views  are collections whose transformation operations are non strict and they no longer extend their corresponding collection type, for example, an IndexedSeqView no longer extends IndexedSeq.
Why do we need a non strict collection here? Can it be a strict collection?
If it must be a non strict collection, may need to define the type of output as scala.collection.SeqView for version compatibility.

Well that is the test case I'm trying to cover.
in 2.12 the input can be lazy Seq, but the current code breaks then.
Perhaps it's enough to say that I manually tested it and not add the UT.

Well that is the test case I'm trying to cover.
in 2.12 the input can be lazy Seq, but the current code breaks then.
Perhaps it's enough to say that I manually tested it and not add the UT.

cc @srowen

cc @srowen

Well that is the test case I'm trying to cover.
in 2.12 the input can be lazy Seq, but the current code breaks then.
Perhaps it's enough to say that I manually tested it and not add the UT.

New UT in this pr will break the compilation of Scala 2.13,  @tanelk want to get a lazy Seq on both 2.12 and 2.13, but SeqView is not Seq in Scala 2.13,  do you have any suggestions ? @srowen

Well that is the test case I'm trying to cover.
in 2.12 the input can be lazy Seq, but the current code breaks then.
Perhaps it's enough to say that I manually tested it and not add the UT.

New UT in this pr will break the compilation of Scala 2.13,  @tanelk want to get a lazy Seq on both 2.12 and 2.13, but SeqView is not Seq in Scala 2.13,  do you have any suggestions ? @srowen

Hm why does Seq vs SeqView matter here? what's the compile error?

Hm why does Seq vs SeqView matter here? what's the compile error?

@srowen
There is a correctness issue, when an lazy Seq is given as the project list (for example SeqView).
I did fix it, but the unit test does not compile on 2.13, because there SeqView does not implement Seq any more.
Would it be okay do not add the UT or is there some lazy Seq present on both 2.12 and 2.13.

@srowen
There is a correctness issue, when an lazy Seq is given as the project list (for example SeqView).
I did fix it, but the unit test does not compile on 2.13, because there SeqView does not implement Seq any more.
Would it be okay do not add the UT or is there some lazy Seq present on both 2.12 and 2.13.

Hm, tough call. Can you invoke the second constructor by adding :_* to the end of the argument? But then I suppose that's not testing what you want to test either. Can you pass a Scala Stream here maybe? not sure if that works. It may work to create a new test-suite in a 2.12-only src/test directory if that's not too much trouble. It sounds like this can't happen in 2.13 anyway so the test isn't relevant.

Hm, tough call. Can you invoke the second constructor by adding :_* to the end of the argument? But then I suppose that's not testing what you want to test either. Can you pass a Scala Stream here maybe? not sure if that works. It may work to create a new test-suite in a 2.12-only src/test directory if that's not too much trouble. It sounds like this can't happen in 2.13 anyway so the test isn't relevant.

Test build #134511 has started for PR 31213 at commit 8dd6da0.

Test build #134511 has started for PR 31213 at commit 8dd6da0.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39096/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39096/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39096/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39096/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39096/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39096/

I went ahead and added version dependent tests. Thanks for the advice.

I went ahead and added version dependent tests. Thanks for the advice.

Test build #134511 has finished for PR 31213 at commit 8dd6da0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ExtractGeneratorSuite extends AnalysisTest
class ExtractGeneratorSuite extends AnalysisTest

Test build #134511 has finished for PR 31213 at commit 8dd6da0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ExtractGeneratorSuite extends AnalysisTest
class ExtractGeneratorSuite extends AnalysisTest

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134511/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134511/

Test build #134689 has started for PR 31213 at commit 82ca8f5.

Test build #134689 has started for PR 31213 at commit 82ca8f5.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39275/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39275/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39275/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39275/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39275/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39275/

Test build #134689 has finished for PR 31213 at commit 82ca8f5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134689 has finished for PR 31213 at commit 82ca8f5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134689/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134689/

Merged to master. I'm satisfied with the motivation and test.

Oh, just saw it. LGTM

authors: AmplabJenkins HyukjinKwon LuciferYang SparkQA srowen tanelk


text:
[SPARK-34256][ML] VectorSlicer refine numFeatures checking and toString method

What changes were proposed in this pull request?
1, update checking of numFeatures;
2, update toString to take names into account;
Why are the changes needed?
1, should use inputAttr.size instead of inputAttr.numAttributes to get numFeatures;
2, this checking is necessary only if $(indices).nonEmpty, otherwise, $(indices).max will throw exception java.lang.UnsupportedOperationException: empty.max
3, in toString, should add length of names;
Does this PR introduce any user-facing change?
Yes, toString now count names;
How was this patch tested?
existing testsuites

Test build #134538 has started for PR 31354 at commit 877ffbf.

Test build #134538 has started for PR 31354 at commit 877ffbf.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39123/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39123/

Test build #134538 has finished for PR 31354 at commit 877ffbf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134538 has finished for PR 31354 at commit 877ffbf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134538/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134538/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39123/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39123/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39123/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39123/

Test build #134577 has started for PR 31354 at commit de3f395.

Test build #134577 has started for PR 31354 at commit de3f395.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39163/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39163/

Test build #134577 has finished for PR 31354 at commit de3f395.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134577 has finished for PR 31354 at commit de3f395.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134577/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134577/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39163/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39163/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39163/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39163/

merged to master, thanks @srowen for reviewing!

authors: AmplabJenkins SparkQA zhengruifeng


text:
[SPARK-36665][SQL][FOLLOWUP] Avoid Optimizing Not(InSubquery)

What changes were proposed in this pull request?
This is a follow-up PR to fix the bug introduced by SPARK-36665. With this fix, NotPropagation optimization does not apply to InSubquery cases.
Why are the changes needed?
NotPropagation optimization previously broke RewritePredicateSubquery so that it does not properly rewrite the predicate to a NULL-aware left anti join anymore.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Unit test added

@aokolnychyi Thank you for finding the issue. cc @viirya

cc @cloud-fan

cc @allisonwang-db FYI

Are there any real-world examples to demonstrate the effectiveness of the rule NotPropagation? I'm a bit worried about making this rule more complicated while this rule has nearly no visible benefit...

Thanks @cloud-fan My goal here is to unblock #35395. As you mentioned your concern about the complexity of this logic, I opened #35428 to remove NotPropagation for now.
For the effectiveness, I put many examples at https://github.com/apache/spark/blob/977dd054ed0946b62e62d2d480dbf25598545a5e/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/NotPropagationSuite.scala#L65-L168 that helps simplifying users' queries.

Closing by preferring https://github.com/apache/spark/pull/35428

authors: HyukjinKwon cloud-fan kazuyukitanimura viirya


text:
[SPARK-38122][Docs] Update the App Key of DocSearch

What changes were proposed in this pull request?

Update the App Key of DocSearch as per the comment https://github.com/algolia/docsearch-configs/pull/5054#issuecomment-1025483446
Why are the changes needed?

DocSearch uses new infra now: https://docsearch.algolia.com/docs/migrating-from-legacy/
The new version allows us to manage indexes, but we have to update the App Key and App id in configuration to make it work.
Does this PR introduce any user-facing change?

No
How was this patch tested?

Manual verified in local setup.

We need to manually update the docs of 3.2.1 after merging this one as well

Merging to master/3.2

Thank you, @gengliangwang .

authors: dongjoon-hyun gengliangwang