text:
[spark streaming] Support spark direct stream

Currently, we only support receiver stream. See https://spark.apache.org/docs/2.2.0/streaming-kafka-0-8-integration.html#approach-2-direct-approach-no-receivers

See comment at https://github.com/apache/pulsar/issues/4585#issuecomment-504836970

@liketic I have to take my previous back. It seems that you were asking for a feature on spark streaming. The one I commented in #4585 is about Spark SQL and Spark Structured Streaming. I don't think anyone is working on a direct stream implementation. If you are interested in contributing, feel free to contribute.

@sijie OK. I'll take a look.

Close this issue since there is already a pulsar-spark connector for structured streaming. https://github.com/streamnative/pulsar-spark

@sijie is this connector https://github.com/streamnative/pulsar-spark based on direct approach or reciever based approach?
I have asked same question here can you please help me understand what use cases both the adaptors are solving?

authors: liketic maneeshbhunwal123 sijie


text:
Messages that have already been acked are redelivered when upgrading Pulsar version

The other day, we upgraded the Pulsar version of our broker servers from 2.3.2 to 2.4.2.
At that time, a large number of messages were redelivered on some topics. These messages should have already been acked in the past. The topics where this happened had some unacked messages in their backlog. I can understand that the unacked messages were redelivered. The strange thing is that some or all of the acked messages that followed the unacked messages were also redelivered.
We have found that this happens when the Pulsar 2.3.2 broker that owns the topic in question is shut down and the topic is moved to the Pulsar 2.4.2 broker. The cause seems to be the difference in behavior of individualDeletedMessages held by ManagedCursorImpl instance.
In Pulsar 2.3.2, individualDeletedMessages is an instance of the TreeRangeSet class. Instances of TreeRangeSet can contain ranges that span different ledgers.
Verification code 1:
RangeSet<PositionImpl> rangeSet = TreeRangeSet.create();
rangeSet.add(Range.openClosed(new PositionImpl(1, 100), new PositionImpl(2, 200)));
System.out.println(rangeSet);

Result 1:
[(1:100..2:200]]

On the other hand, in Pulsar 2.4.2, individualDeletedMessages is an instance of ConcurrentOpenLongPairRangeSet by default. If a range that spans multiple ledgers is added to this instance, the information of the first half will be lost. It seems that adding such a range to ConcurrentOpenLongPairRangeSet is not allowed.
Verification code 2:
LongPairConsumer<PositionImpl> positionRangeConverter = (key, value) -> new PositionImpl(key, value);
LongPairRangeSet<PositionImpl> rangeSet = new ConcurrentOpenLongPairRangeSet<PositionImpl>(4096, positionRangeConverter);
rangeSet.addOpenClosed(1, 100, 2, 200);
System.out.println(rangeSet);

Result 2:
[(2:-1..2:200]]

And if we set managedLedgerUnackedRangesOpenCacheSetEnabled to false on Pulsar 2.4.2 broker, individualDeletedMessages will be an instance of LongPairRangeSet.DefaultRangeSet. Such a range can be added to LongPairRangeSet.DefaultRangeSet.
Verification code 3:
LongPairConsumer<PositionImpl> positionRangeConverter = (key, value) -> new PositionImpl(key, value);
LongPairRangeSet<PositionImpl> rangeSet = new LongPairRangeSet.DefaultRangeSet<>(positionRangeConverter);
rangeSet.addOpenClosed(1, 100, 2, 200);
System.out.println(rangeSet);

Result 3:
[(1:100..2:200]]

This difference in behavior causes redelivery of acked messages. For example, suppose a topic owned by a Pulsar 2.3.2 broker has the following individuallyDeletedMessages:
"individuallyDeletedMessages" : "[(2625703:-1..2625703:9], (2625719:-1..2625727:9]]",

When this Pulsar 2.3.2 broker is shut down and the topic moves to a Pulsar 2.4.2 broker, the individuallyDeletedMessages changes as follows:
"individuallyDeletedMessages" : "[(2625703:-1..2625703:9],(2625727:-1..2625727:9]]",

The messages with ledger ID 2625719 have already been acked, but after the topic moves to the Pulsar 2.4.2 broker, that information is lost and these messages are redelivered to the consumers.

The class of individualDeletedMessages has been changed by the following two PRs:

https://github.com/apache/pulsar/pull/3818
https://github.com/apache/pulsar/pull/3819

@rdhabalia Is the difference in behavior between ConcurrentOpenLongPairRangeSet and TreeRangeSet/LongPairRangeSet.DefaultRangeSet intentional? If so, what is the reason?

@codelipenghui Can you check this issue?

source: gharchive/issue

authors: massakam sijie

repo: apache/pulsar

license_type: permissive

license_source: github-api


text:
Support Message Dispatch Rate on topic level

Is your feature request related to a problem? Please describe.
This is the sub feature of #2688
Describe the solution you'd like
support Get, Set, Remove message dispatch rate on topic level
Describe alternatives you've considered
Additional context

Close by https://github.com/apache/pulsar/pull/8087


text:
[Tests] Fix flaky test GracefulExecutorServicesShutdownTest

Motivation
GracefulExecutorServicesShutdownTest's shouldTerminateWhenFutureIsCancelled remains flaky after #10592 .
Example failure: https://github.com/apache/pulsar/pull/10598/checks?check_run_id=2590214027#step:10:1341
Modifications

fix race condition in test by adding a CountDownLatch to verify that execution has entered the awaitTermination method before the future is cancelled.
use CompletableFuture for awaitTerminationInterrupted check.

What caused the flakiness was that this condition caused the execution to never get to the Thread.sleep when the thread was already interrupted:

Yes, I agree that.
But after call countdown, main thread can cancel the future, that means the Interruption exception can probably be caught by outer catch.
You can test it locally by add some time-consuming action after countdown.

@linlinnn I have merged this patch in order to unblock CI.
We can continue the discussion and find a better fix if think that this is not the best way

ok

But after call countdown, main thread can cancel the future, that means the Interruption exception can probably be caught by outer catch.
You can test it locally by add some time-consuming action after countdown.

I don't see how this could happen. Interrupting a thread doesn't stop execution. It sets the interrupted status and interrupts any blocking methods such as sleep or blocking IO with an InterruptedException. Calling countDown will never be interrupted.
Makes sense?

@lhotari
Oh, I make a wrong understand with the outer catch InterruptedException in GracefulExecutorServicesTerminationHandler#awaitTermination since executor.awaitTermination has been mocked.
Sorry for my mistake.

Oh, I make a wrong understand with the outer catch InterruptedException in GracefulExecutorServicesTerminationHandler#awaitTermination since executor.awaitTermination has been mocked.
Sorry for my mistake.

No worries, the mocking was quite unexpected for executor.awaitTermination.
btw. It seems that this test became very flaky after the switch to JDK11. To get the fix to other PRs, it is either necessary to rebase or close and re-open a PR so that the changes get picked up. When re-running a failed build it won't pick up changes made into master branch. New PR builds or reopened PR builds will pick up changes.

source: gharchive/pull-request

authors: eolivelli lhotari linlinnn


text:
expose producer name to getMessageById for Admin API

Motivation
Current we can get Message from Pulsar Admin API like getMessageById or examineMessage, but Message.getProducerName() return null. In some case we may need to known the producer name of the message.
Modifications
expose producer name to MessageMetadata in Message for Pulsar Admin API.

/pulsarbot run-failure-checks

@wangjialing218 Can you rebase the pull request?

@wangjialing218 Can you rebase the pull request?

@sijie This PR could be close because the feature already included in #11443


text:
[improve][client] Use Jackson's ObjectReader in Client's JSON readers

Motivation

benefits:

no need to lookup the type when reading happens.

The type gets fetched from cache once.
Since a reference is kept, it won't overflow from a busy Jackson TypeFactory LRU cache.

Modifications
Replace ObjectMapper usage with ObjectReader in JacksonJsonReader and JsonReader classes.
Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Codecov Report

Merging #19215 (9cc0894) into master (d7f8f56) will decrease coverage by 36.22%.
The diff coverage is 15.38%.

@@              Coverage Diff              @@
##             master   #19215       +/-   ##
=============================================
- Coverage     47.89%   11.66%   -36.23%
+ Complexity    10871     2432     -8439
=============================================
  Files           713      633       -80
  Lines         69730    59928     -9802
  Branches       7496     6251     -1245
=============================================
- Hits          33394     6988    -26406
- Misses        32631    52186    +19555
+ Partials       3705      754     -2951

Flag
Coverage Δ

unittests
11.66% <15.38%> (-36.23%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...va/org/apache/pulsar/client/impl/ProducerImpl.java
15.62% <0.00%> (-0.04%)
:arrow_down:

...e/pulsar/client/impl/schema/reader/JsonReader.java
0.00% <0.00%> (ø)

...r/client/impl/schema/reader/JacksonJsonReader.java
66.66% <66.66%> (-6.67%)
:arrow_down:

...n/java/org/apache/pulsar/client/api/RawReader.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pulsar/broker/admin/v1/Brokers.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pulsar/broker/admin/v2/Brokers.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pulsar/broker/admin/v2/Tenants.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...va/org/apache/pulsar/broker/admin/v1/Clusters.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...va/org/apache/pulsar/broker/admin/v2/Clusters.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../org/apache/pulsar/broker/admin/v1/Properties.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 335 more

LGTM. @lhotari - do you have documentation indicating this is the preferred approach? Thanks.

Do you mean the use of ObjectReader instead of ObjectMapper?  Or the use of readerFor?

authors: codecov-commenter lhotari


text:
[fix] [broker] Let Pending ack handler can retry to init when encounters a metadata store error

Motivation
The Pending Ack handler can not recover after encountering a MetadataStoreException even the metadata store has been re-established.
2024-07-24T02:56:11,347+0000 [pulsar-transaction-executor-5-1] ERROR org.apache.pulsar.broker.transaction.pendingack.impl.PendingAckHandleImpl - [persistent://{tenant}/{ns}/{topic}] [multiTopicsReader-5c010a1ec8] PendingAckHandleImpl init fail!
org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /managed-ledgers/{tenant}/{namespace}/persistent/{topic}-multiTopicsReader-5c010a1ec8__transaction_pending_ack
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final]
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:101)
	at org.apache.pulsar.broker.transaction.pendingack.impl.PendingAckHandleImpl.lambda$init$1(PendingAckHandleImpl.java:185) ~[io.streamnative-pulsar-broker-3.0.4.6.jar:3.0.4.6]
        ...

Modifications
Add MetadataStoreException and BKException into the retriable error list
Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository: x

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 74.56%. Comparing base (bbc6224) to head (08f0209).
Report is 509 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23153      +/-   ##
============================================
+ Coverage     73.57%   74.56%   +0.98%
- Complexity    32624    33603     +979
============================================
  Files          1877     1919      +42
  Lines        139502   144268    +4766
  Branches      15299    15777     +478
============================================
+ Hits         102638   107570    +4932
+ Misses        28908    28458     -450
- Partials       7956     8240     +284

Flag
Coverage Δ

inttests
27.90% <ø> (+3.32%)
:arrow_up:

systests
24.73% <ø> (+0.41%)
:arrow_up:

unittests
73.91% <ø> (+1.06%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files
Coverage Δ

...nsaction/pendingack/impl/PendingAckHandleImpl.java
72.97% <ø> (+2.12%)
:arrow_up:

... and 504 files with indirect coverage changes

authors: codecov-commenter poorbarcode


text:
Use individual netty-* artifacts instead of netty-all

Motivation
All of Pulsar dependencies are using the individual dependencies instead of getting Netty-all. We should do the same to avoid doing the hacky exclusions stuff. This also reduces a bit the sizes of our shaded jars and dist file.

did you get change to run broker with this change? just want to make sure that we won't have any runtime exception here..

The integration tests job is using the docker images, which are built from the distribution tgz. Any jar missing at runtime will show up there.
(it does actually appear there some issue with the functions runtime deps on https://builds.apache.org/job/pulsar_precommit_integrationtests/6594/ )

run java8 tests

run java8 tests

run java8 tests

run java8 tests

run java8 tests

run java8 tests

run java8 tests
run integration tests

run java8 tests

authors: merlimat rdhabalia


text:
[Bug] The proxy in the cluster mode obtains the wrong address of the broker

Before Creating the Bug Report

[X] I found a bug, not just asking a question, which should be created in GitHub Discussions.

[X] I have searched the GitHub Issues and GitHub Discussions  of this repository and believe that this is not a duplicate.

[X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

Runtime platform environment
OS: CentOS 6.9
RocketMQ version
branch: (develop|tag 5.1.1) version: 5.1.1
JDK Version
JDK: 1.8.0_202
Describe the Bug
When I connect to the clustered proxy to consume messages using the remoting protocol, some messages from one broker cannot be consumed.
After debugging, I found that the obtained offset of broker is wrong.

As shown in the figure above, when ClusterTopicRouteService.getBrokerAddr() obtains the broker address, the brokerName in the parameter is broker-a, but the obtained address is broker-b.
Steps to Reproduce

Deploy a new broker-a, start it.
Create some topics in broker-a for testing, such as message produing and consuming, that is not important.
When I deploy a new broker, such as broker-b,  I need copy topics.json from broker-a to broker-b, then start it.
Now broker-b‘s topics have broker-a and broker-b.

When ClusterTopicRouteService.getBrokerAddr() obtains the broker address, the code's logic did not consider this situation(Topic broker-a exist in broker-a and broker-b), so got the wrong broker addr.
What Did You Expect to See?
Got the right broker addr.
What Did You See Instead?
Got the wrong broker addr.
Additional Context
No response

PR is https://github.com/apache/rocketmq/pull/6909

repo: apache/rocketmq


text:
[Feature][Connector-V2][Clickhouse] Add clickhouse connector time zone key,default system time zone

Purpose of this pull request

Check list

[ ] Code changed are covered with tests, or it does not need tests for reason:
[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update change log that in connector document. For more details you can refer to connector-v2
Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist

[ ] Update the release-note.

[Bug] [zeta] clickhouse to Doris，8 hours difference in datetime type #4935

repo: apache/seatunnel


text:
Added support for enabling debug logs for okhttp

Support setting logging.level to debug to display okhttp detailed log, easy to locate & find issues.
logging.level.org.apache.shenyu.admin.utils.HttpUtils: debug
Make sure that:

[x] You have read the contribution guidelines.
[ ] You submit test cases (unit or integration tests) that back your changes.
[x] Your local test passed ./mvnw clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #4976 (219df2e) into master (a6a56cf) will decrease coverage by 0.13%.
Report is 8 commits behind head on master.
The diff coverage is 57.14%.

@@             Coverage Diff              @@
##             master    #4976      +/-   ##
============================================
- Coverage     63.93%   63.81%   -0.13%
+ Complexity     8528     8527       -1
============================================
  Files          1208     1208
  Lines         35899    35906       +7
  Branches       3217     3213       -4
============================================
- Hits          22953    22913      -40
- Misses        11095    11140      +45
- Partials       1851     1853       +2

Files Changed
Coverage Δ

.../java/org/apache/shenyu/admin/utils/HttpUtils.java
37.50% <57.14%> (+0.77%)
:arrow_up:

... and 10 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

pls add license in shenyu-dist/shenyu-admin-dist

license not right

authors: codecov-commenter kerwin612 moremind

repo: apache/shenyu


text:
getAllService() return duplicated service

Please answer these questions before submitting your issue.

Why do you submit this issue?
[X] Question or discussion
[ ] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?

We found getAllService() return duplicated service?
I want to know how skywalking Guarantee ES has unique ServiceTraffic Data.
getAllService() will query All index which has prefix "service_traffic".

Please search before submitting a new issue.

I searched duplicated service, only found #1124.

The simple version is, OAP query the storage first, if not found, it inserts a new one. If somehow ES server is not fully functional, such as a shard is not working, you will face this case.
We could simply do distinct in the OAP query and pretend this never happens, but considering the reality, we don't. Because when you see this, it means your ES server is not working well as you thought, some metrics are not accurate too.

repo: apache/skywalking

license_source: bigquery


text:
[Core] TSDB basic data management

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
TSDB supports essential storage structure management, containing the below feats:

[x] Open the database. If the path contains nothing, tsdb creates a new file layout. (https://github.com/apache/skywalking-banyandb/pull/64)
[ ] Introduce the lock and manifest files to manage file layout.
[x] Make blocks and segments move forward, which means new entries emerge and old ones are pruned. TSDB will introduce a new strategy component to collect metrics and determine when and how to create new blocks. (https://github.com/apache/skywalking-banyandb/pull/70)
[ ] Implement a block controller which determines how many blocks should be active. It closes blocks based on the LRU algorithm.
[ ] Enhance the data scanner to select blocks based on the query criteria. The scanner has to apply to the block controller for blocks if the blocks are inactive.
[ ] Writing data to the cold blocks which might be closed or never built.

Use case
No response
Related issues
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Due to the downSampling in the OAP server side, we have to support writing data to a block older than the current open ones.
For example, we start the banyandb server at "2022-01-27T01:40:20Z", the oldest block may start from "2022-01-27T01:40:20Z" while the OAP server would send stream entities with timestamp "2022-01-27T01:00:00Z" due to the downSampling at hour level.

Due to the downSampling in the OAP server side, we have to support writing data to a block older than the current open ones.
For example, we start the banyandb server at "2022-01-27T01:40:20Z", the oldest block may start from "2022-01-27T01:40:20Z" while the OAP server would send stream entities with timestamp "2022-01-27T01:00:00Z" due to the downSampling at hour level.

Added this to the task list.

Move the cold writing to

Due to the downSampling in the OAP server side, we have to support writing data to a block older than the current open ones.
For example, we start the banyandb server at "2022-01-27T01:40:20Z", the oldest block may start from "2022-01-27T01:40:20Z" while the OAP server would send stream entities with timestamp "2022-01-27T01:00:00Z" due to the downSampling at hour level.

Added this to the task list.

Moved to https://github.com/apache/skywalking/issues/8815


text:
Add Spring Sleuth dashboard to general service instance template

Please help to check the UI template, then I will upload the screenshot to the website and update the setup documentation.

[ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
[x] Update the CHANGES log.

I think we should update the relative docs accordingly.
This doc is still posting v8's screenshots, https://skywalking.apache.org/docs/main/latest/en/setup/backend/spring-sleuth-setup/

authors: mrproliu wu-sheng


text:
Fix a cross-site scripting vulnerability in the SlingLogPanel.

The log appender name is a user provided value originating here, therefore it should not be written to the output HTML without being sanitized.

I found this issue through lgtm.com, a service that analyzes open-source code to look for potential problems. (Full disclosure: I work for the company that operates it.)

Which versions are exactly affected? We are using 2.11.0. Thank you

From what I can tell, this has been present since mid-2015. I'm not sure what versions that means it affects though.

With the upcoming migration to git this repository will be deleted. If the pull request contains data that should be re-submitted I suggest that you save the changes locally and resubmit them after the migration, which will likely happen on Monday the 9th of October.

authors: chrisgavin h3xspirit rombert

repo: apache/sling


text:
Issue with wrong X-Spam-Date-of-Scan

X-Spam-Date-of-Scan was totally wrong, something like 1965.
I shifted the @_ to correct the problem like said here : http://www.gossamer-threads.com/lists/spamassassin/users/185212

Please use the SpamAssassin Bugzilla to open an issue and attach the patch there. SpamAssassin uses the Apache.org svn repository. For projects that use svn, ASF mirrors the svn repository in a read-only git copy. Your pull request generated an email  to the developers mailing list from the mirrored git repository that got stuck in moderation as not coming from a subscribed address. Since we use svn we are not set up to deal with pull requests sent to the read-only mirrored git repository.
Thanks,
One of the dev@ mailing list moderators for Apache SpamAssassin

Thank you sidney for these informations. I opened a ticket on directly on the bugzilla.
Thanks again :smiley:
Geoffrey

repo: apache/spamassassin


text:
[WIP, DO-NOT-MERGE][SPARK-14473][SQL] Define analysis rules to catch operations not supported in streaming

What changes were proposed in this pull request?
There are many operations that are currently not supported in the streaming execution. For example:

joining two streams
unioning a stream and a batch source
sorting
window functions (not time windows)
distinct aggregates

Furthermore, executing a query with a stream source as a batch query should also fail.
This patch add an additional step after analysis in the QueryExecution which will check that all the operations in the analyzed logical plan is supported or not.
Todo

[ ] Logical Plans

[ ] Decide whether the following are supported

[ ] Pivot

[ ] Tests for each unsupported operation

[ ] Dataset API

[ ] actions like count, collect, etc.
[ ] dedup?

How was this patch tested?
unit tests.

Test build #55267 has started for PR 12246 at commit 87b9782.

Test build #55267 has finished for PR 12246 at commit 87b9782.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait UnsupportedOperationChecker
class QueryExecution(val sqlContext: SQLContext, val logical: LogicalPlan)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55267/
Test FAILed.

Merged build finished. Test FAILed.

Test build #55271 has started for PR 12246 at commit de3e298.

Test build #55271 has finished for PR 12246 at commit de3e298.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55271/
Test FAILed.

Merged build finished. Test FAILed.

Test build #55273 has started for PR 12246 at commit edd1ac1.

Test build #55273 has finished for PR 12246 at commit edd1ac1.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55273/
Test FAILed.

Test build #55306 has started for PR 12246 at commit 6e8c957.

Test build #55306 has finished for PR 12246 at commit 6e8c957.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55306/
Test FAILed.

Test build #55406 has started for PR 12246 at commit bbf9f27.

Test build #55406 has finished for PR 12246 at commit bbf9f27.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55406/
Test PASSed.

Test build #55547 has started for PR 12246 at commit cbcb69d.

Test build #55547 has finished for PR 12246 at commit cbcb69d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55547/
Test FAILed.

Test build #55553 has started for PR 12246 at commit cc7baab.

Test build #55553 has finished for PR 12246 at commit cc7baab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55553/
Test PASSed.

Test build #55572 has started for PR 12246 at commit 64ed243.

Test build #55572 has finished for PR 12246 at commit 64ed243.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55572/
Test FAILed.

Test build #55590 has started for PR 12246 at commit 6aa5554.

Test build #55590 has finished for PR 12246 at commit 6aa5554.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55590/
Test PASSed.

Some minor comments, otherwise LGTM.

There was a critical bug that made it not test the non-supported cases. Fixed it and updated the PR.

Test build #55890 has started for PR 12246 at commit 3f1353e.

Test build #55890 has finished for PR 12246 at commit 3f1353e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55890/
Test PASSed.

Test build #55939 has started for PR 12246 at commit a5d180e.

Test build #55939 has finished for PR 12246 at commit a5d180e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55939/
Test PASSed.

Merged build finished. Test PASSed.

LGTM

Test build #55956 has started for PR 12246 at commit 8a71835.

Test build #55956 has finished for PR 12246 at commit 8a71835.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55956/
Test FAILed.

retest this please

Looks MemorySinkSuite is still flakey...

Test build #55976 has started for PR 12246 at commit 8a71835.

Test build #55976 has finished for PR 12246 at commit 8a71835.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55976/
Test PASSed.

Merged build finished. Test PASSed.

Thanks, merging into master

authors: AmplabJenkins SparkQA marmbrus tdas zsxwing

repo: apache/spark


text:
[SPARK-13547] [SQL] [WEBUI] Add SQL query in web UI's SQL Tab

What changes were proposed in this pull request?
For the web UI, this PR adds the raw SQL query text for each query in the new column "SQL Text" in the SQL Tab. To achieve this, it also passes the SQL text from the parser to the execution UI data, through the logical plan.
If the query has no related query text, show a blank in that field. If there is no query in the table has a query text, not show the column.
If the query text is long (i.e. longer than 140 chars), show the short abstract and display the full version on demand by the link "+more".
If the query text is too long (i.e. longer than 1000 chars), it will be truncated after parsing. This is to avoid downing browsers or web UI servers by running out of memory.
Also, a tooltip is added on the header "SQL Text", saying Shows 140 characters by default. Click "+more" to see more. Long texts are truncated to 1000 characters. Left blank when the query was not issued by SQL..
How was this patch tested?
Tested by trying multiple SQL queries.
A screenshot of the SQL Tab:

(Query 3 is identical to Query 2, but its SQL text is shown in detail by clicking "+more")

Test build #62180 has started for PR 14158 at commit 626f3f7.

Test build #62180 has finished for PR 14158 at commit 626f3f7.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62180/
Test FAILed.

Build finished. Test FAILed.

Checked it out and gave it a test run and it looks good, just one issue. When the user only uses the dataframes api it will never show any SQL Text (only "-"), should we check if there is no SQL Text to display and not show the column in that case?

Thanks, @ajbozarth. That's a great point. When no execution in a table has SQL text, this column shouldn't be shown. I'll fix this.

Test build #62207 has started for PR 14158 at commit beba227.

In the commit above, I made three adjustments:

If the query has no related query text, show a blank in that field.

If there is no query in the table has a query text, not show the column.

If the query text is too long (i.e. longer than 140 chars), show the short abstract and display the full version on demand.

A snapshot of the SQL Tab:

(The detailed SQL text for Query 4 shown after clicking "+more")
The following need to be discussed and implemented (suggested by @yhuai):

Add a ToolTip on "SQL Text", saying "Blank if the query is not issued by SQL" (Please help check this sentence if you are a native English speaker).

For now, we could show the abstract of the string at the front end. But we still need to truncate the query texts which are toooooo long (maybe longer than 1000 bytes?). This is to avoid downing browsers or web UI servers by running out of memory.

Test build #62207 has finished for PR 14158 at commit beba227.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62207/
Test PASSed.

Checked out your changes and I like the updates, plus I think the column looks better on the end. Addressing the questions you raised: i agree we should truncate long queries (1000 chars sounds like a decent number), we should make it clear they were truncated though (maybe with a "..."?) I also think a tool tip would be good. Maybe something like: "Left blank when the query was not issued by SQL"?

@ajbozarth @felixcheung Thanks for viewing and discussing! I plan to implement the two updates I mentioned above. The idea of clipboard sounds awesome, but I think we might need more discussion about this new function.

Test build #62364 has started for PR 14158 at commit 41c2daa.

Updated by truncating long texts and adding a tooltip.
The detail description and the screenshot at https://github.com/apache/spark/pull/14158#issue-165127460 is also updated.

Test build #62364 has finished for PR 14158 at commit 41c2daa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62364/
Test PASSed.

I'll try to take a look at this tomorrow

The tooltip looks odd centered on the column instead of the title text but I believe that's how the whole Web UI works, other than that nit LGTM

@srowen @JoshRosen what do you think of @nblintao 's pr? Some of my team really likes the addition and would love to see it merged

@nblintao Could you rebase? I'd like to try and see if we can get this reviewed and merged.

Please be certain that this works well even for very large queries.  They are not commonplace, but I know that Spark SQL does sometimes get asked to handle SQL queries that are hundreds of lines  long.

Thanks @markhamstra that was already addressed earlier in review

It's great if it is already addressed, but I just didn't see anything explicit in the discussion or examples that showed any query of the magnitude that I am talking about.  Machine-generated queries can be staggeringly large (e.g. I've seen queries that were so long and complicated that they took Spark SQL more than 9 minutes just to parse), and this UI enhancement must be prepared to handle those.

Thanks, @ajbozarth. Sure, I'd love to rebase it and get it merged. But I am really busy with my mid-terms and interviews recently. I'll take a look tonight and fix it if it's easy to do.
I really appreciate your concern about my pull requests.

@markhamstra Thanks. Here is my related explanation at the top of this PR.

If the query text is too long (i.e. longer than 1000 chars), it will be truncated. This is to avoid downing browsers or web UI servers by running out of memory.

Is this what you mean?

@nblintao I really haven't reviewed these changes closely enough to have a specific complaint or concern in mind, but I'm more concerned about what happens when you ask to see "more" when that "more" could be several tens of thousands of characters, not just the few thousand that many would consider to be a very long SQL query.

@markhamstra Queries longer than 1000 chars are truncated to 1000 chars after the parsing. "+more" will only expand 140 chars to at most 1000 chars.

@nblintao Got it; thanks.  There may be distinct queries that will be entirely the same within the first 1000 characters, but that's just the nature of working with these very large queries -- there are lots of things that make them difficult, but it sounds like you've made sure not to make things difficult for more normal queries as a consequence, so that's all good.

@ajbozarth I finally have a chance to rebase it in the winter break. Could you please have a look? Thanks!

Test build #70700 has finished for PR 14158 at commit 83cbb58.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70700/
Test FAILed.

retest this please

Test build #70704 has started for PR 14158 at commit 83cbb58.

Test build #70704 has finished for PR 14158 at commit 83cbb58.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Hi @nblintao, is it still active?

@HyukjinKwon Thanks for reminding. I am finals this week. I will fix this after 15th.

Test build #77510 has started for PR 14158 at commit 69180bd.

I have just rebased. @ajbozarth @HyukjinKwon  @gatorsmile @srowen @vanzin

Test build #77510 has finished for PR 14158 at commit 69180bd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77510/
Test PASSed.

Test build #77518 has started for PR 14158 at commit 114401a.

Test build #77518 has finished for PR 14158 at commit 114401a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77518/
Test FAILed.

Test build #77523 has started for PR 14158 at commit ac96aaa.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77523/
Test FAILed.

retest this please

Test build #78291 has started for PR 14158 at commit ac96aaa.

Merged build finished. Test FAILed.

i'll retrigger this once jenkins is back up.

test this please

Test build #78299 has started for PR 14158 at commit ac96aaa.

Test build #78299 has finished for PR 14158 at commit ac96aaa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class QueryExecution(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78299/
Test FAILed.

retest this please

Test build #78332 has started for PR 14158 at commit ac96aaa.

Test build #78332 has finished for PR 14158 at commit ac96aaa.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class QueryExecution(

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78332/
Test FAILed.

Merged build finished. Test FAILed.

gentle ping @nblintao, is this PR active? If so, I guess the test failure should be fixed if related.

@HyukjinKwon It's still active. I'll fix it when I'm available. Thanks.

hey @nblintao, do you maybe happened to have some time to continue this one?

@HyukjinKwon Sorry for the delay. I'm busy looking for jobs these days. I'll try my best to fix it in October. Thank you for reminding me!

I'm going to close this PR because it goes stale, please feel free to reopen it or open another PR if anyone have more thoughts on this issue.

authors: AmplabJenkins HyukjinKwon SparkQA ajbozarth jiangxb1987 markhamstra nblintao shaneknapp


text:
[SPARK-16810] Refactor registerSinks with multiple constructos

What changes were proposed in this pull request?
For some metrics, it may require some app detailed information from SparkConf. So for those sinks, we need to pass SparkConf via sink constructor. Refactored registerSink class reflection part to allow multiple types of sink constructor to be initialized.
How was this patch tested?
Using existing tests.

Can one of the admins verify this patch?

DId you add a test?

@rxin I was trying to add a unit test for this feature. But the test sinks are not visible for MetricsSystem when it tries to register test sink classes since test sinks are hidden in test packages. Is there any other ways to load test classes for MetricsSystem without messed-up metrics package?

Oops. I kind of messed up here. Going to open another PR for this. Close this for now.

authors: AmplabJenkins lovexi rxin


text:
[SPARK-17590][SQL] Analyze CTE definitions at once and allow CTE subquery to define CTE

What changes were proposed in this pull request?
We substitute logical plan with CTE definitions in the analyzer rule CTESubstitution. A CTE definition can be used in the logical plan for multiple times, and its analyzed logical plan should be the same. We should not analyze CTE definitions multiple times when they are reused in the query.
By analyzing CTE definitions before substitution, we can support defining CTE in subquery.
How was this patch tested?
Jenkins tests.

Test build #65586 has started for PR 15146 at commit 394495d.

Test build #65588 has started for PR 15146 at commit 2f8e79f.

cc @hvanhovell @cloud-fan

Test build #65586 has finished for PR 15146 at commit 394495d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65586/
Test PASSed.

Test build #65588 has finished for PR 15146 at commit 2f8e79f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65588/
Test PASSed.

Seems good, but not quite sure, cc @hvanhovell to confirm

@viirya does this improve performance?

@hvanhovell This is for analyzer change and adds CTE in CTE feature. I don't expect there is performance improvement.

I guess if using the same analyzed plan increases the chance to reuse exchange, then it may improve the performance. Anyway, it is not the purpose of this change. Because the analyzed subquery plan will be changed largely in optimization later, we cannot guarantee this.

Test build #65630 has started for PR 15146 at commit baf239b.

Test build #65630 has finished for PR 15146 at commit baf239b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65630/
Test PASSed.

Merging to master. Thanks!

@hvanhovell @cloud-fan Thanks!

authors: AmplabJenkins SparkQA cloud-fan hvanhovell viirya


text:
[SPARK-21732][SQL]Lazily init hive metastore client

What changes were proposed in this pull request?
This PR changes the codes to lazily init hive metastore client so that we can create SparkSession without talking to the hive metastore sever.
It's pretty helpful when you set a hive metastore server but it's down. You can still start the Spark shell to debug.
How was this patch tested?
The new unit test.

cc @yhuai

Test build #80651 has started for PR 18944 at commit 9eb9149.

lgtm

Test build #80651 has finished for PR 18944 at commit 9eb9149.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80651/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #80655 has started for PR 18944 at commit 9eb9149.

Test build #80655 has finished for PR 18944 at commit 9eb9149.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80655/
Test PASSed.

Thanks! Merging to master.

authors: AmplabJenkins SparkQA gatorsmile yhuai zsxwing


text:
[MINOR][SQL]remove unuse import class

What changes were proposed in this pull request?
this PR describe remove the import class that are unused.
How was this patch tested?
N/A

Can one of the admins verify this patch?

On the one hand, we don't normally just make a change to remove unused imports. But it's a good idea once in a while to clean up. Would you actually try to find all of them? at least, in several whole modules rather than part of one.

ok, i will do it.

Jenkins, test this please.

Test build #3913 has started for PR 19131 at commit e7ce4e5.

Test build #3913 has finished for PR 19131 at commit e7ce4e5.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

What about other components, here you only fixed sql and core module.

wait, I'm still looking for it.

Personally I'm not fond of such fix, this will break lots of existing PRs and force them to rebase again. Usually this could be addressed when fixing other issues. IMHO I don't encourage such PRs.

I'm neutral. Once in a while a cleanup seems helpful, not all the time. I am a little more receptive to people without much contribution history. But yeah I do not want to review things like this regularly. We have a lot of really minor PRs coming through

@srowen, @jerryshao several other components, Include spark streaming, launcher,graphx,core,mllib,mllib-local， except for their tests.
thanks.

Jenkins, test this please.

retest this please

ok to test

Test build #81541 has started for PR 19131 at commit 648ed11.

Test build #81541 has finished for PR 19131 at commit 648ed11.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81541/
Test FAILed.

Test build #81553 has started for PR 19131 at commit bdc1bcf.

Test build #81553 has finished for PR 19131 at commit bdc1bcf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81553/
Test FAILed.

Merged build finished. Test FAILed.

Test build #81563 has started for PR 19131 at commit ee0d848.

Test build #81563 has finished for PR 19131 at commit ee0d848.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

A check for unused imports should be added to scalastyle to prevent these from creeping back in.  If this PR was accompanied with that check (failing before, now passing) I think the merge conflicts are worth it to permanently fix the problem.

I didn't think scalastyle could do that? didn't see a rule at http://www.scalastyle.org/rules-dev.html

I'm going to merge it as a cleanup but yeah let's not do these often.
I would favor adding a style check for this if one can be found, but don't see it in scalastyle.

authors: AmplabJenkins HyukjinKwon SparkQA ash211 heary-cao jerryshao srowen


text:
[SPARK-24013][SQL] Remove unneeded compress in ApproximatePercentile

What changes were proposed in this pull request?
ApproximatePercentile contains a workaround logic to compress the samples since at the beginning QuantileSummaries was ignoring the compression threshold. This problem was fixed in SPARK-17439, but the workaround logic was not removed. So we are compressing the samples many more times than needed: this could lead to critical performance degradation.
This can create serious performance issues in queries like:
select approx_percentile(id, array(0.1)) from range(10000000)

How was this patch tested?
added UT

Test build #89742 has started for PR 21133 at commit 0ac3b4f.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2601/
Test PASSed.

Test build #89742 has finished for PR 21133 at commit 0ac3b4f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89742/
Test PASSed.

Test build #89921 has started for PR 21133 at commit 2fa8da7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2715/
Test PASSed.

Test build #89921 has finished for PR 21133 at commit 2fa8da7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89921/
Test PASSed.

cc @cloud-fan

Test build #89966 has started for PR 21133 at commit d47d9bd.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2743/
Test PASSed.

Maybe we could add the former test as a benchmark to AggregateBenchmark?

@juliuszsompolski I am not sure. This is actually not a performance improvement (strictly speaking that would mean changing an algorithm/code block in order to perform better). Here we are just removing a useless statement which has been wrongly there for legacy reasons. Moreover it is also quite hard to get the benchmark data, since I have not been able to see the query finish without the fix...

Above is my major comment https://github.com/apache/spark/pull/21133#discussion_r185030942
cc @juliuszsompolski @cloud-fan Please see whether it makes sense.

LGTM

retest this please

Test build #90060 has started for PR 21133 at commit aab21a7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2815/
Test PASSed.

authors: AmplabJenkins SparkQA cloud-fan gatorsmile juliuszsompolski mgaido91


text:
[WIP][SPARK-23914][SQL][follow-up] refactor ArrayUnion

What changes were proposed in this pull request?
This PR refactors ArrayUnion based on this suggestion.

Generate optimized code for all of the primitive types except boolean
Generate code using ArrayBuilder or ArrayBuffer
Leave only a generic path in the interpreted path

How was this patch tested?
Existing tests

@kiszk, thanks! I am a bot who has found some folks who might be able to help with the review:@ueshin, @rxin and @colorant

Test build #93840 has started for PR 21937 at commit 738a4a0.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1527/
Test PASSed.

Test build #93840 has finished for PR 21937 at commit 738a4a0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class OpenHashSet[@specialized(Long, Int, Double, Float) T: ClassTag](
 sealed class Hasher[@specialized(Long, Int, Double, Float) T] extends Serializable
 class DoubleHasher extends Hasher[Double]
 class FloatHasher extends Hasher[Float]
case class ArrayUnion(left: Expression, right: Expression) extends ArraySetLike

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93840/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1797/
Test PASSed.

Test build #94208 has started for PR 21937 at commit f48dde0.

Test build #94208 has finished for PR 21937 at commit f48dde0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94208/
Test PASSed.

I'll revisit after the conflict is fixed.

I see. Now, I am rebasing and resolving conflicts.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1850/
Test PASSed.

Test build #94290 has started for PR 21937 at commit 93ec9ec.

Test build #94304 has started for PR 21937 at commit 26c1f8a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1863/
Test PASSed.

Test build #94290 has finished for PR 21937 at commit 93ec9ec.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94290/
Test FAILed.

Test build #94304 has finished for PR 21937 at commit 26c1f8a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94304/
Test PASSed.

Merged build finished. Test PASSed.

LGTM.

Thanks! merging to master.

authors: AmplabJenkins SparkQA holdensmagicalunicorn kiszk ueshin


text:
[SPARK-25683][Core] Updated the log for the firstTime event Drop occurs

What changes were proposed in this pull request?
When the first dropEvent occurs, LastReportTimestamp was printing in the log as
Wed Dec 31 16:00:00 PST 1969
(Dropped 1 events from eventLog since Wed Dec 31 16:00:00 PST 1969.)
The reason is that lastReportTimestamp initialized with 0.
Now log is updated to print "... since the application starts" if 'lastReportTimestamp' == 0.
this will happens first dropEvent occurs.
How was this patch tested?
Manually verified.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

cc @HyukjinKwon ,  @vanzin

Though it looks a little strange, the log content is actually right, I don't think we want to make the last report timestamp to current time (that can confuse users what happened before that timestamp).

@jiangxb1987
Thanks for your comment,
i think printing "since Wed Dec 31 16:00:00 PST 1969" still looks strange,
Instead we can print "since start of the application for first time event Dropping, this looks more appropriate.
so first time log should look like this
18/10/08 17:51:40 WARN AsyncEventQueue: Dropped 1 events since start of the application
Instead of
18/10/08 17:51:40 WARN AsyncEventQueue: Dropped 1 events from eventLog since Wed Dec 31 16:00:00 PST 1969.
please correct me if i am wrong.

@jiangxb1987 @dongjoon-hyun
I have modified accordingly.
Tested locally with existing UT (to make drop the event, locally "LISTENER_BUS_EVENT_QUEUE_CAPACITY" is set to 1), using println.
Tested as in below snap.

ok to test

The PR title is not accurate anymore.

Test build #97412 has started for PR 22677 at commit d71c74f.

Test build #97412 has finished for PR 22677 at commit d71c74f.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97412/
Test FAILed.

Test build #97435 has started for PR 22677 at commit 4b36af3.

Test build #97437 has started for PR 22677 at commit 7105cce.

Test build #97437 has finished for PR 22677 at commit 7105cce.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #97435 has finished for PR 22677 at commit 4b36af3.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97437/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97435/
Test FAILed.

Test build #97446 has started for PR 22677 at commit 3e3ad24.

Test build #97446 has finished for PR 22677 at commit 3e3ad24.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97446/
Test PASSed.

Hi,
I have modified the title and description according to code change. Build also passed.
Kindly check and merge.

cc @vanzin

Retest this please.

Test build #97516 has started for PR 22677 at commit 3e3ad24.

@shivusondur . It seems that your PR almost finished. Please update JIRA title and description together. https://issues.apache.org/jira/browse/SPARK-25683

Test build #97516 has finished for PR 22677 at commit 3e3ad24.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97516/
Test FAILed.

Test build #97518 has started for PR 22677 at commit b1f2e48.

Test build #97518 has finished for PR 22677 at commit b1f2e48.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97518/
Test FAILed.

Test build #97532 has started for PR 22677 at commit df21621.

Test build #97532 has finished for PR 22677 at commit df21621.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97532/
Test PASSed.

Test build #97553 has started for PR 22677 at commit 0f2b2d1.

Merging to master.

Test build #97553 has finished for PR 22677 at commit 0f2b2d1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97553/
Test PASSed.

authors: AmplabJenkins SparkQA dongjoon-hyun jiangxb1987 shivusondur vanzin


text:
[SPARK-28690][SQL] Add the date_part function for timestamps and dates

What changes were proposed in this pull request?
In the PR, I propose new function date_part(). The function is modeled on the traditional Ingres equivalent to the SQL-standard function extract:
date_part('field', source)

and added for feature parity with PostgreSQL (https://www.postgresql.org/docs/11/functions-datetime.html#FUNCTIONS-DATETIME-EXTRACT).
The source can have DATE or TIMESTAMP type. Supported string values of 'field' are:

millennium - the current millennium for given date (or a timestamp implicitly casted to a date). For example, years in the 1900s are in the second millennium. The third millennium started January 1, 2001.
century - the current millennium for given date (or timestamp). The first century starts at 0001-01-01 AD.
decade - the current decade for given date (or timestamp). Actually, this is the year field divided by 10.
year, month, day, hour, minute, second
week - the number of the ISO 8601 week-numbering week of the year. By definition, ISO weeks start on Mondays and the first week of a year contains January 4 of that year.
quarter - the quarter of the year (1 - 4)
dayofweek - the day of the week for date/timestamp (1 = Sunday, 2 = Monday, ..., 7 = Saturday)
dow - the day of the week as Sunday (0) to Saturday (6)
isodow - the day of the week as Monday (1) to Sunday (7)
doy - the day of the year (1 - 365/366)

Here are examples:
spark-sql> select date_part('year', timestamp'2019-08-12 01:00:00.123456');
2019
spark-sql> select date_part('week', timestamp'2019-08-12 01:00:00.123456');
33
spark-sql> select date_part('doy', timestamp'2019-08-12 01:00:00.123456');
224

I changed implementation of extract to re-use date_part() internally.
How was this patch tested?
Added date_part.sql and regenerated results of extract.sql.

Can one of the admins verify this patch?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14012/
Test PASSed.

Test build #108938 has started for PR 25410 at commit af51e52.

Test build #108938 has finished for PR 25410 at commit af51e52.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108938/
Test FAILed.

Test build #108953 has started for PR 25410 at commit e68611a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14024/
Test PASSed.

Test build #108953 has finished for PR 25410 at commit e68611a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108953/
Test FAILed.

jenkins, retest this, please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14028/
Test PASSed.

Test build #108957 has started for PR 25410 at commit e68611a.

Test build #108957 has finished for PR 25410 at commit e68611a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108957/
Test PASSed.

Could you rebase this to the master please, @MaxGekk ?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14175/
Test PASSed.

Test build #109110 has started for PR 25410 at commit efc3ee0.

Test build #109110 has finished for PR 25410 at commit efc3ee0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109110/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14185/
Test PASSed.

Test build #109119 has started for PR 25410 at commit bcf73d2.

Test build #109119 has finished for PR 25410 at commit bcf73d2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109119/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14214/
Test PASSed.

Test build #109145 has started for PR 25410 at commit 1b2c8d4.

Test build #109145 has finished for PR 25410 at commit 1b2c8d4.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109145/
Test FAILed.

jenkins, retest this, please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14219/
Test PASSed.

Test build #109150 has started for PR 25410 at commit 1b2c8d4.

Test build #109150 has finished for PR 25410 at commit 1b2c8d4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109150/
Test PASSed.

Test build #109186 has started for PR 25410 at commit 2fa25b1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14257/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14260/
Test PASSed.

Test build #109189 has started for PR 25410 at commit 494679c.

Test build #109191 has started for PR 25410 at commit 5c8c34d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14262/
Test PASSed.

Test build #109189 has finished for PR 25410 at commit 494679c.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #109186 has finished for PR 25410 at commit 2fa25b1.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #109191 has finished for PR 25410 at commit 5c8c34d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109191/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109189/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109186/
Test FAILed.

Test build #109278 has started for PR 25410 at commit 05b3746.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14347/
Test PASSed.

Test build #109278 has finished for PR 25410 at commit 05b3746.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109278/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14620/
Test PASSed.

@dongjoon-hyun @maropu When I set spark.sql.parser.ansi.enabled to false there: https://github.com/apache/spark/blob/97dc4c0bfc3a15d364a376c6f87cb921d8d6980d/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala#L302
it accepts year without backquotes as well:
-- !query 2
SELECT date_part( 'year', d1) AS year FROM TIMESTAMP_TBL WHERE d1 BETWEEN '1902-01-01' AND '2038-01-01'
-- !query 2 schema
struct<year:int>
-- !query 2 output
1997

How about turning off the mode temporarily like the others? https://github.com/apache/spark/blob/33e45ec7b80934a53f882c309ef8f37762dc0ab8/sql/core/src/test/resources/sql-tests/inputs/pgSQL/text.sql#L49

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14667/
Test PASSed.

Test build #109614 has started for PR 25410 at commit 74a7fec.

Test build #109614 has finished for PR 25410 at commit 74a7fec.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109614/
Test FAILed.

retest this please

Test build #109625 has started for PR 25410 at commit 74a7fec.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14677/
Test PASSed.

Test build #109625 has finished for PR 25410 at commit 74a7fec.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109625/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14704/
Test PASSed.

Test build #109654 has started for PR 25410 at commit ade541d.

Test build #109654 has finished for PR 25410 at commit ade541d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109654/
Test PASSed.

@maropu @dongjoon-hyun Could you take a look at the PR one more time.

Sure, @MaxGekk .

LGTM except for https://github.com/apache/spark/pull/25410#discussion_r316817684
I leave it to the other reviewers.

LGTM except for #25410 (comment)

@maropu Are you not sure about your proposed changes https://github.com/apache/spark/commit/d793f4958f3a9adc69f7a1e04f7fc7a994d5e15a ? Just in case, I cherry picked them https://github.com/apache/spark/pull/25410/commits/bc707dfac3aaccce95a5db2258724469d5ede409 3 days ago.

LGTM except for #25410 (comment)

@maropu Are you not sure about your proposed changes https://github.com/apache/spark/commit/d793f4958f3a9adc69f7a1e04f7fc7a994d5e15a ? Just in case, I cherry picked them https://github.com/apache/spark/pull/25410/commits/bc707dfac3aaccce95a5db2258724469d5ede409 3 days ago.

@MaxGekk Sorry to confuse you! But, I just meant that the other reviewers need to check that part once more. I think the current one is ok though.

@dongjoon-hyun nvm! I think the release has the higher priority.

@dongjoon-hyun Please, take a look at the PR when you have time.

@cloud-fan @HyukjinKwon @srowen Could you take a look at the PR, please.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15141/
Test PASSed.

Test build #110132 has started for PR 25410 at commit b292931.

Test build #110132 has finished for PR 25410 at commit b292931.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
abstract class FileCommitProtocol extends Logging
trait ResourceAllocator
class RpcAbortException(message: String) extends Exception(message)
public final class JavaStructuredKerberizedKafkaWordCount
public final class JavaDirectKerberizedKafkaWordCount
class BindingParquetOutputCommitter(
class PathOutputCommitProtocol(
class DecisionTreeParams(Params):
case class UnresolvedTable(v1Table: CatalogTable) extends Table
 implicit class IdentifierHelper(identifier: TableIdentifier)
class CatalogManager(conf: SQLConf) extends Logging
case class ShowTables(
trait AlterTableStatement extends ParsedStatement
case class ShowTablesStatement(namespace: Option[Seq[String]], pattern: Option[String])
case class ReuseAdaptiveSubquery(
trait CostEvaluator
case class SimpleCost(value: Long) extends Cost
case class DescribeTableExec(
case class ShowTablesExec(
case class AppendDataExecV1(
case class OverwriteByExpressionExecV1(
sealed trait V1FallbackWriters extends SupportsV1Write
 protected implicit class toV1WriteBuilder(builder: WriteBuilder)
trait SupportsV1Write extends SparkPlan
class V2SessionCatalog(sessionState: SessionState) extends TableCatalog with SupportsNamespaces
trait V1WriteBuilder extends WriteBuilder

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110132/
Test PASSed.

Test build #110225 has started for PR 25410 at commit 600eee6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15225/
Test PASSed.

Test build #110225 has finished for PR 25410 at commit 600eee6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110225/
Test PASSed.

LGTM.
@maropu do you know why Spark treats year as reserved keyword? I thought we followed pgsql at that time...

we just follow the SQL-2011 standard and it reserves year. So, we follow it. In fact, I don't know why pgSQL doesn't reserve year...

Thanks! Merged to master.

@maropu @dongjoon-hyun @cloud-fan Thank you for your review.

authors: AmplabJenkins MaxGekk SparkQA cloud-fan dongjoon-hyun maropu


text:
[SPARK-30594][CORE] Do not post SparkListenerBlockUpdated when updateBlockInfo returns false

What changes were proposed in this pull request?

If updateBlockInfo returns false, which means the BlockManager will re-register and report all blocks later. So, we may report two times for the same block, which causes AppStatusListener to count used memory for two times, too. As a result, the used memory can exceed the total memory.
So, this PR changes it to not post SparkListenerBlockUpdated when updateBlockInfo returns false. And, always clean up used memory whenever AppStatusListener receives SparkListenerBlockManagerAdded.
Why are the changes needed?

This PR tries to fix negative memory usage in UI (https://user-images.githubusercontent.com/3488126/72131225-95e37e00-33b6-11ea-8708-6e5ed328d1ca.png, see #27144 ). Though, I'm not very sure this is the root cause for #27144 since known information is limited here.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Honestly, I have no good idea to write a unit test for this issue yet.

@xuanyuanking Can you cherry-pick your commits to this branch? They actually looks good.

During the cherry-pick, please don't forget to fix the invalid UT, @xuanyuanking .

@dongjoon-hyun The UT has been fixed by https://github.com/apache/spark/pull/27418/commits/fecaf20a4b8c88596f6fd60e5033b9b4e328b623

Oh. I missed that. Thanks, @jiangxb1987 ! If that is merged here, I'll test once more.

@jiangxb1987 Thanks Xingbo, the cherry-picks were done in https://github.com/Ngone51/spark/pull/1, waiting for @Ngone51 merging.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117759/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22521/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117760/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22522/
Test PASSed.

Jenkins, retest this please.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117761/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22523/
Test PASSed.

@dongjoon-hyun Would you please help check Jenkins failure?

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22525/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117765/
Test FAILed.

retest this please

Test build #117773 has started for PR 27306 at commit 5f12317.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22535/
Test PASSed.

Test build #117773 has finished for PR 27306 at commit 5f12317.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117773/
Test PASSed.

thanks, merging to master/3.0!

thanks all, especially for @xuanyuanking !

authors: AmplabJenkins Ngone51 SparkQA cloud-fan dongjoon-hyun jiangxb1987 xuanyuanking


text:
[MINOR][DOCS] Fix a typo for a configuration property of resources allocation

What changes were proposed in this pull request?

This PR fixes a typo for a configuration property in the spark-standalone.md.
spark.driver.resourcesfile should be spark.driver.resourcesFile.
I look for similar typo but this is the only typo.
Why are the changes needed?

The property name is wrong.
Does this PR introduce any user-facing change?

Yes. The property name is corrected.
How was this patch tested?

I confirmed the spell of the property name is the correct from the property name defined in o.a.s.internal.config.package.scala.

Test build #124644 has started for PR 28958 at commit d7d6d25.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29261/
Test PASSed.

Test build #124644 has finished for PR 28958 at commit d7d6d25.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124644/
Test PASSed.

authors: AmplabJenkins SparkQA sarutak


text:
[SPARK-32729][SQL][DOCS] Add missing since version for math functions

What changes were proposed in this pull request?
Add missing since version for math functions, including
SPARK-8223 shiftright/shiftleft
SPARK-8215 pi
SPARK-8212 e
SPARK-6829 sin/asin/sinh/cos/acos/cosh/tan/atan/tanh/ceil/floor/rint/cbrt/signum/isignum/Fsignum/Lsignum/degrees/radians/log/log10/log1p/exp/expm1/pow/hypot/atan2
SPARK-8209 conv
SPARK-8213 factorial
SPARK-20751 cot
SPARK-2813 sqrt
SPARK-8227 unhex
SPARK-8218 log(a,b)
SPARK-8207 bin
SPARK-8214 hex
SPARK-8206 round
SPARK-14614 bround
Why are the changes needed?

fix SQL docs
Does this PR introduce any user-facing change?

yes, doc updated
How was this patch tested?

passing doc generation.

cc @cloud-fan @maropu @HyukjinKwon @dongjoon-hyun, thanks a bunch~

Test build #127994 has started for PR 29571 at commit 4700ae9.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32620/
Test PASSed.

Thanks for the work, @yaooqinn ! I think its better to file a jira for the update.

Nice, thanks @yaooqinn for working on this. +1 for a JIRA.

thanks, @maropu @HyukjinKwon, I have filed a Jira ticket SPARK-32729

Merged to master.

Test build #127994 has finished for PR 29571 at commit 4700ae9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127994/
Test PASSed.

authors: AmplabJenkins HyukjinKwon SparkQA maropu yaooqinn


text:
[SPARK-34223][SQL] FIX NPE for static partition with null in InsertIntoHadoopFsRelationCommand

What changes were proposed in this pull request?

with a simple case, the null will be passed to InsertIntoHadoopFsRelationCommand blindly, we should avoid the npe
 test("NPE") {
    withTable("t") {
      sql(s"CREATE TABLE t(i STRING, c string) USING $format PARTITIONED BY (c)")
      sql("INSERT OVERWRITE t PARTITION (c=null) VALUES ('1')")
      checkAnswer(spark.table("t"), Row("1", null))
    }
  }

java.lang.NullPointerException
	at scala.collection.immutable.StringOps$.length(StringOps.scala:51)
	at scala.collection.immutable.StringOps.length(StringOps.scala:51)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:35)
	at scala.collection.IndexedSeqOptimized.foreach
	at scala.collection.immutable.StringOps.foreach(StringOps.scala:33)
	at org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils$.escapePathName(ExternalCatalogUtils.scala:69)
	at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.orig-s0.0000030000-r30676-expand-or-complete(InsertIntoHadoopFsRelationCommand.scala:231)

Why are the changes needed?

a bug fix
Does this PR introduce any user-facing change?

no
How was this patch tested?

new tests

cc @maropu @cloud-fan @dongjoon-hyun @HyukjinKwon, thanks for reviewing

cc @maropu @cloud-fan @dongjoon-hyun @HyukjinKwon, thanks for reviewing

Test build #134432 has started for PR 31320 at commit 211dbe6.

Test build #134432 has started for PR 31320 at commit 211dbe6.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39019/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39019/

Test build #134432 has finished for PR 31320 at commit 211dbe6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134432 has finished for PR 31320 at commit 211dbe6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134432/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134432/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39019/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39019/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39019/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39019/

Test build #134442 has started for PR 31320 at commit c4b18e8.

Test build #134442 has started for PR 31320 at commit c4b18e8.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39028/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39028/

Test build #134450 has started for PR 31320 at commit 452ff11.

Test build #134450 has started for PR 31320 at commit 452ff11.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39028/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39028/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39036/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39036/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134442/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134442/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39028/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39028/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39036/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39036/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39036/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39036/

Test build #134450 has finished for PR 31320 at commit 452ff11.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134450 has finished for PR 31320 at commit 452ff11.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134450/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134450/

Test build #134455 has started for PR 31320 at commit 2d172ab.

Test build #134455 has started for PR 31320 at commit 2d172ab.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39041/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39041/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39041/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39041/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39041/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39041/

Test build #134455 has finished for PR 31320 at commit 2d172ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134455 has finished for PR 31320 at commit 2d172ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134455/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134455/

thanks, merging to master/3.1/3.0!

thanks, merging to master/3.1/3.0!

authors: AmplabJenkins SparkQA cloud-fan yaooqinn