text:
Unable to delete topic when both consumer (subscription) and reader has been connected

Describe the bug
After connecting and closing one subscription and one reader it's not possible to delete a topic through the java admin client. The bug will not manifest if either only subscriptions or only readers are connected, only if a combination of them are.
To Reproduce
Steps to reproduce the behavior:

Start a local standalone pulsar and make sure the admin and broker ports in the test code below match
Run the following java code to reproduce the error:

package sample;

import org.apache.pulsar.client.admin.PulsarAdmin;
import org.apache.pulsar.client.admin.PulsarAdminException;
import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.MessageId;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.Reader;
import org.apache.pulsar.client.api.SubscriptionInitialPosition;
import org.apache.pulsar.client.api.SubscriptionType;
import org.apache.pulsar.client.impl.auth.AuthenticationDisabled;
import org.apache.pulsar.common.policies.data.TenantInfo;
import org.testng.annotations.Test;

import java.util.Collections;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Set;
import java.util.concurrent.TimeUnit;

public class ConsumerAndReaderCombinationTest {

    void deleteTenantIfPresent(PulsarAdmin admin, String tenant) throws PulsarAdminException {
        if (admin.tenants().getTenants().contains(tenant)) {
            for (String namespace : admin.namespaces().getNamespaces(tenant)) {
                List<String> existingTopics = admin.namespaces().getTopics(namespace);
                for (String topic : existingTopics) {
                    admin.topics().delete(topic);
                }
                admin.namespaces().deleteNamespace(namespace);
            }
            admin.tenants().deleteTenant(tenant);
        }
    }

    @Test
    public void provokeBug() throws Exception {
        final String adminUrl = "http://localhost:8080";
        final String brokerUrl = "pulsar://localhost:6650";

        final String tenant = "delete-topic-test-magic48295";
        final String namespace = "ns";
        final String topic = "the-topic";
        final String qualifiedTopic = "persistent://" + tenant + "/" + namespace + "/" + topic;

        /*
         * Ensure a clean tenant
         */
        try (PulsarAdmin admin = PulsarAdmin.builder().serviceHttpUrl(adminUrl).authentication(new AuthenticationDisabled()).build()) {
            deleteTenantIfPresent(admin, tenant);
            Set<String> allowedClusters = new LinkedHashSet<>();
            allowedClusters.add("standalone");
            admin.tenants().createTenant(tenant, new TenantInfo(Collections.emptySet(), allowedClusters));
            admin.namespaces().createNamespace(tenant + "/" + namespace);
        }

        /*
         * Open and close both a consumer on a subscription and a reader on the same topic
         */
        try (PulsarClient client = PulsarClient.builder().serviceUrl(brokerUrl).build()) {
            try (Consumer<byte[]> consumer = client.newConsumer()
                    .topic(qualifiedTopic)
                    .subscriptionType(SubscriptionType.Exclusive)
                    .consumerName("testng")
                    .subscriptionName("s1")
                    .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
                    .subscribe()) {
                // no need to do anything with the consumer, just close it again
            }
            try (Reader<byte[]> reader = client.newReader().topic(qualifiedTopic).startMessageId(MessageId.earliest).create()) {
                // no need to do anything with the reader, just close it again
            }
        }

        try (PulsarAdmin admin = PulsarAdmin.builder()
                .serviceHttpUrl(adminUrl)
                .readTimeout(3, TimeUnit.SECONDS)
                .authentication(new AuthenticationDisabled())
                .build()) {

            /*
             * When bug is present, this call will throw a TimeoutException (read-timeout) after 3 seconds
             */
            admin.topics().delete(qualifiedTopic);
        }
    }
}

Observe lines similiar to the following in the broker log.

16:48:45.579 [pulsar-web-57-3] INFO  org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://delete-topic-test-magic48295/ns/the-topic][s1] Unsubscribing
16:48:45.579 [pulsar-web-57-3] INFO  org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://delete-topic-test-magic48295/ns/the-topic][s1] Successfully closed subscription [ManagedCursorImpl{ledger=delete-topic-test-magic48295/ns/persistent/the-topic, name=s1, ackPos=10:-1, readPos=10:0}]
16:48:45.583 [pulsar-web-57-3] INFO  org.apache.bookkeeper.mledger.impl.MetaStoreImplZookeeper - [delete-topic-test-magic48295/ns/persistent/the-topic] Remove consumer=s1
16:48:45.592 [ProcessThread(sid:0 cport:2181):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x10000f989b00009 type:delete cxid:0xe3 zxid:0xba txntype:-1 reqpath:n/a Error Path:/ledgers/00/0000 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0000
16:48:45.591 [bookkeeper-ml-workers-OrderedExecutor-2-0] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught
java.lang.IllegalArgumentException: inconsistent range
	at java.util.concurrent.ConcurrentSkipListMap$SubMap.<init>(ConcurrentSkipListMap.java:2620) ~[?:1.8.0_212]
	at java.util.concurrent.ConcurrentSkipListMap.subMap(ConcurrentSkipListMap.java:2078) ~[?:1.8.0_212]
	at org.apache.bookkeeper.mledger.util.RangeCache.removeRange(RangeCache.java:132) ~[org.apache.pulsar-managed-ledger-original-2.4.0.jar:2.4.0]
	at org.apache.bookkeeper.mledger.impl.EntryCacheImpl.invalidateEntries(EntryCacheImpl.java:151) ~[org.apache.pulsar-managed-ledger-original-2.4.0.jar:2.4.0]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$7.operationComplete(ManagedLedgerImpl.java:776) ~[org.apache.pulsar-managed-ledger-original-2.4.0.jar:2.4.0]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$7.operationComplete(ManagedLedgerImpl.java:764) ~[org.apache.pulsar-managed-ledger-original-2.4.0.jar:2.4.0]
	at org.apache.bookkeeper.mledger.impl.MetaStoreImplZookeeper.lambda$null$13(MetaStoreImplZookeeper.java:308) ~[org.apache.pulsar-managed-ledger-original-2.4.0.jar:2.4.0]
	at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) ~[org.apache.pulsar-managed-ledger-original-2.4.0.jar:2.4.0]
	at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
16:48:45.610 [BookKeeperClientWorker-OrderedExecutor-5-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [delete-topic-test-magic48295/ns/persistent/the-topic][s1] Deleted cursor ledger 11
16:48:47.557 [pulsar-web-57-11] INFO  org.eclipse.jetty.server.RequestLog - 172.17.0.2 - - [12/Aug/2019:16:48:47 +0000] "GET /admin/v2/persistent/public/functions/coordinate/stats HTTP/1.1" 200 813 "-" "Pulsar-Java-v2.4.0" 5

Expected behavior
it's always expected to be possible to delete a topic that has no connected clients.
Desktop (please complete the following information):

OS: macOS

may be related - cannot unsubscribe (delete a subscription) via the admin tool:

I can confirm this is an issue:  once a Reader as been seen on a topic, it is not possible to delete any Consumer subscription.
This is a massive issue because it means that we basically cannot use Readers and Consumers combined on a topic, because we will never be able to clean that topic and it will be amassing data forever (until quotas are hit).
Here's minimal replication case:

subscribe with a Consumer, then stop the Consumer
delete the subscription with admin tool -- this will work
bin/pulsar-admin topics unsubscribe public/test-seqid/topic-3 -s failover-subscription
subscribe with a Reader, then stop the Reader
subscribe with a Consumer, then stop the Consumer
delete the subscription with admin tool -- this will NOT work (timeout)
bin/pulsar-admin topics unsubscribe public/test-seqid/topic-3 -s failover-subscription

Client code:
    pulsar =
        PulsarClient.builder()
            .serviceUrl(pulsarUrl)
            .connectionsPerBroker(1) // default: 1
            .connectionTimeout(10, TimeUnit.SECONDS)
            .enableTcpNoDelay(true)
            .keepAliveInterval(999, TimeUnit.DAYS)
            .maxBackoffInterval(5, TimeUnit.SECONDS)
            .startingBackoffInterval(1, TimeUnit.SECONDS)
            .statsInterval(60, TimeUnit.SECONDS)
            .ioThreads(1) // default: 1
            .listenerThreads(1) // default: 1
            .maxConcurrentLookupRequests(5000) // default: 5000
            .maxLookupRequests(50000) // default: 50000
            .maxNumberOfRejectedRequestPerConnection(50) // default: 50
            .operationTimeout(30, TimeUnit.SECONDS) // default: 30 sec
            .build();

Consumer code:
        Consumer<Test> consumer = pulsar
            .newConsumer(Schema.PROTOBUF(Test.class))
            .consumerName("consumer1")
            .subscriptionInitialPosition(SubscriptionInitialPosition.Latest)
            .subscriptionType(SubscriptionType.Failover)
            .subscriptionName("failover-subscription")
            .topic("persistent://public/test-seqid/topic-3")
            .subscribe();

    while (true) {
      Message<Test> message = consumer.receive();
      ...

Reader code:
        Reader<Test> reader = pulsar
            .newReader(Schema.PROTOBUF(Test.class))
            .readerName("reader1")
            .topic("persistent://public/test-seqid/topic-3")
            .startMessageId(MessageId.earliest)
            .create();

    while (true) {
      Message<Test> message = reader.readNext();
      ...

(Another separate(?) issue is why the Reader is seeing any previously published data, when retentions on the namespace are zero (both time and size), and all messages were acknowledged. This is confusing.)

@sijie - I think this is not a component/client issue (since the subscription cannot be deleted through the admin cli either)

The problem (inconsistent range) has been fixed as part of fixing #5621.

why the Reader is seeing any previously published data, when retentions on the namespace are zero (both time and size), and all messages were acknowledged. This is confusing.

@youurayy the data is deleted segment by segment. unless the whole segment is deleted, the reader will see the messages of the segment that is not deleted.

@sijie - please is it known when should 2.4.2 proper be out (image pushed)?  There's no new commits on the milestone for last 7 days.

@youurayy the 2.4.2 is under voting. once it passes voting, it will be released. I would expect it is going out around Friday. You can follow the process by subscribing to dev@pulsar.apache.org mailing list, or you can read community weekly updates (https://streamnative.io/weekly/2019/2019-11/2019-11-22-pulsar-weekly/) as well.

authors: kimcs sijie youurayy


text:
Fix: NPE if messageId not present on create-sub admin-api call

Motivation
Right now, while creating subscription if user doesn't pass message-id then broker doesn't handle it properly which gives NPE and gives internal-server error to user.
Message: null

Stacktrace:

java.lang.NullPointerException
        at org.apache.pulsar.broker.admin. PersistentTopicsBase.internalCreateSubscription(PersistentTopicsBase.java:885)
        at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)

Modifications

take earliest-message id while creating subscription if user doesn't provide message-id.

retest this please

run cpp tests

source: gharchive/pull-request

authors: rdhabalia sijie

repo: apache/pulsar

license_type: permissive

license_source: github-api


text:
Added schema information and unittests for primitive schema types

Motivation
Currently primitive schema types(like Long/Double, etc.) don;'t have any schema information attached to them. This pr adds this schema information. We also add the missing unittests
Modifications
Describe the modifications you've done.
Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API: (yes / no)
The schema: (yes / no / don't know)
The default values of configurations: (yes / no)
The wire protocol: (yes / no)
The rest endpoints: (yes / no)
The admin cli options: (yes / no)
Anything that affects deployment: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

@sijie @jerrypeng Based on @sijie feedback and our discussion, I've removed the schema changes and only kept the unittest additions. Please take a look again. Thanks!

run cpp tests
run java8 tests


text:
[Issue 4956]Add default loader for latest pyyaml

Fixes https://github.com/apache/pulsar/issues/4956
Master Issue: https://github.com/apache/pulsar/issues/4956
Motivation
In the latest version of pyyaml, using load will issue a warning and explain some reasons here https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation.
Modifications

Use yaml.FullLoader as default loader

run cpp tests

run cpp tests

run cpp tests

run cpp tests

run cpp tests

authors: sijie tuteng


text:
Fix deduplication cursor does not delete after disabling message deduplication

Motivation
Fix deduplication cursor does not delete after disabling message deduplication. The issue occurs when enabling the message deduplication at the broker.conf and then disable it and restart the broker. The dedup cursor will not be deleted.
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: ( no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation

Does this pull request introduce a new feature? (no)

related to #7345

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

org.apache.pulsar.broker.loadbalance.AntiAffinityNamespaceGroupTest
[ERROR] testBrokerSelectionForAntiAffinityGroup(org.apache.pulsar.broker.loadbalance.AntiAffinityNamespaceGroupTest)  Time elapsed: 0.151 s  <<< FAILURE!
java.lang.AssertionError: null
	at org.testng.Assert.fail(Assert.java:96)
	at org.testng.Assert.assertNotEquals(Assert.java:1157)
	at org.testng.Assert.assertNotEquals(Assert.java:1162)
	at org.apache.pulsar.broker.loadbalance.AntiAffinityNamespaceGroupTest.testBrokerSelectionForAntiAffinityGroup(AntiAffinityNamespaceGroupTest.java:423)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[ERROR] testAckCommand(org.apache.pulsar.broker.service.ServerCnxTest)  Time elapsed: 10.148 s  <<< FAILURE!
java.io.IOException: Failed to get response from socket within 10s
	at org.apache.pulsar.broker.service.ServerCnxTest.getResponse(ServerCnxTest.java:1459)
	at org.apache.pulsar.broker.service.ServerCnxTest.testAckCommand(ServerCnxTest.java:1250)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

/pulsarbot run-failure-checks

authors: codelipenghui wolfstudy


text:
message_magic_code  and blank_magic_code is someone 's phone number

i have found the in the code the param
org.apache.rocketmq.common.message.MessageDecoder#MESSAGE_MAGIC_CODE
org.apache.rocketmq.store.CommitLog#MESSAGE_MAGIC_CODE
and org.apache.rocketmq.store.CommitLog#BLANK_MAGIC_CODE is someone 's code,so we have to protect privacy,but we must Compatibility of previous programs,so calculated  the value replace　it.

this pull request is https://github.com/apache/rocketmq/pull/376

source: gharchive/issue

repo: apache/rocketmq


text:
Introduce Bazel Steward - a tool for keeping dependencies up to date in Bazel

Which Issue(s) This PR Fixes
I am happy to create a Github issue if you find Bazel Steward useful.
Brief Description
Hello everyone,
As one of developers I am submitting this pull request to introduce our new tool for Bazel repositories. Bazel Steward simplifies the process of checking and updating dependencies in your Bazel projects, making it more efficient to keep them up-to-date.
We hope that this tool will be useful to the Bazel community, and we look forward to your feedback.
This pull request integrates Bazel Steward through Github Actions which is currently the easiest way to do this. It will run every day at 12 and create new PRs or resolve conflicts on existing if necessary. You can preview how it looks in a fork that I used to test it. For more details, you can check the project readme.
Bazel Steward is able to correctly update all your maven dependencies, version of Bazel itself as well as Bazel rules.
We hope that Bazel Steward will make it easier for you to manage dependencies. If you have any questions, encounter any issues or have any feedback, please don't hesitate to reach out to me. Thank you!
Changes:

Added GitHub Action Workflow for Bazel Steward
Made rules_jvm_external url explicit - so that Bazel Steward can replace it reliably (as url scheme sometimes changes)
Added config for Bazel Steward to limit concurrently open PRs to 10 to make it more manageable as I think there are 45 outdated dependencies here.

How Did You Test This Change?
I run Bazel Steward on a fork that I mentioned earlier. I had to allow Github Actions for write access and managing PRs in repository settings. Additionally, in order to be able to have workflows triggered on Bazel Steward PRs, you'd need to follow this guide: https://github.com/VirtusLab/bazel-steward#triggering-workflows-for-pull-requests

Codecov Report

Merging #6695 (e1484f7) into develop (57642bc) will decrease coverage by 0.03%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             develop    #6695      +/-   ##
=============================================
- Coverage      43.02%   42.99%   -0.03%
+ Complexity      9023     9017       -6
=============================================
  Files           1109     1109
  Lines          78631    78630       -1
  Branches       10234    10233       -1
=============================================
- Hits           33830    33808      -22
- Misses         40568    40587      +19
- Partials        4233     4235       +2

see 20 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

I am afraid your action VirtusLab/bazel-steward  may not be introduced in RocketMQ due to the ASF limit. But I think Bazel Steward is a helpful tool, can you open a discussion before submitting the pr?

@ShadowySpirits Thanks for response! I opened a discussion here: https://github.com/apache/rocketmq/discussions/6711
Also, could you elaborate a bit on this ASF limit? Maybe something can be done about it?


text:
[RIP-9]update Design_Query.md in RocketMQ/docs/cn

update Design_Query.md

Hi @lyeeeee ,please rebase the codes in you local enviroment.

@lyeeeee It seems that this PR brought some other contributor's work, so cloud you rebase this PR firstly, and as this PR has been no longer active for several months，so I will close this PR temporarily, but please feel free to reopen it.

authors: duhenglucky lyeeeee zongtanghu


text:
skywalking webapp 页面不显示

Search before asking

[X] I had searched in the issues and found no similar issues.

Apache SkyWalking Component
UI (apache/skywalking-booster-ui)
What happened
使用阿里云服务器部署Skywlaking 9.5.0，jdk版本17，oapserver和webapp启动看日志正常启动，但是访问页面无法访问，排除防火墙和端口问题。
浏览器consloe显示static/js/monaco-editor-97fd89d3.js net::ERR_CONTENT_LENGTH_MISMATCH 200 (OK)
What you expected to happen
正常访问页面
How to reproduce
使用Skywalking 9.5.0部署
Anything else
No response
Are you willing to submit a pull request to fix on your own?

[X] Yes I am willing to submit a pull request on my own!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

If you are keeping doing this, we have to block you.
Please follow community requirements.
If you need Chinese channel, we have opened slack channel for that.

repo: apache/skywalking

license_source: bigquery


text:
Fix SQL OTEL document path error.

[ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
[ ] Update the CHANGES log.

I think opentelemetry-receiver.md should be updated as well.

authors: wu-sheng yswdqz


text:
SLING-7140 Support redirects to URLs provided by the underlying datastore.

This pull request contains API and pom changes which is why I am not committing it directly myself.
It should be squashed into a Single commit.
It depends on https://github.com/apache/jackrabbit-oak/pull/70 being merged and OAK 1.6.5 (or 6) being released. It will break a build if Oak 1.6.5-SNAPSHOT does not contain that pull request.
It enables Sling to redirect requests for binaries held in Oak in a S3 DataStore to be served by an appropriately configured CloudFront instance.
For more info and documentation see https://issues.apache.org/jira/browse/SLING-7140 and https://issues.apache.org/jira/browse/OAK-6575

This pull request should not be merged at this time. An alternative patch is being developed that does not depend on Oak.

repo: apache/sling


text:
SOLR-16825: Migrate v2 definitions to 'api' module, pt 6

https://issues.apache.org/jira/browse/SOLR-16825
Description
SOLR-16825 added a new gradle module, 'api', which holds v2 API definitions as interfaces. This allows us to generate an OAS (and SolrRequest implementations from that) as a part of the solrj build.
But these artifacts (the OAS and generated Java code), only cover the v2 APIs that have interfaces in the 'api' module. We need to extract interfaces to live in 'api' for each v2 API in 'core' that doesn't already have one.
Solution
This PR creates 'api' interfaces for a number of v2 APIs, allowing SolrRequest implementations to be generated for them. The following APIs are covered in this PR:

get (entire) schema
get schema version
get schema name
get schema similarity
get schema uniquekey

Tests
PR is a refactor, so doesn't add any additional tests. But manual testing has been done to make sure the affected v2 APIs continue to work, and existing tests continue to pass.
Checklist
Please review the following and check all that apply:

[x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
[x] I have created a Jira issue and added the issue ID to my pull request title.
[x] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
[x] I have developed this patch against the main branch.
[ ] I have run ./gradlew check.

❗ ❗ Naming Emergency ❗ ❗
Many v2 APIs are currently offered at two paths based on whether a user is providing a core or a collection name.  For example, schema APIs are available at both /collections/techproducts/schema and /cores/techproducts_shard1_replica_n1/schema
Trying to define these APIs concisely puts us in the awkward place of needing a generic term for either a core or a collection.
So far in this PR I've made due with "index", but I'm not terribly happy with it.  This choice appears in a few places:

JAX-RS annotations e.g. @Path("/{indexType:collections|cores}/{indexName}/schema")
Committed Java code (mostly constants) e.g. public static final String INDEX_NAME_PATH_PARAMETER = "indexName";
Generated SolrRequest classes e.g. public GetSchemaName(IndexType indexType, String indexName) { ... }

It's a somewhat consequential naming decision, as it'll recur in the generated SolrJ code for most of our per core-or-collection APIs: /schema, /config, /select, /update, etc.
Do we already have some generic term to refer to a core or collection?  Does anyone have any good ideas?

corllection ?   Or, could we use the word collection everywhere, and then just under the covers, if it's a Core then do something smart?    Or, do we need actually seperate routes?   Would there be some value in /collection/bob/schema being a totally different url from /core/bob/schema?   Would having them different open up the door to some how better/cleaner routes?

You know, if you want a single term in place of "core" and "collection" in Apache Solr's RESTful URLs, here are some potential terms you could consider:
Index
Repository
Dataset
Store
Source
Resource
Data
Container
Unit
Entity

These terms can convey the idea of a logical grouping or container for data within Solr, similar to the concept of a core or collection.

^ above was generated ;-). I never write like that.   I think I like anything BUT "index" because I think that is an overloaded term that has specific meaning the closer to disk you get..  A Collection is NEVER a index.    Resource?   Container?

Index: An index is a common term used in information retrieval systems to represent a structured collection of data for efficient searching and retrieval. It implies a storage structure that organizes and facilitates quick access to the data.

Repository: A repository is a centralized storage location that holds a collection of data or resources. It suggests a place where data is stored and managed, implying a sense of organization and structure.

Dataset: A dataset refers to a collection of related data that is treated as a single unit. It implies a cohesive and self-contained set of data that can be queried and analyzed collectively.

Store: A store is a generic term that signifies a place where items or data are kept. It implies a central location where data is stored and accessed.

Source: Source implies the origin or provider of data. It suggests a place where data is obtained, managed, and made available for querying and retrieval.

Resource: Resource is a broad term that encompasses data or information that can be accessed, utilized, or manipulated. It conveys the idea of a pool of data that can be interacted with.

Data: Data is a straightforward term that represents information or facts. It conveys the notion of a collection of structured or unstructured information that can be stored and processed.

Container: A container is a metaphorical term indicating a logical enclosure or compartment that holds and organizes items. It suggests a grouping mechanism that holds data in a cohesive manner.

Unit: Unit implies a self-contained entity or component. It suggests a discrete and independent part that can be managed and accessed as a single entity.

Entity: Entity refers to a distinct and individual object or unit. It implies an identifiable element that can be stored, queried, and manipulated within the system.

Haha, someone's enjoying their LLM's this morning 😛
"Store" is a tad generic for my tastes, but I like it better than "index" I think?  We would have "store type" (which could either be 'collection' or 'core') and "store name" (which would be the actual name of the collection/core).

I'm also cool with resourceType and resourceName, but store is cool too.

I think in the balance, I'm leaning to towards "store", because resources could mean many thing...   "dataStoreType" and "dataStoreName" ?
Or, we could just rip the bandaid off and get rid of /core/ ;-).
What if we had /collection/{collectionName}/core/{coreName} for cores?     I.e, if we can't come up with a great solution to the naming problem, let's just change the problem statement to avoid a crummy answer...    In SolrCloud, why would I ever go direct to a core?   I would always navigate through a collection..  Heck, maybe it gives nicer answers.   And, maybe in the future I could get data about a core regardless of which physical node I'm talking to.    In legacy old should be retired and never spoken of again Solr, you just do /collection/standalone/core/{coreName}.    <--- I'm liking this more...

These API's are all experimental and new, so introducing /collection/standalone/ into the "sorta classic" core urls is actually allowed ;-).   And think about it helping folks in the future with their standalone to cloud migrations...

I think I'll go with "store" as the term for now.  I kindof agree with Eric that "resource" is maybe too generic?  It's also an overloaded term, as JAX-RS/Jersey calls API classes "resources" too.  We'll be able to change this later if folks find it doesn't work well in practice.
(Also, I suspected it from his comments above, but confirmed with @epugh offline that some of the other suggestions, i.e. /collections/standalone, were tongue-in-cheek 😛 )
Will aim to update this PR shortly with the new name, and merge.  Thanks for the help guys!

authors: HoustonPutman epugh gerlowskija

repo: apache/solr


text:
[SPARK-13070][SQL] Better error message when Parquet schema merging fails

Now we also report path and schema of the file in trouble.

Test build #50327 has started for PR 10972 at commit 1ce61a4.

Test build #50331 has started for PR 10972 at commit bfe4987.

Test build #50331 has finished for PR 10972 at commit bfe4987.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50331/
Test FAILed.

Test build #50327 has finished for PR 10972 at commit 1ce61a4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50327/
Test PASSed.

Test build #50354 has started for PR 10972 at commit 2e5eddb.

@viirya do you want to submit a pull request to address your issue? @liancheng is busy with something right now.

Test build #50354 has finished for PR 10972 at commit 2e5eddb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50354/
Test FAILed.

Merged build finished. Test FAILed.

@viirya It would be great if you can help since you are pretty familiar with this part of code :)

I'm closing this one.

@rxin @liancheng yes, I would love to do it. Thanks.

authors: AmplabJenkins SparkQA liancheng rxin viirya

repo: apache/spark


text:
[SPARK-22291][SQL] PostgreSQL UUID[] to StringType: Conversion Error

What changes were proposed in this pull request?
This PR fixes the conversion error when reads data from a PostgreSQL table that contains columns of uuid[], inet[] and cidr[] data types.
For example, create a table with the uuid[] data type, and insert the test data.
CREATE TABLE users
(
    id smallint NOT NULL,
    name character varying(50),
    user_ids uuid[],
    PRIMARY KEY (id)
)

INSERT INTO users ("id", "name","user_ids")
VALUES (1, 'foo', ARRAY
    ['7be8aaf8-650e-4dbb-8186-0a749840ecf2'
    ,'205f9bfc-018c-4452-a605-609c0cfad228']::UUID[]
)

Then it will throw the following exceptions when trying to load the data.
java.lang.ClassCastException: [Ljava.util.UUID; cannot be cast to [Ljava.lang.String;
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$14.apply(JdbcUtils.scala:459)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$14.apply(JdbcUtils.scala:458)
...

How was this patch tested?
Added test in PostgresIntegrationSuite.

cc @viirya Can you help review this? Thanks.

Can one of the admins verify this patch?

Please add tests to https://github.com/apache/spark/blob/master/external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala

@jmchung You can add unit test into PostgresIntegrationSuite.

Seems that we don't even have test against uuid column, when you create test for uuid[], can you also create test for uuid? Thanks.

Besides uuid, other PostgreSQL types such as  "cidr", "inet", they are treated as StringType too, will they work?

Thanks @wangyum and @viirya, I'll add the corresponding tests in PostgresIntegrationSuite.
To @viirya , I'm not sure if the other data types will work,  will consider them into tests, thanks!

gentle ping @wangyum and @viirya, the test case has been added to PostgresIntegrationSuite. As @viirya mentioned early, data types inet[] and cidr[] did not work as for instance of String, they are both mapped to Java's Object class.

@jmchung There are another StringType types "json", "jsonb" in PostgresDialect. Can you also add them into tests?  Thanks.

cc @HyukjinKwon @cloud-fan Please help trigger the jenkins test if you have time. Thanks.

ok to test

add to whitelist

Test build #83103 has started for PR 19567 at commit 040e369.

Thanks @HyukjinKwon :)

Test build #83103 has finished for PR 19567 at commit 040e369.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83103/
Test PASSed.

Merged build finished. Test PASSed.

Test build #83116 has started for PR 19567 at commit 0998a77.

Test build #83116 has finished for PR 19567 at commit 0998a77.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83116/
Test PASSed.

Test build #83145 has started for PR 19567 at commit edc5f6f.

Test build #83145 has finished for PR 19567 at commit edc5f6f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83145/
Test PASSed.

LGTM except one comment, good catch!

This is not just for UUID[] for now. We should update the PR title.

Thanks @viirya, the title has been changed. Please correct me if the modified title is still inappropriate.

gentle ping @cloud-fan and @viirya, there are some feedbacks about the behavior of obj to string.
case StringType =>
  (array: Object) =>
    array.asInstanceOf[Array[java.lang.Object]]
      .map(obj => UTF8String.fromString(obj.toString))

As @HyukjinKwon mentioned before, it may unsafe to covnert the object to string directly, we'll have the java.lang.NullPointerException in test cases. StringType is different from other types use nullSafeConvert method to deal with the null, it process within UTF8String.fromString.
The UTF8String.fromString returns null and UTF8String so that we cannot use String.valueOf(obj) to avoid the NPE, it will raise another errors (because null != "null"):
- Type mapping for various types *** FAILED ***
  Array("a", "null", "b") did not equal List("a", null, "b") (PostgresIntegrationSuite.scala:120)

IMHO, if we don't want to make separation of match cases, we may remain null to meet the original design, e.g.,
case StringType =>
  (array: Object) =>
    array.asInstanceOf[Array[java.lang.Object]]
      .map(obj => UTF8String.fromString(
        if (obj == null) null else obj.toString))

Alternative is we keep the String and Object cases?
Would you mind give some comments? I can keep working on this :)

Test build #83183 has started for PR 19567 at commit f3d45d0.

Test build #83183 has finished for PR 19567 at commit f3d45d0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83183/
Test PASSed.

Test build #83190 has started for PR 19567 at commit 588902d.

LGTM

LGTM too

LGTM

Test build #83195 has started for PR 19567 at commit fae5c45.

Test build #83190 has finished for PR 19567 at commit 588902d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83190/
Test PASSed.

Test build #83195 has finished for PR 19567 at commit fae5c45.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83195/
Test FAILed.

Merged build finished. Test FAILed.

the last commit just change the test name and shouldn't break pyspark tests, I'm merging to master, thanks!

@jmchung can you send a new PR for 2.2? thanks!

authors: AmplabJenkins HyukjinKwon SparkQA cloud-fan jmchung viirya wangyum


text:
[SPARK-19355][SQL][Followup] Remove the child.outputPartitioning check

What changes were proposed in this pull request?
This is based on the discussion https://github.com/apache/spark/pull/16677/files#r212805327.
As SQL standard doesn't mandate that a nested order by followed by a limit has to respect that ordering clause, this patch removes the child.outputPartitioning check.
How was this patch tested?
Unit tests.

cc @hvanhovell

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2559/
Test PASSed.

Test build #95272 has started for PR 22239 at commit b57634b.

@viirya did you try to run TakeOrderedAndProjectSuite? I am pretty sure that will fail now ;)...

cc @cloud-fan for a sanity check.

@viirya did you try to run TakeOrderedAndProjectSuite? I am pretty sure that will fail now ;)...

Not yet. Let me try.

@hvanhovell I can set spark.sql.limit.flatGlobalLimit to false to match TakeOrderedAndProjectExec semantics at the beginning of TakeOrderedAndProjectSuite.  Or you prefer to add an explicit flag in GlobalLimitExec as you mentioned in https://github.com/apache/spark/pull/16677#discussion_r212830045? I think it may not necessarily to add the flag for now.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2560/
Test PASSed.

Test build #95273 has started for PR 22239 at commit 67ed97d.

LGTM - Let's wait a little bit with merging to allow others to comment.

Sure, thank you @hvanhovell

@maropu Thanks. I just added it.

Shall we rename it to: [SPARK-19355][SQL][Followup] Remove the child.outputOrdering check in global limit?

Test build #95272 has finished for PR 22239 at commit b57634b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95272/
Test FAILed.

authors: AmplabJenkins SparkQA hvanhovell viirya


text:
[SPARK-25505][SQL][FOLLOWUP] Fix for attributes cosmetically different in Pivot clause

What changes were proposed in this pull request?
#22519 introduced a bug when the attributes in the pivot clause are cosmetically different from the output ones (eg. different case). In particular, the problem is that the PR used a Set[Attribute] instead of an AttributeSet.
How was this patch tested?
added UT

cc @gatorsmile  @maryannxue

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3568/
Test PASSed.

Test build #96752 has started for PR 22582 at commit 2ed74f3.

Test build #96752 has finished for PR 22582 at commit 2ed74f3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96752/
Test PASSed.

thanks for the review @dongjoon-hyun and @viirya. I think I addressed your comments. Thanks.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3596/
Test PASSed.

Test build #96803 has started for PR 22582 at commit 66206a1.

Test build #96803 has finished for PR 22582 at commit 66206a1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96803/
Test PASSed.

authors: AmplabJenkins SparkQA mgaido91


text:
[SPARK-30200][SQL] Add ExplainMode for Dataset.explain

What changes were proposed in this pull request?

This pr intends to add ExplainMode for explaining Dataset/DataFrame with a given format mode (ExplainMode). ExplainMode has four types along with the SQL EXPLAIN command: Simple, Extended, Codegen, Cost, and Formatted.
For example, this pr enables users to explain DataFrame/Dataset with the FORMATTED format implemented in #24759;
scala> spark.range(10).groupBy("id").count().explain(ExplainMode.Formatted)
== Physical Plan ==
* HashAggregate (3)
+- * HashAggregate (2)
   +- * Range (1)

(1) Range [codegen id : 1]
Output: [id#0L]

(2) HashAggregate [codegen id : 1]
Input: [id#0L]

(3) HashAggregate [codegen id : 1]
Input: [id#0L, count#8L]

This comes from the @cloud-fan suggestion.
Why are the changes needed?

To follow the SQL EXPLAIN command.
Does this PR introduce any user-facing change?

No, this is just for a new API in Dataset.
How was this patch tested?

Add tests in ExplainSuite.

Test build #115091 has started for PR 26829 at commit e8c4af1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19906/
Test PASSed.

Test build #115091 has finished for PR 26829 at commit e8c4af1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115091/
Test PASSed.

Thanks for the check, @dongjoon-hyun !

authors: AmplabJenkins SparkQA maropu


text:
[SPARK-25990][SQL] ScriptTransformation should handle different data types correctly

What changes were proposed in this pull request?

We should convert Spark InternalRows to hive data via HiveInspectors.wrapperFor.
Why are the changes needed?

We may hit below exception without this change:
[info]    org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, 192.168.1.6, executor driver): java.lang.ClassCastException: org.apache.spark.sql.types.Decimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
[info]   	at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaHiveDecimalObjectInspector.getPrimitiveJavaObject(JavaHiveDecimalObjectInspector.java:55)
[info]   	at org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:321)
[info]   	at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:292)
[info]   	at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:247)
[info]   	at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.doSerialize(LazySimpleSerDe.java:231)
[info]   	at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.serialize(AbstractEncodingAwareSerDe.java:55)
[info]   	at org.apache.spark.sql.hive.execution.ScriptTransformationWriterThread.$anonfun$run$2(ScriptTransformationExec.scala:300)
[info]   	at org.apache.spark.sql.hive.execution.ScriptTransformationWriterThread.$anonfun$run$2$adapted(ScriptTransformationExec.scala:281)
[info]   	at scala.collection.Iterator.foreach(Iterator.scala:941)
[info]   	at scala.collection.Iterator.foreach$(Iterator.scala:941)
[info]   	at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
[info]   	at org.apache.spark.sql.hive.execution.ScriptTransformationWriterThread.$anonfun$run$1(ScriptTransformationExec.scala:281)
[info]   	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]   	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1932)
[info]   	at org.apache.spark.sql.hive.execution.ScriptTransformationWriterThread.run(ScriptTransformationExec.scala:270)

Does this PR introduce any user-facing change?

No.
How was this patch tested?

Added new test. But please note that this test returns different result between Hive1.2 and Hive2.3 due to HiveDecimal or SerDe difference(don't know the root cause yet).

Test build #118334 has started for PR 27556 at commit 5a176b3.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23091/
Test PASSed.

cc @cloud-fan

Test build #118334 has finished for PR 27556 at commit 5a176b3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118334/
Test PASSed.

Test build #118370 has started for PR 27556 at commit a07ccff.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23127/
Test PASSed.

Test build #118370 has finished for PR 27556 at commit a07ccff.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118370/
Test FAILed.

Jenkins, retest this please.

Test build #118389 has started for PR 27556 at commit a07ccff.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23146/
Test PASSed.

Test build #118389 has finished for PR 27556 at commit a07ccff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118389/
Test PASSed.

thanks, merging to master/3.0!

@Ngone51
Dose your pr make transform support array/map datatype as script's input column?

Dose your pr make transform support array/map datatype as script's input column?

@AngersZhuuuu No. This doesn't add support for new types but just improve already supported types.

authors: AmplabJenkins AngersZhuuuu Ngone51 SparkQA cloud-fan


text:
[SPARK-33245][SQL] Add built-in UDF - GETBIT

What changes were proposed in this pull request?
GETBIT is a bitwise expression function given an INTEGER value, returns the value of a bit at a specified position.
GETBIT( <integer_expr>, <bit_position> )
Examples
select getbit(11, 100), getbit(11, 3), getbit(11, 2), getbit(11, 1), getbit(11, 0);

GETBIT(11, 3)
GETBIT(11, 2)
GETBIT(11, 1)
GETBIT(11, 0)

1
0
1
1

The mainstream database support this feature show below:
Teradata
https://docs.teradata.com/reader/kmuOwjp1zEYg98JsB8fu_A/PK1oV1b2jqvG~ohRnOro9w
Impala
https://docs.cloudera.com/runtime/7.2.0/impala-sql-reference/topics/impala-bit-functions.html#bit_functions__getbit
Snowflake
https://docs.snowflake.com/en/sql-reference/functions/getbit.html
Yellowbrick
https://www.yellowbrick.com/docs/2.2/ybd_sqlref/getbit.html
Why are the changes needed?
GETBIT is very useful.
Does this PR introduce any user-facing change?
Yes. GETBIT is a new bitwise function.
How was this patch tested?
Jenkins test

Test build #134014 has started for PR 31170 at commit c4b9771.

Test build #134014 has started for PR 31170 at commit c4b9771.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38601/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38601/

Test build #134019 has started for PR 31170 at commit b811f16.

Test build #134019 has started for PR 31170 at commit b811f16.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38601/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38601/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38606/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38606/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38601/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38601/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38606/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38606/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38606/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38606/

Test build #134014 has finished for PR 31170 at commit c4b9771.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class BitGet(left: Expression, right: Expression)

Test build #134014 has finished for PR 31170 at commit c4b9771.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class BitGet(left: Expression, right: Expression)

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134014/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134014/

Test build #134019 has finished for PR 31170 at commit b811f16.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134019 has finished for PR 31170 at commit b811f16.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134019/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134019/

Test build #134038 has started for PR 31170 at commit 5499464.

Test build #134038 has started for PR 31170 at commit 5499464.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38625/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38625/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38625/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38625/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38625/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38625/

Test build #134038 has finished for PR 31170 at commit 5499464.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134038 has finished for PR 31170 at commit 5499464.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134038/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134038/

Test build #134045 has started for PR 31170 at commit fe79dd5.

Test build #134045 has started for PR 31170 at commit fe79dd5.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38631/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38631/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38631/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38631/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38631/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38631/

Test build #134049 has started for PR 31170 at commit bcc5135.

Test build #134049 has started for PR 31170 at commit bcc5135.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38634/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38634/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38634/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38634/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38634/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38634/

Test build #134049 has finished for PR 31170 at commit bcc5135.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134049 has finished for PR 31170 at commit bcc5135.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134049/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134049/

Test build #134053 has started for PR 31170 at commit 24878c9.

Test build #134053 has started for PR 31170 at commit 24878c9.

Test build #134045 has finished for PR 31170 at commit fe79dd5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134045 has finished for PR 31170 at commit fe79dd5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134045/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134045/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38639/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38639/

Test build #134055 has started for PR 31170 at commit f0570c5.

Test build #134055 has started for PR 31170 at commit f0570c5.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38641/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38641/

Test build #134057 has started for PR 31170 at commit 61ded37.

Test build #134057 has started for PR 31170 at commit 61ded37.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38641/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38641/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38643/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38643/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38641/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38641/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38643/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38643/

Test build #134053 has finished for PR 31170 at commit 24878c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134053 has finished for PR 31170 at commit 24878c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134053/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38643/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134053/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38643/

Test build #134057 has finished for PR 31170 at commit 61ded37.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134057 has finished for PR 31170 at commit 61ded37.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134057/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134057/

Test build #134055 has finished for PR 31170 at commit f0570c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134055 has finished for PR 31170 at commit f0570c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134055/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134055/

cc @cloud-fan

cc @cloud-fan

retest this please

retest this please

Test build #134067 has started for PR 31170 at commit 61ded37.

Test build #134067 has started for PR 31170 at commit 61ded37.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38653/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38653/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38653/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38653/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38653/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38653/

Test build #134067 has finished for PR 31170 at commit 61ded37.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134067 has finished for PR 31170 at commit 61ded37.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134067/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134067/

retest this please

retest this please

Test build #134097 has started for PR 31170 at commit 61ded37.

Test build #134097 has started for PR 31170 at commit 61ded37.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38683/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38683/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38683/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38683/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38683/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38683/

Test build #134097 has finished for PR 31170 at commit 61ded37.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134097 has finished for PR 31170 at commit 61ded37.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134097/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134097/

authors: AmplabJenkins SparkQA beliefer


text:
[SPARK-34354][SQL] Fix failure when apply CostBasedJoinReorder on self-join

What changes were proposed in this pull request?

This PR introduces a new analysis rule DeduplicateRelations, which deduplicates any duplicate relations in a plan first and then deduplicates conflicting attributes(which resued the dedupRight of ResolveReferences).
Why are the changes needed?

CostBasedJoinReorder could fail when applying on self-join, e.g.,
// test in JoinReorderSuite
test("join reorder with self-join") {
  val plan = t2.join(t1, Inner, Some(nameToAttr("t1.k-1-2") === nameToAttr("t2.k-1-5")))
      .select(nameToAttr("t1.v-1-10"))
      .join(t2, Inner, Some(nameToAttr("t1.v-1-10") === nameToAttr("t2.k-1-5")))

    // this can fail
    Optimize.execute(plan.analyze)
}

Besides, with the new rule DeduplicateRelations, we'd be able to enable some optimizations, e.g., LeftSemiAnti pushdown, redundant project removal, as reflects in updated unit tests.
Does this PR introduce any user-facing change?

How was this patch tested?

Added and updated unit tests.

cc @cloud-fan @maropu @HyukjinKwon

This commit 2541fd6 fixes the test failure. Previously, the test failed due to:
FlatMapCoGroupsInPandas has a self flatMap. So, when DeduplicateRelations applies on FlatMapCoGroupsInPandas, both leftAttributes and rightAttributes would be rewrite after the right relation deduplicated. But that's not right for the  leftAttributes as it should reference the left relation.

Test build #136819 has started for PR 32027 at commit 2541fd6.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41400/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41400/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41400/

Test build #136819 has finished for PR 32027 at commit 2541fd6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136819/

I will leave it to @cloud-fan for a final look.

thanks, merging to master!

NOTE: This fix improved some TPCDS(sf=20) queries (q14a, q14b ...), , e.g., 180636ms=>141545ms in q14b.

authors: AmplabJenkins HyukjinKwon Ngone51 SparkQA cloud-fan maropu


text:
[SPARK-35639][SQL] Add metrics about coalesced partitions to CustomShuffleReader in AQE

What changes were proposed in this pull request?

CustomShuffleReaderExec reports "number of skewed partitions" and "number of skewed partition splits".
It would be useful to also report "number of partitions to coalesce" and "number of coalesced partitions" and include this in string rendering of the SparkPlan node so that it looks like this

Why are the changes needed?

Improves usability
Does this PR introduce any user-facing change?

Yes, it now provides more information about CustomShuffleReaderExec operator behavior in the UI and logged version of the plan.
How was this patch tested?

Existing tests

Can one of the admins verify this patch?

@cloud-fan could you please take a look

ok to test

Test build #139613 has started for PR 32776 at commit aee1392.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44140/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44140/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44140/

Test build #139613 has finished for PR 32776 at commit aee1392.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139613/

Test build #141100 has started for PR 32776 at commit 5360f51.

@cloud-fan
I rebased after https://github.com/apache/spark/pull/32872 and https://github.com/apache/spark/pull/33342
and removed the changes to CustomShuffleReaderExec.stringArgs that you objected to.
So now this just adds 2 new metrics numCoalescedPartitions and numPartitionsToCoalesce to compliment the existing numSkewedSplits metrics.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45613/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45613/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45613/

Test build #141100 has finished for PR 32776 at commit 5360f51.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141100/

Since skewed partitions and coalesced partitions can appear together in one custom shuffle reader, I think it's reasonable to add the numCoalescedPartitions metric.
numPartitionsToCoalesce is a bit overkill to me. If we can know the original number of partitions of the shuffle (ShuffleExchangeExec can report it), we can get a sense of the custom shuffle reader by looking at num skewed partitions/splits and num coalesced partitions.

That makes sense.

Test build #141526 has started for PR 32776 at commit 83a4082.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46044/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46044/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46044/

Test build #141526 has finished for PR 32776 at commit 83a4082.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141526/

rebased on master after SPARK-36217

Test build #141655 has started for PR 32776 at commit 18f865b.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46171/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46171/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46171/

Test build #141655 has finished for PR 32776 at commit 18f865b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141655/

Test build #141713 has started for PR 32776 at commit 7f51402.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46226/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46226/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46226/

Test build #141713 has finished for PR 32776 at commit 7f51402.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141713/

thanks, merging to master/3.2!

authors: AmplabJenkins SparkQA cloud-fan ekoifman


text:
[SPARK-37613][SQL] Support ANSI Aggregate Function: regr_count

What changes were proposed in this pull request?
This PR used to support ANSI aggregate Function: regr_count
The mainstream database supports regr_count show below:
Teradata
https://docs.teradata.com/r/kmuOwjp1zEYg98JsB8fu_A/1skTm5SD0HD3Z~vii3hXAg
Snowflake
https://docs.snowflake.com/en/sql-reference/functions/regr_count.html#
Oracle
https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/REGR_-Linear-Regression-Functions.html#GUID-A675B68F-2A88-4843-BE2C-FCDE9C65F9A9
H2
http://www.h2database.com/html/functions-aggregate.html#regr_count
Postgresql
https://www.postgresql.org/docs/8.4/functions-aggregate.html
Sybase
https://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.help.sqlanywhere.12.0.0/dbreference/regr-count-function.html
Exasol
https://docs.exasol.com/sql_references/functions/alphabeticallistfunctions/regr_function.htm
Why are the changes needed?
regr_count is very useful.
Does this PR introduce any user-facing change?
'No'. New feature.
How was this patch tested?
New tests.

Test build #146136 has started for PR 34880 at commit 268cc7b.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50610/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50610/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50610/

Test build #146136 has finished for PR 34880 at commit 268cc7b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146136/

Test build #146153 has started for PR 34880 at commit c8bd0ca.

Test build #146154 has started for PR 34880 at commit afe9695.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50626/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50627/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50626/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50627/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50627/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50626/

Test build #146154 has finished for PR 34880 at commit afe9695.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ping @MaxGekk @gengliangwang cc @cloud-fan

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146154/

Test build #146173 has started for PR 34880 at commit f3651d9.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50646/

Test build #146153 has finished for PR 34880 at commit c8bd0ca.

This patch fails from timeout after a configured wait of 500m.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class RegrCount(left: Expression, right: Expression)

retest this please

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50646/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50646/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146153/

Test build #146182 has started for PR 34880 at commit f3651d9.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50655/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50655/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50655/

Test build #146173 has finished for PR 34880 at commit f3651d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146173/

Test build #146182 has finished for PR 34880 at commit f3651d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146182/

Test build #146248 has started for PR 34880 at commit 4e026be.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50722/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50722/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50722/

Test build #146248 has finished for PR 34880 at commit 4e026be.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146248/

retest this please

Test build #146264 has started for PR 34880 at commit 4e026be.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50736/

please also address https://github.com/apache/spark/pull/34880#discussion_r770233966

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50739/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50736/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50736/

Test build #146269 has started for PR 34880 at commit 9dfbae4.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50739/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50739/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50744/

Test build #146274 has started for PR 34880 at commit d0e6bb7.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50749/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50744/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50744/

Test build #146269 has finished for PR 34880 at commit 9dfbae4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50749/

retest this please

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50749/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146269/

please also rerun the github actions (I don't have permissions to do so as the github action runs on your account)

Test build #146283 has started for PR 34880 at commit d0e6bb7.

Test build #146264 has finished for PR 34880 at commit 4e026be.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

please also rerun the github actions (I don't have permissions to do so as the github action runs on your account)

OK

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50757/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146264/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50757/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50757/

Test build #146274 has finished for PR 34880 at commit d0e6bb7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146274/

Test build #146283 has finished for PR 34880 at commit d0e6bb7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/146283/

thanks, merging to master!

@cloud-fan Thanks a lot!

authors: AmplabJenkins SparkQA beliefer cloud-fan


text:
[SPARK-38123][SQL] Unified use DataType as targetType of QueryExecutionErrors#castingCauseOverflowError

What changes were proposed in this pull request?
SPARK-33541 introduces QueryExecutionErrors#castingCauseOverflowError and there are 2 ways for input parameter targetType in Spark code now:

Use DataType.catalogString as targetType, such as use in Cast.scala and IntervalUtils.scala
Use custom literal such as short,int and long in Decimal.scala and numberics.scala

This pr change to unified use DataType as the targetType.
Another change of this pr is to change the targetType from int to LongType.catalogString when FloatExactNumeric#toLong method throw castingCauseOverflowError, this seems to be a issue left over from history.
Why are the changes needed?
Unified use DataType.catalogString as targetType when throwing castingCauseOverflowError and bug fix
Does this PR introduce any user-facing change?
No
How was this patch tested?
Pass GA

cc @srowen @dongjoon-hyun @HyukjinKwon

cc @MaxGekk FYI

I would just pass types into castingCauseOverflowError() and invoke catalogString inside of the method. Don't see any reasons to call catalogString every time.

c36aa59 fix this.
But codegen related code in Cast.scala needs  the original method, for example:
https://github.com/apache/spark/blob/8a559b317f60e2630717a01424a7663e451fb3ce/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L1639-L1657
I didn't find how to call the new method in this code, so I kept the original method @MaxGekk

If the above code is also changed to use the new method, there wil be some UTs failed. For example ANSI mode: Throw exception on casting out-of-range value to long type in AnsiCastSuiteWithAnsiModeOn, the failed reason is
Caused by: java.lang.IllegalArgumentException: Can not interpolate org.apache.spark.sql.types.LongType$ into code block.

the error stack as follows:
(codegen mode) Expected exception java.lang.ArithmeticException to be thrown, but java.lang.IllegalArgumentException was thrown
ScalaTestFailureLocation: org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper at (ExpressionEvalHelper.scala:164)
org.scalatest.exceptions.TestFailedException: (codegen mode) Expected exception java.lang.ArithmeticException to be thrown, but java.lang.IllegalArgumentException was thrown
	at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
	at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
	at org.scalatest.funsuite.AnyFunSuite.newAssertionFailedException(AnyFunSuite.scala:1563)
	at org.scalatest.Assertions.intercept(Assertions.scala:756)
	at org.scalatest.Assertions.intercept$(Assertions.scala:746)
	at org.scalatest.funsuite.AnyFunSuite.intercept(AnyFunSuite.scala:1563)
	at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.$anonfun$checkExceptionInExpression$1(ExpressionEvalHelper.scala:164)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.scalatest.Assertions.withClue(Assertions.scala:1065)
	at org.scalatest.Assertions.withClue$(Assertions.scala:1052)
	at org.scalatest.funsuite.AnyFunSuite.withClue(AnyFunSuite.scala:1563)
	at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkException$1(ExpressionEvalHelper.scala:163)
	at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkExceptionInExpression(ExpressionEvalHelper.scala:184)
	at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkExceptionInExpression$(ExpressionEvalHelper.scala:156)
	at org.apache.spark.sql.catalyst.expressions.CastSuiteBase.checkExceptionInExpression(CastSuiteBase.scala:48)
	at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkExceptionInExpression(ExpressionEvalHelper.scala:153)
	at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.checkExceptionInExpression$(ExpressionEvalHelper.scala:150)
	at org.apache.spark.sql.catalyst.expressions.CastSuiteBase.checkExceptionInExpression(CastSuiteBase.scala:48)
	at org.apache.spark.sql.catalyst.expressions.AnsiCastSuiteBase.$anonfun$testLongMaxAndMin$1(AnsiCastSuiteBase.scala:59)
	at org.apache.spark.sql.catalyst.expressions.AnsiCastSuiteBase.$anonfun$testLongMaxAndMin$1$adapted(AnsiCastSuiteBase.scala:56)
	at scala.collection.immutable.List.foreach(List.scala:431)
	at org.apache.spark.sql.catalyst.expressions.AnsiCastSuiteBase.testLongMaxAndMin(AnsiCastSuiteBase.scala:56)
	at org.apache.spark.sql.catalyst.expressions.AnsiCastSuiteBase.$anonfun$new$31(AnsiCastSuiteBase.scala:119)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:190)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:203)
	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:188)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:200)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:200)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:182)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:64)
	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:64)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:233)
	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
	at scala.collection.immutable.List.foreach(List.scala:431)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:233)
	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:232)
	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1563)
	at org.scalatest.Suite.run(Suite.scala:1112)
	at org.scalatest.Suite.run$(Suite.scala:1094)
	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1563)
	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:237)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:237)
	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:236)
	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:64)
	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:64)
	at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45)
	at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1320)
	at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1314)
	at scala.collection.immutable.List.foreach(List.scala:431)
	at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1314)
	at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:993)
	at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:971)
	at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1480)
	at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:971)
	at org.scalatest.tools.Runner$.run(Runner.scala:798)
	at org.scalatest.tools.Runner.run(Runner.scala)
	at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2or3(ScalaTestRunner.java:38)
	at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:25)
Caused by: java.lang.IllegalArgumentException: Can not interpolate org.apache.spark.sql.types.LongType$ into code block.
	at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotInterpolateClassIntoCodeBlockError(QueryExecutionErrors.scala:318)
	at org.apache.spark.sql.catalyst.expressions.codegen.Block$BlockHelper$.$anonfun$code$1(javaCode.scala:240)
	at org.apache.spark.sql.catalyst.expressions.codegen.Block$BlockHelper$.$anonfun$code$1$adapted(javaCode.scala:237)
	at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
	at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
	at org.apache.spark.sql.catalyst.expressions.codegen.Block$BlockHelper$.code$extension(javaCode.scala:237)
	at org.apache.spark.sql.catalyst.expressions.CastBase.$anonfun$castFractionToIntegralTypeCode$1(Cast.scala:1732)
	at org.apache.spark.sql.catalyst.expressions.CastBase.castCode(Cast.scala:1045)
	at org.apache.spark.sql.catalyst.expressions.CastBase.doGenCode(Cast.scala:995)
	at org.apache.spark.sql.catalyst.expressions.Expression.$anonfun$genCode$3(Expression.scala:151)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.sql.catalyst.expressions.Expression.genCode(Expression.scala:146)
	at org.apache.spark.sql.catalyst.expressions.CastBase.genCode(Cast.scala:986)
	at org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext.$anonfun$generateExpressions$1(CodeGenerator.scala:1272)
	at scala.collection.immutable.List.map(List.scala:293)
	at org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext.generateExpressions(CodeGenerator.scala:1272)
	at org.apache.spark.sql.catalyst.expressions.codegen.GenerateMutableProjection$.create(GenerateMutableProjection.scala:64)
	at org.apache.spark.sql.catalyst.expressions.codegen.GenerateMutableProjection$.generate(GenerateMutableProjection.scala:49)
	at org.apache.spark.sql.catalyst.expressions.MutableProjection$.createCodeGeneratedObject(Projection.scala:84)
	at org.apache.spark.sql.catalyst.expressions.MutableProjection$.createCodeGeneratedObject(Projection.scala:80)
	at org.apache.spark.sql.catalyst.expressions.CodeGeneratorWithInterpretedFallback.createObject(CodeGeneratorWithInterpretedFallback.scala:47)
	at org.apache.spark.sql.catalyst.expressions.MutableProjection$.create(Projection.scala:95)
	at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.evaluateWithMutableProjection(ExpressionEvalHelper.scala:235)
	at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.evaluateWithMutableProjection$(ExpressionEvalHelper.scala:232)
	at org.apache.spark.sql.catalyst.expressions.CastSuiteBase.evaluateWithMutableProjection(CastSuiteBase.scala:48)
	at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.$anonfun$checkExceptionInExpression$5(ExpressionEvalHelper.scala:184)
	at org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf(SQLHelper.scala:54)
	at org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf$(SQLHelper.scala:38)
	at org.apache.spark.sql.catalyst.expressions.CastSuiteBase.withSQLConf(CastSuiteBase.scala:48)
	at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.$anonfun$checkExceptionInExpression$3(ExpressionEvalHelper.scala:167)
	at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.$anonfun$checkExceptionInExpression$3$adapted(ExpressionEvalHelper.scala:165)
	at scala.collection.immutable.List.foreach(List.scala:431)
	at org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper.$anonfun$checkExceptionInExpression$2(ExpressionEvalHelper.scala:165)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.scalatest.Assertions.intercept(Assertions.scala:749)
	... 70 more

@MaxGekk

I didn't find how to call the new method in this code, so I kept the original method @MaxGekk

@LuciferYang Something like this:
val catalogType2 = IntegerType
val ct = ctx.addReferenceObj("catalogType", catalogType2, catalogType2.getClass.getName)
...
throw QueryExecutionErrors.castingCauseOverflowError($c, "$ct");

@MaxGekk Thank you for your guidance, already update the pr, waiting ci

+1, LGTM. Merging to master.
Thank you, @LuciferYang, and @HyukjinKwon @srowen for review.

thanks all ~

authors: HyukjinKwon LuciferYang MaxGekk


text:
[SPARK-39060][SQL][3.0] Typo in error messages of decimal overflow

What changes were proposed in this pull request?

This PR removes extra curly bracket from debug string for Decimal type in SQL.
This is a backport from master branch. Commit: https://github.com/apache/spark/commit/165ce4eb7d6d75201beb1bff879efa99fde24f94
Why are the changes needed?

Typo in error messages of decimal overflow.
Does this PR introduce any user-facing change?

No
How was this patch tested?

By running updated test:
$ build/sbt "sql/testOnly"

@MaxGekk PTAL, last one affected by this typo. Thank you!

Can one of the admins verify this patch?

authors: AmplabJenkins vli-databricks


text:
[SPARK-40326][BUILD] Upgrade fasterxml.jackson.version to 2.13.4

What changes were proposed in this pull request?
upgrade com.fasterxml.jackson.dataformat:jackson-dataformat-yaml from 2.13.3 to 2.13.4
Why are the changes needed?
CVE-2022-25857
SNYK-JAVA-ORGYAML
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA

@LuciferYang

The CVE CVE-2022-25857 seems to be related to package org.yaml:snakeyaml. Does fasterxml.jackson has dependency on org.yaml:snakeyaml and  upgrading to 2.13.4 mitigating it?

@ashutoshcipher Yes, SNYK give me this pr https://github.com/bjornjorgensen/spark/pull/25 where they well upgrade com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:
2.13.3 -> 2.13.4 after I change that and run ./dev/test-dependencies.sh --replace-manifest
org.yaml:snakeyaml is upgraded.

Merged to master/3.3

Can one of the admins verify this patch?

authors: AmplabJenkins ashutoshcipher bjornjorgensen srowen