text:
[cleanup][doc] Remove dangling pages develop-binary-protocol

This closes https://github.com/apache/pulsar-site/issues/172.
Although we can redirect develop- to the proper developing- as:
---
id: develop-binary-protocol
---

import {Redirect} from '@docusaurus/router';

<Redirect to="developing-binary-protocol" />

I notice that we don't always have a develop- page even in the incubating versions. So I regard all those develop- pages as dangling pages. We can just remove them.

develop- total 27:

./site2/website/versioned_docs/version-2.2.0/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.2.1/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.3.0/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.3.1/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.3.2/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.4.0/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.4.1/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.4.2/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.5.0/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.5.1/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.5.2/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.6.0/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.6.1/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.6.2/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.6.3/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.6.4/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.7.0/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.7.1/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.7.3/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.7.4/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.8.0-deprecated/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.8.1-deprecated/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.8.2-deprecated/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.8.x/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.9.0-deprecated/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.9.1-deprecated/develop-binary-protocol.md
./site2/website/versioned_docs/version-2.9.x/develop-binary-protocol.md

developing- total 37:

./site2/docs/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.1.0-incubating/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.1.1-incubating/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.10.0-deprecated/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.10.1-deprecated/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.10.x/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.2.0/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.2.1/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.3.0/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.3.1/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.3.2/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.4.0/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.4.1/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.4.2/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.5.0/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.5.1/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.5.2/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.6.0/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.6.1/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.6.2/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.6.3/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.6.4/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.7.0/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.7.1/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.7.2/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.7.3/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.7.4/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.8.0-deprecated/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.8.1-deprecated/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.8.2-deprecated/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.8.3-deprecated/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.8.x/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.9.0-deprecated/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.9.1-deprecated/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.9.2-deprecated/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.9.3-deprecated/developing-binary-protocol.md
./site2/website/versioned_docs/version-2.9.x/developing-binary-protocol.md

cc @urfreespace @Anonymitaet As we delete content files, https://github.com/apache/pulsar-site/issues/175 this issue can be significant to resolve later.

[x] doc

cc @eolivelli @michaeljmarshall

@michaeljmarshall what do you mean about "source"? I will say all develop-binary-protocol is duplicate of developing-binary-protocol under the same folder.

@tisonkun - thanks for the explanation. I missed those duplicates.

authors: michaeljmarshall tisonkun


text:
[client][javadoc] fix a typo of ConsumerBuilder javadoc

Motivation
Fix typo
Documentation
Check the box below or label this PR directly.
Need to update docs?

[x] doc
This PR changes client's javadoc

/pulsarbot run-failure-checks

source: gharchive/pull-request

repo: apache/pulsar

license_type: permissive

license_source: github-api


text:
[draft] [fix] [client] Could not consume messages after one partition deleted

Motivation
Background:

For the client side, there is a regular task that monitors topic creation and deletion when registering a regex consumer

Stop subscribing after a topic has been deleted.
Start a new subscriber after a new topic has been created.

https://github.com/apache/pulsar/pull/5230 stopped to recreate partitions under the partitioned topic to avoid a partition being created after it was deleted. This led to a part of the partitions being deleted.

Issue:

Regexp Topic & Partitioned Topics: The Pulsar client stopped subscriptions of all partitions when one partition has been deleted, even if other partitions are still there. It causes the messages in other partitions not to be fully consumed.
Regexp Topic: The Pulsar client tries to create subscriptions for all partitions when the consumer starts, even if some partitions have been deleted. It causes the consumer not to start and throws an error "topic not found"( the issue of the current ticket )

You can reproduce the issue by the test testConsumerAfterOnePartDeleted.
Modifications

Instead of stopping all partitions when a partition has been deleted, just stop the partitions which have been deleted.
Split GetTopicsResult into two classes:

GetTopicsResult: the field topics are the collection of the topic names that contains partition name. It is only used by ClientCnx.GET_TOPICS_OF_NAMESPACE.
CollatedGetTopicsResult: merge the topic names that are under the same Partitioned Topic, It is only used in the client internal.

TODO: Since this PR has changed a lot, the following things will be done in the subsequent PR:

[ ] The same test for the Reader API.
[ ] Collate methods for the class MultiTopicsConsumerImpl and PatternMultiTopicsConsumerImpl
[ ] Recheck the behavior of TopicListWatcher.

Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository: x

Split to 3 PRs

[x] https://github.com/apache/pulsar/pull/22766
[ ] (Following PR) Multi Topics Consumer only subscribes to the existing partitions if the config createTopicIfDoesNotExist is false.
[ ] (Following PR) The Patten consumer that is started only removes the partitions that were deleted.


text:
[improve][broker] Recover susbcription creation on the broken schema ledger topic

Motivation
Right now, broker failed to create schema-consumer if the topic's schema ledger is already deleted and broker is not able to recover that ledger. In that case, client will keep receiving below error message and will never be able to consumer on that topic.
2024-04-09T23:17:00,140 - INFO  - [pulsar-client-io-35-3:ConsumerImpl] - [persistent://my-prop/ns1/t1][test] Subscribing to topic on cnx [id: 0x59ed6366, L:/127.0.0.1:60336 - R:localhost/127.0.0.1:60330], consumerId 1
2024-04-09T23:17:00,141 - INFO  - [pulsar-io-8-1:ServerCnx] - [[id: 0x92bca72e, L:/127.0.0.1:60330 - R:/127.0.0.1:60336]] Subscribing on topic persistent://my-prop/ns1/t1 / test. consumerId: 1
2024-04-09T23:17:00,142 - WARN  - [PulsarTestContext-executor-OrderedExecutor-0-0:ServerCnx] - [/127.0.0.1:60336][persistent://my-prop/ns1/t1][test] Failed to create consumer: consumerId=1, No such ledger exists on Bookies -  ledger=6 - operation=Failed to open ledger
2024-04-09T23:17:00,143 - WARN  - [pulsar-client-io-35-3:ClientCnx] - [id: 0x59ed6366, L:/127.0.0.1:60336 - R:localhost/127.0.0.1:60330] Received error from server: No such ledger exists on Bookies -  ledger=6 - operation=Failed to open ledger
2024-04-09T23:17:00,143 - WARN  - [pulsar-client-io-35-3:ConsumerImpl] - [persistent://my-prop/ns1/t1][test] Failed to subscribe to topic on localhost/127.0.0.1:60330
2024-04-09T23:17:00,143 - WARN  - [pulsar-client-io-35-3:ConnectionHandler] - [persistent://my-prop/ns1/t1] [test] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: {"errorMsg":"No such ledger exists on Bookies -  ledger=6 - operation=Failed to open ledger","reqId":1815857183629359419, "remote":"localhost/127.0.0.1:60330", "local":"/127.0.0.1:60336"}
2024-04-09T23:17:00,143 - WARN  - [pulsar-client-io-35-3:ConnectionHandler] - [persistent://my-prop/ns1/t1] [test] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: {"errorMsg":"No such ledger exists on Bookies -  ledger=6 - operation=Failed to open ledger","reqId":1815857183629359419, "remote":"localhost/127.0.0.1:60330", "local":"/127.0.0.1:60336"} -- Will try again in 1.583 s
2024-04-09T23:17:01,727 - INFO  - [pulsar-timer-84-1:ConnectionHandler] - [persistent://my-prop/ns1/t1] [test] Reconnecting after connection was closed

Modifications
Schema should be recovered if schema ledger is failing to open due to non-recoverable ledger error.
Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

[ ] Dependencies (add or upgrade a dependency)
[ ] The public API
[ ] The schema
[ ] The default values of configurations
[ ] The threading model
[ ] The binary protocol
[ ] The REST endpoints
[ ] The admin CLI options
[ ] The metrics
[ ] Anything that affects deployment

Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository:

@rdhabalia @lhotari @Technoboy-
This pr may cause some problems, please look at the third part of this test. For example, we want to evolve schema from PersonTwo to PersonOne. Data with schema PersonTwo cannot be consumed. And only the schema PersonOne is version 1.
    /**
     * This test validates that consumer/producers should recover on topic whose
     * schema ledgers are not able to open due to non-recoverable error.
     *
     * @throws Exception
     */
    @Test
    public void testDeletedSchemaLedgerRecovery() throws Exception {
        final String tenant = PUBLIC_TENANT;
        final String namespace = "test-namespace-" + randomName(16);
        final String topicOne = "test-multi-version-schema-one";
        final String subName = "test";
        final String topicName = TopicName.get(TopicDomain.persistent.value(), tenant, namespace, topicOne).toString();

        admin.namespaces().createNamespace(tenant + "/" + namespace, Sets.newHashSet(CLUSTER_NAME));

        // (1) create schema
        Producer<Schemas.PersonTwo> producer = pulsarClient
                .newProducer(Schema.AVRO(SchemaDefinition.<Schemas.PersonTwo> builder().withAlwaysAllowNull(false)
                        .withSupportSchemaVersioning(true).withPojo(Schemas.PersonTwo.class).build()))
                .topic(topicName).create();

        Schemas.PersonTwo personTwo = new Schemas.PersonTwo();
        personTwo.setId(1);
        personTwo.setName("Pulsar");

        Consumer<Schemas.PersonTwo> consumer = pulsarClient
                .newConsumer(Schema.AVRO(SchemaDefinition.<Schemas.PersonTwo> builder().withAlwaysAllowNull(false)
                        .withSupportSchemaVersioning(true).withPojo(Schemas.PersonTwo.class).build()))
                .subscriptionName(subName).subscriptionType(SubscriptionType.Shared).topic(topicName).subscribe();

        producer.send(personTwo);
        // producer.close();
        // consumer.close();

        // (2) Delete schema ledger
        MetadataCache<SchemaStorageFormat.SchemaLocator> locatorEntryCache = pulsar.getLocalMetadataStore()
                .getMetadataCache(new MetadataSerde<SchemaStorageFormat.SchemaLocator>() {
                    @Override
                    public byte[] serialize(String path, SchemaStorageFormat.SchemaLocator value) {
                        return value.toByteArray();
                    }

                    @Override
                    public SchemaStorageFormat.SchemaLocator deserialize(String path, byte[] content, Stat stat)
                            throws IOException {
                        return SchemaStorageFormat.SchemaLocator.parseFrom(content);
                    }
                });
        String path = "/schemas/public/" + namespace + "/test-multi-version-schema-one";
        SchemaLocator schema = locatorEntryCache.get(path).get().get();
        long ledgerId = schema.getInfo().getPosition().getLedgerId();
        pulsar.getBookKeeperClient().deleteLedger(ledgerId);

        // (3) Topic should recover from deleted schema and should allow to create consumer and producer?????????
        Consumer<Schemas.PersonOne> consumer1 = pulsarClient
                .newConsumer(Schema.AVRO(SchemaDefinition.<Schemas.PersonOne> builder().withAlwaysAllowNull(false)
                        .withSupportSchemaVersioning(true).withPojo(Schemas.PersonOne.class).build()))
                .subscriptionName(subName).subscriptionType(SubscriptionType.Shared).topic(topicName).subscribe();

        Producer<Schemas.PersonOne> producer1 = pulsarClient
                .newProducer(Schema.AVRO(SchemaDefinition.<Schemas.PersonOne> builder().withAlwaysAllowNull(false)
                        .withSupportSchemaVersioning(true).withPojo(Schemas.PersonOne.class).build()))
                .topic(topicName).create();
        assertNotNull(consumer1);
        assertNotNull(producer1);

        Schemas.PersonOne personOne = new Schemas.PersonOne();
        personOne.setId(1);

        producer1.send(personOne);
        Assert.assertEquals(personOne, consumer1.receive().getValue());

        // Old "version" consumer cannot be consumed.
        producer.send(personTwo);
        Assert.assertEquals(personTwo, consumer.receive().getValue());

        consumer.close();
        producer.close();
        consumer1.close();
        producer1.close();
    }

Compared with #18010, there is no configuration control. The default behavior is to automatically skip when the integrity of the schema is destroyed.
Even after this error occurs, whether to skip or not, there should be a configuration that allows users to configure whether to skip or manually repair when the integrity of the schema is damaged.
Of course, the function of manually repairing lost schema does not exist at present. However, I have a pr trying to repair the missing schema.(https://github.com/apache/pulsar/pull/20415） In the future, I will send an email to discuss how to deal with this lost schema.
The current processing of pr does not seem to be a good way.

Even after this error occurs, whether to skip or not, there should be a configuration that allows users to configure whether to skip or manually repair when the integrity of the schema is damaged.

If schema ledger itself is deleted then it means it's similar as having topic without schema, and in that case, we should not make the topic unavailable. This PR simply handles non-recoverable error instead eating and ignoring the exception, and not allowing client to connect. yes, I also agree, we should allow client to connect and create a new schema ledger if needed.

This pr may cause some problems, please look at the third part of this test. For example, we want to evolve schema from PersonTwo to PersonOne. Data with schema PersonTwo cannot be consumed. And only the schema PersonOne is version 1

It's because schema ledger is already deleted and in that case, topic doesn't have schema means it should allow to create a new version. Schema deletion means data loss and you can not do much to avoid such situation other than allowing to add a new schema and make topic available to the client applications.


text:
Debezium: integrate KafkaConnectSource with MySqlConnectorTask

Motivation
This change is to integrate KafkaConnectSource with MySqlConnectorTask.
Steps to run this is like this:
1, use image in debezium tutorial, start mysql server
docker run -it --rm --name mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=debezium -e MYSQL_USER=mysqluser -e MYSQL_PASSWORD=mysqlpw debezium/example-mysql:0.8
2, start pulsar standalone
bin/pulsar standalone
3,  start consume data from connect source target topic, After step 4 is start, it should get data.
bin/pulsar-client consume -s "my-sub-name" kafka-connect-topic -n 0
4, start source connect, It should produce data into Pulsar topics.
bin/pulsar-admin source localrun --tenant public --namespace default --name kafka  --destination-topic-name  kafka-connect-topic --sourceConfigFile  debezium-mysql-source-config.yaml  --archive connectors/pulsar-io-kafka-connect-adaptor-2.2.0-SNAPSHOT.nar --brokerServiceUrl pulsar://127.0.0.1:6650
Modifications
Describe the modifications you've done.
Result
After your change, what will change.

@sijie, Yes, I planed to added a test for this, But this source is a little not compatible with currently currently source test framework.
In current source test, we call produceSourceMessages: Map<String, String> kvs = tester.produceSourceMessages(numMessages)  and return expected kvs to verify later, But in this source, the SourceMessages is created by debezium (convert mysql binlog into KafkaConnectSource), And it seems not easy to save the content as explicit Kvs.

@jiazhai you can consider making a new test case just for debezium related tests.

@sijie opened new issue for integration tests.

authors: jiazhai sijie


text:
[issue #3895] Bugfix for non partitioned topic create

Motivation
Provide a different approach from #3902 trying to fix #3895.
Modifications

Add unit test in order to exploit issue.
Do proper validations for non topic create.
Add missing validateTopicName() method on create non partitioned topic.

Verifying this change

[ ] Make sure that the change passes the CI checks.

well done, this is better resolve this issue.

Thanks @lovelle @ambition119 for working on this issue.

run integration tests

Moving to 2.4 since the feature being fixed was not there in 2.3

authors: ambition119 jiazhai lovelle merlimat


text:
[docs]add parameter for helm init cluster

Motivation
Failed to deploy the pulsar cluster on k8s according to the document.
I checked the pulsar-helm repository, need add parameter --set initialize=true to initialize the cluster metadata.
So maybe the document needs to be updated

@yufan022 Thank you very much for your contribution. Does the change apply only to 2.7.1?

@yufan022 Thank you very much for your contribution. Does the change apply only to 2.7.1?

@Jennifer88huang Maybe more versions,  It depends on the version of Helm.

@@yufan022 Thanks for your info. From this commit "apache/pulsar-helm-chart@b44b523", we can see that it affects versions since 2.6.1. Could you please apply the changes to 2.6.1, 2.6.2, 2.6.3, 2.7.0 and the main branch as well? Thank you.

@Jennifer88huang Hi, some new commits. Please take a look.

@yufan022 Great, thank you. The file in main branch is this file https://github.com/apache/pulsar/blob/master/site2/docs/getting-started-helm.md
Could you apply the changes to this file as well? So it will apply to the next major release (2.8.0). Thank you very much for doing so.

@yufan022 Great, thank you. The file in main branch is this file https://github.com/apache/pulsar/blob/master/site2/docs/getting-started-helm.md
Could you apply the changes to this file as well? So it will apply to the next major release (2.8.0). Thank you very much for doing so.

Completed PTAL

@yufan022 Thank you very much for your contribution~


text:
fix #88

fix #88

Codecov Report

Merging #140 (93a9855) into main (b0d2190) will increase coverage by 0.06%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #140      +/-   ##
==========================================
+ Coverage   47.66%   47.72%   +0.06%
==========================================
  Files          99       99
  Lines        4299     4304       +5
  Branches      663      665       +2
==========================================
+ Hits         2049     2054       +5
  Misses       1896     1896
  Partials      354      354

Impacted Files
Coverage Δ

...rocketmq/mqtt/cs/session/loop/SessionLoopImpl.java
45.53% <100.00%> (+0.85%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter pingww

repo: apache/rocketmq-mqtt


text:
tempBuffer的pid的值会不会被地址值覆盖掉？

ip地址的长度明显长于2，占4位～！

@yuanx1 不会，pid用两个字节就够了，issue用于bug报告，以及新特性讨论，如果有问题的话尽量现在邮件列表（https://rocketmq.apache.org/about/contact/）讨论，这样的话是可以被检索到的，也方便沉淀下来。我先暂时关了这个issue，如果有问题的话，随时reopen。

source: gharchive/issue

repo: apache/rocketmq


text:
Can not understand the documentation

消费者在向Master拉取消息时，Master服务器会根据拉取偏移量与最大偏移量的距离（判断是否读老消息，产生读I/O），以及从服务器是否可读等因素建议下一次是从Master还是Slave拉取。

是什么考虑下，建议下一次是从Master还是Slave拉取，，这个没有看懂，感谢指点。

Issues are generally used for bug reports and feature requests, so if you have any questions about     Apache  RocketMQ, it is recommended to discuss it on the mailing list first, the mailing list [address]    (https://rocketmq.apache.org/about/contact). I will close the Issue first, but please feel free to     reopen it if you have any other issues.

authors: imaffe startjava


text:
[Container] Log split not taking effect for DLedger in container

BUG REPORT

Please describe the issue you observed:

What did you do (The steps to reproduce)?
Container has 3 brokers for each node.

What is expected to see?
Logs belonging to different brokers should be output to different directories.

What did you see instead?
The log is output by defaultAppend, that is, the log is output to the same directory

Please tell us about your environment:

branch: 5.0.0-beta
Linux

Other information (e.g. detailed explanation, logs, related issues, suggestions on how to fix, etc):
The logger for different brokers can't be found because brokerId for dLedger may be changed. We should use dLedgerId to replace brokerId for dLedger mode.

Merged

authors: RongtongJin cserwen


text:
[ISSUE #7560] [RIP-68] Support RocketMQ ACL 2.0

Which Issue(s) This PR Fixes

Fixes #7560
Brief Description

Standardized IP whitelist control: The enhanced ACL design provides a more standardized IP whitelist control mechanism. It effectively restricts user requests to specific IP sources and blocks access from untrusted IP addresses.
Scalable ACL configuration and authentication mechanism: The improved design allows for easy extension and implementation of ACL-related logic. Users can conveniently customize and expand ACL configurations to meet their specific requirements. Additionally, the ACL design includes access control for control-related interfaces, enhancing the overall security of the system.
Effective separation of user and permission management: The optimized design successfully achieves a clear separation between user authentication and permission management, establishing explicit responsibilities and boundaries for each. This enhancement significantly improves the security of the system. Additionally, User passwords should be stored in a non-plaintext format., effectively reducing the risk of password leaks.

How Did You Test This Change?
broker config
authenticationEnabled = true
authenticationProvider = org.apache.rocketmq.auth.authentication.provider.DefaultAuthenticationProvider
initAuthenticationUser = {"username":"rocketmq","password":"12345678"}
innerClientAuthenticationCredentials = {"accessKey":"rocketmq","secretKey":"12345678"}
authenticationMetadataProvider = org.apache.rocketmq.auth.authentication.provider.LocalAuthenticationMetadataProvider
authorizationEnabled = true
authorizationProvider = org.apache.rocketmq.auth.authorization.provider.DefaultAuthorizationProvider
authorizationMetadataProvider = org.apache.rocketmq.auth.authorization.provider.LocalAuthorizationMetadataProvider

proxy config
{
  "authenticationEnabled": true,
  "authenticationProvider": "org.apache.rocketmq.auth.authentication.provider.DefaultAuthenticationProvider",
  "authenticationMetadataProvider": "org.apache.rocketmq.proxy.auth.ProxyAuthenticationMetadataProvider",
  "innerClientAuthenticationCredentials": "{\"accessKey\":\"rocketmq\", \"secretKey\":\"12345678\"}",
  "authorizationEnabled": true,
  "authorizationProvider": "org.apache.rocketmq.auth.authorization.provider.DefaultAuthorizationProvider",
  "authorizationMetadataProvider": "org.apache.rocketmq.proxy.auth.ProxyAuthorizationMetadataProvider"
}

migration from acl v1
migrateAuthFromV1Enabled = true

The PR is too large, it is recommended to split it into multiple submissions.

Codecov Report
Attention: 2493 lines in your changes are missing coverage. Please review.

Comparison is base (f0a3e93) 43.15% compared to head (8b0bb3a) 42.49%.
Report is 8 commits behind head on develop.

:exclamation: Current head 8b0bb3a differs from pull request most recent head 3cb0020. Consider uploading reports for the commit 3cb0020 to get more accurate results

Files
Patch %
Lines

...on/builder/DefaultAuthorizationContextBuilder.java
1.77%
221 Missing :warning:

...ocketmq/broker/processor/AdminBrokerProcessor.java
0.00%
188 Missing :warning:

...g/apache/rocketmq/auth/migration/AuthMigrator.java
0.00%
112 Missing :warning:

...g/apache/rocketmq/client/impl/MQClientAPIImpl.java
2.63%
74 Missing :warning:

...cketmq/tools/command/auth/CreateAclSubCommand.java
0.00%
67 Missing :warning:

...cketmq/tools/command/auth/UpdateAclSubCommand.java
0.00%
67 Missing :warning:

...ion/manager/AuthenticationMetadataManagerImpl.java
41.81%
48 Missing and 16 partials :warning:

...e/rocketmq/broker/auth/converter/AclConverter.java
0.00%
60 Missing :warning:

...rocketmq/tools/command/auth/ListAclSubCommand.java
0.00%
60 Missing :warning:

...tion/manager/AuthorizationMetadataManagerImpl.java
59.15%
34 Missing and 24 partials :warning:

... and 118 more

Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #7725      +/-   ##
=============================================
- Coverage      43.15%   42.49%   -0.66%
- Complexity      9801    10178     +377
=============================================
  Files           1165     1251      +86
  Lines          84485    87843    +3358
  Branches       10969    11377     +408
=============================================
+ Hits           36463    37333     +870
- Misses         43491    45836    +2345
- Partials        4531     4674     +143

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Codecov Report
Attention: 2104 lines in your changes are missing coverage. Please review.

Comparison is base (04eec34) 43.22% compared to head (a728695) 43.06%.
Report is 1 commits behind head on develop.

Files
Patch %
Lines

...g/apache/rocketmq/auth/migration/AuthMigrator.java
0.00%
112 Missing :warning:

...g/apache/rocketmq/client/impl/MQClientAPIImpl.java
2.63%
74 Missing :warning:

...cketmq/tools/command/auth/CreateAclSubCommand.java
0.00%
67 Missing :warning:

...cketmq/tools/command/auth/UpdateAclSubCommand.java
0.00%
67 Missing :warning:

...ion/manager/AuthenticationMetadataManagerImpl.java
41.81%
48 Missing and 16 partials :warning:

...ocketmq/broker/processor/AdminBrokerProcessor.java
69.11%
46 Missing and 17 partials :warning:

...rocketmq/tools/command/auth/ListAclSubCommand.java
0.00%
60 Missing :warning:

.../rocketmq/tools/command/auth/GetAclSubCommand.java
0.00%
57 Missing :warning:

...tion/manager/AuthorizationMetadataManagerImpl.java
59.12%
32 Missing and 24 partials :warning:

...ketmq/tools/command/auth/UpdateUserSubCommand.java
0.00%
50 Missing :warning:

... and 118 more

Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #7725      +/-   ##
=============================================
- Coverage      43.22%   43.06%   -0.16%
- Complexity      9870    10371     +501
=============================================
  Files           1173     1259      +86
  Lines          85034    88372    +3338
  Branches       11013    11416     +403
=============================================
+ Hits           36752    38059    +1307
- Misses         43717    45552    +1835
- Partials        4565     4761     +196

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

缓存部分，打开  cache 自带的 stats 统计能力并输出，性能影响很小，默认缓存大小是否合理？
先试运行一段时间，后期考虑可以加一下
能否快速找到问题用户或者资源？
目前通过审计日志进行支持
this.kvStore.flushWal(true); rocksdb 部分应该保证数据同步落盘。
底层方法使用的就是这个能力
鉴权降级和部分降级能力是否验证过。
可通过修改配置进行降级
RequestHeaderRegistry.getInstance().initialize(); 这个地方能不能去单例，broker container 模式是否可以？
这个功能是扫描所有的RequestHeader然后构建一个映射关系，这个系统级别的数据，用单例还是比较合适的
if (this.authorizationMetadataManager != null) 这里 bug 了，shutdown 了两次。。。
已修复，感谢指出
请求失败时，错误日志过多，会打爆磁盘。
目前通过日志级别来进行日志区分，后面可以有更加高级的策略进行日志打印，支持问题有效排查的同时，避免日志过多
RemotingCode 需要加前缀和别的请求区分。
已修复，感谢指出
AbstractTransactionalMessageCheckListener.java 这里加个 topic 干啥用的？
事务相关的API原来没有透传topic，导致无法在请求面解析用户资源。最新客户端SDK已经加上这个字段
GrpcServerBuilder.java 鉴权 interceptors 和 global interceptors 的顺序是否正确？
interceptor就是倒叙添加的，这个已经检查并确认

It‘s better to have a clear roadmap like how to treat the old version acl.

It‘s better to have a clear roadmap like how to treat the old version acl.

The old version acl will be removed soon, It is good idea to deprecated at enableAcl config.

Are there any plans to reverse this ability into RocketMQ 4.x?

Are there any plans to reverse this ability into RocketMQ 4.x?

There is no such plan at the moment, but you can implement it yourself.

authors: codecov-commenter dingshuangxi888 drpmma frinda lizhimins


text:
[ISSUE #8810] Fix independent execution of e2e and benchmark deployments

Which Issue(s) This PR Fixes

Fixes #8810
Brief Description

This PR modifies the workflow to ensure that the e2e and benchmark deployments are independent. If one deployment fails, it no longer blocks the execution of the other deployment's tasks.
How Did You Test This Change?

The test workflow

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 47.39%. Comparing base (e75554d) to head (cbaab3b).

Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #8812      +/-   ##
=============================================
- Coverage      47.41%   47.39%   -0.02%
+ Complexity     11585    11574      -11
=============================================
  Files           1283     1283
  Lines          89903    89903
  Branches       11571    11571
=============================================
- Hits           42625    42613      -12
- Misses         42041    42044       +3
- Partials        5237     5246       +9

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


text:
[Feature][doc] Add V2 connector jdbc document parameter can speed up data import PR

https://github.com/apache/seatunnel/issues/6167

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

[x] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[x] If you are contributing the connector code, please check that the following files are updated:

Update change log that in connector document. For more details you can refer to connector-v2
Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist

[x] Update the release-note.

The latest PR add : OceanBase、Kingbase、Vertica、Snowflake、SqlServer、PostgreSql jdbc parameter an related information

The code style check failed. Please run mvn spotless:apply to fix it

I have completed formatting

Oops, there are some conflict should be resolved.

repo: apache/seatunnel


text:
omega 开启 hystrix，使用Feign调用bug

有两个服务 A、B，A服务开启hystrix，并且设置threadpool参数，如下：
feign:
  hystrix:
    enabled: true

hystrix:
  threadpool:
    default:
      coreSize: 3
      maximumSize: 5
  command:
    default:
      execution:
        isolation:
          thread:
            timeoutInMilliseconds: ${HYSTRIX_COMMAND_TIMEOUT_IN_MILLISECONDS:40000}

在调用次数大于coreSize时，之后的每次调用，omega设置到feign调用的header中的globalTxId均为第三次（即coreSize的值）的globalTxId，现象如下：
第三次：
2019-10-15 10:17:09.881 DEBUG 5604 --- [nio-8300-exec-6] o.a.s.p.o.transaction.SagaStartAspect    : Initialized context OmegaContext{globalTxId=`2522c732-a7e2-46f2-91c9-7ffcbce2a2b7`, localTxId=2522c732-a7e2-46f2-91c9-7ffcbce2a2b7} before execution of method public java.lang.String com.example.booking.service.BookingServiceImpl.booking(java.lang.String)
 : 开始预定。。。
2019-10-15 10:17:09.883 DEBUG 5604 --- [  hystrix-car-3] .s.p.o.t.f.FeignClientRequestInterceptor : Added X-Pack-Global-Transaction-Id `2522c732-a7e2-46f2-91c9-7ffcbce2a2b7` and X-Pack-Local-Transaction-Id 2522c732-a7e2-46f2-91c9-7ffcbce2a2b7 to request header
2019-10-15 10:17:09.945 DEBUG 5604 --- [hystrix-hotel-3] .s.p.o.t.f.FeignClientRequestInterceptor : Added X-Pack-Global-Transaction-Id `2522c732-a7e2-46f2-91c9-7ffcbce2a2b7` and X-Pack-Local-Transaction-Id 2522c732-a7e2-46f2-91c9-7ffcbce2a2b7 to request header

第四次及之后:
2019-10-15 10:17:12.176 DEBUG 5604 --- [io-8300-exec-10] o.a.s.p.o.transaction.SagaStartAspect    : Initialized context OmegaContext{globalTxId=`5a9a3533-86dc-4631-881f-d402e2895307`, localTxId=5a9a3533-86dc-4631-881f-d402e2895307} before execution of method public java.lang.String com.example.booking.service.BookingServiceImpl.booking(java.lang.String)
 : 开始预定。。。
2019-10-15 10:17:12.176 DEBUG 5604 --- [  hystrix-car-3] .s.p.o.t.f.FeignClientRequestInterceptor : Added X-Pack-Global-Transaction-Id `2522c732-a7e2-46f2-91c9-7ffcbce2a2b7` and X-Pack-Local-Transaction-Id 2522c732-a7e2-46f2-91c9-7ffcbce2a2b7 to request header
2019-10-15 10:17:12.229 DEBUG 5604 --- [hystrix-hotel-3] .s.p.o.t.f.FeignClientRequestInterceptor : Added X-Pack-Global-Transaction-Id `2522c732-a7e2-46f2-91c9-7ffcbce2a2b7` and X-Pack-Local-Transaction-Id 2522c732-a7e2-46f2-91c9-7ffcbce2a2b7 to request header

这个日志是我本地的测试demo；但是在我们的开发环境有时会出现 globalTxId 为null 的情况；
我猜测是不是和OmegaContxt中使用的 InheritableThreadLocal  有关系，或者存在不知的bug？

在调用次数大于coreSize时，之后的每次调用，omega设置到feign调用的header中的globalTxId均为第三次（即coreSize的值）的globalTxId，现象如下：

我发现你的日志中第三次和第四次的 globalTxId 不一样
第三次是 globalTxId=2522c732-a7e2-46f2-91c9-7ffcbce2a2b7
第四次是 globalTxId=5a9a3533-86dc-4631-881f-d402e2895307

这个日志是我本地的测试demo；但是在我们的开发环境有时会出现 globalTxId 为null 的情况；
我猜测是不是和OmegaContxt中使用的 InheritableThreadLocal 有关系，或者存在不知的bug？

你在开发环境运行的时候出现 globalTxId 为 null 是因为设置了断点吗？

第三次和第四次是两次请求，本来就是不一样的。

@pengyu0929 Is it possible to provide a simple reproducer and we can try locally to do the more investigation ? I think there might be some issues when using the thread pool, e.g. clear or reset the globalTxId when releasing the thread and put back to the pool ?

第三次和第四次是两次请求，本来就是不一样的。开发环境运行的时候并没有设置断点

没明白，我看到你说“在调用次数大于coreSize时，之后的每次调用，omega设置到feign调用的header中的globalTxId均为第三次（即coreSize的值）的globalTxId，现象如下” 我以为你举例说的是第三次后以后每次请求的 globalTxId 都和第三次是一样的。

http://www.itmuch.com/spring-cloud-sum/hystrix-threadlocal/
我是参考上面的连接解决问题的，你们可以看下在omega端启用hystrix时，是否存在我描述的这个问题，我使用的saga版本是0.4

Thanks @pengyu0929 for sharing this information and would you like to contribute a PR to fix this issue ?

这个还是你们自己写吧，链接中已经写的比较清楚了。但是要考虑到如果使用者在omega端已经实现了一个策略，可能在代理的时候出现问题。其他的就没什么问题了。

我也碰到用Feign调用时，子事务获取不到globalTxId，导致报错。没有用Hystrix，只是单纯的Feign调用，经测试将Feign改为Dubbo调用就没问题。

@ghost0224 你要确定A服务是否将globalTxId传递下去，如果传递下去了，再判断B服务是否从请求头中获取globalTxId，并且设置到当前服务的OmegaContext中

我也碰到用Feign调用时，子事务获取不到globalTxId，导致报错。没有用Hystrix，只是单纯的Feign调用，经测试将Feign改为Dubbo调用就没问题。

你是用Feign调用Dubbo服务吗？

这个还是你们自己写吧，链接中已经写的比较清楚了。但是要考虑到如果使用者在omega端已经实现了一个策略，可能在代理的时候出现问题。
还有这种方式的扩展性比较差，只要新增一个线程变量就要重写一个类。
其他的就没什么了。

线程上下文切换的时候，需要解决OmegaContext传递的问题，我们需要知道你是如何传递ThreadLocal的， 最好是有一个测试能够复现这个问题，这样可以帮助我们比较快地解这一问题。

@WillemJiang @pengyu0929 @zhfeng 我提交了一个PR解决这个issue,有空可以看一下

Thanks for the @xiangyuQi 's PR to address this issue, now we close this issue.

authors: WillemJiang coolbeevip ghost0224 pengyu0929 xiangyuQi zhfeng

repo: apache/servicecomb-pack


text:
optimize document

[ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.
[ ] Update the CHANGES log.

I noticed you closed the previous PR, what happens?

After submitting the modification, I found that a check was not passed, but I did not know how to modify it, and then I close it

It seems that the proto file has been modified, but I did not modify such files

I can't see proto is modified by you.

Error: fatal: remote error: upload-pack: not our ref 17f05ee21d5bd3d40cfce5ccc8f0174316c47450
Error: fatal: Fetched in submodule path 'apm-protocol/apm-network/src/main/proto', but it did not contain 17f05ee21d5bd3d40cfce5ccc8f0174316c47450. Direct fetching of that commit failed.
fatal:
Error: The process '/usr/bin/git' failed with exit code 128

What's the problem

This may be just a GHA random issue. We could rerun the task to verify.

authors: Oliverwqcwrw wu-sheng

repo: apache/skywalking-java


text:
Support SSL cert files loaded dynamically

By following #4475, SSL could be changed in the same way, so, please adopt that PR in the SSL implementation. Relate to #4470

Duplicated with #5110

repo: apache/skywalking

license_source: bigquery


text:
[Satellite] Support transmit Envoy ALS/Metric protocol

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
Envoy has a lot of observable data that can be analyzed. At present, OAP has provided support for analytical capabilities.
So we should providing protocol transmit ability in Satellite as well. The following is an introduction link for related protocols. Need to provide support for the v2 and v3 versions of ALS and Metrics respectively.
Metrics service: https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/metrics/v3/metrics_service.proto.html
ALS: https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/accesslog/v3/als.proto
Use case
The satellite could transmit Envoy Metrics and ALS protocol data.
Related issues
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

v2 are required for both of them. The OAP backend supports them in the same time, please check whether satellite could be both.

v2 are required for both of them.

Sure, v2 and v3 versions need both of them.

The OAP backend supports them in the same time, please check whether satellite could be both.

I have checked the OAP analysis code, there are combine the message content, but it needs to make sure v2 and v3 versions must be compatible. I think satellite could just transmit them, don't need to combine the message content.

I have checked the OAP analysis code, there are combine the message content, but it needs to make sure v2 and v3 versions must be compatible. I think satellite could just transmit them, don't need to combine the message content.

No, satellite doesn't need to change the content, just forward them.

I have checked the OAP analysis code, there are combine the message content, but it needs to make sure v2 and v3 versions must be compatible. I think satellite could just transmit them, don't need to combine the message content.

No, satellite doesn't need to change the content, just forward them.

Sure.

authors: mrproliu wu-sheng


text:
[SPARK-19960][CORE] Move SparkHadoopWriter to internal/io/

What changes were proposed in this pull request?
This PR introduces the following changes:

Move SparkHadoopWriter to core/internal/io/, so that it's in the same directory with SparkHadoopMapReduceWriter;
Move SparkHadoopWriterUtils to a separated file.

After this PR is merged, we may consolidate SparkHadoopWriter and SparkHadoopMapReduceWriter, and make the new commit protocol support the old mapred package's committer;
How was this patch tested?
Tested by existing test cases.

Test build #74597 has started for PR 17304 at commit eee9678.

Test build #74597 has finished for PR 17304 at commit eee9678.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74597/
Test PASSed.

cc @rxin @mridulm

Merging in master.

authors: AmplabJenkins SparkQA jiangxb1987 rxin

repo: apache/spark


text:
[WIP][SPARK-22063][R] Upgrades lintr to latest commit sha1 ID

What changes were proposed in this pull request?
Currently, we set lintr to jimhester/lintr@a769c0b (see this and SPARK-14074).
I first tested and checked lintr-1.0.1 but it looks many important fixes are missing (for example, checking 100 length). So, I decided to propose it with the sha1 to https://github.com/jimhester/lintr/commit/5431140ffea65071f1327625d4a8de9688fa7e72.
It looks it has fixed many bugs and now finds many instances that I have observed and thought should be caught time to time, here I filed the results.
The downside looks it now takes about 7ish mins, (it was 2ish mins before) in my local.
How was this patch tested?
Manually, ./dev/lint-r after manually updating the lintr package.

Test build #81988 has started for PR 19290 at commit 6e41eff.

@felixcheung and @shivaram,
Should we upgrade this to the latest commit? it increases the time by 5ish mins.
If there is any worry about upgrading, I am willing to leave the script as was. The other fixes look reasonable and I have wondered why these are not being caught. Does this make sense to both of you too?

Test build #81988 has finished for PR 19290 at commit 6e41eff.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
#'                   equivalent to setting thresholds c(1-p, p). In multiclass (or binary)

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81988/
Test FAILed.

Test build #82000 has started for PR 19290 at commit 2baefd9.

Test build #82000 has finished for PR 19290 at commit 2baefd9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82000/
Test PASSed.

@HyukjinKwon Thanks for looking at this. The 5 min addition seems unfortunate though -- does that also happen with lintr-1.0.1 ? I wonder if we are seeing some specific performance slowdown because of not using a release (it might be good to follow up with lint-r authors if that is the case ?)

I think this is great to have, thanks for solving the mystery.

5 min: this is mildly concerning, is it possible this is caused by new checks in lintr? perhaps we could exclude them or something?
lintr version: I see what you say about 1.0.1. According this commit list https://github.com/jimhester/lintr/commits/master the tag for 1.0.1 vs the latest (the git tag you are using) are different only by some insignificant changes?

Test build #82034 has started for PR 19290 at commit 49a166d.

To @shivaram:

does that also happen with lintr-1.0.1 ? I wonder if we are seeing some specific performance slowdown because of not using a release (it might be good to follow up with lint-r authors if that is the case ?)

I just ran it with lintr-1.0.1. This took 2ish mins in my local, which is almost identical with the current, jimhester/lintr@a769c0b. Will open up an issue there and ask this in their repo tomorrow (pending ..).

To @felixcheung:

5 min: this is mildly concerning, is it possible this is caused by new checks in lintr? perhaps we could exclude them or something?

I tried to disable all the linters documented in https://github.com/jimhester/lintr/blob/master/README.md#available-linters:
diff --git a/R/pkg/.lintr b/R/pkg/.lintr
index ae50b28ec61..ca1efcfacc7 100644
--- a/R/pkg/.lintr
+++ b/R/pkg/.lintr
@@ -1,2 +1,2 @@
-linters: with_defaults(line_length_linter(100), multiple_dots_linter = NULL, camel_case_linter = NULL, open_curly_linter(allow_single_line = TRUE), closed_curly_linter(allow_single_line = TRUE))
+linters: with_defaults(commas_linter = NULL, object_usage_linter = NULL, absolute_path_linter = NULL, nonportable_path_linter = NULL, pipe_continuation_linter = NULL, assignment_linter = NULL, closed_curly_linter = NULL, extraction_operator_linter = NULL, implicit_integer_linter = NULL, infix_spaces_linter = NULL, line_length_linter = NULL, no_tab_linter = NULL, object_length_linter = NULL, object_name_linter = NULL, open_curly_linter = NULL, semicolon_terminator_linter = NULL, single_quotes_linter = NULL, spaces_inside_linter = NULL, spaces_left_parentheses_linter = NULL, todo_comment_linter = NULL, trailing_blank_lines_linter = NULL, trailing_whitespace_linter = NULL, T_and_F_symbol_linter = NULL, undesirable_function_linter = NULL, undesirable_operator_linter = NULL, unneeded_concatenation_linter = NULL)
 exclusions: list("inst/profile/general.R" = 1, "inst/profile/shell.R")

and this took 1.5 mins. Looks we could identify some linters that we might not need. Let me check this one tomorrow (pending ..)

lintr version: I see what you say about 1.0.1. According this commit list https://github.com/jimhester/lintr/commits/master the tag for 1.0.1 vs the latest (the git tag you are using) are different only by some insignificant changes?

Diff looks 122 commits - https://github.com/jimhester/lintr/compare/v1.0.1...master . Will take a look though the commit logs (pending ..)

Test build #82034 has finished for PR 19290 at commit 49a166d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82034/
Test FAILed.

isn't it the other way around?
https://github.com/jimhester/lintr/compare/v1.0.1?expand=1

btw, could you check if haven't already, if nolint around the http link, roxygen is going to handle that correctly?

I initially did this, for example,
\href{https://spark.apache.org/docs/latest/sparkr.html#data-type-mapping-between-
r-and-spark}{Spark Data Types} for available data types.

this passes the lint and doc is find but cran check is failed. I actually tried to find out the way but ended up with nolint. ... will try to read the doc one more and out few more cases locally.

Doh, you mean the current status. Yes, I checked.

Test build #82081 has started for PR 19290 at commit 7e6c2c5.

Will update the comment tomorrow.

anyway I mean if we use the git tag 9951084 it should match v1.0.1 release exactly.
the one you listed jimhester/lintr@5431140 is in master I think and perhaps you are saying there are more changes in master (which is true from the diff)?

Ah, yes. It looks some changes are intendedly not into v1.0.1 (vs the master), and given the results from the linter in my local, v1.0.1 missed some important changes. Although I am not used to the policy there, I see the diff in the commit logs (master vs tag v1.0.1).

5 min: this is mildly concerning, is it possible this is caused by new checks in lintr? perhaps we could exclude them or something?

1.0.1 miss the linters below (vs master):
nonportable_path_linter: check that file.path() is used to construct safe and portable paths.
pipe_continuation_linter: Check that each step in a pipeline is on a new line, or the entire pipe fits on one line.
extraction_operator_linter: check that the [[ operator is used when extracting a single element from an object, not [ (subsetting) nor $ (interactive use).
implicit_integer_linter: check that integers are explicitly typed using the form 1L instead of 1.
object_name_linter: check that object names conform to a single naming style, e.g. snake_case or lowerCamelCase.
semicolon_terminator_linter: check that no semicolons terminate statements.
todo_comment_linter: check that the source contains no TODO comments (case-insensitive).
T_and_F_symbol_linter: avoid the symbols T and F (for TRUE and FALSE).
undesirable_function_linter: report the use of undesirable functions, e.g. options or sapply and suggest an alternative.
undesirable_operator_linter: report the use of undesirable operators, e.g. ::: or <<- and suggest an alternative.
unneeded_concatenation_linter: check that the c function is not used without arguments nor with a single constant

I manually disabled the linters above but the elapsed time looks almost the same. So, I think some existing linters were slowed down in the master - I tried to disable everything except for each linter - https://gist.github.com/HyukjinKwon/1f852bcdcc2a13f396dfd2d6a88bea16

I also tried v1.0.1 against the master; however, it looks not detecting many instances this PR fixes. It only detected 3 extra instances:
R/DataFrame.R:2615:22: style: Variable and function names should not be longer than 30 characters.
            colsX <- generateAliasesForIntersectedCols(x, by, suffixes[1])
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
R/DataFrame.R:2616:22: style: Variable and function names should not be longer than 30 characters.
            colsY <- generateAliasesForIntersectedCols(y, by, suffixes[2])
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
R/DataFrame.R:2666:1: style: Variable and function names should not be longer than 30 characters.
generateAliasesForIntersectedCols <- function (x, intersectedColNames, suffix) {
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
lintr checks failed.

So, I would like to propose to use jimhester/lintr@5431140 if possible.
I opened an issue to ask a question about this slowdown - https://github.com/jimhester/lintr/issues/270. Probably, we could wait for few days.  if nothing could be done for now, I guess we could consider just use jimhester/lintr@5431140 as is ...

retest this please

Test build #82120 has started for PR 19290 at commit 7e6c2c5.

Test build #82120 has finished for PR 19290 at commit 7e6c2c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82120/
Test PASSed.

oh, remember to remove WIP and update this line https://github.com/apache/spark/pull/19290/files#diff-74ca2b618d236bbd6faa23e13bff1403R30 before merging in?

Test build #82128 has started for PR 19290 at commit 387228d.

Test build #82129 has started for PR 19290 at commit ee7eb9d.

Ugh.. it failed to install due to permission issue ...
Downloading GitHub repo jimhester/lintr@5431140
from URL https://api.github.com/repos/jimhester/lintr/zipball/5431140
Installing lintr
Downloading GitHub repo hadley/xml2@master
from URL https://api.github.com/repos/hadley/xml2/zipball/master
Installing xml2
'/usr/lib64/R/bin/R' --no-site-file --no-environ --no-save --no-restore  \
  --quiet CMD INSTALL  \
  '/tmp/RtmprFnaBu/devtools438c516b7717/r-lib-xml2-5799cd9'  \
  --library='/usr/lib64/R/library' --install-tests

Error: ERROR: no permission to install to directory '/usr/lib64/R/library'
Installation failed: Command failed (1)
'/usr/lib64/R/bin/R' --no-site-file --no-environ --no-save --no-restore  \
  --quiet CMD INSTALL  \
  '/tmp/RtmprFnaBu/devtools438c286c3792/jimhester-lintr-5431140'  \
  --library='/usr/lib64/R/library' --install-tests

Error: ERROR: no permission to install to directory '/usr/lib64/R/library'
Installation failed: Command failed (1)

Hi @shaneknapp. I am sorry but it's me again ...
Here, this PR tries to upgrade an R package, lintr for static Code analysis for R, which is ran via ./dev/lint-r -> dev/lint-r.R as a part of Jenkins PR build.
This upgrade catches many meaningful instances that should have caught before in SparkR codes and if I understood correctly, we decided to upgrade this if it does not cause a problem.
I tried to fix the dev/lint-r.R script so that lintr can be upgraded if this package is not installed or not the specific version (1.0.1.9000) after testing this in my local, but looks failed in Jenkins due to a permission issue.
I believe the bash command below installs the package correctly:
Rscript -e "devtools::install_github('jimhester/lintr@5431140')"

Maybe, I should have cc'ed you here first before trying this one. I appologise and I will cc you in such cases in the future.

Test build #82128 has finished for PR 19290 at commit 387228d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82128/
Test PASSed.

Merged build finished. Test PASSed.

Test build #82129 has finished for PR 19290 at commit ee7eb9d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82129/
Test PASSed.

@HyukjinKwon -- you will absolutely not have builds install packages on the build system.  this is a really bad idea.
is this absolutely required, or just to fix a warning in the build output?

@shaneknapp Sure, it was my bad. I will be careful next time.
It is required to fix an actual issue in order to to detect R codes that do not follow project's R style.

Do you maybe have some worries about this? If that worry is quite crucial, I think we could also consider an option, not upgrading this, leaving lint-r.R script as was, and only fixing the instances detected by the upgrade here.

For more context, I believe lintr was initially installed in https://github.com/apache/spark/commit/004f57374b98c4df32d9f1e19221f68e92639a49. Upgrade to jimhester/lintr@a769c0b was proposed here in https://github.com/apache/spark/commit/7d1175011c976756efcd4e4e4f70a8fd6f287026 but I believe this didn't actually upgrade the package on Jenkins as lintr was already installed  and "lintr" %in% row.names(installed.packages())  == FALSE gives FALSE.
So, I believe lintr in Jenkins is already not synced to jimhester/lintr@a769c0b as fixed in https://github.com/apache/spark/commit/7d1175011c976756efcd4e4e4f70a8fd6f287026 unless it was manually upgraded afterwards (I can't check but this is just my assumption).
I believe eventually we should upgrade and match with this anyway to detect project's R style in each PR. I at least have been in a PR where I had to manually suggest to fix.

Test build #82236 has started for PR 19290 at commit ee7eb9d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82236/
Test PASSed.

hmm, perhaps open a ticket to ask shane to update Jenkins to this commit checksum manually?

Let me revert "... || packageVersion("lintr") != "1.0.1.9000" this condition back so that it does not try to upgrade it.

Oh! but if we upgrade it first, many builds in PRs should be broken due to this check failure. Probably, let me revert the script first to not upgrade it in order to unblock this.

Test build #82308 has started for PR 19290 at commit 550750e.

I just double checked the newer lintr, lintr@5431140, passes on the top of the master with the current change.

Test build #82308 has finished for PR 19290 at commit 550750e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82308/
Test FAILed.

retest this please

Test build #82312 has started for PR 19290 at commit 550750e.

Test build #82312 has finished for PR 19290 at commit 550750e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82312/
Test PASSed.

hey all, i'm here.  was out sick for the past few days and trying to get
caught up.  sorry about that!
so...  what version of lintr do we need to put on the workers?
On Fri, Sep 29, 2017 at 3:32 AM, UCB AMPLab notifications@github.com
wrote:

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82312/
Test PASSed.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/19290#issuecomment-333091984, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABiDrC3TwFkXTT_wInCvG0W450khEmugks5snMdJgaJpZM4PdlZS
.

Ah, thank you @shaneknapp. The command below:
Rscript -e "devtools::install_github('jimhester/lintr@5431140')"

should upgrades lintr to the desired version here.
Actually, looks we should merge this one first before upgrading lintr in Jenkins. I realised upgrading it first could make R lintr check failures in other PRs up to my knowledge ..
I updated the change here back to not try to upgrade lintr. So, merging this one would not cause a problem on the build system. Let me go ahead with this one first and then I guess we could upgrade.
Thanks again and I am sorry for this, @shaneknapp.

btw, lintr once upgraded will run on all builds on all branches right? wouldn't the upgrade break other branches?

@felixcheung  --  yes, this is the system default lintr, meaning all calls to lintr will be against this version.
as for other branches, i think it could possibly break them.
@shivaram thoughts?

Let me merge this one first. This shouldn't cause any problem to built system for now.

Merged to master.

So, @felixcheung, @shaneknapp and @shivaram, looks we have comments, https://github.com/apache/spark/pull/19290#issuecomment-333321991 and https://github.com/apache/spark/pull/19290#issuecomment-333322653, to discuss further.
The problem looks upgrading lintr to jimhester/lintr@5431140 could break other builds in other branches.
Should I rather try to install this for each build, for example, via checking the environment variables as a workaround? I checked those before - https://github.com/apache/spark/pull/17669#issue-222376466

authors: AmplabJenkins HyukjinKwon SparkQA felixcheung shaneknapp shivaram


text:
[SPARK-11034] [LAUNCHER] [MESOS] Launcher: add support for monitoring Mesos apps

What changes were proposed in this pull request?
Added Launcher support for monitoring Mesos apps in Client mode. SPARK-11033 can handle the support for Mesos/Cluster mode since the Standalone/Cluster and Mesos/Cluster modes use the same code at client side.
How was this patch tested?
I verified it manually by running launcher application, able to launch, stop and kill the mesos applications and also can invoke other launcher API's.

Can one of the admins verify this patch?

ok to test

Test build #82744 has started for PR 19385 at commit 28f511e.

Test build #82744 has finished for PR 19385 at commit 28f511e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82744/
Test PASSed.

This only solves half the problem, right? What about cluster mode?

Thanks @vanzin for looking into this.

This only solves half the problem, right? What about cluster mode?

Yes, it solves the Mesos/Client mode. For Mesos/Cluster mode, I think the Standalone/Cluster(SPARK-11033) implementaton(or similar way) would handle since both use the same logic at the client side to submit the application.

Test build #82750 has started for PR 19385 at commit ced3e58.

Test build #82750 has finished for PR 19385 at commit ced3e58.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82750/
Test PASSed.

Merged build finished. Test PASSed.

Merging to master.

authors: AmplabJenkins SparkQA devaraj-kavali vanzin


text:
[SPARK-23984][K8S] Initial Python Bindings for PySpark on K8s

What changes were proposed in this pull request?
Introducing Python Bindings for PySpark.

[x] Running PySpark Jobs
[x] Increased Default Memory Overhead value
[ ] Dependency Management for virtualenv/conda

How was this patch tested?
This patch was tested with

[x] Unit Tests
[x] Integration tests with this addition

KubernetesSuite:
- Run SparkPi with no resources
- Run SparkPi with a very long application name.
- Run SparkPi with a master URL without a scheme.
- Run SparkPi with an argument.
- Run SparkPi with custom labels, annotations, and environment variables.
- Run SparkPi with a test secret mounted into the driver and executor pods
- Run extraJVMOptions check on driver
- Run SparkRemoteFileTest using a remote data file
- Run PySpark on simple pi.py example
Run completed in 4 minutes, 3 seconds.
Total number of tests run: 9
Suites: completed 2, aborted 0
Tests: succeeded 9, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

Can one of the admins verify this patch?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2416/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2365/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2365/

Thanks @ifilonenko !
I'm interested in figuring out what it means for the container images to be "python 2/3 generic" - does that imply being able to run either, based on submit parameters?

cc @holdenk

Integration tests are meant to be in this repository but we haven't gotten there yet. See https://github.com/apache/spark/pull/20697

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2798/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2729/

@holdenk I think your comment above gets at a use-case "ambiguity" that containerization causes. There are now at least two choices of channel for supplying dependencies: from the command line, or by customized container (and here there are at least two sub-cases: manually created customizations, or via source-to-image tooling).
When specifying deps via the command line, particularly in cluster mode, we have backed out of staging local files via init-container; does pulling from URI suffice?

@shaneknapp @ssuchter integration tests seem to be failing not due to this PR, but in general. Please investigate, because this PR does pass integration tests + an extra PySpark test.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2822/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2753/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2753/

@erikerlandson I think pulling from URI is fine for now. The actual comment was just focused on the usage of spark-submit in that case, but I agree longer term we should think about dependencies, especially things which can't just be shipped as zip or pyfiles (but I think that is vNext).

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2974/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2894/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2894/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2975/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2895/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2895/

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3057/
Test PASSed.

Merged build finished. Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2973/

jenkins, ok to test

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3078/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2993/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2993/

jenkins, ok to test

Test build #90421 has started for PR 21092 at commit c59068d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3080/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2995/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2995/

jenkins, ok to test

jenkins, retest this please

Test build #90424 has started for PR 21092 at commit 306f3ed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3082/
Test PASSed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2997/

Test build #90424 has finished for PR 21092 at commit 306f3ed.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90424/
Test PASSed.

Test build #90561 has started for PR 21092 at commit 72953a3.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3182/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3088/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3088/

Test build #90561 has finished for PR 21092 at commit 72953a3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90561/
Test FAILed.

jenkins, retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3186/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3091/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3091/

jenkins, retest this please

Test build #90660 has started for PR 21092 at commit 72953a3.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3242/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3146/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3146/

Test build #90660 has finished for PR 21092 at commit 72953a3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90660/
Test PASSed.

@ifilonenko ping on this patch?

Will resolve comments today @mccheah

Test build #91373 has started for PR 21092 at commit 7bedeb6.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3748/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3612/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3612/

Test build #91374 has started for PR 21092 at commit 1801e96.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3749/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3613/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3613/

Test build #91373 has finished for PR 21092 at commit 7bedeb6.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91373/
Test PASSed.

Test build #91374 has finished for PR 21092 at commit 1801e96.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91374/
Test PASSed.

Test build #91394 has started for PR 21092 at commit 24a704e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3761/
Test PASSed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3623/

Test build #91394 has finished for PR 21092 at commit 24a704e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91394/
Test FAILed.

Test build #91530 has started for PR 21092 at commit 6a6d69d.

Test build #91530 has finished for PR 21092 at commit 6a6d69d.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91530/
Test FAILed.

Merged build finished. Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3835/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3692/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3692/

Test build #91537 has started for PR 21092 at commit ab92913.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3837/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3694/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3694/

KubernetesSuite:

Run SparkPi with no resources
Run SparkPi with a very long application name.
Run SparkPi with a master URL without a scheme.
Run SparkPi with an argument.
Run SparkPi with custom labels, annotations, and environment variables.
Run SparkPi with a test secret mounted into the driver and executor pods
Run extraJVMOptions check on driver
Run SparkRemoteFileTest using a remote data file
Run PySpark on simple pi.py example
Run PySpark with Python2 to test a pyfiles example
Run PySpark with Python3 to test a pyfiles example
Run completed in 4 minutes, 28 seconds.
Total number of tests run: 11
Suites: completed 2, aborted 0
Tests: succeeded 11, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:24 min
[INFO] Finished at: 2018-06-07T18:54:42-04:00
[INFO] Final Memory: 21M/509M
[INFO] ------------------------------------------------------------------------

For new addition to: https://github.com/apache-spark-on-k8s/spark-integration/pull/46

Test build #91573 has started for PR 21092 at commit a61d897.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3857/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3712/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/3712/

awesome!

Test build #91573 has finished for PR 21092 at commit a61d897.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91573/
Test PASSed.

Sorry in advance if this is the wrong place to be asking this!
Does this PR mean that we'll be able to create SparkContexts using PySpark's SparkSession.Builder with master set to k8s://<...>:<...>, and have the resulting jobs run on spark-on-k8s, instead of on local/standalone?
E.g.:
from pyspark.sql import SparkSession
spark = SparkSession.builder.master('k8s://https://kubernetes:443').getOrCreate()

I'm trying to use PySpark in a Jupyter notebook that's running inside a Kubernetes pod, and have it use spark-on-k8s instead of resorting to using local[*] as master.
Till now, I've been getting an error saying that:

Error: Python applications are currently not supported for Kubernetes.

whenever I try to use k8s://<...> as master.
Thanks!

@lucashu1 please send your question to stackoverflow or user@spark.apache.org!

authors: AmplabJenkins SparkQA erikerlandson felixcheung foxish holdenk ifilonenko lucashu1 mccheah


text:
[SPARK-24812][SQL] Last Access Time in the table description is not valid

What changes were proposed in this pull request?
Last Access Time will always displayed wrong date Wed Dec 31 15:59:59 PST 1969 when user run  DESC FORMATTED table command
In hive its displayed as "UNKNOWN" which makes more sense than displaying wrong date. seems to be a limitation as of now even from hive, better we can follow the hive behavior unless the limitation has been resolved from hive.
How was this patch tested?
UT has been added which makes sure that the wrong date "Wed Dec 31 15:59:59 PST 1969 "
shall not be added as value for the Last Access  property

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

Seems making sense.

seems to be a limitation as of now even from hive, better we can follow the hive behavior unless the limitation has been resolved from hive.

Do you maybe know related Hive side ticket or can you point me out the related codes?

Test build #93311 has started for PR 21775 at commit b527fdc.

Test build #93311 has finished for PR 21775 at commit b527fdc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93311/
Test PASSed.

This is an external change. Please add one more point in the migration guide.

sure, i will update the PR based on the comments,  Thanks for suggestions.

Test build #93406 has started for PR 21775 at commit e9b0f91.

Test build #93406 has finished for PR 21775 at commit e9b0f91.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93406/
Test FAILed.

Test build #93407 has started for PR 21775 at commit 2008c21.

Test build #93407 has finished for PR 21775 at commit 2008c21.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93407/
Test FAILed.

@HyukjinKwon
seems to be a limitation as of now even from hive, better we can follow the hive behavior unless the limitation has been resolved from hive.

Hive-2526 is the JIRA related to this issue , could not figure out any other open JIRA's mentioning this problem

Test build #93409 has started for PR 21775 at commit 140c4ce.

Test build #93409 has finished for PR 21775 at commit 140c4ce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93409/
Test PASSed.

retest this please

Test build #93421 has finished for PR 21775 at commit 140c4ce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93421/
Test PASSed.

Merged build finished. Test PASSed.

Test build #93431 has started for PR 21775 at commit 76a34c6.

Test build #93431 has finished for PR 21775 at commit 76a34c6.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93431/
Test FAILed.

Merged build finished. Test FAILed.

ok to test

retest this please

Test build #93437 has finished for PR 21775 at commit 76a34c6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93437/
Test PASSed.

@HyukjinKwon @gatorsmile All issues has been addressed, please let me know how this patch looks like. Thanks .

retest this please

The commit has been tested.
LGTM Thanks! Merged to master.

Test build #93513 has started for PR 21775 at commit 76a34c6.

Test build #93513 has finished for PR 21775 at commit 76a34c6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93513/
Test FAILed.

Merged build finished. Test FAILed.

authors: AmplabJenkins HyukjinKwon SparkQA gatorsmile sujith71955


text:
Documentation improvement: More detailed explanation of possible "deploy-mode"s

What changes were proposed in this pull request?
Documentation improvement: the current documentation of "deploy-mode" argument does not explain in detail the behaviour or effects of using client vs cluster. One could start wondering how one (a beginner) is supposed to know and understand the consequences of these.
This PR extends the description with some additional details / explanation.
How was this patch tested?
N/A: documentation only

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

(See our conventions for PR titles)

Test build #4577 has started for PR 23919 at commit a0e64ca.

Test build #4577 has finished for PR 23919 at commit a0e64ca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

This is a good job.

I think this is accurate enough for all resource managers, but would remove the sentence per https://github.com/apache/spark/pull/23919#discussion_r261484458

Closing this due to no feedback from its author.

authors: AmplabJenkins HyukjinKwon SparkQA beliefer peter-gergely-horvath srowen


text:
[SPARK-27997][K8S] Add support for kubernetes OAuth Token refresh

What changes were proposed in this pull request?
This change allows a spark user to provide a class which implements fabric's OAuthTokenProvider to refresh tokens throughout the life of the spark app.
spark.kubernetes.authenticate.submission.oauthTokenProvider=<token>
spark.kubernetes.authenticate.driver.oauthTokenProvider=<token>
spark.kubernetes.authenticate.oauthTokenProvider=<token>

https://javadoc.io/doc/io.fabric8/kubernetes-client/5.6.0/io/fabric8/kubernetes/client/OAuthTokenProvider.html
Why are the changes needed?
Currently, while running spark on kubernetes, one should specify oauth tokens via config before starting an application.
spark.kubernetes.authenticate.submission.oauthToken=<token>
spark.kubernetes.authenticate.driver.oauthToken=<token>
spark.kubernetes.authenticate.oauthToken=<token>

The token has an expiration time (usually an hour, for GKE) and there is no way to update the token in the runtime. The spark app starts to throw exceptions.
io.fabric8.kubernetes.client.KubernetesClientException: Unauthorized
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:202)
	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

Does this PR introduce any user-facing change?
Yes, a configuration option spark.kubernetes.client.oauth.token.provider.class is added.
How was this patch tested?
A class which implements OAuthTokenProvider interface[0] was added into the classpath on driver node with no other spark options for tokens specified
It was also tested with expired tokens specified, and the token was updated via the user-provided class.
--conf spark.kubernetes.authenticate.submission.oauthToken=<expired>
--conf spark.kubernetes.authenticate.driver.oauthToken=<expired>
--conf spark.kubernetes.authenticate.oauthToken=<expired>

There is no need to use any other token-related configuration options if this class is provided.
An example of the user-provided class for GKE
[0] https://gist.github.com/haodemon/5490fefdb258275c1f805d584319090b
import io.fabric8.kubernetes.client.OAuthTokenProvider

class OAuthGoogleTokenProvider extends OAuthTokenProvider {
  private val binary = "gcloud"
  private val args = "config config-helper --format=json"

  override def getToken: String = {
    val response = (binary + " " + args).!!
    val token = new ObjectMapper().readTree(response)
      .get("credential")
      .get("access_token")
    token.getTextValue
  }
}

Can one of the admins verify this patch?

ok to test

Test build #142193 has started for PR 33675 at commit 7b0f6c3.

Test build #142193 has finished for PR 33675 at commit 7b0f6c3.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
      .doc(\"A class that implements fabric's OAuthTokenProvider interface to \" +

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46705/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142193/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46705/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46705/

Can one of the admins verify this patch?

Test build #142197 has started for PR 33675 at commit a20229b.

Test build #142197 has finished for PR 33675 at commit a20229b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
      .doc(\"A class that implements OAuthTokenProvider interface to \" +

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142197/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46709/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46709/

@dongjoon-hyun, answering inline.

Apache Spark mainly runs the unit tests on your GitHub Action. Could you enable the GitHub Action build_and_test job in your Spark fork?

We also have a Jenkins sever. I'll trigger it though.

I'm not sure why they didn't start before, but seems like now they're passing.

Apache Spark 3.3.0 will use 5.6.0, could you update the link to 5.6.0 link in the PR description?

https://javadoc.io/doc/io.fabric8/kubernetes-client/4.12.0/io/fabric8/kubernetes/client/OAuthTokenProvider.html

Updated, thank you.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46709/

BTW, I embedded your example into the PR description to provide it as a commit message, @haodemon . Please let me know if that is not what you want.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142329/

Test build #142330 has started for PR 33675 at commit 5830123.

Test build #142330 has finished for PR 33675 at commit 5830123.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46838/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46838/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142330/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46841/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46841/

Jenkins retest this please.
@haodemon have you run the K8s integration tests locally on your env?

Test build #143220 has started for PR 33675 at commit 5830123.

Test build #143220 has finished for PR 33675 at commit 5830123.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143220/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47722/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47722/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47722/

Test build #143289 has started for PR 33675 at commit 56fc132.

Test build #143289 has finished for PR 33675 at commit 56fc132.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143289/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47792/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47792/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47792/

Test build #143293 has started for PR 33675 at commit 58ce2f4.

Test build #143293 has finished for PR 33675 at commit 58ce2f4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143293/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47796/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47796/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47796/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48455/

Can one of the admins verify this patch?

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50658/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50658/

@haodemon Can we reopen this and I'll help with anything needed to get the tests to pass?

This is reopened according to @tiagovrtr 's request.

@haodemon Can we reopen this and I'll help with anything needed to get the tests to pass?

Thank you, @tiagovrtr. I'll take a look tomorrow and will try to update the PR. In any case, I'll post an update here.

Can one of the admins verify this patch?

this patch seems only to bring the latest changes from master, anything else to do here?

Looping in @renato-farias as I'm leaving the BBC and he'll very much welcome this fix in a future release 🙂

Here is a rebased PR.

https://github.com/apache/spark/pull/42432

authors: AmplabJenkins SparkQA dongjoon-hyun haodemon holdenk tiagovrtr


text:
[SPARK-44803][BUILD] Replace publishOrSkip with publish in SparkBuild to eliminate warnings

What changes were proposed in this pull request?
The pr aims to replace publishOrSkip with publish in SparkBuild to eliminate warnings.
Why are the changes needed?
After we upgraded SBT to 1.9.1+, the publish method has been marked as deprecated, to reduce compilation warnings.

Does this PR introduce any user-facing change?
No.
How was this patch tested?
Manual testing.

We executed the following tests at the code level based on the changes made by SBT.
1.The logic of SBT is as follows:
https://github.com/sbt/sbt/blob/1.9.x/main/src/main/scala/sbt/Defaults.scala#L3642-L3653
2.Test is as follows:

cc @LuciferYang

1.Before
sh build/sbt

Maven:

Ivy:

2.After
sh build/sbt

Maven:

Ivy:

cc @dongjoon-hyun

authors: LuciferYang panbingkun


text:
[SPARK-47045][SQL] Replace IllegalArgumentException by SparkIllegalArgumentException in sql/api

What changes were proposed in this pull request?
In the PR, I propose to replace all IllegalArgumentException by SparkIllegalArgumentException in sql/api code base, and introduce new legacy error classes with the _LEGACY_ERROR_TEMP_ prefix.
Why are the changes needed?
To unify Spark SQL exception, and port Java exceptions on Spark exceptions with error classes.
Does this PR introduce any user-facing change?
Yes, it can if user's code assumes some particular format of IllegalArgumentException messages.
How was this patch tested?
By running existing test suites like:
$ build/sbt "core/testOnly *SparkThrowableSuite"

Was this patch authored or co-authored using generative AI tooling?
No.

@HyukjinKwon @LuciferYang @panbingkun @srielau @hvanhovell @beliefer Could you review this PR, please.

Merging to master. Thank you, @srielau for review.