text:
Caused by: org.apache.distributedlog.exceptions.BKTransmitException: Failed to open ledger handle for log segment

Describe the bug
Perform error bin/pulsar-admin brokers list pulsar-cluster
null
Reason: javax.ws.rs.ProcessingException: pulsar.example.com: 未知的名称或服务
To Reproduce
Steps to reproduce the behavior:

After the start zookeepe bookie broker
exec bin/pulsar-admin brokers list pulsar-cluster
output
null

Reason: javax.ws.rs.ProcessingException: pulsar.example.com: 未知的名称或服务
4. See error
aused by: org.apache.bookkeeper.statelib.api.exceptions.StateStoreException: Failed to restore rocksdb 000000000000000011/000000000000000001/000000000000000000
at org.apache.bookkeeper.statelib.impl.rocksdb.checkpoint.RocksCheckpointer.restore(RocksCheckpointer.java:84) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.kv.RocksdbKVStore.loadRocksdbFromCheckpointStore(RocksdbKVStore.java:161) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.kv.RocksdbKVStore.init(RocksdbKVStore.java:223) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal.lambda$initializeLocalStore$5(AbstractStateStoreWithJournal.java:202) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal.lambda$executeIO$16(AbstractStateStoreWithJournal.java:471) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
... 12 more
Caused by: org.apache.distributedlog.exceptions.BKTransmitException: Failed to open ledger handle for log segment [LogSegmentId:3413, firstTxId:59, lastTxId:59, version:VERSION_V5_SEQUENCE_ID, completionTime:1626933253327, recordCount:1, regionId:0, status:0, logSegmentSequenceNumber:1, lastEntryId:0, lastSlotId:0, inprogress:false, minActiveDLSN:DLSN{logSegmentSequenceNo=1, entryId=0, slotId=0}, startSequenceId:0] : -7
at org.apache.distributedlog.impl.logsegment.BKLogSegmentEntryStore$1.openComplete(BKLogSegmentEntryStore.java:258) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerOpenOp.openComplete(LedgerOpenOp.java:232) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerOpenOp.lambda$initiate$0(LedgerOpenOp.java:117) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_121]
at org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:396) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:575) ~[org.apache.pulsar-pulsar-zookeeper-2.4.0.jar:2.4.0]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) ~[org.apache.pulsar-pulsar-zookeeper-2.4.0.jar:2.4.0]
15:43:46.072 [io-write-scheduler-OrderedScheduler-1-0] INFO  org.apache.bookkeeper.stream.storage.impl.sc.StorageContainerRegistryImpl - Release resources hold by StorageContainer ('11') during de-register
15:43:46.072 [io-write-scheduler-OrderedScheduler-1-0] INFO  org.apache.bookkeeper.stream.storage.impl.sc.StorageContainerImpl - Stopping storage container (11) ...
15:43:46.072 [io-write-scheduler-OrderedScheduler-1-0] INFO  org.apache.bookkeeper.stream.storage.impl.store.MVCCStoreFactoryImpl - scStores for 11 on store factory is null, return directly
15:43:46.072 [io-write-scheduler-OrderedScheduler-1-0] INFO  org.apache.bookkeeper.stream.storage.impl.sc.StorageContainerImpl - Successfully stopped storage container (11).
15:43:46.072 [io-write-scheduler-OrderedScheduler-1-0] WARN  org.apache.bookkeeper.stream.storage.impl.sc.ZkStorageContainerManager - Failed to start storage container (11)
java.util.concurrent.CompletionException: org.apache.bookkeeper.statelib.api.exceptions.StateStoreException: Failed to restore rocksdb 000000000000000011/000000000000000001/000000000000000000
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_121]
at org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal.lambda$executeIO$16(AbstractStateStoreWithJournal.java:474) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111) [com.google.guava-guava-21.0.jar:?]
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58) [com.google.guava-guava-21.0.jar:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75) [com.google.guava-guava-21.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: org.apache.bookkeeper.statelib.api.exceptions.StateStoreException: Failed to restore rocksdb 000000000000000011/000000000000000001/000000000000000000
at org.apache.bookkeeper.statelib.impl.rocksdb.checkpoint.RocksCheckpointer.restore(RocksCheckpointer.java:84) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.kv.RocksdbKVStore.loadRocksdbFromCheckpointStore(RocksdbKVStore.java:161) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.kv.RocksdbKVStore.init(RocksdbKVStore.java:223) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal.lambda$initializeLocalStore$5(AbstractStateStoreWithJournal.java:202) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal.lambda$executeIO$16(AbstractStateStoreWithJournal.java:471) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
... 12 more
Caused by: org.apache.distributedlog.exceptions.BKTransmitException: Failed to open ledger handle for log segment [LogSegmentId:3413, firstTxId:59, lastTxId:59, version:VERSION_V5_SEQUENCE_ID, completionTime:1626933253327, recordCount:1, regionId:0, status:0, logSegmentSequenceNumber:1, lastEntryId:0, lastSlotId:0, inprogress:false, minActiveDLSN:DLSN{logSegmentSequenceNo=1, entryId=0, slotId=0}, startSequenceId:0] : -7
at org.apache.distributedlog.impl.logsegment.BKLogSegmentEntryStore$1.openComplete(BKLogSegmentEntryStore.java:258) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerOpenOp.openComplete(LedgerOpenOp.java:232) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerOpenOp.lambda$initiate$0(LedgerOpenOp.java:117) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_121]
at org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:396) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:575) ~[org.apache.pulsar-pulsar-zookeeper-2.4.0.jar:2.4.0]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) ~[org.apache.pulsar-pulsar-zookeeper-2.4.0.jar:2.4.0]
15:43:46.072 [io-write-scheduler-OrderedScheduler-1-0] INFO  org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal - Initializing the local state for mvcc store 000000000000000012/000000000000000001/000000000000000000
15:43:46.077 [io-write-scheduler-OrderedScheduler-1-0] INFO  org.apache.distributedlog.BKDistributedLogManager - Create sync reader starting from DLSN{logSegmentSequenceNo=1, entryId=0, slotId=0}
15:43:46.078 [main-EventThread] INFO  org.apache.distributedlog.logsegment.PerStreamLogSegmentCache - 000000000000000012/000000000000000001/000000000000000000/checkpoints/0024c9ed-75d4-455b-8994-c95b743f29c6/metadata added log segment (logrecs_000000000000000001 : [LogSegmentId:3452, firstTxId:59, lastTxId:59, version:VERSION_V5_SEQUENCE_ID, completionTime:1626933263335, recordCount:1, regionId:0, status:0, logSegmentSequenceNumber:1, lastEntryId:0, lastSlotId:0, inprogress:false, minActiveDLSN:DLSN{logSegmentSequenceNo=1, entryId=0, slotId=0}, startSequenceId:0]) to cache.
15:43:46.078 [main-EventThread] INFO  org.apache.distributedlog.logsegment.PerStreamLogSegmentCache - 000000000000000012/000000000000000001/000000000000000000/checkpoints/0024c9ed-75d4-455b-8994-c95b743f29c6/metadata added log segment (logrecs_000000000000000001 : [LogSegmentId:3452, firstTxId:59, lastTxId:59, version:VERSION_V5_SEQUENCE_ID, completionTime:1626933263335, recordCount:1, regionId:0, status:0, logSegmentSequenceNumber:1, lastEntryId:0, lastSlotId:0, inprogress:false, minActiveDLSN:DLSN{logSegmentSequenceNo=1, entryId=0, slotId=0}, startSequenceId:0]) to cache.
15:43:46.079 [main-EventThread] INFO  org.apache.distributedlog.ReadAheadEntryReader - Starting the readahead entry reader for 000000000000000012/000000000000000001/000000000000000000/checkpoints/0024c9ed-75d4-455b-8994-c95b743f29c6/metadata: : segments = [[LogSegmentId:3452, firstTxId:59, lastTxId:59, version:VERSION_V5_SEQUENCE_ID, completionTime:1626933263335, recordCount:1, regionId:0, status:0, logSegmentSequenceNumber:1, lastEntryId:0, lastSlotId:0, inprogress:false, minActiveDLSN:DLSN{logSegmentSequenceNo=1, entryId=0, slotId=0}, startSequenceId:0]]
15:43:46.079 [io-write-scheduler-OrderedScheduler-1-0] ERROR org.apache.bookkeeper.statelib.impl.rocksdb.checkpoint.RocksCheckpointer - Failed to restore rocksdb 000000000000000012/000000000000000001/000000000000000000
org.apache.distributedlog.exceptions.BKTransmitException: Failed to open ledger handle for log segment [LogSegmentId:3452, firstTxId:59, lastTxId:59, version:VERSION_V5_SEQUENCE_ID, completionTime:1626933263335, recordCount:1, regionId:0, status:0, logSegmentSequenceNumber:1, lastEntryId:0, lastSlotId:0, inprogress:false, minActiveDLSN:DLSN{logSegmentSequenceNo=1, entryId=0, slotId=0}, startSequenceId:0] : -7
at org.apache.distributedlog.impl.logsegment.BKLogSegmentEntryStore$1.openComplete(BKLogSegmentEntryStore.java:258) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerOpenOp.openComplete(LedgerOpenOp.java:232) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerOpenOp.lambda$initiate$0(LedgerOpenOp.java:117) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_121]
at org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:396) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:575) ~[org.apache.pulsar-pulsar-zookeeper-2.4.0.jar:2.4.0]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) ~[org.apache.pulsar-pulsar-zookeeper-2.4.0.jar:2.4.0]
15:43:46.080 [io-write-scheduler-OrderedScheduler-1-0] INFO  org.apache.bookkeeper.stream.storage.impl.store.MVCCStoreFactoryImpl - Clearing resources hold by stream(1)/range(0) at storage container (12)
15:43:46.080 [io-write-scheduler-OrderedScheduler-1-0] WARN  org.apache.bookkeeper.stream.storage.impl.sc.StorageContainerRegistryImpl - De-registered StorageContainer ('12') when failed to start
java.util.concurrent.CompletionException: org.apache.bookkeeper.statelib.api.exceptions.StateStoreException: Failed to restore rocksdb 000000000000000012/000000000000000001/000000000000000000
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:943) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:926) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_121]
at org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal.lambda$executeIO$16(AbstractStateStoreWithJournal.java:474) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111) [com.google.guava-guava-21.0.jar:?]
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58) [com.google.guava-guava-21.0.jar:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75) [com.google.guava-guava-21.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: org.apache.bookkeeper.statelib.api.exceptions.StateStoreException: Failed to restore rocksdb 000000000000000012/000000000000000001/000000000000000000
at org.apache.bookkeeper.statelib.impl.rocksdb.checkpoint.RocksCheckpointer.restore(RocksCheckpointer.java:84) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.kv.RocksdbKVStore.loadRocksdbFromCheckpointStore(RocksdbKVStore.java:161) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.kv.RocksdbKVStore.init(RocksdbKVStore.java:223) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal.lambda$initializeLocalStore$5(AbstractStateStoreWithJournal.java:202) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.statelib.impl.journal.AbstractStateStoreWithJournal.lambda$executeIO$16(AbstractStateStoreWithJournal.java:471) ~[org.apache.bookkeeper-statelib-4.9.2.jar:4.9.2]
... 12 more
Caused by: org.apache.distributedlog.exceptions.BKTransmitException: Failed to open ledger handle for log segment [LogSegmentId:3452, firstTxId:59, lastTxId:59, version:VERSION_V5_SEQUENCE_ID, completionTime:1626933263335, recordCount:1, regionId:0, status:0, logSegmentSequenceNumber:1, lastEntryId:0, lastSlotId:0, inprogress:false, minActiveDLSN:DLSN{logSegmentSequenceNo=1, entryId=0, slotId=0}, startSequenceId:0] : -7
at org.apache.distributedlog.impl.logsegment.BKLogSegmentEntryStore$1.openComplete(BKLogSegmentEntryStore.java:258) ~[org.apache.distributedlog-distributedlog-core-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerOpenOp.openComplete(LedgerOpenOp.java:232) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.client.LedgerOpenOp.lambda$initiate$0(LedgerOpenOp.java:117) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_121]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_121]
at org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:396) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:575) ~[org.apache.pulsar-pulsar-zookeeper-2.4.0.jar:2.4.0]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) ~[org.apache.pulsar-pulsar-zookeeper-2.4.0.jar:2.4.0]
Expected behavior
Please help to find the cause of the problem
Screenshots
Desktop (please complete the following information):

OS: [e.g. iOS]
centOS7
pulsar-2.4.0

The issue had no activity for 30 days, mark with Stale label.


text:
Java Sinks/Sources fail due to noexec on /tmp - no way to override JVM options for spawned processes

There is no way to provide additional (and configurable) -D options to java processes started by RuntimeUtils.java of pulsar-functions-runtime.
More specifically, when attempting to start a source/sink using Pulsar 2.8 / Centos7 there are Java JNI binaries that appear to extract to /tmp and execute from /tmp. Our corporate policy is to have noexec specified for /tmp and so alternative paths need to be specified using -D options. There is currently no way to pass these options to the spawned processes however.
I have tried various other methods such as setting _JAVA_OPTIONS but with no success.
Currently the only workaround I have found is to rebuild pulsar-functions-runtime with the additional options hardcoded and replacing the package in the lib folder.

Duplicate of https://github.com/apache/pulsar/issues/12051

source: gharchive/issue

authors: thesquiff tisonkun

repo: apache/pulsar

license_type: permissive

license_source: github-api


text:
[Bug] when delete namespace bundle-data in metastore was not deleted

Search before asking

[X] I searched in the issues and found nothing similar.

Version
master branch
Minimal reproduce step
just add a test case
org.apache.pulsar.broker.namespace.NamespaceServiceTest
testModularLoadManagerRemoveBundleAndLoad
 @Test
    public void testModularLoadManagerRemoveBundleAndLoad() throws Exception {
        final String BUNDLE_DATA_PATH = "/loadbalance/bundle-data";
        final String namespace = "prop/ns-abc";
        final String topic1 = "persistent://" + namespace + "/topic1";
        final String topic2 = "persistent://" + namespace + "/topic2";

        // configure broker with ModularLoadManager
        conf.setLoadManagerClassName(ModularLoadManagerImpl.class.getName());
        conf.setForceDeleteNamespaceAllowed(true);
        restartBroker();

        LoadManager loadManager = spy(pulsar.getLoadManager().get());
        Field loadManagerField = NamespaceService.class.getDeclaredField("loadManager");
        loadManagerField.setAccessible(true);
        doReturn(true).when(loadManager).isCentralized();
        SimpleResourceUnit resourceUnit = new SimpleResourceUnit(pulsar.getSafeWebServiceAddress(), null);
     //   Optional<ResourceUnit> res = Optional.of(resourceUnit);
    //    doReturn(res).when(loadManager).getLeastLoaded(any(ServiceUnitId.class));
        loadManagerField.set(pulsar.getNamespaceService(), new AtomicReference<>(loadManager));

        NamespaceName nsname = NamespaceName.get(namespace);

        @Cleanup
        PulsarClient pulsarClient = PulsarClient.builder().serviceUrl(pulsar.getBrokerServiceUrl()).build();
        @Cleanup
        Consumer<byte[]> consumer1 = pulsarClient.newConsumer().topic(topic1)
                .subscriptionName("my-subscriber-name1").subscribe();
        @Cleanup
        Consumer<byte[]> consumer2 = pulsarClient.newConsumer().topic(topic2)
                .subscriptionName("my-subscriber-name2").subscribe();

        NamespaceBundle bundle =
                pulsar.getNamespaceService().getNamespaceBundleFactory().getBundle(TopicName.get(topic1));

        loadManager.getLeastLoaded(bundle);

        //create znode for bundle-data
        pulsar.getBrokerService().updateRates();
        loadManager.writeLoadReportOnZookeeper();
        loadManager.writeResourceQuotasToZooKeeper();

        String path = BUNDLE_DATA_PATH + "/" + nsname.toString() + "/0x00000000_0xffffffff";

        Optional<GetResult> getResult = pulsar.getLocalMetadataStore().get(path).get();
        assertTrue(getResult.isPresent());

        //delete namespace which will remove bundle and load
        pulsar.getAdminClient().namespaces().deleteNamespace(nsname.toString(),true);

        TimeUnit.SECONDS.sleep(5);
        getResult = pulsar.getLocalMetadataStore().get(path).get();
        assertFalse(getResult.isPresent());

        // update broker bundle report to zk
        loadManager.writeLoadReportOnZookeeper();
        loadManager.writeResourceQuotasToZooKeeper();

        getResult = pulsar.getLocalMetadataStore().get(path).get();
        assertTrue(getResult.isPresent());

    }

What did you expect to see?
after delete namespace
   getResult = pulsar.getLocalMetadataStore().get(path).get();
        assertFalse(getResult.isPresent());

What did you see instead?
true
Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

BundleStats was not deleted ,then task will rewrite the bundle data again


text:
Not enough non-faulty bookies available when sending a message that exceeds defaultMaxMessageSize

Describe the bug
All configurations are the default configuration.
After starting, send a message that exceeds defaultMaxMessageSize, pulsar's broker is unavailable.
To Reproduce
test code:
func main() {
    client, err := pulsar.NewClient(pulsar.ClientOptions{
        URL:       "pulsar://localhost:6650",
        IOThreads: 5,
    })

    if err != nil {
        log.Fatal(err)
    }

    defer client.Close()

    producer, err := client.CreateProducer(pulsar.ProducerOptions{
        Topic:           "my-topic",
        CompressionType: pulsar.NoCompression,
        Batching:        false,
    })
    if err != nil {
        log.Fatal(err)
    }

    defer producer.Close()

    ctx := context.Background()
    buf := make([]byte, 10*1024*1024)
    arrays := [1024*1024*5]byte{}
    buf = arrays[:]
    fmt.Printf("the buf length is:[%d]\n", len(buf))

    producer.SendAsync(ctx, pulsar.ProducerMessage{
        Payload: buf,
    }, func(producerMessage pulsar.ProducerMessage, e error) {
        fmt.Printf("send complete. err=%v,\n", e)
    })
}

error as follows:
16:37:42.874 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN  org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (7213, 0): Bookie handle is not available
16:37:42.874 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:127.0.0.1:3181>], allBookies [<Bookie:127.0.0.1:3181>].
16:37:42.874 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [<Bookie:127.0.0.1:3181>], fallback to choose bookie randomly from the cluster.
16:37:42.874 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:127.0.0.1:3181>], allBookies [<Bookie:127.0.0.1:3181>].
16:37:42.874 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:127.0.0.1:3181>], allBookies [<Bookie:127.0.0.1:3181>].
16:37:42.874 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [<Bookie:127.0.0.1:3181>], fallback to choose bookie randomly from the cluster.
16:37:42.874 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:127.0.0.1:3181>], allBookies [<Bookie:127.0.0.1:3181>].
16:37:42.874 [BookKeeperClientWorker-OrderedExecutor-1-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=7213,loopId=3f741063) Exception updating
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:989) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:907) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:797) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:200) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:757) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:221) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:659) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:114) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:295) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1908) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:122) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:111) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1927) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1876) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:360) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.writeComplete(PerChannelBookieClient.java:2028) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.proto.PerChannelBookieClient$AddCompletion.lambda$errorOut$0(PerChannelBookieClient.java:2051) ~[bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.proto.PerChannelBookieClient$CompletionValue$1.safeRun(PerChannelBookieClient.java:1606) [bookkeeper-server-4.9.2.jar:4.9.2]
	at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [bookkeeper-common-4.9.2.jar:4.9.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.32.Final.jar:4.1.32.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
16:37:42.874 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN  org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (7213, 1): Bookie handle is not available
16:37:42.874 [BookKeeperClientWorker-OrderedExecutor-1-0] WARN  org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:7213, change-id:0000000001)][attempt:1] Exception changing ensemble

Expected behavior
When I send a message that exceeds the maximum message allowed by pulsar, I expect pulsar's broker to fail the message instead of causing the entire broker to be unavailable.
Screenshots

Desktop (please complete the following information):

OS: [Mac OS]
Go version: 1.11.4
Pulsar version: latest

Any updates on this? I'm facing the same issue.

@wolfstudy it seems that it is a bookkeeper client issue. Do you mind creating an issue at bookkeeper project side?

@wolfstudy it seems that it is a bookkeeper client issue. Do you mind creating an issue at bookkeeper project side?

Sure, will do this.

authors: goku321 sijie wolfstudy


text:
Tests of pulsar-functions-runtime are not passing on 2.7.0rc1

Describe the bug
Tests are not passing due to an error on Kubernetes client runtime, it created to a default configuration, failing with a NullPointerException.
To Reproduce

build pulsar from 2.7.0rc1 sources
run tests in pulsar-functions-runtime

Expected behavior
Tests are passing
Desktop (please complete the following information):

OS: Fedora
Maven 3.6.3
JDK 1.8 (OpenJDK)

Additional context
[ERROR] testResources(org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeTest)  Time elapsed: 0.011 s  <<< FAILURE!
java.lang.RuntimeException: java.lang.NullPointerException
	at org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactory.initialize(KubernetesRuntimeFactory.java:211)
	at org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeTest.createKubernetesRuntimeFactory(KubernetesRuntimeTest.java:217)
	at org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeTest.createKubernetesRuntimeFactory(KubernetesRuntimeTest.java:225)
	at org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeTest.testResources(KubernetesRuntimeTest.java:340)
	at org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeTest.testResources(KubernetesRuntimeTest.java:324)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at io.kubernetes.client.util.ClientBuilder.kubeconfig(ClientBuilder.java:259)
	at io.kubernetes.client.util.ClientBuilder.getClientBuilder(ClientBuilder.java:108)
	at io.kubernetes.client.util.ClientBuilder.standard(ClientBuilder.java:87)
	at io.kubernetes.client.util.ClientBuilder.standard(ClientBuilder.java:79)
	at io.kubernetes.client.util.Config.defaultClient(Config.java:113)
	at org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactory.setupClient(KubernetesRuntimeFactory.java:339)
	at org.apache.pul

The same error is on pulsar-functions-worker module

java.lang.RuntimeException: java.lang.NullPointerException
	at org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactory.initialize(KubernetesRuntimeFactory.java:211)
	at org.apache.pulsar.functions.worker.FunctionRuntimeManager.<init>(FunctionRuntimeManager.java:211)
	at org.apache.pulsar.functions.worker.FunctionRuntimeManagerTest.testExternallyManagedRuntimeUpdate(FunctionRuntimeManagerTest.java:674)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at io.kubernetes.client.util.ClientBuilder.kubeconfig(ClientBuilder.java:259)
	at io.kubernetes.client.util.ClientBuilder.getClientBuilder(ClientBuilder.java:108)
	at io.kubernetes.client.util.ClientBuilder.standard(ClientBuilder.java:87)
	at io.kubernetes.client.util.ClientBuilder.standard(ClientBuilder.java:79)
	at io.kubernetes.client.util.Config.defaultClient(Config.java:113)
	at org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactory.setupClient(KubernetesRuntimeFactory.java:339)
	at org.apache.pulsar.functions.runtime.kubernetes.KubernetesRuntimeFactory.initialize(KubernetesRuntimeFactory.java:208)
	... 14 more

@wolfstudy is helping on this issue.

Hello @eolivelli More info reference to: https://apache-pulsar.slack.com/archives/C5ZSVEN4E/p1606187284156200?thread_ts=1606147723.154600&cid=C5ZSVEN4E

I have resolved my local problem by starting a minikube instance
This is something we should document in some "how-to-build" document.
As we are releasing a source package the users must able to build it and all tests MUST pass, otherwise it is impossible for downstream users to check if the code is in good shape.
cc @ivankelly @merlimat @sijie @rdhabalia @jiazhai

authors: eolivelli jiazhai wolfstudy


text:
Optimize memory usage: support to  shrink for pendingAcks map

Modifications
issue: https://github.com/apache/pulsar/issues/14268
In order to optimize memory consumption, in the consumer class, the map used by the pendingAcks variable needs to support automatic shrink.
The original pendingAcks variable is defined by the ConcurrentLongLongPairHashMap class of bookkeeper. In order to be independent of the bookkeeper release version, this class is extracted into pulsar as a common class, similar to the ConcurrentLongHashMap class.
See the PR corresponding to bookkeeper: https://github.com/apache/bookkeeper/pull/3061.
Documentation
Check the box below or label this PR directly (if you have committer privilege).
Need to update docs?

[ ] doc-required
(If you need help on updating docs, create a doc issue)

[x] no-need-doc
(Please explain why)

[ ] doc
(If this PR contains doc changes)

In order to allow pendingAcks in the consumer to support the automatic shrink capability,
Here I extract the ConcurrentLongLongPairHashMap class into pulsar separately. @eolivelli @merlimat

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

@Jason918 PTAL,thanks!

@codelipenghui PTAL,thanks!

@merlimat PTAL,thanks!

@merlimat PTAL,thanks!

ping

source: gharchive/pull-request


text:
[improve][broker] Make some operation SubscribeRate methods in Namespaces async

Motivation
See PIP https://github.com/apache/pulsar/issues/14365 and change tracker https://github.com/apache/pulsar/issues/15043.
Make NamespacesBase deleteSubscribeRate / setSubscribeRate / getSubscribeRate methods to pure async.
Documentation

[x] no-need-doc

@shibd  All suggested fixed. PTAL.

@mattisonchao OK, I'll try it. There is no unit test for the v2 Namespaces yet, I will try to add it, it may take some time.

Do these non-existent files need to be generated locally？

@shibd  @mattisonchao  PTAL.

Hi @mattisonchao Can you help trigger the ci ？

Hi @Technoboy-  Can you help trigger the ci ？

Hi @Technoboy- Can you help trigger the ci ？

Yes, I have triggered.

Yes, I have triggered.

Thanks

/pulsarbot run-failure-checks

authors: Technoboy- gaoyanliang shibd


text:
[improve] [broker] PIP-240: new public method unloadSubscription in PersistentTopic

Motivation
PIP-240: A new API to unload subscriptions
Modifications

part-1: add a new method unloadSubscription( String subName ) for PersistentTopic

Follow-up work (not included in current PR)

part-2: add a new method unloadSubscription( String subName ) for NonPersistentTopic
part-3: the API unload subscription in v2/PerssitentTopics
part-4: the CMD unload subscription in CmdTopics

Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository:

https://github.com/poorbarcode/pulsar/pull/75

After discussing with @congbobo184, we feel it is too heavy to use a map to track the running unload task, I will find another way to do this.

Codecov Report

Merging #19737 (ec20578) into master (3b118b6) will increase coverage by 0.05%.
The diff coverage is 33.33%.

@@             Coverage Diff              @@
##             master   #19737      +/-   ##
============================================
+ Coverage     72.86%   72.91%   +0.05%
- Complexity    31628    31819     +191
============================================
  Files          1861     1865       +4
  Lines        137500   138083     +583
  Branches      15141    15173      +32
============================================
+ Hits         100187   100681     +494
- Misses        29351    29397      +46
- Partials       7962     8005      +43

Flag
Coverage Δ

inttests
24.16% <0.00%> (-0.23%)
:arrow_down:

systests
24.93% <0.00%> (-0.08%)
:arrow_down:

unittests
72.20% <33.33%> (+0.07%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

.../pulsar/broker/service/BrokerServiceException.java
86.84% <0.00%> (-2.25%)
:arrow_down:

...sar/broker/service/persistent/PersistentTopic.java
80.08% <40.00%> (+0.04%)
:arrow_up:

... and 158 files with indirect coverage changes

authors: codecov-commenter poorbarcode


text:
[Pulsar Client Tools]Support generate documentation of pulsar admin cli automatically

Motivation
Currently, the commands on the page http://pulsar.apache.org/docs/en/pulsar-admin/ are all manually added. when the document changes, there are often wrong contents. in order to solve this problem, we expect the commands on this page to be automatically generated. this pr is the first step. If it passes, we will add a new page on the website to show the automatically generated commands later.
Modifications

Add a hidden command --generate-doc to automatically generate command line documents
Add integration tests

Usage
./bin/pulsar-admin --generate-doc tenants
./bin/pulsar-admin --generate-doc namespaces
.....

The following is what we expect:

Verifying this change
Integration test pass

removed the milestone for now. If we can make it before releasing 2.5.0, we can mark it for 2.5.0.

Thank you for your suggestion. The situation you mentioned is my initial implementation plan. However, when I tested it, I found that different command objects jcommand was used in the PulsarAdminTool and CmdBase.
PulsarAdminTool:
https://github.com/apache/pulsar/blob/master/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/PulsarAdminTool.java#L87
CmdBase:
https://github.com/apache/pulsar/blob/master/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdBase.java#L38
All the child commands were inherited from the class CmdBase. In subclasses such as CmdTopics, we cannot get the commandMap object. Therefore, it may be necessary to pass down the commandmap object and the jcommand object of PulsarAdminTool. therefore, it is necessary to add some logic to the PulsarAdmintool. I am not sure if my understanding is correct. I hope to get your reply, but I will try this implementation as much as possible.
@sijie

I found that different command objects jcommand was used in the PulsarAdminTool and CmdBase.

The jcommand in PulsarrAdminTool is the root command.The jcommand in CmdBase is for a command group.

All the child commands were inherited from the class CmdBase. In subclasses such as CmdTopics, we cannot get the commandMap object.

You can get the command groups from the commandMap of PulsarAdminTool. Then you can iterate the command group. For each command group, you can get its sub commandMap.

run java8 tests

run integration tests

run java8 tests

@sijie PTAL

authors: sijie tuteng


text:
Add GitHub issue template for reporting flaky tests

Motivation
It was suggested on the mailing list by @sijie to add a GitHub issue template for reporting flaky tests.
By having an issue template for flaky tests, it's easier to report flaky tests consistently.

/pulsar-bot run-failure-tests

/pulsar-bot run-failure-tests

Failed test:
Error:  Tests run: 21, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 108.4 s <<< FAILURE! - in org.apache.pulsar.broker.service.ReplicatorTest
Error:  testResetCursorNotFail(org.apache.pulsar.broker.service.ReplicatorTest)  Time elapsed: 30.209 s  <<< FAILURE!
org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.broker.service.ReplicatorTest.testResetCursorNotFail() didn't finish within the time-out 30000
	at org.testng.internal.MethodInvocationHelper.invokeWithTimeoutWithNewExecutor(MethodInvocationHelper.java:371)
	at org.testng.internal.MethodInvocationHelper.invokeWithTimeout(MethodInvocationHelper.java:282)
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:605)
	at org.testng.internal.TestInvoker.retryFailed(TestInvoker.java:214)
	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:58)
	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.testng.TestRunner.privateRun(TestRunner.java:764)
	at org.testng.TestRunner.run(TestRunner.java:585)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
	at org.testng.SuiteRunner.run(SuiteRunner.java:286)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
	at org.testng.TestNG.runSuites(TestNG.java:1069)
	at org.testng.TestNG.run(TestNG.java:1037)
	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99)
	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

[INFO]
[INFO] Results:
[INFO]
Error:  Failures:
Error:  org.apache.pulsar.broker.service.ReplicatorTest.testResetCursorNotFail(org.apache.pulsar.broker.service.ReplicatorTest)
[INFO]   Run 1: PASS
Error:    Run 2: ReplicatorTest.testResetCursorNotFail ? ThreadTimeout Method org.apache.pulsar...
[INFO]
[INFO]
Error:  Tests run: 20, Failures: 1, Errors: 0, Skipped: 0

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

authors: lhotari zymap


text:
Golang: Make each 'Client' have an independent 'ClientManager'

Currently a process globally shares a 'ClientManager', so only one 'Conn' is established for each endpoint. When users create a new client, their expectations should be completely independent.

@guyinyou https://github.com/apache/rocketmq-clients/pull/235 could you reopen this pr？

@guyinyou #235 could you reopen this pr？

@aaron-ai

authors: guyinyou leizhiyuan

repo: apache/rocketmq-clients


text:
请问如何联系commiter合并代码

Anybody merge the code

pls leave a PR id

pls leave a PR id

#3085,thank you

authors: francisoliverlee yglovewms

repo: apache/rocketmq


text:
[Bug] Exception when pop messages with multiple LMQ indexes

Before Creating the Bug Report

[X] I found a bug, not just asking a question, which should be created in GitHub Discussions.

[X] I have searched the GitHub Issues and GitHub Discussions  of this repository and believe that this is not a duplicate.

[X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

Runtime platform environment
all
RocketMQ version
develop
JDK Version
No response
Describe the Bug
NumberFormatException is thrown when pop messages with multiple LMQ indexes.

Steps to Reproduce
/
What Did You Expect to See?
/
What Did You See Instead?
/
Additional Context
No response

The property PROPERTY_INNER_MULTI_DISPATCH and PROPERTY_INNER_MULTI_QUEUE_OFFSET may be strings containing multiple elements splitted by comma.
eg:
PROPERTY_INNER_MULTI_DISPATCH="%LMQ%lmq1,%LMQ%lmq2,%LMQ%lmq3"
PROPERTY_INNER_MULTI_QUEUE_OFFSET="1,2,3"

You can submit a pull request to fix this issue.

authors: leizhiyuan redlsz


text:
[ROCKETMQ-207]Correct spelling mistake in RouteInfoManager.java

There is several spelling mistakes, it's a little confusing and i think we should fix them,like this one :
if (null == prevBrokerLiveInfo) {
                   log.info("new broker registerd, {} HAServer: {}", brokerAddr, haServerAddr);
}

and here is the JIRA link of this issue : https://issues.apache.org/jira/browse/ROCKETMQ-207 ,thanks for your time

Coverage decreased (-0.03%) to 38.636% when pulling bfa7f1ac8bfac3334e6b307db4aabc66ad22f5e9 on aCoder2013:feature/correct-spelling-mistake into 0fe947173a85d8931b1068805713e89dbba4125a on apache:develop.

Coverage decreased (-0.03%) to 38.636% when pulling bfa7f1ac8bfac3334e6b307db4aabc66ad22f5e9 on aCoder2013:feature/correct-spelling-mistake into 0fe947173a85d8931b1068805713e89dbba4125a on apache:develop.

Coverage decreased (-0.03%) to 38.636% when pulling bfa7f1ac8bfac3334e6b307db4aabc66ad22f5e9 on aCoder2013:feature/correct-spelling-mistake into 0fe947173a85d8931b1068805713e89dbba4125a on apache:develop.

Thank you! Please relate JIRA  link in the discriptions.
Also, there are multiple typo string literal in rocektmq code base , it's welcome to fix them in this pr too.

Coverage decreased (-0.03%) to 38.632% when pulling 9422bdb38ba8871b87c4b5172f1dc08b207d901b on aCoder2013:feature/correct-spelling-mistake into 0fe947173a85d8931b1068805713e89dbba4125a on apache:develop.

Coverage decreased (-0.03%) to 38.632% when pulling 9422bdb38ba8871b87c4b5172f1dc08b207d901b on aCoder2013:feature/correct-spelling-mistake into 0fe947173a85d8931b1068805713e89dbba4125a on apache:develop.

Coverage decreased (-0.03%) to 38.632% when pulling 9422bdb38ba8871b87c4b5172f1dc08b207d901b on aCoder2013:feature/correct-spelling-mistake into 0fe947173a85d8931b1068805713e89dbba4125a on apache:develop.

LGTM @zhouxinyu @vongosling please help a review

Coverage increased (+0.1%) to 38.819% when pulling 7b256ebeb48c99494f1c1d8eddd7b58ea2039992 on aCoder2013:feature/correct-spelling-mistake into 845830865fc37d0364a19cbd89ceaf8a30b37e1c on apache:develop.

@zhouxinyu I've fixed the code style , please check it out ,thanks

Changes Unknown when pulling eabe47fd600c6e1b9f36023943d0491a5f920161 on aCoder2013:feature/correct-spelling-mistake into ** on apache:develop**.

Changes Unknown when pulling eabe47fd600c6e1b9f36023943d0491a5f920161 on aCoder2013:feature/correct-spelling-mistake into ** on apache:develop**.

authors: Jaskey aCoder2013 coveralls dongeforever


text:
Try to make CI test more stable

Make sure set the target branch to develop
What is the purpose of the change
XXXXX
Brief changelog
XX
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage increased (+0.05%) to 47.636% when pulling ec583336975fe1751db1d5f864dc692b419b2325 on RongtongJin:develop-try into 00da3e7249d2339717d96f7ea2ab8e4a2fd4242c on apache:develop.

authors: RongtongJin coveralls


text:
add message property PROPERTY_BORN_TIMESTAMP

add bornTimeStamp property to message when calling MessagingProcessor::sendMessage

Coverage decreased (-0.1%) to 48.035% when pulling 8b609e4cf51998fae3a8cc8ff80794cdfc036a4a on ingdex:develop into f4fd55b92164cda18aa99d108f87ef0f643270e2 on apache:develop.

Codecov Report

Merging #4643 (8b609e4) into develop (f4fd55b) will increase coverage by 0.02%.
The diff coverage is 66.66%.

@@              Coverage Diff              @@
##             develop    #4643      +/-   ##
=============================================
+ Coverage      44.04%   44.07%   +0.02%
- Complexity      6934     6938       +4
=============================================
  Files            923      923
  Lines          63012    63015       +3
  Branches        8288     8289       +1
=============================================
+ Hits           27755    27775      +20
+ Misses         31796    31782      -14
+ Partials        3461     3458       -3

Impacted Files
Coverage Δ

...g/apache/rocketmq/common/message/MessageConst.java
97.56% <ø> (ø)

...mq/proxy/grpc/v2/producer/SendMessageActivity.java
77.88% <66.66%> (-0.18%)
:arrow_down:

...nt/impl/consumer/ConsumeMessageOrderlyService.java
49.47% <0.00%> (-0.71%)
:arrow_down:

...apache/rocketmq/store/queue/BatchConsumeQueue.java
69.50% <0.00%> (-0.63%)
:arrow_down:

...ent/impl/consumer/DefaultLitePullConsumerImpl.java
69.81% <0.00%> (-0.51%)
:arrow_down:

...lient/impl/consumer/DefaultMQPushConsumerImpl.java
35.01% <0.00%> (-0.39%)
:arrow_down:

...org/apache/rocketmq/store/DefaultMessageStore.java
52.69% <0.00%> (-0.25%)
:arrow_down:

...cketmq/broker/processor/PopBufferMergeService.java
40.47% <0.00%> (-0.24%)
:arrow_down:

...rocketmq/client/impl/factory/MQClientInstance.java
47.30% <0.00%> (-0.14%)
:arrow_down:

...main/java/org/apache/rocketmq/store/CommitLog.java
68.46% <0.00%> (+0.09%)
:arrow_up:

... and 10 more

authors: codecov-commenter coveralls ingdex


text:
SEP-19 : Basic Standby task/container implementation

This PR brings a basic standby container implementation, that uses Standby tasks using
SideInputs

Closing this in lieu of https://github.com/apache/samza/pull/912 and a follow on to that.

repo: apache/samza

license_source: bigquery


text:
[Feature][Zeta engine] Added the metrics information of table statistics in multi-table mode

…ics in multi-table mode #6959

Purpose of this pull request

Solve the problem of historical submission confusion  #7088
Does this PR introduce any user-facing change?

How was this patch tested?

Check list

[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update change log that in connector document. For more details you can refer to connector-v2
Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist

[ ] Update the release-note.

@Hisoka-X

please resolve conflicts

repo: apache/seatunnel


text:
[Bug]  No connection established

Search before asking

[X] I had searched in the issues and found no similar issues.

Apache SkyWalking Component
NodeJS Server Side Agent (apache/skywalking-nodejs)
What happened
{
file: '/Volumes/development/projects/app_pc_pduiv2/node_modules/.pnpm/skywalking-backend-js@0.6.0/node_modules/skywalking-backend-js/lib/agent/protocol/grpc/clients/HeartbeatClient.js',
code: 14,
details: 'No connection established',
metadata: Metadata { internalRepr: Map(0) {}, options: {} },
level: 'error',
message: 'Failed to send heartbeat 14 UNAVAILABLE: No connection established',
stack: 'Error: 14 UNAVAILABLE: No connection established\n' +
'    at Object.callErrorFromStatus (/Volumes/development/projects/app_pc_pduiv2/node_modules/.pnpm/@grpc+grpc-js@1.7.3/node_modules/@grpc/grpc-js/build/src/call.js:32:19)\n' +
'    at Object.onReceiveStatus (/Volumes/development/projects/app_pc_pduiv2/node_modules/.pnpm/@grpc+grpc-js@1.7.3/node_modules/@grpc/grpc-js/build/src/client.js:195:52)\n' +
'    at Object.onReceiveStatus (/Volumes/development/projects/app_pc_pduiv2/node_modules/.pnpm/@grpc+grpc-js@1.7.3/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:365:141)\n' +
'    at Object.onReceiveStatus (/Volumes/development/projects/app_pc_pduiv2/node_modules/.pnpm/@grpc+grpc-js@1.7.3/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:328:181)\n' +
'    at /Volumes/development/projects/app_pc_pduiv2/node_modules/.pnpm/@grpc+grpc-js@1.7.3/node_modules/@grpc/grpc-js/build/src/call-stream.js:188:78\n' +
'    at processTicksAndRejections (node:internal/process/task_queues:78:11)\n' +
'for call at\n' +
'    at ServiceClientImpl.makeUnaryRequest (/Volumes/development/projects/app_pc_pduiv2/node_modules/.pnpm/@grpc+grpc-js@1.7.3/node_modules/@grpc/grpc-js/build/src/client.js:163:34)\n' +
'    at ServiceClientImpl.reportInstanceProperties (/Volumes/development/projects/app_pc_pduiv2/node_modules/.pnpm/@grpc+grpc-js@1.7.3/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)\n' +
'    at Timeout._onTimeout (/Volumes/development/projects/app_pc_pduiv2/node_modules/.pnpm/skywalking-backend-js@0.6.0/node_modules/skywalking-backend-js/lib/agent/protocol/grpc/clients/HeartbeatClient.js:63:43)\n' +
'    at listOnTimeout (node:internal/timers:557:17)\n' +
'    at processTimers (node:internal/timers:500:7)'
}
What you expected to happen
No error
How to reproduce
start agent
Anything else
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

This is your local env issue. Check the npm logs.

authors: code-farmer-i wu-sheng

repo: apache/skywalking


text:
Can the page of  "Span Info" be expanded manually?

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
[ ] Bug
[ ] Requirement
[x] Feature or performance improvement

Question

What do you want to know?

Bug

Which version of SkyWalking, OS and JRE?

Which company or project?

What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.

Requirement or improvement

Please describe about your requirements or improvement suggestions.

I wanna check for the exceptions of the "Span Info",  I can't see the full information  because the page so small, is it possible to expand manually?

This is good improvement. Contributions are welcome.

Can you describe the requirements in detail?If you can, please take a screenshot, thank you!

ok, I will finish.

I think manual extension is not convenient. It is better to add a button to pop up a dialog box large enough. What do you think?
@zhaoxunyong

I think manual extension is not convenient. It is better to add a button to pop up a dialog box large enough. What do you think?
@zhaoxunyong

or  Or add a button that automatically changes the box to the size of the content and retracts to its original size.

@x22x22 I remembered the previous version has a right arrow on the top right corner, I can expand it manually.

I think manual extension is not convenient. It is better to add a button to pop up a dialog box large enough. What do you think?
@zhaoxunyong

I think manual extension is not convenient. It is better to add a button to pop up a dialog box large enough. What do you think?
@zhaoxunyong

I remembered the previous version has a right arrow on the top right corner, I can expand it manually.

I remembered the previous version has a right arrow on the top right corner, I can expand it manually.

I think manual extension is not convenient. It is better to add a button to pop up a dialog box large enough. What do you think?
@zhaoxunyong

Or add a button that automatically changes the box to the size of the content and retracts to its original size.

It's good idea~

@zhaoxunyong see https://github.com/apache/skywalking-rocketbot-ui/pull/152

@zhaoxunyong see apache/skywalking-rocketbot-ui#152

Great, you're so efficient!

I just tag this as the last feature PR in 6.4.0, if this could get approve by @TinyAllen

authors: wu-sheng x22x22 zhaoxunyong


text:
[Rover] license issue

Search before asking

[X] I had searched in the issues and found no similar issues.

Apache SkyWalking Component
OAP server (apache/skywalking)
What happened
I happened to find the license of the source file protocol_analyze.h is GPL-2.0 but we declare it as Apache 2.0, which is not allowed as per ASF’s policy.
we should fix this as soon as possible before the next release.
What you expected to happen
No license conflict
How to reproduce
No need to reproduce.
Anything else
Note that the GPL files are used in pixie and pixie being Apache2.0 licensed doesn’t mean the GPL files become Apache2.0
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

@mrproliu As a protocol analyzer, I think it should be very easy to rebuild one, and we just need HTTP 1.1/2 identifying.

Another thing I found is that this file is not a simple copy of the original file. It adds some other content from other file but doesn't mention the changes and the other files. I'm not sure whether there are other similar cases or not. Please overhaul the files that are copied from other repos.


text:
Add changelog for 6.1

Because we were preparing graduation, 6.1 comes a little late :) But we are very close to it. Here is a very long changelog for SkyWalking 6.1, and no incubating anymore.
6.1.0
Project
SkyWalking graduated as Apache Top Level Project.

Support compiling project agent, backend, UI separately.

Java Agent

Support Vert.x Core 3.x plugin.
Support Apache Dubbo plugin.
Support use_qualified_name_as_endpoint_name and use_qualified_name_as_operation_name configs in SpringMVC plugin.
Support span async close APIs in core. Used in Vert.x plugin.
Support MySQL 5,8 plugins.
Support set instance id manually(optional).
Support customize enhance trace plugin in optional list.
Support to set peer in Entry Span.
Support Zookeeper plugin.
Fix Webflux plugin created unexpected Entry Span.
Fix Kafka plugin NPE in Kafka 1.1+
Fix wrong operation name in postgre 8.x plugin.
Fix RabbitMQ plugin NPE.
Fix agent can't run in JVM 6/7, remove module-info.class.
Fix agent can't work well, if there is whitespace in agent path.
Fix Spring annotation bug and inheritance enhance issue.
Fix CPU accessor bug.

Backend
Performance improved, especially in CPU limited environment. 3x improvement in service mesh scenario(no trace) in 8C16G VM.
Significantly cost less CPU in low payload.

Support database metric and SLOW SQL detection.
Support to set max size of metadata query. And change default to 5000 from 100.
Support ElasticSearch template for new feature in the future.
Support shutdown Zipkin trace analysis, because it doesn't fit production environment.
Support log type, scope HTTP_ACCESS_LOG and query. No feature provided, prepare for future  versions.
Support .NET clr receiver.
Support Jaeger trace format, no analysis.
Support group endpoint name by regax rules in mesh receiver.
Support diable statement in OAL.
Support basic auth in ElasticSearch connection.
Support metric exporter module and gRPC implementor.
Support >, <, >=, <= in OAL.
Support role mode in backend.
Support Envoy metric.
Support query segment by service instance.
Support to set host/port manually at cluster coordinator, rather than based on core settings.
Make sure OAP shutdown when it faces startup error.
Support set separated gRPC/Jetty ip:port for receiver, default still use core settings.
Fix JVM receiver bug.
Fix wrong dest service in mesh analysis.
Refactor ScopeDeclaration annotation.
Refactor register lock mechanism.
Add SmartSql component for .NET
Add integration tests for ElasticSearch client.
Add test cases for exporter.
Add test cases for queue consume.

UI

RocketBot UI has been accepted and bind in this release.
Support CLR metric.

Document

Documents updated, matching Top Level Project requirement.
UI licenses updated, according to RocketBot UI IP clearance.
User wall and powered-by list updated.
CN documents removed, only consider to provide by volunteer out of Apache.

All issues and pull requests are here

Coverage remained the same at 16.009% when pulling 1525c05d642c29801ab0bf875ba315899ac403d1 on changelog into 660943a23e01e8b58c3c8333518cc022589fe983 on master.

authors: coveralls wu-sheng


text:
Fix wrong logger names

Please answer these questions before submitting pull request

Why submit this pull request?
[x] Bug fix
[ ] New feature provided
[ ] Improve performance

Logger name differentiate from the real class may be troublesome when debugging, and we have several pull requests that try to correct some of them (#3054), but not all of them.
So I just wrote a simple shell script to find out all such cases and fix them once and for all.
Since we are using lombok, to avoid such problem in the future, I'd recommend using @Slf4j annotation in future PRs when possible.

Since we are using lombok, to avoid such problem in the future, I'd recommend using @slf4j annotation in future PRs when possible.

@kezhenxu94 You could change this in OAP side, I am just saying, the check may be still necessary.

Are we going to merge this first?

Are we going to merge this first?

I've just fixed those in agent side, and can be merged first now.
As for the check in CI, let me work on it in another PR

/run e2e

/run e2e


text:
Ephermeral storage should extend to ZK instance

When you specify the ephemeral storage type for a solrcloud, that should also extend to the Zookeeper instance that the solr-operator manages (specified like this).
(We noted this when the ZK pod did not start on AWS Fargate because it was trying to use a persistent volume claim, so the SolrCloud pod did not start either despite using emptyDir.)

Agreed. This is a good feature that should be included in v0.4.0.
We should expose all options for the Zookeeper storage, but default to ephemeral/persistent depending on what the SolrCloud uses.

We should expose all options for the Zookeeper storage, but default to ephemeral/persistent depending on what the SolrCloud uses.

This is important, in our use-case we probably will have solr data on ephemeral but ZK on persistent.

authors: HoustonPutman jward-bw mogul

repo: apache/solr-operator


text:
Expose k8s topology labels to Solr PODs

We support topologySpreadConstraint in #350, so Solr PODs are spread across AZs. But for Solr to be able to place replicas across AZs with the Affinity placement plugin, Solr needs access to node label topology.kubernetes.io/zone as a java sysprop availability_zone.
This is non-trivial, as node labels are not exposed to PODs, and you'd need special permissions to fetch it from ApiServer. Instead of each user going through hoops of setting up Init containers to pull in this info, let's have the operator handle it and inject topology.kubernetes.io/zone as a system property into Solr.
Elastic has done a similar thing, referenced in https://the-asf.slack.com/archives/C01JR8WE1M5/p1669108868939589?thread_ts=1668633426.926589&cid=C01JR8WE1M5

Hey Jan, did you see this ancient PR https://github.com/apache/solr-operator/pull/90? Its basically hacking the node labels in as sysprops. But yes, you need node permissions which is very unfortunate. I really wish Kubernetes solved this via the Downwards API years ago.
Given SIP-18 what do you think about a plugin that allows loading "sysProp" like key-values from external sources, like kubernetes? Getting away from hacky initContainers is the best path for us. You can see how we add new kube-friendly features to Solr, but make those features available in earlier versions through the operator doing some hacky stuff. And eventually the minimum Solr version will let us get rid of those hacks.
So I can definitely see a hacky initContainer doing the work for us here, but eventually it would be great to just use the KubePropertiesPlugin (just an example), and not have to have a complex setup process.
Eventually maybe https://github.com/kubernetes/kubernetes/issues/40610 will make it in and this gets even easier for us...

Thanks for the link, did not find it.
While a KubePropertiesPlugin is nice long term for several things, I think it makes sense to start with the more targeted feature of propagating zone info as sys prop.
Re the configuring of placement plugins, would make sense if /clusterprops.json could also be read from / owned by a CM, then it would be simple to provide such config without touching ZK.

Cycling back to this, I had a look at #90 to see if it is something I could incorporate into my running clusters with plain helm templating. I'm sure I could extract each part from the go code into scripts and initcontainer yaml to make it work until it is a feature of the operator. Do you forsesee a variant of #90 every being merged?

It could be, though I absolutely hate that it is necessary. Let's not do it for v0.8.0, because we just need to fix TLS and get it out. But let's target it for the next version. I'll ping you, so that we can work on it together.

I have most of the moving parts incorporated in a private helm chart, except some access permission trouble for the sa token, almost there. Please ping me after 0.8 release and I'm happy to discuss and review/test the feature. If we at the same time could make it much simpler to configure the affinity (or other) placement plugin it would be a nice combo.

So, for information, here are the fragments I ended up with in my setup to do this manually: https://gist.github.com/janhoy/dfc24bb128bd5a44a114a10d266b9e56
I've just cut out the relevant parts from a helm template command, not sure if this will apply, but people could copy/paste into their chart or yaml. This is particularly done to get hold of topology.kubernetes.io/zone, don't care about other labels.

authors: HoustonPutman janhoy


text:
[SPARK-13597][PySpark][ML] Python API for GeneralizedLinearRegression

What changes were proposed in this pull request?
Python API for GeneralizedLinearRegression
JIRA: https://issues.apache.org/jira/browse/SPARK-13597
How was this patch tested?
The patch is tested with Python doctest.

Test build #52317 has started for PR 11468 at commit af5f7eb.

cc @mengxr @yanboliang

Test build #52317 has finished for PR 11468 at commit af5f7eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class GeneralizedLinearRegression(JavaEstimator, HasLabelCol, HasFeaturesCol, HasPredictionCol,
class GeneralizedLinearRegressionModel(JavaModel):

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52317/
Test PASSed.

Ping!  Is this still active?  It will be great to get into 2.0

@vectorijk Do you have time to update this PR? If not, I can help.

Sorry about late response. Yes, i will catch this today.
On Fri, Apr 1, 2016, 09:02 Yanbo Liang notifications@github.com wrote:

@vectorijk https://github.com/vectorijk Do you have time to update this
PR? If not, I can help.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/11468#issuecomment-204450309

Test build #54974 has started for PR 11468 at commit 822c844.

Test build #54974 has finished for PR 11468 at commit 822c844.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class GeneralizedLinearRegressionModel(JavaModel, JavaMLWritable, JavaMLReadable):

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54974/
Test FAILed.

Merged build finished. Test FAILed.

Jenkins, retest this please.

Test build #54977 has started for PR 11468 at commit 822c844.

Test build #54977 has finished for PR 11468 at commit 822c844.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class GeneralizedLinearRegressionModel(JavaModel, JavaMLWritable, JavaMLReadable):

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54977/
Test PASSed.

@vectorijk This PR looks good overall, please address my last comments. Thanks!

Test build #55402 has started for PR 11468 at commit f60e48c.

Test build #55402 has finished for PR 11468 at commit f60e48c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55402/
Test PASSed.

@yanboliang Thanks! I have addressed your comments.

LGTM except the last minor issue.

Test build #55534 has started for PR 11468 at commit 4199c93.

Test build #55534 has finished for PR 11468 at commit 4199c93.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55534/
Test PASSed.

@jkbradley I have addressed all the comments. Could you review this again?

Merged into master. Thanks!

authors: AmplabJenkins SparkQA jkbradley mengxr vectorijk yanboliang

repo: apache/spark


text:
[SPARK-15449][MLlib][Example]:Wrong Data Format - Documentation Issue

What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
In the MLLib naivebayes example, scala and python example doesn't use libsvm data, but Java does.
I make changes in scala and python example to use the libsvm data as the same as Java example.
How was this patch tested?
Manual tests

Test build #59291 has started for PR 13301 at commit fa3656e.

Test build #59291 has finished for PR 13301 at commit fa3656e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59291/
Test PASSed.

Test build #59292 has started for PR 13301 at commit 5bf30dd.

Test build #59292 has finished for PR 13301 at commit 5bf30dd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59292/
Test PASSed.

Hm, I guess I don't see why this should use the libsvm sample? 2 of the 3 examples used the naive bayes sample, and it's a naive bayes example. Shouldn't Java be fixed?

@srowen ML uses the sample_libsvm_data.txt in all three examples.
sample_naive_bayes_data.txt  is not in libsvm format. The format is shown below:
0,1 0 0
0,2 0 0
0,3 0 0
0,4 0 0
1,0 1 0
1,0 2 0
1,0 3 0
1,0 4 0
2,0 0 1
2,0 0 2
2,0 0 3
2,0 0 4

Test build #59297 has started for PR 13301 at commit 16c6f6c.

Test build #59297 has finished for PR 13301 at commit 16c6f6c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59297/
Test PASSed.

OK got it. Now sample_naive_bayes_data.txt is no longer used, I think. I think it can be deleted?

@srowen I think we can delete it. Let me double check it and update this PR. Thanks!

I grep all scala/java/py files and there is no reference to the data file.

Test build #59300 has started for PR 13301 at commit 2e15ec9.

Test build #59300 has finished for PR 13301 at commit 2e15ec9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59300/
Test PASSed.

Minor comment otherwise LGTM

@MLnick I am on travel now. I will update it on Saturday. Thanks!

@MLnick Done. removed seed in python and java

Test build #59507 has started for PR 13301 at commit 7d3b5b0.

LGTM pending tests

Test build #59507 has finished for PR 13301 at commit 7d3b5b0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59507/
Test PASSed.

Merged to master/2.0

authors: AmplabJenkins MLnick SparkQA srowen wangmiao1981


text:
[BUILD] Close stale PRs

Closes #18522
Closes #17722
Closes #18879
Closes #18891
Closes #18806
Closes #18948
Closes #18949
Closes #19070
Closes #19039
Closes #19142
Closes #18515
Closes #19154
Closes #19162
Closes #19187

Test build #81675 has started for PR 19203 at commit 6386e0c.

@srowen, it looks 19091 is missed. The rest of mine is a subset of the current list.

Test build #81675 has finished for PR 19203 at commit 6386e0c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81675/
Test PASSed.

authors: AmplabJenkins HyukjinKwon SparkQA srowen


text:
[SPARK-22462][SQL] Make rdd-based actions in Dataset trackable in SQL UI

What changes were proposed in this pull request?
For the few Dataset actions such as foreach, currently no SQL metrics are visible in the SQL tab of SparkUI. It is because it binds wrongly to Dataset's QueryExecution. As the actions directly evaluate on the RDD which has individual QueryExecution, to show correct SQL metrics on UI, we should bind to RDD's QueryExecution.
How was this patch tested?
Manually test. Screenshot is attached in the PR.

The screenshot for running sql("select * from range(10)").foreach(a => Unit) on spark-shell:

Test build #83581 has started for PR 19689 at commit ac539cd.

Test build #83581 has finished for PR 19689 at commit ac539cd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83581/
Test FAILed.

Test build #83584 has started for PR 19689 at commit f0c6399.

Test build #83584 has finished for PR 19689 at commit f0c6399.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83584/
Test FAILed.

Merged build finished. Test FAILed.

retest this please.

Test build #83589 has started for PR 19689 at commit f0c6399.

Test build #83591 has started for PR 19689 at commit f0c6399.

Test build #83589 has finished for PR 19689 at commit f0c6399.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83589/
Test PASSed.

Test build #83591 has finished for PR 19689 at commit f0c6399.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83591/
Test PASSed.

cc @juliuszsompolski

Thanks for the fix @viirya! But I'm not a Spark committer to approve it :-).

@juliuszsompolski No problem. Non-committer can still review. :)

cc @cloud-fan for review too.

and also I believe anyone can leave the sign-off too if it looks good :).

Test build #83688 has started for PR 19689 at commit 1b1699a.

LGTM

Test build #83688 has finished for PR 19689 at commit 1b1699a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83688/
Test PASSed.

Merged build finished. Test PASSed.

Test build #83711 has started for PR 19689 at commit 20dc217.

Test build #83711 has finished for PR 19689 at commit 20dc217.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83711/
Test PASSed.

thanks, merging to master!

authors: AmplabJenkins HyukjinKwon SparkQA cloud-fan juliuszsompolski viirya


text:
[SPARK-21322][SQL][followup] support histogram in filter cardinality estimation

What changes were proposed in this pull request?
some code cleanup/refactor and naming improvement.
How was this patch tested?
existing tests

cc @ron8hu @wzhfy

Test build #84773 has started for PR 19952 at commit ebcd6d1.

Test build #84773 has finished for PR 19952 at commit ebcd6d1.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84773/
Test FAILed.

Test build #84784 has started for PR 19952 at commit ebcd6d1.

Test build #84784 has finished for PR 19952 at commit ebcd6d1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84784/
Test PASSed.

Test build #84822 has started for PR 19952 at commit 8fe0c49.

Test build #84827 has started for PR 19952 at commit 4e35c43.

Test build #84822 has finished for PR 19952 at commit 8fe0c49.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84822/
Test PASSed.

Merged build finished. Test PASSed.

Test build #84827 has finished for PR 19952 at commit 4e35c43.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84827/
Test PASSed.

Merged build finished. Test PASSed.

thanks, merging to master!

authors: AmplabJenkins SparkQA cloud-fan


text:
[SPARK-22844][R] Adds date_trunc in R API

What changes were proposed in this pull request?
This PR adds date_trunc in R API as below:
> df <- createDataFrame(list(list(a = as.POSIXlt("2012-12-13 12:34:00"))))
> head(select(df, date_trunc("hour", df$a)))
  date_trunc(hour, a)
1 2012-12-13 12:00:00

How was this patch tested?
Unit tests added in R/pkg/tests/fulltests/test_sparkSQL.R.

Test build #85175 has started for PR 20031 at commit c54abe9.

cc @felixcheung, could you take a look when you have some time?

Test build #85177 has started for PR 20031 at commit 1c3e956.

Test build #85175 has finished for PR 20031 at commit c54abe9.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85175/
Test FAILed.

Test build #85177 has finished for PR 20031 at commit 1c3e956.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85177/
Test PASSed.

Test build #85308 has started for PR 20031 at commit 6aa835e.

Test build #85308 has finished for PR 20031 at commit 6aa835e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85308/
Test PASSed.

Thanks for review and approval, @dongjoon-hyun and @felixcheung.

Merged to master.

authors: AmplabJenkins HyukjinKwon SparkQA


text:
[SPARK-17967][SQL] Support for array as an option in SQL parser

What changes were proposed in this pull request?
This PR targets to add the ability for dealing with an array (JSON array) in tablePropertyValue rule.
SQL
CREATE TEMPORARY TABLE tableA USING csv
OPTIONS (nullValue [2012, 1.1, 'null'], ...)

How was this patch tested?
Manually tested and test cases added in DDLParserSuite.scala.

cc @gatorsmile could you take a look please?

Test build #85555 has started for PR 20125 at commit 5cae64b.

Is this a special feature for SparkSQL only? Seems Hive doesn't have such support.

Yup, I was thinking of SparkSQL only feature.
For more details, the original intention was to support multiple values for nullValue but I realised such option support can be generallised - there were several issues about this since CSV is thirdparty library (I will find and give some links if requested). Also, there is one reference in R too:
> d <- "col1,col2
+ 1,3
+ 2,4"
> df <- read.csv(text=d, na.strings=c("3", "2"))
> df

  col1 col2
1    1   NA
2   NA    4

For more context, original proposal (Scala/SQL/Python/Java) here - https://github.com/apache/spark/pull/16611 touched many files and I received an advice to make this smaller, which I liked.

Btw, is this any difference than using string? Like:

Nope, they will be the same but I was thinking this is a simplest fix.

I actually think https://github.com/apache/spark/pull/20125#issuecomment-354604768 are good points and I was hesitant about it. Although IMHO I think it's fine but let me cc @hvanhovell and @rxin too, who reviewed my related PRs before.

Test build #85555 has finished for PR 20125 at commit 5cae64b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85555/
Test PASSed.

Merged build finished. Test PASSed.

I am sorry it's been inactive. Let me update this one within a week.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6299/
Test PASSed.

Two general questions:

Any behavior change in the parser?
What is the corresponding interface for DataFrameReader and DataFrameWriter APIs?

``

authors: AmplabJenkins HyukjinKwon SparkQA gatorsmile viirya


text:
[SPARK-23782][CORE][UI] SHS should list only applications with read permissions

What changes were proposed in this pull request?
Before the PR, all applications were returned to all users.
The PR, instead, changes the behavior, so that, If an authentication method is specified, the remote user is used to check the read permissions of the applications and only the applications the user can read are returned.
How was this patch tested?
added UT

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1726/
Test PASSed.

Test build #88545 has started for PR 20891 at commit bc87945.

Test build #88545 has finished for PR 20891 at commit bc87945.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88545/
Test FAILed.

Test build #88560 has started for PR 20891 at commit 126e6a8.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1735/
Test PASSed.

Build finished. Test PASSed.

Test build #88561 has started for PR 20891 at commit cefd672.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1736/
Test PASSed.

Test build #88561 has finished for PR 20891 at commit cefd672.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #88560 has finished for PR 20891 at commit 126e6a8.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88560/
Test FAILed.

I think I fixed this issue before, but the community rejected such proposal. They thought that listing all the applications for any user is OK without checking the permission.

jerry is correct, the community rejected a similar pr in the past

@jerryshao @ajbozarth also @vanzin told me the same in the JIRA. Honestly, I think that it was an error to reject it in the past.
There is no other system which allows users without read permissions to list what other users are doing. You can check the behavior of any DB for instance (Postgres, Oracle, ...).
In this way, we are showing to all users which other users are on the system, when they are running applications and so on. Some example of information which users can know in this way, while they shouldn't:

the name of the other users on the system (if I am a company and I have 2 consultant companies working on the same cluster, I might not want each company to know that the other is working there too);
if the name of the applications are explicit, I can understand what another user is doing on the cluster, even though I do not have read permissions for their applications; again I might know that a competitor company is working on that cluster for doing a specific work....
non-admin users can see how many and which users are currently using the cluster.

All these things should not be disclosed to non-admin users. I think this is especially critical in situation where a company has a cluster but many other consulting companies are working on it. The one who owns the cluster is likely not willing to disclose to its consultant which other consultant are there and what they are doing. In this way, we are letting them know.
Moreover, it is semantically wrong. You are saying that a user has no read permission for an application, but actually it can see that the application exists, its name, duration, submitting user, and so on. So you can see some details for something you have no read access to.

There is no other system which allows users without read permissions to list what other users are doing.

I just ran ps on my machine and saw everything the root user was doing.

@mgaido91 I really think it's wrong to try to draw a parallel to something like Oracle. Oracle is completely unlike Spark - it's a self-contained system where you don't have any outside visibility except through what Oracle gives you. Spark relies on a bunch of other systems to do things like run processes on a cluster, store data, etc. And the things you're trying to hide here are all visible in those different layers.
Even with Oracle, you could check whether people are running certain tools on client machines and say "hey, user foo is connecting to Oracle". You may not know which DB they're connecting to, and you definitely won't know what it is that they're doing. But you also don't know that with Spark.
To go through your examples:

user names are not sensitive information. You can see them in /etc/passwd. You can see them by listing files on your fs - even if you don't have read permissions on the file itself, or reading ACLs for those files. If you want two companies to not see each other, you deploy different clusters (or, in this case, different SHS reading from different event log directories, with different authentication for each).

The app name is arguable. But it's always been public in Spark, so people shouldn't be using that for anything sensitive. If they are, well, they already have a security problem right there, today, and your patch won't fix it, since that data has already leaked. And better hope that app name was not set in any command line, since those are visible to anyone who can log into the same machine.

Who's using the cluster. Again, not sensitive information.

If you want to draw a parallel to something like Oracle, you should be looking at the thrift server. That one is supposed to be a multi-user service that shouldn't leak information to users other than the one that submitted a specific job. I have no idea whether that is the case today, but if it's not, it would be a completely different change from what you have here.
If you still think this is important, at the very least this needs to be opt-in. But I'm still very skeptical about the need for this at all.

@mgaido91 what is the status of Hadoop, for example like YARN RM UI, will it show apps which is run by other users, while this user doesn't have permission to see?

@vanzin thanks for your comment. I think it is important, but if I am the only one thinking this in the community of course I can close this PR.
Therefore, I'd like to know also others' opinion on this topic.
@jerryshao the YARN RM UI shows the applications to all the users, as SHS is doing now, indeed.

@mgaido91 should probably close this PR for now.

authors: AmplabJenkins SparkQA ajbozarth jerryshao mgaido91 vanzin


text:
[SPARK-24067][BACKPORT-2.3][STREAMING][KAFKA] Allow non-consecutive offsets

What changes were proposed in this pull request?
Backport of the bugfix in SPARK-17147
Add a configuration spark.streaming.kafka.allowNonConsecutiveOffsets to allow streaming jobs to proceed on compacted topics (or other situations involving gaps between offsets in the log).
How was this patch tested?
Added new unit test
justinrmiller has been testing this branch in production for a few weeks

Test build #90513 has started for PR 21300 at commit 84648e0.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3145/
Test PASSed.

Test build #90513 has finished for PR 21300 at commit 84648e0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90513/
Test PASSed.

Merging to branch-2.3

cc @zsxwing

@gatorsmile this is identical to the original PR which was reviewed by @srowen and discussion on the jira to backport it had not raised any objections since April

@koeninger Not all the committers check the JIRA regularly. For such a big backport PR, we need to check the impact and sometimes even we need to change the implementation to reduce the risk and impacts. Thus, it would be always good to let the others double check and review it instead of directly merging it.
cc @zsxwing @srowen @rxin. Could you double check the PRs and confirm whether the risk is low for backporting it to 2.3?

See the JIRA; this still looks like a bug fix and seems OK for back port to me.

this is ok to me since it's turned off by default

Thanks for your confirmation.

authors: AmplabJenkins SparkQA gatorsmile koeninger srowen zsxwing