text:
[type:refactor] remove SpEL and Groovy plugins

// Describe your PR here; eg. Fixes #issueNo

Make sure that:

[ ] You have read the contribution guidelines.
[ ] You submit test cases (unit or integration tests) that back your changes.
[ ] Your local test passed mvn clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #2576 (a78133b) into master (832f0ba) will decrease coverage by 0.41%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #2576      +/-   ##
============================================
- Coverage     60.80%   60.38%   -0.42%
+ Complexity     4213     4178      -35
============================================
  Files           654      652       -2
  Lines         18493    18485       -8
  Branches       1660     1659       -1
============================================
- Hits          11244    11163      -81
- Misses         6375     6448      +73
  Partials        874      874

Impacted Files
Coverage Δ

...a/org/apache/shenyu/common/enums/OperatorEnum.java
95.00% <ø> (-0.46%)
:arrow_down:

...che/shenyu/sync/data/http/HttpSyncDataService.java
61.38% <0.00%> (-19.81%)
:arrow_down:

...henyu/plugin/grpc/resolver/ShenyuNameResolver.java
51.57% <0.00%> (-12.64%)
:arrow_down:

...a/org/apache/shenyu/common/utils/VersionUtils.java
67.85% <0.00%> (-10.72%)
:arrow_down:

...apache/shenyu/admin/utils/ShenyuDictH2Trigger.java
63.63% <0.00%> (-9.10%)
:arrow_down:

...henyu/admin/service/impl/UpstreamCheckService.java
54.40% <0.00%> (-8.00%)
:arrow_down:

.../apache/shenyu/admin/model/dto/BatchCommonDTO.java
50.00% <0.00%> (-7.15%)
:arrow_down:

...apache/shenyu/admin/model/dto/AuthPathWarpDTO.java
57.14% <0.00%> (-7.15%)
:arrow_down:

...org/apache/shenyu/admin/model/dto/AuthPathDTO.java
58.82% <0.00%> (-5.89%)
:arrow_down:

...in/response/strategy/NettyClientMessageWriter.java
69.56% <0.00%> (-4.35%)
:arrow_down:

... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 832f0ba...a78133b. Read the comment docs.


text:
does not support spring cloud gagteway

Hello,
I found in use that skywalking could not collect data from the Spring Cloud Gateway project (Spring Cloud Gateway is a gateway service through which all requests are routed to specific microservices with an official address of Https://spring.io/projects/spring-cloud-gateway). But for the spring boot project, it can be collected.

Yes. Duplicate requirement #2126 cc @ascrutae

source: gharchive/issue

authors: dean-hu wu-sheng

repo: apache/incubator-skywalking

license_type: permissive

license_source: github-api


text:
Restore Zipkin receiver based on new core

This PR includes old codes at 5.x. I just add Zipkin format supported back and make it passed. Also I add new document for new receiver. Simply, you could open Zipkin reciever by this
receiver_zipkin:
  default:
    host: 0.0.0.0
    port: 9411
    contextPath: /

Zipkin receiver wouldn't use core HTTP port/ip to listen, to avoid an unexpected situation.
FYI @adriancole

Coverage increased (+0.5%) to 13.24% when pulling ae4d4ed689a59c8474f9b3c4de1e9705f4b642bb on zipkin-receiver into afd616433588644addb8498bbe6734e1f5e85803 on master.

source: gharchive/pull-request

authors: coveralls wu-sheng


text:
[Feature] Unified code checkstyle

Search before asking

[X] I had searched in the feature and found no similar feature requirement.

Description
Currently, java in the streampark project uses checkstyle for code style checking,
scala does not have any code style checking,
So we need to use a unified codestyle rule for code checking, such as: spotless
Usage Scenario
No response
Related issues
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

done

repo: apache/incubator-streampark


text:
Add ability to modify size of query window

Make sure these boxes are checked before submitting your issue - thank you!

[ ] I have checked the superset logs for python stacktraces and included it here as text if any
[x] I have reproduced the issue with at least the latest released version of superset
[x] I have checked the issue tracker for the same issue and I haven't found one similar

Superset version
0.17.4
Expected results
See a draggable/expandable box corner.
Actual results
There is no such draggable, expandable box corner.
Steps to reproduce
There are 25 more lines in this query :(

Notice: this issue has been closed because it has been inactive for 375 days. Feel free to comment and request for this issue to be reopened.

Please reopen this issue.  The query window should be expandable so I can see more than 12 lines of code at a time.

@charlesdguthrie this is fixed in master

authors: charlesdguthrie mistercrunch silashundt

repo: apache/incubator-superset


text:
[js-testing] write more tests for javascripts/explore/reducers/exploreReducer.js

current coverage: 34%. bring to 70%.

Notice: this issue has been closed because it has been inactive for 331 days. Feel free to comment and request for this issue to be reopened.

authors: ascott mistercrunch


text:
How can I start a Superset web server rather than development server in this project?

Question : When I type superset --help , there is no option for "runserver " but only "run" . Who knows how to deal with it? Thank you ! #question

https://apache-superset.readthedocs.io/en/latest/installation.html#a-proper-wsgi-http-server

have you solved this?I get the same question like you.Hope you can help,Thanks a lot!

have you solved this?I get the same question like you.Hope you can help,Thanks a lot!

sorry , I just get your question.....the link above tells us the great way, you can run "gunicorn
-w 10
-k gevent
--timeout 120
-b  0.0.0.0:6666
--limit-request-line 0
--limit-request-field_size 0
--statsd-host localhost:8125
superset:app",

authors: Demonya Robinub mistercrunch


text:
Inconsistency in descriptions

Changes are all in descriptions. Didn't touch code.

Corrected inconsistency of description and examples in argument.

In the explanation of argument 'sv', example 1 and 2 contradicted the statement: "sum(sv) must be 1".

Example 2 (sv = (0.25, 0.25, 0.25, 0.25, 0.25)) seems like a typo(?), because when i == 5 in parfor, T2 (line55) would be a zero column, so P (line57) would be empty.

Corrected location of 'sample.dml' in the example: From algorithms/utils/sample.dml to ./scripts/utils/sample.dml.

Added slightly more wordy explanations, as new users may try to learn dml from sample scripts like this one.

Can an Admin verify this patch?

thanks @ethanyxu that's a good catch - the example of 5 partitions is indeed wrong.
Could you please slightly change the description: (1) it would be good to relax statements like "equal sized disjoint subsets" as the the outputs are not necessarily equal sized (they are in expectation equal sized), (2) also, "randomly split" might create the wrong impression that sum(sv)=1 although you might very well have sum(sv)<1 and use it for sampling, (3) could you please break very long lines (e.g., line 23) and fix the indentation of lines 28-30? Thanks.

add to whitelist.

Thank you @mboehm7 . Changes modified according to your comment.

LGTM - in addition, I would just slightly modify line 29 to "less than or equal 1"

Done.

great - thanks @ethanyxu

authors: akchinSTC dusenberrymw ethanyxu mboehm7

repo: apache/incubator-systemml

license_source: bigquery


text:
Parameter APIs: CRUD

[x] get /api/$version/parameters
[ ] get /api/$version/parameters/:id
[ ] post /api/$version/parameters
[ ] put /api/$version/parameters/:id
[ ] delete /api/$version/parameters/:id

@dangogh - see #1363

fixed in #1950

repo: apache/incubator-trafficcontrol


text:
Remove Cava-Toml subproject

Following deprecation in the 0.7 release branch (#16), remove Cava-Toml from master.

As per https://github.com/apache/incubator-tuweni/pull/16#issuecomment-491569392, I'll close this until the discussions are concluded.

repo: apache/incubator-tuweni


text:
Need armeabi-v7a, arm64-v8a and x86_64 architectures support in Android Weex SDK

Describe the bug
Currently, Android Weex SDK doesn’t support armeabi-v7a, arm64-v8a and x86_64 architectures. From August 2019 onwards Google will not allow apps on Playstore which does not have support for these architectures. So we need these architectures to support in Weex SDK.
Expected behavior
Weex SDK should support x64 architectures eg. armeabi-v7a, arm64-v8a and x86_64.
Additional context
For more info please check the following link:
https://developer.android.com/distribute/best-practices/develop/64-bit

We are working on this issue now, and I think we will solve the problem in the early of June.
Thanks.

I'm working on this issue, and arm64-v8a so has been built out now.  As YorkShen said,  it will be released with Weex 0.26

ARM 64 is supported now.
#2561

so it released with Weex 0.26?
YorkShen notifications@github.com 于2019年6月18日周二 下午8:49写道：

Closed #2389 https://github.com/apache/incubator-weex/issues/2389.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-weex/issues/2389?email_source=notifications&email_token=AADZOGDLGUKI2LWMN56TNMTP3DKUNA5CNFSM4HKTBJC2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOSBESFOY#event-2420712123,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADZOGHEQ3OWXE5BSEBLG2LP3DKUNANCNFSM4HKTBJCQ
.

What is the latest arm64-v8a version?

Please subscribe the mailing list. According to ASF's policy, Weex 0.26 will not be release until the LGPL issue is solved. The information will be updated in the mailing list, not there.

Ok，Thank you very much!
YorkShen notifications@github.com 于2019年6月24日周一 下午7:50写道：

Please subscribe the mailing list
https://weex.apache.org/guide/contribute/how-to-contribute.html#mailing-list.
According to ASF's policy
https://apache.org/legal/resolved.html#category-x, Weex 0.26 will not
be release until the LGPL issue
https://issues.apache.org/jira/browse/LEGAL-464 is solved. The
information will be updated in the mailing list, not there.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-weex/issues/2389?email_source=notifications&email_token=AADZOGDTMXD3YSGA2OUU7YTP4CYHTA5CNFSM4HKTBJC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYMVHZY#issuecomment-504976359,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADZOGENOEM2QENOUL26XM3P4CYHTANCNFSM4HKTBJCQ
.

When will version 26 be available? I can download the SDK reference directly, but it didn't work

authors: Darin726 YorkShen ginuim nikkulshrestha yanghuaxiong

repo: apache/incubator-weex


text:
[Feature][Dashboard] Batch Data Stream Management

Description
No response
Use case
No response
Are you willing to submit PR?

[ ] Yes, I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

duplicated with https://github.com/apache/inlong/issues/9813, close it.

repo: apache/inlong


text:
[IOTDB-3003]  data sync pipe recover bug & other issues

https://issues.apache.org/jira/browse/IOTDB-3003

@Cpaulyz   Could you help review this code change?

@

LGTM

thank you

@HTHou  could you help merge this PR?     :-)

@HTHou   thank you!

repo: apache/iotdb


text:
[IOTDB-3922] Fix incorrect column name in some scenarios for GroupByLevel

see IOTDB-3922

Coverage decreased (-0.002%) to 43.147% when pulling 3e6cdaf48a0bd1191dedc1bf33e3185915bfe33b on lmh/FixGroupByLevelHeader into 22f28663259ebbdee74332e84dad1a3073ccc65a on master.


text:
OAK-11075: Add a Buffer#getShort() API

Modifies org.apache.jackrabbit.oak.commons.Buffer to also support getting a short from the current position of the buffer, similarly to other APIs already provided such as Buffer#getLong().
Closes OAK-11075

This change requires a minor version increase for package org.apache.jackrabbit.oak.commons.
The build failed for oak-commons with:
[ERROR] org.apache.jackrabbit.oak.commons: Version increase required; detected 2.1.0, suggested 2.2.0

Thanks @mreutegg, I updated my pull request.

Thanks for your review!

authors: Nicolapps mreutegg

repo: apache/jackrabbit-oak


text:
MINOR: Remove ZK dependency for coordinator topics' partition counts

GroupMetadataManager currently uses ZooKeeper to obtain any non-default value for the __consumer_offsets partition count. ZooKeeper will not be available when the broker uses a Raft-based metadata quorum. This PR removes the hard dependency on ZooKeeper by allowing a function returning an Int to be provided instead. Providing a ZooKeeper client is still supported in apply() methods, but those methods transparently wrap the client with a function and pass that function instead.
TransactionStateManager currently uses ZooKeeper to obtain any non-default value for the __transaction_state partition count.  ZooKeeper will not be available when the broker uses a Raft-based metadata quorum.  This PR removes the hard dependency on ZooKeeper by allowing a function returning an Int to be provided instead.  Providing a ZooKeeper client is still supported in apply() methods, but those methods transparently wrap the client with a function and pass that function instead.
Existing tests are sufficient to prevent bugs and regressions.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Did we consider passing a function instead of zk client?

Did we consider passing a function instead of zk client?

What is the motivation for making transactionTopicPartitionCount and groupMetadataTopicPartitionCount lazy? Is it because in the KIP-500 world, we may not yet have this metadata from the Raft quorum so we can't look this up on startup?
If that's the case, are we guaranteed to have loaded the metadata before the first usages of GroupMetadataManager and TransactionManager?

We discussed this offline and the current suggestion is to pass the function in the start method.

We finally got a non-red build, some unrelated flaky tests in JDK 11 and 15. JDK 8 passed.

Failing tests are
org.apache.kafka.connect.integration.RebalanceSourceConnectorsIntegrationTest.testMultipleWorkersRejoining
and org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationSSLTest.testOneWayReplicationWithAutoOffsetSync() which are not related.

authors: cmccabe ijuma mumrah rondagostino

repo: apache/kafka


text:
KAFKA-14104: Add CRC validation when iterating over Metadata Log Records

This commit adds a check to ensure the RecordBatch CRC is valid when
iterating over a Batch of Records using the RecordsIterator. The
RecordsIterator is used by both Snapshot reads and Log Records reads in
Kraft. The check can be turned off by a class parameter and is on by default.

The build failures are a little cryptic. The same tests all pass locally, and the failing step in the build only complains of the script exiting with a non-zero status, but there are no actual failing tests. Will just try to trigger another run.


text:
KAFKA-14172: Should clear cache when active recycled from standby

This fix is inspired by https://github.com/apache/kafka/pull/12540.

Added a clearCache function for CachedStateStore, which would be triggered upon recycling a state manager.
Added the integration test inherited from #12540 .
Improved some log4j entries.
Found and fixed a minor issue with log4j prefix.

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

LGTM. Feel free to merge (if you don't want to do the rewrite of the test).

Managed to do that rewrite :) though it requires some unchecked conditional get still..

@mjsax cherry-picked to 3.4 so far; 3.3 gets a lot of conflicts so I'm still working on it.

Thank you for looking at the bug report - much appreciated.

authors: chorslev guozhangwang


text:
KAFKA-7420: Global store surrounded by read only implementation

Global store surrounded by read-only KeyValueStore implementation.
All methods that try to write into Store will throw UnsupportedOperationException
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Hello, @mjsax! Can you take a look at my changes?

Failure unrelated.
Retest this, please

Failure unrelated.
Retest this, please

Failure unrelated.
Retest this, please

@nizhikov We stabilized a couple of test recently. Can you rebase this PR? This should help to get green builds.

@mjsax

We should add a new ProcessorContextImplTest

Please, tell me, what should be tested by this?

@mjsax

We should add a new ProcessorContextImplTest

Can you help me with this? What should be tested?

We should test, that if a globalstore is returned from the context, calling any of the protected methods throws an exception.

Tests passed

Call for second review. Anyof @guozhangwang @bbejeck @vvcephei

@bbejeck Thank you for the review.
I've added tests for non-exception methods of KeyValueStore.
Please, take a look.

Build failed with a checkstyle error

@mjsax
@bbejeck
Thanks for the review.

Build failed with a checkstyle error
LGTM pending the checkstyle fix

:man_facepalming: Sorry about that. Fixed

Tests passed. Can we merge this PR?

Thus, I think, we need to check the type of the global store, and wrap it accordingly if it implements one of the three store interfaces.

We should wrap in read only decorator instances that implements 3 following interfaces:

KeyValueStore
WindowStore
SessionStore

Matthias, am I understand you correctly?
I will provide this fix and tests shortly.

Yes. Check the type, for KeyValueStore do what the current PR does, and for Window/Session-Store apply the same pattern.

@mjsax I've implement your proposal. Please, review.

Tests are green.

Tests are green.

Hello, @guozhangwang. Please, review my PR.

@mjsax Do I need to improve this PR? Can you approve it?

Hello @mjsax
I got 2 OKs from you and bbejeck
Do I need one more review from guozhangwang?
No rush, I just want to know, how it's work in Kafka community
Can you merge this PR by yourself? :)

@nizhikov Committers can merge PRs.
I would like to get a confirmation from @bbejeck because we did some mayor changes after he approved. (Or a second review from @guozhangwang @vvcephei).
We want two +1 before merging.

@bbejeck Thank you.
@mjsax  There are two +1. Let's merge it?! :)

can you rebase this PR one more time to make sure the build is green (just as a guard to not break trunk if we merge). Thanks.

Done

Hello, @mjsax
Is there anything that should be improved prior merge?

Hello, @mjsax
Do you have a change do look at the tests results?
Can we merge PR?
Should I ask Bill Bejeck or Guozhang Wang to review and merge this patch?

Thank you, @mjsax
Sorry for pinging you too many times.
WIll try to avoid it in the next contributions.

authors: mjsax nizhikov


text:
MINOR: Fix stale comment in partition reassignment javadoc

During the cleanup phase of an overriding reassignment, we failed to mention that we still add replicas in the adding_replicas field for the initial LeaderAndIsr request

As said in the javadoc:
* Cleanup Phase: In the cases where this reassignment has to override an ongoing reassignment.
   *   . The ongoing reassignment is in phase A
   *   . ORS denotes the original replica set, prior to the ongoing reassignment
   *   . URS denotes the unnecessary replicas, ones which are currently part of the AR of the ongoing reassignment but will not be part of the new one
   *   . OVRS denotes the overlapping replica set - replicas which are part of the AR of the ongoing reassignment and will be part of the overriding reassignment
   *       (it is essentially (RS - ORS) - URS)

We do want to keep AR=OVRS because the overriding reassignment will have it too - no need to risk shrinkage of ISR during the race between this first LAIR and the subsequent LAIR that adds the proper AR/RR fields
Here is the code which shows that we send AR=OVRS
https://github.com/apache/kafka/blob/f99bb0466e9c8845963b350f3ad9b353db72fd5a/core/src/main/scala/kafka/controller/KafkaController.scala#L693
cc @viktorsomogyi @cmccabe

Yes, good catch. Thanks!

LGTM

authors: cmccabe stanislavkozlovski


text:
KAFKA-9921: explicit handling of null values with retainDuplicates

In general the behavior of window stores with retainDuplicates is not well documented or enforced, so we should attempt to clarify things better in the javadocs and in the code itself. This explicitly skips the put/delete when the value is null and duplicates are allowed, and specifies this behavior in the docs.
Also adds in a test I left out in the earlier PR https://github.com/apache/kafka/pull/8564

Retest this please

Retest this please

Retest this please.

Merged to trunk and cherry-picked to 2.5.

authors: ableegoldman mjsax


text:
MINOR: Fix timestampDelta type in doc

Type of timestampDelta is written as varint in the doc, but actually it's varlong

https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/record/DefaultRecord.java#L47

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

retest this please

Ok to test

@ocadaruma Could you rebase to trigger QA?

@ocadaruma Could you rebase it again? Our Jenkins was on vacation :(


text:
MINOR: add ableegoldman and cadonna to whitelist

So we don't have to bug the committers to kick off tests 😄

cc @vvcephei @guozhangwang


text:
KAFKA-10692: Add delegation.token.secret.key, deprecate ...master.key

Add delegation.token.secret.key broker config and deprecate delegation.token.master.key as described in KIP-681

@omkreddy, @mimaison please could one of you review?


text:
Bump rocksdb to v8.3.3

This is a bugfix release in 8.3 line (8.6.х are development now).
Bugfix

Fix a bug where if there is an error reading from offset 0 of a file from L1+ and that the file is not the first file in the sorted run, data can be lost in compaction and read/scan can return incorrect results.
Fix a bug where iterator may return incorrect result for DeleteRange() users if there was an error reading from a file.
Fixed a race condition in GenericRateLimiter that could cause it to stop granting requests

Original release notice: https://github.com/facebook/rocksdb/releases/tag/v8.3.3

The bug fixes LGTM
Also I notice lots of behavior changes and some new API/changes about compaction in 8.4 (Notice that some are DeleteRange related, some are new statistics)
level_compaction_dynamic_level_bytes is default to true, and it's false in kvrocks config. Should we try to switch that? Or keep the current behavior?

Thanks, @mapleFU!  I think, we can change our defaults to rocksdb defaults. I can add in this patch, if other reviewers are OK.

Yes. All these bugfixes LGTM. But there are some changes over compaction, I think having a benchmark or others would be better

@aleksraiden @mapleFU How about upgrading to v8.3.3 first? since those critical fixes are also fixed in this version. And then we can have more tests before upgrading to the next minor version.

Maybe we can upgrade to 8.3.3 before releasing our 2.6.0, and then upgrade to 8.5 in the unstable branch.

OFF TOPIC: I think we now need a release manager for 2.6.0.

@apache/kvrocks-committers Not sure if any committer has time to drive this process recently? Can refer how to create a Kvrocks release

Yeah, I'm not an expert on RocksDB, so I don't know how changes in 8.5.3 will affect kvrocks, though it doesn't meens we're locked in 8.3, but I think we should take upgrade carefully...
But 8.3.3 is always ok to me...

Yeah, I'm not an expert on RocksDB, so I don't know how changes in 8.5.3 will affect kvrocks, though it doesn't meens we're locked in 8.3, but I think we should take upgrade carefully...

Agree we shouldn't be too aggressive when it's related to storage(rocksdb), we need more tests before upgrading to the minor version since it may introduce some unexpected behaviors.

Yes, I also think we upgrade the underlying engine is too radical. I agreed to upgrade to 8.3.3, one of the bugs about GenericRateLimiter we encountered in production and I reported it(https://github.com/facebook/rocksdb/issues/11742).
Like @mapleFU suggested, we should set level_compaction_dynamic_level_bytes to true in kvrocks, this has a lot of benefits, and rocksdb8.2.0 can be easily to migrate level_compaction_dynamic_level_bytes from false to true( https://github.com/facebook/rocksdb/pull/11321)

Okey, many thanks about discussion. I changed PR to support only 8.3.3 and send new PR in next release. Also, config options level_compaction_dynamic_level_bytes changed to true.

authors: PragmaTwice aleksraiden caipengbo git-hulk mapleFU

repo: apache/kvrocks


text:
[2.17.1] AsyncLogger thread is stuck on 100% CPU on lmax-34. queue polling

Description
Some low probability race condition causes lmax to loop indefinitely on Thread.yield
Configuration
-DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector
-Dlog4j2.enable.threadlocals=true
-Dlog4j2.enable.direct.encoders=true
-XX:CompileCommand=dontinline,org.apache.logging.log4j.core.async.perftest.NoOpIdleStrategy::idle
Version: 2.17.1
Operating system: Centos 8
JDK: OpenJDK 64-Bit Server VM GraalVM LIBGRAAL 20.3.3 (build 11.0.12+5-jvmci-20.3-b06)
Logs
It would seem that the queue datastructure becomes corrupted and cas=uses the polling thread to loop infinitely with the following stacktrace:
"Log4j2-TF-1-AsyncLogger[AsyncContext@46f5f779]-1" #18 daemon prio=5 os_prio=0 cpu=4404755.82ms elapsed=4733.22s tid=0x00007fed7ade9800 nid=0xcd runnable  [0x00007fed7bffd000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.Thread.yield(java.base@11.0.12/Native Method)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.transferAfterCancelledWait(java.base@11.0.12/AbstractQueuedSynchronizer.java:1752)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.12/AbstractQueuedSynchronizer.java:2119)
        at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.lang.Thread.run(java.base@11.0.12/Thread.java:829)

Reproduction
This was found on a large production system with very low probability. Using log4j with the above settings. We don't know the exact case where the polling thread reaches this state

@shloim, the upcoming Log4j 2 release (i.e., 2.23.0) will be the very first Log4j 2 version that ships Disruptor 4 support. (Disruptor 3 will still work too.) Is that something you can try?

Alternatively, can you upgrade your Log4j version to 2.18.0 or later?
In 2.18.0 @remkop backported the TimeoutBlockingWaitStrategy from Disruptor 4.x with such a comment:
https://github.com/apache/logging-log4j2/blob/252f1a65e1b44fa78c622b4140d9116bf57d3617/log4j-core/src/main/java/org/apache/logging/log4j/core/async/TimeoutBlockingWaitStrategy.java#L34-L40
The difference in the locking mechanism seems relevant to your case.

Alternatively, can you upgrade your Log4j version to 2.18.0 or later?
In 2.18.0 @remkop backported the TimeoutBlockingWaitStrategy from Disruptor 4.x

I'm working on testing it with log4j=2.22.1 slf4j=1.7.36 lmax=3.4.4

Alternatively, can you upgrade your Log4j version to 2.18.0 or later?
In 2.18.0 @remkop backported the TimeoutBlockingWaitStrategy from Disruptor 4.x with such a comment:
https://github.com/apache/logging-log4j2/blob/252f1a65e1b44fa78c622b4140d9116bf57d3617/log4j-core/src/main/java/org/apache/logging/log4j/core/async/TimeoutBlockingWaitStrategy.java#L34-L40
The difference in the locking mechanism seems relevant to your case.

Do I need to set a custom WaitStrategy or can I keep the default behavior?

@shloim,
"Our" TimeoutBlockingWaitStrategy should be the default one.

After switching to log4j 2.18.0 the issue persists but in a different stacktrace:
"Log4j2-TF-1-AsyncLogger[AsyncContext@46f5f779]-1" #20 daemon prio=5 os_prio=0 cpu=3228328.37ms elapsed=3495.56s tid=0x00007fea233e9000 nid=0xd0 runnable  [0x00007fea23dfd000]
   java.lang.Thread.State: RUNNABLE
        at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.lang.Thread.run(java.base@11.0.12/Thread.java:829)

@shloim, can you give 2.23.0-SNAPSHOT a try in combination with Disruptor 4? (See docs on how to configure Maven/Gradle/SBT for snapshot builds.)

Hi,
I'm seeing this with 2.23.1 and Disruptor 4.0.0 in my application.
It's happening when i use

Perhaps the Log4j async manual could be more clear

We can indeed do this. Updated the #2534 description accordingly.

The documentation of Yield was fixed in #2577 .

same issues. and cpu 100%. and t version is 3.4.2 .

@Baixiu-code,

Are you using Log4j Core 2.24.1?
Did you try using com.lmax:disruptor version 4.0.0?
What is the status of the polling thread? The polling thread's name starts with Log4j2-TF-<number>-AsyncLogger as showed in the initial comment for this issue. Your screenshot only covers the logging threads, which are blocked, because the queue is full. Can you provide the stack trace of the polling thread as text instead of an image?

authors: AndreKurait Baixiu-code ppkarwasz shloim vy


text:
LOG4J2-1216: Nested pattern layout options broken

Change is for the most part as suggested by the original reported of the issue in the Jira, I only added a unit test. I made one small modification:
if (depth > 0) {          // option not closed, continue with pattern on opening "{"
    return begin-1;
}

Turned to:
if (depth > 0) {          // option not closed, continue with pattern after closing bracket
   return pattern.indexOf('}', start) + 1;
}

Because the original version was breaking PatternParserTest#testCustomPattern.
Also it seems there are 3 unit test in core that are broken on head.

Reference: https://issues.apache.org/jira/browse/LOG4J2-1216

Is this PR still relevant? @garydgregory ?

authors: garydgregory mikaelstaldal zsomborklara


text:
Add note about versions supporting log4j.xml

Would be good to add this to more of the features, but I don't know have enough context to label everything correctly.

Thanks for the additional updates @HoustonPutman. It looks good to me, anything else we would want before committing?

I think it's good to go!

authors: HoustonPutman madrob

repo: apache/lucene-solr-operator


text:
MAHOUT-1642 fix missing first element in SimilarItemsIterator

based on @OlegZotov's fix #77

ok, without being in context of this algorithmically, this looks like a valid fix to me. I think I'd rely on the opinion on contributors and commit it even though it is to a taste recommender. If @srowen happens to come and say yay or nay, it'd be great.

authors: dlyubimov fape

repo: apache/mahout


text:
Update contributors.yaml for gradywang.

Update contributors.yaml for gradywang.

@vinodkone, cloud you help to review this again? Thanks!

repo: apache/mesos


text:
MINIFICPP-1504: followup review changes

Renamed CPU to Cpu
Removed static_errors
Replaced c style casts to static_cast
Replaced implicit narrowing
Added type aliases
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically main)?

[ ] Is your initial contribution a single, squashed commit?

For code changes:

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file?
[ ] If applicable, have you updated the NOTICE file?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.

I messed up the commit msg, but meant to close this with 4fc6f49d

authors: martinzink szaszm

repo: apache/nifi-minifi-cpp


text:
NIFI-2752 Correct ReplaceText default pattern and unit tests (0.x)

Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[X] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[X] Has your PR been rebased against the latest commit within the target branch (typically master)?

[X] Is your initial contribution a single, squashed commit?

For code changes:

[X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[X] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

reviewing

+1 contrib-check, verified unit tests before and after the fix, ran a simple flow, and looks like it matches master branch.  Thanks @jskora will merge to 0.x branch.

Am I missing something, or did this pr get merged in, but is still open?

@trkurc, I believe this has been merged in 0.x branch and consequently the PR is not automatically closed. @jskora do you mind closing it?

@trkurc @pvillard Sorry, my bad for not closing this.

authors: jskora mosermw pvillard31 trkurc

repo: apache/nifi


text:
NIFI-4814 - Add distinctive attribute to S2S reporting tasks

Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit?

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

+1 LGTM, ran all three reporting tasks and verified the attributes are present and correct. Thanks for the improvement! Merging to master


text:
NIFI-10319: Update Calcite Core

Summary
NIFI-10319
Upgrade calcite core to remediate vulnerabilities in the tool.
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[X] Apache NiFi Jira issue created

Pull Request Tracking

[X] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[X] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[X] Pull Request based on current revision of the main branch
[X] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[ ] Build completed using mvn clean install -P contrib-check

[X] JDK 8
[X] JDK 11
[X] JDK 17

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

Will review

+1 looks good to me, tested this out with QueryRecord, will merge.

authors: UcanInfosec thenatog


text:
fs: support openat/fchmodat/mkfifoat/fstatat/...at api

Summary
fs: support openat/fchmodat/mkfifoat/fstatat/...at api
Refs to:
https://linux.die.net/man/2/openat
Impact
support file api with relative directory path spicified dirfd
Testing
local test

@yamt openat api is ready for review.

repo: apache/nuttx


text:
Implemented: New service definition attribute "verb" to denote the corresponding HTTP method (OFBIZ-11328)

Implemented: New service definition attribute "verb" to denote the corresponding HTTP method (OFBIZ-11328)
Explanation
In order for exportable services in OFBiz to be called via REST interface, it is necessary for the service to tell what HTTP verb should be used to call it. This implementation adds the attribute and makes it available in ModelService class via property verb.
Thanks:
Girish

I'm keeping it open for review by the reviewers. I will merge it during the upcoming weekend.

If we're talking about an action here, why don't we call it 'action'?

@PierreSmits : "action" makes more sense. I'll change the attribute name and then merge. Thanks.

authors: PierreSmits girishvasmatkar

repo: apache/ofbiz-framework


text:
Diversifying the reflection API, for environment.go

https://groups.google.com/g/golang-nuts/c/S2gBW3BV4QU/m/I4gWtrPxBwAJ
https://github.com/apache/openwhisk-wskdeploy/issues/1130

See comments in https://github.com/apache/openwhisk-wskdeploy/pull/1134

authors: advancedwebdeveloper mrutkows

repo: apache/openwhisk-wskdeploy


text:
HDDS-7315. SCM order of close of dbstore and other services

What changes were proposed in this pull request?
scmMetaStore to be close at last in sequence, as other service like metric may be using this to access tables.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7315
How was this patch tested?
Running Unit test and integration test for the impact.

@ChenSammi @nandakumar131  Please review

LGTM + 1.

authors: ChenSammi sumitagrawl

repo: apache/ozone


text:
HDDS-7600. [Snapshot] Mark all new proto message fields as optional

What changes were proposed in this pull request?
Mark all new proto message fields as optional before snapshot feature branch is merged to master branch.
Context: https://github.com/apache/ozone/pull/3985/files#r1041505724
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7600
How was this patch tested?

Existing tests should pass.

@smengcl, why do we need to mark the field as optional? If the message itself is optional, then the fields are never used.
If someone wants to use that message, we should have the mandatory fields as required.

@smengcl, why do we need to mark the field as optional? If the message itself is optional, then the fields are never used. If someone wants to use that message, we should have the mandatory fields as required.

Hi Nanda, as I mentioned previously in this PR comment:
https://github.com/apache/ozone/pull/3824#discussion_r1011376445
required field is an outdated concept in proto2, where protobuf 3 has completely ditched this concept, so every single field in proto3 is implicitly optional, reason being that user can always check the whether the field is set with hasABCField().
A significant drawback of required is that it makes it significantly harder to deprecate and remove a field in the future, which IIRC was also blocking Hadoop from moving to pure proto3. Although we are still using proto2 syntax, I would suggest strongly moving away from required and embrace optional.

Thanks @nandakumar131 and @prashantpogde for the reviews.


text:
HDDS-8526. Intermittent failure: Directory is not empty

What changes were proposed in this pull request?
When key is deleted, it firsts inserts into cache as deleted and later flush to db which deletes entry from db. In case of Bucket empty check we first check in cache and then check in table whether any entry exist for the prefix. But during this iteration there is chance that cache gets flushed into db and when we read cache it returns NULL and at this point we assume that table has entry but cache don't have. This shows that bucket is not empty at this point.
So to fix this in this case we should check in table whether really table entry is removed.
Similar problem is for Volume empty check. Making volume empty check also follow above criteria to ensure it is empty or not.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8526
How was this patch tested?
Before Fix: Test fails Intermittently in latest master for TestRootedOzoneFileSystemWithFSO.
https://github.com/ashishkumar50/ozone/actions/runs/5529154450
After Fix: Succeeded in all 10 * 10 run for TestRootedOzoneFileSystemWithFSO.
https://github.com/ashishkumar50/ozone/actions/runs/5531565163

@ashishkumar50 Thanks for working over this, there is an improvement possible, BucketTable is cache CacheType.FULL_CACHE, i.e. entire bucket table is cached. So this actually do not need to check the Table over disk.
The bug here is fixing that, if entry in disk table is present and validating with cache entry exist and reverifying with disk. But check with disk table can be avoided.

@sumitagrawl, You mean for checking isVolumeEmpty we should avoid calling
isKeyPresentInTable(volumePrefix) ? And we can just depend on cache?
Old existing code was checking even in DB. Is it safe here just depend on cache and not checking on disk?

recently we have done same for container delete, even though we have in cache that shows container is empty but still we check from db to really ensure it is empty before delete

Only volume and bucket tables use FULL_CACHE, so other table may have entries not in the cache.
100x run passed with the change @sumitagrawl suggested.
Regular CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/5540255300
To be on the safe side, we can make the change in two steps (for easier revert if problems surface): first merge this one as is, then create new PR for the additional performance improvement.

recently we have done same for container delete, even though we have in cache that shows container is empty but still we check from db to really ensure it is empty before delete

Only volume and bucket tables use FULL_CACHE, so other table may have entries not in the cache.
100x run passed with the change @sumitagrawl suggested.
Regular CI: https://github.com/adoroszlai/hadoop-ozone/actions/runs/5540255300
To be on the safe side, we can make the change in two steps (for easier revert if problems surface): first merge this one as is, then create new PR for the additional performance improvement.

Ok we can do improvement in separate PR, currently we can keep as old behaviour to check both Cache and DB for bucket and key table. Later we can use just cache for bucket table.

Thanks @ashishkumar50 for the fix, @smengcl, @sumitagrawl for the review.


text:
HDDS-9131. [hsync] A freon tool to focus on hsync/hflush performance

What changes were proposed in this pull request?
Add a -sync option for ozone freon dfsg which does hsync/hflush after write.
Example:
$ ozone freon dfsg --path ofs://ozone1/vol1/bucket1/a1 -s 1024 --buffer 1024 -sync hflush -t 8 -n 10000
starts 8 threads to write 10000 files, each file is hflush'ed when reaching the end of buffer.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9131
How was this patch tested?
Unit tests, manual tests

CI's failing, UT:
 Tests run: 6, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 0.807 s <<< FAILURE! - in org.apache.hadoop.ozone.freon.TestContentGenerator
Error:  org.apache.hadoop.ozone.freon.TestContentGenerator.writeWrite  Time elapsed: 0.015 s  <<< ERROR!
java.lang.NullPointerException
	at org.apache.hadoop.ozone.freon.ContentGenerator.doFlushOrSync(ContentGenerator.java:106)
	at org.apache.hadoop.ozone.freon.ContentGenerator.write(ContentGenerator.java:99)
	at org.apache.hadoop.ozone.freon.TestContentGenerator.writeWrite(TestContentGenerator.java:42)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

Fixing that now

The failed test was only merged in master a few hours ago and doesn't look related.

Thanks @adoroszlai . I am curious how did you merge commits and update my PR :) Can any committers update someone else's PR?

I am curious how did you merge commits and update my PR :) Can any committers update someone else's PR?

When creating the PR, there is a checkbox Allow edits and access to secrets by maintainers, enabled by default.  If it is left enabled, committers can update the PR.

Thanks @jojochuang for the patch, @duongkame for the review.

authors: adoroszlai duongkame jojochuang


text:
HDDS-11482. EC Checksum throws IllegalArgumentException because the buffer limit is negative

What changes were proposed in this pull request?
The parity checksum bytes calculation for the EC stripe checksum should be based on each stripe's first chunk's length. But right now it is calculated only on the first stripe's chunk length which results in incorrect results and, in many cases causes the byte buffer limit to become negative. Further, the checksum calculation has been simplified in this patch.
The checksum calculation for the last stripe is a little tricky, We have two cases for it.

If the last stripe has only one chunk at replica index 1, we can get the chunk length to calculate checksum.
If the last stripe has more than one chunk, Then the last chunk will be partial.

For EC, We store the stripe checksum for the entire stripe in replica index 1 and the parity blocks. We use this stripe checksum to calculate the checksum during reading. The chunks we get during checksum calculation will always be from replica index 1 or parity blocks. And the chunk length will always be the stripe's first chunk's length.
Hence, we don't get the length of the last chunk which may be partial in most of the cases. We calculate the last chunk's size from the block size.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11482
How was this patch tested?
Updated the existing test.
Test fails without the fix:https://github.com/aswinshakil/ozone/actions/runs/11002573679/job/30550687378
Test passed with the fix: https://github.com/aswinshakil/ozone/actions/runs/11002650413/job/30551032116

@sadanand48 @sodonnel Can you take another look at this? Thank you!

@sodonnel I have updated the PR. Can you please take a look at it?

Thanks @aswinshakil for the patch, @sadanand48, @sodonnel for the review.

authors: adoroszlai aswinshakil


text:
【Hive】To avoid excessive HMS memory usage, when executing AlterTable for a Paimon table containing a large number of fields

Purpose
When metastore=hive, if the number of table fields is large (1600+) and there are many files, MetaStoreUtils.updateTableStatsSlow is executed by default to update table statistics, but Paimon has its own statistics and does not need to rely on hive.
Therefore, an option is provided to control whether to update Hive table statistics by the business
Linked issue: Issue-4507
API and Format
org.apache.paimon.hive.HiveCatalog#alterTableToHms

Please change your title 【Hive】to [Hive] and make your code style with format, you can execute 'mvn spotless:apply' for it

Please change your title 【Hive】to [Hive] and make your code style with format, you can execute 'mvn spotless:apply' for it

Thank you. All the tests can be passed locally, but there are still errors now. It seems that the errors have nothing to do with my changes.

+1

authors: GangYang-HX JingsongLi xuzifu666

repo: apache/paimon


text:
Pulsar test instability

Error is occurring to Pulsar test case randomly on multiple of my PRs
PulsarConsumerTest.testPartitionLevelConsumerBatchMessages:246 expected [500] but found [491]

CI Logs. Click to expand
Error:  Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.324 s <<< FAILURE! - in org.apache.pinot.plugin.stream.pulsar.PulsarConsumerTest
Error:  org.apache.pinot.plugin.stream.pulsar.PulsarConsumerTest.testPartitionLevelConsumer  Time elapsed: 0.845 s
Error:  org.apache.pinot.plugin.stream.pulsar.PulsarConsumerTest.testPartitionLevelConsumerBatchMessages  Time elapsed: 0.48 s  <<< FAILURE!
java.lang.AssertionError: expected [500] but found [491]
    at org.testng.Assert.fail(Assert.java:93)
    at org.testng.Assert.failNotEquals(Assert.java:512)
    at org.testng.Assert.assertEqualsImpl(Assert.java:134)
    at org.testng.Assert.assertEquals(Assert.java:115)
    at org.testng.Assert.assertEquals(Assert.java:388)
    at org.testng.Assert.assertEquals(Assert.java:398)
    at org.apache.pinot.plugin.stream.pulsar.PulsarConsumerTest.testPartitionLevelConsumerBatchMessages(PulsarConsumerTest.java:246)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:661)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:869)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1193)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:126)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
    at org.testng.TestRunner.privateRun(TestRunner.java:744)
    at org.testng.TestRunner.run(TestRunner.java:602)
    at org.testng.SuiteRunner.runTest(SuiteRunner.java:380)
    at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:375)
    at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
    at org.testng.SuiteRunner.run(SuiteRunner.java:289)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1301)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1226)
    at org.testng.TestNG.runSuites(TestNG.java:1144)
    at org.testng.TestNG.run(TestNG.java:1115)
    at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:136)
    at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
    at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99)
    at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:145)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
    at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)

[INFO]
[INFO] Results:
[INFO]
Error:  Failures:
Error:    PulsarConsumerTest.testPartitionLevelConsumerBatchMessages:246 expected [500] but found [491]

Fixed by #8554

Re-opening the issue as it seems the flakiness is reduced but still occurs

OK. so the branch wasn't rebased in the PR :P

this seems to be still recurring: https://github.com/apache/pinot/runs/6120269495?check_suite_focus=true.
if I understood correctly github action always merge master before running tests. so PR base doesn't matter. see the log in the above test link
refs/remotes/pull/8580/merge

I'm still seeing the exact same failure. @KKcorps Please take another look

Another failure: https://github.com/apache/pinot/actions/runs/4366896907/jobs/7637509337
Error:  Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 68.865 s <<< FAILURE! - in org.apache.pinot.plugin.stream.pulsar.PulsarConsumerTest
Error:  org.apache.pinot.plugin.stream.pulsar.PulsarConsumerTest.testPartitionLevelConsumer  Time elapsed: 0.507 s  <<< FAILURE!
java.lang.AssertionError: expected [500] but found [400]

Should be solved with #12812

Still flaky..

Closing this one and track it in #13008 which contains more details

authors: Jackie-Jiang KKcorps walterddr

repo: apache/pinot


text:
add metric for time retention failing due to end time

This is a new metric to catch when segment start/end time is invalid and there time based segment retention is failing.

Codecov Report
Attention: Patch coverage is 0% with 5 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (59551e4) to head (6f5866c).
Report is 931 commits behind head on master.

Files
Patch %
Lines

...core/retention/strategy/TimeRetentionStrategy.java
0.00%
3 Missing :warning:

...g/apache/pinot/common/metrics/ControllerMeter.java
0.00%
1 Missing :warning:

...troller/helix/core/retention/RetentionManager.java
0.00%
1 Missing :warning:

:exclamation:  There is a different number of reports uploaded between BASE (59551e4) and HEAD (6f5866c). Click for more details.
HEAD has 53 uploads less than BASE

Flag
BASE (59551e4)
HEAD (6f5866c)

integration
7
1

integration2
3
1

temurin
12
1

java-21
7
1

skip-bytebuffers-true
3
0

skip-bytebuffers-false
7
1

unittests
5
0

unittests1
2
0

java-11
5
0

unittests2
3
0

integration1
2
0

custom-integration1
2
0

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #13879       +/-   ##
=============================================
- Coverage     61.75%    0.00%   -61.76%
=============================================
  Files          2436     2502       +66
  Lines        133233   138574     +5341
  Branches      20636    21540      +904
=============================================
- Hits          82274        0    -82274
- Misses        44911   138574    +93663
+ Partials       6048        0     -6048

Flag
Coverage Δ

custom-integration1
?

integration
0.00% <0.00%> (-0.01%)
:arrow_down:

integration1
?

integration2
0.00% <0.00%> (ø)

java-11
?

java-21
0.00% <0.00%> (-61.63%)
:arrow_down:

skip-bytebuffers-false
0.00% <0.00%> (-61.75%)
:arrow_down:

skip-bytebuffers-true
?

temurin
0.00% <0.00%> (-61.76%)
:arrow_down:

unittests
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

We do have a check in TimeValidationTransformer to check if the time value is in valid range. Can you double check why it is not triggered since you don't have continueOnError set to true?

Because there's no ingestionConfig set, we hit this branch that sets _enableTimeValueCheck to false.
But this code also has a similar problem. https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/TimeValidationTransformer.java#L93 sets the time value to null without incrementing any metrics. Pinot is basically failing to do something quite important (ingest the data) without offering any way to alert users.

I still don't agree that SegmentStatusChecker is the right place for this, but I've added all "invalid time" metrics to it. I didn't duplicate the "is retention-able" logic, because that would add a 3rd this is duplicated. I've added a unit test on the metric as well.

Good point on emitting metrics when error is captured. The reason why we didn't emit metrics before is because there was no way to pass in a metric object into the RecordTransformer as it can be running in different components (e.g. Server/Minion/Hadoop/Spark). Since we have added singleton metric object, we can emit metrics now.

authors: Jackie-Jiang codecov-commenter jadami10


text:
make data dir accessible from ClusterInfoAccessor

Make data dir accessible from ClusterInfoAccessor so that minion can put data into the same data dir when pushing segments using SegmentPushType.metadata

Codecov Report

Merging #8881 (952290a) into master (754e807) will decrease coverage by 0.03%.
The diff coverage is 0.00%.

:exclamation: Current head 952290a differs from pull request most recent head 9277450. Consider uploading reports for the commit 9277450 to get more accurate results

@@             Coverage Diff              @@
##             master    #8881      +/-   ##
============================================
- Coverage     69.65%   69.62%   -0.04%
  Complexity     4679     4679
============================================
  Files          1806     1806
  Lines         94201    94202       +1
  Branches      14050    14050
============================================
- Hits          65620    65588      -32
- Misses        24021    24055      +34
+ Partials       4560     4559       -1

Flag
Coverage Δ

integration1
26.19% <0.00%> (-0.06%)
:arrow_down:

integration2
24.79% <0.00%> (-0.05%)
:arrow_down:

unittests1
66.36% <ø> (-0.02%)
:arrow_down:

unittests2
15.41% <0.00%> (+0.01%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...troller/helix/core/minion/ClusterInfoAccessor.java
79.31% <0.00%> (-2.84%)
:arrow_down:

...data/manager/realtime/DefaultSegmentCommitter.java
0.00% <0.00%> (-80.00%)
:arrow_down:

...ller/helix/core/minion/TaskTypeMetricsUpdater.java
80.00% <0.00%> (-20.00%)
:arrow_down:

...er/api/resources/LLCSegmentCompletionHandlers.java
43.56% <0.00%> (-18.82%)
:arrow_down:

...ore/query/scheduler/resources/ResourceManager.java
84.00% <0.00%> (-12.00%)
:arrow_down:

...data/manager/realtime/SegmentCommitterFactory.java
88.23% <0.00%> (-11.77%)
:arrow_down:

...roller/helix/core/relocation/SegmentRelocator.java
72.97% <0.00%> (-8.11%)
:arrow_down:

...altime/ServerSegmentCompletionProtocolHandler.java
51.88% <0.00%> (-6.61%)
:arrow_down:

.../pinot/core/query/scheduler/PriorityScheduler.java
78.08% <0.00%> (-5.48%)
:arrow_down:

.../filter/predicate/InPredicateEvaluatorFactory.java
70.45% <0.00%> (-4.55%)
:arrow_down:

... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 754e807...9277450. Read the comment docs.

authors: codecov-commenter zhtaoxiang


text:
[Pulsar Function Development]  java.lang.ClassNotFoundException: org.apache.pulsar.client.impl.schema.JSONSchema

Describe the bug
Develop a Pulsar function which contains the library "org.apache.pulsar.client.impl.schema.JSONSchema" in it. The pulsar function could be executed successfully under the localrun mode, but the system will show the error message "java.lang.NoClassDefFoundError: org/apache/pulsar/client/impl/schema/JSONSchema" while under the cluster mode.
To Reproduce
Steps to reproduce the behavior:

Prepare the pulsar function, object class and pom.xml;

Function:
package holystone.pulsar;

import org.apache.pulsar.functions.api.Context;
import org.apache.pulsar.functions.api.Function;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.PulsarClientException;
import org.apache.pulsar.client.impl.schema.JSONSchema;
import org.apache.pulsar.client.api.Producer;
import models.DeliveryNote;

   public class DeliverynoteDispatchingFucntion implements Function<DeliveryNote, String> {

	   String sd10_warehouse  = "persistent://demo/pulsar_function/dn-sd10";
	   String hh10_warehouse  = "persistent://demo/pulsar_function/dn-hh10";
	   String error 		  = "persistent://demo/pulsar_function/error";
	   String final_warehouse = "";

	public String process(DeliveryNote deliverynote, Context context) throws PulsarClientException {

		    String warehouse =  deliverynote.getWarehouse();

	    	switch(warehouse) {
	    	case "sd10":
	    		final_warehouse = sd10_warehouse;
	    		break;
	    	case "hh10":
	    		final_warehouse = hh10_warehouse;
	    		break;
	    	default:
	    		final_warehouse = error;
	    		break;
	    	}

	        PulsarClient pulsarClient = PulsarClient.builder()
	                                   .serviceUrl("pulsar://192.168.60.53:6650")
	                                   .build();

	  	    Producer<DeliveryNote> producer =
	  	       		pulsarClient.newProducer(JSONSchema.of(DeliveryNote.class))
	  	                        .topic(final_warehouse)
	  	                        .create();

	  	    producer.newMessage(JSONSchema.of(DeliveryNote.class)).value(deliverynote).send();
	 	    producer.flush();
	  	    producer.close();
	  	    pulsarClient.close();

	  	    String json_delivery_note = deliverynote.toString();

		    return json_delivery_note ;

	    }

	}

The DeliveryNote class is as follow
package models;

public class DeliveryNote   {

	private String deliveryno;
	private String materialno;
	private int    quantity;
	private String warehouse;

	public String getDeliveryno() {
		return deliveryno;
	}
	public void setDeliveryno(String deliveryno) {
		this.deliveryno = deliveryno;
	}
	public String getMaterialno() {
		return materialno;
	}
	public void setMaterialno(String materialno) {
		this.materialno = materialno;
	}
	public int getQuantity() {
		return quantity;
	}
	public void setQuantity(int quantity) {
		this.quantity = quantity;
	}
	public String getWarehouse() {
		return warehouse;
	}
	public void setWarehouse(String warehouse) {
		this.warehouse = warehouse;
	}

}

The POM.xml is as follow
<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>com.holystone.pulsar</groupId>
	<artifactId>DeliveryNoteDispatchingFunction</artifactId>
	<version>1.0</version>

	<properties>
		<pulsar.version>2.8.1</pulsar.version>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<maven.compiler.source>1.8</maven.compiler.source>
		<maven.compiler.target>1.8</maven.compiler.target>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.apache.pulsar</groupId>
			<artifactId>pulsar-client-original</artifactId>
			<version>${pulsar.version}</version>
		</dependency>

		<!-- https://mvnrepository.com/artifact/org.apache.pulsar/pulsar-functions-api -->
		<dependency>
			<groupId>org.apache.pulsar</groupId>
			<artifactId>pulsar-functions-api</artifactId>
			<version>${pulsar.version}</version>
		</dependency>

	</dependencies>

	<build>
		<!-- <defaultGoal>clean compile assembly:single</defaultGoal> -->
		<pluginManagement>
			<plugins>
				<plugin>
					<artifactId>maven-clean-plugin</artifactId>
					<version>3.1.0</version>
				</plugin>
				<plugin>
					<artifactId>maven-resources-plugin</artifactId>
					<version>3.0.2</version>
				</plugin>
				<plugin>
					<artifactId>maven-compiler-plugin</artifactId>
					<version>3.8.0</version>
					<configuration>
						<release>8</release>
					</configuration>
				</plugin>
				<plugin>
					<artifactId>maven-surefire-plugin</artifactId>
					<version>2.22.1</version>
				</plugin>
				<plugin>
					<artifactId>maven-jar-plugin</artifactId>
					<version>3.0.2</version>
				</plugin>
				<plugin>
					<artifactId>maven-install-plugin</artifactId>
					<version>2.5.2</version>
				</plugin>
				<plugin>
					<artifactId>maven-deploy-plugin</artifactId>
					<version>2.8.2</version>
				</plugin>

				<plugin>
					<artifactId>maven-site-plugin</artifactId>
					<version>3.7.1</version>
				</plugin>
				<plugin>
					<artifactId>maven-project-info-reports-plugin</artifactId>
					<version>3.0.0</version>
				</plugin>
				<plugin>
					<artifactId>maven-assembly-plugin</artifactId>
					<configuration>
						<finalName>DeliveryNoteDispatching</finalName>
						<appendAssemblyId>false</appendAssemblyId>
						<archive>
							<manifest>
								<mainClass>holystone.pulsar.DeliverynoteDispatchingFunction</mainClass>
							</manifest>
						</archive>
						<descriptorRefs>
							<descriptorRef>jar-with-dependencies</descriptorRef>
						</descriptorRefs>
					</configuration>
					<executions>
						<execution>
							<id>make-assembly</id>
							<phase>package</phase>
							<goals>
								<goal>assembly</goal>
							</goals>
						</execution>
					</executions>
				</plugin>
			</plugins>
		</pluginManagement>
	</build>
</project>

Function.txt
Class.txt
pom.txt

Package the developed function into the jar file  (DeliveryNoteDispatchingFunction-1.0.jar)

Copy the jar file to the $PULSAR_HOME/functions

Execute the command :

bin/pulsar-admin functions create --classname holystone.pulsar.DeliverynoteDispatchingFucntion --jar functions/DeliveryNoteDispatchingFunction-1.0.jar --inputs persistent://demo/pulsar_function/delivery_note --tenant demo --namespace pulsar_function --name dispatch --output persistent://demo/pulsar_function/debug

Get the status by executing the command :

bin/pulsar-admin functions getstatus --tenant demo --namespace pulsar_function --name dispatch.

the information is
{
  "numInstances" : 1,
  "numRunning" : 0,
  "instances" : [ {
    "instanceId" : 0,
    "status" : {
      "running" : false,
      "error" : "UNAVAILABLE: io exception",
      "numRestarts" : 0,
      "numReceived" : 0,
      "numSuccessfullyProcessed" : 0,
      "numUserExceptions" : 0,
      "latestUserExceptions" : [ ],
      "numSystemExceptions" : 0,
      "latestSystemExceptions" : [ ],
      "averageLatency" : 0.0,
      "lastInvocationTime" : 0,
      "workerId" : "tomcs"
    }
  } ]
}

Get the error from the  logs of the function and the error is shown as follow :

java.lang.NoClassDefFoundError: org/apache/pulsar/client/impl/schema/JSONSchema
        at holystone.pulsar.DeliverynoteDispatchingFucntion.process(DeliverynoteDispatchingFucntion.java:48) ~[DeliveryNoteDispatchingFunction-1.0.jar:?]
        at holystone.pulsar.DeliverynoteDispatchingFucntion.process(DeliverynoteDispatchingFucntion.java:1) ~[DeliveryNoteDispatchingFunction-1.0.jar:?]
        at org.apache.pulsar.functions.instance.JavaInstance.handleMessage(JavaInstance.java:95) ~[org.apache.pulsar-pulsar-functions-instance-2.8.1.jar:?]
        at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:271) [org.apache.pulsar-pulsar-functions-instance-2.8.1.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Caused by: java.lang.ClassNotFoundException: org.apache.pulsar.client.impl.schema.JSONSchema
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_292]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_292]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_292]
        ... 5 more

Expected behavior
The Pulsar function could be executed in the cluster mode (The function could be executed successfully under the localrun mode)
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Pulsar 2.8.1
Linux : VERSION="18.04.5 LTS (Bionic Beaver)"
Java version:  openjdk version "1.8.0_292"
Additional context
Add any other context about the problem here.

Should i also pack the jar which contains the class "org.apache.pulsar.client.impl.schema.JSONSchema" in my jar of pulsar-function

Hi , I know the issue

In the maven , I only use "package" in my Eclipse IDE
The correct goal will be "clean compile assembly:single"

Now, I could use JSONSchema in the pulsar function ! Thanks the help from the community

repo: apache/pulsar


text:
Flaky-test: Integration tests: PulsarStateTest#testSinkState

I have seen this temporary error on current master 6987d911897e66c384f8791535681f90887436b4
Additional context
Add any other context about the problem here.
pulsar-standalone-suite(org.apache.pulsar.tests.integration.functions.PulsarStateTest)  Time elapsed: 15.033 s  <<< FAILURE!
java.lang.AssertionError: expected [val1-9] but found [val1-7]
at org.testng.Assert.fail(Assert.java:99)
at org.testng.Assert.failNotEquals(Assert.java:1037)
at org.testng.Assert.assertEqualsImpl(Assert.java:140)
at org.testng.Assert.assertEquals(Assert.java:122)
at org.testng.Assert.assertEquals(Assert.java:629)
at org.testng.Assert.assertEquals(Assert.java:639)
at org.apache.pulsar.tests.integration.functions.PulsarStateTest.testSinkState(PulsarStateTest.java:182)

Closed as stale.
It should be resolved by #9870.

authors: eolivelli tisonkun


text:
[Build] Failed to build pulsar-build:manylinux-3.9

Describe the bug
PR #9389 introduces the python 3.9 client wheel supports. But it's will failed to build the base image pulsar-build:manylinux-3.9.
Logs:
Step 23/31 : RUN pip install twine
 ---> Running in 826d0f3ffa14
Collecting twine
  Downloading twine-3.3.0-py3-none-any.whl (34 kB)
Collecting rfc3986>=1.4.0
  Downloading rfc3986-1.4.0-py2.py3-none-any.whl (31 kB)
Collecting requests-toolbelt!=0.9.0,>=0.8.0
  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Requirement already satisfied: setuptools>=0.7.0 in /opt/_internal/cpython-3.9.2/lib/python3.9/site-packages (from twine) (54.1.1)
Collecting pkginfo>=1.4.2
  Downloading pkginfo-1.7.0-py2.py3-none-any.whl (25 kB)
Collecting requests>=2.20
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting readme-renderer>=21.0
  Downloading readme_renderer-29.0-py2.py3-none-any.whl (15 kB)
Collecting tqdm>=4.14
  Downloading tqdm-4.59.0-py2.py3-none-any.whl (74 kB)
Collecting keyring>=15.1
  Downloading keyring-23.0.0-py3-none-any.whl (32 kB)
Collecting colorama>=0.4.3
  Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting SecretStorage>=3.2
  Downloading SecretStorage-3.3.1-py3-none-any.whl (15 kB)
Collecting jeepney>=0.4.2
  Downloading jeepney-0.6.0-py3-none-any.whl (45 kB)
Collecting importlib-metadata>=3.6
  Downloading importlib_metadata-3.7.2-py3-none-any.whl (11 kB)
Collecting zipp>=0.5
  Downloading zipp-3.4.1-py3-none-any.whl (5.2 kB)
Collecting Pygments>=2.5.1
  Downloading Pygments-2.8.1-py3-none-any.whl (983 kB)
Collecting six
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting docutils>=0.13.1
  Downloading docutils-0.16-py2.py3-none-any.whl (548 kB)
Collecting bleach>=2.1.0
  Downloading bleach-3.3.0-py2.py3-none-any.whl (283 kB)
Collecting webencodings
  Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: packaging in /opt/_internal/cpython-3.9.2/lib/python3.9/site-packages (from bleach>=2.1.0->readme-renderer>=21.0->twine) (20.9)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting cryptography>=2.0
  Downloading cryptography-3.4.6.tar.gz (546 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting cffi>=1.12
  Using cached cffi-1.14.5-cp39-cp39-manylinux1_x86_64.whl (406 kB)
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/_internal/cpython-3.9.2/lib/python3.9/site-packages (from packaging->bleach>=2.1.0->readme-renderer>=21.0->twine) (2.4.7)
Building wheels for collected packages: cryptography
  Building wheel for cryptography (PEP 517): started
  Building wheel for cryptography (PEP 517): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /opt/_internal/cpython-3.9.2/bin/python3.9 /opt/_internal/cpython-3.9.2/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpjpj578gf
       cwd: /tmp/pip-install-vkopdabc/cryptography_145bc764456f4bf5bf46f276ebb0054e
  Complete output (161 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.9
  creating build/lib.linux-x86_64-3.9/cryptography
  copying src/cryptography/exceptions.py -> build/lib.linux-x86_64-3.9/cryptography
  copying src/cryptography/fernet.py -> build/lib.linux-x86_64-3.9/cryptography
  copying src/cryptography/utils.py -> build/lib.linux-x86_64-3.9/cryptography
  copying src/cryptography/__init__.py -> build/lib.linux-x86_64-3.9/cryptography
  copying src/cryptography/__about__.py -> build/lib.linux-x86_64-3.9/cryptography
  creating build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/general_name.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/name.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/ocsp.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/base.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/extensions.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/certificate_transparency.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  copying src/cryptography/x509/oid.py -> build/lib.linux-x86_64-3.9/cryptography/x509
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat
  copying src/cryptography/hazmat/_types.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
  copying src/cryptography/hazmat/_der.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
  copying src/cryptography/hazmat/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
  copying src/cryptography/hazmat/_oid.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/backends
  copying src/cryptography/hazmat/backends/interfaces.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends
  copying src/cryptography/hazmat/backends/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings
  copying src/cryptography/hazmat/bindings/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/_cipheralgorithm.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/keywrap.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/_asymmetric.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/padding.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/cmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/hashes.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/constant_time.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/_serialization.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/hmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  copying src/cryptography/hazmat/primitives/poly1305.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/ed25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/x448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/ciphers.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/x25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/cmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/hashes.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/backend.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/x509.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/ec.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/dsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/rsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/decode_asn1.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/hmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/ocsp.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/utils.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/ed448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/encode_asn1.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/aead.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/dh.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  copying src/cryptography/hazmat/backends/openssl/poly1305.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
  copying src/cryptography/hazmat/bindings/openssl/_conditional.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
  copying src/cryptography/hazmat/bindings/openssl/binding.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
  copying src/cryptography/hazmat/bindings/openssl/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/kbkdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/scrypt.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/hkdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/concatkdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/x963kdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  copying src/cryptography/hazmat/primitives/kdf/pbkdf2.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
  copying src/cryptography/hazmat/primitives/twofactor/hotp.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
  copying src/cryptography/hazmat/primitives/twofactor/totp.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
  copying src/cryptography/hazmat/primitives/twofactor/utils.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
  copying src/cryptography/hazmat/primitives/twofactor/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
  copying src/cryptography/hazmat/primitives/ciphers/algorithms.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
  copying src/cryptography/hazmat/primitives/ciphers/base.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
  copying src/cryptography/hazmat/primitives/ciphers/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
  copying src/cryptography/hazmat/primitives/ciphers/aead.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
  copying src/cryptography/hazmat/primitives/ciphers/modes.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/ed25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/x448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/x25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/padding.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/ec.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/dsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/rsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/utils.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/ed448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  copying src/cryptography/hazmat/primitives/asymmetric/dh.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
  creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
  copying src/cryptography/hazmat/primitives/serialization/pkcs12.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
  copying src/cryptography/hazmat/primitives/serialization/ssh.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
  copying src/cryptography/hazmat/primitives/serialization/base.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
  copying src/cryptography/hazmat/primitives/serialization/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
  copying src/cryptography/hazmat/primitives/serialization/pkcs7.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
  running egg_info
  writing src/cryptography.egg-info/PKG-INFO
  writing dependency_links to src/cryptography.egg-info/dependency_links.txt
  writing requirements to src/cryptography.egg-info/requires.txt
  writing top-level names to src/cryptography.egg-info/top_level.txt
  reading manifest file 'src/cryptography.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching 'docs/_build'
  warning: no previously-included files found matching 'vectors'
  warning: no previously-included files matching '*' found under directory 'vectors'
  warning: no previously-included files matching '*' found under directory '.github'
  warning: no previously-included files found matching 'release.py'
  warning: no previously-included files found matching '.coveragerc'
  warning: no previously-included files found matching 'codecov.yml'
  warning: no previously-included files found matching '.readthedocs.yml'
  warning: no previously-included files found matching 'dev-requirements.txt'
  warning: no previously-included files found matching 'tox.ini'
  warning: no previously-included files found matching 'mypy.ini'
  warning: no previously-included files matching '*' found under directory '.zuul.d'
  warning: no previously-included files matching '*' found under directory '.zuul.playbooks'
  writing manifest file 'src/cryptography.egg-info/SOURCES.txt'
  copying src/cryptography/py.typed -> build/lib.linux-x86_64-3.9/cryptography
  running build_ext
  generating cffi module 'build/temp.linux-x86_64-3.9/_padding.c'
  creating build/temp.linux-x86_64-3.9
  generating cffi module 'build/temp.linux-x86_64-3.9/_openssl.c'
  running build_rust

      =============================DEBUG ASSISTANCE=============================
      If you are seeing a compilation error please try the following steps to
      successfully install cryptography:
      1) Upgrade to the latest pip and try again. This will fix errors for most
         users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
      2) Read https://cryptography.io/en/latest/installation.html for specific
         instructions for your platform.
      3) Check our frequently asked questions for more information:
         https://cryptography.io/en/latest/faq.html
      4) Ensure you have a recent Rust toolchain installed:
         https://cryptography.io/en/latest/installation.html#rust
      5) If you are experiencing issues with Rust for *this release only* you may
         set the environment variable `CRYPTOGRAPHY_DONT_BUILD_RUST=1`.
      =============================DEBUG ASSISTANCE=============================

  error: can't find Rust compiler

  If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.

  To update pip, run:

      pip install --upgrade pip

  and then retry package installation.

  If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.

  This package requires Rust >=1.41.0.
  ----------------------------------------
  ERROR: Failed building wheel for cryptography
Failed to build cryptography
ERROR: Could not build wheels for cryptography which use PEP 517 and cannot be installed directly
The command '/bin/sh -c pip install twine' returned a non-zero code: 1

reproduce

checkout to the master code,
cd pulsar/pulsar-client-cpp/create-images.sh

I removed the python3.9 wheel file build temporarily so we can build the master branch successfully. After the pulsar-build:manylinux-3.9 build out I will add the python 3.9 build back

@zymap  Can we add back the python 3.9 support...please :)
We need it so much 👍

Sure. I will take a look ASAP.

This appears to be due to cryptography dropping its manylinux1 builds. They do produce manylinux2010 and manylinux2014 wheels.
Per https://github.com/pypa/manylinux,

PEP 513 defined manylinux1_x86_64 and manylinux1_i686 platform tags and the wheels were built on Centos5. Centos5 reached End of Life (EOL) on March 31st, 2017.

The simplest fix is likely to bump https://github.com/apache/pulsar/blob/f800fde9f27b2e14994f822b8ddac4d01583fb08/pulsar-client-cpp/docker/Dockerfile#L21 to manylinux2010 (although I haven't tried that personally). However, that image doesn't have any python2 interpreters.
If supporting python2 is still important, one option may be to continue to produce manylinux1 wheels for 2.7, and use manylinux2010 for the newer interpreters

authors: alexku7 jeffreytolar zymap


text:
[improve][common] Add thread name for RateLimiter.

Contribution Checklist
Motivation
Add thread name for RateLimiter
Documentation

[x] doc-not-needed

Matching PR in forked repository
PR in forked repository: https://github.com/Technoboy-/pulsar/pull/11

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@7e420c6). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #18013   +/-   ##
=========================================
  Coverage          ?   27.37%
  Complexity        ?     3568
=========================================
  Files             ?      393
  Lines             ?    43418
  Branches          ?     4462
=========================================
  Hits              ?    11884
  Misses            ?    29684
  Partials          ?     1850

Flag
Coverage Δ

unittests
27.37% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

authors: Technoboy- codecov-commenter


text:
[Issue 7675][Pulsar SQL] Presto connector OffloadPolicies deserialization

Master Issue: #7675
Motivation
When using the pulsar-presto connector as a plugin for an existing presto cluster, presto fails to create PulsarSplits.
Modifications

The OffloadPolicy is passed as a String to the PulsarSplit.
The OffloadPolicy json String is converted to Parameters and passed to OffloadPolicy::create
com.fasterxml.jackson.core:jackson-annotations jars are removed from the pulsar-presto-connector tarball per PrestoSQL documentation.
A configured ManagedLedgerConfig is used when finding the bounds in getPredicatePushdownInfo
The offloaderDirectory in the PulsarConnectorConfig is always used when searching for offloaders

@wolfstudy I think that this is a continuation on the work that you did in #7701

/pulsarbot run-failure-checks

Hi @enriched  :)
Any updates about that issue ?
It just prevents us today to work with offloading and presto :(

@alexku7 I have built this locally and am able to use it as a plugin for presto v332 with pulsar offloading enabled to S3 buckets. I just have not found time yet to fix the tests.

Thanks @enriched
Let me understand. will it work for me  in the 2.6.2 version even without that fix since the version 2.6.2 has the updated v332 version of presto-sql ?

@alexku7 I have added a the pulsar-presto-connector built from this source to a standard prestosql v332 installation (from https://repo1.maven.org/maven2/io/prestosql/presto-server/332/presto-server-332.tar.gz). That presto instance is able to connect to a pulsar broker running the streamnative/pulsar-all:2.6.0 docker image and query data in offloaded ledgers.

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

@enriched Seems the failed CI tests related to this change, could you please help take a look?

Sorry, not pay attention to this PR, I also added a PR to fix the problem. refer to #9300, PTAL.

@enriched Could you please rebase the code?

authors: alexku7 codelipenghui enriched gaoran10 sijie zymap


text:
NO-JIRA: create .asf.yaml

https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features

can you please squash the changes

authors: gemmellr jiridanek


text:
Fix CONSUME_FROM_LAST_OFFSET mode may pull data from 0L

Problem View
When we set CONSUME_FROM_LAST_OFFSET to pull data, if it is the first time to start the consumer group, data may be pulled from the 0L. This is not what we expected.
I built a demo to reproduce this situation.
First sent the data using the producer：

Set consumer mode as CONSUME_FROM_LAST_OFFSET.

At the first startup, the consumption situation obtained should theoretically not pull the previous message, but the message is still pulled from the beginning:

Location the problem
When a consumer group needs to pull messages, it obtains the initial location to be consumed according to the startup mode.

In cluster mode, the offset is read from the broker side. (cluster mode has the above problem)
In broadcast mode, the offset of the local record is read, and -1 is returned if it is not read.
In broker, in  ConsumerManageProcessor#queryConsumerOffset, the broker will query the consumeoffset by consumegroup, topic and queueId.
● If the offsetTable records the consumption information of this consumer group under this topic and this queueId, return this recorded consumeoffset
● If there is no record in the offsetTable, the minimum offset in the consumequeue will be obtained. (When the minimum offset is less than or equal to 0 && offset 0 is in memory, it returns to 0) -> This leads that when consumequeue does not clean up expired files, the minimum offset is 0 and it is in memory , 0 will be returned, and the client receives an offset of 0, so it starts to consume from the starting position.

Solution
When querying consumeoffset, set ZeroIfNotFound false in the requestHeader.
So, when the previous record is not obtained, it will no longer return 0.

Another Question
The above solution can solve this problem, but I still have some doubts about the original implementation method
In  ConsumerManageProcessor#queryConsumerOffset，why we need to set offset as 0L, when minOffset <= 0 and offset 0L is in memory. Can we return response just setCode as ResponseCode.QUERY_NOT_FOUND?

@duhenglucky @RongtongJin I'm also confused about this question.  Can you explain it?

Another Question
The above solution can solve this problem, but I still have some doubts about the original implementation method
In ConsumerManageProcessor#queryConsumerOffset，why we need to set offset as 0L, when minOffset <= 0 and offset 0L is in memory. Can we return response just setCode as ResponseCode.QUERY_NOT_FOUND?

Does it related to topic auto create?

authors: shengminw zhanghaou

repo: apache/rocketmq