text:
ORC-1378: [Java] Add slf4j impl to avoid warning message in example module

What changes were proposed in this pull request?
This PR aims to add slf4j-impl to avoid warning message in example module.
Why are the changes needed?
When we run orc java examples, the following error will show in output:
java -cp ./examples/target/orc-examples-1.9.0-SNAPSHOT-uber.jar org.apache.orc.examples.CoreWriter

SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info

After #1132, we use slf4j-simple instead of log4j. This PR remove the log4j dependecies and add slf4j-simple.
This one is very similar to https://github.com/apache/orc/pull/1266.
After the PR:
java -cp ./examples/target/orc-examples-1.9.0-SNAPSHOT-uber.jar  org.apache.orc.examples.CoreWriter

org.apache.orc.examples.CoreWriter
Feb 23, 2023 7:03:48 PM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[main] INFO org.apache.orc.impl.HadoopShimsCurrent - Can't get KeyProvider for ORC encryption from hadoop.security.key.provider.path.
[main] INFO org.apache.orc.impl.PhysicalFsWriter - ORC writer created for path: my-file.orc with stripeSize: 67108864 blockSize: 268435456 compression: Compress: ZLIB buffer: 262144
[main] INFO org.apache.orc.impl.WriterImpl - ORC writer created for path: my-file.orc with stripeSize: 67108864 options: Compress: ZLIB buffer: 262144

How was this patch tested?
manual testing.

Hi @zabetak, thank you for your suggestion, I update the PR. If you have time please take a look thanks~

cc @dongjoon-hyun @guiyanakuang

Sorry, I made a mistake. Fixed.

Sorry, I made a mistake. Fixed.

It's okay, other modules did it this way before. It's just that now I think @zabetak suggestion is better.

https://github.com/guiyanakuang/orc/commit/66675a9bddecb76e3d19c4dfdf8934244ffe0e03
@deshanxiao , I made some attempts based on your pull request, what do you think about it.
cc @zabetak

guiyanakuang@66675a9
@deshanxiao , I made some attempts based on your pull request, what do you think about it. cc @zabetak

+1 for the change, Thank you. Please let me know if you think this deserves a new PR

If we remove the slf4j-simple from my PR, the slf4j may not be able to find the corresponding implementation when executing. This may have a warn log in the PR

I understand  @zabetak means to set it to runtime:
If there is a specific implementation class in the compiler phase, there will be a compilation error.

@deshanxiao Please continue with the PR.

The advantage of setting it to runtime is to avoid transitive dependencies, as projects that depend on Orc have their own logging implementation. We only set it in the parent pom.xml to avoid setting it in each submodule, and this way the dependencies won't be as tightly coupled.

The advantage of setting it to runtime is to avoid transitive dependencies, as projects that depend on Orc have their own logging implementation. We only set it in the parent pom.xml to avoid setting it in each submodule, and this way the dependencies won't be as tightly coupled.

I found that there are two slf4j-simple dependencies in the pom file, one in dependency and one in dependencymanagement. I remove the dependencymanagement one.
If we add dependencies in parent pom. All modules (tools bench example) that depend on the pom.xml file will automatically inherit this dependency

I think it would be simpler if I would show you what I had in mind:
https://github.com/zabetak/orc/tree/ORC-1378

The current PR also contains some files that shouldn't be there:

.my-file.orf.crc
my-file.orc
I somehow missed to spot them in the first pass.

yeah, It is good to fix git ignore to solve the problem.

If we add dependencies in parent pom (not dependencymanagement). All modules (tools bench example) that depend on the pom.xml file will automatically inherit this dependency.

Note that adding the dependency in the global dependency section it will also lead to many log (INFO) messages appearing when running tests; not sure if this is desired.

I think it would be simpler if I would show you what I had in mind: https://github.com/zabetak/orc/tree/ORC-1378

@guiyanakuang @zabetak The biggest difference between this PR and @zabetak 's PR ’is whether to add dependency (not dependency management) to pom.

I think it would be simpler if I would show you what I had in mind: https://github.com/zabetak/orc/tree/ORC-1378

@guiyanakuang @zabetak The biggest difference between this PR and @zabetak 's PR ’is whether to add dependency (not dependency management) to pom.

Sorry, in the example I provided, I added the slf4j-simple with the runtime scope in the parent pom.xml dependencies instead of the dependency management section. I now realize that this could be harmful. Before this change, orc-core did not transitively pass any log implementation jar. After adding slf4j-simple to the parent pom.xml dependencies, it will transitively pass it, which may cause a log library conflict for projects that depend on orc-core. Therefore, I think @zabetak's approach is better.
Additionally, we can add slf4j-simple with the provided scope to orc-core, which can avoid transitive dependencies and also enable logging output when testing orc-core.

Thank you @zabetak. I have merged your commit and add provicde scope in orc-core.

Test has passed:

@guiyanakuang @zabetak I have changed the title and description quoted by @zabetak. Please take a look.
This PR should own to @zabetak.
Thank you very much @guiyanakuang @zabetak

@deshanxiao , please modify the PR title to match the content. Because this PR not only solves the problem in the example module. Perhaps the title should be "Optimize logging dependency definition for ORC project."

cc @dongjoon-hyun

I don't think the following is a precise claim because #1132 only changed bench modules which is not published to the Maven Central. It's just one of the test module which is irrelevant to the example module.

When we run orc java examples, the following error will show in output:
...
After #1132, we use slf4j-simple instead of log4j.

Could you revise the PR description a little more? Or, did I miss something?

I don't think the following is a precise claim because #1132 only changed bench modules which is not published to the Maven Central. It's just one of the test module which is irrelevant to the example module.

When we run orc java examples, the following error will show in output:
...
After #1132, we use slf4j-simple instead of log4j.

Could you revise the PR description a little more? Or, did I miss something?

Thank you @dongjoon-hyun.  You are right, let me fix it.

@zabetak Thank you for the reminder. You're right. At that time, I found that only orc-core was unable to output log information during testing (because other modules already had runtime simple-slf4j dependencies). Using "test" instead of "provided" is better. I'm sorry for providing misleading information.

@zabetak @guiyanakuang
The following is the latest slf4j implementation class of each module:

orc-core : slf4j-simple:jar:2.0.6:test
orc-tools : slf4j-simple:jar:2.0.6:runtime
orc-examples : slf4j-simple:jar:2.0.6:runtime

Thank you @dongjoon-hyun

@dongjoon-hyun @guiyanakuang @zabetak Please let me know if you have any comments. Thank you!

@deshanxiao thank you for your efforts, thank you @zabetak and @dongjoon-hyun for the review.  I just merged it into the main branch.

authors: deshanxiao guiyanakuang zabetak

repo: apache/orc


text:
[C++] read encrypted ORC file

What changes were proposed in this pull request?
you can use C++ version code to read encrypted ORC files
ORC file structure：links
Why are the changes needed?
The C++ version is not yet able to read encrypted ORC files now
How was this patch tested?
orc::RowReaderOptions rowReaderOptions;
orc::ReaderOptions readerOpts;
std::shared_ptr<orc::InMemoryKeystore> keyStore = std::make_shared<orc::InMemoryKeystore>();
std::string passKey ="secret123";
keyStore->addKey("pii", 0, orc::EncryptionAlgorithm::AES_CTR_128,passKey);
readerOpts.setKeyProvider(keyStore);
printContents("/root/encryption.orc",readerOpts, rowReaderOptions);

Thanks @zxf216! Let me take a first pass. cc @coderex2522 @ffacs

Thank you for making a PR, @zxf216 .
We have been tracking this via

Add C++ API for Columnar Encryption #1183

BTW, could you make the CI happy by fixing all failures?
Okay

I converted it to Draft because CIs fails still.

Gentle ping, @zxf216 .

source: gharchive/pull-request

authors: dongjoon-hyun wgtmac zxf216

license_type: permissive

license_source: bigquery


text:
HDDS-4494. Provide Guice module to inject configuration to annotated fields

What changes were proposed in this pull request?
hadoop-hdds/config project provides a light-weight annotation based configuration interface. It supports to create an object with injection:
ReplicationConfig replicationConfig := ozoneConfig.getObject(replicationConfig);

However, it seems to be hard to inject configuration to the service itself as usually we inject a lot of other dependencies to the constructor, not just the configuration.
One possible solution is using a Guice module. Guice is already used by recon, and this patch adds some optional modules to inject configuration variables to any service / instance if required.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4494
How was this patch tested?
New utility is not yet used by production code, but covered with a new unit test.

Two months and 3 days without any answer.
It seems that nobody is interested to check or comment this PR.
I am closing it now.

repo: apache/ozone

license_source: github-api


text:
HDDS-4995. Skip CI for draft pull requests

What changes were proposed in this pull request?
Github pull requests can be created as (or converted to) draft. This is useful for gathering early feedback from other contributors about proof-of-concept code. Such code may not even compile, and style issues are usually ignored.
Since further commits are generally expected in this case, I think it makes sense to skip CI build/test until the PR is marked as "ready for review".  The goal is to save CI cycles for other issues.
If CI feedback is desired, the author should enable workflows in their fork so we can check push builds there.
https://issues.apache.org/jira/browse/HDDS-4995
How was this patch tested?
Tested with a draft PR in my fork: https://github.com/adoroszlai/hadoop-ozone/pull/12
Builds:

commit 1

push: https://github.com/adoroszlai/hadoop-ozone/actions/runs/665327555 (cancelled manually)
PR: https://github.com/adoroszlai/hadoop-ozone/actions/runs/665361303

commit 2

push: https://github.com/adoroszlai/hadoop-ozone/actions/runs/665365286 (cancelled manually)
PR: https://github.com/adoroszlai/hadoop-ozone/actions/runs/665366214

"ready for review"

https://github.com/adoroszlai/hadoop-ozone/actions/runs/665370947 (cancelled manually)

Also with this PR:

draft: https://github.com/apache/ozone/actions/runs/665634620
"ready for review": https://github.com/apache/ozone/actions/runs/665636793

@adoroszlai , how can github actions be enabled on forks ?

@adoroszlai , how can github actions be enabled on forks ?

https://github.com/mukul1987/hadoop-ozone/actions/workflows/post-commit.yml - I think you as repo owner should see a button: "Enable workflow".

Thanks @dineshchitlangia and @elek for the review.


text:
HDDS-11183. Keys from DeletedTable and DeletedDirTable of AOS should be deleted on batch operation while creating a snapshot

What changes were proposed in this pull request?
Keys from DeletedTable and DeletedDirTable of AOS should be deleted on batch operation while creating a snapshot and not directly from the table. The transactions have to be flushed to delete the keys. If the transaction is not flushed then the deleted keys wouldn't be present in the next transaction replay leading to having unreferenced blocks in the system.
Scenario:
Snapshot S1 create is fired.
create S1 transaction has not been flushed yet, but it has created the snapshot directory and deleted the entries from the table.
This OM has received a bootstrap request from another OM. The bootstrap command will only copy the OM instance and the snapshots that have been flushed to the AOS. S1 will not be a part of it.
The boostrapped OM will replay unflushed transactions(Create S1 will be a part of it). Now it will create a checkpoint out of the bootstrapped AOS which doesn’t have the delete entries. Now the snapshot instance will also not have the deletedKey entries
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11183
How was this patch tested?
Existing unit tests and addition of unit tests.

Thanks @swamirishi for the patch. The idea makes sense to me. It should have used putWithBatch() in the first place.

https://github.com/apache/ozone/actions/runs/9964095152/job/27532079656
Error:  Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 180.071 s <<< FAILURE! - in org.apache.hadoop.ozone.om.snapshot.TestSnapshotBackgroundServices
Error:  org.apache.hadoop.ozone.om.snapshot.TestSnapshotBackgroundServices.testSnapshotAndKeyDeletionBackgroundServices  Time elapsed: 41.556 s  <<< ERROR!
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: Cannot invoke "org.apache.ratis.server.protocol.TermIndex.toProto()" because the return value of "org.apache.ratis.server.raftlog.RaftLog.getTermIndex(long)" is null
	at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268)
	at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249)
	at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167)
	at org.apache.ratis.proto.grpc.AdminProtocolServiceGrpc$AdminProtocolServiceBlockingStub.groupInfo(AdminProtocolServiceGrpc.java:496)
...

Thanks @swamirishi for the patch and @smengcl for the review.

authors: hemantk-12 smengcl swamirishi


text:
feat: add catalog API

fix: #52

Also invite @Aitozi for a reivew.

authors: QuakeWang Xuanwo

repo: apache/paimon-rust


text:
[Bug] Tag createTime is null

Search before asking

[X] I searched in the issues and found nothing similar.

Paimon version
0.8.0
Compute Engine
spark
Minimal reproduce step
auto create tag using process-time

What doesn't meet your expectations?
Tag createTime should always be set.
Anything else?
No response
Are you willing to submit a PR?

[X] I'm willing to submit a PR!

Hi @wForget , this is because we need to be compatible to old readers. Very regretful, the JsonIgnoreProperties(ignoreUnknown = true) not be set in Snapshot for old versions.

Hi @wForget , I closed #3557, we can discuss here.

Hi @wForget , this is because we need to be compatible to old readers. Very regretful, the JsonIgnoreProperties(ignoreUnknown = true) not be set in Snapshot for old versions.

Thanks for the explanation, but I still don't understand why this is blocking, it is always possible to write to the new added tagTimeRetained and tagCreateTime fields.

@wForget

New writer write new tag, there is a tagCreateTime field.
Old reader read this tag, failed, because of json unknown field exception.

I add this comments:
// When timeRetained is not defined, please do not write the tagCreatorTime field,
// as this will cause older versions (<= 0.7) of readers to be unable to read this
// tag.
// When timeRetained is defined, it is fine, because timeRetained is the new
// feature.

@wForget

New writer write new tag, there is a tagCreateTime field.
Old reader read this tag, failed, because of json unknown field exception.

Got it, thanks.

source: gharchive/issue

authors: JingsongLi wForget

repo: apache/paimon


text:
[cdc] paimon cdc database sync support computed columns

Purpose
Currently, there are still many paimon users who need to support computed columns when synchronizing multiple tables in the entire database. This PR will implement this feature.
If the referenced columned column exists in the table, the columned column function will be triggered. For example,
Table A column:
col1 col2 col3 create_sys_tm

Table B column:
col1 col4 col4 create_sys_tm

Define the calculated function:
--computed_column="_substring=substring(col3,2), pt=date_format(create_sys_tm,yyyyMMdd)"

The final table structure is as follows:
Table A column:
col1 col2 col3 _substring pt

Table B column:
col1 col4 col4 pt

The main changes of PR:
Move the implementation of xxxRecordParser#evalComputedColumns down to RichCdcMultiplexRecordEventParser#evalComputedColumns
Tests
MySqlSyncDatabaseTableListITCase#testComputedColumn
MongoDBSyncDatabaseActionITCase#testComputedColumn
KafkaOggSyncDatabaseActionITCase#testComputedColumn
KafkaMaxwellSyncDatabaseActionITCase#testComputedColumn
KafkaCanalSyncDatabaseActionITCase#testComputedColumn
API and Format

Documentation

@JingsongLi @yuzelin @zhuangchong  can you help review this pr? tks~


text:
Remove hive modules

Remove the hive modules as discusses in the Parquet sync.
Reporter: Fokko Driesprong / @Fokko
Assignee: Fokko Driesprong / @Fokko
Related issues:

Remove Unused Modules  (is a child of)

PRs and other links:

GitHub Pull Request #691

Note: This issue was originally created as PARQUET-1676. Please see the migration documentation for further details.

Gabor Szadovszky / @gszadovszky:
The parent PARQUET-1666 is targeted to 1.12.0. Is it really targeted to 1.11.0?

Fokko Driesprong / @Fokko:
Great catch @gszadovszky I'm fine with targeting this for 1.12

Gabor Szadovszky / @gszadovszky:
I've removed Hive modules in one step under PARQUET-1666.

Gabor Szadovszky / @gszadovszky:
@Fokko, any updates on this?

Gabor Szadovszky / @gszadovszky:
@Fokko, do you want to pick this up for 1.12.0?

repo: apache/parquet-java


text:
Need to add documentation on how to specify avro schema inline for realtime ingestion config

If user wants to use the Avro message decoder for e.g below config
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.consumer.type": "lowlevel",
"stream.kafka.topic.name": "",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.SimpleAvroMessageDecoder",
"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.broker.list": "",
"stream.kafka.consumer.prop.auto.offset.reset": "largest"
}
then it will throw an error: "the avro schema must be provided", however there is no documentation on how to do that.
Relevant field is "stream.kafka.decoder.prop.schema", please document this.

@mneedham is this something you can help with ? cc @mayankshriv

repo: apache/pinot


text:
Allow moveToFinalLocation in METADATA push based on config

METADATA push didn't allow the option of moveSegmentToFinalLocation. This meant that if someone had generated segments in a location that was not the deep store, there was absolutely no way to move those segments into deep store without manual scripting.
Chatting with @xiangfu0 , I understood that this was done on purpose, in order to keep METADATA push very light weight and not involve copying from outputDir to dataDir in the push. But this is a very valid scenario where users would want to generate segments in a location other than deep store, and still want to use METADATA push (reasons could be multiple, as listed in https://github.com/apache/pinot/issues/7328).
While there have been more elaborate solutions proposed in the issue above (like periodic background tasks), this PR simply enables one to allow METADATA push to so moveToFinalLocation. And while this would increase the data push time, it is a better solution than asking users to switch to SegmentTarPush or SegmentUriPush if the issue was disparate permissions between outputDir ad dataDir (one of the reasons mentioned in the issue).
Backward compatible behavior or this being false by default is maintained.

Codecov Report

Merging #8823 (9e6cea8) into master (07b3ee6) will decrease coverage by 35.90%.
The diff coverage is 47.61%.

@@              Coverage Diff              @@
##             master    #8823       +/-   ##
=============================================
- Coverage     62.93%   27.03%   -35.91%
+ Complexity     4609        1     -4608
=============================================
  Files          1689     1729       +40
  Lines         89307    91133     +1826
  Branches      13415    13636      +221
=============================================
- Hits          56209    24636    -31573
- Misses        29053    64158    +35105
+ Partials       4045     2339     -1706

Flag
Coverage Δ

integration1
27.03% <47.61%> (?)

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...e/pinot/common/utils/FileUploadDownloadClient.java
46.28% <ø> (+28.00%)
:arrow_up:

...he/pinot/segment/local/utils/SegmentPushUtils.java
0.00% <0.00%> (-12.71%)
:arrow_down:

...he/pinot/spi/ingestion/batch/spec/PushJobSpec.java
0.00% <0.00%> (-59.10%)
:arrow_down:

...ces/PinotSegmentUploadDownloadRestletResource.java
50.00% <52.63%> (+9.07%)
:arrow_up:

...apache/pinot/controller/api/upload/ZKOperator.java
55.78% <58.82%> (-3.93%)
:arrow_down:

...in/java/org/apache/pinot/spi/utils/StringUtil.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/org/apache/pinot/spi/utils/BooleanUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/spi/trace/BaseRecording.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/spi/trace/NoOpRecording.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/config/table/FSTType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 1402 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 07b3ee6...9e6cea8. Read the comment docs.

authors: codecov-commenter npawar


text:
Add healthcheck metric to pinot-server

What does this PR do?
This code allows pinot-server emit metric based upon health check results similar to pinot-controller (see PinotControllerHealthCheck#L84.
Two meters were added in correspondence with pinot-controller meters:

healthcheckOkCalls
healthcheckBadCalls

How was the code tested?

Unit tests have passed
Verified cluster restart

I couldn't apply labels due to lack of permissions, would appreciate someone applying "feature" or "enhancement" as necessary.

Codecov Report

Merging #9168 (0d5155f) into master (22843c6) will decrease coverage by 43.72%.
The diff coverage is 60.00%.

@@              Coverage Diff              @@
##             master    #9168       +/-   ##
=============================================
- Coverage     69.98%   26.25%   -43.73%
+ Complexity     4759       44     -4715
=============================================
  Files          1847     1835       -12
  Lines         98535    98187      -348
  Branches      14961    14923       -38
=============================================
- Hits          68959    25779    -43180
- Misses        24732    69850    +45118
+ Partials       4844     2558     -2286

Flag
Coverage Δ

integration1
26.25% <60.00%> (+0.02%)
:arrow_up:

integration2
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...inot/server/api/resources/HealthCheckResource.java
0.00% <0.00%> (ø)

...a/org/apache/pinot/common/metrics/ServerMeter.java
100.00% <100.00%> (ø)

...inot/server/starter/helix/AdminApiApplication.java
85.48% <100.00%> (+0.23%)
:arrow_up:

...in/java/org/apache/pinot/spi/utils/BytesUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/org/apache/pinot/spi/utils/BooleanUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/spi/trace/BaseRecording.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/spi/trace/NoOpRecording.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/config/table/FSTType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/config/user/RoleType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/data/MetricFieldSpec.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 1346 more

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

authors: aidar-st codecov-commenter


text:
[multistage] [bugfix] fix agg hash collision bug

Fix the bug where group by key hashcode has collision, we will produce wrong  agg result.
I'll follow up with a test.
This fixes #9675

lgtm please fix linter and check ci

Done. I reverted my refactoring for easier review.

Codecov Report

Merging #9676 (6373423) into master (229c55e) will increase coverage by 4.51%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master    #9676      +/-   ##
============================================
+ Coverage     63.02%   67.53%   +4.51%
+ Complexity     5229     5169      -60
============================================
  Files          1938     1447     -491
  Lines        104130    75795   -28335
  Branches      15780    12070    -3710
============================================
- Hits          65625    51188   -14437
+ Misses        33640    20947   -12693
+ Partials       4865     3660    -1205

Flag
Coverage Δ

integration1
?

integration2
?

unittests1
67.53% <100.00%> (-0.04%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

.../pinot/query/runtime/blocks/TransferableBlock.java
70.45% <ø> (+3.06%)
:arrow_up:

...inot/query/runtime/operator/AggregateOperator.java
85.00% <100.00%> (+0.78%)
:arrow_up:

.../pinot/query/runtime/plan/PhysicalPlanVisitor.java
96.87% <100.00%> (ø)

...va/org/apache/pinot/core/routing/RoutingTable.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...va/org/apache/pinot/common/config/NettyConfig.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...a/org/apache/pinot/common/metrics/MinionMeter.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...g/apache/pinot/common/metrics/ControllerMeter.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../apache/pinot/common/metrics/BrokerQueryPhase.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../apache/pinot/common/metrics/MinionQueryPhase.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ache/pinot/server/access/AccessControlFactory.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 742 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

oops. merged as part of #9729 already . closing this one

authors: 61yao codecov-commenter walterddr


text:
Waiting for ack

I'm trying to consume 1 message at a time, and i'm acknowledging the message outside of the listener. But it's just grabbing the next message without waiting for the ack.
code sample:
let msg;
let msgConsumer;

const consumer = client.subscribe({
                    topic: `persistent://public/default/test1`,
                    subscription: `sub1`,
                    subscriptionType: 'Exclusive',
                    receiverQueueSize:0,
                    ackTimeoutMs:300000,
                    listener: (msg,msgConsumer) => {
                        console.log(msg.getData().toString());
                        msg = msg;
                        msgConsumer = msgConsumer;

                    }
                  });

function someWhereElse(){

    msgConsumer.acknowledge(msg);

}

Is something like this possible?

Since listener is used for asynchronous message processing, it doesn't seem to meet your requirement (1 message at a time).

Why are you using listener?
Could you try receive() instead? e.g.

const consumer = client.subscribe( /* without listener */ );

async function somewhereElse() {
  const msg = await consumer.receive();
  consumer.acknowledge(msg);
}

I'm not receiving in the same place I'm acknowledging. I'm trying to find a way to make 1 process add jobs to queues, and acknowledge them, and the other process needs to just process 1 at a time and wait for acknowledgement.

process 1 adds a message to the queue.
process 2 starts processing message
process 1 acknowledges message
process 2 can receive next message

if a second message gets added to the queue, it has to wait for the first message to be acknowledged,
I haven't found a way to make this work with the manual receive(). But I'm new to queueing so maybe I'm looking over something.

I'm trying to find a way to make 1 process add jobs to queues, and acknowledge them, and the other process needs to just process 1 at a time and wait for acknowledgement.

It seems we can't implement it with only the functionality of pulsar-client.

authors: equanz kimula taefed

repo: apache/pulsar-client-node


text:
[Bug] pulsarctl library does not have any option of backlog type option.

Search before asking

[X] I searched in the issues and found nothing similar.

Version
v2.8.3.2
https://github.com/streamnative/pulsarctl/releases/download/v2.8.3.2/pulsarctl-amd64-linux.tar.gz
Minimal reproduce step
/home/pulsar/pulsarctl-amd64-linux/pulsarctl namespaces set-backlog-quota --limit-size 1G --limit-time  3600 type "message age" --policy producer_request_hold <namespace_name> --token  --admin-service-url  --tls-allow-insecure
What did you expect to see?
Backlog has to set with the corresponding topic.
What did you see instead?

Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

Refer https://github.com/streamnative/pulsarctl

https://github.com/streamnative/pulsarctl/issues/914

Closed as superseded by https://github.com/streamnative/pulsarctl/issues/914.

authors: jyoti112 mattisonchao tisonkun

repo: apache/pulsar


text:
Flaky-test: PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer

Search before asking

[X] I searched in the issues and found nothing similar.

Example failures

2023-09-29T17:36:04.3978122Z
2023-09-27T06:21:08.7783211Z
2023-09-25T09:49:23.6018203Z

Exception stacktrace
java.lang.AssertionError: expected object to not be null
	at org.testng.Assert.fail(Assert.java:110)
	at org.testng.Assert.assertNotNull(Assert.java:1483)
	at org.testng.Assert.assertNotNull(Assert.java:1467)
	at org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer(PersistentDispatcherFailoverConsumerTest.java:379)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

Flaky-test: org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer
Number of failures: 11
org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest is flaky. The testAddRemoveConsumer test method fails sporadically.
java.lang.AssertionError: expected object to not be null
	at org.testng.Assert.fail(Assert.java:110)
	at org.testng.Assert.assertNotNull(Assert.java:1483)
	at org.testng.Assert.assertNotNull(Assert.java:1467)
	at org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer(PersistentDispatcherFailoverConsumerTest.java:380)

Usage tip: To enable automatic navigation to failure message, open the following links with CTRL/CMD-click.
example failure 2023-11-04T22:14:28.4440428Z
example failure 2023-11-03T22:48:36.2722672Z
example failure 2023-11-03T01:34:07.0048022Z
example failure 2023-11-02T11:52:27.8191887Z

Full exception stacktrace

java.lang.AssertionError: expected object to not be null
	at org.testng.Assert.fail(Assert.java:110)
	at org.testng.Assert.assertNotNull(Assert.java:1483)
	at org.testng.Assert.assertNotNull(Assert.java:1467)
	at org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer(PersistentDispatcherFailoverConsumerTest.java:380)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

example failure 2023-11-02T06:24:37.9019343Z
example failure 2023-11-01T22:24:59.2165302Z
example failure 2023-11-01T19:48:30.9955372Z
example failure 2023-10-30T06:24:49.6468755Z
example failure 2023-10-30T05:40:12.4651414Z

Full exception stacktrace

java.lang.AssertionError: expected object to not be null
	at org.testng.Assert.fail(Assert.java:110)
	at org.testng.Assert.assertNotNull(Assert.java:1483)
	at org.testng.Assert.assertNotNull(Assert.java:1467)
	at org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer(PersistentDispatcherFailoverConsumerTest.java:380)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

java.lang.AssertionError: expected object to not be null
	at org.testng.Assert.fail(Assert.java:110)
	at org.testng.Assert.assertNotNull(Assert.java:1483)
	at org.testng.Assert.assertNotNull(Assert.java:1467)
	at org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer(PersistentDispatcherFailoverConsumerTest.java:379)

Usage tip: To enable automatic navigation to failure message, open the following links with CTRL/CMD-click.
example failure 2023-10-30T05:31:57.2389215Z
example failure 2023-10-29T09:11:53.7295314Z

Full exception stacktrace

java.lang.AssertionError: expected object to not be null
	at org.testng.Assert.fail(Assert.java:110)
	at org.testng.Assert.assertNotNull(Assert.java:1483)
	at org.testng.Assert.assertNotNull(Assert.java:1467)
	at org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer(PersistentDispatcherFailoverConsumerTest.java:379)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

Recent failure:
Flaky-test: org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer
Number of failures: 1
org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest is flaky. The testAddRemoveConsumer test method fails sporadically.
java.lang.AssertionError: expected object to not be null
	at org.testng.Assert.fail(Assert.java:110)
	at org.testng.Assert.assertNotNull(Assert.java:1483)
	at org.testng.Assert.assertNotNull(Assert.java:1467)
	at org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer(PersistentDispatcherFailoverConsumerTest.java:380)

Usage tip: To enable automatic navigation to failure message, open the following links with CTRL/CMD-click.
example failure 2023-11-14T09:59:00.0584287Z

Full exception stacktrace

java.lang.AssertionError: expected object to not be null
	at org.testng.Assert.fail(Assert.java:110)
	at org.testng.Assert.assertNotNull(Assert.java:1483)
	at org.testng.Assert.assertNotNull(Assert.java:1467)
	at org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer(PersistentDispatcherFailoverConsumerTest.java:380)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)

Flaky-test: org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer
Number of failures: 5
org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest is flaky. The testAddRemoveConsumer test method fails sporadically.
java.lang.AssertionError: expected object to not be null
	at org.testng.Assert.fail(Assert.java:110)
	at org.testng.Assert.assertNotNull(Assert.java:1483)
	at org.testng.Assert.assertNotNull(Assert.java:1467)
	at org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer(PersistentDispatcherFailoverConsumerTest.java:379)

Usage tip: To enable automatic navigation to failure message, open the following links with CTRL/CMD-click.
example failure 2024-08-12T09:07:27.7408489Z
example failure 2024-08-12T09:06:57.9752609Z
example failure 2024-08-10T06:50:13.6782019Z
example failure 2024-08-09T06:02:39.2387029Z
example failure 2024-08-08T11:59:57.8564839Z

Full exception stacktrace

java.lang.AssertionError: expected object to not be null
	at org.testng.Assert.fail(Assert.java:110)
	at org.testng.Assert.assertNotNull(Assert.java:1483)
	at org.testng.Assert.assertNotNull(Assert.java:1467)
	at org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer(PersistentDispatcherFailoverConsumerTest.java:379)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)


text:
Fix ConsumerBuilder and PulsarClient document

Motivation
There are few document error in code. We should fix them.
Modifications
Fixes document in ConsumerBuilder and PulsarClient.

Thank you!


text:
Branch 2.2

Motivation
Maven build fails due to posix non-compliance.
Modifications
Per jerrypeng, #3069,
Added following line to the configurations section in the assembly plugin in following pom.xml.
<tarLongFileMode>posix</tarLongFileMode>

https://github.com/apache/pulsar/blob/v2.2.0/pulsar-sql/presto-distribution/pom.xml#L128
Result
Successful Maven build.

@drdrill Thanks for your contribution. Could you please rebase with master and resolve the conflicts?
Thanks!

@drdrill thanks for the contribution.  I don't actually see:
<tarLongFileMode>posix</tarLongFileMode>

being added in this PR

authors: drdrill jerrypeng srkukarni


text:
Remove redundant readMoreEntries call in PersistentDispatcherMultiple…

Motivation
Remove redundant readMoreEntries call in PersistentDispatcherMultipleConsumers and PersistentStickyKeyDispatcherMultipleConsumers
Currently, readMoreEntries() will be called in readEntriesComplete(), may be some redundancy here. It's better to call readMoreEntries() in the sendMessagesToConsumers().
For shared subscription, no need to wait send messages to consumers complete, so call readMoreEntries() in the end of sendMessagesToConsumers().
For key_shared subscription, need to wait send messages to consumers complete, so call readMoreEntries() while all consumers writeAndFlush() completely.

run java8 tests
run Integration Tests


text:
Updated storage client

(If this PR fixes a github issue, please add Fixes #<xyz>.)
Fixes #
(or if this PR is one task of a github issue, please add Master Issue: #<xyz> to link to the master issue.)
Master Issue: #
Motivation
Explain here the context, and why you're making that change. What is the problem you're trying to solve.
Modifications
Describe the modifications you've done.
Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API: (yes / no)
The schema: (yes / no / don't know)
The default values of configurations: (yes / no)
The wire protocol: (yes / no)
The rest endpoints: (yes / no)
The admin cli options: (yes / no)
Anything that affects deployment: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

@david-streamlio Can you help to update the description of pull request?

move to 2.8.0 first

Closed as stale and conflict. Please rebase and submit a new patch if it's still relevant.

authors: codelipenghui david-streamlio tisonkun wolfstudy


text:
netty bug CPU 100%

BUG REPORT

Please describe the issue you observed:

What did you do (The steps to reproduce)?

场景：一个JVM里面会启动很多的DefaultMQAdminExt连接去查看不同集群的状态
"TaskWorker_466" #753 daemon prio=5 os_prio=0  nid=0x29ef9 ...    java.lang.Thread.State: RUNNABLE 	at io.netty.util.Recycler$Stack.scavengeSome(Recycler.java:504) 	at io.netty.util.Recycler$Stack.scavenge(Recycler.java:454) 	at io.netty.util.Recycler$Stack.pop(Recycler.java:435) 	at io.netty.util.Recycler.get(Recycler.java:144) 	at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.newInstance(AbstractChannelHandlerContext.java:1115) 	at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.access$1600(AbstractChannelHandlerContext.java:1104) 	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:825) 	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:799) 	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:835) 	at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1017) 	at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:256) 	at org.apache.rocketmq.remoting.netty.NettyRemotingAbstract.invokeSyncImpl(NettyRemotingAbstract.java:370) 	at org.apache.rocketmq.remoting.netty.NettyRemotingClient.invokeSync(NettyRemotingClient.java:374) 	at org.apache.rocketmq.client.impl.MQClientAPIImpl.getBrokerRuntimeInfo(MQClientAPIImpl.java:1121) 	at org.apache.rocketmq.tools.admin.DefaultMQAdminExtImpl.fetchBrokerRuntimeStats(DefaultMQAdminExtImpl.java:235) 	at org.apache.rocketmq.tools.admin.DefaultMQAdminExt.fetchBrokerRuntimeStats(DefaultMQAdminExt.java:202)

What did you expect to see?

What did you see instead?
大量线程阻塞在Recycler类，通过Top查看线程CPU使用率100%

Please tell us about your environment:
version:4.3.0
Other information (e.g. detailed explanation, logs, related issues, suggestions how to fix, etc):
https://github.com/netty/netty/pull/6180
这个是netty github上面对这个bug的描述

test

@davidpdw It is welcome submitting a PR to fix it.

可以通过DefaultMQAdminExt#setClientCallbackExecutorThreads，把Netty的网络线程数量限制一下，默认情况下会开启 n  * DefaultMQAdminExt  *  cpucores 个网络线程！

authors: davidpdw dongeforever hdchen whathi

repo: apache/rocketmq


text:
负载均衡的实际效果和官方文档不对应

Before Creating the Bug Report

[X] I found a bug, not just asking a question, which should be created in GitHub Discussions.

[X] I have searched the GitHub Issues and GitHub Discussions  of this repository and believe that this is not a duplicate.

[X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

Runtime platform environment
MacOS
RocketMQ version
5.2.0
JDK Version
1.8
Describe the Bug
官方文档的描述是5.X版本以后是消息级别的负载：
源码版本为：

源码中PushConsumer使用的还是AllocateMessageQueueAveragely负载均衡类

实际使用和测试都发现，超过队列数以后得消费者都不会被分配对应的队列，那此时不是还是队列粒度的负载均衡么？为何官方文档提到的是消息粒度的？
Steps to Reproduce
`List cidAll = Arrays.asList("cid-1", "cid-2", "cid-3", "cid-4", "cid-5", "cid-6", "cid-7");
List mqAll = Arrays.asList(
new MessageQueue("topic", "broker-a", 0),
new MessageQueue("topic", "broker-a", 1),
new MessageQueue("topic", "broker-a", 2),
new MessageQueue("topic", "broker-a", 3)
);
    List<MessageQueue> testallocate = testallocate("group-1", "cid-7", mqAll, cidAll);
    System.out.println(testallocate);`

以上是简单的测试用例，发现后面的消费者都是没有分配队列的
What Did You Expect to See?
我希望看到的是符合官方文档中提到的消息级别的负载均衡
What Did You See Instead?
无
Additional Context
No response

应该相信官方文档描述还是相信实际实验效果和源码？

需要设置消费模式为pop消费

官方文档说的默认的push和pull都支持啊

authors: jiao-duan yiduwangkai


text:
[Issue #5448] [RIP-46] Implement metrics for default message store

link to #5448

Codecov Report

Merging #5528 (b2d0530) into develop (14cff19) will increase coverage by 0.00%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             develop    #5528   +/-   ##
==========================================
  Coverage      42.70%   42.71%
- Complexity      8023     8026    +3
==========================================
  Files           1029     1031    +2
  Lines          72681    72735   +54
  Branches        9604     9610    +6
==========================================
+ Hits           31041    31066   +25
- Misses         37704    37741   +37
+ Partials        3936     3928    -8

Impacted Files
Coverage Δ

.../rocketmq/broker/metrics/BrokerMetricsManager.java
11.57% <0.00%> (-0.16%)
:arrow_down:

...he/rocketmq/proxy/metrics/ProxyMetricsManager.java
7.07% <0.00%> (-0.20%)
:arrow_down:

...org/apache/rocketmq/store/DefaultMessageStore.java
53.58% <0.00%> (-0.05%)
:arrow_down:

...tmq/store/metrics/DefaultStoreMetricsConstant.java
0.00% <0.00%> (ø)

...etmq/store/metrics/DefaultStoreMetricsManager.java
0.00% <0.00%> (ø)

.../rocketmq/broker/filter/ConsumerFilterManager.java
72.19% <0.00%> (-0.90%)
:arrow_down:

...ent/impl/consumer/DefaultLitePullConsumerImpl.java
68.38% <0.00%> (-0.63%)
:arrow_down:

.../org/apache/rocketmq/store/ha/DefaultHAClient.java
61.53% <0.00%> (-0.52%)
:arrow_down:

...rocketmq/broker/processor/PopMessageProcessor.java
37.31% <0.00%> (-0.51%)
:arrow_down:

.../org/apache/rocketmq/proxy/config/ProxyConfig.java
46.92% <0.00%> (ø)

... and 11 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: ShadowySpirits codecov-commenter


text:
[ISSUE #6373] Optimized log printing and fixed null pointer exceptions

close #6373

Codecov Report

Merging #6375 (0ef8b40) into develop (5ac9119) will decrease coverage by 0.02%.
The diff coverage is 53.33%.

@@              Coverage Diff              @@
##             develop    #6375      +/-   ##
=============================================
- Coverage      43.07%   43.06%   -0.02%
+ Complexity      8914     8912       -2
=============================================
  Files           1105     1105
  Lines          77943    77956      +13
  Branches       10144    10145       +1
=============================================
- Hits           33575    33570       -5
- Misses         40151    40176      +25
+ Partials        4217     4210       -7

Impacted Files
Coverage Δ

...n/java/org/apache/rocketmq/proxy/ProxyStartup.java
44.34% <0.00%> (ø)

...tmq/proxy/service/mqclient/MQClientAPIFactory.java
0.00% <0.00%> (ø)

...org/apache/rocketmq/store/DefaultMessageStore.java
46.75% <63.63%> (+0.14%)
:arrow_up:

...rg/apache/rocketmq/proxy/config/Configuration.java
72.41% <100.00%> (+0.98%)
:arrow_up:

... and 19 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

@xdkxlk PTAL

authors: codecov-commenter socutes


text:
Issue in MaskedDateField extends DateField

Go to Inventory from Setup in Menu
Go to Setups
Carriage/Freight - > contractor
file : dbzCarriageContractorSetup at path D:\DBIZSOL\Main\Royale\src\com\dbz\modules\INV
file : MaskedDateField at path  D:\DBIZLib\src\com\dbz\controls

Did the Flex app also get a Fault event in this Setup?  It looks to me that the Fault Handler (fn_HandleFault) is not written correctly.  I don't think the Fault Handler can expect that currentTarget is set as the Fault Handler is called from the AsyncToken responders and not called via dispatchEvent (until later).
The Fault being sent to the Fault Handler is:
"Detected duplicate HTTP-based FlexSessions, generally due to the remote host disabling session cookies.  Session cookies must be enabled to manage the client connection correctly"
So if the Flex application did not get a Fault, but the Royale application is, then you will have to resolve why that is as well as fix up the code in fn_faultHandler

@aharui
Issue is because of maskedDateField and maskedTextInput . see the above given screenshots

I made some changes and now there is some new error

OK, the Contractor setup appears without throwing uncaught exceptions.

authors: aharui alinakazi

repo: apache/royale-asjs


text:
[Feature][Connector-V2] connector-kafka source support data conversion extracted by kafka connect source

Purpose of this pull request
#4137
Check list

[ ] Code changed are covered with tests, or it does not need tests for reason:
[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update change log that in connector document. For more details you can refer to connector-v2
Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist

[ ] Update the release-note.

@hailin0 @TyrantLucifer

@hailin0  PTAL

@sunxiaojian please resolve the conflict

@sunxiaojian please resolve the conflict

@hailin0 done

@sunxiaojian please resolve the conflict

@hailin0 done

@sunxiaojian please resolve the conflict

@hailin0 done

@sunxiaojian Duplicate with #5066 feature?

It is also compatible with Debezium and can be removed

@sunxiaojian please resolve the conflict

@hailin0 done

@sunxiaojian Duplicate with #5066 feature?

@hailin0  It is also compatible with debezium and can be removed

@hailin0 PTAL

CI error is not related to this modification

repo: apache/seatunnel


text:
Enable Solr Operator metrics when using the helm chart.

Resolves #307
This does not use the protected metrics-endpoint described in the kubebuilder docs. This is also just using the default operator metrics. We can add additional metrics in future PRs.
The metrics have been enabled since #321 was implemented, but not when using the helm chart. This fixes that.

Tested manually and was able to query the operator metrics through kubectl port-forward.

repo: apache/solr-operator


text:
[SOLR-15737]: Create v2 equivalent of v1 'CREATESNAPSHOT', 'LISTSNAPSHOT' and 'DELETESNAPSHOT' (collection level)

https://issues.apache.org/jira/browse/SOLR-16468

Description
This PR contains the V2 API implementations of CREATE, LIST, and DELETE collection snapshot resources.
Solution
I created Jersey implementations of the existing CREATE, LIST, and DELETE collection snapshot resources. I then refactored the existing V1 endpoints to call the new V2 resources.
Tests
TODO: I plan to write tests that excercise the V2 resources.
Checklist
Please review the following and check all that apply:

[x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
[x] I have created a Jira issue and added the issue ID to my pull request title.
[x] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
[x] I have developed this patch against the main branch.
[x] I have run ./gradlew check.
[x] I have added tests for my changes.
[x] I have added documentation for the Reference Guide

TODO:

Add javadoc documentation and possibly reference guide documentation.
Add tests for the new Jersey Resources.

Hi @gerlowskija - I believe I've addressed the concerns listed above along with adding some unit tests that cover the Create and Delete collection snapshot endpoints. You mentioned in the jira ticket comments that the actual creation and deletion of snapshots was tested elsewhere, so I think just supplying unit tests may be all that's needed here. Spinning up a full cloud test case may be a little overkill!
I'm still looking into the documentation portion of this work. It may be worth adding the API docs to collection-management.adoc like you mentioned, but I'll need to read through the backup-restore.adoc to see how I'd want to work them into there.

I'm doing a little digging into this serialization issue, and it looks like we are hitting the else block (line 108) of org.apache.solr.common.util.TextWriter when trying to write CollectionSnapshotMetadata. From looking at the code leading up to this point, I'm starting to think that the JSON message writer for the V1 API assumes that any "objects" it sees are NamedLists or SimpleOrderedMaps. I'm not sure if that's correct or not, but that's my current assumption

New unit tests and other tweaks look great!  We're just down to the docs at this point I think.

It may be worth adding the API docs to collection-management.adoc like you mentioned, but I'll need to read through the backup-restore.adoc to see how I'd want to work them into there.

Sounds good.  Thinking on it a bit more, it might make sense to split those pieces up.  Adding coverage to collection-management.adoc would be relatively rote small and rote, so it seems "in scope" for this PR.  backup-restore.adoc probably needs some larger restructing, so I'd be fine creating a separate JIRA ticket for it and leaving it for another day.  But that's just my 2c; happy to handle that however you'd like.

Jdurham2843 force-pushed the SOLR-15737-snapshot-collection-v2 branch from d461741 to 875d7a7
last week

One small nitpick process-wise: please avoid force-pushes if you can.  It makes it slightly harder for others from adding small fixes/improvements to your branch, and it makes things a little harder to review in github by doing weird things to ongoing line-level review comments and nullifying Github's really nice "Diff files since your last review" feature.

Collection-management.adoc has been updated with three sections on CREATESNAPSHOT, LISTSNAPSHOTS, and DELETESNAPSHOT. I think it's a good idea to do the backup-restore.adoc separately, since it wouldn't be nearly as simple to update and would need to consider other APIs aside from the three worked on here. Let me know what you think!
Regarding the force push: sorry about that! I needed to get some code in from the Apache Solr main branch, so I did a rebase and force push. That's normally the approach I reach for when needing to pull in changes, and can't do a fast-forward. Is there a another way to pull in changes, without needing to force push? My git-fu is fairly rudimentary

Is there another way to pull in changes, without needing to force push?

Yep, definitely!  IMO this stuff can be tricky to explain (or read about) through text alone.  Everyone's git settings and repo tends to be just different enough that sharing workflow-commands alone can cause as much confusion as it clears up.  Anyway, if this explanation doesn't help I'm happy to do a screenshare to discuss as well. (My email address should be on my GH profile)
But here goes!

Let's say I have a feature branch (my-feature) on a Github fork repo (myfork), and I want to pull in some changes from the main branch in the Apache repo (origin).

The first step is to update your local copy of main so that it matches the latest stuff upstream:

git checkout main
`git pull origin main

Now that your local copy of main is up to date, you can merge those changes into your feature branch

git checkout my-feature
git merge main
(Optionally fix any merge conflicts that might've arisen and git commit when you're happy)

Your local my-feature branch now has the changes from main, we just need to publish your updated branch upstream:

git push myfork my-feature:my-feature

Hope that helps!

Awesome, and thank you for the review and help on this!

Is there any problem with the CHANGES.txt?
I see some broken checks here, and a contribution from a colleague of mine seems to have a broken crave build caused by the CHANGES.txt ?
Still investigating, so I am sorry if I said any nonsense!

repo: apache/solr


text:
SOLR-16989: Optimize and consolidate reuse of DocValues iterators for value retrieval

See: SOLR-16989

Proposed CHANGES.txt entry under Optimizations for 9.4.0 release:
* SOLR-16989: Optimize and consolidate reuse of DocValues iterators for value retrieval (Michael Gibney)

I would have loved the review, sorry this didn't get on your radar at the time. Please lmk if you have any concerns about what was committed, or what could/should have been committed alongside; I certainly don't mind revisiting.


text:
[CORE] Refactoring: Use pattern matching and dedicated method

It should be slightly easier to see what really happens (without questions like what other state a task can be here).
BTW, I was thinking about introducing TaskState.isFailed or similar method so it's even clearer that the order of cases in the pattern match matters as the only case left out in TaskState.isFinished is FINISHED. With such a method, TaskState.isFinished would be TaskState.isFailed + FINISHED state. Perhaps, TaskState.isCompleted would be more appropriate.

Can one of the admins verify this patch?

I think the logic is equivalent, yes, but it isn't any shorter or clearer. Yes, the issue is that it looks strange to check if the state is FINISHED and then otherwise check if it's a finished state (which means, a finished state that isn't FINISHED). I don't think this is worth it.

I disagree since Spark core is hard due to what it does and making it harder can hinder contributions. There are just such little things that can change how welcoming the code is. Please reconsider your vote (I spent the time on it already so if it's lost aka "I don't think this is worth it", let's turn it back when the change hits master). You've just agreed that the logic is convoluted.

The question is whether this makes the code more 'welcoming' or not, of course, and I don't think this does. I did not agree that the current logic is convoluted; I'm talking about your change, and agreeing with your assessment that relying on isFinished to mean "finished but not FINISHED" is funny.
Of course you don't want to spend time on changes that won't be merged. However, the amount of effort that goes into a change alone doesn't make it a better change. No, it would never be appropriate to just merge it and then think later about whether it was worth it.
A lot of my time and others' is spent reviewing changes that won't be merged, so rejected PRs/JIRAs cost everyone time*. Ideally we'd all know what kinds of change will get buy-in from people who can merge ahead of time; the list at https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-PreparingtoContributeCodeChanges was written to help describe what makes that likely. I'd also look for JIRAs opened by committers, which are much more likely to have buy-in.

Well, they have some value: they're a way of communicating about what won't be accepted and have value if this saves time in the future. I think many of your PRs have been rejected on similar grounds, so I'd also look to you to tailor your choices appropriately, rather than push back further. For example, https://github.com/apache/spark/pull/10432 is also pretty trivial but is also fixes with no downside that I can see. Although it's not the most important thing I'd encourage people to work on, myself, that seems OK. Here the issue is that it seems to do nothing but make the code marginally less clear.

Doh, you're again inviting me to make more involved changes (which is great, but don't think helps others to contribute). What I'm gonna do is to improve the change with changes with TaskState. Would that make sense? Would you approve such a change? I know the code quite well now, and believe that it needs such small changes to ultimately be more open for newcomers. There's simply far too much to learn in Spark Core, and such naming conventions where isFinished should've been isCompleted instead (following the event CompleteEvent sent later) don't make it any easier.
Thanks a lot for helping me find the right way! I'm greatly indebted for the effort.

I'm not suggesting any changes at all. If there were a few places in the code that call for checking for a state that is "finished, but not FINISHED" then I'd say it's worth making a name and method for that and refactoring this and the other code to use it. For this alone, it doesn't seem like there's an improvement.
I'm not arguing against improvements; I'm arguing against things that don't appear to be an improvement.

Your debate aside, I am not sure whether it makes sense to change the current one. It has virtually zero impact on anything, and as a result it is more or less changing it for the sake of changing it, which in general isn't a great idea.

More substantively, if...else generally performs better than pattern matching, and this is a fairly hot code path.

@jaceklaskowski can you close the pr?

authors: AmplabJenkins jaceklaskowski markhamstra rxin srowen

repo: apache/spark


text:
[SPARK-19265][SQL] make table relation cache general and does not depend on hive

What changes were proposed in this pull request?
We have a table relation plan cache in HiveMetastoreCatalog, which caches a lot of things: file status, resolved data source, inferred schema, etc.
However, it doesn't make sense to limit this cache with hive support, we should move it to SQL core module so that users can use this cache without hive support.
It can also reduce the size of HiveMetastoreCatalog, so that it's easier to remove it eventually.
main changes:

move the table relation cache to SessionCatalog
SessionCatalog.lookupRelation will return SimpleCatalogRelation and the analyzer will convert it to LogicalRelation or MetastoreRelation later, then HiveSessionCatalog doesn't need to override lookupRelation anymore
FindDataSourceTable will read/write the table relation cache.

How was this patch tested?
existing tests.

cc @yhuai @gatorsmile

Test build #71522 has started for PR 16621 at commit a7845cc.

Test build #71522 has finished for PR 16621 at commit a7845cc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class QualifiedTableName(database: String, name: String)
class FindHiveSerdeTable(session: SparkSession) extends Rule[LogicalPlan]

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71522/
Test FAILed.

Test build #71549 has started for PR 16621 at commit 98a5483.

Start the review. Will remove this comment when the review is finished.

Test build #71549 has finished for PR 16621 at commit 98a5483.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class QualifiedTableName(database: String, name: String)
class FindHiveSerdeTable(session: SparkSession) extends Rule[LogicalPlan]

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71549/
Test FAILed.

Test build #71567 has started for PR 16621 at commit 919aaa2.

Could we rename SimpleCatalogRelation to UnresolvedCatalogRelation? The current name looks very confusing to me.

can we do it later? We are going to merge CatalogRelation implementations and unify the table relation representations soon.

Test build #71567 has finished for PR 16621 at commit 919aaa2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class QualifiedTableName(database: String, name: String)
class FindHiveSerdeTable(session: SparkSession) extends Rule[LogicalPlan]

Sure. No problem.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71567/
Test FAILed.

Test build #71572 has started for PR 16621 at commit 2883c8b.

Test build #71574 has started for PR 16621 at commit bbccdae.

No more comments. It looks pretty good! Let us see whether all the test cases can pass.

Test build #71576 has started for PR 16621 at commit d636389.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71576/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71574/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71572/
Test FAILed.

Test build #71583 has started for PR 16621 at commit 096fcc8.

Test build #71583 has finished for PR 16621 at commit 096fcc8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class QualifiedTableName(database: String, name: String)
class FindHiveSerdeTable(session: SparkSession) extends Rule[LogicalPlan]

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71583/
Test PASSed.

LGTM

Thanks! Merging to master.

authors: AmplabJenkins SparkQA cloud-fan gatorsmile


text:
[SPARK-20694][DOCS][SQL] Document DataFrameWriter partitionBy, bucketBy and sortBy in SQL guide

What changes were proposed in this pull request?

Add Scala, Python and Java examples for partitionBy, sortBy and bucketBy.
Add Bucketing, Sorting and Partitioning section to SQL Programming Guide
Remove bucketing from Unsupported Hive Functionalities.

How was this patch tested?
Manual tests, docs build.

Test build #76748 has started for PR 17938 at commit 20c7ca6.

Test build #76748 has finished for PR 17938 at commit 20c7ca6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76748/
Test PASSed.

@HyukjinKwon Sounds good. SPARK-20694.
Should we document the difference between buckets (metastore based) and partitions (file system based)? The latter one could by done by referencing Partition Discover.

(I think I am not supposed to decide this and probably the best is the confirmation from a commiter)

@cloud-fan @tejasapatil Could you please help review this PR?

Test build #76813 has started for PR 17938 at commit a14296a.

Test build #76813 has finished for PR 17938 at commit a14296a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76813/
Test PASSed.

Test build #76825 has started for PR 17938 at commit 7bf4bbc.

Test build #76825 has finished for PR 17938 at commit 7bf4bbc.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76825/
Test FAILed.

Test build #76828 has started for PR 17938 at commit cc1bfcf.

Test build #76830 has started for PR 17938 at commit 606f1e3.

Test build #76830 has finished for PR 17938 at commit 606f1e3.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76830/
Test FAILed.

Test build #76828 has finished for PR 17938 at commit cc1bfcf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76828/
Test PASSed.

Test build #76831 has started for PR 17938 at commit a7aff81.

Test build #76831 has finished for PR 17938 at commit a7aff81.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76831/
Test PASSed.

Merged build finished. Test PASSed.

@zero323 Could you also document SQL interface for creating bucket and partition tables in this PR?

@gatorsmile Sure, but I assume you mean only PARTITION BY, right? I don't think that CLUSTER BY or  SORT BY is supported in SQL (should it be supported after  #17644 is resolved?).

Test build #76867 has started for PR 17938 at commit c4d7856.

Test build #76867 has finished for PR 17938 at commit c4d7856.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76867/
Test PASSed.

@zero323 We already support it for data source tables. Below is just an example.
CREATE TABLE tbl(a INT, b INT) USING parquet CLUSTERED BY (a) SORTED BY (b) INTO 5 BUCKETS")

@gatorsmile  Huh...  in that case it looks like parser (?) needs a little bit of work, unless of course following are features.

Omitting USING doesn't work
CREATE TABLE user_info_bucketed(user_id BIGINT, firstname STRING, lastname STRING)
CLUSTERED BY(user_id) INTO 256 BUCKETS

with:
Error in query:
Operation not allowed: CREATE TABLE ... CLUSTERED BY(line 1, pos 0)

== SQL ==
CREATE TABLE user_info_bucketed(user_id BIGINT, firstname STRING, lastname STRING)
^^^
CLUSTERED BY(user_id) INTO 256 BUCKETS

Omitting USING adding PARTITION BY with column not present in the main clause (valid Hive DDL) doesn't work:
CREATE TABLE user_info_bucketed(user_id BIGINT, firstname STRING, lastname STRING)
PARTITIONED BY (department STRING)
CLUSTERED BY(user_id) INTO 256 BUCKETS

with
Error in query:
Operation not allowed: CREATE TABLE ... CLUSTERED BY(line 1, pos 2)

== SQL ==
  CREATE TABLE user_info_bucketed(user_id BIGINT, firstname STRING, lastname STRING)
--^^^
  PARTITIONED BY (department STRING)
  CLUSTERED BY(user_id) INTO 256 BUCKETS

PARTITION BY alone works:
CREATE TABLE user_info_bucketed(user_id BIGINT, firstname STRING, lastname STRING)
PARTITIONED BY (department STRING)

PARTITION BY with USING when partition column is in the main spec works:
CREATE TABLE user_info_bucketed(
 user_id BIGINT, firstname STRING, lastname STRING, department STRING)
USING parquet
PARTITIONED BY (department)

CLUSTERED BY +  PARTITION BY with USING when partition column is in the main spec works:
CREATE TABLE user_info_bucketed(
   user_id BIGINT, firstname STRING, lastname STRING, department STRING)
USING parquet
PARTITIONED BY (department)
CLUSTERED BY(user_id) INTO 256 BUCKETS

PARTITION BY when parition column is in the main spec, USING omitted:
CREATE TABLE user_info_bucketed(
 user_id BIGINT, firstname STRING, lastname STRING, department STRING)
PARTITIONED BY (department)

with:
Error in query:
mismatched input ')' expecting {'SELECT', 'FROM', 'ADD', 'AS', 'ALL', 'DISTINCT', 'WHERE', 'GROUP', 'BY', 'GROUPING', 'SETS', 'CUBE', 'ROLLUP', 'ORDER', 'HAVING', 'LIMIT', 'AT', 'OR', 'AND', 'IN', NOT, 'NO', 'EXISTS', 'BETWEEN', 'LIKE', RLIKE, 'IS', 'NULL', 'TRUE', 'FALSE', 'NULLS', 'ASC', 'DESC', 'FOR', 'INTERVAL', 'CASE', 'WHEN', 'THEN', 'ELSE', 'END', 'JOIN', 'CROSS', 'OUTER', 'INNER', 'LEFT', 'SEMI', 'RIGHT', 'FULL', 'NATURAL', 'ON', 'LATERAL', 'WINDOW', 'OVER', 'PARTITION', 'RANGE', 'ROWS', 'UNBOUNDED', 'PRECEDING', 'FOLLOWING', 'CURRENT', 'FIRST', 'AFTER', 'LAST', 'ROW', 'WITH', 'VALUES', 'CREATE', 'TABLE', 'VIEW', 'REPLACE', 'INSERT', 'DELETE', 'INTO', 'DESCRIBE', 'EXPLAIN', 'FORMAT', 'LOGICAL', 'CODEGEN', 'COST', 'CAST', 'SHOW', 'TABLES', 'COLUMNS', 'COLUMN', 'USE', 'PARTITIONS', 'FUNCTIONS', 'DROP', 'UNION', 'EXCEPT', 'MINUS', 'INTERSECT', 'TO', 'TABLESAMPLE', 'STRATIFY', 'ALTER', 'RENAME', 'ARRAY', 'MAP', 'STRUCT', 'COMMENT', 'SET', 'RESET', 'DATA', 'START', 'TRANSACTION', 'COMMIT', 'ROLLBACK', 'MACRO', 'IGNORE', 'IF', 'DIV', 'PERCENT', 'BUCKET', 'OUT', 'OF', 'SORT', 'CLUSTER', 'DISTRIBUTE', 'OVERWRITE', 'TRANSFORM', 'REDUCE', 'USING', 'SERDE', 'SERDEPROPERTIES', 'RECORDREADER', 'RECORDWRITER', 'DELIMITED', 'FIELDS', 'TERMINATED', 'COLLECTION', 'ITEMS', 'KEYS', 'ESCAPED', 'LINES', 'SEPARATED', 'FUNCTION', 'EXTENDED', 'REFRESH', 'CLEAR', 'CACHE', 'UNCACHE', 'LAZY', 'FORMATTED', 'GLOBAL', TEMPORARY, 'OPTIONS', 'UNSET', 'TBLPROPERTIES', 'DBPROPERTIES', 'BUCKETS', 'SKEWED', 'STORED', 'DIRECTORIES', 'LOCATION', 'EXCHANGE', 'ARCHIVE', 'UNARCHIVE', 'FILEFORMAT', 'TOUCH', 'COMPACT', 'CONCATENATE', 'CHANGE', 'CASCADE', 'RESTRICT', 'CLUSTERED', 'SORTED', 'PURGE', 'INPUTFORMAT', 'OUTPUTFORMAT', DATABASE, DATABASES, 'DFS', 'TRUNCATE', 'ANALYZE', 'COMPUTE', 'LIST', 'STATISTICS', 'PARTITIONED', 'EXTERNAL', 'DEFINED', 'REVOKE', 'GRANT', 'LOCK', 'UNLOCK', 'MSCK', 'REPAIR', 'RECOVER', 'EXPORT', 'IMPORT', 'LOAD', 'ROLE', 'ROLES', 'COMPACTIONS', 'PRINCIPALS', 'TRANSACTIONS', 'INDEX', 'INDEXES', 'LOCKS', 'OPTION', 'ANTI', 'LOCAL', 'INPATH', 'CURRENT_DATE', 'CURRENT_TIMESTAMP', IDENTIFIER, BACKQUOTED_IDENTIFIER}(line 3, pos 30)

== SQL ==
    CREATE TABLE user_info_bucketed(
      user_id BIGINT, firstname STRING, lastname STRING, department STRING)
    PARTITIONED BY (department)
------------------------------^^^

Test build #76899 has started for PR 17938 at commit b5babf6.

When you omit USING, it's hive style CREATE TABLE syntax, which is very different from Spark. We should encourage users to use the spark style CREATE TABLE syntax and only document it(with USING statement).

Test build #76899 has finished for PR 17938 at commit b5babf6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76899/
Test PASSed.

Test build #76900 has started for PR 17938 at commit 92fb3b3.

@cloud-fan Thanks for the clarification. Just a thought - shouldn't we either support it consistently or don't support at all? Current behaviour is quite confusing and I don't think that documentation alone will cut it.

Test build #76900 has finished for PR 17938 at commit 92fb3b3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76900/
Test PASSed.

Merged build finished. Test PASSed.

we are going to support bucketing in hive style CREATE TABLE syntax soon.

In the current 2.2 docs, we already updated all the syntax to CREATE TABLE ... USING.... This is the new change delivered in 2.2
Thus, it is OK to document like what you just committed. Let me review them carefully now. Thanks for your work!

LGTM except a few minor comments.
cc @tejasapatil @cloud-fan

Test build #76905 has started for PR 17938 at commit 65ac310.

Test build #76905 has finished for PR 17938 at commit 65ac310.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76905/
Test PASSed.

Test build #76911 has started for PR 17938 at commit 3a8b6e9.

Test build #76911 has finished for PR 17938 at commit 3a8b6e9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76911/
Test PASSed.

Will merge it when my minor comment is resolved.
Thanks for working on it!

Test build #77436 has started for PR 17938 at commit bea0676.

Test build #77436 has finished for PR 17938 at commit bea0676.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77436/
Test PASSed.

Thanks @gatorsmile

authors: AmplabJenkins HyukjinKwon SparkQA cloud-fan gatorsmile zero323


text:
[SPARK-20844] Remove experimental from Structured Streaming APIs

Now that Structured Streaming has been out for several Spark release and has large production use cases, the Experimental label is no longer appropriate.  I've left InterfaceStability.Evolving however, as I think we may make a few changes to the pluggable Source & Sink API in Spark 2.3.

Test build #77197 has started for PR 18065 at commit fe03241.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77197/
Test FAILed.

test this please

Test build #77203 has started for PR 18065 at commit fe03241.

Test build #77203 has finished for PR 18065 at commit fe03241.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77203/
Test PASSed.

all the python apis also need to be marked not experimental!

Other scala class that are still marked experimental are

FlatMapGroupsWithStateFunction
GroupState
GroupStateTimeout
Not sure if you missed or you intentionally kept them as experimental. Since the mapGroupsWithState is new, I think its better to actually keep them experimental.

Good catch on python!  Fixed but please help me make sure I didn't miss anything.
Leaving all the GroupState stuff experimental was on purpose, since this will be the first release including it.

Test build #77317 has started for PR 18065 at commit 8221c01.

Test build #77317 has finished for PR 18065 at commit 8221c01.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77317/
Test PASSed.

Test build #77431 has started for PR 18065 at commit 98e8326.

Test build #77431 has finished for PR 18065 at commit 98e8326.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77431/
Test PASSed.

LGTM. Merging to master and 2.2.

authors: AmplabJenkins SparkQA marmbrus tdas zsxwing


text:
[SPARK-22469][SQL] Accuracy problem in comparison with string and numeric

What changes were proposed in this pull request?
This fixes a problem caused by #15880
select '1.5' > 0.5; // Result is NULL in Spark but is true in Hive.
When compare string and numeric, cast them as double like Hive.

@cloud-fan Would you please look at this when you have time?

Can one of the admins verify this patch?

@liutang123 can you update your PR description to say this fixes a problem caused by #15880 .

ok to test

Test build #83816 has started for PR 19692 at commit aada8ee.

Test build #83816 has finished for PR 19692 at commit aada8ee.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83816/
Test FAILed.

Test build #83820 has started for PR 19692 at commit cf09358.

Test build #83820 has finished for PR 19692 at commit cf09358.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83820/
Test PASSed.

Merged build finished. Test PASSed.

Test build #83829 has started for PR 19692 at commit 9c5bba0.

Test build #83833 has started for PR 19692 at commit cb1c8b8.

Test build #83833 has finished for PR 19692 at commit cb1c8b8.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #83829 has finished for PR 19692 at commit 9c5bba0.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83833/
Test FAILed.

Merged build finished. Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83829/
Test FAILed.

Jenkins, retest this please

retest this please

Test build #83891 has started for PR 19692 at commit cb1c8b8.

Test build #83891 has finished for PR 19692 at commit cb1c8b8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83891/
Test PASSed.

LGTM

backported to 2.2 and manually fixed the conflict.

Sorry, I just saw it.
Thank fan for doing this.

authors: AmplabJenkins SparkQA cloud-fan gatorsmile liutang123


text:
[PYTHON][WIP] pyspark schema.toDDL() and DataType.fromDDL(ddl)

What changes were proposed in this pull request?
Add a pyspark equivalent to schema.toDDL
Why are the changes needed?
Spark in Scala has a schema.toDDL method, but pyspark lacks an analog.
Does this PR introduce any user-facing change?
Yes.  This change adds DataType.toDDL and DataType.fromDDL to pyspark, matching
the equivalent methods in core Spark.
How was this patch tested?
pyspark/sql/tests/test_types.py now has test_to_ddl()

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can you file a JIRA please?
Also, please show examples of this new APIs in "Does this PR introduce any user-facing change?".

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Closing this due to inactivity here.

authors: AmplabJenkins HyukjinKwon dougbateman