text:
Consider the StringBuilder's capacity instead of content length when being trimmed

The trim operation aims at releasing the too much memory occupied by the StringBuilder. So when the StringBuilder really contains few effective characters, but with large capacity, should also be trimmed.

@garydgregory Yes, I noticed that and fixed it.

Hi,
When I build with 'mvn clean install' from a clean git master, all the tests pass. Note that I am on Windows 10 and that the test HttpAppenderTest fails randomly for me, so I usually annotate it with @Ignored in my local workspace. It looks like you have a debugging session ahead of you in your favorite IDE ;-)

Well, the case runs well on my computer. But I notice that the MAX_STRING_BUILDER_SIZE is not a constant, instead it may be loaded from the sysstem properties. So I changed my test case, to define the size upon the value of MAX_STRING_BUILDER_SIZE, instead of a hard coded number. Let's see, whether it will work.

Still reports this problem? Hmm... interesting. I also have ran this case in my IDE, nothing went wrong. Would you please show me the git repository version, maybe we have to check if we are running the same version of code.

I merged the latest master to my branch just now, Would you please run the case again, and show me the result? @garydgregory

I reappeared the failure! when I run the tests with mvn clean test -pl log4j2-core, I got the following failure (it is just the failure you provided)
[ERROR] Failures:
[ERROR]   AbstractStringLayoutTest.testGetStringBuilderCapacityRestrictedToMax:74 capacity, trimmed to MAX_STRING_BUILDER_SIZE expected:<2048> but was:<4096>

Then I run mvn in debug mode, and find out the log4j2-api is not my local code, but some code downloaded from the maven snapshot repository.

command for running test in debug mode:
mvn test -pl log4j-core -Dmaven.surefire.debug="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 -Xnoagent -Djava.compiler=NONE" -Dtest=AbstractStringLayoutTest

Well, the correct command for running the test is mvn clean test -am -pl log4j2-core. It will compile log4j2-core and all its dependent modules, but I didn't have java9 installed, so it will always fail for the log4j-api-java9. If you have java9 installed, please check it to see whether it fails.

I have the whole project sources and all modules (sources) loaded in
Eclipse so I run the one test directly from Eclipse.
The commands you show will never work (did you try them?) since there is no
log4j2-core module, it is log4j-core.
So if I run:
mvn clean install -pl log4j-api
and:
mvn clean test -pl log4j-core -Dtest=AbstractStringLayoutTest
I get:
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.logging.log4j.core.layout.AbstractStringLayoutTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
0.032 s <<< FAILURE! - in
org.apache.logging.log4j.core.layout.AbstractStringLayoutTest
[ERROR]
testGetStringBuilderCapacityRestrictedToMax(org.apache.logging.log4j.core.layout.AbstractStringLayoutTest)
Time elapsed: 0.032 s  <<< FAILURE!
java.lang.AssertionError: capacity, trimmed to MAX_STRING_BUILDER_SIZE
expected:<2048> but was:<4096>
at
org.apache.logging.log4j.core.layout.AbstractStringLayoutTest.testGetStringBuilderCapacityRestrictedToMax(AbstractStringLayoutTest.java:74)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
AbstractStringLayoutTest.testGetStringBuilderCapacityRestrictedToMax:74
capacity, trimmed to MAX_STRING_BUILDER_SIZE expected:<2048> but was:<4096>
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 53.099 s
[INFO] Finished at: 2017-07-17T08:53:05-07:00
[INFO] Final Memory: 43M/686M
Make sure you have the latest from git master.
Thank you!
Gary
On Mon, Jul 17, 2017 at 6:09 AM, Jerry notifications@github.com wrote:

I reappeared the failure! when I run the tests with mvn clean test -pl
log4j2-core, I got the following failure (it is just the failure you
provided)
[ERROR] Failures:
[ERROR]   AbstractStringLayoutTest.testGetStringBuilderCapacityRestrictedToMax:74 capacity, trimmed to MAX_STRING_BUILDER_SIZE expected:<2048> but was:<4096>
Then I run mvn in debug mode, and find out the log4j2-api is not my local
code, but some code downloaded from the maven snapshot repository.
command for running test in debug mode:
mvn test -pl log4j-core -Dmaven.surefire.debug="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 -Xnoagent -Djava.compiler=NONE" -Dtest=AbstractStringLayoutTest
Well, the correct command for running the test is mvn clean test -am -pl
log4j2-core. It will compile log4j2-core and all its dependent modules,
but I didn't have java9 installed, so it will always fail for the
log4j-api-java9. If you have java9 installed, please check it to see
whether it fails.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/logging-log4j2/pull/92#issuecomment-315750578,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABIfN_X6J_VNqM93qO3459iVX2_RUJHLks5sO1z0gaJpZM4OUANs
.

After this failure, could you please check your local maven repository for the log4j-api-2.9-SNAPSHOT version, decompile the jar and check the StringBuilders class. After I got the failure, I found it is not compiled from my code.
I didn't check the command I provided above, I just wrote it on my experience. My point is that the two modules may have to be compiled in only 1 command. For me, my settings for maven will download the dependent SNAPSHOT artifacts every time I build a module, so even when I installed the SNAPSHOT version, it will be rewritten when I try to build the other module. I'm not sure what's your settings, so I suggest to check the jar file in the repository.
public static void trimToMaxSize(final StringBuilder stringBuilder, final int maxSize) {
    if (stringBuilder != null && stringBuilder.capacity() > maxSize) { // I modified this line, but I found it remains unchanged and still check the stringBuilder.length() in the decompiled code.
        stringBuilder.setLength(maxSize);
        stringBuilder.trimToSize();
    }
}

I followed the tips to run command mvn -U clean test -pl log4j-core -Dtest=AbstractStringLayoutTest, and then I checked my local maven repository, but I'm afraid it didn't work. The following picture is a screen shot when I decompile the log4j-api-2.9-SNAPSHOT.jar with Java Decompiler. Please have a look on the red rectangle I draw here, it is where different from my code, and also the reason cause the failures above.

Please accept my apologies, I made a mistake saying the patch was partly applied to log4j-api. It was not. The new test passes.

With this patch, I now get:
[ERROR] Failures:
[ERROR]   AsyncLoggerThreadContextGarbageFreeTest>AbstractAsyncThreadContextTestBase.testAsyncLogWritesToLog:159->AbstractAsyncThreadContextTestBase.checkResult:185 AsyncLoggerTest.log: line 18 expected:<...yncLoggerContext i=1[]8> but was:<...yncLoggerContext i=1[2]8>
[INFO]
[ERROR] Tests run: 1889, Failures: 1, Errors: 0, Skipped: 21
[INFO]
[INFO] ------------------------------------------------------------------------
Gary

I runned this test, but it didn't fail on my machine.

authors: garydgregory xnslong


text:
PR 13757 follow-up: add missing with-discountOverlaps Similarity constructor variants, CHANGES.txt entries (#13845)

(cherry picked from commit dab731175c86cc25741f37845136d38a69a9d165)
Resolved Conflicts:
lucene/CHANGES.txt

@cpoerschke would you mind getting this in, but backing out the changes to 9.12.0 section of the change log?

source: gharchive/pull-request

authors: cpoerschke javanna

repo: apache/lucene

license_type: permissive

license_source: github-api


text:
Some refactoring/cleanup of AbstractSortedSetDocValueFacetCounts

A little refactoring/cleanup of common functionality in AbstractSortedSetDocValueFacetCounts. No functional change.

Thanks @shaie !


text:
NIFI-5449: Added Base64 Encode/Decode functions to RecordPath

Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit?

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Merged. Thanks @mattyb149

repo: apache/nifi


text:
NIFI-5673 Fixing connection handling in MQTT processors

Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit?

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

@bbende can you change the title to refer to the correct jira, you got me all excited about the other jira ;)

authors: bbende ottobackwards


text:
NIFI-6546 - Add JsonPath set value support

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Enables ability to set JsonPath values.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

I'll try to add this to my todo list.

@tpalfy - I've updated the code based on your review.
@tpalfy @MikeThomsen - Let me know if you any additional advice for me.
Thanks for your feedback.
Mans

@mans2singh you need to rebase off of master because you have a merge conflict.

@MikeThomsen -
I've rebased to master.
Please let me know if you have any additional advice/recommendations.
Thanks

@MikeThomsen -
I've updated the code based on your review.
Please let me know if you have any additional comments.
Thanks


text:
NIFI-8263: Maximum Thread Pool Size property introduced in ListenHTTP

https://issues.apache.org/jira/browse/NIFI-8263
Description of PR
Introduces Maximum Thread Pool Size property in the ListenHTTP processor to be able to set the maximum number of threads used by the Jetty server.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically main)?

[x] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[x] Have you verified that the full build is successful on JDK 8?
[ ] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

Thank you @exceptionfactory !
You are right, there is no simple way to check the thread pool size within the Server instance. After your comment, I thought about it and found that we do not need to check the server's thread pool size as long as we can test what value the server's constructor is called with. So I organized the server instantiation into a separate method and created unit tests for it. My modifications are in the recent commit.

Waiting for the build checks to complete. If all green, will merge.


text:
NIFI-13147: Remove include-new-ui profile

NIFI-13147:

Remove include-new-ui profile and include it by default always.

reviewing

authors: joewitt mcgilman


text:
NIFI-13755 On NiFi restart, re-enable as many previously enabled controller services as possible

(meaning all the ones which are not dependent on another disabled service).
Only skip trying to re-enable a previously disabled controller service IF it depends on another service which is disabled.

Summary
NIFI-13755
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[X] Apache NiFi Jira issue created

Pull Request Tracking

[X] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-13755
[X] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-13755

Pull Request Formatting

[X] Pull Request based on current revision of the main branch
[X] Pull Request refers to a feature branch with one commit containing changes

Verification

I added and updated automated tests to exercise the new behavior
I tested with running NiFi server for several disabled and enabled state services/combinations and it worked as ticket describes it should.

Build

[X] Build completed using mvn clean install -P contrib-check

[X] JDK 21

Licensing
No new dependencies
Documentation
No documentation changes

+1 LGTM, verified the expected behavior. @exceptionfactory do you want another look at this? If not I'll merge

Thanks @mattyb149, I will take another look and follow up soon.

@exceptionfactory Can you please let me know what you think of this updated PR  when you have a chance?

@exceptionfactory Do you think you'll be able to finish your review this week? Or if you're satisfied I can do the merge, thanks!

@exceptionfactory Do you think you'll be able to finish your review this week? Or if you're satisfied I can do the merge, thanks!

Thanks for the reminder @mattyb149, yes, I should be able to finish a review this week.

Thanks for pinging @exceptionfactory. I do think the change is reasonable. Looking at how the method gets used, I don't see anywhere that there is a strict requirement for atomicity. So I'm fine with making these changes. I do see @exceptionfactory that Github still shows '1 change requested' from you, which I assume is no longer relevant. But I'll hold off on clicking the "Squash and merge" button to make sure. But I'm a +1. Thanks all!


text:
RISC-V32 MMU Implementation

We're currently looking at configuring a kernel build for NuttX for using 32 bit risc-v architecture.
I can see that there is a MMU implementation for 64-bit risc-v in arch/risc-v/src/common/riscv_mmu.h.
/* SvX definitions, only Sv39 is currently supported, but it should be
 * trivial to extend the driver to support other SvX implementations
 *
 * Sv39 has:
 * - 4K page size
 * - 3 page table levels
 * - 9-bit VPN width
 */

#ifdef CONFIG_ARCH_MMU_TYPE_SV39
#define RV_MMU_PTE_PADDR_SHIFT  (10)
#define RV_MMU_PTE_PPN_MASK     (((1ul << 44) - 1) << RV_MMU_PTE_PADDR_SHIFT)
#define RV_MMU_PTE_PPN_SHIFT    (2)
#define RV_MMU_VPN_WIDTH        (9)
#define RV_MMU_VPN_MASK         ((1ul << RV_MMU_VPN_WIDTH) - 1)
#define RV_MMU_PT_LEVELS        (3)
#define RV_MMU_VADDR_SHIFT(_n)  (RV_MMU_PAGE_SHIFT + RV_MMU_VPN_WIDTH * \
                                 (RV_MMU_PT_LEVELS - (_n)))
#define RV_MMU_SATP_MODE        (SATP_MODE_SV39)
#define RV_MMU_L1_PAGE_SIZE     (0x40000000) /* 1G */
#define RV_MMU_L2_PAGE_SIZE     (0x200000)   /* 2M */
#define RV_MMU_L3_PAGE_SIZE     (0x1000)     /* 4K */

/* Minimum alignment requirement for any section of memory is 2MB */

#define RV_MMU_SECTION_ALIGN        (RV_MMU_L2_PAGE_SIZE)
#define RV_MMU_SECTION_ALIGN_MASK   (RV_MMU_SECTION_ALIGN - 1)
#else
#error "Unsupported RISC-V MMU implementation selected"
#endif /* CONFIG_ARCH_MMU_TYPE_SV39 */

Is there any guidance or plans for implementing something similar for 32-bit? Or possibly someone has attempted something similar in the past?

64bit MMU is added by @pussuw. @pussuw could you give some hint?

The implementation I need is 64-bit only so at least I don't plan to do / will not do anything regarding riscv32.
But it should be pretty easy to extend the implementation for other SvXX types also (including Sv32). I intentionally wrote the driver keeping scalability for other architectures in mind.
In theory you only need to do the above definitions for Sv32 and the implementation "should work". Might be a good idea to check the macros outside of that #ifdef as well. Obviously this has never been tested with Sv32 but I use the Sv39 implementation daily and it works just fine.
When I did the implementation riscv32 and riscv64 had separate implementations, which were merged into one later. So the driver and the public interface might not be 100% compatible with Sv32 as-is.
The most important document obviously is the riscv privileged specification which tells you how the address translations work.
Please do keep in mind that when running riscv in supervisor mode you need some sort of SBI implementation. That implementation is out-of-scope here and you need to choose and integrate it yourself.

source: gharchive/issue

repo: apache/nuttx


text:
mm/alloc: remove all unnecessary cast for alloc

Summary
mm/alloc: remove all unnecessary cast for alloc
Fix the minor style issue and remove unnecessary cast
Signed-off-by: chao an anchao@xiaomi.com
Impact
N/A
Testing
ci-check

please add a commit message

Done

@pkarashchenko  all addressed, thank you!

Please fix the leftovers and we can move to merge

@pkarashchenko Done, please review again.


text:
Hide version in activation's path generated by generateFallbackActivation

Description
If the activation is generated by the generateFallbackActivation method in the InvokerReactive, it contains the version in the path annotation.
In this case, the ES activation store can't search for activations by the name without version.
And also, all other components that generate activation don't store version, but only InvokerReactive stores the version in the path annotation. It must be consistent.
https://github.com/apache/openwhisk/blob/a1ba98778e3db80a06fe707c47c5f4c05d8b85ee/core/controller/src/main/scala/org/apache/openwhisk/core/controller/actions/PrimitiveActions.scala#L589
https://github.com/apache/openwhisk/blob/a1ba98778e3db80a06fe707c47c5f4c05d8b85ee/core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/ContainerProxy.scala#L1070
https://github.com/apache/openwhisk/blob/8759cadeea6101a3693aac1d34d4069f516d5558/core/controller/src/main/scala/org/apache/openwhisk/core/controller/actions/SequenceActions.scala#L243
Path annotation
{
    "annotations": [
        {
            "key": "kind",
            "value": "nodejs:10"
        },
        {
            "key": "path",
            "value": "seonghyun/a2@0.0.154"
        }
    ]
}

Path annotation (expected)
{
    "annotations": [
        {
            "key": "kind",
            "value": "nodejs:10"
        },
        {
            "key": "path",
            "value": "seonghyun/a2"
        }
    ]
}

Related issue and scope

None

My changes affect the following components

[x] Data stores (activation format)

Types of changes

[x] Enhancement or new feature (adds new functionality).

Checklist:

[x] I signed an Apache CLA.
[x] I reviewed the style guides and followed the recommendations (Travis CI will check :).
[ ] I added tests to cover my changes.
[x] My changes require further changes to the documentation.
[ ] I updated the documentation where necessary.

Codecov Report

Merging #5025 (06fc8b6) into master (cb16450) will decrease coverage by 6.85%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #5025      +/-   ##
==========================================
- Coverage   83.77%   76.91%   -6.86%
==========================================
  Files         202      202
  Lines        9787     9787
  Branches      432      432
==========================================
- Hits         8199     7528     -671
- Misses       1588     2259     +671

Impacted Files
Coverage Δ

...pache/openwhisk/core/invoker/InvokerReactive.scala
80.00% <ø> (ø)

...core/database/cosmosdb/RxObservableImplicits.scala
0.00% <0.00%> (-100.00%)
:arrow_down:

...ore/database/cosmosdb/cache/CacheInvalidator.scala
0.00% <0.00%> (-100.00%)
:arrow_down:

...e/database/cosmosdb/cache/ChangeFeedConsumer.scala
0.00% <0.00%> (-100.00%)
:arrow_down:

...core/database/cosmosdb/CosmosDBArtifactStore.scala
0.00% <0.00%> (-95.85%)
:arrow_down:

...sk/core/database/cosmosdb/CosmosDBViewMapper.scala
0.00% <0.00%> (-93.90%)
:arrow_down:

...tabase/cosmosdb/cache/CacheInvalidatorConfig.scala
0.00% <0.00%> (-92.31%)
:arrow_down:

...enwhisk/connector/kafka/KamonMetricsReporter.scala
0.00% <0.00%> (-83.34%)
:arrow_down:

...e/database/cosmosdb/cache/KafkaEventProducer.scala
0.00% <0.00%> (-78.58%)
:arrow_down:

...whisk/core/database/cosmosdb/CosmosDBSupport.scala
0.00% <0.00%> (-74.08%)
:arrow_down:

... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cb16450...06fc8b6. Read the comment docs.

authors: codecov-io upgle

repo: apache/openwhisk


text:
SIGSEGV When read statistics of an all null boolean stripe.

orc-statistic is a tool using Apache Orc Reader[C++] to get statistics of orc file,
and I got a SIGSEGV when processing a stripe with an all null boolean columns.
When BooleanColumnStatisticsImpl::toProtoBuf, mutable_bucketstatistics() is called.
https://github.com/apache/orc/blob/0aba9875be4396c40adedae5a257e622d43b64eb/c%2B%2B/src/Statistics.hh#L463
The Protocol Buffers Docs[https://protobuf.dev/reference/cpp/cpp-generated/] saying:
Bar* mutable_foo(): Returns a pointer to the mutable Bar object that stores the field’s value. If the field was not set prior to the call, then the returned Bar will have none of its fields set (i.e. it will be identical to a newly-allocated Bar). After calling this, has_foo() will return true and foo() will return a reference to the same instance of Bar.

So in  BooleanColumnStatisticsImpl::BooleanColumnStatisticsImpl, https://github.com/apache/orc/blob/0aba9875be4396c40adedae5a257e622d43b64eb/c%2B%2B/src/Statistics.cc#L201, pb.has_bucketstatistics() is true. But pb.bucketstatistics().count(0) will access to a invalid field.

Thanks for reporting the issue! Would you like to create a PR with a fix?

Thanks for reporting the issue! Would you like to create a PR with a fix?

ok

Do you have an JIRA account? You may need to create an JIRA first and submit an PR linking to the JIRA title. @oushu1zhouzhanghua1

Which Apache ORC versions are affected by this, @oushu1zhouzhanghua1 ?

Gentle ping, @oushu1zhouzhanghua1 .

Since there is not much activity, let's close this issue. I hope Java-version orc-tools can help the situations, @oushu1zhouzhanghua1 .

repo: apache/orc

license_source: bigquery


text:
HDDS-6173. Merge master branch including s3g persistent connection changes into feature branch

What changes were proposed in this pull request?
Merge requested in PR #2945 to merge Apache Ozone master branch with HDDS-5881 changes INTO HDDS-4400-s3-performance feature branch.  Merge allows s3 gateway gRPC to work with s3 gateway persistent connection changes made in HDDS-5881 found in master branch.
see major code files affected in attachment.
merge_conflicts.txt
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6173
How was this patch tested?
Unit tests for s3 gateway and acceptance tests for s3 gateway.
Passes both s3 gateway specific unit tests and acceptance tests with the ozonesecure cluster.

Hey Neil, 800 files seems like a lot of files to review.  I wonder if we could make it easier.
If I'm understanding things correctly, there are really just 15 files that really need to be reviewed.  Here are those diffs:
https://github.com/GeorgeJahad/ozone/compare/HDDS-4440-updated-feature-branch..neils-dev:5881_merge
Note that diff is between a branch I created from the old feature branch, and your 5881_merge branch.
To understand the diff better let me describe how the two branches were created.
To create your 5881_merge branch, you:

started from the feature branch,
merged in the latest master, 30cb5e7644fc3fcb524ba9ede9059a118b03f2fd,
resolved the merge conflicts, and
integrated the s3 grpc changes with the hadoop rpc changes.

To create my branch, I:

got the s3grpc diffs from the old feature branch and
patched 30cb5e7644fc3fcb524ba9ede9059a118b03f2fd with it.

I let the merge conflicts be rejected, and didn't try to integrate the hrpc changes.
Here is the code:
# tag the last master merged into the feature branch
git tag oldGrpcMaster db574d1cfffc8c0a57bef4dcdec008bb6c6c3fba

# get the s3grpc changes in the old feature branch
git diff oldGrpcMaster  HDDS-4440-s3-performance > /tmp/s3grpc.diff

# tag the master that will be the basis of the new feature branch
git tag newGrpcMaster 30cb5e7644fc3fcb524ba9ede9059a118b03f2fd

#create updated feature branch
git checkout newGrpcMaster
git checkout -b HDDS-4440-updated-feature-branch

# Now patch the new feature branch with the s3grpc changes
# note that the changes that don't merge cleanly will be rejected
# and will be resolved by hand in the next commit
patch -p1 -f < /tmp/s3grpc.diff
git add .
git commit -m "added s3grpc changes not including merge conflicts"

Now that branch has all the s3grpc changes commited up to now, without resolving any merge conflicts, or integrating the grpc changes with hadoop rpc.
Those are things you added, and I think are the main things the reviewers want to see.
They get those diffs by comparing my branch with yours.
What do you think?

Thanks @GeorgeJahad, I like the idea, however there are things shown in the diffs generated that are incorrect, also missing changes ie. OzoneClientProto.  It does give a good reference to some of the changes.  This is linked to PR #2945. and was asked for there, so a better idea of the changes are seen in that PR (excluding the TLS).  In that PR the hrpc changes (HDDS-5581) are 'patched' into the 4440-feature-branch and clearer to see the integration between the feature branch and the 5581.  Basically, this PR is a merge from the master into the feature branch.  As a result some changes were necessary to the already reviewed s3gateway gRPC code  in the feature branch.  No additional code was added as the TLS code (2945) will be added to this merge.

Added a file to the pull-request description: merge_integration_diffs.txt  based on followup discussions with @GeorgeJahad to highlight the changes made to the s3g feature branch to integrate with the merge of Master to HDDS-4440-s3-performance.

Taking a look @neils-dev

I went over the commits that might be in conflict and looks good.
LGTM
cc @adoroszlai can you please merge this PR.

authors: GeorgeJahad kerneltime neils-dev

repo: apache/ozone


text:
HDDS-8023. [HTTPFSGW] Remove support for satisfyStoragePolicy operation

What changes were proposed in this pull request?
I removed the support for satisfyStoragePolicy operation.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8023
How was this patch tested?
Built the project successfully, green CI on my fork: https://github.com/dombizita/ozone/actions/runs/4258119003

Thanks Zita for working on this. It's looking good to me, I just have 1 question/remark. SATISFYSTORAGEPOLICY enum value is no longer used for practical purposes only for throwing the UnsupportedOperationException. Is it still needed, or is it possible to remove that operation entirely? I'm not too familiar with how HTTPFS works, and what side effects it could have.

hi @Galsza, thanks for the review! we decided to keep SATISFYSTORAGEPOLICY as it is a valid operation in WebHDFS, check here, so we want to throw a specific exception in that case, as it is not just a general not supported operation.


text:
HDDS-8258. RocksIterator not closed properly in KeyValueHandler.logBlocksIfNonZero

What changes were proposed in this pull request?
Close BlockIterator in KeyValueHandler#logBlocksIfNonZero.
https://issues.apache.org/jira/browse/HDDS-8258
How was this patch tested?
https://github.com/adoroszlai/hadoop-ozone/actions/runs/4509916244

Thanks @duongkame for the review.


text:
HDDS-10370. Recon - Handle the pre-existing missing empty containers in clusters.

What changes were proposed in this pull request?
This PR addresses the corner case when a Recon has earlier identified some MISSING (UNHEALTHY) containers, but they all were EMPTY (No keys mapped), so Recon should filter out such existing MISSING (UNHEALTHY) containers.
In a running cluster, Recon may have identified some MISSING (UNHEALTHY) containers before the HDDS-9695 was applied. After the fix is applied, in a running cluster, existing MISSING (UNHEALTHY) containers which were actually EMPTY were still shown as MISSING, so this PR change is to filter out such existing MISSING (UNHEALTHY) containers which were actually EMPTY.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10370
How was this patch tested?
Existing Junit integration test org.apache.hadoop.ozone.recon.TestReconTasks#testEmptyMissingContainerDownNode was updated and tested.

@sumitagrawl @dombizita @fapifta , pls review.

Working on fixing TestContainerHealthTask test case failure.

Working on fixing TestContainerHealthTask test case failure.

This is fixed.


text:
Phoenix 7211 : Identify IT tests that can be run successfully against real distributed cluster

Jira: PHOENIX-7211

Thanks @Divneet18 for your work. @virajjasani @stoty @tkhurana, Please review when you get the chance.

Please also fix the markdownlint error.

Awaiting final approval from @stoty

authors: Divneet18 jpisaac stoty virajjasani

repo: apache/phoenix


text:
change default range index version to v2

The new range index has been shown to decrease query latency significantly and this enables it by default.

The test failure seems completely unrelated to the change. Could a committer rerun please?

@richardstartin Triggered a rerun, but this might be related to the segment size change for the new index version

Codecov Report

Merging #7815 (a1a9b95) into master (b7b8258) will decrease coverage by 6.43%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #7815      +/-   ##
============================================
- Coverage     71.66%   65.23%   -6.44%
+ Complexity     4086     4082       -4
============================================
  Files          1578     1533      -45
  Lines         80773    78903    -1870
  Branches      12002    11799     -203
============================================
- Hits          57888    51471    -6417
- Misses        18993    23766    +4773
+ Partials       3892     3666     -226

Flag
Coverage Δ

integration1
?

integration2
?

unittests1
68.67% <ø> (-0.02%)
:arrow_down:

unittests2
14.61% <ø> (-0.01%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

.../apache/pinot/spi/config/table/IndexingConfig.java
92.20% <ø> (ø)

...a/org/apache/pinot/common/metrics/MinionMeter.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...g/apache/pinot/common/metrics/ControllerMeter.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../apache/pinot/common/metrics/BrokerQueryPhase.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../apache/pinot/common/metrics/MinionQueryPhase.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...he/pinot/common/messages/SegmentReloadMessage.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...pinot/core/data/manager/realtime/TimerService.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...pinot/minion/exception/TaskCancelledException.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...not/common/exception/HttpErrorStatusException.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...t/core/startree/plan/StarTreeDocIdSetPlanNode.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 363 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b7b8258...a1a9b95. Read the comment docs.

@Jackie-Jiang I see, that's a different error, it didn't get that far before I force pushed the first time.

To confirm, reader should still be able to read existing segments with old version right ? Are we planning to convert existing segments to V3 during segment preprocessor / reload ?

@siddharthteotia Yes, reader can read all different versions, and the reader support is included in 0.9.0 release

authors: Jackie-Jiang codecov-commenter richardstartin siddharthteotia

repo: apache/pinot


text:
Add swagger authorization header support

Description
This PR adds the Authorization: xxx header support into the Swagger UI in Pinot.
Now when the users opens Pinot Swagger, they will see an extra "Authorize" button.

If the user does not authorize, the Swagger will behave like before, send all the API requests through
If the user clicks "Authorize" button, the user can pass in one extra bearer token and send them into Pinot

Below is a gif for authorization (Note that the Authorization is not mandatory, it just plainly adds a header)

Release Notes

Added Authentication header option into Swagger UI for all endpoints

Tests Done
Run the PinotQuickStart example and recorded the screen like above shows

cc @apucher @xiangfu0

Codecov Report

Merging #8830 (cd923d4) into master (7b9e16b) will decrease coverage by 35.63%.
The diff coverage is 16.66%.

@@              Coverage Diff              @@
##             master    #8830       +/-   ##
=============================================
- Coverage     69.82%   34.19%   -35.64%
+ Complexity     4658      185     -4473
=============================================
  Files          1741     1741
  Lines         91474    91472        -2
  Branches      13674    13674
=============================================
- Hits          63875    31279    -32596
- Misses        23172    57626    +34454
+ Partials       4427     2567     -1860

Flag
Coverage Δ

integration1
?

integration2
25.60% <16.66%> (+0.11%)
:arrow_up:

unittests1
?

unittests2
14.18% <0.00%> (+0.06%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...ot/broker/api/resources/PinotBrokerAppConfigs.java
0.00% <ø> (ø)

...e/pinot/broker/api/resources/PinotBrokerDebug.java
85.71% <ø> (ø)

...t/broker/api/resources/PinotBrokerHealthCheck.java
0.00% <ø> (ø)

...pinot/broker/api/resources/PinotBrokerRouting.java
0.00% <ø> (ø)

...pinot/broker/api/resources/PinotClientRequest.java
50.94% <ø> (-7.55%)
:arrow_down:

...sources/PinotAccessControlUserRestletResource.java
61.97% <ø> (ø)

...ller/api/resources/PinotBrokerRestletResource.java
60.24% <ø> (ø)

.../controller/api/resources/PinotClusterConfigs.java
34.21% <ø> (ø)

...oller/api/resources/PinotControllerAppConfigs.java
100.00% <ø> (ø)

...ler/api/resources/PinotControllerAuthResource.java
0.00% <ø> (ø)

... and 1099 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cd55015...cd923d4. Read the comment docs.

authors: codecov-commenter dongxiaoman


text:
add validation for proto decoder

Add validation for protobuf decoder on the necessary properties

Codecov Report

Merging #9588 (5616c12) into master (cea282c) will decrease coverage by 2.65%.
The diff coverage is 62.50%.

@@             Coverage Diff              @@
##             master    #9588      +/-   ##
============================================
- Coverage     69.99%   67.34%   -2.66%
+ Complexity     5261     5110     -151
============================================
  Files          1934     1443     -491
  Lines        103303    75578   -27725
  Branches      15694    12056    -3638
============================================
- Hits          72306    50897   -21409
+ Misses        25896    21014    -4882
+ Partials       5101     3667    -1434

Flag
Coverage Δ

integration1
?

integration2
?

unittests1
67.34% <62.50%> (-0.03%)
:arrow_down:

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...he/pinot/segment/local/utils/TableConfigUtils.java
68.01% <62.50%> (-0.15%)
:arrow_down:

...va/org/apache/pinot/core/routing/RoutingTable.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...va/org/apache/pinot/common/config/NettyConfig.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...a/org/apache/pinot/common/metrics/MinionMeter.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...g/apache/pinot/common/metrics/ControllerMeter.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../apache/pinot/common/metrics/BrokerQueryPhase.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../apache/pinot/common/metrics/MinionQueryPhase.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ache/pinot/server/access/AccessControlFactory.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...he/pinot/common/messages/SegmentReloadMessage.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...he/pinot/common/messages/TableDeletionMessage.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 856 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


text:
Maven profile configuration problem

Describe the bug
Maven profile configuration problem.
<profile>
      <id>main</id>
      <activation>
        <activeByDefault>true</activeByDefault>
        // Why add jdk activation mode in main profile?
        // If so. When I want to use the core-modules profile to package,
        // some modules that I don't want to activate will also be introduced
        <jdk>[8,)</jdk>
      </activation>
      <modules>
     ...
      </modules>
</profile>

To Reproduce
Steps to reproduce the behavior:

Master branch
Use command 'mvn -Pcore-modules -DskipTests clean package'

Expected behavior
Than some modules like test or docker modules will be packaged, this is not what I want.
Screenshots
No.
Desktop (please complete the following information):

OS: macos

Additional context
Add any other context about the problem here.

The issue had no activity for 30 days, mark with Stale label.

Closed as stale. Please create a new issue if it's still relevant to the maintained versions.
Now, you can ask questions in https://github.com/apache/pulsar/discussions/categories/q-a instead, or send feature request which you don't have idea to implement in https://github.com/apache/pulsar/discussions/categories/general, since we almost use issues to track actionable tasks.

authors: codelipenghui tisonkun viviel

repo: apache/pulsar


text:
PulsarClientException should not extends IOException

Is your enhancement request related to a problem? Please describe.
Currenlty the PulsarClientException extends the IOException class which is a checked-exception. Of-course checked vs unchecked exception has been a long debate in the Java community but I see two main issues with this is Pulsar :
1. Bad Developer Experience
As most of the methods provided by the client/producer/consumer/reader APIs can throw a PulsarClientException, this forces developers to eventually wrap the code with a try-catch block or simply rethrow the exception, which can make future refactoring painful.
Furthermore, in many cases we just don't have the choice to reject the exception again because we can't do anything else but stop ourapplication, eg: we failed to initialise the PulsarClient.
Finally, it also limits the use of Java lambda expressions or portability to other JVM languages such as Kotlin.
2. Inconsistency
For example, the InvalidConfigurationException extends the PulsarClientException and hence the IOException but is not related to any interrupted I/O operations.
Describe the solution you'd like
Create a top-level exception class PulsarException that extends the RuntimeException class. Both PulsarClientException and PulsarAdminException should then extend the PulsarException class.
Describe alternatives you've considered
Additional context
This is a breaking-change.

Closed as stale and no one worked on it. Please open a new issue if it's still relevant to the maintained versions.

authors: fhussonnois tisonkun


text:
[function] fix-typo

Motivation
fix-typo

/pulsarbot run-failure-checks


text:
[fix][broker] Fix the managed ledger factory shutdown stuck due to scheduled executor shutdown first

Fixes #16665
Motivation
Fix the managed ledger factory shutdown stuck due to scheduled executor shutdown first
2022-07-18T14:03:01,708 - ERROR - [main:MockedBookKeeperTestCase@107] - tearDown Error
java.util.concurrent.TimeoutException: null
	at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095) ~[?:?]
	at org.apache.bookkeeper.test.MockedBookKeeperTestCase.tearDown(MockedBookKeeperTestCase.java:101) ~[test-classes/:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132) ~[testng-7.3.0.jar:?]
	at org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:61) ~[testng-7.3.0.jar:?]
	at org.testng.internal.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:366) ~[testng-7.3.0.jar:?]
	at org.testng.internal.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:320) ~[testng-7.3.0.jar:?]
	at org.testng.internal.TestInvoker.runConfigMethods(TestInvoker.java:701) ~[testng-7.3.0.jar:?]
	at org.testng.internal.TestInvoker.runAfterGroupsConfigurations(TestInvoker.java:677) ~[testng-7.3.0.jar:?]
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:661) ~[testng-7.3.0.jar:?]
	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174) ~[testng-7.3.0.jar:?]
	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46) ~[testng-7.3.0.jar:?]
	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822) ~[testng-7.3.0.jar:?]
	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147) ~[testng-7.3.0.jar:?]
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146) ~[testng-7.3.0.jar:?]
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128) ~[testng-7.3.0.jar:?]
	at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]
	at org.testng.TestRunner.privateRun(TestRunner.java:764) [testng-7.3.0.jar:?]
	at org.testng.TestRunner.run(TestRunner.java:585) [testng-7.3.0.jar:?]
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384) [testng-7.3.0.jar:?]
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378) [testng-7.3.0.jar:?]
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337) [testng-7.3.0.jar:?]
	at org.testng.SuiteRunner.run(SuiteRunner.java:286) [testng-7.3.0.jar:?]
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53) [testng-7.3.0.jar:?]
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96) [testng-7.3.0.jar:?]
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218) [testng-7.3.0.jar:?]
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140) [testng-7.3.0.jar:?]
	at org.testng.TestNG.runSuites(TestNG.java:1069) [testng-7.3.0.jar:?]
	at org.testng.TestNG.run(TestNG.java:1037) [testng-7.3.0.jar:?]
	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135) [surefire-testng-3.0.0-M3.jar:3.0.0-M3]
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112) [surefire-testng-3.0.0-M3.jar:3.0.0-M3]
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:123) [surefire-testng-3.0.0-M3.jar:3.0.0-M3]
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:90) [surefire-testng-3.0.0-M3.jar:3.0.0-M3]
	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146) [surefire-testng-3.0.0-M3.jar:3.0.0-M3]
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) [surefire-booter-3.0.0-M3.jar:3.0.0-M3]
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) [surefire-booter-3.0.0-M3.jar:3.0.0-M3]
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) [surefire-booter-3.0.0-M3.jar:3.0.0-M3]
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) [surefire-booter-3.0.0-M3.jar:3.0.0-M3]
2022-07-18T14:03:01,710 - INFO  - [main:MockedBookKeeperTestCase@70] - >>>>>> starting public void org.apache.bookkeeper.mledger.impl.ManagedLedgerTest.avoidUseSameOpAddEntryBetweenDifferentLedger() throws java.lang.Exception

The root cause is the scheduled executor shutdown before all the ledgers closed. The close ledger operation will be stuck due to the scheduled executor being shut down.
https://github.com/apache/pulsar/blob/c60e063fcab18a2da9757ca5f13479e268e0681d/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java#L241
Modification
Shut down the scheduled executor after all the ledgers and cursors are closed.
Documentation
Check the box below or label this PR directly.
Need to update docs?

[ ] doc-required
(Your PR needs to update docs and you will update later)

[x] doc-not-needed
(Please explain why)

[ ] doc
(Your PR contains doc changes)

[ ] doc-complete
(Docs have been already added)

When we called  ManagedLedgerImpl#updateLedgersIdsComplete ->  OpAddEntry#initiate, and  then  If ledger.asyncAddEntry(L140) callback successfully, the OpAddEntry state will be COMPLETED  instead of INITIATED .  I think this is the root cause to make the test failure.
https://github.com/apache/pulsar/blob/a1b8476e40b6a40e82062915d13e4dc9d7dff9cb/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java#L125-L140

When we called ManagedLedgerImpl#updateLedgersIdsComplete -> OpAddEntry#initiate, and then If ledger.asyncAddEntry(L140) callback successfully, the OpAddEntry state will be COMPLETED instead of INITIATED . I think this is the root cause to make the test failure.

Oh yes. I forgot to update the PR description. This is an issue I found during the ManageLedger test. Can't fix https://github.com/apache/pulsar/issues/16665

authors: AnonHxy codelipenghui


text:
[fix][test] Fix PrometheusMetricsTest.testBrokerConnection

Fixes #18535
Motivation
in #17905, we enable transaction coordinator in the broker, this will lead to opening a connection in broker, so connection_count should recalculate.
Modifications
Enable transaction coordinator in broker only in testDuplicateMetricTypeDefinitions.
Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository: https://github.com/coderzc/pulsar/pull/33

Codecov Report

Merging #18539 (4362c4b) into master (aeb4503) will increase coverage by 5.36%.
The diff coverage is 33.33%.

@@             Coverage Diff              @@
##             master   #18539      +/-   ##
============================================
+ Coverage     31.39%   36.75%   +5.36%
+ Complexity     6651     1956    -4695
============================================
  Files           697      208     -489
  Lines         68015    14430   -53585
  Branches       7285     1582    -5703
============================================
- Hits          21353     5304   -16049
+ Misses        43667     8550   -35117
+ Partials       2995      576    -2419

Flag
Coverage Δ

unittests
36.75% <33.33%> (+5.36%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...va/org/apache/pulsar/client/impl/ConsumerImpl.java
14.84% <0.00%> (ø)

...he/pulsar/client/impl/MultiTopicsConsumerImpl.java
22.26% <0.00%> (ø)

...lsar/client/impl/conf/ClientConfigurationData.java
96.66% <100.00%> (+0.11%)
:arrow_up:

...apache/pulsar/client/impl/AutoClusterFailover.java
70.00% <0.00%> (-6.12%)
:arrow_down:

...che/bookkeeper/mledger/impl/ManagedLedgerImpl.java

...pulsar/broker/admin/impl/PersistentTopicsBase.java

...sistent/PersistentDispatcherMultipleConsumers.java

...ersistentStickyKeyDispatcherMultipleConsumers.java

...ersistentStreamingDispatcherMultipleConsumers.java

...apache/pulsar/broker/admin/v1/SchemasResource.java

... and 484 more

/pulsarbot run-failure-checks

authors: codecov-commenter coderzc


text:
[Issue 6024][pulsar_storm] Support alternate output streams in PulsarSpout

Fixes #6024
Motivation
This is all described in detail in https://github.com/apache/pulsar/issues/6024, but in short, an insurmountable obstacle to using Pulsar in our storm topology is the fact that PulsarSpout only emits to the "default" stream. In our environment, we need to emit on different streams based on the content of each received message. This change extends PulsarSpout to recognize a Values extension that specifies an alternate output stream, and uses that stream when given.
Modifications
A new PulsarTuple class is added. It extends Values and adds a method to return the output stream.
When emitting a tuple after calling toValues(msg), PulsarSpout checks if the returned Values is a PulsarTuple. If so, it emits to the designated stream, otherwise it emits as before.
Verifying this change

[x] Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

A test case was added to PulsarSpoutTest to verify that when PulsarTuple us used, the tuple is emitted in the specified stream.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API: (yes?) Existing APIs don't change, but a new class is added that carries additional information.
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation

Does this pull request introduce a new feature? (yes)
If yes, how is the feature documented? JavaDocs

Any guidance on what's needed for this would be appreciated? For example, should I do a separate PR to master?

Hi @dawillcox, we are working on the master branch. Can you please send the pull request based on master?
See the contribution guide: http://pulsar.apache.org/en/contributing/

OK, new PR created: https://github.com/apache/pulsar/pull/6039
This one presumably should be closed.

authors: dawillcox sijie


text:
[ISSUE #44] Support shared subscription

fix: #44
TODO:
1.pop ack
2. rocketmq support pop on lmq

look forward ...

We plan to develop all development based on the develop branch, can we submit the pr to the develop branch?

We plan to develop all development based on the develop branch, can we submit the pr to the develop branch?

sure, done

Can you link the place（issue/pr） where rocketmq supports the lmq pop

ref: https://github.com/apache/rocketmq/pull/6577

authors: ferrirW pingww

repo: apache/rocketmq-mqtt


text:
rocketmq message hasnot been sent, Caused by ConcurrentModificationException

Use rocketMQTemplate.asyncSend to send msg; throw org.springframework.messaging.MessagingException: nested exception is java.util.ConcurrentModificationException, failedMessage=GenericMessage [payload=byte[142], headers={id=9f951e70-dbae-c369-3877-d8438f5f22bd, contentType=application/json, TAGS=sign, timestamp=1621476904559}]
rocketmq-client version is 4.4.0

Is not a problem will now, is the incidental question, 100 requests in 1 or 2 times, lead to the failure message is sent

@catalpa-tree could you leave more information in here or submit a new issue in this repo(if you used this project instead of rocketmq-client):https://github.com/apache/rocketmq-spring

@duhenglucky  same problem!  use rocketmq-spring-boot-starter（2.1.0），broker version is 4.7.1，1 master 2 slave dleger

authors: catalpa-tree duhenglucky dujingsong

repo: apache/rocketmq


text:
[Issue #3576]：Optimization: In the case of high concurrency，ThreadLocalRandom is better than Random

DefaultMQProducerImpl .java
code1:
public class DefaultMQProducerImpl implements MQProducerInner { private final InternalLogger log = ClientLogger.getLog(); private final Random random = new Random();
code2:
private SendResult sendDefaultImpl(
Message msg,
final CommunicationMode communicationMode,
final SendCallback sendCallback,
final long timeout
) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
this.makeSureStateOK();
Validators.checkMessage(msg, this.defaultMQProducer);
final long invokeID = random.nextLong();

Coverage increased (+0.004%) to 55.154% when pulling 0f05762e845080316ce243a6c9f04a3df39103f3 on mrhbj:develop into 4daade3c1ac4ad0650b6e77430cd93ed9eeba0a7 on apache:develop.

authors: coveralls mrhbj


text:
[ISSUE #8517] client send UNREGISTER_CLIENT request twice may cause proxy NPE

Which Issue(s) This PR Fixes

Fixes #8517
Brief Description
client send UNREGISTER_CLIENT request twice may cause proxy NPE
check channel is null before use it, if channel is null, log warn instead of NPE

How Did You Test This Change?

Codecov Report
Attention: Patch coverage is 0% with 16 lines in your changes missing coverage. Please review.

Project coverage is 46.86%. Comparing base (fa0cd49) to head (b3f7df8).

Files
Patch %
Lines

...proxy/remoting/activity/ClientManagerActivity.java
0.00%
16 Missing :warning:

Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #8528      +/-   ##
=============================================
- Coverage      46.95%   46.86%   -0.09%
+ Complexity     11325    11296      -29
=============================================
  Files           1275     1275
  Lines          89061    89067       +6
  Branches       11454    11456       +2
=============================================
- Hits           41815    41739      -76
- Misses         42081    42155      +74
- Partials        5165     5173       +8

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


text:
Compiler Build Failed

runt ant wipe-all then run ant again.

authors: aharui pashminakazi

repo: apache/royale-asjs


text:
s:List change event is not being triggered

Project Details are :
https://paste.apache.org/cpo8y
Test case is available on VDI.
Go to Import from Select in Menu
Go to Import -> Import Chargesfrom Setups in Menu
File : com/dbz/modules/IMP/dbzImportChargesSetup.mxml
Click on this button :
https://user-images.githubusercontent.com/42200979/77047368-6ec4b080-69e6-11ea-9fda-6f316d7c0ddc.png
New window will be opened which is : com/dbz/modules/IMP/dbzimportChargerFormulaPopup
In Flex
when we double click any element from s:List with ID lstSysVariable named system Columns a change event is triggered fn_addSysColumns
private function fn_addSysColumns(e:Event):void {
				selectedListItem = List(e.currentTarget).selectedItem;
				impchg_Formula_Disp.text = impchg_Formula_Disp.text + " " + selectedListItem.label;
				impchg_Formula.text = impchg_Formula.text + " " + selectedListItem.id;

			}

and that element will be added in Formula Editor which is mx:TextArea
<mx:TextArea id="impchg_Formula_Disp" maxChars="400" width="593" tabIndex="1" x="1" y="2" height="146" valueCommit="bValidate=false;"/>

In Royale
when i double click on any element from system column nothing is added in Formula Editor TextArea because change event is not being triggered

On the first click of the double-click, does the list selection change?  If you make a simple test case with s:List, does the list selection change?

@aharui Thanks now change event is being triggered..
but there's an exception in function which is called on change event.

Change function :
private function fn_addSysColumns(e:Event):void {
				selectedListItem = List(e.currentTarget).selectedItem;
				impchg_Formula_Disp.text = impchg_Formula_Disp.text + " " + selectedListItem.label;
				impchg_Formula.text = impchg_Formula.text + " " + selectedListItem.id;

			}

selectedListItem is null that's why selectedListItem .label is null
why selectedListItem is setted null?
selectedListItem is object variable

@aharui Thank u so much.
On a single click value is added in formula editor 2 times


text:
SAMZA-1725: Set travis build idle time out to 20 minutes.

Problem
Currently, average build time of samza codebase is 15 to 20 minutes. However, travis has a build idle timeout of 10 minutes and fails the build if the gradle build command doesn't log anything to console
for 10 minutes(occurs when running tests in samza-test module).
Sample travis build failure error:
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

Fix:
Increasing the build idle-wait timeout value to 20 minutes.

@dnishimura
Sample successful build on top of this patch: https://travis-ci.org/shanthoosh/samza/jobs/382416654
Please take a look when you get a chance.

Does increasing the timeout just mask the issues with all the connection and Kafka connection exceptions I see in the build log? Or are those expected?

authors: dnishimura shanthoosh

repo: apache/samza


text:
[Bug][Seatunnel-web] Support encyrption of datasource data

Currently Datasource passwords are stored as it is in database and also is sent to seatunnel-engine in .profile file. All sensitive information like password should be encrypted. Also the API response should be encrypted for datasource password.
This patch handles the encryption of data using the encryption type available in seatunnel-engine. We can use base64 encryption type.
When we enable this encryption type, we need it configure it in job env like below:
env {
"shade.identifier" =base64
}
add shade.identifier to your env. you can refer to below doc for more understanding
https://seatunnel.apache.org/docs/2.3.2/connector-v2/Config-Encryption-Decryption/

I will review and test this PR in 1-2 days

How this feature works? I see encryption only, I do not see any decryption. How it will be able to connnect to databases, it will fail.
Can you please create  datasource, create a job, execute the job and attache the snapshot here.

How this feature works? I see encryption only, I do not see any decryption. How it will be able to connnect to databases, it will fail. Can you please create datasource, create a job, execute the job and attache the snapshot here.

Thank you for reviewing. Decryption is done on the engine side. There is a ConfigShadeUtils which takes care of the decryption of password while connecting to the datasource. Sure, i can share the details.

Decryption is done on the engine side.
We are addressing two components in this context: the datasource and the generated job configuration.

In the Seatunnel-web, the datasource is used during job design via the UI. It's essential that it can establish a connection with the configured database. During this connection process, the encrypted password should be decrypted for successful connectivity.
The generated job configuration file needs to include the 'datasourceEncryptType'. In the env section, it should specify something like 'shade.identifier = "base64"'.

Hi @arshadmohammad
i have taken care of the scenarios you mentioned. Can you please review the updated patch

authors: BilwaST arshadmohammad

repo: apache/seatunnel-web


text:
maven cannot find shardingsphere-plugin-mode-cluster-repository-nacos

maven cannot find shardingsphere-plugin-mode-cluster-repository-nacos

Yes, we need to compile the shardingsphere-plugin project locally, the plugin will not be published to maven.

Since this issue has been resolved, I will close this issue.

authors: RaigorJiang Yule-Momoko strongduanmu

repo: apache/shardingsphere-plugin