text:
Make Lucene90 postings format to write FST off heap

Description
Only the root block will be written off-heap while the sub blocks won't be (They are small so it might not be worth it: We would need to have 1 IndexOutput for each of the sub blocks)
Note: The FSTCompiler change should be merged as part of #12981

@mikemccand I'm wondering if there is already some benchmark that can show the RAM saved by this change

BlockTree is kinda crazy how it builds up the final FST: each little backwards-recursive chunk of term-space stores its subset of terms into a baby FST, and then on grouping N such child blocks into a new parent block, decodes those N baby FSTs, appending them into a new big-baby FST.  This backwards recursive process continues for all chunks of term space, finally ending in the root block, where all the giant-baby FSTs are finally logically appended to one another to make the final FST.
This change makes only that final FST construction run off-heap, which is a good baby step.
But what if in my index all terms start with say a?  I think that will mean we do on-heap construction of basically the full sized FST?  Hmm, or, will the root block have the a prefix pointing to it, so we will in fact build the whole FST off-heap, from baby FSTs for aa*, ab*, ac*, ad*, etc.?
We might instead just switch to off-heap building once the expected FST size crosses a threshold?  We can use createTempOutput to make temporary files as needed for the non-root FSTs that are "too big"?

all the giant-baby FSTs

Heh, this made me remember the awesome character, Bôh, from the incredible movie Spirited Away:

We might instead just switch to off-heap building once the expected FST size crosses a threshold? We can use createTempOutput to make temporary files as needed for the non-root FSTs that are "too big"?

I think this is a good idea. Wondering how should we choose a reasonable threshold? Maybe it could be a parameter? (Was afraid introducing another parameter would also increase the configuration complexity of the system).
One of the trade-off here is that they could potentially slow down the indexing: Apart from the root node, we need to traverse and iterate through the whole FST, and off-heap traversal might be slower than on-heap traversal (I think we saw 17% increases in the Synonym off-heap reading https://github.com/apache/lucene/pull/13054). For root node, it doesn't need to be traversed, and we need to save it to IndexOutput anyway, so doing it off-heap actually save time: There's no need to construct the on-heap FST.

@mikemccand I tried to use createTempOutput when the size is beyond some threshold. The logic works, but I got this exception
java.lang.IllegalArgumentException: invalid codec filename '_field_temp_2.tmp', cannot end with .tmp extension
	at __randomizedtesting.SeedInfo.seed([7608D2C3E960BC5A:CF77AADE6D1859B5]:0)

It seems the Index will check every files created under the index directory and will disallow any tmp file. I think we could delete the tmp file after use, but there doesn't seem a functionality to delete IndexInput (maybe due to the abstraction and not all IndexInput is actually a File?)

If there is a way to create temp file outside of the search index, then it would work too, but I can't find it as all I/O are accessible from SegmentWriteState.directory.

Hmm, the .tmp file should never be part of the file set for the actual segment?  It should be transient, and then re-copied into the final result (the actual block tree file that stores the FST today), and only that final result file is seen by SegmentInfo?

And you can use the IndexOutput.getName() to get the String name of the temp file, and delete that file (prolly in a finally clause in case disaster strikes) using Directory.deleteFile once its contents have been saved away to the final file?

Oh Directory.deleteFile is exactly what I needed! It's silly that I missed that. I'll post another revision soon.

I've updated to use temp IndexOutput and modify the test. It seems to be working now. I'm open for suggestion of the default block heap threshold and how to configure it.

@mikemccand I've added the suggestion so that baby-giant FST will also be written off-heap if they are above some threshold. Let me know if there is other changes needed.


text:
MAHOUT-1802: Capture attached checkpoints (if cached)

Currently, the optimizer generates checkpoints and attaches them to actual logical elements of the DAG via CheckpointAction$cp.
ie:
drmC = drmA+ drmB

val cp1 = drmC.checkpoint() // checkpoint
val cp2 = drmC.checkpoint() // cp2 == cp1

drmD = cp1 + drmE // cp1 + drmE

but, in:
 drmD = drmC + drmE // computes drmA + drmB + drmC all over
drmC already has cp1 attached to it so we should assume the common computational path is the intent here regardless and should be used, instead of building plans that recompute it. That is,
drmD = drmC + drmE should imply cp1 + drmE as well even if checkpoint is not used explicitly.
This PR allows us to avoid excessive declarations like
drmAcp = drmA.checkpoint

drmB = drmAcp %*%...

and instead just use
drmA.checkpoint()

drmB = drmA %*% ....

LGTM

source: gharchive/pull-request

authors: andrewpalumbo smarthi

repo: apache/mahout

license_type: permissive

license_source: github-api


text:
NIFI-3363: PutKafka NPE with User-Defined partition

Marked PutKafka Partition Strategy property as deprecated, as Kafka 0.8 client doesn't use 'partitioner.class' as producer property, we don't have to specify it.
Changed Partition Strategy property from a required one to a dynamic property, so that existing processor config can stay in valid state.
Fixed partition property to work.
Route a flow file if it failed to be published due to invalid partition.

Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit?

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Reviewing. . .

Tested with Kafka 0.8 broker, looks good, will merge once build completes

authors: ijokarumawak olegz

repo: apache/nifi


text:
NIFI-3547 Wait Processor should not WARN for expected behavior

Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit?

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

change looks good, will get this merged.  thanks for adjusting!

authors: apiri josephxsxn


text:
NIFI-5377 prevent infinite loop if a controller service circular refe…

…rence exists
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit?

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Will review....

@markobean Thanks for the PR! I've tried running it and it does indeed address the issue called out in the JIRA. Previously, the infinite loop occurred when trying to configure the services in question. With the PR in place, I can now configure the services to build the reference cycle. However, if I try to enable them I encountered a different infinite loop here:
at org.apache.nifi.controller.service.StandardControllerServiceReference.findRecursiveReferences(StandardControllerServiceReference.java:125)
I also seem to recall that previously the framework would identify when a reference cycle was detected and it would set that flag in the ControllerServiceReferencingComponentDTO. I noticed in the UI after establishing the reference cycle, the framework did not appear to detect it. This may or may not be related to the infinite loop referenced above. I do not believe this infinite loop is a result of this PR, but it's something we should probably address now as well.
In my scenario service A did not reference any other service, service B referenced service A and C, and service C referenced service B. In your testing, did you happen to hit the same loop I did when trying to enable them?

@mcgilman Did you experience this behavior on the graph or in a unit test? After fixing the infinite loop locally I was never able to enable the controller services on the graph because they were never valid: A depends on B which depends on A. A is not enabled so B cannot be enabled and vice versa.

@JonathanKessler I experienced it on the graph. If you add another service (like I described) you should be able to enable the service with no dependencies. As part of that action, you can indicate whether you want referencing services to also be enabled.

@mcgilman I did a quick review as well and I agree with your assessment that this does in fact address the issue brought up but there are still similar issues. I attached a .patch file to the JIRA that I believe addresses those. If you want to review that .patch file and you're okay with it then we can go ahead and merge in this PR and that patch file to close this JIRA out. Thanks!

Thanks @markap14! I'll pull down this PR and apply your patch for reviewing purposes.

Thanks @markobean and @markap14. These have been merged to master.

authors: JonathanKessler markap14 markobean mcgilman


text:
NIFI-8111: Add JSLTTransformJSON processor

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Provides an alternative to JoltTransformJSON using JSLT as the transformation language. The JOLT DSL can have a steep learning curve, where JSLT is inspired by other popular tools such as jq, XPath and XQuery. The resulting NAR is < 2MB and only has Jackson as a dependency so I included it in the assembly.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically main)?

[x] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[x] Have you verified that the full build is successful on JDK 8?
[ ] Have you verified that the full build is successful on JDK 11?
[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[x] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

I'd like to reconsider this as a PR on its own without the JSLTTransformRecord processor, that one is more complicated as we'd need a full implementation of Jackson's JsonParser and related classes for NiFi record-based parsing, which is a significant effort versus this quick win for JSLT parsing if the input is JSON. If accepted I will write a follow-up Jira to add JSLTTransformRecord

Had to force-push this because of the irrelevant file changes plus I rebased against the latest main. Hopefully this is good to go after a final review @Lehel44 @MikeThomsen @exceptionfactory  Thanks!


text:
NIFI-8260 Implement an Upload File capability in the NiFi UI for flow definitions.

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_This feature allows the user to upload a previously exported JSON flow definition into the NiFi UI and it will create a new Process Group, bypassing NiFi Registry.
NIFI-8260 - [WIP] Created Upload File capability for front end and server side logic.
Created a new API endpoint to upload a JSON flow definition and transform it to a Versioned Flow Snapshot.
Fixed the client side file form to reset after submit.
Fixed the canvas to instantly update and show the process group after submitting the file.
Changed the Add Processor Group dialog UI based on design notes.
Changed the Upload File link to an icon and moved to the name input field.
Changed the 'Registry Import' link to say 'Import from Registry' and moved to the bottom of the dialog.
Display the filename when a file is selected.
Added a cancel file button to the Process Group dialog._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically main)?

[x] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[x] Have you verified that the full build is successful on JDK 8?
[ ] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

The PR is ready for initial review. The overall functionality works. I have one more UI enhancement that I'm working on now - it's for disabling the "Add Process Group" dialog "Add" button until the dialog is ready for submission.
cc: @pvillard31 @mcgilman

When I read this PR title I was thinking this was a very different thing.  Reading the JIRA I think it is more specific to being able to upload a 'flow definition' which will create that flow on the NiFi it is uploaded to.  Can we please clarify the description?
The term 'flow file' is a specific thing in NiFi and it means 'data in the flow' as represented by a set of attributes and content.

@joewitt Yes! Changed the term to flow definition.

Built the PR and gave it a try. When trying to import a flow (json file) that I just exported from somewhere else on the canvas by right clicking on a process group / download flow definition, I always got the below stack trace:
2021-02-25 23:42:36,617 ERROR [NiFi Web Server-234] o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: java.lang.NullPointerException. Returning Internal Server Error response.
java.lang.NullPointerException: null
        at org.apache.nifi.web.api.ProcessGroupResource.lambda$uploadProcessGroup$106(ProcessGroupResource.java:4373)
        at org.apache.nifi.web.api.ApplicationResource.withWriteLock(ApplicationResource.java:719)
        at org.apache.nifi.web.api.ProcessGroupResource.uploadProcessGroup(ProcessGroupResource.java:4328)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1452)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
        at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
        at org.apache.nifi.web.filter.RequestLogger.doFilter(RequestLogger.java:66)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
...

@pvillard31 Good catch - I just pushed a commit that should fix this. If uploading a flow that was downloaded from the canvas, the flow does not contain snapshot metadata. I was trying to access the metadata, but it's not necessary. Thanks.

@exceptionfactory Thanks for reviewing! I've addressed your feedback and committed the changes.

My last commit includes the disabled buttons, which completes this PR and is ready for merge once it's approved. Thanks.
@pvillard31

@exceptionfactory Thanks for reviewing! I've addressed your feedback and committed the changes.

Thanks for making the changes @mtien-apache.  I will take another look following the updates. I noticed that StandardProcessGroup, NiFiServiceFacade, and StandardNiFiServiceFacade still show up with reordered imports, can you remove those changes?

Note sure if this has been addressed but I noticed that: if selecting a flow on the canvas ans clicking on the "Group" icon (to make a process group from the selection), then the icon to upload a JSON file appears when it should just ask for the new process group to create from the selection.

I don't think this is expected. And if I do select a JSON file at this step (instead of just giving a name for my selection to be added in a process group), the process group for my JSON file will be added instead of putting my selection into a process group.
So... I guess it's just about removing the option to upload the file when we click on the "Group" button. It should only be available when drag-and-dropping a process group on the canvas. Thoughts?

@pvillard31 Agreed the upload file button should be removed when a user clicks on the "Group" button. Will make the change.

Just pushed a commit -

Hide the Upload File button when grouping components @pvillard31
Refactored uploadProcessGroup and exception handling @exceptionfactory
Refactored nf-ng-group-component.js @sardell

Thanks

@sardell Thanks for reviewing!

@markap14 I made revisions from your feedback. Thanks for looking at this in detail!

Merged (and took care of reverting import ordering changes)

authors: exceptionfactory joewitt mtien-apache pvillard31


text:
NIFI-14005 - Support Custom Content from External File in GenerateFlowFile

Summary
NIFI-14005 - Support Custom Content from External File in GenerateFlowFile
Add a property in GenerateFlowFile allowing the user to specify a local path to an external file. If specified the content of the external file will be used as content of the generated flow files.
This makes things easier compared to a GetFile for example. Besides, coupled with the Asset feature, it makes it easy to load a test dataset as content for GenerateFlowFile processor in a cloud native environment (container based).
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[x] Apache NiFi Jira issue created

Pull Request Tracking

[x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[x] Pull Request based on current revision of the main branch
[x] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[x] Build completed using mvn clean install -P contrib-check

[x] JDK 21

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

That is an extremely valid point that I completely missed. Given that some users definitely want to restrict processors with access to local file system, it feels like we shouldn't change expectations around GenerateFlowFile and a dedicated processor for this specific usage would be better. What do you think?

That is an extremely valid point that I completely missed. Given that some users definitely want to restrict processors with access to local file system, it feels like we shouldn't change expectations around GenerateFlowFile and a dedicated processor for this specific usage would be better. What do you think?

Yes, under the circumstances, it seems like a separate Processor would be better. I think it would be good to scope out what it looks like on the Jira issue and then move to consider the implementation.

a dedicated processor for this specific usage would be better. What do you think?

Thinking about it, a separate processor might make sense insofar that the processor is not actually "generating" content but rather injecting pre-defined content when using the contents of a file (although the same could be said about the existing "Custom Text")


text:
stm32l562 secure fault occurs when running NUTTX

Nuttx operating system with stm32l562 board, compile the bin file of the board, and then according to the board directory, readme.txt in the operation through the TFM load the bin run, will give the following secure fault problem:
[Sec Thread] Secure image initializing! ABCDF [Sec Handler] Oops... Secure fault!!! You're not going anywhere! [INF] Starting bootloader [INF] Checking BL2 NV area [INF] Checking BL2 NV area header [INF] Checking BL2 NV Counter consistency [INF] Consistent BL2 NV Counter 3 = 0x1000000 [INF] Consistent BL2 NV Counter 4 = 0x1000000 [INF] Swap type: none [INF] Swap type: none [INF] verify counter 0 1000000 1000000 [INF] counter 0 : ok [INF] verify sig key id 0 [INF] signature OK [INF] verify counter 1 1000000 1000000 [INF] counter 1 : ok [INF] verify sig key id 1 [INF] signature OK [INF] Bootloader chainload address offset: 0x17000 [INF] Jumping to the first image slot set to BL2 SHARED DATA2XX_HUKCUSTOMIZATION [INF] Code c002900 c011bfe [INF] hash TFM_SBSFU_Boot 12f2c685 .. 57deacd1 [INF] otfdec key b8, c0, 35, 4f, 78, 9e, e4, 62, [INF] otfdec key 37, 7a, f0, 8, 51, 96, 70, 2b
I have completely followed the steps of readme.txt, but there is still this problem, I would like to ask anyone who has encountered this problem, or how to find the author who created the board code, thank you.

@michi-jung could you @zhutouwangzha ?

@michi-jung could you @zhutouwangzha ?

Hello!When i only open Binary Loader Debug and informational output in the nuttx debug, a new error message appears:
ABCDF
stm32l5_nmi: PANIC!!! NMI received
_assert: error file:chip/stm32l5_irq.c line:128
_assert: Current Version: NuttX  0.0.0 0 May 12 2023 13:15:41 arm
_assert: Assertion failed panic: at file: chip/stm32l5_irq.c:128 task: 0x90002315
up_dump_register: R0: 9000164d R1: 00000004 R2: 20001888  R3: e000ed04
up_dump_register: R4: 80000000 R5: 90021313 R6: 900005f4  FP: 900005f4
up_dump_register: R8: 900005f4 SB: 900005f4 SL: 200018a4 R11: 20001888
up_dump_register: IP: 900005f4 SP: 20001950 LR: 9000165f  PC: 900018b8
up_dump_register: xPSR: 81000200 PRIMASK: 00000000 CONTROL: 00000000
up_dump_register: EXC_RETURN: ffffffa8
dump_stack: IRQ Stack:
dump_stack: sp:     0x20000a08
dump_stack:   base: 0x20000368
dump_stack:   size: 00002048
stack_dump: 0x20000a00: 00000800 00000001 00000000 20000ba0 900018b8 20000a08 00010000 9000a123
stack_dump: 0x20000a20: 90001fb5 900005f4 900005f4 9001fe91 900278a4 00000080 90002315 20000acd
stack_dump: 0x20000a40: 00000000 9002789c 00000080 900278a4 7474754e 90000058 9002124b 900221f7
stack_dump: 0x20000a60: 00000000 200000a0 20000a80 00000000 9000c93b 9000c8ff 9002124b 9002fe00
stack_dump: 0x20000a80: 20000a98 302e3000 0000302e 20000adc 90027880 00000003 20307880 2079614d
stack_dump: 0x20000aa0: 32203231 20333230 313a3331 31343a35 00000000 00000000 00000000 9000c93b
stack_dump: 0x20000ac0: 90001fb5 9000c8ff 00000000 6d726108 90027800 00000003 00000000 20000b08
stack_dump: 0x20000ae0: 90001911 20000ba0 00000000 01000003 9000209b 9000a123 00000000 9002789c
stack_dump: 0x20000b00: 00000080 900278a4 90027948 9000191b 00000000 00000000 2000187c 00000002
stack_dump: 0x20000b20: 00000003 00000000 90002247 9000293b 2000187c 00000002 90002077 00000002
stack_dump: 0x20000b40: 00000000 900018e9 80000000 9000209b 00000000 00000002 00000000 80000000
dump_stack: User Stack:
dump_stack: sp:     0x20001950
dump_stack:   base: 0x2000188c
dump_stack:   size: 00001000
stack_dump: 0x20001940: deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef 00000062 deadbeef
stack_dump: 0x20001960: deadbeef deadbeef 90025ff9 90026109 90021313 00000062 2000015c deadbeef
stack_dump: 0x20001980: 90022a25 90026109 90025ff9 00000012 9002f788 00000000 00000000 deadbeef
stack_dump: 0x200019a0: 90022a25 9000b989 90026109 00000012 9002f788 deadbeef 9000bc45 9000b989
stack_dump: 0x200019c0: 90022a25 deadbeef 00000012 9002f788 20001bac deadbeef 20001bac deadbeef
stack_dump: 0x200019e0: 9000bc45 9000bf93 9000b989 9002f732 00000000 00000000 20001bac deadbeef
stack_dump: 0x20001a00: deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef
stack_dump: 0x20001a20: deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef
stack_dump: 0x20001a40: deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef 00000012 deadbeef
stack_dump: 0x20001a60: 00000012 9002f788 00000000 00000000 73ad0000 deadbeef 9000bc45 9000bf93
stack_dump: 0x20001a80: 9000bf93 20001c18 deadbeef deadbeef 20001c14 9002f730 20001bac deadbeef
stack_dump: 0x20001aa0: 00000000 9000bf93 9000bc45 9002fe03 00000000 00000000 20001bac deadbeef
stack_dump: 0x20001ac0: 20001c14 deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef
stack_dump: 0x20001ae0: deadbeef 20001ba4 deadbeef 20001ba4 00000000 deadbeef deadbeef deadbeef
stack_dump: 0x20001b00: deadbeef deadbeef 00adbeef deadbeef 90017cd3 900043cb 00000000 deadbeef
stack_dump: 0x20001b20: 00001c78 00000000 00000000 00000000 56000200 deadbeef 900221f7 9002124b
stack_dump: 0x20001b40: 9000bf93 20001b90 00001c78 00000000 20001b8c 9002fe00 20001bac 00000001
stack_dump: 0x20001b60: 9000c93b 9002124b 900221f7 00000001 20001bac 20001b8c 90002a99 9000c93b
stack_dump: 0x20001b80: 9000c8ff 9002124b 9002fe00 20001ba4 20001bac 9000de71 20001be8 9002f730
stack_dump: 0x20001ba0: 00000006 9002f730 20001be8 00000000 9002294d 900229cd 9000c67d 00000065
stack_dump: 0x20001bc0: 9000d339 200025f8 9000c93b 9001e7c1 9000c8ff 9000cf47 20001c14 9002f730
stack_dump: 0x20001be0: 00000006 20002778 20001c14 9001e8c7 9000c93b 00000188 00000006 0002da08
stack_dump: 0x20001c00: 20001c14 900005f4 9000263b 9001e8c7 9002f730 9002f788 9001e7c1 0008020c
stack_dump: 0x20001c20: 90002637 900005f4 9000263b 900005f4 9001e7c1 00000008 20002778 900005f4
stack_dump: 0x20001c40: 9000263b 0002e38c 20001c74 20000ba0 20000b6c 00000000 00000001 900005f4
show_tasks:    PID GROUP PRI POLICY   TYPE    NPX STATE   EVENT      SIGMASK  STACKBASE  STACKSIZE      USED   FILLED    COMMAND
show_tasks:   ----   --- --- -------- ------- --- ------- ---------- -------- 0x20000368      2048      2048   100.0%!   irq
dump_task:       0     0   0 FIFO     Kthread N-- Running            0000000000000000 0x2000188c      1000       992    99.2%!

@zhutouwangzha can you use GDB with SWD (i.e with OpenOCD, JLink GDB, etc) and do some step by step debugging to reveal the root cause?
Just to make things clear, could you please show the steps you are doing?
Example:
First I'm configuring the board and compiling NuttX:
$ ./tools/configure.sh stm32l562e-dk:nsh
$ make -j
Then I'm flashing it using the command ....
It will help others with same board to follow your steps.

According to the Readme file, I followed these steps:
$./tools/configure.sh stm32l562e-dk:nsh
$make -j
$python3 ~/compile/STM32CubeL5/Middlewares/Third_Party/mcuboot/scripts/imgtool.py sign -k ~/compile/STM32CubeL5/Middlewares/Third_Party/trustedfirmware/bl2/ext/mcuboot/root-rsa-2048_1.pem -E ~/compile/STM32CubeL5/Middlewares/Third_Party/trustedfirmware/bl2/ext/mcuboot/enc-rsa2048-pub.pem -o 0x90000400 -e little -S 0x100000 -H 0x400 --pad-header -v 1 -s auto --align 8 nuttx.bin tfm_ns_enc_sign.bin
Then follow the prompts to change the sample code in 1.3.1TF-M, and then go to the Projects/STM32L562E-DK/Applications/TFM/TFM_SBSFU_Boot/STM32CubeIDE directory and modify line 30 in TFM_UPDATE.sh,$stm32programmercli $connect -d/my_home/nuttxspace/nuttx/tfm_ns_enc_sign.bin $slot1 -v.
After the modification, execute regression.sh and TFM_UPDATE.sh.
But I can't debug with GDB at the moment.

@zhutouwangzha Can you please have a look at https://github.com/secore-ly/nuttx-upstreaming/tree/issue-9316 and let me know whether this works for you?
Note that you need to update STM32CubeL5 to version 1.5.0 and that a third patch is required as described in the updated board's README.

@zhutouwangzha Can you please have a look at https://github.com/secore-ly/nuttx-upstreaming/tree/issue-9316 and let me know whether this works for you?
Note that you need to update STM32CubeL5 to version 1.5.0 and that a third patch is required as described in the updated board's README.

Thanks for your help, I pulled your updated NUTTX code and updated TF-M to version 1.5.0 according to the new README.txt, and it finally works. But I don't know if I'm using GDB to debug the steps correctly, my operation is as follows:
$openocd -f interface/stlink-v2.cfg -f target/stm32l5x.cfg
$gdb-multiarch nuttx/nuttx
$(gdb)target extended-remote :3333
$(gdb)break nsh_main
$(gdb)continue
When the last step is executed, gdb prompts: The program is not being run.

Thanks for your help, I pulled your updated NUTTX code and updated TF-M to version 1.5.0 according to the new README.txt, and it finally works.

Do you mean that your board boots successfully to the NSH prompt now?  I.e. is the fix good to go into a pull request?

But I don't know if I'm using GDB to debug the steps correctly

Did you set OB_RDP_LEVEL_0 in Projects/STM32L562E-DK/Applications/TFM/TFM_SBSFU_Boot/Inc/boot_hal_cfg.h?
You have to apply the regression.sh script, before you flash the new firmware to the device (This will reset all Option Byte settings, disable TrustZone and erase flash).
I am not running with openocd.  I am using the ST-LINK_gdbserver from STM32CubeCLT like this:
$ ST-LINK_gdbserver -d -cp /opt/secore.ly/compute-secore.ly-sdk/bin

STMicroelectronics ST-LINK GDB server. Version 6.1.0
Copyright (c) 2022, STMicroelectronics. All rights reserved.

Starting server with the following options:
        Persistent Mode            : Disabled
        Logging Level              : 31
        Listen Port Number         : 61234
        Status Refresh Delay       : 15s
        Verbose Mode               : Disabled
        SWD Debug                  : Enabled

COM frequency = 24000 kHz
Target connection mode: Default
Reading ROM table for AP 0 @0xe00fefd0
Hardware watchpoint supported by the target
ST-LINK Firmware version : V3J10M3B5S1
Device ID: 0x481
PC: 0xc0154c0
ST-LINK device status: HALT_MODE
ST-LINK detects target voltage = 3.30 V
TrustZone: Active
ST-LINK device status: HALT_MODE
ST-LINK device initialization OK
Stm32Device, pollAndNotify running...
SwvSrv state change: 0 -> 1
Waiting for connection on port 61235...
Waiting for debugger connection...
Waiting for connection on port 61234...

$ arm-nuttx-eabi-gdb
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=arm-nuttx-eabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) target remote :61234
Remote debugging using :61234
warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
0x0c0154c0 in ?? ()
(gdb) symbol-file nuttx_user.elf
Reading symbols from nuttx_user.elf...
(gdb) break nsh_main
Breakpoint 1 at 0x818073c: file nsh_main.c, line 57.
(gdb) monitor reset
STM32 Successfully completed reset operation (System reset)
(gdb) continue
Continuing.

Breakpoint 1, nsh_main (argc=1, argv=0x200d0c18) at nsh_main.c:57
57        sched_getparam(0, &param);
(gdb)

Yes, the NSH prompt now boots successfully. Thanks for your help, I'll look into GDB debugging myself later.

source: gharchive/issue

authors: acassis michi-jung xiaoxiang781216 zhutouwangzha

repo: apache/nuttx


text:
nuttx/libm: Fix powl wrong if first parameter is negative

Summary
This PR applies the pattern of calculation used for pow and powf to powl
Impact
Has no impact on current implementation
Testing
Pass CI

[Experimental Bot, please feedback here]
This PR appears to not meet the NuttX requirements fully.
Here's why:

Insufficient Detail: The summary is too brief.  While it mentions the change applies a pattern from pow and powf to powl, it lacks critical context:

Why is this change necessary? Is there a bug in powl? Is it for performance optimization? Code clarity?
How does the pattern from pow and powf differ from the current powl implementation?
What functional part of the code is being changed? Be specific - mention the file(s) or modules involved.

Testing Lacks Specificity: "Pass CI" is not enough information.

Which CI environments are relevant? Specify the target architectures and configurations tested.
"Testing logs before/after" are missing. Provide concrete examples of how the output or behavior differs due to this change.
Consider edge cases: Were any specific inputs or scenarios tested where powl's behavior might be different?

To improve this PR:

Expand the Summary: Provide the missing context about the why and how of the change.
Detailed Testing:

List the specific CI environments (targets) where the tests passed.
Include relevant snippets of logs or output demonstrating the change's effect (or lack thereof, if that's the intended outcome).
If possible, highlight test cases that specifically exercise the powl function with various inputs.

authors: nuttxpr pkarashchenko


text:
ORC-373: Option to disable dictionary encoding

disables dictionary encoding at the time of string tree writer initialization.

@omalley  could you please take a look?

@prasanthj no unit test?

@damiencarol added junit tests. my branches got messed up. So had to give another PR. #279

@prasanthj yeah, in general you should make a dev branch for the pull request (eg. prasanthj:orc-373).  Then you can do a force-push to the branch to update the PR.

authors: damiencarol omalley prasanthj

repo: apache/orc

license_source: bigquery


text:
ORC-975: Avoid double counting closestFixedBits

What changes were proposed in this pull request?

310  int idx = encodeBitWidth(findClosestNumBits(data[i]));

public int encodeBitWidth(int n) {
    n = getClosestFixedBits(n);
    .....
}

public int findClosestNumBits(long value) {
  final int numBits = 64 - Long.numberOfLeadingZeros(value);
  return getClosestFixedBits(numBits);
}

getClosestFixedBits is called twice.
Extracts the new method findNumBits from findClosestNumBits. Avoid double counting closestFixedBits.
Why are the changes needed?

Avoid redundant calls.
How was this patch tested?

Pass the CIs.

This is not backported to branch-1.7 because there is a conflict.

authors: dongjoon-hyun guiyanakuang


text:
HDDS-6200. Provide OM#getSupportedBucketLayout RPC call for BucketLayout upgrade finalization.

What changes were proposed in this pull request?
The idea here is to add OM#getSupportedLayouts() RPC call to get the bucket layout values.
Before finalization OM#getSupportedLayouts() RPC call should return empty values. Now, the new client's OzoneClientAdapter and CreateBucketHandler would get an empty [] list then continue with the existing older version behavior.
After finalization OM#getSupportedLayouts() RPC call will have values FSO/OBS and the client will have the new behavior.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6200
How was this patch tested?
Added related Integration Test.

@rakeshadr @errose28 Could you review these changes? This PR is a part of the upgrade effort for the BucketLayout feature.
One of the downsides of my current implementation seems to be as follows:
If we are performing an upgrade on an Ozone installation that already supports OBS and FSO layouts, to a more recent version - the OM would still return an empty list of supported layouts as the response to the OzoneManager.getSupportedBucketLayouts(), and not allow its clients to create a new bucket with OBS/FSO layout (which were already supported before the upgrade process began).

@errose28 thanks for reviewing this PR.

Let me know what the current requirements for FSO client interactions are and we can decide whether this is the best way to handle them.

Following is the requirements of the Ozone:
OM Pre-finalized state
Client CreateBucket behavior in different client end points:
[A] Ozone Shell Command #createBucket:-
       case-1) New-Shell-Client uses default value OBS:
             "ozone sh  create bucket myBucket"
              Now, ozone client will send the default value OBS to OM server
      case-2) New-Shell-Client sends explicit value:
              "ozone sh create bucket --layout=FSO myFSOBucket"

[B] BasicRootedFileSystem #createBucket:-
      case-3) if the bucket doesn’t exists then ozone client will create a bucket with "FSO" layout.

IIUC the simplest option is to handle it by ignoring the bucket layout argument at the OM side and silently creates LEGACY bucket layout with a WARN message.
On a second thought, I was trying an approach to throw exception if user explicitly specify FSO or OBS layout value in the above mentioned "case-2". But I like your idea of unifying the pre-finalized code for upcoming features. That makes code more maintainable. Since this is a new feature, we have the flexibility to decide the behavior of different FSO cases.
I agree to pause this PR now and would like to see HDDS-6213 proposal patch.

⬆  cc @kerneltime who is working on the client versioning.

authors: JyotinderSingh errose28 rakeshadr

repo: apache/ozone


text:
HDDS-8407. Intermittent failure in TestOzoneFileSystem#testTrash

What changes were proposed in this pull request?
Allow multiple trash checkpoints to account for items deleted for other test cases.
https://issues.apache.org/jira/browse/HDDS-8407
How was this patch tested?
No failures of testTrash in 100 runs (only testListStatusWithIntermediateDir failed once, being fixed in HDDS-8370):
https://github.com/adoroszlai/hadoop-ozone/actions/runs/4691530928
Regular CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/4692326817

100/100 passed: https://github.com/adoroszlai/hadoop-ozone/actions/runs/4712265767

Thanks @ayushtkn for the review.


text:
HDDS-8444. Increase timeout of CI build

What changes were proposed in this pull request?
build check in CI is frequently timing out in forks due to Maven cache miss (but not in apache/ozone).  Dependent checks (e.g. acceptance, kubernetes) can only be run after re-try.  To avoid wasting time, I propose increasing time limit for build check from 30 to 45 minutes.
https://issues.apache.org/jira/browse/HDDS-8444
How was this patch tested?
CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/4734224440/jobs/8402851285
build check succeeded in 36m 6s, i.e. would have failed without this change.

This has been killing me @adoroszlai !  Thanks for fixing it!

Thanks @ayushtkn, @GeorgeJahad for the review.

authors: GeorgeJahad adoroszlai


text:
HDDS-8722. Improve thread names in Datanode.

What changes were proposed in this pull request?
Add the ability to specify a prefix for threads created for the datanode.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8722
How was this patch tested?

@Galsza will be continuing this work and post a new PR


text:
HDDS-10612. Add Robot test to verify Container Balancer for RATIS containers

What changes were proposed in this pull request?
HDDS-10612. Add Robot test to verify Container Balancer for RATIS containers
Currently there are only unit tests for Container Balancer and no acceptance tests at all. At a minimum, we should add a Robot test to verify Container Balancer for RATIS containers. And probably in the future we should add robot test for EC case.
Test case:

Move 1 datanode to maintenance mode (we use 4 datanodes in this test)
Create multiple keys  (after loading the data, we check that 3 datanodes are ~60% busy, and the one that is in maintenance mode is empty)
Start datanode recommission (wait until datanode recommissioning is completed)
Start container balancer (wait until container balancer is completed)
Check results (after balancing on all 4 datanodes, we should see approximately the same data distribution.)

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10612
How was this patch tested?
Added Robot test

Seems there is a intermittent failure on the acceptance test
https://github.com/apache/ozone/actions/runs/8546730074/job/23418032793
@afilpp Could you take a look?


text:
[core] Fix numeric literals cast in filter pushdown after schema evolution

Purpose

Reproduce case:
CREATE TABLE T (
  id INT,
  f DECIMAL(5,2),
);
INSERT INTO T VALUES (1, 111.32);
ALTER TABLE T MODIFY (f DECIMAL(6, 3));
SELECT * FROM T WHERE f < 111.321;

The result is empty but expected 111.320
There are also same problem for other numeric field cast Scenario.

Tests

API and Format

Documentation

+1

The predicate conversion of decimal/floating type seems to be too complicated and there may be bad cases. Is it possible to set up a whitelist to only allow integer conversion?

The predicate conversion of decimal/floating type seems to be too complicated and there may be bad cases. Is it possible to set up a whitelist to only allow integer conversion?

I think this is a good idea. Actually, decimal type and related schema evolution is rarely used in industry.

+1

authors: BitFunnelComp JingsongLi wwj6591812 yuzelin

repo: apache/paimon


text:
[Multi-stage] Stage stats are very misleading

Currently the stage stats reported in the query response is the aggregated stats over all the operators, which can be quite confusing.
When tracing is disabled, we collect stats for leaf and send operator, and reporting doubled numDocs and numBlocks than actually sent by the leaf stage.
When tracing is enabled, we collect stats for every operator, and aggregate numDocs and numBlocks for all the operators.
Instead, we might want to just record the stats for blocks sent out from each stage. We should also revisit other stats, and see if it makes sense to aggregate them.

cc @gortiz @KKcorps

That is weird. What's causing the doubling of leaf data? iirc, the stats are only sent when EOS block is received

By default, we collect stats for leaf operator and send operator. The docs gathered for each stage is the aggregated value. IMO we should only count the rows sent by the send operator.

Ohhh, iirc, I did that to ensure that all the existing stats for V1 engine are also available for each table.  I must missed the tracing duplication part. Assigning this to myself

authors: Jackie-Jiang KKcorps

repo: apache/pinot


text:
Updating schema with new MV column causes index failures and stops consumption

One of the users added a new MV column to the schema and reloaded the segments. Fyi, the data in the topic doesn't have any values for this new field. It failed with the following exception:
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
        at org.apache.pinot.segment.local.segment.index.readers.constant.ConstantMVForwardIndexReader.getDictIdMV(ConstantMVForwardIndexReader.java:48) ~[startree-pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.
11.0-SNAPSHOT-b92cbb4df88eee0bf0fbb39dfc434387370edc8e]
        at org.apache.pinot.segment.local.segment.readers.PinotSegmentColumnReader.getValue(PinotSegmentColumnReader.java:89) ~[startree-pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-b92cbb4df88e
ee0bf0fbb39dfc434387370edc8e]
        at org.apache.pinot.segment.local.segment.readers.PinotSegmentRecordReader.getRecord(PinotSegmentRecordReader.java:226) ~[startree-pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-b92cbb4df8
8eee0bf0fbb39dfc434387370edc8e]
        at org.apache.pinot.segment.local.segment.readers.PinotSegmentRecordReader.next(PinotSegmentRecordReader.java:215) ~[startree-pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-b92cbb4df88eee0
bf0fbb39dfc434387370edc8e]
        at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:218) ~[startree-pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSH
OT-b92cbb4df88eee0bf0fbb39dfc434387370edc8e]
        at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:123) ~[startree-pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-b92cbb4df88
eee0bf0fbb39dfc434387370edc8e]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:839) [startree-pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-b92cbb4df88eee0bf0fbb39dfc434387370edc8e]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:766) [startree-pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-b92cbb4df88eee0bf0fbb39dfc434387370edc8e]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:665) [startree-pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-b92cbb4df88eee0bf0fbb39dfc434387370edc8e]
        at java.lang.Thread.run(Thread.java:829) [?:?]

Steps to reproduce

Run the RealtimeQuickstart
Load some data
Edit schema to add a new MV column
Reload the segments

Things to note

The quickstart is using High level consumer, where as , the customer was using low level consumer.
When reproducing, the same exception happens. However, this time the stacktrace is from the query path.

2022/06/02 11:31:20.041 ERROR [BaseCombineOperator] [pqw-6] Caught exception while processing query: QueryContext{_tableName='meetupRsvp_REALTIME', _subquery=null, _selectExpressions=[event_id, event_name, event_tags, event_time, group_city, group_country, group_id, group_lat, group_lon, group_name, location, mtime, rsvp_count, venue_name], _aliasList=[null, null, null, null, null, null, null, null, null, null, null, null, null, null], _filter=null, _groupByExpressions=null, _havingFilter=null, _orderByExpressions=null, _limit=10, _offset=0, _queryOptions={responseFormat=sql, groupByMode=sql, timeoutMs=10000}, _expressionOverrideHints={}, _explain=false}
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
	at org.apache.pinot.segment.local.segment.index.readers.constant.ConstantMVForwardIndexReader.getDictIdMV(ConstantMVForwardIndexReader.java:48) ~[classes/:?]
	at org.apache.pinot.core.common.DataFetcher$ColumnValueReader.readStringValuesMV(DataFetcher.java:701) ~[classes/:?]
	at org.apache.pinot.core.common.DataFetcher.fetchStringValues(DataFetcher.java:391) ~[classes/:?]
	at org.apache.pinot.core.common.DataBlockCache.getStringValuesForMVColumn(DataBlockCache.java:462) ~[classes/:?]
	at org.apache.pinot.core.operator.docvalsets.ProjectionBlockValSet.getStringValuesMV(ProjectionBlockValSet.java:179) ~[classes/:?]
	at org.apache.pinot.core.common.RowBasedBlockValueFetcher.createFetcher(RowBasedBlockValueFetcher.java:84) ~[classes/:?]
	at org.apache.pinot.core.common.RowBasedBlockValueFetcher.<init>(RowBasedBlockValueFetcher.java:33) ~[classes/:?]
	at org.apache.pinot.core.operator.query.SelectionOnlyOperator.getNextBlock(SelectionOnlyOperator.java:97) ~[classes/:?]
	at org.apache.pinot.core.operator.query.SelectionOnlyOperator.getNextBlock(SelectionOnlyOperator.java:40) ~[classes/:?]
	at org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:39) ~[classes/:?]
	at org.apache.pinot.core.operator.combine.BaseCombineOperator.processSegments(BaseCombineOperator.java:158) ~[classes/:?]
	at org.apache.pinot.core.operator.combine.BaseCombineOperator$1.runJob(BaseCombineOperator.java:101) [classes/:?]
	at org.apache.pinot.core.util.trace.TraceRunnable.run(TraceRunnable.java:40) [classes/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111) [guava-20.0.jar:?]
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58) [guava-20.0.jar:?]
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75) [guava-20.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]

Another exception seen, tangential to this issue is a NumberFormatException in SegmentsValidationAndRetentionConfig and SegmentStatusChecker

2022/06/02 11:32:53.528 ERROR [SegmentStatusChecker] [pool-17-thread-4] Caught exception while updating segment status for table meetupRsvp_REALTIME
java.lang.NumberFormatException: null
	at java.lang.Integer.parseInt(Integer.java:614) ~[?:?]
	at java.lang.Integer.parseInt(Integer.java:770) ~[?:?]
	at org.apache.pinot.spi.config.table.SegmentsValidationAndRetentionConfig.getReplicasPerPartitionNumber(SegmentsValidationAndRetentionConfig.java:176) ~[classes/:?]
	at org.apache.pinot.controller.helix.SegmentStatusChecker.updateTableConfigMetrics(SegmentStatusChecker.java:143) ~[classes/:?]
	at org.apache.pinot.controller.helix.SegmentStatusChecker.processTable(SegmentStatusChecker.java:113) ~[classes/:?]
	at org.apache.pinot.controller.helix.SegmentStatusChecker.processTable(SegmentStatusChecker.java:56) ~[classes/:?]
	at org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.processTables(ControllerPeriodicTask.java:116) ~[classes/:?]
	at org.apache.pinot.controller.helix.core.periodictask.ControllerPeriodicTask.runTask(ControllerPeriodicTask.java:85) ~[classes/:?]
	at org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:150) ~[classes/:?]
	at org.apache.pinot.core.periodictask.BasePeriodicTask.run(BasePeriodicTask.java:135) ~[classes/:?]
	at org.apache.pinot.core.periodictask.PeriodicTaskScheduler.lambda$start$0(PeriodicTaskScheduler.java:87) ~[classes/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]

Looks like I can't assign it to myself. But I will pick this up!

This issue can be reproduced by adding the following test query in BaseClusterIntegrationTestSet#testReload :
SELECT SUMMV(NewIntMVDimension) FROM " + rawTableName
The problem wasn't caught in the existing integration tests because the simple select * type of queries is by default limited to 10 records. If we increase limit in the query, it is very expensive and likely to run out of heap space. So, using an aggregated function on a column ensures that the query fetches from the consuming segment as well.
The test has been updated in the PR #9205


text:
Fix githubEvents table for quickstart recipes

The PR addresses the bug raised in the issue #10675
The changes include:

Rename the examples/stream/githubEvents directory to examples/stream/pullRequestMergedEvents as the files it contains creates a table named pullRequestMergedEvents
Copy the examples/minions/stream/githubEvents table to examples/stream/githubEvents so that we avoid the RealtimeToOfflineSegmentsTask which is present in the minions sub path table config and leads to the exception logs mentioned in the linked issue
Remove the retention rule for githubEvents table as the table contains data from 2021.

Codecov Report

Merging #10716 (008b3dc) into master (cad764d) will decrease coverage by 38.24%.
The diff coverage is n/a.

:exclamation: Current head 008b3dc differs from pull request most recent head db86638. Consider uploading reports for the commit db86638 to get more accurate results

@@              Coverage Diff              @@
##             master   #10716       +/-   ##
=============================================
- Coverage     70.32%   32.08%   -38.24%
+ Complexity     6429      453     -5976
=============================================
  Files          2112     2112
  Lines        114056   114056
  Branches      17226    17226
=============================================
- Hits          80213    36600    -43613
- Misses        28244    74150    +45906
+ Partials       5599     3306     -2293

Flag
Coverage Δ

integration1
24.42% <ø> (+0.07%)
:arrow_up:

integration2
?

unittests1
?

unittests2
13.85% <ø> (+0.02%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1332 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


text:
In Producer batching is not working with synchronous mode using send api

Search before asking

[X] I searched in the issues and found nothing similar.

Read release policy

[X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

Version
3.2.x
Minimal reproduce step
ProducerBuilder producerBuilderBatching = client.newProducer(Schema.STRING)
								.accessMode("Shared")
								.compressionType("NONE")
								.enableBatching(true)
								.blockIfQueueFull(true)
								.batchingMaxMessages(5)
								.batchingMaxBytes(13270)
								.batchingMaxPublishDelay(70000, TimeUnit.MILLISECONDS);

ProducerBase<String> producerStringBatching = (ProducerBase<String>) producerBuilderBatching.topic("batch-test-topic")
								.sendTimeout(20, TimeUnit.SECONDS).create();

MessageId msgId = producerStringBatching.newMessage().value("Check batching").send();

What did you expect to see?
number of messages should increase from 0 to 4 in a single batch(as max number of messages configured in a single batch was 5) and below are the pulsar SDK logs which should print in console

2024-04-02T14:54:08,963 DEBUG [pool-22-thread-2] org.apache.pulsar.client.impl.BatchMessageContainerImpl - [batch-test-topic] [standalone_onessl-4-800] add message to batch, num messages in batch so far 0
2024-04-02T14:54:08,964 DEBUG [pool-22-thread-15] org.apache.pulsar.client.impl.BatchMessageContainerImpl - [batch-test-topic] [standalone_onessl-4-800] add message to batch, num messages in batch so far 1
2024-04-02T14:54:08,964 DEBUG [pool-22-thread-9] org.apache.pulsar.client.impl.BatchMessageContainerImpl - [batch-test-topic] [standalone_onessl-4-800] add message to batch, num messages in batch so far 2
2024-04-02T14:54:08,965 DEBUG [pool-22-thread-6] org.apache.pulsar.client.impl.BatchMessageContainerImpl - [batch-test-topic] [standalone_onessl-4-800] add message to batch, num messages in batch so far 3
2024-04-02T14:54:08,965 DEBUG [pool-22-thread-20] org.apache.pulsar.client.impl.BatchMessageContainerImpl - [batch-test-topic] [standalone_onessl-4-800] add message to batch, num messages in batch so far 4

just like working in async mode
What did you see instead?
number of messages in a single batch is always 0 while using the synchronous mode
2024-04-02T14:43:26,850 DEBUG [pool-22-thread-15] org.apache.pulsar.client.impl.BatchMessageContainerImpl - [batch-test-topic] [standalone_onessl-4-792] add message to batch, num messages in batch so far 0
2024-04-02T14:43:26,852 DEBUG [pool-22-thread-8] org.apache.pulsar.client.impl.BatchMessageContainerImpl - [batch-test-topic] [standalone_onessl-4-792] add message to batch, num messages in batch so far 0
2024-04-02T14:43:26,854 DEBUG [pool-22-thread-7] org.apache.pulsar.client.impl.BatchMessageContainerImpl - [batch-test-topic] [standalone_onessl-4-792] add message to batch, num messages in batch so far 0

Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

For the purpose of reduce the sync-api send message latency, client will flush messages to broker when you call the sync-api.
And sync-api will block your app's threads, if we don't flush messages to broker but waiting until the batch is full, it may block all the threads of your app.
/cc @gaoran10 @codelipenghui

Thanks for the good issue report @ragaur-tibco !

For the purpose of reduce the sync-api send message latency, client will flush messages to broker when you call the sync-api.

@dao-jun  Yes, this is reasonable, but the problem is that this doesn't seem to be documented any where.
I found the flushing logic here:
https://github.com/apache/pulsar/blob/ffff639a1b73a34bbb5115503d4c7783bb2a2770/pulsar-client/src/main/java/org/apache/pulsar/client/impl/TypedMessageBuilderImpl.java#L82-L86
I think @ragaur-tibco's issue report is great and the minimum resolution is to document the behavior.
It seems that there could be a use case where one would want to achieve batching while using the synchronous API. That's not currently supported. We have 2 options for resolving this:

Documenting the behavior and making it explicit that synchronous send will trigger flushing of the current batch message.
Adding support for controlling the triggering behavior so that batching would be possible also with the synchronous API.

@lhotari
I think support batch message for sync-api has potential risks, as I mentioned before, if we don't flush messages but waiting until the batch is full, it may block users' app threads for a long while.
Especially there are may producers send message in batch in a single app, it may block all the threads.
I prefer option 1, and I'll fix the doc later.

authors: dao-jun lhotari ragaur-tibco

repo: apache/pulsar


text:
[ManagedLedger] Compress managed ledger info

Motivation
Currently, the ManagedLedgerInfo contains offload context info, if there are too many ledgers in one ManagedLedger, a Zookeeper ZNode data will increase, it's hard for Zookeeper to manage these data. We could compress the ManagedLedgerInfo data, this will decrease the data size.
For example, if one ManagedLedgerInfo contains 30000 ledgers and each LedgerInfo contains offload context, the uncompressed data size is about 6MB, after compress with ZSTD, the compression data size could be decreased to about 1.3MB.
Modifications
Add a ManagedLedgerInfoMetadata before ManagedLedgerInfo, the data structure as below.
[MAGIC_NUMBER] (2) + [METADATA_SIZE] (4) + [METADATA_PAYLOAD] + [MANAGED_LEDGER_INFO_PAYLOAD]

Add a new configuration managedLedgerInfoCompressionType to control ManagedLedgerInfo compression type, if not set this configuration, then don't compress ManagedLedgerInfo data.
Migration
When reading data from the Zookeeper, check the magic number of the data, if the head data match the magic number then try to parse metadata and uncompress data, if encounter errors or not match fall back to parse ManagedLedgerInfo directly.
Verifying this change
Verify compress and uncompress ManagedLedgerInfo data.
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation
For committer
For this PR, do we need to update docs?

If yes,

if you update docs in this PR, label this PR with the doc label.

if you plan to update docs later, label this PR with the doc-required label.

if you need help on updating docs, create a follow-up issue with the doc-required label.

If no, label this PR with the no-need-doc label and explain why.

@codelipenghui this is a new feature, with a significant impact, as you cannot rollback if you enable this feature.
I believe that we cannot cherry pick this feature to 2.8.x release line.
We should cherry pick to released versions only bug fixes and small features that have a low impact.
does this make sense to you ?

@eolivelli: the upgrade requires two steps:

upgrade the binary without enabling this feature.
after all brokers are upgraded, enable this feature.

Because it is a 2-steps upgrade story, we can add the code to 2.8.1 because this feature would be turned off. With the code available in 2.8.1, it provides a smooth path to enable this feature.

With the code available in 2.8.1, it provides a smooth path to enable this feature.
makes sense to me

we have to document it carefully, because it will ease switching from 2.8.1 to 2.9.0,
but then the problem goes down to the upgrade from 2.8.0 to 2.8.1: you have to take care of enabling this feature by upgrading to 2.8.1 all the brokers and then enable the flag.
if you feel strong then I am not against cherry picking this into 2.8.x.
I just wanted to point out that we should start porting less "new features" to released branches, in order to preserve stability as much as possible, but this is not the right thread for discussing this point.

we have to document it carefully, because it will ease switching from 2.8.1 to 2.9.0,
but then the problem goes down to the upgrade from 2.8.0 to 2.8.1: you have to take care of enabling this feature by upgrading to 2.8.1 all the brokers and then enable the flag.

Marked it for doc-required and releasenote-required.

@eolivelli @sijie If users enable this feature, they could upgrade the Pulsar broker normally, but if they want to degrade the Pulsar broker, the old Pulsar broker couldn't parse the compression ManagedLedgerInfo data. Do you have any idea about this, or maybe we only need to add a warning in the release note?

@gaoran10 I don't think we need to worry about the downgrade. We just need to document this clearly.

@gaoran10 I don't think we need to worry about the downgrade. We just need to document this clearly.

Ok

Should we change their positions so as to avoid NPE ?
metadata.getCompressionType().equals(CompressionType.ZLIB.name())

change to
CompressionType.ZLIB.name().equals(...)

Good catch, I'll change it.

We should try to use a magic number with the most significant bit set to 1 because that should be guaranteed to be a non-valid protobuf sequence

@merlimat I'm not sure about how to generate a non-valid protobuf sequence. I take a look at some docs, the basic protobuf encode format is field_number + wire_type + value, currently, the wire type range is 0-5, so I use a code 111. Because I want to use a short value to present the magic number, so the MSB couldn't be 1, could you provide some suggestions? Or we could use a magic number > 111 1111 1111 1111, an int value?

authors: eolivelli gaoran10 sijie


text:
[fix][broker] Fix creating system namespace topic failure.

Fixes #14935
Master Issue: #14935
Motivation
When creating system namespace topic, it should always be successful.
Documentation

[x] no-need-doc

https://github.com/apache/pulsar/blob/ca436a96f0ab1ba7c9af4e613d2bf844a2c315e6/pulsar-transaction/coordinator/src/main/java/org/apache/pulsar/transaction/coordinator/impl/MLTransactionLogImpl.java#L61
https://github.com/apache/pulsar/blob/ca436a96f0ab1ba7c9af4e613d2bf844a2c315e6/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStore.java#L64
https://github.com/apache/pulsar/blob/ca436a96f0ab1ba7c9af4e613d2bf844a2c315e6/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java#L66
this three topic can't be created to persistent topic, so I think we don't use isSystemTopic() directly

https://github.com/apache/pulsar/blob/ca436a96f0ab1ba7c9af4e613d2bf844a2c315e6/pulsar-transaction/coordinator/src/main/java/org/apache/pulsar/transaction/coordinator/impl/MLTransactionLogImpl.java#L61
https://github.com/apache/pulsar/blob/ca436a96f0ab1ba7c9af4e613d2bf844a2c315e6/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStore.java#L64
https://github.com/apache/pulsar/blob/ca436a96f0ab1ba7c9af4e613d2bf844a2c315e6/pulsar-common/src/main/java/org/apache/pulsar/common/naming/TopicName.java#L66
these three topic can't be created to persistent topic, so I think we don't use isSystemTopic() directly

@congbobo184  We should update isSystemTopic  method then.

authors: Technoboy- congbobo184 gaozhangmin


text:
TEST: fix modify env

DO NOT MERGE / TEST ONLY
See https://github.com/liudezhi2098/pulsar/pull/11 and https://github.com/apache/pulsar/pull/16884 instead.

[x] doc-not-needed

"Pulsar CI / CI - Unit - Other" passed. Closed.


text:
[improve][test] Disable OTel autoconfigured exporters in tests

Motivation
Running InflightReadsLimiterTest is slow and prints these warnings
Nov 01, 2024 4:37:19 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Failed to export metrics. Server responded with gRPC status code 2. Error message: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4317
Nov 01, 2024 4:37:21 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Failed to export metrics. Server responded with gRPC status code 2. Error message: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4317
Nov 01, 2024 4:37:26 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Failed to export metrics. Server responded with gRPC status code 2. Error message: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4317
Nov 01, 2024 4:37:31 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Failed to export metrics. Server responded with gRPC status code 2. Error message: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4317

Default OTel exporters should be disabled in tests by default.
Modifications
Disable default OTel exporters in tests.
Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

@dragosvictor @heesung-sn  Please review

ref: https://opentelemetry.io/docs/languages/java/configuration/#properties-exporters

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 74.29%. Comparing base (bbc6224) to head (f902ce2).
Report is 706 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23540      +/-   ##
============================================
+ Coverage     73.57%   74.29%   +0.72%
- Complexity    32624    34402    +1778
============================================
  Files          1877     1943      +66
  Lines        139502   147050    +7548
  Branches      15299    16209     +910
============================================
+ Hits         102638   109256    +6618
- Misses        28908    29355     +447
- Partials       7956     8439     +483

Flag
Coverage Δ

inttests
27.59% <ø> (+3.01%)
:arrow_up:

systests
24.37% <ø> (+0.04%)
:arrow_up:

unittests
73.67% <ø> (+0.82%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.
see 656 files with indirect coverage changes

authors: codecov-commenter heesung-sn lhotari


text:
issue#9345 -- bug fix for RuntimeUtils.getCmd not setting pending-async-requests

Fixes #9345
Motivation
When I enable either the Process or Kubernetes function Runtimes and set the parameter "maxPendingAsyncRequests" to a value other than 1000 in the function worker configuration (e.g. functions_worker.yml), any function instances launched by the worker continue to have maxPendingAsyncRequests = 1000. I'd like to be able to adjust this parameter to guard against my function running out of memory -- without increasing the function heap size -- due to too many publish operations in flight, as can occur when the output topic's backlog quota has been reached.
Modifications
Modified RuntimeUtils.getCmd to add --pending-sync-requests to the list of arguments passed to JavaInstanceStarter
Verifying this change
This change added tests and can be verified as follows:

Updated pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/kubernetes/KubernetesRuntimeTest.java to test for presence of --pending-async-requests argument in generated commandline, and increased expected arg counts by 2 to account for expected presence of --pending-async-requests
Updated pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/process/ProcessRuntimeTest.java to test for presence of --pending-async-requests argument in generated commandline, and increased expected arg counts by 2 to account for expected presence of --pending-async-requests
Updated pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/worker/WorkerApiV2ResourceConfigTest.java to verify that the configured value of "maxPendingAsyncRequests" was being properly loaded into the WorkerConfig

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): no
The public API: no
The schema: no
The default values of configurations: no
The wire protocol: no
The rest endpoints: no
The admin cli options: no
Anything that affects deployment: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable
If a feature is not applicable for documentation, explain why?

bug fix to make an existing configuration parameter operational

If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

@nlu90 Please also help review this PR

@nlu90 Please also help review this PR

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

Please merge the newest master and rerun tests

Please merge the newest master and rerun tests

@csthomas1 thanks for fix!

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

@csthomas1 Looks like the test failed by this change. Could you please take a look?
move this to the next release.

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

@csthomas1 Awesome! Thank you for your contribution!

this patch does not apply to branch-7.2

authors: Renkai codelipenghui csthomas1 eolivelli jerrypeng sijie zymap


text:
Fix: increase wait time in recovery tests to avoid flaking

Currently the tests for recovering after TelemetryCommand transmission failure wait 3 seconds after the initial setup for the telemetry client goroutine to run. This is at the edge of expected wait time for the goroutine to go through all of the required code. Increasing the wait time for five seconds guarantees that by the time that we check the expected logs all of the testing scenario will have already completed.
Fixes #410

Codecov Report

Merging #411 (359a670) into master (550f969) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #411   +/-   ##
=========================================
  Coverage     35.06%   35.06%
  Complexity      663      663
=========================================
  Files           220      220
  Lines         11511    11511
  Branches        279      279
=========================================
  Hits           4036     4036
  Misses         7213     7213
  Partials        262      262

Flag
Coverage Δ

golang
37.07% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

It seems like this issue is not related to the wait time of the consumer goroutine. I'm going to try to reproduce in on Windows

@Anakin100100 Thank you for your contribution! I am not familiar with Golang, and I have a question: does Golang have a tool similar to Java's awaitility that can help us avoid using the sleep method and increase the speed of test execution?

@aaron-ai I'm not aware that such tool exists @guyinyou is something like that typical in Go? In this case skipping the sleep period is a bad idea because the function that we are testing is running possibly on another process as a goroutine so there is no guarantee of execution order. This would result in flaking. Running the tests in parallel is a viable idea. I will look into that.

I kinda suspect this could be a deadlock somewhere that is hit more often on windows maybe due to different goroutine implementation. I will try to use go 1.17 locally just to make sure.

I managed to reproduce the issue only once on a completely fresh installation of Go on my desktop. Subsequent builds and test runs run fine.

@aaron-ai could you give me access to run GitHub actions in my pull requests? Currently I can't see if the new changes fix the build issue in cicd. (They seem to work locally)

@aaron-ai could you give me access to run GitHub actions in my pull requests? Currently I can't see if the new changes fix the build issue in cicd. (They seem to work locally)

I am trying to investigate the Github Action approval requirement of ASF project, it is a little bit different from other GitHub project.

@aaron-ai thanks for pointing out the version change. I think we can merge this now that the tests run fine. To sum up in this pr we:

corrected 2 flaky tests
fixed flaking on windows in cicd
corrected some cosmetic changes pointed out by staticcheck.

I'm going to work on the data races in #294 next.

@Anakin100100  Could you comment on this issue https://github.com/apache/rocketmq-clients/issues/294? then I can assign it to you.

authors: Anakin100100 aaron-ai codecov-commenter

repo: apache/rocketmq-clients


text:
DefaultLitePullConsumer seek(MessageQueue,Offset) report   An error occurred in pull message process.

code
	DefaultLitePullConsumer consumer =new DefaultLitePullConsumer("abc","testConsumer", null);
	consumer.setNamesrvAddr("192.168.64.7:9876");
	consumer.start();
	consumer.setAutoCommit(true);

        Collection<MessageQueue> messageQueue2 =  consumer.fetchMessageQueues("topic");

consumer.assign(list);
	for(MessageQueue mq:list){
		consumer.seek(mq,0);
		System.out.println(mq);
	}

	while (true){
		List<MessageExt> messageExtList =consumer.poll();
		System.out.println(messageExtList.size());
        }

report error:
22-02-15 18:13:05,976 INFO RocketmqRemoting - createChannel: begin to connect remote host[192.168.64.7:9876] asynchronously
2022-02-15 18:13:05,984 INFO RocketmqRemoting - NETTY CLIENT PIPELINE: CONNECT  UNKNOWN => 192.168.64.7:9876
2022-02-15 18:13:05,986 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:9876] success, AbstractBootstrap$PendingRegistrationPromise@68ad99fe(success)
2022-02-15 18:13:05,992 DEBUG io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@78f76823
2022-02-15 18:13:05,992 DEBUG io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@78f76823
2022-02-15 18:13:05,992 DEBUG io.netty.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@78f76823
2022-02-15 18:13:06,060 INFO RocketmqClient - the topic[TBW102] route info changed, old[null] ,new[TopicRouteData [orderTopicConf=null, queueDatas=[QueueData [brokerName=rocketmq, readQueueNums=8, writeQueueNums=8, perm=7, topicSysFlag=0]], brokerDatas=[BrokerData [brokerName=rocketmq, brokerAddrs={0=192.168.64.7:10911}]], filterServerTable={}]]
2022-02-15 18:13:06,060 INFO RocketmqClient - updateTopicPublishInfo prev is not null, TopicPublishInfo [orderTopic=false, messageQueueList=[], sendWhichQueue=ThreadLocalIndex{threadLocalIndex=null}, haveTopicRouterInfo=false]
2022-02-15 18:13:06,060 INFO RocketmqClient - topicRouteTable.put. Topic = TBW102, TopicRouteData[TopicRouteData [orderTopicConf=null, queueDatas=[QueueData [brokerName=rocketmq, readQueueNums=8, writeQueueNums=8, perm=7, topicSysFlag=0]], brokerDatas=[BrokerData [brokerName=rocketmq, brokerAddrs={0=192.168.64.7:10911}]], filterServerTable={}]]
2022-02-15 18:13:06,149 INFO RocketmqRemoting - createChannel: begin to connect remote host[192.168.64.7:10911] asynchronously
2022-02-15 18:13:06,149 INFO RocketmqRemoting - NETTY CLIENT PIPELINE: CONNECT  UNKNOWN => 192.168.64.7:10911
2022-02-15 18:13:06,150 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,150 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,150 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,150 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,150 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,150 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,150 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,150 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,150 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,150 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,150 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,150 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,150 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,151 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,151 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,151 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,151 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,151 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,151 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,151 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,151 INFO RocketmqRemoting - createChannel: connect remote host[192.168.64.7:10911] success, AbstractBootstrap$PendingRegistrationPromise@1752d4ed(success)
2022-02-15 18:13:06,162 ERROR RocketmqClient - An error occurred in pull message process.
java.lang.InterruptedException
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081)
at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:276)
at org.apache.rocketmq.remoting.netty.ResponseFuture.waitResponse(ResponseFuture.java:71)
at org.apache.rocketmq.remoting.netty.NettyRemotingAbstract.invokeSyncImpl(NettyRemotingAbstract.java:434)
at org.apache.rocketmq.remoting.netty.NettyRemotingClient.invokeSync(NettyRemotingClient.java:377)
at org.apache.rocketmq.client.impl.MQClientAPIImpl.pullMessageSync(MQClientAPIImpl.java:772)
at org.apache.rocketmq.client.impl.MQClientAPIImpl.pullMessage(MQClientAPIImpl.java:726)
at org.apache.rocketmq.client.impl.consumer.PullAPIWrapper.pullKernelImpl(PullAPIWrapper.java:200)
at org.apache.rocketmq.client.impl.consumer.DefaultLitePullConsumerImpl.pullSyncImpl(DefaultLitePullConsumerImpl.java:910)
at org.apache.rocketmq.client.impl.consumer.DefaultLitePullConsumerImpl.pull(DefaultLitePullConsumerImpl.java:885)
at org.apache.rocketmq.client.impl.consumer.DefaultLitePullConsumerImpl.pull(DefaultLitePullConsumerImpl.java:880)
at org.apache.rocketmq.client.impl.consumer.DefaultLitePullConsumerImpl.access$1400(DefaultLitePullConsumerImpl.java:77)
at org.apache.rocketmq.client.impl.consumer.DefaultLitePullConsumerImpl$PullTaskImpl.run(DefaultLitePullConsumerImpl.java:831)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
2022-02-15 18:13:06,162 WARN RocketmqClient - The Pull Task is cancelled after doPullTask, MessageQueue [topic=abc%4131412277592745308, brokerName=rocketmq, queueId=3]
2022-02-15 18:13:06,165 ERROR RocketmqClient - An error occurred in pull message process.

consumer.seek(mq,0) will interrupt polling thread starting at  consumer.start();
It is just an error log, not an exception.
This log maybe ignored in DefaultLitePullConsumer.

repo: apache/rocketmq


text:
[ISSUE #3684] change client jdk version to 1.6

Make sure set the target branch to develop
What is the purpose of the change
XXXXX
Brief changelog
XX
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage decreased (-0.2%) to 53.207% when pulling 1be3cc363fea0404a3b9c25ec74161d79e0ac42a on duhenglucky:develop_jdk into 9f9b1091722189eedcf07a4dfdaa4e2f061385d3 on apache:develop.

Codecov Report

Merging #3683 (1be3cc3) into develop (9f9b109) will decrease coverage by 0.14%.
The diff coverage is 19.44%.

@@              Coverage Diff              @@
##             develop    #3683      +/-   ##
=============================================
- Coverage      47.37%   47.23%   -0.15%
+ Complexity      5044     5030      -14
=============================================
  Files            627      627
  Lines          41348    41371      +23
  Branches        5372     5374       +2
=============================================
- Hits           19588    19540      -48
- Misses         19332    19406      +74
+ Partials        2428     2425       -3

Impacted Files
Coverage Δ

...rocketmq/client/impl/factory/MQClientInstance.java
52.96% <0.00%> (-0.33%)
:arrow_down:

...mq/client/impl/producer/DefaultMQProducerImpl.java
44.98% <0.00%> (-0.97%)
:arrow_down:

...g/apache/rocketmq/client/impl/MQClientAPIImpl.java
12.84% <50.00%> (-0.03%)
:arrow_down:

...ent/impl/consumer/DefaultLitePullConsumerImpl.java
67.99% <100.00%> (-0.52%)
:arrow_down:

.../rocketmq/client/producer/RequestFutureHolder.java
81.39% <100.00%> (ø)

.../trace/hook/ConsumeMessageOpenTracingHookImpl.java
74.35% <100.00%> (ø)

...rg/apache/rocketmq/common/stats/StatsSnapshot.java
84.61% <0.00%> (-15.39%)
:arrow_down:

...tmq/logappender/log4j2/RocketmqLog4j2Appender.java
35.00% <0.00%> (-10.00%)
:arrow_down:

.../apache/rocketmq/common/stats/MomentStatsItem.java
38.09% <0.00%> (-9.53%)
:arrow_down:

...ache/rocketmq/common/stats/MomentStatsItemSet.java
39.13% <0.00%> (-8.70%)
:arrow_down:

... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9f9b109...1be3cc3. Read the comment docs.

What is this? For traditional cloud customers, you could create a release branch. Or say, you cancel JDK8 some special semantics, release two packages, one of them is backporting 1.6.

If there is no other significant reason, it is better to keep 1.6 for the client, to be compatible with traditional enterprise users.

You can calculate the percentage of traditional enterprises in your customers？ If you could learn more life-cycle governance about production, you could not insist on this solution. I have given you another backport way to support and educate enterprise customers. I think we have many of these customers in ours. Pls remember, Open source doesn't kidnap by our business. You can support innovation and smooth upgrades in a more collaborative way.

@vongosling I insist on it unless there is a significant reason.
If it doesn't gain too much value in the upgrading of the client version,  why should we do that?  Why not give more convenience to users?
If you want to change the version, please show the use cases and explain the values.

authors: codecov-commenter coveralls dongeforever duhenglucky vongosling