text:
Distributed client/server test couldn't fixate the port when using RMI SSL

Liu XP (Bug 64425):
Steps:

Generated RMI keystore and put it on JMeter client and server side.
Because of firewall， I need to fixate RMI port in jmeter.properties for both JMeter client and JMeter server
server_port=1099
client.rmi.localport=2096
server.rmi.localport=2099

Sub caseA:
3.1. If I ignore security with server.rmi.ssl.disable=true in jmeter.properties and then start JMeter server & JMeter client.
4.1. I can run distribution test without RMI SSL.
Sub caseB:
3.2. If I setted security with server.rmi.ssl.disable=false in jmeter.properties and then start JMeter server & JMeter client.
4.2. I got following log message when starting distribution test.
jmeter -n -R 10.100.159.156 -t Debug.jmx
2020-05-12 15:48:16,899 INFO o.a.j.JMeter: Creating summariser <summary>
2020-05-12 15:48:16,902 INFO o.a.j.e.DistributedRunner: Configuring remote engine: 10.100.159.156
2020-05-12 15:48:17,735 INFO o.a.j.e.DistributedRunner: Starting distributed test with remote engines: [10.100.159.156] @ Tue May 12 15:48:17 CST 2020 (1589269697733)
2020-05-12 15:48:17,735 INFO o.a.j.e.ClientJMeterEngine: running clientengine run method
2020-05-12 15:48:17,739 INFO o.a.j.e.u.CompoundVariable: Note: Function class names must contain the string: '.functions.'
2020-05-12 15:48:17,739 INFO o.a.j.e.u.CompoundVariable: Note: Function class names must not contain the string: '.gui.'
2020-05-12 15:48:17,912 INFO o.a.j.r.RmiUtils: System property 'java.rmi.server.hostname' is not defined, using localHost address
2020-05-12 15:48:17,912 INFO o.a.j.r.RmiUtils: Local IP address=10.101.22.54
2020-05-12 15:48:17,914 INFO o.a.j.s.SampleEvent: List of sample_variables: []
2020-05-12 15:48:18,055 INFO o.a.j.r.SSLRMIServerSocketFactory: Created SSLSocket: [SSL: ServerSocket[addr=tcn1010365/10.101.22.54,localport=2098]]
2020-05-12 15:48:18,063 INFO o.a.j.s.BatchSampleSender: Using batching (client settings) for this run. Thresholds: num=100, time=60000
2020-05-12 15:48:18,063 INFO o.a.j.s.DataStrippingSampleSender: Using DataStrippingSampleSender for this run
2020-05-12 15:48:18,063 INFO o.a.j.r.RmiUtils: System property 'java.rmi.server.hostname' is not defined, using localHost address
2020-05-12 15:48:18,063 INFO o.a.j.r.RmiUtils: Local IP address=10.101.22.54
2020-05-12 15:48:18,204 ERROR o.a.j.e.ConvertListeners: RemoteException occurred while replacing Remotable item.
java.rmi.server.ExportException: Listen failed on port: 2098; nested exception is:
java.io.IOException: Could not bind to tcn1010365/10.101.22.54 using port 2098
at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:337) ~[?:?]
at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:243) ~[?:?]
at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:412) ~[?:?]
at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147) ~[?:?]
at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:234) ~[?:?]
at java.rmi.server.UnicastRemoteObject.exportObject(UnicastRemoteObject.java:470) ~[?:?]
at java.rmi.server.UnicastRemoteObject.exportObject(UnicastRemoteObject.java:381) ~[?:?]
at java.rmi.server.UnicastRemoteObject.<init>(UnicastRemoteObject.java:260) ~[?:?]
at org.apache.jmeter.samplers.RemoteSampleListenerImpl.<init>(RemoteSampleListenerImpl.java:44) ~[ApacheJMeter_core.jar:5.2.1]
at org.apache.jmeter.engine.ConvertListeners.addNode(ConvertListeners.java:78) [ApacheJMeter_core.jar:5.2.1]
at org.apache.jorphan.collections.HashTree.traverse(HashTree.java:976) [jorphan.jar:5.2.1]
at org.apache.jmeter.engine.ClientJMeterEngine.runTest(ClientJMeterEngine.java:137) [ApacheJMeter_core.jar:5.2.1]
at org.apache.jmeter.engine.DistributedRunner.start(DistributedRunner.java:135) [ApacheJMeter_core.jar:5.2.1]
at org.apache.jmeter.engine.DistributedRunner.start(DistributedRunner.java:157) [ApacheJMeter_core.jar:5.2.1]
at org.apache.jmeter.JMeter.runNonGui(JMeter.java:1094) [ApacheJMeter_core.jar:5.2.1]
at org.apache.jmeter.JMeter.startNonGui(JMeter.java:992) [ApacheJMeter_core.jar:5.2.1]
at org.apache.jmeter.JMeter.start(JMeter.java:564) [ApacheJMeter_core.jar:5.2.1]
It looks like the following code couldn't be executed again for RMI SSL mode when adding node in org.apache.jmeter.engine.ConvertListeners.java
super(DEFAULT_LOCAL_PORT, RmiUtils.createClientSocketFactory(),  RmiUtils.createServerSocketFactory()); // RemoteSampleListenerImpl.java
Refer fixed Bug: https://github.com/apache/jmeter/issues/4799
Created attachment jmeter.log: JMeter client log
Severity: normal
OS: All

I have the same issue in Jmeter 5.5
I believe this is a bug - when RMI SSL is enabled we are not able to set client.rmi.localport to a specific value without error

Hi. I have the same issue in Jmeter 5.6.3.
server_port=1099
client.rmi.localport=4003
I guarantee that the ports are not occupied by anything.
2024-06-13 16:41:26,582 INFO o.a.j.r.SSLRMIServerSocketFactory: Created SSLSocket: [SSL: ServerSocket[addr=ServerName/10.xx.xx.xx,localport=4005]]
2024-06-13 16:41:26,582 INFO o.a.j.r.RmiUtils: System property 'java.rmi.server.hostname' is not defined, using localHost address
2024-06-13 16:41:26,582 INFO o.a.j.r.RmiUtils: Local IP address=10.xx.xx.xx
2024-06-13 16:41:26,724 INFO o.a.j.r.SSLRMIServerSocketFactory: Created SSLSocket: [SSL: ServerSocket[addr=ServerName/10.xx.xx.xx,localport=4004]]
2024-06-13 16:41:26,724 INFO o.a.j.r.RmiUtils: System property 'java.rmi.server.hostname' is not defined, using localHost address
2024-06-13 16:41:26,724 INFO o.a.j.r.RmiUtils: Local IP address=10.xx.xx.xx
2024-06-13 16:41:26,866 ERROR o.a.j.e.ConvertListeners: RemoteException occurred while replacing Remotable item.
java.rmi.server.ExportException: Listen failed on port: 4005; nested exception is:
java.io.IOException: Could not bind to ServerName/10.xx.xx.xx using port 4005
at java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:337) ~[?:?]
at java.rmi/sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:243) ~[?:?]
at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:412) ~[?:?]
at java.rmi/sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147) ~[?:?]
at java.rmi/sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:234) ~[?:?]
at java.rmi/java.rmi.server.UnicastRemoteObject.exportObject(UnicastRemoteObject.java:470) ~[?:?]
at java.rmi/java.rmi.server.UnicastRemoteObject.exportObject(UnicastRemoteObject.java:381) ~[?:?]
at java.rmi/java.rmi.server.UnicastRemoteObject.(UnicastRemoteObject.java:260) ~[?:?]
at org.apache.jmeter.samplers.RemoteSampleListenerImpl.(RemoteSampleListenerImpl.java:43) ~[ApacheJMeter_core.jar:5.6.3]
at org.apache.jmeter.engine.ConvertListeners.addNode(ConvertListeners.java:77) ~[ApacheJMeter_core.jar:5.6.3]
at org.apache.jorphan.collections.HashTree.traverseInto(HashTree.java:1029) ~[jorphan.jar:5.6.3]
at org.apache.jorphan.collections.HashTree.traverseInto(HashTree.java:1030) ~[jorphan.jar:5.6.3]
at org.apache.jorphan.collections.HashTree.traverse(HashTree.java:1012) ~[jorphan.jar:5.6.3]
at org.apache.jmeter.engine.ClientJMeterEngine.runTest(ClientJMeterEngine.java:137) ~[ApacheJMeter_core.jar:5.6.3]
at org.apache.jmeter.engine.DistributedRunner.start(DistributedRunner.java:146) ~[ApacheJMeter_core.jar:5.6.3]
at org.apache.jmeter.gui.action.RemoteStart.doAction(RemoteStart.java:83) ~[ApacheJMeter_core.jar:5.6.3]
at org.apache.jmeter.gui.action.ActionRouter.performAction(ActionRouter.java:83) ~[ApacheJMeter_core.jar:5.6.3]
at org.apache.jmeter.gui.action.ActionRouter.lambda$actionPerformed$0(ActionRouter.java:65) ~[ApacheJMeter_core.jar:5.6.3]
at java.desktop/java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:313) [?:?]
at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:770) [?:?]
at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721) [?:?]
at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:715) [?:?]
at java.base/java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85) [?:?]
at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:740) [?:?]
at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203) [?:?]
at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124) [?:?]
at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113) [?:?]
at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109) [?:?]
at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101) [?:?]
at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:90) [?:?]
Caused by: java.io.IOException: Could not bind to ServerName/10.xx.xx.xx using port 4005
at org.apache.jmeter.rmi.SSLRMIServerSocketFactory.createServerSocket(SSLRMIServerSocketFactory.java:138) ~[ApacheJMeter_core.jar:5.6.3]
at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:670) ~[?:?]
at java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:324) ~[?:?]
... 30 more
Caused by: java.net.BindException: Address already in use: NET_Bind
at java.base/java.net.PlainSocketImpl.bind0(Native Method) ~[?:?]
at java.base/java.net.PlainSocketImpl.socketBind(PlainSocketImpl.java:132) ~[?:?]
at java.base/java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:452) ~[?:?]
at java.base/java.net.ServerSocket.bind(ServerSocket.java:381) ~[?:?]
at java.base/java.net.ServerSocket.(ServerSocket.java:243) ~[?:?]
at java.base/javax.net.ssl.SSLServerSocket.(SSLServerSocket.java:181) ~[?:?]
at java.base/sun.security.ssl.SSLServerSocketImpl.(SSLServerSocketImpl.java:78) ~[?:?]
at java.base/sun.security.ssl.SSLServerSocketFactoryImpl.createServerSocket(SSLServerSocketFactoryImpl.java:87) ~[?:?]
at org.apache.jmeter.rmi.SSLRMIServerSocketFactory.createServerSocket(SSLRMIServerSocketFactory.java:136) ~[ApacheJMeter_core.jar:5.6.3]
at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:670) ~[?:?]
at java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:324) ~[?:?]
... 30 more

authors: asfimport georgejdli vdvarlamov

repo: apache/jmeter


text:
Misc Sonar Fixes

JSONPathAssertion: Merge nested if
SMIMEAssertion: add NOSONAR to getLogger class
XMLSchemaAssertion: Remove throws IllegalArgumentException
XMLSchemaAssertion: Formatting and removal of useless comments
AssertionGui: Replace String literal with constant
AssertionGui: Comment tidy and method simplification

Codecov Report

Merging #528 into master will decrease coverage by <.01%.
The diff coverage is 75%.

@@             Coverage Diff              @@
##             master     #528      +/-   ##
============================================
- Coverage     56.19%   56.19%   -0.01%
  Complexity    10020    10020
============================================
  Files          1024     1024
  Lines         62915    62912       -3
  Branches       7065     7065
============================================
- Hits          35358    35352       -6
- Misses        25078    25083       +5
+ Partials       2479     2477       -2

Impacted Files
Coverage Δ
Complexity Δ

...a/org/apache/jmeter/assertions/SMIMEAssertion.java
43.57% <100%> (ø)
19 <1> (ø)
:arrow_down:

...g/apache/jmeter/assertions/XMLSchemaAssertion.java
73.13% <100%> (ø)
10 <1> (ø)
:arrow_down:

...org/apache/jmeter/assertions/gui/AssertionGui.java
65.41% <50%> (+0.48%)
17 <0> (ø)
:arrow_down:

...rg/apache/jmeter/assertions/JSONPathAssertion.java
86.36% <75%> (-0.16%)
37 <0> (ø)

...main/java/org/apache/jorphan/gui/MenuScroller.java
30.53% <0%> (-4.2%)
8% <0%> (-2%)

...n/java/org/apache/jmeter/reporters/Summariser.java
86.25% <0%> (+1.52%)
19% <0%> (+2%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3263a5d...3243c8e. Read the comment docs.

source: gharchive/pull-request

license_type: permissive

license_source: bigquery


text:
KAFKA-15853: Move OffsetConfig to group-coordinator module

blocker for #15158
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@mimaison
@ijuma
This PR are ready for merge for me. Please, take a look.

@dajac Can you confirm if org.apache.kafka.coordinator.group is the best place to put the OffsetConfig class?

That seems fine. We also have GroupCoordinatorConfig. I hope that we can eventually consolidate them but we can look into this afterwards. We already have a Jira for reworking the group coordinator configs: https://issues.apache.org/jira/browse/KAFKA-15089.

authors: OmniaGM dajac mimaison nizhikov

repo: apache/kafka

license_source: github-api


text:
KAFKA-15717: Added KRaft support in LeaderEpochIntegrationTest

This PR adds KRaft support in LeaderEpochIntegrationTest
JIRA : https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-15717
Test Result

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@benstopford Please if you can review and provide the feedback.

Thanks for the contribution!
Do you have a Jira account? If so I can assign https://issues.apache.org/jira/browse/KAFKA-15717 to you.

Thanks for the review! @mimaison
Yes, I have
About my jira account, Username and Fullname are appchemist.

Done, thanks

authors: appchemist mimaison


text:
KAFKA-15853: Move KafkaConfig log properties and docs out of core

part 1 for #15501
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@nizhikov can you have a look please?

Now that https://github.com/apache/kafka/pull/15075 got merged, this needs rebasing.

rebased the pr

Since Mickael is already reviewing it, I'll focus on other PRs. Thanks @mimaison !

@OmniaGM I think this is almost ready, but there are a couple of minor comments to address and some conflicts to fix.

Will be addressing this soon after another pr

@fvaleri, @nizhikov @mimaison just rebased the pr and addressed the feedback will wait for this one to be merged before rebasing the other pr for KafkaConfig

@OmniaGM Is this the next PR in line for moving broker configs?

@OmniaGM Is this the next PR in line for moving broker configs?

Yes, I rebased the PR and also pushed a refactor to follow the same pattern we adopted in https://github.com/apache/kafka/pull/15656#discussion_r1554518847, https://github.com/apache/kafka/pull/15684, https://github.com/apache/kafka/pull/15670

The last commit renamed the class to KafkaLogConfigs instead similar to KafkaSecurityConfigs and ZKConfigs. I tried to move some to LogConfig.ServerLogConfig similar to LogConfig.RemoteLogConfig specially that some default configs are defined in LogConfig. However I'm bit concern that LogConfig seems already huge. What others prefer? Keep it in  KafkaLogConfigs  or move them to LogConfig.ServerLogConfig`
cc: @mimaison, @chia7712

Keep it in KafkaLogConfigs

I prefer this way since LogConfig includes too many stuff now ... maybe we should move the constants out of LogConfig

maybe we should move the constants out of LogConfig

We can try to do this in a follow up after we merge all PRs for KafkaConfig as am worried about potential circular dependency. And it would better to see the full pic before do improvement for LogConfig.

There are build error. please fix it :)

hopefully for the last time :)

I feel #15684 will cause many conflicts to this PR, and vice versa :_

Let's just let Omnia decides which PR she prefers doing first. Then we can all review that PR, and once merged move onto the next one. This would avoid unnecessary churn as I expect she's getting tired of rebasing all of these PRs every day!

Thanks @mimaison this is really thoughtful! breaking KafkaConfig out of core is one of the hardest cleanup specially if we want to move with smaller/medium PRs. So I don't see any other way than going through it. I think it might be better to take them in order so we can get them merged and the rebases can be easier.
Let's merge https://github.com/apache/kafka/pull/15684 as

it is smaller
more contained than the log config
and group configs are more likely to change as part of KIP-848 and any of its followups which make the conflicts much harder and risker to get messed up.

However I'm bit concern that LogConfig seems already huge. What others prefer? Keep it in KafkaLogConfigs or move them to LogConfig.ServerLogConfig

the most default values of KafkaLogConfigs are in LogConfig, and they are in different module. That pattern is no similar to ReplicationConfigs, KafkaSecurityConfigs. Is server-common module more suitable to collect those server side configs since bothstorage and server depend on server-common. Also, server-common has org.apache.kafka.server.config package too.

However I'm bit concern that LogConfig seems already huge. What others prefer? Keep it in KafkaLogConfigs or move them to LogConfig.ServerLogConfig

the most default values of KafkaLogConfigs are in LogConfig, and they are in different module. That pattern is no similar to ReplicationConfigs, KafkaSecurityConfigs. Is server-common module more suitable to collect those server side configs since bothstorage and server depend on server-common. Also, server-common has org.apache.kafka.server.config package too.

I think it might be better and simpler to move KafkaLogConfigs into server-common and make LogConfig use them. I'll push an update with this soon

[2024-04-18T17:26:16.149Z] [ant:checkstyle] [ERROR] /home/jenkins/workspace/Kafka_kafka-pr_PR-15569/server-common/src/main/java/org/apache/kafka/server/config/ServerLogConfigs.java:22:1: Disallowed import - org.apache.kafka.server.common.MetadataVersion.IBP_3_0_IV1. [ImportControl]

[2024-04-18T17:26:16.149Z] [ant:checkstyle] [ERROR] /home/jenkins/workspace/Kafka_kafka-pr_PR-15569/server-common/src/main/java/org/apache/kafka/server/config/ServerLogConfigs.java:23:1: Disallowed import - org.apache.kafka.server.config.ServerTopicConfigSynonyms.LOG_PREFIX. [ImportControl]

there are imports error :_

The failed test testParseAndValidateAddressesWithReverseLookup will get fixed by #15758. I will merge this PR

authors: OmniaGM chia7712 mimaison showuon


text:
KAFKA-16711: make sure update highestOffsetInRemoteStorage after log dir change

We use topicIdPartition as the key for each RLM task. After supporting JBOD feature, it will cause highestOffsetInRemoteStorage in log not updated after logDir alter completion. The reproducing flow is like this:

tp-0 locating in dirA is the leader of the partition
tp-0 is altering logDir to dirB
tp-0 is copying segments to remote storage (note: the log in dirA)
The logDir altering for tp-0 is completed
remoteLogManager#onLeadershipChange is invoked, copiedOffsetOption is reset to Optional.empty()
The copying segments to remote storage in step 3 for tp-0 is completed, updating copiedOffsetOption to new offset, as well as the log#highestOffsetInRemoteStorage. (note: the log in dirA)
In the next run of RLMTask, the log will be the one in target dir (dirB), and the log#highestOffsetInRemoteStorage (dirB) will be the default value (-1), which will block the log cleanup operation.

This PR makes sure after log dir changed, the RLMTask state will be reset. Also adding a test for the above scenario.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@kamalcph @satishd @clolov , call for review. Thanks.

In the next run of RLMTask, the log will be the one in target dir (dirB), and the log#highestOffsetInRemoteStorage (dirB) will be the default value (-1), which will block the log cleanup operation.

If there are some segments to upload from the dirB, will the log#highestOffsetInRemoteStorage updated?
Trying to understand how the issue surfaced with the AlterLogDirTest, there we had one log segment (002.log) to upload post the alterLogDir operation.

@kamalcph ,

If there are some segments to upload from the dirB, will the log#highestOffsetInRemoteStorage updated?

Yes, it will! That means, the inconsistent state will be fixed after next log segment copied to remote from dirB.

Trying to understand how the issue surfaced with the AlterLogDirTest, there we had one log segment (002.log) to upload post the alterLogDir operation.

Yes, it should not have the issue, but in edge case, there's a race condition like this:

alter from dirA to dirB completed, but leader is still in dirA because we need to wait until controller update metadata log
The condition we're waiting for alterLogDirAction is that when the dirB contains logs, and dirA is gone, we consider it is completed
We started to produce data to the topic, and dirB (follower) is replicating 002.log from dirA(leader)
log segment 002.log is uploaded to remote (in dirA)
received leadership change to the dirB, and now it doesn't have any logs to uploaded.
The earliest offset check failed for dirB (expect 3, but it's 2 since 002.log is existed)

@satishd , call for review. Thanks.

There are unrelated test failures, merging it to trunk.

authors: kamalcph satishd showuon


text:
KAFKA-4720: add a KStream#peek(ForeachAction<K, V>)

https://issues.apache.org/jira/browse/KAFKA-4720
Peek is a handy method to have to insert diagnostics that do not affect the stream itself, but some external state such as logging or metrics collection.

Thanks for this PR. Because this is a public API change, it is required to do a KIP first: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
If you need any help with preparing a KIP, please ask for help at the dev mailing list.

Added final to everywhere it made sense, and added test coverage that the value is passed as well as the key.

New version up with a rewritten test that should address all above comments.

Okay, another pass, hopefully there are not many nits left to pick...

I think the test failure is unrelated:
java.lang.AssertionError: Heartbeat thread active after close
	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinatorTest.testHeartbeatThreadClose(ConsumerCoordinatorTest.java:1425)

LGTM, thanks @stevenschlansker

Yes, this was introduced by removing a this. which was actually necessary :( pushed a fix.

The KIP vote passed with +8 votes.

Merged to trunk.

@guozhangwang Created https://issues.apache.org/jira/browse/KAFKA-4772

How is this different from map with some side-effecting operation?

@xiongtx:
peek is similar to map, just like foreach is similar to map.  An important reason to have peek and foreach is that it allows a developer to clearly communicate the intent of an operation:

map: "When I use map, my intent is to modify the input stream, i.e. I want to an output stream that contains the modified data so that I can perform additional operations on the modified data."
foreach: "When I use foreach, my intent is to cause some side effects based on the input stream AND then I want to terminate the processing." (that's why foreach returns void and thus is labeled as a terminal operation)
peek: "When I use peek, my intent is to cause some side effects based on the input data AND then I want to continue processing the (unmodified) input data."

The PR includes the following javadocs snippet:

Peek is a non-terminal operation that triggers a side effect (such as logging or statistics collection) and returns an unchanged stream.

See also the Java 8 documentation on java.util.stream.Stream#peek():
https://docs.oracle.com/javase/8/docs/api/java/util/stream/Stream.html#peek-java.util.function.Consumer-
Does that help?

@miguno Fair enough, thanks.

authors: enothereska guozhangwang miguno mjsax stevenschlansker xiongtx


text:
[LIBCLOUD-711] Fixed occasional CRC check failure when decompressing …

…large responses
fixes issue: https://issues.apache.org/jira/browse/LIBCLOUD-711

Thanks for the report.
Looking at the issue it seems more like a problem with the response getting corrupted and not an issue with the actual library we use to decompress the response - that is why thee CRC exists in the first place - to make sure the data is intact and not corrupted.
If it was an issue with a library, it would fail all of the time, not just some times. It could be a some kind of network issue on your or CloudSigma's side, a loadbalancer at CloudSigma corrupting the response or something else in the middle causing this issue to happen.
Only thing I find weird is that using zlib doesn't cause this issue to happen. Python zlib library should still include a CRC.
In any case, I will dig in and also try to reproduce the issue myself.
/cc @vpetersson

I would agree with you, but we've been polling CloudSigma's API with gzip
request/response headers (specifically the /servers/detail/?limit=0 action)
every few minutes for a while now using curl's decompression library, and
haven't had any issues.  Also, while debugging this issue, the compressed
response data is complete when decompressed with command line gzip (but
it's missing the EOF?).
Granted, it is strange that there's no complaints about it with other
drivers. If you need any other info to help troubleshoot, let me know.  In
the meantime I'll do some testing in other environments on the off chance
that something's off with mine.
On Sat, May 16, 2015, 21:12 Tomaz Muraus notifications@github.com wrote:

Thanks for the report.
Looking at the issue it seems more like a problem with the response
getting corrupted and not an issue with the actual library we use to
decompress the response - that is why thee CRC exists in the first place -
to make sure the data is intact and not corrupted.
If it was an issue with a library, it would fail all of the time, not just
some times. It could be a some kind of network issue on your or
CloudSigma's side, a loadbalancer at CloudSigma corrupting the response or
something else in the middle causing this issue to happen.
Only thing I find weird is that using zlib doesn't cause this issue to
happen. Python zlib library should still include a CRC.
In any case, I will dig in and also try to reproduce the issue myself.
/cc @vpetersson https://github.com/vpetersson
—
Reply to this email directly or view it on GitHub
https://github.com/apache/libcloud/pull/519#issuecomment-102667197.

@chrisob Alright, thanks.
I will also apply try to reproduce the issue you have encountered and test your patch.

I did a bit more research and it looks like the problem could be related to the server using "deflate" compression and incorrectly advertising it as "gzip" to the client via content-encoding header (see http://stackoverflow.com/a/9856879).
If that's the case, it's really an issue with the server and not issue in Libcloud. We could add try / catch and fall back to zlib if gzip throws checksum error exceptions, but this seems hacky.
In any case, I will also test out your changes.

Also, this might be relevant - https://stackoverflow.com/questions/3122145/zlib-error-error-3-while-decompressing-incorrect-header-check/22310760#comment42261099_22310760
In any case, we already check for zlib / deflate content-encoding header and use zlib to decompress data if this header is returned. Also using zlib when gzip header is returned seems wrong since it's really a server misconfiguration / issue and we shouldn't silently accept it (although, it does look like a lot of clients do that :/)

This patch solves the problem for me (libcloud 0.20.1, Python 2.7.6 )!

(@Kami just a side-note: I'm no longer with CloudSigma :)

Works for me too.

@Kami why can't we do it like requests do https://github.com/kennethreitz/requests/commit/6d6252aa9fc4bb38d7f68073b41092c31ddb146b#commitcomment-19841258 ?
Python gzip module states that data compression is provided by the zlib module.
I'm not an experrt on decompression topic, but I want to use libcloud to manage cloudsigma and it's unusable right now :(

authors: Kami chrisob kamilchm pka vpetersson

repo: apache/libcloud


text:
LOG4J2-2785: Added support for pattern Layout to abbreviate logger or class names except n rightmost words

Hi, I've added a specialized abbreviator named DynamicWordAbbreviator to address the specific requirement of this feature request. Javadoc added and JUnit test updated to cover the new feature.

When I have some time to tinker, I think this feature could be added to the existing abbreviator. All existing patterns effectively use rightWordCount=1, however we could make that configurable and allow existing options to work with this.
Something along the lines of
int limit = input.length();
for (int i = 0; I < rightWordCount; I++) {
    limit = input.lastIndexOf('.', limit);
    if (limit < 0) {
        return input;
    }
}
// call into standard abbreviator.

@carterkozak Here's another small update to my pull request.
Can you merge it into the release branch please?

@sman-81 we are all volunteers and have limited time to devote to review. Your most recent comment comes across as entitled and rude. Please work on that. In your position, I might have replied on the PR with "Is there anything I can do to get this into a mergable state?"

you listen to their criticism
you incorporate their feedback

I asked you twice to avoid string.split in favor of indexOf/lastIndexOf, but it's still used.

@carterkozak
I'm a volunteer just like you.
My pull request is from July 1st, thus seven months old by now!
The last comment from anyone other than me was August 12, 2021.
Such discourages people from contributing to the project.

I asked you twice to avoid string.split in favor of indexOf/lastIndexOf, but it's still used.

And you are wrong. String#split does not create a Pattern object for certain one-character patterns such as here, as I have pointed out to you (see comment on August 5, 2021).

The point is that string.split is inefficient, regardless of internal details because it's allocating several new strings unnecessarily.

Hi there @carterkozak now String#split is gone. I have also squashed commits into one. Please take a look.

This patch was manually extracted and merged since the backing repo has been deleted.

Hard to believe my PR actually got merged after a little over 1,5 years :)

authors: carterkozak rgoers sman-81


text:
[MSKINS-157] Fix footer formatting changes in 1.8

(introduced by MSKINS-148 and MSKINS-149)

Looking into...

Thanks, could you please share some insights about the roadmap to 2.0?

authors: michael-o twogee

repo: apache/maven-fluido-skin


text:
METRON-1968: Messages are lost when a parser produces multiple messages and batch size is greater than 1

Contributor Comments
This PR represents a fairly significant shift in the Writer class architecture.  Currently these classes do not support tuples that result in multiple messages, mainly due to a limitation in the BulkMessageWriter interface.  The write method accepts separate lists of tuples and messages so there is no way to know which tuples are associated with which message.  It has worked so far with parsers that only emit a single message from a tuple because a 1 to 1 relationship is assumed in these classes.
I experimented with several different approaches to fixing this and tried to follow a strategy that would avoid having to significantly rewrite multiple classes (this was unavoidable in certain places).  I change the BulkMessageWriter.write interface from:
BulkWriterResponse write(String sensorType, WriterConfiguration configurations, Iterable<Tuple> tuples, List<MESSAGE_T> messages) throws Exception;

to:
BulkWriterResponse write(String sensorType, WriterConfiguration configurations, Map<String, MESSAGE_T> messages) throws Exception;

The messages being passed in are now represented as a map where the keys are message ids and the values are the messages.  This made changes to the BulkMessageWriter implementations easier because they continue to assume a 1 to 1 relationship between message ids (formerly tuples) and messages.  Now the writers report which message ids succeeded/failed instead of tuples.  The BulkWriterComponent continues track batch sizes and timeouts but no longer manages tuples or error handling.  Instead it calls BulkWriterResponseHandler.handleFlush when messages are flushed.  This interface is injected into BulkWriterComponent and allows us to properly commit messages according to the requirements of the platform the classes are running on.  For now a Storm implementation of BulkWriterResponseHandler is setup and passed into the BulkWriterComponent.   Managing tuple to message relationships as well as tuple acking and error handling are now done in a single class.  The bolts now have to setup a StormBulkWriterResponseHandler object and add tuples/messages to it.
Changes Included

The BulkMessageWriter is updated and all implementations have been adjusted to conform to the new interfaces.  Changes to the implementations are small and straightforward.
Tuple acking and error handling is moved out of BulkWriterComponent and into StormBulkWriterResponseHandler.
Bolts have been updated with the new StormBulkWriterResponseHandler pattern
I noticed HBaseWriter and `PcapWriter are no longer being used.  Rather than go to the trouble of updating their tests I removed them.
Removed WriterHandler.handleAck since acks are no longer handled here.
Added logging to make troubleshooting writer issues easier

There were also several significant changes that needed to be made to the tests:

The writer tests are updated to match the new BulkMessageWriter.write interface.  In most cases these changes were simple however I noticed there were no unit tests for KafkaWriter.write so I added them.
The parser integration tests now verify the all tuples were acked.
There is now a parser integration test that simulates the use case described in this Jira.  The jsonMapQuery parser integration test now produces multiple messages from a single tuple and sets the batch size to 5.
A test was added for StormBulkWriterResponseHandler that also simulates the use case described in this Jira.  Error handling tests that were originally in BulkWriterComponentTest were migrated here.

Testing
This has been tested in full dev both for regression and for the use case described in this Jira.  There are 3 test cases:

bro sensor with a batch size set to 5
snort sensor with a batch size set to 1
jsonMapQuery sensor that produces multiple messages from a tuple with a batch size set to both less than and greater than the number of messages

Setup

Spin up full dev and verify data is landing in both Elasticsearch and HDFS.
Stop all sensors with service sensor-stubs stop.
Create the jsonMapQuery sensor topic:

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic jsonMapQuery --partitions 1 --replication-factor 1

Copy the jsonMapQuery sample data to full dev:

cd /metron-deployment/development/centos6
vagrant scp ../../../metron-platform/metron-integration-test/src/main/sample/data/jsonMapQuery/raw/jsonMapExampleOutput /tmp

Stop Metron Parsers in Ambari.
Navigate to Ambari > Metron > Configs > Parsers and change "Metron Parsers" from bro,snort,yaf to bro,snort,jsonMapQuery.
Start Metron Parsers in Ambari.
Clear the Elasticsearch indices:

curl -XDELETE http://node1:9200/bro_index*
curl -XDELETE http://node1:9200/snort_index*

Stop Metron Indexing in Ambari.
Clear the HDFS files:

su hdfs -c "hdfs dfs -rm /apps/metron/indexing/indexed/bro/*"
su hdfs -c "hdfs dfs -rm /apps/metron/indexing/indexed/snort/*"

Start Metron Indexing in Ambari.

Verify there is no bro or snort data in Elasticsearch or HDFS:

[root@node1 vagrant]# curl -s -XGET http://node1:9200/_cat/indices?h=index,docs.count | grep bro
[root@node1 vagrant]# curl -s -XGET http://node1:9200/_cat/indices?h=index,docs.count | grep snort
[root@node1 vagrant]# hdfs dfs -cat /apps/metron/indexing/indexed/bro/*.json | wc -l
cat: `/apps/metron/indexing/indexed/bro/*.json': No such file or directory
0
[root@node1 vagrant]# hdfs dfs -cat /apps/metron/indexing/indexed/snort/*.json | wc -l
cat: `/apps/metron/indexing/indexed/snort/*.json': No such file or directory
0

Each topology (parsing, enrichment, indexing) can be configured with a batch size and timeout.  The batch timeout defaults to 1/2 the tuple timeout so tests described next should be done reasonably quick (before the timeout happens).  To make the test results easier to understand we also need to adjust the batch sizes to 1 for topologies we are not directly testing.  Set the bro parser batch size to 1:

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "parserClassName": "org.apache.metron.parsers.bro.BasicBroParser",
  "sensorTopic": "bro",
  "parserConfig": {
    "batchSize": 1
  }
}' 'http://user:password@node1:8082/api/v1/sensor/parser/config/bro'

Set the snort parser batch size to 1:

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "parserClassName": "org.apache.metron.parsers.snort.BasicSnortParser",
  "sensorTopic": "snort",
  "parserConfig": {
    "batchSize": 1
  }
}' 'http://user:password@node1:8082/api/v1/sensor/parser/config/snort'

Set the enrichment batch size to 1:

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "es.clustername": "metron",
  "es.ip": "node1:9200",
  "es.date.format": "yyyy.MM.dd.HH",
  "parser.error.topic": "indexing",
  "update.hbase.table": "metron_update",
  "update.hbase.cf": "t",
  "es.client.settings": {},
  "profiler.client.period.duration": "15",
  "profiler.client.period.duration.units": "MINUTES",
  "user.settings.hbase.table": "user_settings",
  "user.settings.hbase.cf": "cf",
  "bootstrap.servers": "node1:6667",
  "source.type.field": "source:type",
  "threat.triage.score.field": "threat:triage:score",
  "enrichment.writer.batchSize": "1",
  "profiler.writer.batchSize": "15",
  "profiler.writer.batchTimeout": "0",
  "geo.hdfs.file": "/apps/metron/geo/default/GeoLite2-City.tar.gz",
  "asn.hdfs.file": "/apps/metron/asn/default/GeoLite2-ASN.tar.gz"
}' 'http://user:password@node1:8082/api/v1/global/config'

Set the jsonMapQuery indexing batch size to 1 (since we will test this sensor in parsers):

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "hdfs": {
    "index": "jsonmapquery",
    "batchSize": 1,
    "enabled": true
  },
  "elasticsearch": {
    "index": "jsonmapquery",
    "batchSize": 1,
    "enabled": true
  },
  "solr": {
    "index": "jsonmapquery",
    "batchSize": 1,
    "enabled": false
  }
}' 'http://user:password@node1:8082/api/v1/sensor/indexing/config/jsonMapQuery'

It can be helpful to keep a Kafka console consumer running for each topic:
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper node1:2181 --topic bro
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper node1:2181 --topic snort
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper node1:2181 --topic jsonMapQuery
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper node1:2181 --topic enrichments
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper node1:2181 --topic indexing

bro

Bro has a batch size of 5 in indexing by default.  Write 4 bro messages:

shuf -n 4 /opt/sensor-stubs/data/bro.out | sed -e "s/\"ts\"\:[0-9]\+\./\"ts\"\:`date +%s`\./g" | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic bro

There should be no data in Elasticsearch or HDFS:
[root@node1 vagrant]# curl -s -XGET http://node1:9200/_cat/indices?h=index,docs.count | grep bro
[root@node1 vagrant]# hdfs dfs -cat /apps/metron/indexing/indexed/bro/*.json | wc -l
cat: `/apps/metron/indexing/indexed/bro/*.json': No such file or directory
0

Write an additional bro message:

shuf -n 1 /opt/sensor-stubs/data/bro.out | sed -e "s/\"ts\"\:[0-9]\+\./\"ts\"\:`date +%s`\./g" | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic bro

There should now be 5 messages in Elasticsearch and HDFS:
[root@node1 vagrant]# curl -s -XGET http://node1:9200/_cat/indices?h=index,docs.count | grep bro
bro_index_2019.02.07.22    5
[root@node1 vagrant]# hdfs dfs -cat /apps/metron/indexing/indexed/bro/*.json | wc -l
5

snort

Snort has a batch size of 1 in indexing by default:  Write a snort message:

shuf -n 1 /opt/sensor-stubs/data/snort.out | sed -e "s/[^,]\+ ,/`date +'%m\/%d\/%y-%H:%M:%S'`.000000 ,/g" | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic snort

There should now be 1 message in Elasticsearch and HDFS:
[root@node1 vagrant]# curl -s -XGET http://node1:9200/_cat/indices?h=index,docs.count | grep snort
snort_index_2019.02.07.22    1
[root@node1 vagrant]# hdfs dfs -cat /apps/metron/indexing/indexed/snort/*.json | wc -l
1

jsonMapQuery

The jsonMapQuery sensor produces 10 messages for every tuple.  Set the batch size to 5 in the parser topology:

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "parserClassName": "org.apache.metron.parsers.json.JSONMapParser",
  "sensorTopic": "jsonMapQuery",
  "parserConfig": {
    "jsonpQuery": "$.foo",
    "batchSize": 5
  }
}' 'http://user:password@node1:8082/api/v1/sensor/parser/config/jsonMapQuery'

Writer a jsonMapQuery message:

cat /tmp/jsonMapExampleOutput | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic jsonMapQuery

This should cause the issue described in the Jira.  Before there was only a single message being written for each tuple.  With this PR there should be 10 messages in Elasticsearch and HDFS:
[root@node1 vagrant]# curl -s -XGET http://node1:9200/_cat/indices?h=index,docs.count | grep jsonmapquery
jsonmapquery_index_2019.02.07.22   10
[root@node1 vagrant]# hdfs dfs -cat /apps/metron/indexing/indexed/jsonMapQuery/*.json | wc -l
10

Change the batch size to 15:

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
  "parserClassName": "org.apache.metron.parsers.json.JSONMapParser",
  "sensorTopic": "jsonMapQuery",
  "parserConfig": {
    "jsonpQuery": "$.foo",
    "batchSize": 15
  }
}' 'http://user:password@node1:8082/api/v1/sensor/parser/config/jsonMapQuery'

Writer another jsonMapQuery message:

cat /tmp/jsonMapExampleOutput | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic jsonMapQuery

There should still be 10 messages in Elasticsearch and HDFS (no additional messages written):
[root@node1 vagrant]# curl -s -XGET http://node1:9200/_cat/indices?h=index,docs.count | grep jsonmapquery
jsonmapquery_index_2019.02.07.22   10
[root@node1 vagrant]# hdfs dfs -cat /apps/metron/indexing/indexed/jsonMapQuery/*.json | wc -l
10

Writer another jsonMapQuery message:

cat /tmp/jsonMapExampleOutput | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic jsonMapQuery

This should cause a batch to flush so we should see 15 additional messages:
[root@node1 vagrant]# curl -s -XGET http://node1:9200/_cat/indices?h=index,docs.count | grep jsonmapquery
jsonmapquery_index_2019.02.07.22   25
[root@node1 vagrant]# hdfs dfs -cat /apps/metron/indexing/indexed/jsonMapQuery/*.json | wc -l
25

Writer another jsonMapQuery message:

cat /tmp/jsonMapExampleOutput | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic jsonMapQuery

This should cause another batch to flush since there were 5 messages still in the batch.  We should see another 15 messages added:
[root@node1 vagrant]# curl -s -XGET http://node1:9200/_cat/indices?h=index,docs.count | grep jsonmapquery
jsonmapquery_index_2019.02.07.22   40
[root@node1 vagrant]# hdfs dfs -cat /apps/metron/indexing/indexed/jsonMapQuery/*.json | wc -l
40

The various topologies can be tested by adjusting their batch sizes and timeouts.  The previous instructions focus on testing batch sizes for bro in indexing, snort in indexing and jsonMapQuery in parsing.  Other topologies and scenarios can and should be tested with different batch sizes and timeouts.
Feedback Requested
Outside of the standard code review and function testing, they are a few areas that are not 100% clear and I would like feedback on:

Is the fundamental architectural approach solid?  Are there any holes I'm not thinking of?
The message ids generated by the bolts and used to track messages in the writer classes are Java UUIDs.  Is this good enough?  Any id that uniquely identifies a message could be used here.  Is there something that would perform better?
There is the potential for messages to pile up in StormBulkWriterResponseHandler (and probably other BulkWriterComponent too) when tuples time out and are replayed.  I think this is also an issue now but should we explore some kind of cache that can evict messages after timeouts?
I have added javadocs and am planning on taking another pass and adding more.  Are there areas you feel need more/better explanation?  My goal is for these classes to be easy to understand and navigate.  Is the interaction between the bolts and response handler clear?

Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:

[x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
[x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

[x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?

[x] Have you included steps or a guide to how the change may be verified and tested manually?

[x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh

[x] Have you written or updated unit tests and or integration tests to verify your changes?

[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

[x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:
cd site-book
mvn site

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

This looks really good.  I think that the way this works is important, maybe important enough to be in the architecture documentation.

@merrimanr Looked like the build failed.  Could you look into it when you have a chance?

The latest commit converts Map<String, MESSAGE_T> to List<BulkWriterMessage<MESSAGE_T>>  in the BulkMessageWriter.write method (where BulkWriterMessage contains and id and message).  This should make it clearer what the BulkMessageWriter interface is expecting.

really like that last change @merrimanr, nice

The latest commit implements @nickwallen's suggestion of refactoring the flush decision logic in BulkWriterComponent.write method into separate policies.  I think it was a great suggestion and I like it better than what we had.
I also added several javadocs to the various classes included in this PR.  I am still reviewing but if you see classes that need some javadoc love, just let me know.
Since this is a fairly significant change I tested everything again in full dev and things are still working.

The implementation of this seems pretty reasonable, from taking a look at things.  I didn't dig as deep as the other reviewers, but I'm good with where this is at code-wise.
@ottobackwards requested some architecture documentation and I didn't see any updates there. This feels like a good opportunity to get a core part of our architecture well documented and understood to make issues like this easier to understand in the future. Would you be able to write up some docs around this?

@justinleet yes was planning on circling back to that once we made it through the code review.  Now that this PR is in a good spot I'll start on that documentation.  Thanks.

If you want to put out a discuss for how to document this and like things and control scope, I would not be opposed to a follow on, as long as it get's done before next release.

Thanks @ottobackwards I appreciate that.  Would save me from the difficult task of keeping this big PR up to date with master.  I will start a discuss thread now on the topic.  If there are no objections there I will create a follow on Jira and commit to getting it done within a few days.

I think we need some way to document architectural concepts that is not as onerous as a full architecture document.  I can imagine almost an architectural FAQ type of thing:
Q. How to the Indexers batch writes?
A. index_batching.md

@ottobackwards just started a discuss thread on the architectural documentation topic.

Follow on Jira is here.

I'm +1, with the follow-on for documentation (and can you post the Jira here for posterity?)

https://issues.apache.org/jira/browse/METRON-2014

I am +1 as well. Thanks for working through this with us @merrimanr

Nice work @merrimanr, I am +1 pending agreement from @JonZeolla in the DISCUSS thread put out about documentation. Also, @merrimanr would you be able to link that thread to the Jira and tag it as required for our next release?

Here is the Jira from the DISCUSS thread:  https://issues.apache.org/jira/browse/METRON-2014

authors: justinleet merrimanr mmiklavc ottobackwards

repo: apache/metron


text:
Encrypted flash

This is done with a pseudo flash device, which provides hal_flash interface.
Device is split into common part, and architecture specific part. Only nrf51/nrf52 implementation is within this PR. So the current split might need adjusting when/if other platforms are added. Split is done so that you can take advantage of crypto acceleration when available.
Internally it uses AES-128 CTR mode where counter values comes from block offset.
At the moment the use cases are a bit limited; we'd need to augment hal_flash interface a bit to allow use of this device for things like FCB and image storage. Specifically, they'd need a routine with which to query the flash device whether underlying flash is empty or not.
As a example I modified one BSP to show how to set this up.

My useful comment of the day: it should have been called crypt_flash!

Thanks for taking a look, good points in all comments.

authors: mkiiskila utzig

repo: apache/mynewt-core


text:
Remove I2C0 initialization code from the EE-02 BSP.

This code assumed that pins 26 and 28 would be used for I2C, but this is not dictated by the EE-02 design.  It does not belong in the generic BSP.

Furthermore, prior to removing it herein, the call to hal_i2c_init seems to have had no effect.  It was still necessary to call hal_i2c_init from application code.  Out of curiosity, can this be explained?


text:
mesh: Fix and clean up iv update procedure

This patch fixes issue when receiving iv index greater than current
index + 42 in update mode. According to Specification when node is in update
state it should only accept iv index equal to the current iv index. When
node is in normal mode it should ignore index that is greater than current
index + 42.
This allows to pass MESH/NODE/IVU/BI-02-C.
Also this patch cleans up the iv update procedure, to make it easier to
read.

@jhedberg FYI

Updates according to @jhedberg's comments.


text:
MINIFI-235: Use the appropriate connection when requested_hostname_ i…

…s not localhost
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with MINIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit?

For code changes:

[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[x] If applicable, have you updated the LICENSE file?
[x] If applicable, have you updated the NOTICE file?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

reviewing

@phrocker looks good. Thank you for helping out with this issue!

repo: apache/nifi-minifi-cpp


text:
NIFI-1833 - Azure Storage processors

Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit?

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Initial PR submission, will inquire about checkstyle rules

Reviewing...

Reviewing after squash...

@aperepel Can you add usage documentation about the risks of allowing the account name and key to be stored as flowfile attributes?  Mentioning that sensitive attributes are not supported, and that anyone with provenance access for the processors will be able to see the attribute values.  Also, the flowfile repository, where the attributes are stored, does not support encryption internally, and unless the repository is being written to a TDE partition, the attribute values could possibly be accessed there, as well.

@aperepel Integration tests are written for the processors, but aren't included in the build.  There should probably be some tests that do get run with the build, using the TestRunner.  The CloudStorageAccount would have to be mocked, which at a minimum you could do by making an extension of the class under test in which the createStorageAccount method used by the test returns a mock.

authors: aperepel jtstorck

repo: apache/nifi


text:
NIFI-5667: Add nested record support for PutORC

The basic approach here is that I removed all references to Avro schemas/fields and replaced them with NiFi Record API concepts. This allows us to not have to switch back and forth, since Avro is not the de facto standard for schemas or flow file content (although we are still fairly tightly coupled to Avro schemas, but that's a different issue :) )
I'll comment in the PR on various parts of the code to note the "real" changes to fix the reported issue.
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit?

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Hi, I have been verifying the pull request and I came across this error for one of my Avro messages,
I am currently working to figure out which part of the message caused this.
2018-10-10 14:14:42,489 ERROR [Timer-Driven Process Thread-6] org.apache.nifi.processors.orc.PutORC PutORC[id=0430e7ab-99f1-3e25-c491-935245567fa3] Failed to write due to java.lang.ClassCa
stException: org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo: java.lang.ClassCastException: org.apache.hadoop
.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo
java.lang.ClassCastException: org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo cannot be cast to org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo
        at org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:127)
        at org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:177)
        at org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.lambda$convertToORCObject$0(NiFiOrcUtils.java:129)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.convertToORCObject(NiFiOrcUtils.java:130)
        at org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:73)
        at org.apache.nifi.processors.orc.record.ORCHDFSRecordWriter.write(ORCHDFSRecordWriter.java:94)
        at org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$null$0(AbstractPutHDFSRecord.java:324)
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2235)
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2203)
        at org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.lambda$onTrigger$1(AbstractPutHDFSRecord.java:305)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1662)
        at org.apache.nifi.processors.hadoop.AbstractPutHDFSRecord.onTrigger(AbstractPutHDFSRecord.java:272)
        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
        at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Can you share the schema of the data, and possibly a JSON export of your Avro file? I couldn't reproduce this with an array of ints, and @bbende ran successfully with an array of records.

Sorry for the late answer got bogged down.
Avro:
{
  "name": "IOlist",
  "namespace": "analytics.models.its",
  "type": "record",
  "fields": [
    {
      "name": "OItems",
      "type": [
        "null",
        {
          "type": "array",
          "items": {
            "name": "ISC",
            "namespace": "analytics.models.its.iolist.oitems",
            "type": "record",
            "fields": [
              {
                "name": "Od",
                "type": [
                  "null",
                  "long"
                ]
              },
              {
                "name": "HS",
                "type": [
                  "null",
                  "bytes"
                ]
              },
              {
                "name": "AS",
                "type": [
                  "null",
                  "bytes"
                ]
              },
              {
                "name": "NS",
                "type": [
                  "null",
                  "string"
                ]
              }
            ]
          }
        }
      ]
    }
  ]
}

JSON
{
  "OItems" : {
    "array" : [ {
      "Od" : {
        "long" : 99999
      },
      "HS" : {
        "bytes" : "/w=="
      },
      "AS" : {
        "bytes" : "AA=="
      },
      "NS" : {
        "string" : "0"
      }
    } ]
  }
}

@VikingK in your schema it has OItems defined as an array, but then in the JSON OItems is not an array, its an object with a field called array. So running with that schema and example JSON I get:

Caused by: java.lang.ClassCastException: org.codehaus.jackson.node.ObjectNode cannot be cast to org.codehaus.jackson.node.ArrayNode
	at org.apache.nifi.json.JsonTreeRowRecordReader.convertField(JsonTreeRowRecordReader.java:188)
	at org.apache.nifi.json.JsonTreeRowRecordReader.convertJsonNodeToRecord(JsonTreeRowRecordReader.java:118)
	at org.apache.nifi.json.JsonTreeRowRecordReader.convertJsonNodeToRecord(JsonTreeRowRecordReader.java:83)
	at org.apache.nifi.json.JsonTreeRowRecordReader.convertJsonNodeToRecord(JsonTreeRowRecordReader.java:74)
	at org.apache.nifi.json.AbstractJsonRowRecordReader.nextRecord(AbstractJsonRowRecordReader.java:92)

Which makes sense because the OItems field is not an array, but the schema says it is.
I'm trying to figure out how to reproduce the other error you showed.

@bbende wierd, I tried it again and I got the same error, here is the output from avro tools and I also attached my test.avro message
java -jar avro-tools-1.8.2.jar getschema test.avro
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
{
  "type" : "record",
  "name" : "IOlist",
  "namespace" : "analytics.models.its",
  "fields" : [ {
    "name" : "OItems",
    "type" : [ "null", {
      "type" : "array",
      "items" : {
        "type" : "record",
        "name" : "ISC",
        "namespace" : "analytics.models.its.iolist.oitems",
        "fields" : [ {
          "name" : "Od",
          "type" : [ "null", "long" ]
        }, {
          "name" : "HS",
          "type" : [ "null", "bytes" ]
        }, {
          "name" : "AS",
          "type" : [ "null", "bytes" ]
        }, {
          "name" : "NS",
          "type" : [ "null", "string" ]
        } ]
      }
    } ]
  } ]
}

java -jar avro-tools-1.8.2.jar tojson test.avro
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
{"OItems":{"array":[{"Od":{"long":99999},"HS":{"bytes":"/w=="},"AS":{"bytes":"AA=="},"NS":{"string":"0"}}]}}

test.zip

Ok using your test.avro going into PutORC with an AvroReader that uses embedded schema I can get the error you showed earlier.
I also tested ConvertRecord using the AvroReader and JsonWrtier and that works and produces the following JSON:
[
  {
    "OItems": [{
          "Od" : 99999,
          "HS" : [47,119,61,61],
          "AS" : [65,65,61,61],
          "NS" : "0"
        }]
   }
]

I found the issue and pushed a new commit with the fix. Also your test data exercised another part of the code I hadn't thought of, your "as" field is a keyword in Hive so when I used the generated hive.ddl attribute to create the table on top of the ORC files, it didn't work. The other change in this commit is to backtick-quote the field names to protect against field names that are reserved words.

@mattyb149 asome work, I was gonna suggest the backtick feature since its a pain when downstream systems sends all kinds of weird names, like Timestamp etc. My solution right now is an ugly groovy script that eliminates them.
I'll run some testing tomorrow on the fixes.

@mattyb149 I checked out the new backtick ddl patch you applied. Its seems it only backticks the complex data structures, the "top" level ones are for example not ticked.
For example ProductId,Items,PropertyMap,Serial and Metadata should be backticked as well to protect from bad names.
hive.ddl
CREATE EXTERNAL TABLE IF NOT EXISTS Sales (ProductId INT, Items ARRAY<STRUCT<`DNumber`:INT, `NSales`:BIGINT>>, PropertyMap MAP<STRING, STRING>, Serial STRUCT<`Serial`:BIGINT, `Date`:TIMESTAMP, `SystemId`:BIGINT, `T`:BIGINT, `IncludeStartTx`:BOOLEAN>, Metadata STRUCT<`AvroTs`:TIMESTAMP>) STORED AS ORC

Updated the rest of the backticks, thanks @VikingK and @bbende for your reviews!

Looks good to me, was able to verify the functionality, going to merge

authors: VikingK bbende mattyb149


text:
NIFI-6766: Escape policy label

NIFI-6766:

Ensuring policy label is properly escaped when populating the user's access policy listing.

+1 merged to master

authors: joewitt mcgilman


text:
NIFI-9871 Correct Component Stack Trace Logging

Description of PR
NIFI-9871 Corrects stack trace logging for extension components to avoid sending summarized stack trace causes to the SLF4J Logger. This change preserves the stack trace summary in bulletin messages and avoids writing duplicate information to application log files.
The ConnectableTask includes log wording adjustments to indicate the difference between termination, failure, and halted because of an uncaught exception.
The updated TestSimpleProcessLogger includes several new test methods to improve coverage and ensure consistent formatting.
The updated SimpleProcessLogger implementation of ComponentLog behaves as follows when logging an error:

Bulletin Messages contain a simple message with a summary of causes:

ListSFTP[id=d4ee5196-b059-3bab-4416-9f59a45cd7b0] Processing failed: org.apache.nifi.processors.standard.socket.ClientConnectException: SSH Client connection failed [192.168.1.1:22]
- Caused by: com.exceptionfactory.socketbroker.BrokeredConnectException: Proxy Address [/192.168.1.100:8888] connection failed
- Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)

Application Log Messages contain a simple message and full stack trace without duplicating the summary of causes:

2022-04-07 12:30:45,500 ERROR [Timer-Driven Process Thread-7] o.a.nifi.processors.standard.ListSFTP ListSFTP[id=d4ee5196-b059-3bab-4416-9f59a45cd7b0] Processing failed
org.apache.nifi.processors.standard.socket.ClientConnectException: SSH Client connection failed [192.168.1.1:22]
	at org.apache.nifi.processors.standard.ssh.StandardSSHClientProvider.getClient(StandardSSHClientProvider.java:115)
	at org.apache.nifi.processors.standard.util.SFTPTransfer.getSFTPClient(SFTPTransfer.java:598)
	at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:302)
	at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:264)
	at org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:120)
	at org.apache.nifi.processors.standard.ListSFTP.performListing(ListSFTP.java:151)
	at org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:112)
	at org.apache.nifi.processor.util.list.AbstractListProcessor.listByNoTracking(AbstractListProcessor.java:562)
	at org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:532)
	at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
	at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1283)
	at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
	at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
	at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.exceptionfactory.socketbroker.BrokeredConnectException: Proxy Address [/192.168.1.100:8888] connection failed
	at com.exceptionfactory.socketbroker.BrokeredSocket.connect(BrokeredSocket.java:100)
	at net.schmizz.sshj.SocketClient.connect(SocketClient.java:138)
	at org.apache.nifi.processors.standard.ssh.StandardSSHClientProvider.getClient(StandardSSHClientProvider.java:112)
	... 20 common frames omitted
Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:607)
	at com.exceptionfactory.socketbroker.BrokeredSocket.connect(BrokeredSocket.java:97)
	... 22 common frames omitted

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[X] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[X] Has your PR been rebased against the latest commit within the target branch (typically main)?

[X] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[X] Have you written or updated unit tests to verify your changes?
[X] Have you verified that the full build is successful on JDK 8?
[ ] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

I tested this by forcing a processor to generate an error. I used GenerateFlowFile to generate a FlowFile with the content "hello" and then sent to CompressContent. Configured CompressContent to decompress gzip. This should fail since the content is not gzipped, it's just the text "hello". But the error message written to the logs was:
2022-04-11 09:39:24,963 ERROR [Timer-Driven Process Thread-5] o.a.n.p.standard.CompressContent [CompressContent[id=33f515be-1000-1180-fa23-4e3424088fe0], decompress, StandardFlowFileRecord[uuid=770a869d-a200-4eb4-bd6d-093a75ad3fb8,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1649684358971-1, container=default, section=1], offset=0, length=58],offset=0,name=770a869d-a200-4eb4-bd6d-093a75ad3fb8,size=58], gzip] Unable to {} {} using {} compression format due to {}; routing to failure
org.apache.nifi.processor.exception.ProcessException: IOException thrown from CompressContent[id=33f515be-1000-1180-fa23-4e3424088fe0]: java.io.IOException: java.io.IOException: Input is not in the .gz format
	at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3437)
	at org.apache.nifi.processors.standard.CompressContent.onTrigger(CompressContent.java:282)
	at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
	at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1283)
	at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
	at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63)
	at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.io.IOException: Input is not in the .gz format
	at org.apache.nifi.processors.standard.CompressContent$1.process(CompressContent.java:373)
	at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3416)
	... 13 common frames omitted
Caused by: java.io.IOException: Input is not in the .gz format
	at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.init(GzipCompressorInputStream.java:193)
	at org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream.<init>(GzipCompressorInputStream.java:167)
	at org.apache.nifi.processors.standard.CompressContent$1.process(CompressContent.java:351)
	... 14 common frames omitted

So some variables didn't get substituted in:
Unable to {} {} using {} compression format due to {}; routing to failure

Thanks for the feedback @markap14! I pushed an update to correct the handling of Throwable references in Object[] parameters passed to log methods. I also updated the unit tests to increase of amount of argument verification.

Thanks @exceptionfactory changes look good. As do the logs. :) +1 merged to main


text:
fs/inode: combine multiple file list allocs into one to improve performance

Summary
fs/inode: combine multiple file list allocs into one to improve performance
combine multiple file list allocs into one to improve the task create performance
Signed-off-by: chao an anchao@lixiang.com
Impact
sim/nsh
Testing
ci-check

@anchao what is the impact of this change? Could you please show the output of toolchain-size command (i.e. arm-none-eabi-size) before and after this modification? And the "nsh> free" before and after this modification.
I think this modification is very good, but we need to see how it will impact small microcontrollers as well. Depending on results it is better to have an way to keep the original behavior.

@anchao what is the impact of this change? Could you please show the output of toolchain-size command (i.e. arm-none-eabi-size) before and after this modification? And the "nsh> free" before and after this modification.
I think this modification is very good, but we need to see how it will impact small microcontrollers as well. Depending on results it is better to have an way to keep the original behavior.

@acassis
I just measured the related performance of Task Termination. The focus in the process is mainly on the time consumption in interval B:

After applying this PR, the time consumption will be reduced ~2us (Tricore TC397 300MHZ):
15.97(us) -> 13.55(us)
But after testing about this change, I found that this PR has a regression issue. The file pointer held by the user will be invalid due to memory re-allocation after the file is expanded, so I will close this PR later.

authors: acassis anchao

repo: apache/nuttx


text:
sched: fix compile error when SCHED_CRITMONITOR is enabled

Summary
Compilation error occurs after SCHED_CRITMONITOR is enabled
sched/sched_critmonitor.c:315: undefined reference to `serr'
Impact
Testing

@Gary-Hobson it is a good idea to include the info from this Summary in the commit log message, this way people will see it in the git local repository.

authors: Gary-Hobson acassis


text:
Update GeoData_ID.xml

Hi,
I believe geoTypeId for "ID-YO" should be "REGION" instead of "REGIOIN"

That is correct! Typos can easily happen.
Please be aware that, while the project is moving towards Git, we're still in the phase of discussion, documenting and planning.
Best regards,

Fixed at 4ac9ae097e

authors: JacquesLeRoux PierreSmits RoyIrwan

repo: apache/ofbiz-framework


text:
[OODT-986] Add OODT NodeJS File Viewer Prototype

This PR links with the JIRA Issue OODT-986
This PR Contains a sample React based Prototype for OODT File Manager "File Object Viewer" functionality and a sample swagger implementation for API visualisation.

Thanks @NGimhana
Few things that stand out at first glance, can you rename react-webapp, react-components? Can you merge your .gitignore with the top level one?
Thanks
Tom

Sorry @NGimhana can you also stick AS2.0 License headers at the top of the files also please.

Thanks for reviewing @buggtb.
Added License headers and other suggestions to files.
Extremely Sorry for the unnecessary commits I made.

No problem @NGimhana we can squash it all at merge time anyway...
I just took a look at your .gitignore, can you check it works as it should? The paths look like it wouldn't look in the correct place.
Can you also do something about the output location of the tarball or whatever that stuff on line 134 is, because that clearly wont stand the test of time when it comes to versioning and needs to preferably end up in a target/ build/ or something directory and then get excluded by a wildcard on that directory.
Thanks!

"Tarball" is used to verify the package will include only the files you intend it to when published.
Run the npm pack command locally which will generate a tarball in the working directory, the same way it does for publishing. However that command will not execute automatically when we build the package. So I removed that verification from .gitignore.
However I suggest to run that command inside dist/. Then we can exclude "dist" directory when it comes to versioning.

LGTM, thanks!

authors: NGimhana buggtb

repo: apache/oodt


text:
fix(services/memory): MemoryConfig implement Debug

from: https://github.com/apache/opendal/issues/4941

Thanks for this change! Would you like to add Debug for Configurator to avoid this happen of other services?

sure!

repo: apache/opendal


text:
There was an SSL error when I deployed the environment

I execution instruction "ansible-playbook -i environments/local openwhisk.yml"
And  when to execute the task ——“TASK [cli : Download release archive to build directory ...] ”，
there is an SSLError ——“ An exception occurred during task execution. To see the full traceback, use -vvv. The error was: SSLError: ('The read operation timed out',)”
the details are as follows.
TASK [cli : Download release archive to build directory ...] ***********************************************************************************************************
Friday 17 April 2020  13:43:15 +0800 (0:00:00.261)       0:04:01.627 **********
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: SSLError: ('The read operation timed out',)
fatal: [172.17.0.1]: FAILED! => {"changed": false, "msg": "failed to create temporary content file: ('The read operation timed out',)"}
failed to create temporary content file: ('The read operation timed
out',)
PLAY RECAP *************************************************************************************************************************************************************
172.17.0.1                 : ok=11   changed=2    unreachable=0    failed=1
controller0                : ok=26   changed=3    unreachable=0    failed=0
invoker0                   : ok=29   changed=7    unreachable=0    failed=0
kafka0                     : ok=10   changed=4    unreachable=0    failed=0
Friday 17 April 2020  13:48:47 +0800 (0:05:31.380)       0:09:33.008 **********

The logs show

The read operation timed out

This might indicate the nginx container failed to start up or crashed. Check docker ps output to see which containers are up. How did you deploy your stack?

source: gharchive/issue

repo: apache/openwhisk


text:
HDDS-7359. Remove Ozone 0.5.0-beta download link from the website

https://issues.apache.org/jira/browse/HDDS-7359
Tested locally with hugo serve. Verified that 0.5.0-beta row is gone from download page with this change.

Thanks @smengcl for the patch, @ayushtkn for the review.

authors: adoroszlai smengcl

repo: apache/ozone-site


text:
HDDS-5033. SCM may not be able to know full port list of Datanode after Datanode is started.

What changes were proposed in this pull request?
When SCM HA is enabled, after restart DN, the SCM may not know the full ports of that DN.
The issue is:
SCMNodeManager just record the DatanodeDetails once during register. But for DN, it won’t record the admin, server, client port into DatanodeDetails until its ratis server is up. Thus there is contention here: if the register request is reported before ratis server is up, SCM won’t know full port list of that DN.
The solution is
If SCM HA is enabled, OzoneContainer#start() will be called multi-times from VersionEndpointTask. The first call should do the initializing job, the successive calls should wait until OzoneContainer is initialized.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5033
How was this patch tested?
CI and integration test inside tencent.

@bshashikant @nandakumar131 Please take a look at this bug fix of SCM HA. Thanks

@GlenGeng , can you explain why OzoneContainer#start() needs to be called multiple times if SCM HA is enabled?

@GlenGeng , can you explain why OzoneContainer#start() needs to be called multiple times if SCM HA is enabled for registering to the same SCM?

Please check
public class SCMConnectionManager
    implements Closeable, SCMConnectionManagerMXBean {
  private static final Logger LOG =
      LoggerFactory.getLogger(SCMConnectionManager.class);

  private final ReadWriteLock mapLock;
  private final Map<InetSocketAddress, EndpointStateMachine> scmMachines;

If SCM HA is enabled, there will be 3 VersionEndpointTask created, one for each SCM.

@GlenGeng , can you explain why OzoneContainer#start() needs to be called multiple times if SCM HA is enabled for registering to the same SCM?

Actually no, each call is for different SCM.

authors: GlenGeng bshashikant

repo: apache/ozone


text:
HDDS-5709. do not call removeTransactionsFromDB if nothing to remove

What changes were proposed in this pull request?
for now, blockingDeleting service will delete keys periodically and thus removeTransactionsFromDB will  be called periodically  , which is generate a raft log when scm ha is enabled.  if nothing to remove, a raft log will also be generated , which will delele nothing. this is not needed.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5709
How was this patch tested?
no need

LGTM, +1.

@ChenSammi @ayushtkn thanks for the reivew！


text:
HDDS-7519. Reserve Space for Allocated Blocks

What changes were proposed in this pull request?
changes to support reserve block at SCM with expiry and ICR with block commit

expiry block duration config
closer of container on in-sufficient size (EC already have this feature)
block expiry time set on allocate block and release of expired block on space check

TODO: ICR for blockCommit and release of block
Design doc attached as part of JIRA
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7519
How was this patch tested?
// IN-PROGRESS

@errose28

HDDS-8329 has fixed problem of write failure. Now problem of oversubscription may not be important but need look separately

authors: kerneltime sumitagrawl


text:
HDDS-7597. Improve ambiguous messages

What changes were proposed in this pull request?
Change error message when user does not have SCM superuser priviledge.
What is the link to the Apache JIRA
HDDS-7597
How was this patch tested?
Manual test.
How to recreate behavior described in ticket?

mvn clean install -DskipTests -DskipShade
cd hadoop-ozone/dist/target/ozone-*/compose/ozonesecure
docker-compose up -d --scale datanode=3
docker exec -it ozonesecure_om_1 /bin/bash
a. kinit -k testuser/om@EXAMPLE.COM -t /etc/security/keytabs/testuser.keytab
b. ozone admin container report

@neils-dev It seems you can't create an account anymore. Someone from ASF has to create one for you. Below quote is from the link @mladjan-gadzic proviced.

In November, 2022, due to an influx of false Jira accounts creating a flood of spam tickets, Infra ended public signups to ASF Jira accounts.


text:
HDDS-8627. Recon - API for Count of deletePending directories and amount of data mapped to such directories.

What changes were proposed in this pull request?
To retrieve the count of deletePending directories and the corresponding data size, we need to implement the DU space calculation logic. I am currently utilizing the NameSpaceSummaryTable to fetch the information regarding sub-directories and files linked to the DeletedDirectory.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8627
How was this patch tested?
Unit Testing

@devmadhuu @dombizita @ashishkumar50 can you please take a look !

@devmadhuu @adoroszlai @ashishkumar50
Could you please review this.

Initially, we attempted to implement the DU space calculation of deleted directories using NSSummaries. However, Sumit highlighted the difficulty in determining the total size of all deleted directories using our original approach. Currently, we iterate through the DeletedDirectoryTable, calculate the size of each deleted directory, and then find the sum total size. The issue arises when both the parent and child entries are encountered in the DeletedDirectoryTable, resulting in the same size being calculated twice.
To address the challenge of calculating the size of deleted directories, especially when encountering both parent and child entries in the DeletedDirectoryTable, we are adjusting our approach. To obtain the count of deletePending directories and their corresponding data size, we can derive the data size from the keys of the deleted directories. These keys are already being calculated and are an integral part of the deleted directories.

Please adjust the Jira (HDDS-8627) title and description to the changes discussed during the PR review @ArafatKhan2198 @sumitagrawl

Please adjust the Jira (HDDS-8627) title and description to the changes discussed during the PR review @ArafatKhan2198 @sumitagrawl

Thanks for pointing it out will do!


text:
HDDS-9660. NPE at OzoneManagerRatisServer.getRaftLeaderId

What changes were proposed in this pull request?
There is a possibility that first call of method returns non-null value but second time call can return null value, so the way check is done is wrong.
So first get value in local variable and use same for check null and use further.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9660
How was this patch tested?

impact verified with existing cases

@adoroszlai updated...

Thanks @sumitagrawl for the patch, @sadanand48 for the review.

authors: adoroszlai sumitagrawl


text:
HDDS-10072. Update Container interface for named RwLock operations

What changes were proposed in this pull request?
Container extends the RwLock interface from Hadoop, probably only for convenience.  Hadoop 3.4 adds two new methods in RwLock (HDFS-16434) without default implementation.  We need to

either implement these new methods,
or inline the methods used into Container.

No code in Ozone makes use of the RwLock abstraction, so I think we can go with (2).
Q: should we keep methods implemented but not used in Ozone? (readLockInterruptibly, writeLockInterruptibly, hasReadLock)
https://issues.apache.org/jira/browse/HDDS-10072
How was this patch tested?
CI:
https://github.com/adoroszlai/ozone/actions/runs/7426301421

Thanks @jojochuang for the review.


text:
HDDS-10217. Speed up TestKeyDeletingService

What changes were proposed in this pull request?
Speed up TestKeyDeletingService:
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 162.082 s - in org.apache.hadoop.ozone.om.service.TestKeyDeletingService

Start OM in per-class setup methods instead of per-test.  Tests are moved into two nested classes, because some of them test normal operations, others simulate failing block delete requests.  Most assertions had to be changed to not assume "empty" OM DB, similarly to #6107.
Reduce some service intervals.
Extract some helper methods.

https://issues.apache.org/jira/browse/HDDS-10217
How was this patch tested?
Total time is reduced to 40-45 seconds:
[INFO] Running org.apache.hadoop.ozone.om.service.TestKeyDeletingService
[INFO] Running org.apache.hadoop.ozone.om.service.TestKeyDeletingService$Failing
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.812 s - in org.apache.hadoop.ozone.om.service.TestKeyDeletingService$Failing
[INFO] Running org.apache.hadoop.ozone.om.service.TestKeyDeletingService$Normal
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.254 s - in org.apache.hadoop.ozone.om.service.TestKeyDeletingService$Normal
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.151 s - in org.apache.hadoop.ozone.om.service.TestKeyDeletingService

CI:
https://github.com/adoroszlai/ozone/actions/runs/7729212630
Passed in 200 runs:
https://github.com/adoroszlai/ozone/actions/runs/7755051816

Thanks @smengcl for the review.


text:
PDFBOX-4032: Handle correctly write special characters '\n' (LF), '\r…

Pull request for PDFBOX-4032

Looks good to me.

repo: apache/pdfbox


text:
Allow specifying custom metadata when uploading Google Cloud Storage files

The resumableUpload in the Google Cloud Storage connector allows specifying metadata of type  Map[String, String] which are added to the uploaded file.
As per the GCS documentation, metadata properties like contentType or cacheControl are predefined by GCS.
However, there is also custom metadata, which is arbitrary user-provided key-value pairs. They are sent in the HttpRequest in the same way, but the value is not a string but a json object:
{
    // metadata
    "contentType": "...",
    // custom metadata
    "metadata": {
        "customKey": "customData"
    }
}

However this cannot be modelled currently because metadata is Map[String, String] - the value needs to be specified as Map[String, String] instead of String to handle this.
(According to the documentatoin, GCS should also allow the notation metadata.customKey = customData but this did not work for me when I tried)

Yes, makes sense. I will check again if I can find out what's the issue.

repo: apache/pekko-connectors


text:
perf: Reduce loops in when clean queue in BroadcastHub (#1628)

(cherry picked from commit 9596ea4c1827701ab7f540897bba9f6209ceef15)

I don't think we should backport this. It is not semver friendly to be putting so many changes in patch releases.
We are taking too many risks in the patch releases as it is.

Ok, let's only get it in the next release.

@pjfanning Btw, this is a kind of bug too, because the loop can take seconds.

Okay, let me close it, anyway ,ship it in next version will be safer.

authors: He-Pin pjfanning

repo: apache/pekko


text:
Added option to set the method name for Basic authentication

Motivation
Added option to set the method name for Basic authentication

Do we need to extend the c_Authentication.cc?

Do we need to extend the c_Authentication.cc?

Yes, we can add it later. Just want to get this included in the release and python too.

authors: merlimat nodece

repo: apache/pulsar-client-cpp


text:
Authentication support for transaction perf

Is your enhancement request related to a problem? Please describe.
Currently, bin/pulsar-perf transaction does not support pass authentication params.
So, if the broker side enabled authentication, we are not able to use the transaction perf tool to test transactions.
Describe the solution you'd like
Support authentication for transaction perf tool which similar with perf producer and perf consumer does.
Make sure the transaction perf tool can apply the authentication configurations from client.conf

Hi, @codelipenghui
I think i can work on it, this feature can let me know authentication part.
Thanks ~

I am dealing with this issue.
And I found there may be more than that missing, I'll check and add it.

I am dealing with this issue. And I found there may be more than that missing, I'll check and add it.

Sure, Thanks for your notification.

repo: apache/pulsar


text:
Go client: consumer.Unsubscribe() places topic in unusable state.

Describe the bug
My unit tests use a static subscription name and topic name.  My test does a: create, publish, consume messages and then unsubscribe at the end.  This works the first time with no problem, but subsequent runs fail and the subscription is in an odd state.   I am able to clear the issue be running: pulsar-client consume --subscription-name my-sub  --num-messages 0 my-topic
Then the next iteration of my test succeeds... and we repeat the process all over again.
If you don't Unsubscribe() then the test always succeeds.
If you create a new subscription name every time, the test always succeed.
To Reproduce
Steps to reproduce the behavior:
I created a repo with my Go tests: https://github.com/Jim-Lambert-Bose/go-test-pulsar-consumer
To recreate:

clone repo
run go test -v multiple times.

Expected behavior
A clear and concise description of what you expected to happen.
I believe that you should be able to subscribe/unsubscribe as many times as you wish from the same static subscription name.  As long as you create a new subscriber, it should just work.
Desktop (please complete the following information):

OS: macOS Mojave v10.14.4

It seems like a question than a bug. @wolfstudy can you help check the issue here?

You can comment out the operation of seek() and continue to execute it.
	err = consumer.Seek(pulsar.EarliestMessage)
	assert.Nil(t, err)

It seems that the question has been answered. Closed the issue now. If there are any other questions, please create a new issue.

authors: Jim-Lambert-Bose sijie wolfstudy