text:
Streams 306

loosens criteria for ThroughputQueueSingleThreadTest
adopts Uninterruptibles.sleepUninterruptibly in
ThroughputQueueMulitThreadTest and BroadcastMonitorThread
(also includes STREAMS-305)

tests still failing

repo: apache/incubator-streams


text:
Request for a filter box with user-level filtering capabilities.

Imagine you've connected a data source to Superset, containing data on all your clients and their corresponding account managers. To help the account managers perform their jobs better, you decide to build a beautiful dashboard containing all the right charts and visuals. However, these account managers shouldn't be able to see each other's client data (sensitive, security, etc.)
If you only have a handful of account managers (AMs), it's easy to write a script to split the master data set into account manager-specific subsets before importing it to Superset, and then create tables from there. Problem solved.
What if you have hundreds of AMs? One way to go about this is create a single dashboard with slices associated to the master data set, add a filter box, select the AM you want to narrow the data down to, and then export the report as a static image. This has several drawbacks:

The dashboard the AM receives is a static image.
Someone has to do apply the filter and perform the export manually each time.
AMs have to wait for someone to send them the report each time they want it.

Now, if it were possible to adapt the existing filter box to auto-detect the Superset user and auto-apply a preset filter, then disable the dropdown menu from being changed, you could do away with those 3 drawbacks.
Of course there are millions of other ways to go about this, but thought I'd throw this out there.

If you have hundreds of AMs maybe you have an IT that can support providing separated views for the same data and find a way to build slices / dashboard programmatically?

Right; that could work, but maintaining a single view has 2 key benefits I can think of:

Less clutter. 100 AMs = 100 views = at least 100 slices = at least 100 dashboards (in this scenario where data separation is important).

Any updates or enhancements made to a master view or master dashboard would propagate to all AMs.

I'm wondering how hard could it be to add "virtual" slice / dashboard that you can connect to different datasources.

Sounds like row level security to me (other issues talk about this). Though the mechanisms for row-level-security would be around enforcing a WHERE clause for people on certain tables, as opposed to enforcing a filter on a dashboard.

hi,
Can you please, explain more about  row level security? Someone have already implement this?

Has there been any progress on this request?  I have tried to find the "recommended" approach for data level security but not sure if anything has been published. For instance, we use Druid+Superset and there are some requests for multi-tenant id but unsure if this works as expected.  I would think having a user level permission on a specific data source is a requirement for most organizations.  Also, on FilterBoxes, allowing a user to select specific dimension values based on Role/Group would be preferable.
In the meantime, how have others worked around this?  Is using a Slice or Datasource per user really feasible? How does the networking work? Proxy? Kubernetes INGRES?
Any thoughts on this are appreciated. While we use Superset internally, we cannot expand to a broader user base without this functionality.  If a User A is authorized to see a subset of a Druid Data source, how is this controlled exactly?
Thanks

@ahsanshah I hear you. I am facing a similar issue.
AIUI the current permission systems are about access to dashboards, charts and data-sources. That's a level above what we want (i.e we want to have granular permissions based access to the data within those charts).
Here is an initial thought on how maybe superset could be patched to make this work:

When creating users, have a set of arbitrary key/values that you can populate for that user. It could be things like 'user_id', 'department' etc. The point here is to add key/values that we would want to later filter on.
When creating Charts, where the 'Add Filter' is, have an 'Add User Filter' option. This might look the same as the regular Filter option, except the value would be a key that we had previously entered as the user. It might look something like storing a bindparam for later interpolation.
Table.get_sqla_query() would then build our query just like it does now, although in addition to a filter key, it could also look for user_filter (or whatever), and would do a getattr(flask.g.user, user_filter.val) or similar to get the user specific value that we are filtering against.

You'd obviously have to consider what to do if the User doesn't have the attribute, as well as other possible complexities I haven't thought of.
If those who know Superset better than I think this is a reasonable approach I'd be happy to get on and write a patch, because as much as I'd like to use Superset, without this kind of feature it's of limited use to me :(

Also, here is a way you can implement your own Dashboard level filtering without touching the Superset codebase. Roughly speaking you need to do the following:

Create a custom view and override the /superset/explore_json endpoint
Modify the Viz query_obj() method to include any additional filtering you want.
Import the custom view via the FLASK_APP_MUTATOR.

It looks something like this.
from flask import g
from flask_appbuilder import expose
from superset import appbuilder
from superset.views.core import Superset, log_this, has_access_api

class MyCustomView(Superset):

    @log_this
    @has_access_api
    @expose('/explore_json/<datasource_type>/<datasource_id>/', methods=['GET', 'POST'])
    @expose('/explore_json/', methods=['GET', 'POST'])
    def explore_json(self, *args, **kwargs):
        """ This is a copy of the method definition from Superset """
        return super(MyCustomView, self).explore_json(*args, **kwargs)

    def get_viz(self, *args, **kwargs):
        """ Return the Vizualisation object

        get_viz() overrides the Superset.get_viz() method
        so it can wrap it and alter the filter pattern
        that it produces. This is so we can filter
        viz objects by whatever we want.
        """
        viz_obj = super(DAView, self).get_viz(*args, **kwargs)
        # The viz_obj has a query_obj() method which will be called
        # and filters added before being fetched from the DB.
        # We want to wrap this function, so when called we add our own
        # filtering
         def wrapped(f):
            def inner(*args, **kwargs):
                query_obj = f(*args, **kwargs)
                # If we want here we can check the dashboard first
                dashboard = urllib.urlparse(request.headers['referer']).path
                if dashboard == '/superset/dashboard/dashboardX'/:
                    # Then filter by username
                    query_obj['filter'].append({'val': f'{g.user.username}', 'op': '==', 'col': 'username'})
                return query_obj
            return inner
        viz_obj.query_obj = wrapped(viz_obj.query_obj)
        return viz_obj

# Here we are making sure that any calls to explore_json()
# will call our custom view, thus giving us control of the response
for rule in appbuilder.app.url_map.iter_rules():
    if rule.rule == '/superset/explore_json/':
        rule.endpoint = 'MyCustomView.explore_json'

appbuilder.app.url_map.update()
appbuilder.add_view_no_menu(DAView)

Then in your superset config.py file add the following:

def import_stuff(app):
    from my_app import views
FLASK_APP_MUTATOR = import_stuff

Here the views module has our previosuly defined MyCustomView class etc.
I tried this on 0.26 and it seemed to work fine. This will get called for every chart AFAICT, so you might have to be clever about what filters you are applying for what charts etc. The above is just a basic example.

This request is definitely a must in my opinion. Like @anthonymhho said, the solution the way it is right now is not scalable to different level  of data access which is general the case in many organisation.
Any status on this?

source: gharchive/issue

authors: ahsanshah amoussoubaruch anthonymhho kalium99 kwims mistercrunch xrmx

repo: apache/incubator-superset

license_type: permissive

license_source: github-api


text:
fix: choose language link for local dev

CATEGORY

[x] Bug Fix

SUMMARY
@superset-ui/connection will add protocol and host to an "endpoint" call. e.currentTarget.href may return the full URL instead of the relative url. This sometimes results in invalid URLs for change language API calls.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
In local dev environment, change language will result in a 404 error:

The result should be gone after this fix.
TEST PLAN
Change language in local dev box will work and should not throw an error in the browser console.
ADDITIONAL INFORMATION
N/A
REVIEWERS
@kristw

Codecov Report

Merging #9215 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #9215   +/-   ##
=======================================
  Coverage   58.91%   58.91%
=======================================
  Files         372      372
  Lines       11996    11996
  Branches     2937     2937
=======================================
  Hits         7068     7068
  Misses       4750     4750
  Partials      178      178

Impacted Files
Coverage Δ

superset-frontend/src/setup/setupApp.ts
0% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ca2bc8b...d29e8a3. Read the comment docs.

@rusackas this looks like a nice fix, might be up your alley for a review

source: gharchive/pull-request

authors: codecov-io ktmud willbarrett


text:
[SYSTEMML-604] Undefined function error message line, col, filename

Previously, the following DML:
print("howdy");
a=foo();

produced error message with null filename and no line number or column number set:
Caused by: org.apache.sysml.parser.LanguageException: ERROR: null -- line 0, column 0 -- function foo is undefined in namespace .defaultNS
	at org.apache.sysml.parser.StatementBlock.isMergeableFunctionCallBlock(StatementBlock.java:201)

This update populates the filename, line number, and column number:
Caused by: org.apache.sysml.parser.LanguageException: ERROR: example.dml -- line 2, column 0 -- function foo is undefined in namespace .defaultNS
	at org.apache.sysml.parser.StatementBlock.isMergeableFunctionCallBlock(StatementBlock.java:208)

@mboehm7 Is this the proper way to set this information?

Although it works, I would expect the setup of positions and filenames during initial parsing (e.g., DmlSyntacticValidator) for atomic expressions and the propagation for composite expressions either in the syntactic parser or during validate. Also please use the one-liner 'setAllPositions'. Thanks @deroneriksson

@mboehm7 Moved to CommonSyntacticValidator. Let me know if this is satisfactory.

LGTM, but it would be great to double check if this fix also applies to external functions or if additional extensions are required.

Not sure if I entirely understand WRT external functions, but if I try to cause an issue with a function defined with externalFunction (use wrong number of args, have the java function throw an exception, etc), typically an error such as "Runtime error in program block generated from statement block between lines 7 and 7" is generated.  Let me know if you'd like me to handle a specific external function situation. I will go ahead and merge this PR.

that's fine, please go ahead and merge the PR.

repo: apache/incubator-systemml

license_source: bigquery


text:
Adding tenancy to the database

WIP
On this PR the tenancy is only added to the relavant tables. The already existing APIs were not modified.
Default tenancy of items is "NULL".
API for tenancy was added, including UT.
Additionally, an "upgrade" script was added to create the root tenant and assign it to the specified admin user.

Closing the PR as the content would be added by another PR

repo: apache/incubator-trafficcontrol


text:
[Relay][Fix] i64 support

Previously GetShape produces Tensor with i32 shape even with relay.var('a', shape=(tvm.tir.const(1, "int64"))).  This pr modifies this behavior so it repects user's input.

@hzfan can you followup?

@tqchen ok, I will follow up.

CI's green. @tqchen @kazum

cc @mbrookhart @jroesch @junrushao1994 @ZihengJiang @antinucleon

@jwfromm please take another look and approve explicitly if everything looks good to you.

LGTM

Thanks @hzfan @jwfromm @tqchen @kazum

authors: hzfan jwfromm tqchen zhiics

repo: apache/incubator-tvm


text:
[BYOC] FTVMAnnotateTarget method signature update

This PR changes signature of FTVMAnnotateTarget to runtime::TypedPackedFunc<bool(const Expr& expr)>
which allows to utilise extra information (e.g. an operation output type) from passed expr argument.

tensorrt.py changes look good

@mbaret PTAL and approve and request changes explicitly. Thanks.

Thanks @d-smirnov @mbaret @lhutton1 @trevor-m

authors: comaniac d-smirnov trevor-m


text:
[Improvement] Shade Netty Native lib

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

What would you like to be improved?
Our Netty only shade jar package, never shade netty package, If we use multiple version Netty, our client may be influenced.
How should we improve?
We can refer to
https://github.com/apache/spark/pull/24502
https://github.com/apache/incubator-celeborn/pull/908
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Hi @jerqi
Do we still have this issue?  I'd like to work on it if yes.

Hi @jerqi
Do we still have this issue? I'd like to work on it if yes.

Yes. You can go ahead.

repo: apache/incubator-uniffle


text:
[iOS] -[WXListComponent _insertSubcomponent:atIndex:] crash

-[WXListComponent _insertSubcomponent:atIndex:] crash
-[__NSArrayM insertObject:atIndex:]: index 2 beyond bounds [0 .. 0]

Please provide more information, there is nothing we can do with such little infomation.

authors: YorkShen zhanglinfeng

repo: apache/incubator-weex


text:
README typo, build profile addition and version bump

Added interpreter/ folder to .gitignore
README.md
Fixed a typo in README.md
Updated mvn build command for Hadoop 2.7
In spark/pom.xml:
Bumped Spark default version number from 1.4.0 to 1.4.1
Added -PHadoop-2.7 build profile

+1

Looks good to me!

LGTM!

authors: Leemoonsoo felixcheung jongyoul randerzander

repo: apache/incubator-zeppelin


text:
[Bug][DataProxy] DataProxy is not sending data to Pulsar

What happened
dp is not sending data to pulsar.

What you expected to happen
There is a record of data consumption in pulsar, and the last data generated in clickhouse.
How to reproduce
When I finished the approval and kept adding data to the local source file, no data was generated in clickhouse. Finally, after finding the problem, I found that there was no data in pulsar.

Environment
centos 7
InLong version
master
InLong Component
InLong DataProxy
Are you willing to submit PR?

[X] Yes, I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Message has been sent to dp? and  If the message has been sent to dp, please confirm whether the expected topic configuration information is contained in  conf/topics.properties,

In addition, please check whether the log contains the following information。

@baomingyu

There is this error in inlong-dataproxy/logs/.

Please check if this topic exists in the pulsar cluster. and check the initial configuration steps of the cluster. If the initial configuration information of the cluster in the Manager is incorrect, the topic may not be created in advance.

this issue may be the same with https://github.com/apache/inlong/issues/5163, and we will look into it.

fixed in https://github.com/apache/inlong/pull/5189. Please check again.

When I use the latest code, I still can't get the data to pulsar, and new errors occurs.

这个问题你是否已经解决？我碰到了同样的问题，束手无策。1.4.0版本

@ly100120 please add the steps to reproduce the issue.

@ly100120 please try the latest code, and create a new Issue if it appears again.

authors: baomingyu dadalongvs dockerzhang ly100120

repo: apache/inlong


text:
Remove the usage of Validate.isTrue method to avoid unnecessary construction of string

Description
Avoid the usage of method Validate.isTrue.

coverage: 44.522% (+0.03%) from 44.492% when pulling 8ecf6f8953689ed49f0cec70991db5a316424917 on beyyes/imporve_statistic_judgement into 0fa2300e5825aabd1c3716ef0e48ad6e66b3311a on master.

authors: Beyyes coveralls

repo: apache/iotdb


text:
a test

Description
Content1 ...
Content2 ...
Content3 ...

This PR has:

[ ] been self-reviewed.

[ ] concurrent read
[ ] concurrent write
[ ] concurrent read and write

[ ] added documentation for new or modified features or behaviors.
[ ] added Javadocs for most classes and all non-trivial methods.
[ ] added or updated version, license, or notice information
[ ] added comments explaining the "why" and the intent of the code wherever would not be obvious
for an unfamiliar reader.
[ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold
for code coverage.
[ ] added integration tests.
[ ] been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

coverage: 48.059% (-0.02%) from 48.083% when pulling 0e1eb5ed30a39a5d09b81355ea3e9edd1e0f8f4e on beyyes/schema_region_per_data_node into 1f321df57b7b535b2055d0e0237fd474a26aa39e on master.


text:
[IOTDB-1836] Fix Query Exception Bug after deleting all sgs

Description
[IOTDB-1836]
Reasons
Delete storage group didn't clear cache, so the cache still contains the stale data. However, the files has already been deleted, so while querying, File not Found error occurs.

Coverage decreased (-0.02%) to 67.704% when pulling e95fbeb64eacfbe501547d3c72025c4748d7ce8b on DeleteBugM into cbd1317d194c81c0bc4818c7748d377d62d0a13a on master.


text:
[ISIS-2117] Replace instances of Math.random with Random.nextDouble

ISIS-2117 Replace Math.random with Random.nextDouble. There is a performance overhead cost when using Math.random over Random.nextDouble. Even moreso, the Random class can allow for easier random manipulation in the future if needed.

I can see 2 enhancements to this class

let 'random' be a final field
use Random.nextInt(int bound) instead, to improve code readability

I'm using this inspiration to update the class in the 'v2' development branch. Thx!
    private static final int NUMBER_CHARACTERS = 10;
    private static final String CHARACTERS = "ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890";

    private final Random random = new Random();

    @Override
    public String generateRandomCode() {
        final StringBuilder buf = new StringBuilder(NUMBER_CHARACTERS);
        for (int i = 0; i < NUMBER_CHARACTERS; i++) {
            final int pos = random.nextInt(CHARACTERS.length());
            buf.append(CHARACTERS.charAt(pos));
        }
        return buf.toString();
    }

Optimization applied to '2112_Spring' branch, which is designated to be merged into the final 2.0.0 release.

repo: apache/isis


text:
Updated checkstyle and added MissingDeprecated check, also formatted

Description
Added extra checkstyle rule and updated to 8.23 (8.24 is the current latest but it causes a lot more errors, especially usage of < and > in JavaDoc).
Types of changes

Bug fix (non-breaking change which fixes an issue)

@ham1 , have you seen https://github.com/diffplug/spotless?
I guess we might want to move some of checkstyle rules to spotless since the latter enables ./gradlew spotlessApply to automatically apply the fixes like "import order", "spacing", "wrong file header", etc, etc.

I had not seen that before. Auto-fixing of spacing, import order and header etc. sounds good!

Opps I missed the checksums.

@ham1 , I see you contribute a lot of cleanup, and that is nice.
Re Spotless, it might look as follows:  https://travis-ci.org/vlsi/vlsi-release-plugins/jobs/584025923#L410
That is way better than Checkstyle.

I agree it's nice because fixes can be auto applied, but they can't be applied in CI, and it doesn't seem to have as many rules, so I do not yet see how it is much better than Checkstyle, maybe I've missed something? Maybe a topic for the mailing list?

Both tools can't autofix code in CI
Spotless can autofix code in dev machine
Spotless prints much more meaningful messages.

=> Spotless is better

Adding final all over the place might easily break backward compatibility.
Is there a good reason for retrofitting finals all over the place?

Good point. I believe the rule is there for those classes who cannot be extended anyway because they have private default constructors and therefore either can't be or probably shouldn't be extended.
From the docs:
Checks that a class which has only private constructors is declared as final. Doesn't check for classes nested in interfaces or annotations, as they are always final there.

Codecov Report

Merging #494 into master will increase coverage by <.01%.
The diff coverage is 70%.

@@             Coverage Diff              @@
##             master     #494      +/-   ##
============================================
+ Coverage     54.84%   54.84%   +<.01%
  Complexity     9800     9800
============================================
  Files          1018     1018
  Lines         62734    62731       -3
  Branches       7058     7059       +1
============================================
+ Hits          34404    34405       +1
+ Misses        25900    25897       -3
+ Partials       2430     2429       -1

Impacted Files
Coverage Δ
Complexity Δ

...ter/protocol/http/parser/JsoupBasedHtmlParser.java
92.06% <ø> (ø)
2 <0> (ø)
:arrow_down:

...che/jmeter/protocol/tcp/sampler/TCPClientImpl.java
46% <ø> (ø)
5 <0> (ø)
:arrow_down:

...in/java/org/apache/jmeter/timers/TimerService.java
92.85% <ø> (ø)
8 <0> (ø)
:arrow_down:

...java/org/apache/jmeter/util/JSR223TestElement.java
53.96% <ø> (ø)
19 <0> (ø)
:arrow_down:

.../jmeter/extractor/json/jmespath/JMESPathCache.java
86.66% <ø> (ø)
3 <0> (ø)
:arrow_down:

...va/org/apache/jmeter/gui/util/JSyntaxTextArea.java
47.82% <ø> (+1.34%)
8 <0> (ø)
:arrow_down:

.../main/java/org/apache/jmeter/util/JMeterUtils.java
47.88% <ø> (ø)
69 <0> (ø)
:arrow_down:

...org/apache/jmeter/samplers/AsynchSampleSender.java
37.5% <ø> (ø)
5 <0> (ø)
:arrow_down:

...r/protocol/http/parser/LagartoBasedHtmlParser.java
83.67% <ø> (ø)
4 <0> (ø)
:arrow_down:

.../org/apache/jorphan/logging/Slf4jLogkitLogger.java
4.76% <ø> (ø)
1 <0> (ø)
:arrow_down:

... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6dfd0d4...92a85e3. Read the comment docs.

authors: codecov-io ham1 vlsi

repo: apache/jmeter


text:
KAFKA-15492: Enable spotbugs when building with Java 21

spotbugs 4.8.0 has a number of new warnings that are enabled by default:

CT_CONSTRUCTOR_THROW: disable due to too many false positives
PI_DO_NOT_REUSE_PUBLIC_IDENTIFIERS_*: disable due to too many false positives
PA_PUBLIC_MUTABLE_OBJECT_ATTRIBUTE: disable for Scala and for a few Streams
classes, fix other instances.
RV_EXCEPTION_NOT_THROWN: disable due to too many false positives.
SS_SHOULD_BE_STATIC: disable for jmh code, fix other instances.
DM_DEFAULT_ENCODING: fix every case.

Changelog: https://github.com/spotbugs/spotbugs/blob/4.8.0/CHANGELOG.md
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Looks like the new version is not in the Maven repo yet. Will remove draft once that's resolved.

Thanks for the review. In general, we have assumed UTF-8 since that's the default for Linux and MacOS. It has become the default for all platforms since Java 18: https://openjdk.org/jeps/400

Build is successful. Unrelated test failures:
[Build / JDK 21 and Scala 2.13 / org.apache.kafka.streams.integration.NamedTopologyIntegrationTest.shouldAddNamedTopologyToRunningApplicationWithSingleInitialNamedTopology()](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14533/4/testReport/junit/org.apache.kafka.streams.integration/NamedTopologyIntegrationTest/Build___JDK_21_and_Scala_2_13___shouldAddNamedTopologyToRunningApplicationWithSingleInitialNamedTopology__/)
[Build / JDK 21 and Scala 2.13 / org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskRequestWithOldStartMsGetsUpdated()](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14533/4/testReport/junit/org.apache.kafka.trogdor.coordinator/CoordinatorTest/Build___JDK_21_and_Scala_2_13___testTaskRequestWithOldStartMsGetsUpdated__/)
[Build / JDK 17 and Scala 2.13 / org.apache.kafka.connect.integration.ConnectorRestartApiIntegrationTest.testMultiWorkerRestartOnlyConnector](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14533/4/testReport/junit/org.apache.kafka.connect.integration/ConnectorRestartApiIntegrationTest/Build___JDK_17_and_Scala_2_13___testMultiWorkerRestartOnlyConnector/)
[Build / JDK 17 and Scala 2.13 / kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest.testCreateTokenForOtherUserFails(String).quorum=kraft](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14533/4/testReport/junit/kafka.api/DelegationTokenEndToEndAuthorizationWithOwnerTest/Build___JDK_17_and_Scala_2_13___testCreateTokenForOtherUserFails_String__quorum_kraft/)
[Build / JDK 17 and Scala 2.13 / kafka.api.SaslScramSslEndToEndAuthorizationTest.testAuthentications(String).quorum=zk](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14533/4/testReport/junit/kafka.api/SaslScramSslEndToEndAuthorizationTest/Build___JDK_17_and_Scala_2_13___testAuthentications_String__quorum_zk/)
[Build / JDK 17 and Scala 2.13 / kafka.server.DescribeClusterRequestTest.testDescribeClusterRequestIncludingClusterAuthorizedOperations(String).quorum=kraft](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14533/4/testReport/junit/kafka.server/DescribeClusterRequestTest/Build___JDK_17_and_Scala_2_13___testDescribeClusterRequestIncludingClusterAuthorizedOperations_String__quorum_kraft/)
[Build / JDK 17 and Scala 2.13 / org.apache.kafka.tiered.storage.integration.TransactionsWithTieredStoreTest.testBumpTransactionalEpoch(String).quorum=kraft](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14533/4/testReport/junit/org.apache.kafka.tiered.storage.integration/TransactionsWithTieredStoreTest/Build___JDK_17_and_Scala_2_13___testBumpTransactionalEpoch_String__quorum_kraft/)
[Build / JDK 17 and Scala 2.13 / org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskRequestWithOldStartMsGetsUpdated()](https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14533/4/testReport/junit/org.apache.kafka.trogdor.coordinator/CoordinatorTest/Build___JDK_17_and_Scala_2_13___testTaskRequestWithOldStartMsGetsUpdated__/)

authors: divijvaidya ijuma

repo: apache/kafka


text:
KAFKA-17642: PreVote response handling and ProspectiveState

KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-996%3A+Pre-Vote
Jira: https://issues.apache.org/jira/browse/KAFKA-16164
Implements items 2-4 which cover response handling and new ProspectiveState:

https://issues.apache.org/jira/browse/KAFKA-17642
https://issues.apache.org/jira/browse/KAFKA-17643
https://issues.apache.org/jira/browse/KAFKA-16607

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

It makes sense to me that after the resign state the replica should always increase its epoch. The replica resigned from leadership at epoch X so eventually the epoch will be at least X + 1. Did you consider transitioning to candidate and relaxing the transition functions to allow both resigned and prospective to transition to candidate?

yes, I decided not to list that as an option because I felt it was equal to if not worse than the option of having resigned transition to prospective in epoch X + 1. personally I felt it was nicer to have less edge cases to the invariant that only prospective should transition to candidate


text:
KAFKA-4269 extracted code updating topics when regex pattern specifie…

…d out of topicGroups method. The topicGroups method only called from StreamPartitionAssignor when KafkaStreams object  is the leader, needs to be executed for clients.

@guozhangwang  @mjsax @dguy
Please review when you have a chance.
The previous change for multiple consumers missed extracting a block of code in the topicGroups() method only called for the leader KafkaStreams instance.

@dguy
Added unit test for method

A few minor comments otherwise LGTM. @bbejeck could you rebase?

@guozhangwang
rebased and updated per comments

Merged to trunk.

@bbejeck Could you file a follow-up PR for 0.10.1 branch just in case we have a bug-fix release later?

@guozhangwang - Sure thing.  I attempted opening the PR against the 0.10.1 branch but it looked like there were a lot of conflicts, is there another approach ?

You can try 1) checkout 0.10.1 first and checkout a new branch from there, and 2) use git cherry-pick to pull in that commit from trunk, 3) upon showing conflicts, fixing them forward, 3) git commit to create a new PR.

authors: bbejeck guozhangwang


text:
MINOR: clean stateDirectory in TopologyTestDriver

Specifically, on TopologyTestDriver#close()
Allow the test driver to clean up any persistant files that may
have been written by a state store in the topology.
Add a test verifying that persistent state does get cleaned up.

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@mjsax @guozhangwang @bbejeck ,
I'd appreciate any feedback you have on this.

just one minor nit, otherwise LGTM. Thanks for the patch @vvcephei!

I went ahead and fixed some other things that could be final and also stopped using double-brace initialization for Properties.

Ok, @mjsax and @guozhangwang , what do you think about this version?

Merged to trunk and cherry-picked to 1.1.
Thanks for the patch and congrats to your first contribution @vvcephei !

authors: bbejeck mjsax vvcephei


text:
stop wasting CPU copying java collections in scala

Signed-off-by: radai-rosenblatt radai.rosenblatt@gmail.com
a profiler run on one of our brokers under load showed that ~11% of the global cpu time for the entire broker was taken by these 3 .map() calls:
5.36% in replicas.asScala.map(.toInt) (line 79)
2.31% in offlineReplicas.asScala.map(.toInt) (line 81)
3.58% in isr.asScala.map(_.toInt) (line 97)
for a total of 11.25%

this is also expected to have a positive impact on GC (less garbage produced in copied Buffer[Int])

One more thing, let's add a comment explaining this or else someone will revert the change.

@ijuma - commit edited to explain the issue a little better.

@ijuma - i've added a comment explaining why the method signature for getEndpoints() retains the java types.

authors: ijuma radai-rosenblatt


text:
Test2

More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3862/
Test FAILed (JDK 7 and Scala 2.10).


text:
KAFKA-9410: Make groupId Optional in KafkaConsumer

More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

ok to test

retest this please

authors: belugabehr hachikuji


text:
KAFKA-8245 Flaky Test DeleteConsumerGroupsTest#testDeleteCmdAllGroups

the root cause is the “shutdown” may obstruct consumer to complete the group join. If the consumer wakes up before completing the partition assignments, the consumer is in a state of UNJOINED (so it won’t send LeaveGroupRequest when closing) and the broker is in a state of CompletingRebalance until timeout.
To fix the flaky, we should wait all groups are in Stable state to make sure all Consumers have completed the group joining (so they are able to send LeaveGroupRequest when closing).
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

looped the test 50 times, all pass

Retest this please.

the flaky is traced by https://issues.apache.org/jira/browse/KAFKA-7965
ping for reviews

Call for review @bob-barrett @cmccabe

retest this please

If these tests are present in 2.4 and 2.5, I suggest cherry picking to those branches too.

@vahidhashemian thanks for merging

Merged to 2.4 and 2.5 branches too.

authors: chia7712 ijuma mjsax vahidhashemian


text:
KAFKA-10227: Enforce cleanup policy to only contain compact or delete once

https://issues.apache.org/jira/browse/KAFKA-10227
This patch fixes both TopicCommand and ConfigCommand. When duplicate items are specified in cleanup.policy, they will be removed automatically and then applied.
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Seems it failed by an unrelated issue tracked in KAFKA-10017 .

@huxihx Thanks for the PR. Have you considered doing this in the broker? Doing it in the broker has the advantage that it would work for all the clients in the ecosystem.

@dajac Thank you for the comment. Well, it is quite tempting to put all the check things on the broker side. However, we should still have the current tools refined to be robust enough. TopicCommand and ConfigCommand are the most popular tools to add configs for topics. They are definitely deserved to be optimized to build up the basic fault tolerance. Maybe we could open a new PR to do something great on the broker side later on. What do you think?

@huxihx I agree that improving TopicCommand and ConfigCommand is a good thing. I think that we have to be cautious with having generic validations in the clients though as it could bit us in the future. By doing the validation in the broker, we also improve them indirectly ;).

@huxihx As @dajac mentioned, I also envisioned doing this check on the broker side.
I actually initially hit this issue with a 3rd party client and just included the kafka-topics command in the ticket to easily reproduce it.

@mimaison @dajac If that's the case, I agree to move the logic into broker side. However, I suggest we keep the current refinement in TopicCommand and ConfigCommand since these tools should be capable of doing some pre-processing. What do you think？

@huxihx I am not against improving the command line but I think that the approach that you are proposing in this PR is a bit risky for the future. Please, take a look at my previous comment and let me know what you think.

@dajac @mimaison  Thanks for the comments. Please review again.

Ping @omkreddy @mimaison for review.

@abbccdda Hi, do you have any time to review this patch? It's been quite a long time since no one had a look at it :(

@huxihx can you rebase your PR? Thanks

authors: dajac huxihx mimaison


text:
Happy New Year 2024

:mag: Description
Issue References 🔗
Update the NOTICE files for coming 2024.
Types of changes :bookmark:

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪
Review

Checklist 📝

[x] This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (5d59cf1) 61.10% compared to head (25e85e5) 61.14%.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #5933      +/-   ##
============================================
+ Coverage     61.10%   61.14%   +0.04%
  Complexity       23       23
============================================
  Files           621      621
  Lines         36864    36864
  Branches       5014     5014
============================================
+ Hits          22527    22542      +15
+ Misses        11895    11879      -16
- Partials       2442     2443       +1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Thanks, merged to all active branches

repo: apache/kyuubi


text:
[WIP] Migrate from setup.py to pyproject.toml

This pull request is a work in progress attempt at trying to migrate package and build metadata from setup.py to pyproject.toml.
It will take some time to polish out all the kinks to ensure produced sdist and wheel artifacts match what would have been produced by the previous setup.oy config.
As part of this change, running tests directly via python setup.py test will be deprecated and removed (https://docs.pytest.org/en/7.2.x/explanation/goodpractices.html#do-not-run-via-setuptools) -  I will document that in the changelog and update the affected readme.
TODO

[ ] Verify new .tar.gz and wheel artifacts match what would be generated with setup.py in the past
[ ] Update dist.sh script to utilize build package (https://github.com/pypa/build)
[ ] Research potentially using hatchling instead of setuptools for the build backend (may want to avoid this for now to keep the scope of the changes smaller)
[ ] Update development docs (release management section)
[ ] To reduce build artifact sizes, possibly consider excluding tests + test fixture files again (we've did that a long time ago and then changed to including tests with the distribution artifact - so the end user can run the tests on the downloaded artifact and verify the artifact is in order)

Codecov Report

Merging #1929 (9d15c48) into trunk (e26f012) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##            trunk    #1929   +/-   ##
=======================================
  Coverage   83.12%   83.12%
=======================================
  Files         353      353
  Lines       81365    81365
  Branches     8598     8598
=======================================
  Hits        67630    67630
  Misses      10931    10931
  Partials     2804     2804

^ @rsnk96 Just a heads up since you made a similar change recently by moving most of the tooling config into pyproject.toml.
This change is still very much work in progress, but I just wanted to give you a heads up.

I checked and tests the artifacts and they appear to be looking and working fine.
For the time being, I will leave simple mainimal setup.py file there so we can still rely on python setup.py --version.
Alternative without that minimal setup.py file would be much more complicated (especially since we rely on dynamic version functionality in pyproject.toml).

authors: Kami codecov-commenter

repo: apache/libcloud


text:
CloudSigma - add Honolulu, HI endpoint data

The recently added Honolulu, HI endpoint has not found its way into libcloud yet.  There are no associated JIRA tickets with this change.
Here it is.

Merged into trunk. Thanks.
/cc @vpetersson

I've failed at the commit message (mentioned the wrong ticket number) - 925613a875cf848bfcf4815b2f58fdc6286a4510.
@gladiatr72 - can you please manually close this ticket?

Awesome. Thanks @Kami!

No problem! Thanks for getting this merged.
On Jan 26, 2015 3:07 PM, "Viktor Petersson" notifications@github.com
wrote:

Awesome. Thanks @Kami https://github.com/Kami!
--
Reply to this email directly or view it on GitHub
https://github.com/apache/libcloud/pull/439#issuecomment-71538387.

authors: Kami gladiatr72 vpetersson


text:
[Bug] jdbc task cannot obtain the set variable value when the task takes a long time

Search before asking

[X] I searched the issues and found no similar issues.

Linkis Component
linkis-engineconnn-plugin
Steps to reproduce
1, create a jdbc job ,then publish to schedulis;
2,  the jdbc job have set variable like below:
set  yesterday= date_sub(current_date,1);
3, used variable like :
select * from xxx where  usetime>= ${hiveconf:yesterday}
4, server get the code right.
select * from xxx where  usetime>= ${hiveconf:yesterday}
5, but the final conversion sql is:
select * from xxx where  usetime>=

Expected behavior
final conversion sql is:
select * from xxx where  usetime>= date_sub(current_date,1)
Your environment

Linkis version used: 1.1.1
Environment name and version:

cdh-6.3.2
hive-2.1.1
spark-2.4 & 3.2
jdk 1.8.0_121

Anything else
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

提供最新版DataSphereStudio全家桶安装服务。
包括（DataSphereStudio, Linkis, Exchangeis(Datax, Sqoop), Scriptis, Streamis(Flink), Visualis, DolphinScheduler)

另外包含 元数据，数据安全，数据质量，数据血缘。（这些都是DataSphereStudio不自带的服务）
包含hadoop集群搭建（hive, spark, flink, kafak, kylin）
提供上述所有系统二次定制开发服务
提供系统搭建运维疑难问题解决

添加备注，DSS大数据
持续为DataSphereStudio开源贡献力量

repo: apache/linkis


text:
Unable to send query to Mailing List log4j-user@logging.apache.org

Hi,
I'm trying to understand a scenario and as per documentation and recommendation I tried to email to the Mailing list log4j-user@logging.apache.org which returns unsuccessfully.
Following is the response I get when I email.
**Hi. This is the qmail-send program at apache.org.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.
<log4j-user@logging.apache.org>:
Must be sent from an @apache.org address or a subscriber address or an address in LDAP.**
Can anyone guide how can I ask my query and through which platform?

@danishmarif, these are mailing lists. You first need to subscribe and then post your questions. Otherwise, you cannot see the answers to your question.

authors: danishmarif vy


text:
Increase build environment requirement to Java 17

While Maven plugins have started requiring Java 11 a while ago (e.g. Spotless 2.30+), some new plugins like bnd-maven-plugin have bumped the requirement to Java 17.
Most of our code is Java 17-ready, but we must update some of our tests that modify final private fields.
This is a blocker for apache/logging-parent#41.

Fixed by #2021


text:
[DISCUSS] Could we have a different ANN algorithm for Learned Sparse Vectors?

Description
Learned Sparse Vectors claim to combine the benefits of sparse (i.e. lexical) and dense (i.e. vector) representations
From https://en.wikipedia.org/wiki/Learned_sparse_retrieval:

Learned sparse retrieval or sparse neural search is an approach to text search which uses a sparse vector representation of queries and documents.[1] It borrows techniques both from lexical bag-of-words and vector embedding algorithms, and is claimed to perform better than either alone. The best-known sparse neural search systems are SPLADE[2] and its successor SPLADE v2

From https://zilliz.com/learn/enhancing-information-retrieval-learned-sparse-embeddings:

Learn sparse embeddings denote sparse vector representations of data crafted through sophisticated machine learning models such as SPLADE and BGE-M3. Unlike traditional sparse vectors, which rely solely on statistical methods like BM25, learned sparse embeddings enrich the sparse representation with contextual information while retaining keyword search capabilities. They can discern the significance of adjacent or correlated tokens, even if not explicitly present in the text, resulting in a "learned" sparse representation adept at capturing relevant keywords and classes.

A famous model for such sparse representations of documents is SPLADE (https://github.com/naver/splade)
Paper
Came across Efficient Inverted Indexes for Approximate Retrieval over Learned Sparse Representations (https://arxiv.org/pdf/2404.18812) which shows promising benchmarks for KNN search over learned sparse vectors:

Experimental results show average per-query latency in microsecond territory on various sparse embeddings of Ms Marco. Impressively, Seismic outperforms the graph-based winning solutions of the BigANN Challenge by a factor of at least 3.4 at 95% accuracy on Splade and 12 on Efficient Splade, with the margin widening substantially as accuracy increases. Other baselines, including state-of-the-art inverted index-based algorithms, are consistently one to two orders of magnitude slower than Seismic

Figure 3 in the linked paper summarizes the design of the algorithm:

The design of Seismic. Inverted lists are independently partitioned into geometrically-cohesive blocks. Each block is a set of document identifiers with a summary vector. The inner product of a query with the summary approximates the inner product attainable with the documents in that block. The forward index stores the complete vectors (including values).

Learned Sparse Vectors seem to be naturally compatible with inverted indexes, and many aspects of the algorithm are already implemented in Lucene
Could we use this for faster KNN search when sparse vectors are used?

What I wonder is: how can Lucene help with this? I feel like we have all the primitives available to enable Splade-style search and retrieval, but maybe there is something missing? IIRC there needs to be a per-term score, but I think we do have the ability to store custom term frequencies and to override similarity in order to do the appropriate combination of these term scores, so I think those are the ingredients needed for this. Maybe it's a case of trying it and seeing if there are some features it would be helpful to embed in the Lucene layer, or if indeed we can build this "on top"?

There might be a better format than just terms. But I would assume the bipartite graph stuff would help here.
Additionally, I would expect the most benefits to be made at query time. Looking at the newer research out of the splade folks, they are optimizing the query time by adjusting the scoring, not really the storage.
Maybe a better query would be a good first step.

I found this recent paper by well-known people in the IR efficiency space quite interesting: https://arxiv.org/pdf/2405.01117. It builds on inverted indexes and simple/intuitive ideas:

BP reordering, that Ben alluded to and that Lucene already supports, it naturally clusters documents with similar terms together,
Block-max WAND, which Lucene supports,
Anytime ranking on document ordered indexes (https://arxiv.org/pdf/2104.08976), ie. ranking ranges of doc IDs that have the best impact scores first in order to optimize pruning. Something Lucene doesn't support at the moment but that look doable and generally useful.
Unsafe top-k search via termination conditions and skipping blocks that are barely more competitive than the current top-k-th hit.
Query term pruning, which sounds like a good idea in general for learned sparse retrieval when the model generates many terms.
Scoring high-frequency / low-scoring terms via a forward index instead of an inverted index.

I have recently been interested in this direction and plan on spending non trivial amount of time on this over the next few weeks. Assuming we haven't started dev on this, I am assigning it to myself.
What I have been hacking on is impact sorted prioritisation of docIDs during the ranking phase (especially for custom rankers), so that would probably be the first thing that comes out of this ticket.

authors: atris benwtrent jpountz kaivalnp msokolov

repo: apache/lucene


text:
Address backward compat test issues after upgrading main to Lucene 11

There are several issues that need to be addressed now that main has moved on to accumulating changes for 11.

[x] Move 9.x bw test indices to "unsupported"
[x] Add 10.0.0 bw indices after 10.0.0 has been released
[ ] Resolve spurious 9.x bw test indices like more terms

Heads up: I pushed 10.0.0 backwards indices to main, and re-enabled backwards compatibility tests for indices >= 10.0, see https://github.com/apache/lucene/commit/352d85cbe4d3e64e3ac08588a816ae603de736df .

authors: ChrisHegarty javanna


text:
Add IntervalsSource for range and regexp queries

We already have convenient functions for contructing IntervalsSource for wildcard and fuzzy functions. This adds functions for regexp and range as well.

@romseygeek @dweiss Thank you for the review.


text:
host: sm: ignore security request, add test

Ignore security request that was recived after pairing has already started - fixes SM/CEN/PIS/BV-03-C
Add and fix SM unit tests

#AutoPTS run mynewt SM

Scheduled PR https://github.com/apache/mynewt-nimble/pull/1826#issuecomment-2263272206, board: nrf52, estimated start time: 16:52:19, test case count: 62, estimated duration: 0:30:06Test cases to be runSM/CEN/PROT/BV-01-CSM/CEN/JW/BV-05-CSM/CEN/JW/BI-04-CSM/CEN/JW/BI-01-CSM/CEN/PKE/BV-01-CSM/CEN/PKE/BV-04-CSM/CEN/PKE/BI-01-CSM/CEN/PKE/BI-02-CSM/CEN/OOB/BV-01-CSM/CEN/OOB/BV-03-CSM/CEN/OOB/BV-09-CSM/CEN/OOB/BI-01-CSM/CEN/EKS/BV-01-CSM/CEN/EKS/BI-01-CSM/CEN/KDU/BV-05-CSM/CEN/KDU/BV-06-CSM/CEN/KDU/BV-10-CSM/CEN/KDU/BI-01-CSM/CEN/PIS/BV-02-CSM/CEN/PIS/BV-03-CSM/CEN/SCJW/BV-01-CSM/CEN/SCJW/BV-04-CSM/CEN/SCJW/BI-01-CSM/CEN/SCPK/BV-01-CSM/CEN/SCPK/BV-04-CSM/CEN/SCPK/BI-01-CSM/CEN/SCPK/BI-02-CSM/CEN/SCOB/BV-01-CSM/CEN/SCOB/BV-04-CSM/CEN/SCOB/BI-01-CSM/CEN/SCOB/BI-04-CSM/PER/PROT/BV-02-CSM/PER/JW/BV-02-CSM/PER/JW/BI-03-CSM/PER/JW/BI-02-CSM/PER/PKE/BV-02-CSM/PER/PKE/BV-05-CSM/PER/PKE/BI-03-CSM/PER/OOB/BV-02-CSM/PER/OOB/BV-04-CSM/PER/OOB/BV-10-CSM/PER/OOB/BI-02-CSM/PER/EKS/BV-02-CSM/PER/EKS/BI-02-CSM/PER/KDU/BV-01-CSM/PER/KDU/BV-02-CSM/PER/KDU/BV-07-CSM/PER/KDU/BV-08-CSM/PER/KDU/BI-01-CSM/PER/PIS/BV-01-CSM/PER/PIS/BV-02-CSM/PER/SCJW/BV-02-CSM/PER/SCJW/BV-03-CSM/PER/SCJW/BI-02-CSM/PER/SCPK/BV-02-CSM/PER/SCPK/BV-03-CSM/PER/SCPK/BI-03-CSM/PER/SCPK/BI-04-CSM/PER/SCOB/BV-02-CSM/PER/SCOB/BV-03-CSM/PER/SCOB/BI-02-CSM/PER/SCOB/BI-03-C

AutoPTS Bot results:
No failed test found.
Successful testsSM      SM/CEN/EKS/BI-01-C              PASSSM      SM/CEN/EKS/BV-01-C              PASSSM      SM/CEN/JW/BI-01-C               PASSSM      SM/CEN/JW/BI-04-C               PASSSM      SM/CEN/JW/BV-05-C               PASSSM      SM/CEN/KDU/BI-01-C              PASSSM      SM/CEN/KDU/BV-05-C              PASSSM      SM/CEN/KDU/BV-06-C              PASSSM      SM/CEN/KDU/BV-10-C              PASSSM      SM/CEN/OOB/BI-01-C              PASSSM      SM/CEN/OOB/BV-01-C              PASSSM      SM/CEN/OOB/BV-03-C              PASSSM      SM/CEN/OOB/BV-09-C              PASSSM      SM/CEN/PIS/BV-02-C              PASSSM      SM/CEN/PIS/BV-03-C              PASSSM      SM/CEN/PKE/BI-01-C              PASSSM      SM/CEN/PKE/BI-02-C              PASSSM      SM/CEN/PKE/BV-01-C              PASSSM      SM/CEN/PKE/BV-04-C              PASSSM      SM/CEN/PROT/BV-01-C             PASSSM      SM/CEN/SCJW/BI-01-C             PASSSM      SM/CEN/SCJW/BV-01-C             PASSSM      SM/CEN/SCJW/BV-04-C             PASSSM      SM/CEN/SCOB/BI-01-C             PASSSM      SM/CEN/SCOB/BI-04-C             PASSSM      SM/CEN/SCOB/BV-01-C             PASSSM      SM/CEN/SCOB/BV-04-C             PASSSM      SM/CEN/SCPK/BI-01-C             PASSSM      SM/CEN/SCPK/BI-02-C             PASSSM      SM/CEN/SCPK/BV-01-C             PASSSM      SM/CEN/SCPK/BV-04-C             PASSSM      SM/PER/EKS/BI-02-C              PASSSM      SM/PER/EKS/BV-02-C              PASSSM      SM/PER/JW/BI-02-C               PASSSM      SM/PER/JW/BI-03-C               PASSSM      SM/PER/JW/BV-02-C               PASSSM      SM/PER/KDU/BI-01-C              PASSSM      SM/PER/KDU/BV-01-C              PASSSM      SM/PER/KDU/BV-02-C              PASSSM      SM/PER/KDU/BV-07-C              PASSSM      SM/PER/KDU/BV-08-C              PASSSM      SM/PER/OOB/BI-02-C              PASSSM      SM/PER/OOB/BV-02-C              PASSSM      SM/PER/OOB/BV-04-C              PASSSM      SM/PER/OOB/BV-10-C              PASSSM      SM/PER/PIS/BV-01-C              PASSSM      SM/PER/PIS/BV-02-C              PASSSM      SM/PER/PKE/BI-03-C              PASSSM      SM/PER/PKE/BV-02-C              PASSSM      SM/PER/PKE/BV-05-C              PASSSM      SM/PER/PROT/BV-02-C             PASSSM      SM/PER/SCJW/BI-02-C             PASSSM      SM/PER/SCJW/BV-02-C             PASSSM      SM/PER/SCJW/BV-03-C             PASSSM      SM/PER/SCOB/BI-02-C             PASSSM      SM/PER/SCOB/BI-03-C             PASSSM      SM/PER/SCOB/BV-02-C             PASSSM      SM/PER/SCOB/BV-03-C             PASSSM      SM/PER/SCPK/BI-03-C             PASSSM      SM/PER/SCPK/BI-04-C             PASSSM      SM/PER/SCPK/BV-02-C             PASSSM      SM/PER/SCPK/BV-03-C             PASS

Scheduled PR https://github.com/apache/mynewt-nimble/pull/1826#issuecomment-2263378919, board: nrf52, estimated start time: 17:39:59, test case count: 142, estimated duration: 1:26:58Test cases to be runGAP/BROB/BCST/BV-01-CGAP/BROB/BCST/BV-02-CGAP/BROB/BCST/BV-03-CGAP/BROB/BCST/BV-04-CGAP/BROB/OBSV/BV-01-CGAP/BROB/OBSV/BV-02-CGAP/BROB/OBSV/BV-05-CGAP/BROB/OBSV/BV-06-CGAP/DISC/NONM/BV-01-CGAP/DISC/NONM/BV-02-CGAP/DISC/LIMM/BV-03-CGAP/DISC/LIMM/BV-04-CGAP/DISC/GENM/BV-03-CGAP/DISC/GENM/BV-04-CGAP/DISC/LIMP/BV-01-CGAP/DISC/LIMP/BV-02-CGAP/DISC/LIMP/BV-03-CGAP/DISC/LIMP/BV-04-CGAP/DISC/LIMP/BV-05-CGAP/DISC/GENP/BV-01-CGAP/DISC/GENP/BV-02-CGAP/DISC/GENP/BV-03-CGAP/DISC/GENP/BV-04-CGAP/DISC/GENP/BV-05-CGAP/DISC/RPA/BV-01-CGAP/IDLE/NAMP/BV-01-CGAP/IDLE/NAMP/BV-02-CGAP/CONN/NCON/BV-01-CGAP/CONN/NCON/BV-02-CGAP/CONN/NCON/BV-03-CGAP/CONN/DCON/BV-01-CGAP/CONN/UCON/BV-01-CGAP/CONN/UCON/BV-02-CGAP/CONN/UCON/BV-03-CGAP/CONN/UCON/BV-06-CGAP/CONN/ACEP/BV-01-CGAP/CONN/ACEP/BV-03-CGAP/CONN/ACEP/BV-04-CGAP/CONN/GCEP/BV-01-CGAP/CONN/GCEP/BV-02-CGAP/CONN/GCEP/BV-05-CGAP/CONN/GCEP/BV-06-CGAP/CONN/SCEP/BV-01-CGAP/CONN/SCEP/BV-03-CGAP/CONN/DCEP/BV-01-CGAP/CONN/DCEP/BV-03-CGAP/CONN/DCEP/BV-05-CGAP/CONN/DCEP/BV-06-CGAP/CONN/CPUP/BV-01-CGAP/CONN/CPUP/BV-02-CGAP/CONN/CPUP/BV-03-CGAP/CONN/CPUP/BV-04-CGAP/CONN/CPUP/BV-05-CGAP/CONN/CPUP/BV-06-CGAP/CONN/CPUP/BV-08-CGAP/CONN/CPUP/BV-10-CGAP/CONN/TERM/BV-01-CGAP/CONN/PRDA/BV-01-CGAP/CONN/PRDA/BV-02-CGAP/BOND/NBON/BV-01-CGAP/BOND/NBON/BV-02-CGAP/BOND/NBON/BV-03-CGAP/BOND/BON/BV-01-CGAP/BOND/BON/BV-02-CGAP/BOND/BON/BV-03-CGAP/BOND/BON/BV-04-CGAP/SEC/SEM/BV-21-CGAP/SEC/SEM/BV-37-CGAP/SEC/SEM/BV-38-CGAP/SEC/SEM/BV-22-CGAP/SEC/SEM/BV-39-CGAP/SEC/SEM/BV-40-CGAP/SEC/SEM/BV-23-CGAP/SEC/SEM/BV-24-CGAP/SEC/SEM/BV-56-CGAP/SEC/SEM/BV-57-CGAP/SEC/SEM/BV-58-CGAP/SEC/SEM/BV-59-CGAP/SEC/SEM/BV-60-CGAP/SEC/SEM/BV-61-CGAP/SEC/SEM/BV-26-CGAP/SEC/SEM/BV-41-CGAP/SEC/SEM/BV-42-CGAP/SEC/SEM/BV-27-CGAP/SEC/SEM/BV-43-CGAP/SEC/SEM/BV-44-CGAP/SEC/SEM/BV-28-CGAP/SEC/SEM/BV-29-CGAP/SEC/SEM/BV-62-CGAP/SEC/SEM/BV-63-CGAP/SEC/SEM/BV-64-CGAP/SEC/SEM/BV-65-CGAP/SEC/SEM/BV-66-CGAP/SEC/SEM/BV-67-CGAP/SEC/SEM/BI-09-CGAP/SEC/SEM/BI-10-CGAP/SEC/SEM/BV-45-CGAP/SEC/SEM/BI-20-CGAP/SEC/SEM/BI-21-CGAP/SEC/SEM/BI-22-CGAP/SEC/SEM/BI-23-CGAP/SEC/AUT/BV-11-CGAP/SEC/AUT/BV-12-CGAP/SEC/AUT/BV-13-CGAP/SEC/AUT/BV-14-CGAP/SEC/AUT/BV-17-CGAP/SEC/AUT/BV-18-CGAP/SEC/AUT/BV-19-CGAP/SEC/AUT/BV-20-CGAP/SEC/AUT/BV-21-CGAP/SEC/AUT/BV-22-CGAP/SEC/AUT/BV-23-CGAP/SEC/AUT/BV-24-CGAP/SEC/AUT/BV-25-CGAP/SEC/AUT/BV-27-CGAP/PRIV/CONN/BV-10-CGAP/PRIV/CONN/BV-11-CGAP/PRIV/CONN/BV-12-CGAP/ADV/BV-01-CGAP/ADV/BV-02-CGAP/ADV/BV-03-CGAP/ADV/BV-04-CGAP/ADV/BV-05-CGAP/ADV/BV-08-CGAP/ADV/BV-10-CGAP/ADV/BV-11-CGAP/ADV/BV-14-CGAP/ADV/BV-17-CGAP/ADV/BV-18-CGAP/PADV/PASM/BV-01-CGAP/PADV/PAM/BV-01-CGAP/PADV/PASE/BV-01-CGAP/PADV/PASE/BV-02-CGAP/PADV/PASE/BV-03-CGAP/PADV/PASE/BV-04-CGAP/PADV/PASE/BV-05-CGAP/PADV/PASE/BV-06-CGAP/PADV/PAST/BV-01-CGAP/PADV/PAST/BV-02-CGAP/GAT/BV-04-CGAP/GAT/BV-05-CGAP/GAT/BV-06-C

AutoPTS Bot results:
No failed test found.
Successful testsSM      SM/CEN/EKS/BI-01-C              PASSSM      SM/CEN/EKS/BV-01-C              PASSSM      SM/CEN/JW/BI-01-C               PASSSM      SM/CEN/JW/BI-04-C               PASSSM      SM/CEN/JW/BV-05-C               PASSSM      SM/CEN/KDU/BI-01-C              PASSSM      SM/CEN/KDU/BV-05-C              PASSSM      SM/CEN/KDU/BV-06-C              PASSSM      SM/CEN/KDU/BV-10-C              PASSSM      SM/CEN/OOB/BI-01-C              PASSSM      SM/CEN/OOB/BV-01-C              PASSSM      SM/CEN/OOB/BV-03-C              PASSSM      SM/CEN/OOB/BV-09-C              PASSSM      SM/CEN/PIS/BV-02-C              PASSSM      SM/CEN/PIS/BV-03-C              PASSSM      SM/CEN/PKE/BI-01-C              PASSSM      SM/CEN/PKE/BI-02-C              PASSSM      SM/CEN/PKE/BV-01-C              PASSSM      SM/CEN/PKE/BV-04-C              PASSSM      SM/CEN/PROT/BV-01-C             PASSSM      SM/CEN/SCJW/BI-01-C             PASSSM      SM/CEN/SCJW/BV-01-C             PASSSM      SM/CEN/SCJW/BV-04-C             PASSSM      SM/CEN/SCOB/BI-01-C             PASSSM      SM/CEN/SCOB/BI-04-C             PASSSM      SM/CEN/SCOB/BV-01-C             PASSSM      SM/CEN/SCOB/BV-04-C             PASSSM      SM/CEN/SCPK/BI-01-C             PASSSM      SM/CEN/SCPK/BI-02-C             PASSSM      SM/CEN/SCPK/BV-01-C             PASSSM      SM/CEN/SCPK/BV-04-C             PASSSM      SM/PER/EKS/BI-02-C              PASSSM      SM/PER/EKS/BV-02-C              PASSSM      SM/PER/JW/BI-02-C               PASSSM      SM/PER/JW/BI-03-C               PASSSM      SM/PER/JW/BV-02-C               PASSSM      SM/PER/KDU/BI-01-C              PASSSM      SM/PER/KDU/BV-01-C              PASSSM      SM/PER/KDU/BV-02-C              PASSSM      SM/PER/KDU/BV-07-C              PASSSM      SM/PER/KDU/BV-08-C              PASSSM      SM/PER/OOB/BI-02-C              PASSSM      SM/PER/OOB/BV-02-C              PASSSM      SM/PER/OOB/BV-04-C              PASSSM      SM/PER/OOB/BV-10-C              PASSSM      SM/PER/PIS/BV-01-C              PASSSM      SM/PER/PIS/BV-02-C              PASSSM      SM/PER/PKE/BI-03-C              PASSSM      SM/PER/PKE/BV-02-C              PASSSM      SM/PER/PKE/BV-05-C              PASSSM      SM/PER/PROT/BV-02-C             PASSSM      SM/PER/SCJW/BI-02-C             PASSSM      SM/PER/SCJW/BV-02-C             PASSSM      SM/PER/SCJW/BV-03-C             PASSSM      SM/PER/SCOB/BI-02-C             PASSSM      SM/PER/SCOB/BI-03-C             PASSSM      SM/PER/SCOB/BV-02-C             PASSSM      SM/PER/SCOB/BV-03-C             PASSSM      SM/PER/SCPK/BI-03-C             PASSSM      SM/PER/SCPK/BI-04-C             PASSSM      SM/PER/SCPK/BV-02-C             PASSSM      SM/PER/SCPK/BV-03-C             PASS

#AutoPTS run mynewt GAP/BROB/BCST/BV-04-C

Scheduled PR https://github.com/apache/mynewt-nimble/pull/1826#issuecomment-2275673088, board: nrf52, estimated start time: 14:15:22, test case count: 1, estimated duration: 0:01:37Test cases to be runGAP/BROB/BCST/BV-04-C

AutoPTS Bot results:
Failed testsGAP     GAP/BROB/BCST/BV-04-C           FAIL

#AutoPTS run mynewt SM/CEN/PIS/BV-03-C

Scheduled PR https://github.com/apache/mynewt-nimble/pull/1826#issuecomment-2275928684, board: nrf52, estimated start time: 16:11:14, test case count: 1, estimated duration: 0:02:54Test cases to be runSM/CEN/PIS/BV-03-C

AutoPTS Bot results:
No failed test found.
Successful testsSM      SM/CEN/PIS/BV-03-C              PASS

@sjanc please review

last commit should be squashed so that tests always build

@sjanc done

authors: codecoup-tester piotrnarajowski sjanc

repo: apache/mynewt-nimble


text:
MINIFICPP-1736 Add MiNiFi agent C2 operation capabilities to agent manifest

Added MiNiFi supported C2 operation data to the agent manifest in the following format:
{
  ..
  "agentInfo": {
      "agentManifest": {
          ...
          "supportedOperations": [
              {
                  "properties": {
                      <OPERAND1>: {},
                      <OPERAND2>: {
                          <METADATA_KEY>: [
                            <METADATA_VALUE_OBJECTS>
                          ]
                      }
                  },
                  "type": <OPERATION>
              }
          ],
          ...
      },
      ...
  }
}

https://issues.apache.org/jira/browse/MINIFICPP-1736

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically main)?

[ ] Is your initial contribution a single, squashed commit?

For code changes:

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file?
[ ] If applicable, have you updated the NOTICE file?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.

The JSON structure of the supported operations was simplified in 9a29abe81ce6b918dcb1b9f8924117dc19a2be59, with some additional bugfixes/new features of the serialization:

removed merging of parent-child nodes with the same name in HeartbeatJsonSerializer.cpp that would cause a bug of losing non-collapsible JSON nodes
added keep_empty flag to Value nodes signaling to not drop empty JSON objects or arrays while serializing

repo: apache/nifi-minifi-cpp


text:
NIFI-9958 Add Framework Support for Sensitive Dynamic Properties

Summary
NIFI-9958 Adds framework support for Sensitive Dynamic Properties in Processors, Controller Services and Reporting Tasks.
The implementation adds a SupportsSensitiveDynamicProperties behavior annotation, which must be applied to component classes to indicate support for Sensitive Dynamic Properties. No other changes are necessary at the component implementation level.
Changes include updates to REST Resource methods and models to indicate component support for sensitive dynamic properties, as well as the ability for a client to indicate names of dynamic properties that should be marked sensitive. The Property Descriptor REST Resource methods for components include an new optional sensitive query parameter to support subsequent user interface integration.
New system tests include basic component creation and updating with sensitive dynamic property names and values. Component auditor changes ensure the Flow Configuration History masks sensitive dynamic property values.
The flow persistence approach infers sensitive property value status based on whether the property value matches the standard encrypted pattern. This strategy avoids the need to introduce structural changes to the flow.xml.gz or flow.json.gz files.
This implementation provides the basic capabilities, and adds the SupportsSensitiveDynamicProperties annotation to InvokeHTTP for the purpose of testing REST Resource methods. Subsequent pull requests will incorporate user interface and documentation updates.
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[X] Apache NiFi Jira issue created

Pull Request Tracking

[X] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[X] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[X] Pull Request based on current revision of the main branch
[X] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[X] Build completed using mvn clean install -P contrib-check

[X] JDK 8
[ ] JDK 11
[ ] JDK 17

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

Thanks for the initial feedback @markap14! Pushed an update implementing the changes.

@exceptionfactory I created an InvokeHTTP processor and set a breakpoint so that I could update the DTO to indicate that my newly added property was sensitive. This worked well!
But then I copied & pasted the processor and I found that the newly created Processor had the property value but didn't have the property marked as sensitive. Looks like there needs to be an update to DtoFactory.copy(ProcessorConfigDTO), .copy(ControllerServiceDTO) and .copy(ReportingTaskDTO) to ensure that any copies of the components have the sensitive property names copied as well.

Also ran into an issue where I pushed that same flow into a nifi registry (built on the same branch) and then changed the sensitive property to use a sensitive parameter. When I checked out the flow again, the value was NOT marked sensitive.

Thanks for catching the snippet and sensitive parameter handling issues @markap14! I updated StandardSnippetDAO and DtoFactory to maintain the sensitive dynamic property names when copying and pasting. I also updated the synchronizer to consider the Versioned Property Descriptor sensitive status to ensure that status will be preserved when loading a versioned flow.

Thanks for the update @exceptionfactory. I think we've still got something not quite right, though. I With the latest updated, I have an InvokeHTTP processor with 2 sensitive dynamic properties. One is named "SecretHeader" with a value of "SecretValue" and the other is "SecretParamHeader" with a value of "#{SecretParam}". When I push to registry or download, we are now preserving the "SecretParamHeader" property as we want. But now it's also exporting "SecretHeader" = "SecretValue". So it didn't strip out the sensitive dynamic property.

But now it's also exporting "SecretHeader" = "SecretValue". So it didn't strip out the sensitive dynamic property.

Thanks for the additional feedback @markap14, I will take another look at the export handling.

@exceptionfactory I think I ran into one more issue. I created an InvokeHTTP processor and added a new sensitive property. Did NOT use a parameter. Pushed to registry. Then checked it out. The sensitive property was gone, which we expected (of note, we should describe this behavior clearly in the user guide!). But then I added the property back into my flow, and it still shows as a local modification. I think we need to filter that out as a local modification.

I walked through this scenario with pushing a flow to Registry and then importing it again. It appeared to work as expected without showing any local modifications.

@exceptionfactory thanks for checking. I think that I messed up my test. Because I was trying to run this in cluster mode and there's no UI, I was updating DTO fields manually and I think I accidentally left the property as non-sensitive. Sorry for the confusion!

repo: apache/nifi


text:
NIFI-11653 Add Connection URL Validation for Database Services

Summary
NIFI-11653 Adds a Connection URL Validator for Database URL properties in the DBCP and HikariCP Connection Pool Controller Services. The Connection URL Validator introduces the concept of unsupported URL schemes for JDBC connections with an initial value listing H2.
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[X] Apache NiFi Jira issue created

Pull Request Tracking

[X] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[X] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[X] Pull Request based on current revision of the main branch
[X] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[X] Build completed using mvn clean install -P contrib-check

[X] JDK 11
[ ] JDK 17

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

@exceptionfactory changes look good.  wrapping local build now.  recommend an entry in the migration guidance for 1.22 stating that interaction with h2 databases will be prevented in these components going forward.

authors: exceptionfactory joewitt


text:
NIFI-13997 Load the requested python processor only

Summary
NIFI-13997
Load the requested python processor only
Before this change: when multiple python processors were present in a package,
we only installed the dependencies from requirements.txt plus those from
ProcessorDetails.dependencies of the requested processor. Then we tried to
load all the python files in the package, including other processors which may
have other dependencies. The result was that the requested processor could not
be loaded.
After the change: we only load the requested processor and any non-processor
helper files which are present in the package.
Enable py4j integration tests in system-tests
There are 26 tests (+1 added now) in PythonControllerInteractionIT.java, which
is in nifi-py4j-integration-tests, but these were not run in any of the CI
jobs. I have added them to system-tests, because integration-tests explicitly
skips nifi-py4j-integration-tests with a comment that it is run as part of
system-tests.
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[x] Apache NiFi Jira issue created

Pull Request Tracking

[x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[x] Pull Request based on current revision of the main branch
[ ] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[x] Build completed using mvn clean install -P contrib-check

[x] JDK 21

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

I recommend building the tests using much more narrowly-scoped dependencies

fixed in 19f2f345fde48a733a6a2732933417ac01d7e451

[...] The fact that there are ways to support both inline and external dependencies with Python Processors is a special feature, but I'm not sure whether this is something we should promote in general. [...]

Yes, shared dependencies per package is the way to go in most cases. We did, however, run into a case of a package of "parser" processors, and while most of their dependencies were common, some of the format-specific libs were quite large. We would like to avoid loading all format-specific libs if only one format is used.


text:
tools/Wasm.mk problems

the idea to build wasm modules using nuttx headers is broken
it produces very WAMR specific modules (it seems targeting WAMR's LIBC_BUILTIN). if we want to build wasm modules, it's better to target wasm32-wasi.
WCC detection is broken. for example, github ubuntu-22.04 runners have /bin/clang, which confuses the detection.
CONFIG_INTERPRETERS_WASM vs CONFIG_INTERPRETERS_WASM3 typo

my proposal: remove it
an alternative:

target it to wasm32-wasi
have an explicit config to enable it (disable it by default)

@yamt
We are not using the implementation of wasi/wasi-libc. We have a python script to parse nuttx csv files(syscall/libc/libm) to automatically generate nuttx syscall/libc/libm glue codes. Currently, this feature works as expected without boundary checking in wamr, but it still needs some work if enable boundary check.

@yamt We are not using the implementation of wasi/wasi-libc. We have a python script to parse nuttx csv files(syscall/libc/libm) to automatically generate nuttx syscall/libc/libm glue codes. Currently, this feature works as expected without boundary checking in wamr, but it still needs some work if enable boundary check.

where can i find the python script?

where can i find the python script?

I will upload a draft ASAP

why do you prefer the "no sandbox" approach?
doesn't it spoil many of benefits of wasm?

@yamt Please refer to this PR https://github.com/apache/nuttx-apps/pull/2058

authors: anchao no1wudi yamt

repo: apache/nuttx-apps


text:
[vcos:bugfix] armv8-r:optimize libc string apis with asm

Optimize libc string apis(memcpy/memset/memmove/memchr/strcmp/strlen) with arm32 assembly instruction including vfp and neon. Add arch releated elf parsing
Note: Please adhere to Contributing Guidelines.
Summary
optimize armv8r cortex-r52 libc apis.
Impact
armv8r cpu libc apis.
Use arm32 assemble / neon instructions to optimize libc apis (memcpy/memset/memmove/memchr/strcmp/strlen)
Testing
Tested on armv8r cortex-r52 cpu.

[Experimental Bot, please feedback here]
No, this PR description does not fully meet the NuttX requirements.  Here's why and how to fix it:
Missing Information/Insufficient Detail:

Summary:  While it mentions optimization, it lacks crucial details.

Why is this change necessary? Is there a performance bottleneck? Are the current implementations inefficient?
What functional part of the code is being changed? Be specific – mention the files/functions modified (e.g., libc/string/memcpy.c).
How does the change exactly work?  "Use arm32 assemble / neon instructions" is too vague. Briefly explain the optimization techniques used (e.g., loop unrolling, SIMD instructions).  Mention if any new functions were added.  The mention of "arch related elf parsing" seems unrelated to string functions and needs further explanation or belongs in a separate PR.

Impact:  The impact descriptions are too brief.

Is new feature added?  If the ELF parsing is new, say so explicitly. Otherwise, it's likely "Is existing feature changed?" is YES.
Impact on user: Probably NO, but if there are any changes to function behavior (however subtle), explain them.
Impact on build:  Possibly YES, as assembly code is being added. Specify any new build flags, dependencies, or changes to the build system.
Impact on hardware:  Specify the specific ARM architectures affected (armv8r, cortex-r52 is a good start, but mention if others are impacted).  Are there any implications for other architectures?
Impact on documentation: Likely YES if new features are added or if the behavior of existing functions changes. State whether documentation updates are included in the PR.
Impact on security:  Consider potential security implications of optimized code (e.g., buffer overflows if optimizations change memory access patterns). If none, state "NO."
Impact on compatibility:  Are there any compatibility concerns with older versions of NuttX or other ARM architectures?

Testing:  The testing description is inadequate.

Build Host(s): Provide details about the build host OS, CPU architecture, and compiler version used.
Target(s):  Be more specific about the target board and configuration.
Testing logs:  Absolutely essential! Include actual logs demonstrating the functionality before and after the change.  Show performance benchmarks if performance improvement is the goal.  Demonstrate correctness by showing input and output.

Example of an Improved Summary:
This PR optimizes several libc string functions (memcpy, memset, memmove, memchr, strcmp, strlen) for the armv8r architecture (specifically cortex-r52) to improve performance.  The current C implementations are inefficient, particularly for larger data sizes. This optimization utilizes ARM32 assembly language and NEON SIMD instructions to perform operations on multiple bytes simultaneously. Key optimizations include loop unrolling and specialized instructions for memory operations.  This PR also adds a new ELF parsing function [...] (Explain its purpose and connection, or remove this if unrelated).
By providing more specific details in each section, the PR will be much clearer and easier for reviewers to evaluate.  Remember to follow the contributing guidelines and provide concrete evidence of testing.

authors: jinliangli nuttxpr

repo: apache/nuttx


text:
ORC-284: [C++] add missing tests for C++ tools

Added two tool tests for FileStatistics and FileContents.

@majetideepak Please help review this patch, thanks!

+1 LGTM. Thanks @wgtmac !

authors: majetideepak wgtmac

repo: apache/orc


text:
HDDS-5121. fix download page to use ozone space instead of hadoop

Ozone tar files uploaded to the old hadoop namespace. As Ozone is not Hadoop any more they supposed to be uploaded to  the ozone project space.
I moved the artifacts:
svn mv https://dist.apache.org/repos/dist/release/hadoop/ozone/ozone-1.1.0 https://dist.apache.org/repos/dist/release/ozone/1.1.0

But proper linking requires this one flag turning off.

Thanks the addendum (and the review) . (cc #4, I think the column order was broken there)

repo: apache/ozone-site


text:
HDDS-6991. Setting bucket owner may throw NPE

What changes were proposed in this pull request?
Change the order of two params to avoid null
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6991
How was this patch tested?
Tested on prod cluster.

Thank you for your contribution @ferhui

@kerneltime @aswinshakil @adoroszlai Thanks for your reviews.

authors: ferhui kerneltime

repo: apache/ozone


text:
HDDS-9295. Avoid overriding finalize() in CodecBuffer.

What changes were proposed in this pull request?
CodecBuffer overrides the finalize() method for leak detection. It turns out causing a much longer GC time. (Thanks
@duongkame for finding it out.)
This JIRA is to change CodecBuffer not to override finalize(). In the test, we will create a subclass to override CodecBuffer for leak detection.
What is the link to the Apache JIRA
HDDS-9295
How was this patch tested?
Added a new test.

@szetszwo @duongkame Could you please update the JIRA description about the background of this issue?

@umamaheswararao , we do have the background information in the first two comments of HDDS-9295.

... A finalizer alternative, like reference queue, can be considered (ref: https://www.oracle.com/technical-resources/articles/javase/finalization.html).

ReferenceQueue sounds like a very good idea.

authors: szetszwo umamaheswararao


text:
[cdc] Adjust-CdcJsonDeserializationSchema

Purpose
The serialization entry of cdc kafka_table/database_sync is CdcJsonDeserializationSchema, but CdcJsonDeserializationSchema#deserialize returns CdcSourceRecord(JsonNode). When users extend other format, there will be limitations. Most data serialization is based on byte[], not JsonNode.
For example:
Users need to use other serialization protocols to serialize data based on byte[], such as binlog of other databases.

Tests

API and Format

Documentation

closed, recommended to use flink-cdc Paimon Pipeline Connector first

repo: apache/paimon


text:
[core] Drop stats in manifest file reading

Purpose

We planned to remove the statistical information when reading the file, but this optimization was hindered by the need for statistical information later for whole bucket filter.
In this PR, we refactor this logical to drop stats and compute filter selected result, in this way, we can keep whole bucket filter.

Tests

API and Format

Documentation

Very thanks for @JingsongLi do this work!
Our company needs this feature now!

+1


text:
Support for Dead Letter Topics in Consumers?

Does the nodeJS client currently support dead letter topics?

Unfortunately the node.js client doesn't support the dead letter topic for now.

@nkurihar, thank you for confirming. Is there a timeline on when this would be added, if it exists?
Thanks for all your contributions to this client.

Is there a timeline on when this would be added, if it exists?

No, unfortunately we have not decided the timeline to add this feature yet...

repo: apache/pulsar-client-node


text:
msgDropRate metric is not available through non-deprecated Java interface, CLI, or Pulsar Manager

Describe the bug
https://github.com/apache/pulsar/pull/1634 consolidated the Admin API's persistentTopics() and nonPersistentTopics() to topics(), but the non-persistent msgDropRate metric available in nonPersistentTopics().stats() is not available in the consolidated topics().stats(). Since nonPersistentTopics() is deprecated, so there is no supported API for getting msgDropRate. There is also no way to get this statistic from the pulsar-admin CLI tool or Pulsar Manager, presumably because they are using topics().stats().
To Reproduce
Steps to reproduce the behavior:

Create a non-persistent topic nptopic
Use CLI tool: pulsar-admin topics stats non-persistent://public/default/nptopic -- msgDropRate is not available
Use Java client: admin.topics().getStats(topic) -- msgDropRate is not available
Use Java client: admin.nonPersistentTopics().getStats(topic) -- msgDropRate is available, but this interface is deprecated

Expected behavior
This metric should be available through topics().stats(). OR a new method should be added to topics that returns non-persistent topic stats, similar to the getPartitionedStats() method that gets stats for partitioned topics.
Version
2.7.1

the MsgDropRate is also missing from stats in mainline after the interface update, figure out it's a weird serialization issue, I'll try to fix it.

Thanks @MarvinCai,
The new behavior in 2.8.1 is much better. At least as far as I can tell, it still requires a type cast, but that's not so bad. This code is working:
String topicName = "non-persistent://public/default/my_topic";
TopicStats topicStats = admin.topics().getStats(topicName);
SubscriptionStats subStats = stats.getSubscriptions().get("drop_test");
double dropRate = 0.0;
if (subStats instanceof NonPersistentSubscriptionStats)
{
    NonPersistentSubscriptionStats npSubStats = (NonPersistentSubscriptionStats)subStats;
    dropRate = npSubStats.getMsgDropRate();
}

This beats my previous work-around in 2.7.1 (which no longer worked in 2.8.0):
String topicName = "non-persistent://public/default/my_topic";
TopicStats topicStats;
if (topicName.startsWith("non-persistent")
{
    topicStats = admin.nonPersistentTopics().getStats(topicName);
}
else
{
    topicStats = admin.topics().getStats(topicName);
}

double dropRate = 0.0;
if (topicStats instanceof NonPersistentTopicStats)
{
    NonPersistentTopicStats npTopicStats = (NonPersistentTopicStats)topicStats;
    dropRate = npTopicStats.getSubscriptions().get("drop_test").getMsgDropRate();
}

authors: MarvinCai megfigura

repo: apache/pulsar