text:
Resharding in sharding-proxy?

I would like to know if sharding-proxy supports resharding automatically as a new node is added ?

ShardingSphere can't resharding automatically for now.
Auto scaling out feature is developing, sharding-scaling will be merged after release 4.0.0-RC3.
Detail see #3077 .

authors: KomachiSion parthibd

license_type: permissive


text:
I want to extend ParameterData by SPI to analyse body in json

Question
I want to extend ParameterData by SPI to analyse body in json, but I find the body can only be read once. If I read just like this:
@Join
public class BodyParameterData implements ParameterData {
@Override
public String builder(final String paramName, final ServerWebExchange exchange) {
ServerRequest serverRequest = ServerRequest.create(exchange, HandlerStrategies.withDefaults().messageReaders());
Mono reqBody = serverRequest.bodyToMono(String.class);
String bodyStr = reqBody.block();
DocumentContext context = JsonPath.parse(bodyStr);
return context.read(paramName);
}
}
and the WebClientPlugin get a error like this:
[2022-03-30 10:58:31.055] [ERROR] [] [WebClientPlugin:lambda$handleRequestBody$1:119] [reactor-http-nio-1] [-[null]-]
io.netty.handler.timeout.ReadTimeoutException: null
[2022-03-30 10:58:31.061] [ERROR] [] [GlobalErrorHandler:handle:52] [reactor-http-nio-1] [-[[6e08505e-1] Resolved [ReadTimeoutException: null] for HTTP POST /shenyu-gateway/v1/pay-center/v1/pay/config]-]
I judge the bytebuffer in the body may exist some problem.
Can I get some sugggestion?

No， you can impl ParameterData， you can see FileSizeFilter.java

OK, thank you so much!

source: gharchive/issue

repo: apache/incubator-shenyu

license_source: github-api


text:
[fix: bug] Custom spi config

// Describe your PR here; eg. Fixes #issueNo

Make sure that:

[ ] You have read the contribution guidelines.
[ ] You submit test cases (unit or integration tests) that back your changes.
[ ] Your local test passed mvn clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #2586 (e637c79) into master (017b655) will decrease coverage by 0.69%.
The diff coverage is 100.00%.

:exclamation: Current head e637c79 differs from pull request most recent head b9c70a9. Consider uploading reports for the commit b9c70a9 to get more accurate results

@@             Coverage Diff              @@
##             master    #2586      +/-   ##
============================================
- Coverage     61.15%   60.46%   -0.70%
+ Complexity     4237     4195      -42
============================================
  Files           653      653
  Lines         18532    18532
  Branches       1659     1658       -1
============================================
- Hits          11334    11205     -129
- Misses         6305     6441     +136
+ Partials        893      886       -7

Impacted Files
Coverage Δ

...a/org/apache/shenyu/common/enums/OperatorEnum.java
90.00% <100.00%> (-5.00%)
:arrow_down:

.../org/apache/shenyu/common/enums/ParamTypeEnum.java
95.65% <100.00%> (ø)

...ache/shenyu/plugin/grpc/cache/GrpcClientCache.java
62.50% <0.00%> (-37.50%)
:arrow_down:

...n/alibaba/dubbo/cache/AlibabaDubboConfigCache.java
45.88% <0.00%> (-29.42%)
:arrow_down:

...gin/apache/dubbo/cache/ApacheDubboConfigCache.java
45.23% <0.00%> (-27.39%)
:arrow_down:

...che/shenyu/sync/data/http/HttpSyncDataService.java
61.38% <0.00%> (-19.81%)
:arrow_down:

...henyu/plugin/grpc/resolver/ShenyuNameResolver.java
45.26% <0.00%> (-17.90%)
:arrow_down:

...a/org/apache/shenyu/common/utils/VersionUtils.java
67.85% <0.00%> (-10.72%)
:arrow_down:

...apache/shenyu/admin/utils/ShenyuDictH2Trigger.java
63.63% <0.00%> (-9.10%)
:arrow_down:

...ache/shenyu/admin/utils/PluginHandleH2Trigger.java
63.63% <0.00%> (-9.10%)
:arrow_down:

... and 31 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 017b655...b9c70a9. Read the comment docs.

source: gharchive/pull-request


text:
fix missing ojdbc component id in component-libraries.yml

Please answer these questions before submitting pull request

Why submit this pull request?

[X] Bug fix

[ ] New feature provided

[ ] Improve performance

Related issues

Bug fix

Bug description.
发现oracle 的组件解析出来是ojdbc， id： 34，然后这几个配置的地方没有34这个id。
How to fix?

New feature or improvement

Describe the details and related test reports.

Coverage decreased (-0.002%) to 14.345% when pulling b31bd33da65f04e75b5222c47279ba3e08ef7426 on langyan1022:master into ee6841fc13ffb246566283184e955381ae243644 on apache:master.

repo: apache/incubator-skywalking


text:
Druid lookup type extraction failing in pandas

When you have a druid lookup type column defined as
{
 "type": "extraction",
 "dimension": "coderizedColumn",
 "outputName": "prettyNameColumn",
 "outputType": "STRING",
 "extractionFn": {
   "type": "lookup",
   "dimension": "dimensionName",
   "outputName": "dimensionOutputName",
   "replaceMissingValueWith": "missing_value",
   "retainMissingValue": false,
   "lookup": {
     "type": "map",
     "map": {
       "Code1": "PrettyName1",
       "Code2": "PrettyName2",
     },
     "isOneToOne": false
   }
 }
}

When trying to query on it, superset now breaks with the following exception
2017-11-06 16:57:59,864:ERROR:root:"None of [[u'extractionFn', u'dimension', u'outputName', u'outputType', u'type']] are in the [index]"
Traceback (most recent call last):
 File "/app/.heroku/python/lib/python2.7/site-packages/superset/viz.py", line 275, in get_payload
   df = self.get_df()
 File "/app/.heroku/python/lib/python2.7/site-packages/superset/viz.py", line 98, in get_df
   self.results = self.datasource.query(query_obj)
 File "/app/.heroku/python/lib/python2.7/site-packages/superset/connectors/druid/models.py", line 1069, in query
   client=client, query_obj=query_obj, phase=2)
 File "/app/.heroku/python/lib/python2.7/site-packages/superset/connectors/druid/models.py", line 865, in get_query_str
   return self.run_query(client=client, phase=phase, **query_obj)
   qry['dimensions'], filters)
 File "/app/.heroku/python/lib/python2.7/site-packages/superset/connectors/druid/models.py", line 874, in _add_filter_from_pre_query_data
   f = Dimension(dim) == row[dim]
 File "/app/.heroku/python/lib/python2.7/site-packages/superset/connectors/druid/models.py", line 1006, in run_query
 File "/app/.heroku/python/lib/python2.7/site-packages/pandas/core/series.py", line 642, in __getitem__
   return self._get_with(key)
 File "/app/.heroku/python/lib/python2.7/site-packages/pandas/core/series.py", line 683, in _get_with
   return self.loc[key]
 File "/app/.heroku/python/lib/python2.7/site-packages/pandas/core/indexing.py", line 1328, in __getitem__
   return self._getitem_axis(key, axis=0)
 File "/app/.heroku/python/lib/python2.7/site-packages/pandas/core/indexing.py", line 1541, in _getitem_axis
   return self._getitem_iterable(key, axis=axis)
 File "/app/.heroku/python/lib/python2.7/site-packages/pandas/core/indexing.py", line 1081, in _getitem_iterable
   self._has_valid_type(key, axis)
 File "/app/.heroku/python/lib/python2.7/site-packages/pandas/core/indexing.py", line 1418, in _has_valid_type
   (key, self.obj._get_axis_name(axis)))
KeyError: "None of [[u'extractionFn', u'dimension', u'outputName', u'outputType', u'type']] are in the [index]"

This broke in the 0.19 series of releases.

This https://github.com/apache/incubator-superset/pull/3920 should fix issue with dimension spec.

authors: ghulands kkalyan

repo: apache/incubator-superset


text:
style: fix checkbox color

Max, is this PR related to https://github.com/apache/incubator-superset/pull/10799 ? Replaces the use of font-awesome for checkboxes with the svg components that already exist.

Yes @JunlinC it's related, it's just fixing the color

Codecov Report

Merging #10970 into master will decrease coverage by 4.48%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master   #10970      +/-   ##
==========================================
- Coverage   65.85%   61.37%   -4.49%
==========================================
  Files         815      815
  Lines       38324    38336      +12
  Branches     3600     3599       -1
==========================================
- Hits        25238    23528    -1710
- Misses      12984    14622    +1638
- Partials      102      186      +84

Flag
Coverage Δ

#cypress
?

#javascript
61.76% <100.00%> (+0.01%)
:arrow_up:

#python
61.14% <ø> (-0.31%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

superset-frontend/src/components/CheckboxIcons.tsx
85.71% <ø> (ø)

...uperset-frontend/src/components/Checkbox/index.tsx
100.00% <100.00%> (ø)

superset-frontend/src/SqlLab/App.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/explore/App.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/dashboard/App.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/explore/index.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/dashboard/index.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/setup/setupColors.js
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/chart/ChartContainer.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/setup/setupFormatters.js
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 182 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3d8f757...c87a610. Read the comment docs.

this PR breaks the filter_box control:

authors: JunlinC codecov-commenter graceguo-supercat mistercrunch


text:
[SECURITY] Bump pyarrow to 0.15.1 due to CVE

CATEGORY
Choose one

[x] Bug Fix
[ ] Enhancement (new features, refinement)
[ ] Refactor
[ ] Add tests
[ ] Build / Development Environment
[ ] Documentation

SUMMARY
Updating PyArrow to latest version due to CVE: https://snyk.io/vuln/SNYK-PYTHON-PYARROW-483024
Note that the RESULTS_BACKEND_USE_MSGPACK config has been defaulted to False in https://github.com/apache/incubator-superset/pull/8060, but the intention is to re-enable this once https://github.com/apache/incubator-superset/issues/8225 is resolved.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN
CI
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

REVIEWERS

Looks like travis was having issues when this was pushed, restarted 2 builds in the matrix...

authors: mistercrunch robdiciuccio


text:
[core] Fix, sanitize errors returned from testconn

CATEGORY
Choose one

[X] Bug Fix
[X] Enhancement (new features, refinement)
[ ] Refactor
[ ] Add tests
[ ] Build / Development Environment
[ ] Documentation

SUMMARY
Test connection API testconn does not sanitize errors generated from SQLAlchemy or dbapi drivers. This PR distinguish between URI parse errors, drInivers not installed on the system and OperationalError.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Before:
Connection failed:

Invalid URI:

After:
Connection failed (OperationalError):

Invalid URI:

Invalid driver:

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

REVIEWERS
@robdiciuccio

Thank you, once more @villebro, addressed the comments

Codecov Report

Merging #9178 into master will decrease coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #9178      +/-   ##
==========================================
- Coverage   59.16%   59.13%   -0.04%
==========================================
  Files         372      372
  Lines       11935    11938       +3
  Branches     2927     2925       -2
==========================================
- Hits         7061     7059       -2
- Misses       4694     4699       +5
  Partials      180      180

Impacted Files
Coverage Δ

...ontend/src/components/ListView/TableCollection.tsx
88.88% <0%> (-3.71%)
:arrow_down:

...frontend/src/views/dashboardList/DashboardList.tsx
69.72% <0%> (-3.43%)
:arrow_down:

...uperset-frontend/src/views/chartList/ChartList.tsx
74.48% <0%> (-0.26%)
:arrow_down:

...et-frontend/src/components/ConfirmStatusChange.tsx
100% <0%> (ø)
:arrow_up:

...rset-frontend/src/components/ListView/ListView.tsx
92.4% <0%> (+0.19%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e55fe43...df5ad14. Read the comment docs.

authors: codecov-io dpgaspar


text:
[iOS]Cannot fire [refresh] event when pullingdown on list component ! weex version is 0.26.0

Describe the bug

Cannot fire [refresh] event when pullingdown on a list use [refresh] component. pullingdown event is works fine.
IOS上 refresh 组件不能触发refresh事件，pullingdown事件工作正常
To Reproduce

Environment

Device: [iPhone6/6s/6sPlus/7.........]
OS: [iOS12.1]

Expected behavior

Screenshots

Additional context

There is nothing I can do without a demo.
If this still bothers you and you find a solution, you could give us a PR if you can, I am very happy to talk with you implemenation detail or review you PR in mailing list.
I have a busy schedule and I can't read Github issue every day, but I check mailing list every day. I am sorry if this bothers you.

repo: apache/incubator-weex


text:
Reintegrate Ping

Ping feature to keep websocket was introduced in #109 .
But it was deleted during the merge of #56.
This is bringing it back

This is ready, let's wait for CI

LGTM

authors: Leemoonsoo corneadoug

repo: apache/incubator-zeppelin

license_source: bigquery


text:
allow mapping of complex user properties that implement the Supplier interface

Add support for synchronising user profile properties that implement Supplier interface.

hi @daniancu , i will look into this tomorrow. one trivial thing just before i get started: could you please create JIRA ticket in https://issues.apache.org/jira/browse/OAK-9745?jql=project %3D "Jackrabbit Oak" ?
we need a dedicated jira for any kind of modifications to the code base in order to be able to make sure we have a comprehensive list for releases (and also for awareness of the community). thanks a lot.

authors: anchela daniancu

repo: apache/jackrabbit-oak


text:
[REFACTORING] Use standard configuration for webadmin integration tests

This is closer to runtime conditions thus prevents some bug.
Modify guice injections to associate value 0 with a random port.

Force pushed to squash commits...

Resolve conflicts please

authors: Arsnael chibenwa

repo: apache/james-project


text:
The counter function calculation error in HTTP Request.

Expected behavior
Increment by 1 each time
Actual behavior
Increment by 2 each time
Steps to reproduce the problem

JMeter Version
5.6.2
Java Version
1.8.0_371
OS Version
10.0.19045.3324

I've found the same issues, it occurred in version 5.6.2 but not in 5.5
Possible cause:
The following changes were made in jmeter.properties file, as it says it will be removed in future. For now, uncommenting line 1063 and setting it true works just as fine as in JMeter 5.5

jmeter.properties file is located under bin directory

The Counter config element works fine in both versions. If it fits your use case, you can try it:

Make sure to use at right scope

According to the above method, it works normally.

authors: Ninja-Cyborg xmlbw

repo: apache/jmeter


text:
KAFKA-15632: Drop the invalid remote log metadata events

__remote_log_metadata topic cleanup policy is set to DELETE and default retention is set to unlimited.
The expectation is that the user will configure the maximum retention time for this internal topic compared to all the other user created topics in the cluster. We cannot keep it to unlimited as the contents of this internal topic need to be in the local storage.
RemoteLogMetadata cache expect the events to be in the order of valid transition as defined in
RemoteLogSegmentState#isValidTransition
Once the retention got expired for this topic say after 30 days due to breach by size/time, then there can be partial metadata events and the RemoteLogMetadataCache starts to throw remote-resource-not-found-error and being logged as warn statement while starting up the broker.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

For my understanding. Previously whenever we encountered an invalid transition the consumer threw and exception and exited. This did not resolve itself upon broker restart. What you want to do instead is to disregard the invalid state transition and just log it when encountered?

yes, this is an unrecoverable error so we want to log an error and proceed further. Assume that there is an invalid transition happened only for one segment due to producer/network retries (or) late-arrival message, then we don't want to block the remote operations for other partitions.

Will take a look next week. Thanks.

@showuon @clolov It looks like the comments have been addressed. Do you have any other comments on this PR?

authors: kamalcph satishd showuon

repo: apache/kafka


text:
MINOR: fix docs typo

More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Call for review @guozhangwang @bbejeck @vvcephei
Need to be cherry-picked to 2.1 branch, too.

@bbejeck Pushed to more commits...

Merged to trunk and cherry-picked to 2.1 branch.


text:
[KAFKA-10705]: Make state stores not readable by others

Change permissions on the folders for the state store so they're no readable or writable by "others", but still accessible by owner and group members.
Committer Checklist (excluded from commit message)

[x] Verify design and implementation
[x] Verify test coverage and CI build status
[x] Verify documentation (including upgrade notes)

@wcarlson5 @cadonna @ableegoldman for review

retest this please

Cherrypicked to 2.6, will wait for 2.7.0 release to finish up before merging to 2.7

@ableegoldman I see you merged this in 2.6. How critical is this fix? Do I need to spin a new RC?

I don't think so. It would be nice to have if you happen to end up cutting a new RC, but I wouldn't delay the ongoing release over this

Cherrypicked to 2.7

Cherrypicked to 2.7


text:
[Umbrella] Kyuubi Spark lineage extension supports Spark 3.4

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the proposal
Make Kyuubi Spark lineage extension supports Spark 3.4.
Task list

[x] #4925
[x] #5170

Are you willing to submit PR?

[x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
[ ] No. I cannot submit a PR at this time.

@iodone is this completed?

@iodone is this completed?

In development.

Thanks for your great work! @iodone

repo: apache/kyuubi


text:
Bump Scala to 2.12.20

:mag: Description
Issue References 🔗

Scala 2.12.20 release https://github.com/scala/scala/releases/tag/v2.12.20

Describe Your Solution 🔧
Types of changes :bookmark:

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪
Behavior Without This Pull Request :coffin:
Behavior With This Pull Request :tada:
Related Unit Tests

Checklist 📝

[x] This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

cc @bowenliang123

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 0.00%. Comparing base (d7219fc) to head (8736f2c).
Report is 2 commits behind head on master.

Additional details and impacted files
@@          Coverage Diff           @@
##           master   #6676   +/-   ##
======================================
  Coverage    0.00%   0.00%
======================================
  Files         683     684    +1
  Lines       42213   42233   +20
  Branches     5755    5755
======================================
- Misses      42213   42233   +20

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Thanks, merged to master (1.10.0).


text:
[Feature][ECM] Support download ec log file

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Problem Description
No response
Description
No response
Use case
No response
Solutions
No response
Anything else
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Duplicate issue

repo: apache/linkis


text:
Lucene's facets should tap into IndexSearcher's TaskExecutor too?

Description
Spinoff from the exciting discussion on https://github.com/apache/lucene/pull/13472:
Lucene has made great gains recently on intra-query concurrency: using multiple threads (with a "slice" work unit = one or more segments) to reduce latency of queries.  Besides faster wall clock time, since CPU is running concurrently, Lucene also can gain efficiency because segments can terminate earlier / start using skipping as the more competitive results from other segments arrive sooner/concurrently, causing less total CPU to be spent to get the top hits for the query.
But I think Lucene's doc values and taxonomy facets do not use any concurrency?  Even if you pass a TaskExecutor to IndexSearcher, facet counting will still run single threaded.  Can we fix this to also make facet counting faster (net elapsed wall clock time)?  It's tricky because some facet counting aggregate into data structures (like int[] or an HPPC int->int map) that are not easily made thread safe?
Note: we do have ConcurrentSortedSetDocValuesFacetCounts which does use concurrency, but the other facet counting (numeric ranges, taxonomy facets) do not.  Also, ConcurrentSortedSetDocValuesFacetCounts takes its own ExecutorService not a TaskExecutor.

Also, ConcurrentSortedSetDocValuesFacetCounts takes its own ExecutorService

It also creates one task per segment instead of reusing the IndexSearcher slices, which would be nice to fix too.
(More generally, I wish the facet module behaved a bit more like a regular Lucene Collector, instead of first loading all hits into a bitset and doing the work in a second phase, which is memory-intensive and means that it doesn't automatically benefit from IndexSearcher's existing features like inter-segment concurrency, dynamic pruning, timeout support, or upcoming features like I/O concurrency and intra-segment concurrency.)

(More generally, I wish the facet module behaved a bit more like a regular Lucene Collector, instead of first loading all hits into a bitset and doing the work in a second phase, which is memory-intensive and means that it doesn't automatically benefit from IndexSearcher's existing features like inter-segment concurrency, dynamic pruning, timeout support, or upcoming features like I/O concurrency and intra-segment concurrency.)

+1 -- then facets would auto-magically get these awesome concurrency improvements in Lucene.

authors: jpountz mikemccand

repo: apache/lucene


text:
Terminate automaton after matched the whole prefix for PrefixQuery.

For PrefixQuery, we can terminate the automaton on current term if we have matched the whole prefix, and match this term directly.
Furthermore, if there is a subBlock, we could match all its' sub terms.

@jpountz Please take a look when you get a chance.

IntersectsTermsEnum is a bit scary to me, maybe @mikemccand can take a look, I expect him to be more familiar with it.

I will have a look -- thanks for the ping @jpountz.

@mikemccand Thanks for you suggestion, I am trying to implement it.

Have you tried to measure any performance change with this? E.g. you could run a luceneutil benchy with just PrefixQuery, or, Regexp/WildcardQuery that also have this property (match-all states in their automata).

I am working on this too.
@jpountz Thanks for you reply.

I think the optimization may be similar to the one done in AutomatonTermsEnum?
https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/AutomatonTermsEnum.java#L149-L153
When "ping-ponging" the term dictionary against the automaton, it tracks visited bitset and looks for such loops in the automaton. when it finds one, it temporarily acts like a TermRangeQuery.
I think, it works a bit more general than just prefixquery and also helps with regex and wildcard queries too.

I think the optimization may be similar to the one done in AutomatonTermsEnum?

Thanks for reminding that, I will dig into AutomatonTermsEnum's optimization.

There is still problem in the implementation of detecting whether accept state can match all suffix (RunAutomaton.canMatchAllSuffix), I am trying to figure it out.
On the other hand, I will dig into AutomatonTermsEnum's optimization.

@mikemccand
I renamed the field used to indicate wether an accept state can match all suffixes, and detected it in RunAutomaton.
Please take a look when you get a chance.

I think the optimization may be similar to the one done in AutomatonTermsEnum?

It seems the optimization of AutomatonTermsEnum is to improve iterating term mode, from seeking bytes from DFA and seek termsEnum(seekCeil), to simply sequential reads the termsEnum, after finding a loop(setLinear).
In both mode, AutomatonTermsEnum needs to check if the term is accepted by running automaton.
If I am not mistaken, this optimization is different from AutomatonTermsEnum's. It directly matches all reminding suffixes and sub blocks, after detecting an accept state with a .* transition back onto itself.
Or maybe you mean we can improve AutomatonTermsEnum's optimization, to implement this optimization's effect? @rmuir

Have you tried to measure any performance change with this? E.g. you could run a luceneutil benchy with just PrefixQuery, or, Regexp/WildcardQuery that also have this property (match-all states in their automata).

I measured it with current implementation with wikimedium1m, the result is unstable:
TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
Prefix3      879.20      (9.1%)      995.37      (4.2%)   13.2% (   0% -   29%) 0.062
Prefix3      924.98      (9.9%)     1042.17      (7.9%)   12.7% (  -4% -   33%) 0.083

TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
Prefix3     1480.94      (8.8%)     1559.89      (6.4%)    5.3% (  -9% -   22%) 0.195
Prefix3     1242.80      (6.9%)     1307.30      (5.3%)    5.2% (  -6% -   18%) 0.299
Prefix3      177.54      (1.3%)      202.74      (6.3%)   14.2% (   6% -   22%) 0.000

@mikemccand
I think this change is close for PrefixQuery, TermRangeQuery, and custom binary AutomatonQuery.
As for RegexpQuery and WildcardQuery, whose automata converted by UTF32ToUTF8. We just get accept transitions [0, 127] in RunAutomaton, I am not sure whether it is enough that just check the accept state is transition is [0, 127].
Another approach is set matchAllSuffix in UTF32ToUTF8, but need to add a unFundamental member to Automaton.
Or, can we just push this change for Prefix/TermRangeQuery, and leave a TODO for Regexp/WildcardQuery temporarily?

Conflicts resolved.

Hi @mikemccand , Can we push on this change?

authors: jpountz mikemccand rmuir vsop-479


text:
MINIFICPP-849 - Fixed dynamic properties with expression language sup…

…port
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit?

For code changes:

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file?
[ ] If applicable, have you updated the NOTICE file?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

We only got the dynamic expression once and cached the compiled expression, thereby effectively ignoring changes to the dynamic property.
Now we recompile the expression if we detect the change in the value of the property.

I misunderstood how dynamic properties work, sorry about that.
If you don't have any objections I will close this pull request.

@bakaid It's a bit of a pain to go through online wikis to find that info, so no fault at all. I don't object to it being closed.

authors: bakaid phrocker

repo: apache/nifi-minifi-cpp


text:
NIFI-4353 Implemented new JAXB logic.

Added unit test and template test resource.
Added RAT exclusion.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit?

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Will review...

Thanks @alopresto! This has been merged to master.

authors: alopresto mcgilman

repo: apache/nifi


text:
NIFI-6791 Add UUID3 and UUID5 functions to Expression Language

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
This adds type 3 and type 5 namespace and name-based UUID functions to the Expression Language.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[X] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[X] Has your PR been rebased against the latest commit within the target branch (typically master)?

[X] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[X] Have you written or updated unit tests to verify your changes?
[X] Have you verified that the full build is successful on both JDK 8 and JDK 11?
[X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[x] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[X] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

would love to see @markap14 get some eyes on here but it looks good.  I'm just not quite comfortable merging myself as I dont feel confident enough in the EL world

authors: jfrazee joewitt


text:
Follow o_cloexe semantics when calling the exec function

Summary
sched/task: close file descriptor with O_CLOEXEC before active task or exec
modify:

Ensure that the child task copies all fds of the parent task,
including those with O_CLOEXE.
Make sure spawn_file_action is executed under fd with O_CLOEXEC,
otherwise it will fail.
When a new task is activated or exec is called, close all fds
with O_CLOEXEC flags.

fs/dup3: impletement dup3/nx_dup3_from_tcb function
 refs: https://man7.org/linux/man-pages/man2/dup.2.html

fs/dup/dup2: don't add O_CLOEXEC for new file descriptor
refs: https://man7.org/linux/man-pages/man2/dup.2.html

The two file descriptors do not share file descriptor flags (the
close-on-exec flag).  The close-on-exec flag (FD_CLOEXEC; see
fcntl(2)) for the duplicate descriptor is off.

libs/lib_psfa_adddup2.c: remove workaround aboud dup2
Because popen no longer needs to rely on intermediate
tasks to start command tasks.

Impact
posix_spawn/exec_module/task_create/... with exec or start new task
Testing
Vela

@Donny9 I think it should be a nice idea to include a popen test to apps/testing/ostest. Testing only on Vela will be like "It works on my machine" TM! :-D

@Donny9 I think it should be a nice idea to include a popen test to apps/testing/ostest. Testing only on Vela will be like "It works on my machine" TM! :-D

https://github.com/apache/nuttx-apps/pull/2163 contain the related change to make popen more conform the standard

@Donny9 I think it should be a nice idea to include a popen test to apps/testing/ostest. Testing only on Vela will be like "It works on my machine" TM! :-D

apache/nuttx-apps#2163 contain the related change to make popen more conform the standard

@xiaoxiang781216 actually it only created the popen() function. I'm talking about adding a popen() test into ostest

@acassis OK, I will add popen test to ostest in another PR soon~

@Donny9 please rebase the patch

@Donny9 please rebase the patch

Done~

repo: apache/nuttx


text:
Fix-SAMA5D2-LCDC-framebuffer-artefacts-(LVGL)

The SAMA5D2 LCDC driver has memory mapped framebuffers that LVGL (for example) can write to directly causing issues. LVGL - and presumably other applications - need a buffer that they can render to and only when done should that be copied to the "real" framebuffers of the LCDC.
I have added a function CONFIG_FB_UPDATE  that can be called via the relevant IOCTL that does the copy and it has fixed the issue I was seeing in relation to screen artefacts I noted as an issue in the nuttx-apps repo here.
I am not 100% sure what the correct way to address this is, so for now I have added a new Kconfig entry to allow an "EXT" memory buffer to be declared, and that is what is reported as the pinfo->fbmem memory location but it doesn't really feel right - and it seems there are no other architectures that address this so it is perhaps unique to the SAMA5D2?
There must be a better/neater/proper way to be doing this?
Impact
It allows LVGL V9 to behave as expected, on this processor. I have tried to ensure that the driver reverts to the previous methodology if the new Kconfig entry is not present.
Testing
Custom board with SAMA5D27-D1G and 800x480 LCDC. Manually deleted new Kconfig entry to check the driver reverts to as before and, similarly, deleted the newly enforced CONFIG FB_UPDATE entry to check.

@TimJTi I think it is fine. Just let @patacongo say if he thinks this approach is fine (don't know if it is compatible with FLAT, PROTECTED and KERNEL modes).
@TimJTi please algo remember to update SAMA5D2 Documentation/ to include this information that they need to enable CONFIG_FB_UPDATE to avoid tearing/flickering as you was facing.

Drivers should not write directly into hardware frame buffers.  For the case of a single graphics user thread, that would work but it will not work.  The result would be a disaster.
When the NX windowing system is used, for example, numerous threads may be accessing the display concurrently.  LVGL will not work in that mode, but other graphics do.  NX will serialize all accesses across multiple threads and all will work well.
This is not a good solution and I ask that you do not commit it.  Instead, please consider using the pre-window framebuffer logic.  I think this does whay you want to do, but might require that you think a little differently:
``CONFIG_NX_RAMBACKED``
   Enables RAM backed window support. If this option is selected, then
   windows may be optionally created with a RAM framebuffer backing up
   the window content. Rending into the window will result in rending
   into the backup framebuffer, then updating the physical display from
   the framebuffer.

   The advantage of this option is that the application that manages
   window will no longer receive redraw() callbacks. Those calls
   normally occur, for example, when a window "above" moves exposing a
   portion of the window below. If this option is selected, then the
   system will redraw the exposed portion of the window from the backup
   framebuffer without intervention of the window applications. This
   greatly reduces the complexity of the application and performance of
   the window at the expense of increased memory usage.

   An exception is the case when the window is resized to a wider and/or
   taller size. In that case, the redraw callback will till occur. It is
   necessary in that case to provide new graphic content for the
   extended window area.

   Redraw requests in other cases are also suppressed: Changes to window
   position, size, etc.

The is a superior, bug-free alternative.

This is not a good solution and I ask that you do not commit it. Instead, please consider using the pre-window framebuffer logic.

In the case of LVGL, you would not use all windows but only the background window.  I do think you would have to enable the NX subsystem which would provide the multithreaded support.
I think some other work would be required to do this without NX.

Thank you @patacongo !!! @XuNeo since you are working on LVGL V9 could you please take a look and help Tim on that?

Thank you @patacongo !!! @XuNeo since you are working on LVGL V9 could you please take a look and help Tim on that?

A truly great solution would be to implement a LGVL compatible framebuffer driver for the NX windows.  That would allow to solve BOTH the tearing problem to to support multithreaded concurrent windows with LVGL.  How cool would that be?

Thank you @patacongo !!! @XuNeo since you are working on LVGL V9 could you please take a look and help Tim on that?

Thank you for your comments. Having slept on it I have realised my solution is right...but in completely the wrong place. It belongs in LVCGL not NuttX which is what I had thought until convincing myself otherwise.
I am right that LVGL was "messing" with the framebuffer and homed in on a SAMA5D2 driver issue; but the LVGL NuttX framebuffer is rendering DIRECT to the framebuffer, not to it's own draw buffer. This is contrary to the basic principles of LVGL - set up a buffer or buffers for LVGL to render to; flush it to the display; notify that the flush is complete; repeat.
I will reopen my issue on LVGL as this needs to be addressed there. I will keep this PR open for a short while as it is a useful place for a discussion and NuttX issues get more attention than LVGL ones do, sadly.
But I like the idea of NX and RAM backed windows but I am not sure what "a LGVL compatible framebuffer driver for the NX windows" would be like?
Since LVGL simply renders to a buffer that is setup in user-land, can an app not simply use LVGL to render to an NX RAM-backed window without any change to the NX/NuttX ecosystem?
Very interested to discuss this further - when I asked a question that was 100% related to this issus @gregory-nutt  essentially said the same: LVGL on multiple NX windows would be super cool; and my nascent App could easily use multiple NX windows, many of which have independent LVGL graphics rendered to them. Bring it on!!

Since LVGL simply renders to a buffer that is setup in user-land, can an app not simply use LVGL to render to an NX RAM-backed window without any change to the NX/NuttX ecosystem?

The RAM backed windows are an option and it sounds like you don't need the option since the buffering will occur in user space.

Very interested to discuss this further - when I asked a question that was 100% related to this issus @gregory-nutt essentially said the same: LVGL on multiple NX windows would be super cool; and my nascent App could easily use multiple NX windows, many of which have independent LVGL graphics rendered to them. Bring it on!!

An window framebuffer would be a minor extension to the existing full display framebuffer.  If the framebuffer is not RAM backed then this driver Window framebuffer driver would just replace the direct writes into the framebuffer with calls to NX APIs to write the image.  NX would write to the correct position on the display, clip, and handle intersections.
Documentation: https://cwiki.apache.org/confluence/display/NUTTX/NX+Graphics+Subsystem
Other than the initialization of the window and selection of the driver, I don't think that the change would affect the application.
The driver would also have to handle a few callbacks like redraw (just redraw the window if it a new part of the window is exposed) and other events (like moving the window or keyboard or mouse input).  Keyboard / mouse input is another topic.  These events go to the window that has focus.  I would need to think about that more.
I am interested in this topic.  I am very retired now, but still willing to help out.

I am interested in this topic. I am very retired now, but still willing to help out.
Thanks! I spent the morning trying to fix the LVGL nuttx framebuffer driver but it misbehaves. I added local buffers - as there should be - for the rendering, and a flush function to memcpy the specific "dirty" area of the local render buffer to the nuttX framebuffer but it has odd, and repeatable, side effects. I just hope someone in LVGL-land will advise/comment.

I am on the cusp of being too deep into my current LVGL path to  invite NX to the party, but I will if I have to, short term - and will certainly look into it mid-term if there is a miraculous lvgl fix for now. Bear with me.

Hi @TimJTi ,
There are two issues with your current setup. I have explained a bit in https://github.com/lvgl/lvgl/issues/6389.
Issue
From the video, it looks like the arc head is flicking. This happens because LVGL's intermediate drawing result is sent to display by hardware LCD controller directly without waiting.
The second issue is tearing. With single frame buffer, this will happen for sure.
Solution
LVGL has a special refreshing mode called PARTIAL mode. It allows LVGL to draw on a temporal buffer and memcpy to FB when it's really finished(thus no flicking). It can still suffer from screen tearing with single FB.
Current LVGL NuttX porting(lv_nuttx_fbdev.c) does not support this mode. lv_nuttx_lcd.c has support but I have no experience using it.
Instead of hacking into FB driver, we should upgrading  lv_nuttx_fbdev.c to add support to PARTIAL mode.
Multiple instance
LVGL V9 introduces support for multiple instances, allowing multiple windows to run simultaneously. The server allocates virtual framebuffers to clients, which can be easily composed using the LVGL Image widget. Clients draw on the virtual FB and send update events to the server.
The NX Graphic system sounds like a good candidate for managing the window framebuffers, and input events. The recent LVGL QNX porting PR has something similar that using QNX Screen window as a display.

Hi. If you look at the latest updates to the LVGL issue you linked to, you will see that I am 99% sure LVGL is rendering DIRECTLY to the framebuffer which, if I am right, is WRONG. I have implemented  a variant (a link to my fork is in that issue) that uses LVGL Linux-type framebuffer code, with dedicated buffer(s) that lvgl can render to that are only flushed to the real hardware framebuffer via the flush callback - but it still has issues and I included a video of them. This suggests lvgl is calling a flush BEFORE it has completely finished rendering. If I am right with that, then it is also WRONG!!
I have tried partial mode but, to be fair, not before I did the linux-style framebuffer driver. If the declaration of the buffer as PARTIAL rather than DIRECT is a trigger for lvgl to only call the flush after rendering is definitely complete, it would be useful to know, and I will try it when I get a moment.
But, for now, LVGL V9 is broken when it comes to using a framebuffer rather than arch or board-specfic device driver.

Perhaps the answer is lv_disp_flush_is_last(&disp_drv) ? That - I think - can be used to not flush until rendering is finished...I just came across this in the porting guide (which until now I hadn't expected to need to read!)
flush_cb a callback function to copy a buffer's content to a specific area of the display. lv_disp_flush_ready(&disp_drv) needs to be called when flushing is ready. LVGL might render the screen in multiple chunks and therefore call flush_cb multiple times. To see if the current one is the last chunk of rendering use lv_disp_flush_is_last(&disp_drv).

This is 100% LVGL, so I will close this and work on a fix and PR at the LVGL repo. Once I get time to look at an NX+LVGL combo I will open a new discussion and/or PR.
Thanks for everyone's help here.

authors: TimJTi XuNeo acassis patacongo


text:
Improved: list and grids (OFBIZ-11345)

changed:
FinAccountForms.xml: changed forms with list attributed to grids, additional cleanup
FinAccountScreens.xml: changed form reference for lists to grid reference

Thanks Jacques for your assistance to get this merged.

repo: apache/ofbiz-framework


text:
ORC-1608: Upgrade Hadoop to 3.4.0

This is a clone of #1783

This is resolved via 2f2d0ba8a

repo: apache/orc


text:
ORC-829: Optimize Serialization percentileBits

What changes were proposed in this pull request?
Optimize Serialization percentileBits method
Why are the changes needed?
Speed and simplicity
How was this patch tested?
Existing unit tests, no functionality changes.

Hey @belugabehr changes LGTM -- just wondering if we can/should incorporate the Perf tests you did when discovered this?

@pgaref Hey, thanks for taking a look.
Nothing special here.  I am just slowly working through the core-benchmark module.  I've examined the write side a bit (see: #734 #735 #736) and hope to find time next week to look at the read side.
To be clear, I think the biggest savings of this PR comes from using Long.numberOfLeadingZeros instead of implementing a loop that can loop as many as 64-times per value written.  The rest of the stuff I changed based on manual inspection of the code.  I did not test each piece individually, but as I mentioned, after making these changes this method didn't even register in the performance capture.


text:
HDDS-6129. OM has thread unsafe issues that can cause OM shutdown.

What changes were proposed in this pull request?
See: https://issues.apache.org/jira/browse/HDDS-6129
How was this patch tested?
Existing UT

I have modified this PR by referring to the OmMultipartKeyInfo modification in HDDS-2322. Also added UT. Can you help take antoher look? @bharatviswa504 @adoroszlai

Now with this change, not sure the added test makes sense.

Agreed, I posted the test method only as a simple way to repro the problem.

Now with this change, not sure the added test makes sense.

Agreed, I posted the test method only as a simple way to repro the problem.

Tnanks @adoroszlai and @bharatviswa504 for the review.
I changed the original test, where I passed subject.copyObject(). This will get an error before this patch,  after we re-new the object there will be no errors here.

Merged this. Thanks @adoroszlai and @bharatviswa504 for the review.

authors: adoroszlai captainzmc

repo: apache/ozone


text:
HDDS-6239. ozonesecure-mr failing with No URLs in mirrorlist

What changes were proposed in this pull request?
CentOS 8 is EOL, package repository moved to vault.centos.org on Jan/31.  ozonesecure-mr test needs to install the krb5-workstation package.  This is a quick and dirty workaround to update repository config to access the new location.  (Base image needs to be updated in the long run.)
https://issues.apache.org/jira/browse/HDDS-6239
How was this patch tested?
https://github.com/adoroszlai/hadoop-ozone/runs/5018254007#step:5:660

Thanks @umamaheswararao for the review.

you want to file another JIRA for longer term solution as you mentioned this is 'quick and dirty solution" ?

The image affected is not part of Ozone (it's https://github.com/flokkr/docker-hadoop), so I don't think we need a Jira issue there.

Sure, thanks I have merged this.

authors: adoroszlai umamaheswararao


text:
HDDS-6127. file checksum to support both CRC32 and CRC32C.

What changes were proposed in this pull request?
Support both CRC32 an CRC32C for Ozone to make it possible to replicate files between HDFS and Ozone.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6127
How was this patch tested?
Unit tests.

Just a question - does this actually help if the block sizes are different? Eg HDFS is mostly 128MB, Ozone is 256MB by default. Combining checksums with different blocks sizes will give a different combined checksum, right?

Just a question - does this actually help if the block sizes are different? Eg HDFS is mostly 128MB, Ozone is 256MB by default. Combining checksums with different blocks sizes will give a different combined checksum, right?

md5md5crc32 will be influenced by chunk size and block size.
but for composite-crc32c,  since crc32c support incremental computation, block size and chunk size will take no effect on the final file checksum.

@JacksonYao287 - thanks for the info.

md5md5crc32 will be influenced by chunk size and block size.
but for composite-crc32c, since crc32c support incremental computation, block size and chunk size will take no effect on the final file checksum.

So far in Ozone, we only have md5md5crc32, right?

@sodonnel
seems we already have composite-crc32 in Ozone,  but we do not have composite-crc32c.
please  take a look at BaseFilechecksumHelper#makeCompositeCrcResult()

The coding frame is fine, but the result of computed Composite-crc32c is different from hdfs's. Could you take a deep look?

thanks @jojochuang for this patch. can you add a test for ChecksumCombineMode.COMPOSITE_CRC?

thanks @jojochuang for the work. i have committed this patch to master branch!

authors: JacksonYao287 Xushaohong jojochuang sodonnel


text:
HDDS-9081. Handling Netty back pressure when streaming containers for replication.

What changes were proposed in this pull request?
Replication sends messages faster than the network/receiver can consume. This results in many messages being queued by Netty and the amount allocated memory surges and results in OOME.
The change in this PR eliminates messages backlog and memory surge for both pull and push, thus allowing multiple concurrent replications.
It can be optimized to fully utilize manual control to avoid unnecessary delay (or sleep), following these examples. I don't have data to say if it's worth it yet.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9081
How was this patch tested?
Tested locally and verified it eliminates direct memory surges during replication.

[INFO] Running org.apache.hadoop.ozone.container.replication.TestGrpcContainerUploader
Error:  Tests run: 3, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 1.942 s <<< FAILURE! - in org.apache.hadoop.ozone.container.replication.TestGrpcContainerUploader
Error:  org.apache.hadoop.ozone.container.replication.TestGrpcContainerUploader.successfulReplication  Time elapsed: 1.841 s  <<< ERROR!
java.lang.ClassCastException: org.apache.hadoop.ozone.container.replication.TestGrpcContainerUploader$1 cannot be cast to org.apache.ratis.thirdparty.io.grpc.stub.CallStreamObserver

authors: duongkame kerneltime


text:
PHOENIX-7441 Integrate the Spotless plugin and update the code template

Update dev/PhoenixCodeTemplate.xml to use latest format as in hbase.
Copied license-header file from hbase, the one which phoenix was using was a little different
Fix file having misplaced package block as we do not want any manual code change in the commit where we run spotless
Disable upToDateChecking which is true by default since spotless 2.35.0

CC: @stoty @lfrancke

I would either discuss the import order changes in the existing dev thread, or open a new one for them.
We should have consensus on that before the big bang reformat commit. (Even if it's just no objection/replies)
I personally support copying everything from HBase.
I'm fine with renaming the xml, but if we do that, we also need to update the website, which refers to it.

PTAL @virajjasani @gjacoby126

authors: NihalJain stoty

repo: apache/phoenix


text:
Bug fix: Start counting operator execution time from first NoOp block

Instructions:

The PR has to be tagged with at least one of the following labels (*):

feature
bugfix
performance
ui
backward-incompat
release-notes (**)

Remove these instructions before publishing the PR.

(*) Other labels to consider:

testing
dependencies
docker
kubernetes
observability
security
code-style
extension-point
refactor
cleanup

(**) Use release-notes label for scenarios like:

New configuration options
Deprecation of configurations
Signature changes to public methods/interfaces
New plugins added or old plugins removed

Codecov Report

Merging #10450 (8e652bd) into master (3687b2b) will increase coverage by 5.04%.
The diff coverage is 77.77%.

@@             Coverage Diff              @@
##             master   #10450      +/-   ##
============================================
+ Coverage     63.23%   68.28%   +5.04%
- Complexity     5905     6104     +199
============================================
  Files          2036     2067      +31
  Lines        110973   111933     +960
  Branches      16892    17027     +135
============================================
+ Hits          70177    76435    +6258
+ Misses        35606    30034    -5572
- Partials       5190     5464     +274

Flag
Coverage Δ

integration1
24.37% <0.00%> (-0.12%)
:arrow_down:

integration2
?

unittests1
67.91% <77.77%> (-0.05%)
:arrow_down:

unittests2
13.92% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...he/pinot/query/runtime/operator/OperatorStats.java
87.75% <75.00%> (-4.93%)
:arrow_down:

...not/query/runtime/operator/MultiStageOperator.java
79.54% <100.00%> (ø)

... and 439 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: KKcorps codecov-commenter

repo: apache/pinot


text:
Bump com.tdunning:t-digest from 3.2 to 3.3

Bumps com.tdunning:t-digest from 3.2 to 3.3.

Changelog
Commits

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 0.00%. Comparing base (59551e4) to head (a2d6504).
Report is 269 commits behind head on master.

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #12882       +/-   ##
=============================================
- Coverage     61.75%    0.00%   -61.76%
=============================================
  Files          2436     2427        -9
  Lines        133233   132771      -462
  Branches      20636    20557       -79
=============================================
- Hits          82274        0    -82274
- Misses        44911   132771    +87860
+ Partials       6048        0     -6048

Flag
Coverage Δ

custom-integration1
?

integration
0.00% <ø> (-0.01%)
:arrow_down:

integration1
?

integration2
0.00% <ø> (ø)

java-11
?

java-21
0.00% <ø> (-61.63%)
:arrow_down:

skip-bytebuffers-false
0.00% <ø> (-61.75%)
:arrow_down:

skip-bytebuffers-true
0.00% <ø> (-27.73%)
:arrow_down:

temurin
0.00% <ø> (-61.76%)
:arrow_down:

unittests
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Please take a look at #7076. We didn't upgrade it because the accuracy has regression

authors: Jackie-Jiang codecov-commenter xiangfu0


text:
Clean Google Dependencies

TODO

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 0.00%. Comparing base (59551e4) to head (b763d05).
Report is 537 commits behind head on master.

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #13297       +/-   ##
=============================================
- Coverage     61.75%    0.00%   -61.76%
=============================================
  Files          2436     2459       +23
  Lines        133233   135540     +2307
  Branches      20636    20971      +335
=============================================
- Hits          82274        0    -82274
- Misses        44911   135540    +90629
+ Partials       6048        0     -6048

Flag
Coverage Δ

custom-integration1
?

integration
0.00% <ø> (-0.01%)
:arrow_down:

integration1
?

integration2
0.00% <ø> (ø)

java-11
?

java-21
0.00% <ø> (-61.63%)
:arrow_down:

skip-bytebuffers-false
0.00% <ø> (-61.75%)
:arrow_down:

skip-bytebuffers-true
?

temurin
0.00% <ø> (-61.76%)
:arrow_down:

unittests
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: abhioncbr codecov-commenter


text:
update log4j2.xml to write helix logs into dedicated files

This PR changes log4j2.xml files to write helix logs into dedicated files so logs are more readable.

Codecov Report

Merging #9786 (c9d349b) into master (dc49b2c) will decrease coverage by 48.41%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master    #9786       +/-   ##
=============================================
- Coverage     64.08%   15.67%   -48.42%
+ Complexity     4970      175     -4795
=============================================
  Files          1903     1903
  Lines        102553   102553
  Branches      15605    15605
=============================================
- Hits          65722    16076    -49646
- Misses        32062    85279    +53217
+ Partials       4769     1198     -3571

Flag
Coverage Δ

unittests1
?

unittests2
15.67% <ø> (-0.04%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...src/main/java/org/apache/pinot/sql/FilterKind.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ain/java/org/apache/pinot/core/data/table/Key.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...in/java/org/apache/pinot/spi/utils/BytesUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...n/java/org/apache/pinot/core/data/table/Table.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/org/apache/pinot/core/data/table/Record.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/org/apache/pinot/core/util/GroupByUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/spi/trace/BaseRecording.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/spi/trace/NoOpRecording.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/config/table/FSTType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/config/user/RoleType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 1208 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter zhtaoxiang


text:
Fix bad_weak_ptr when close a ClientConnection during construction

Fixes https://github.com/apache/pulsar-client-cpp/issues/348
Fixes https://github.com/apache/pulsar-client-cpp/issues/349
Motivation
When close is called in ClientConnection's constructor,
shared_from_this() will be called, which results in a
std::bad_weak_ptr error. This error does not happen before
https://github.com/apache/pulsar-client-cpp/pull/317 because
shared_from_this() could only be called when the producers or
consumers field is not empty.
Modifications
Add a ResultException and throw it if there is a failure in
ClientConnection's constructor and catch it in
ConnectionPool::getConnectionAsync. Then retrieve the result and
return the failed future.
Since close is now always called on a constructed ClientConnection
object, remove the 2nd parameter.
In addition, check authentication_ even for non-TLS URLs. Otherwise,
the null authentication will be used to construct CommandConnect.
Add testInvalidPlugin and testTlsConfigError to verify the changes.

@RobertIndie I used an exception-based way to fix this issue now so close is always called on a ClientConnection object that completes the construction successfully.

The Oauth2Test failed, I will check the failed test first.

repo: apache/pulsar-client-cpp


text:
PIP 194 : Pulsar client: seek command add epoch

Status: Proposal
Authors: Qiang Huang
Pull Request:
Mailing List discussion:
Release:

Motivation
Reader belongs to exclusive subscription type, and it uses nonDurable cursor. After receiving messages, Reader will ack cumulatively immediately.
The flowPermits are triggered in multiple scenarios from the client side and it is isolated from seek of Consumer. Therefore, it is possibile that flowPermits will execute after seek from the client side, like the following flow chart.

When handleSeek processing is delay from the server side, the MarkDelete position is modified in a wrong way.
The expected result is that Readercan re-consume messages from mark delete:(1,1) after seek. But it doesn't work.
Pulsar read message and seek position is not a synchronous operation, the seek request can't prevent an in-process entry reading operation. The client-side also has an opportunity to receive messages after the seek position.
Pulsar client make read messages operation and seek position operation synchronized so add an epoch into server and client consumer.  After client reader consumer invoke seek , the epoch increase 1 and send seek  command carry the epoch and then server consumer will update the epoch. When dispatcher messages to client will carry the epoch which the cursor read at the time. Client consumer will filter the send messages command which is smaller than current epoch.
In this way, after the client consumer send seek command successfully, because it has passed the epoch filtering, the consumer will not receive a message with a messageID greater than the user previously seek position.
Current implementation details
CommandSeek Protocal
// Reset an existing consumer to a particular message id
message CommandSeek {
    required uint64 consumer_id = 1;
    required uint64 request_id  = 2;

    optional MessageIdData message_id = 3;
    optional uint64 message_publish_time = 4;
}

CommandMessage
message CommandMessage {
    required uint64 consumer_id       = 1;
    required MessageIdData message_id = 2;
    optional uint32 redelivery_count  = 3 [default = 0];
    repeated int64 ack_set = 4;
    optional uint64 epoch = 5 [default = 0];
}

CommandMessage already add epoch by PIP-84 , when client receive CommandMessage will compare the command epoch and local epoch to handle this command.
Goal
Add epoch into seek command.
API Changes
Protocal change: CommandSeek
// Reset an existing consumer to a particular message id
message CommandSeek {
    required uint64 consumer_id = 1;
    required uint64 request_id  = 2;

    optional MessageIdData message_id = 3;
    optional uint64 message_publish_time = 4;
    optional uint64 consumer_epoch = 5;
}

CommandSeek command add epoch field, when client send seek command to server successfully, the server will change the server consumer epoch to the command epoch. The epoch only can bigger than the old epoch in server. Now the client can filter out the message which contains less consumer epoch.
Implementation

stage 1: Check the current cursor status when handling flowPermits from the server side.
stage 2: Add epoch into seek command, and server update the consumer epoch. It can prevent an in-process entry reading operation after the seek request.

Reject Alternatives
None yet.
Note

Consumer reconnect need reset epoch.

It looks like this PIP can fix the problem https://github.com/apache/pulsar/issues/13788 in the "Reader with seeking" case.

Looks like this PIP can fix the problem https://github.com/apache/pulsar/issues/13788 in the "Reader with seeking" case.
Because of the problem described from that PIP, for the reader, its Ack is out of order.

This PIP should be able to fix #16171.

This PIP should be able to fix #16171.

I believe so.

Cool. This would fix the duplicated message issues when using seek().

authors: HQebupt mattisonchao nodece syhily

repo: apache/pulsar


text:
[Bug]  Delayed message delivery receive duplicate messages when unload topic

Search before asking

[X] I searched in the issues and found nothing similar.

Version
2.7.5 / 2.8.4 /master
Minimal reproduce step

Run the code for 20 minutes

public static void main(String[] args) {
        String url = "pulsar://10.98.242.210:6650";
        String t = "persistent://components/test/test_deliver";
        new Thread(() -> {
            try {
                PulsarClient client = PulsarClient.builder()
                    .serviceUrl(url)
                    .ioThreads(1)
                    .listenerThreads(2)
                    .build();
                Producer<String> producerA = client.newProducer(Schema.STRING)
                    .topic(t)
                    .producerName("test")
                    .create();
                Random random = new Random();
                while (true) {
                    String message = String.valueOf(System.currentTimeMillis());
                    producerA.newMessage(Schema.STRING).value(message).deliverAfter(random.nextInt(30), TimeUnit.MINUTES).send();
                    Thread.sleep(10L);
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }).start();
        Set<String> stringSet = new ConcurrentHashSet<>();
        try {
            PulsarClient client = PulsarClient.builder()
                .serviceUrl(url)
                .ioThreads(1)
                .listenerThreads(2)
                .build();
            Consumer consumer = client.newConsumer(Schema.STRING)
                .topic(t)
                .subscriptionName("frozen")
                .subscriptionType(SubscriptionType.Shared)
                .subscribe();
            while (true) {
                Message<String> message = consumer.receive();
                if (stringSet.contains(message.getValue())) {
                    log.info("duplicate messages thread={} message={}", Thread.currentThread().getName(), message.getValue());
                } else {
                    stringSet.add(message.getValue());
                }
                consumer.acknowledge(message);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

After 20 minutes, execute unload cmd:

bin/pulsar-admin topics unload persistent://components/test/test_deliver

See consumer log, duplicate messages

What did you expect to see?
no duplicate messages
What did you see instead?
duplicate messages
Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

cc @coderzc since you're working on delay message heavily recently, can you help with this issue to see if it's a usage issue?

I maybe understand what caused the issue, try set isAckReceiptEnabled(true) to true when create consumer.
            Consumer consumer = client.newConsumer(Schema.STRING)
                .topic(t)
                .subscriptionName("frozen")
                // enable ack receipt
                .isAckReceiptEnabled(true)
                .subscriptionType(SubscriptionType.Shared)
                .subscribe();

Then, try you test again.

@dao-jun  My client is pulsar-client-go. Is there similar consumer option like isAckReceiptEnabled. Maybe AckWithResponse? Could you explain the reason for the duplicate consume?

@dao-jun I tried but it did not solve the promblem.

func test() {
	clientOps := pulsar.ClientOptions{
		URL: "pulsar://localhost:6650",
	}
	client, err := pulsar.NewClient(clientOps)
	if err != nil {
		return
	}
	defer client.Close()
	consumerOps := pulsar.ConsumerOptions{
		Topic:            "persistent://public/default/my-topic",
		SubscriptionName: "test",
		AckWithResponse:  true,
	}
	consumer, err := client.Subscribe(consumerOps)
	if err != nil {
		return
	}
	defer consumer.Close()
	for {
		msg, err := consumer.Receive(context.Background())
		if err != nil {
			return
		}
		// Do your business here
		consumer.Ack(msg)
	}
}

@SennoYuki

The problem is Pulsar optimized consumer.ack.
When you call consumer.ack, the ack request will not send to the broker immediately, but cached in the client. When the cached ack requests reached the threshold, send them to the broker in batch.
If you unload the topic when the ack requests didn't send to the broker, you will receive duplicated messages.

enable AckWithResponse, the consumer will wait until broker received the ack

@dao-jun I think it's only part reson for small batch duplicated messages. But in my scene the duplicated messages is huge. It's not possible causing the 20M and more message duplicated consume. And in topics stats the non ack messages shows smaller than 1k

Few unacked message also have a chance to cause many duplicated messages, you can take a try @SennoYuki

ok @dao-jun , I will retry tomorrow. Hope a good result.

Mark:
The root cause is the conflicts between cursor.readEntries and cursor.asyncReplayEntries.

@dao-jun I retry serveral times . It dose not work

I guess your topic have non-delayed delivery messages and delay delivery messages, if your topic only have delayed delivery messages, it will be work. As I marked before, it's the conflicts between cursor.readEntries and cursor.asyncReplayEntries, one for delayed messages, another fro non-delayed delivery messages. We don't have a suitable solution yet to solve the problem. @SennoYuki

@dao-jun I confirm it dose not contain non-delayed delivery message. In my program any message has more than 5 minute delay time. Sad, I choose Pulsar because it supports delayed messages. Any plan for this bug?

authors: SennoYuki dao-jun jdfrozen tisonkun


text:
Improve branch-2.6 action CI

Is your enhancement request related to a problem? Please describe.
In the current branch-2.6 branch, some test cases are difficult to pass in action CI, but these test cases can be passed multiple local verifications. This may be caused by ignoring the flaky test and not including cherry-pick in the pull request of cherry-pick 2.6.2.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Closed as stale and no one worked on it. Please open a new issue if it's still relevant to the maintained versions.

authors: tisonkun wolfstudy


text:
Migrate the python2.7 to python3.7

Motivation
PIP-56 proposed we should upgrade our python version in the Docker image, we did that in the PR #7963.
But I still find some python2.7 usage in the pulsar repository:
function module:
https://github.com/apache/pulsar/blob/master/pulsar-functions/scripts/python/generate.sh
cpp module:
https://github.com/apache/pulsar/blob/master/pulsar-client-cpp/CMakeLists.txt#L175
https://github.com/apache/pulsar/blob/master/pulsar-client-cpp/python/CMakeLists.txt#L40https://github.com/apache/pulsar/blob/master/pulsar-client-cpp/python/pkg/osx/vagrant-build.sh#L38
Should we remove them as well?

The C++ related build and test scripts rely on the docker image apachepulsar/pulsar-build:ubuntu-16.04.
Dockerfiles under https://github.com/apache/pulsar/blob/master/pulsar-client-cpp/docker directory also need changes.

It's already done by PIP-155.

authors: BewareMyPower zymap


text:
Fix message buffer being copied from direct memory to heap memory

Motivation
When I tested pulsar with AppendIndexMetadataInterceptor configured, i.e.
brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor

I found the heap memory increased very fast so that GC happened frequently. After the analysis from the dump info, I found many messages, which have BrokerEntryMetadata in the head, are stored in heap memory instead of direct memory.
When I removed the above configuration, the heap memory became slow to increase.
Modifications

Copy two buffers into a single buffer in direct memory instead of using CompositeByteBuf. Because when a CompositeByteBuf that has over 1 components goes to BookKeeper's checksum logic, the nioBuffer() method will be called and the CompositeByteBuf's internal buffers will be concatenated into a single buffer in heap memory.
Add the reference count check to ensure that after the change, the input and output buffers of addBrokerEntryMetadata will be released finally after the output buffer is released.
Add a slightly code refactor that makes ManagedLedgerInterceptor#beforeAddEntry's returned value be used.

Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
After this change, the heap memory increasing problem was solved.

Here're the monitor statistics after and before this change. The first case (before 00:15) uses this PR's addBrokerEntryMetadata method.

I used a producer with nearly 200 MB/s write speed. As we can see, the heap memory of second case (after 00:15) increased very fast and the GC happened very frequently.
The difference of these two cases are only the pulsar-common.jar.
I'm not an expert of Netty and also very confused why the CompositeByteBuf make the whole buffer be copied into the heap memory.
Here's a dumped result that was generated by jmap -dump:format=b.

We can see there're many messages in heap memory. The first two bytes are [14, 2], which are equivalent to magicBrokerEntryMetadata. See https://github.com/apache/pulsar/blob/37e3180f06188c4d34cd347d46d66bcde3516445/pulsar-common/src/main/java/org/apache/pulsar/common/protocol/Commands.java#L117

@codelipenghui @hangc0276 @wuzhanpeng  @dockerzhang @aloyszhang PTAL. Also it would be appreciated if you can compare the heap memory usage before and after this PR.
I'm not sure if there's something wrong with my test environment. The VM is AWS EC2 Ubuntu Server 18.04 LTS (HVM) ami-05248307900d52e3a i3.4xlarge.

It looks like the heap memory problem is a BK side problem. I doubt that when LedgerHandle#asyncAddEntry(ByteBuf data, AddCallback cb, Object ctx) accepts a CompositeByteBuf as the data, it will finally concat CompositeByteBuf's buffers to a single buffer in heap memory.

@dlg99  PTAL

Seems the problem is related to the crc32 checksum in the bookkeeper client. Since we have more than 1 component in the CompositeByteBuf, CompositeByteBuf.hasMemoryAddress() will return false, details to see implement at the CompositeByteBuf:
public boolean hasMemoryAddress() {
        switch(this.componentCount) {
        case 0:
            return Unpooled.EMPTY_BUFFER.hasMemoryAddress();
        case 1:
            return this.components[0].buf.hasMemoryAddress();
        default:
            return false;
        }
    }

For the crc32 checksum in the bookkeeper will use nioBuffer() to resume the CRC hash, due to CompositeByteBuf..nioBuffer() will copy the data to the HeapByteBuffer if the component size > 1.
public static int resumeChecksum(int previousChecksum, ByteBuf payload) {
        if (payload.hasMemoryAddress() && (CRC32C_HASH instanceof Sse42Crc32C)) {
            return CRC32C_HASH.resume(previousChecksum, payload.memoryAddress() + payload.readerIndex(),
                payload.readableBytes());
        } else if (payload.hasArray()) {
            return CRC32C_HASH.resume(previousChecksum, payload.array(), payload.arrayOffset() + payload.readerIndex(),
                payload.readableBytes());
        } else {
            return CRC32C_HASH.resume(previousChecksum, payload.nioBuffer());
        }
    }

@codelipenghui Thanks for you help, I'll update the PR description after I changed the beforeAddEntry back.
@eolivelli As what @codelipenghui mentioned, here we cannot avoid copying data because in BookKeeper side CompositeByteBuf#nioBuffer will be called. I think we can open an issue in https://github.com/apache/bookkeeper. So after
BookKeeper side fixed the problem, we could change the code here to avoid copying data.

I've updated the code and PR description, PTAL again. @eolivelli @codelipenghui

@BewareMyPower Could you please help add an issue for tracking the Pulsar side and an issue for tracking the bookkeeper side? Before the 2.8.0 release, we need to fix both of them, this also needs a release of the bookkeeper which contains this fix. I agree with we can copying the data for now to avoid copy data into the JVM,  since this only happens when the broker entry metadata enabled, if this can not be fixed completely, we'd better disable this feature by default in 2.8.0.
@eolivelli WDYT?

@dlg99 please follow up on BK.

@codelipenghui nice catch. The checksum part should be easy to fix, DigestManager.computeDigestAndPackageForSending can check if ByteBuf is an instance of CompositeByteBuf, treat it as Iterable (implemented by CompositeByteBuf), update digest in a loop. Or something along these lines.
PCBC.addEntry is doing something similar with ByteBufList.

since this only happens when the broker entry metadata enabled, if this can not be fixed completely, we'd better disable this feature by default in 2.8.0.

This feature is disabled by default now because the default brokerEntryMetadataInterceptors is empty and managedLedgerInterceptor is null.

BTW, I'm not sure if BK could be released in time so that Pulsar 2.8.0 won't be delayed too long. Currently I suggest applying this PR's patch first. If BK can't be released in time, we could apply the BK's fix to Pulsar 2.8.1.

https://github.com/apache/bookkeeper/pull/2701 should fix it.
Existing tests should be enough, we are not unit-testing memory allocations.
I didn't wait for the test run locally, let's see what CI checks say.

@dlg99 Thanks for your quick fix.
And what I concern is if the PR is merged, when will BK be released and then we can upgrade Pulsar's BK dependency.

@BewareMyPower I think we have a few other interesting changes since bk 4.13.0 to justify 4.13.1 release.
@eolivelli what do you think? We'll need to review pending PRs, what can be merged etc. Release process (+voting etc) easily takes a week.
@BewareMyPower meanwhile, can I ask you to build BK locally and confirm that the fix in fact improves the situation with heap allocations in your scenario? Let's assume this is a pre-req for starting the release process.

@dlg99 I agree with you

@dlg99 OK, I'll verify it soon.

@dlg99 It looks like there's not significant improvement.

I'll add some logs to check if the BK dependency is updated. BTW, my build steps are:

Build latest BK with PR 2701

mvn clean install -DskipTests

Upgrade Pulsar's BK version

<bookkeeper.version>4.14.0-SNAPSHOT</bookkeeper.version>

Build Pulsar

mvn clean install -DskipTests -Pcore-modules

Then upload the distribution/server/target/apache-pulsar-2.8.0-SNAPSHOT-bin.tar.gz to my VM and start it.

@BewareMyPower Looking at original chart, heap utilization used to go up to 4.5G, now about 3G (assuming I interpret these charts correctly, the tests and configuration are are the same etc). I'd expect these to be the same but grow slower and less frequent GC pauses but I have no idea what these charts actually show.
You may need to run a profiler to check where the allocations happen. IIRC JFR can record data over some interval and then JMC can show you where the most allocations happened, with stacktraces etc.

I'll try to debug BK dependencies first. Maybe it was not caused by the checksum.

@dlg99 I just found there's something wrong with @codelipenghui 's analysis, which leads to a result that your PR didn't work.
We can debug BrokerEntryMetadataE2ETest and go to https://github.com/apache/pulsar/blob/5a7ba52193c069bc40705bcdc74287aae1066b17/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/OpAddEntry.java#L123-L128

The original data is io.netty.buffer.CompositeByteBuf
However, duplicateBuffer is io.netty.buffer.UnpooledDuplicatedByteBuf

So the argument type, which is passed to LedgerHandle#asyncAddEntry, is UnpooledDuplicatedByteBuf not CompositeByteBuf.

Though I tried to change the code above to
            ByteBuf duplicateBuffer = data.retainedDuplicate();
            final ByteBuf originalData = data;
            data = duplicateBuffer;

            // internally asyncAddEntry() will take the ownership of the buffer and release it at the end
            addOpCount = ManagedLedgerImpl.ADD_OP_COUNT_UPDATER.incrementAndGet(ml);
            lastInitTime = System.nanoTime();
            ledger.asyncAddEntry(originalData, this, addOpCount);

But it still didn't work. I'll take a look later.

So the argument type, which is passed to LedgerHandle#asyncAddEntry, is UnpooledDuplicatedByteBuf not CompositeByteBuf.

@BewareMyPower We might need to use ByteBuf.unwrap() to access to the wrapped buffer.

I tried to change the code above to
ByteBuf duplicateBuffer = data.retainedDuplicate();
But it still didn't work (even worse). I've added some debug logs to BK:

There's an extra retain, that would also require another release on the buffer.

@BewareMyPower I updated the pr to unwrap DuplicatedByteBuf

@dlg99 Really thanks for your help. It should work now.
But there's still a little problem that is not related to your code change. When I run Pulsar with BK 4.14-SNAPSHOT, it looks like that the JNI library can't be loaded:

WARN  com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider

This is the reason that I cannot get a good test result. However, when I run the Pulsar with BK 4.13, it succeeded to load the JNI library:

INFO  com.scurrilous.circe.checksum.Crc32cIntChecksum - SSE4.2 CRC32C provider initialized

They are on the same VM so I wondered what could cause this?

Also I have a question. Should we add an option to configure that if the CompositeByteBuf could be merged? Because for the platforms that can't load the Circe JNI library, the performance impact that is caused by frequent GC is much more than the data copy in memory. With the Java based CRC32c provider, the merged ByteBuf is still in direct memory but the CompositeByteBuf will call nioBuffer() and copy data into heap memory.

BK 4.14 is basically equals to 4.13 in this scope.
So probably it is a problem about how you have built it locally.
On which platform you are? Arw you on Linux? Which JDK?

@eolivelli I built BK on my MacOS with JDK 8, see https://github.com/apache/pulsar/pull/10330#issuecomment-825801576
And I run the Pulsar with BK 4.14 on AWS EC2 Ubuntu Server 18.04 LTS (HVM) ami-05248307900d52e3a, with JDK 11.

Probably (I am not sure)  the problem is that you have to build BK on Linux, in order to build the CirceChecksum library correctly.
When we release BK, even while using Mace's, we run the build in a docker env, you can use the script you find in the 'dev' folder for instance

@eolivelli Sounds reasonable. I'll give it a shot.

@BewareMyPower can you confirm that the copy to heap issue is resolved by the bookkeeper change?

@dlg99 Yeah, it's resolved. Sorry I forgot to update the result. Here's the compare with the same workload of https://github.com/apache/pulsar/pull/10330#issuecomment-825801576

The heap memory is stable and there's no GC.

Hi, @eolivelli I see https://github.com/apache/bookkeeper/pull/2701 has been merged for a few days. Is it any plan for BookKeeper's release now? Then the Pulsar's BK dependency could be upgraded to fix https://github.com/apache/pulsar/pull/10330.

Can you please start a discussion on dev@bookkeeper.apache.org ?
I am fine with cutting a release

@eolivelli Sorry for the late reply because I'm on vacation now. I sent an email just now, PTAL.

authors: BewareMyPower codelipenghui dlg99 eolivelli merlimat


text:
Bump netty version to 4.1.75

Motivation
Changelog: https://netty.io/news/2022/03/10/4-1-75-Final.html
Modifications

Upgrade Netty from 4.1.74.Final to 4.1.75.Final
Netty 4.1.75.Final depends on netty-tc-native 2.0.50, also updates

/pulsarbot run-failure-checks

https://github.com/grpc/grpc-java/issues/8981

we need to update bookkeeper to bump netty version, closed this for now.


text:
[Improve][txn] Add admin api updateTransactionCoordinatorNumber

Motivation & Modification
Add admin api updateTransactionCoordinatorNumber to change the partition number of tc
Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API: (yes / no)
The schema: (yes / no / don't know)
The default values of configurations: (yes / no)
The wire protocol: (yes / no)
The rest endpoints: (yes / no)
The admin cli options: (yes / no)
Anything that affects deployment: (yes / no / don't know)

Documentation
Check the box below or label this PR directly.
Need to update docs?

[ ] doc-required
(Your PR needs to update docs and you will update later)

[x] no-need-doc
(Please explain why)

[ ] doc
(Your PR contains doc changes)

[ ] doc-added
(Docs have been already added)

/pulsarbot run-failure-checks


text:
[improve][websocket] Add ping support

Motivation
When the WebSocket connection is idle timeout, the server disconnects the client. We need to keep alive between the client and the server.
There are two options:

Increasing the idle timeout, but be very unfriendly to users who are not sensitive to time
Sending the ping frame to the client

Some client(browser) doesn't support sending the ping frame to the server, which causes idle timeout, so we only add enable ping feature on the server.
Note: The Jetty automatically processes ping/pong frames from the client by default.
Modifications

Implement sending the ping frame to the client
Add webSocketPingDurationSeconds config to enable the ping feature

Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository:

Ping @poorbarcode

Codecov Report

Merging #19203 (27b92cd) into master (246c270) will decrease coverage by 5.74%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master   #19203      +/-   ##
============================================
- Coverage     43.01%   37.27%   -5.75%
+ Complexity    10167     1990    -8177
============================================
  Files           747      209     -538
  Lines         72255    14442   -57813
  Branches       7786     1578    -6208
============================================
- Hits          31080     5383   -25697
+ Misses        37577     8477   -29100
+ Partials       3598      582    -3016

Flag
Coverage Δ

unittests
37.27% <ø> (-5.75%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...g/apache/pulsar/client/impl/ConnectionHandler.java
50.00% <0.00%> (-5.32%)
:arrow_down:

.../org/apache/pulsar/client/impl/ConnectionPool.java
37.43% <0.00%> (-1.03%)
:arrow_down:

.../timeout/TransactionTimeoutTrackerFactoryImpl.java

...lsar/broker/loadbalance/BrokerFilterException.java

...tion/buffer/impl/InMemTransactionBufferReader.java

...r/broker/service/schema/SchemaRegistryService.java

...keeper/mledger/impl/cache/PendingReadsManager.java

...org/apache/bookkeeper/mledger/util/RangeCache.java

...n/java/org/apache/pulsar/broker/service/Topic.java

...rg/apache/pulsar/metadata/impl/batching/OpGet.java

... and 530 more

/pulsarbot rerun-failure-checks

/pulsarbot rerun-failure-checks

/pulsarbot rerun-failure-checks

/pulsarbot rerun-failure-checks

Remove release/2.9.5, using #19245 instead.

authors: codecov-commenter nodece


text:
Update functions-quickstart.md

classname -> className
<--
Contribution Checklist

Name the pull request in the form "[Issue XYZ][component] Title of the pull request", where XYZ should be replaced by the actual issue number.
Skip Issue XYZ if there is no associated github issue for this pull request.
Skip component if you are unsure about which is the best component. E.g. [docs] Fix typo in produce method.

Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.

Each pull request should address only one issue, not mix up code from multiple issues.

Each commit in the pull request has a meaningful commit message

Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.

(The sections below can be removed for hotfixes of typos)
-->
(If this PR fixes a github issue, please add Fixes #<xyz>.)
Fixes #
(or if this PR is one task of a github issue, please add Master Issue: #<xyz> to link to the master issue.)
Master Issue: #
Motivation
Explain here the context, and why you're making that change. What is the problem you're trying to solve.
Modifications
Describe the modifications you've done.
Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API: (yes / no)
The schema: (yes / no / don't know)
The default values of configurations: (yes / no)
The wire protocol: (yes / no)
The rest endpoints: (yes / no)
The admin cli options: (yes / no)
Anything that affects deployment: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

@yjmyzz which version of pulsar are you using? the option has been changed to lower cases.

authors: sijie yjmyzz


text:
事务消息回查

rocketmq 4.6.0的版本
在写事务消息的demo时发现几个问题，求解答
1、producer发送几条pre消息，本地事务返回unknown的情况下 ，只有部分消息进行了回查的回调
2、同一事务消息下，checkLocalTransaction模拟返回unkonw的情况下 ，没有再次进行事务消息的回查调用，默认不是15次吗？

@xiebin2015 首先确定是不是同一个group启动了客户端；其次可以根据服务端transaction.log的日志看下是不是发送了回查请求。最好是能够提供更多的信息，来一起看下这个问题。

你好 ，我是在一个jvm里面同时有producer和comsumer，就一个demo项目，本地事务处理模拟未决事务返回unkown，设置是10秒回查一次 ，第一次进行回查进入到check的方法了，在check方法里面也模拟未决事务返回unkown，此时没有再进行回查。transaction.log日志中也没看到有 回查的日志具体如下
brocker端每10秒有检查 ，但从日志上看 ，没有prepare message
------------------ 原始邮件 ------------------
发件人: "Heng Du"<notifications@github.com>;
发送时间: 2020年5月7日(星期四) 上午8:48
收件人: "apache/rocketmq"<rocketmq@noreply.github.com>;
抄送: "一页书"<407708958@qq.com>;"Mention"<mention@noreply.github.com>;
主题: Re: [apache/rocketmq] 事务消息回查 (#1974)
@xiebin2015 首先确定是不是同一个group启动了客户端；其次可以根据服务端transaction.log的日志看下是不是发送了回查请求。最好是能够提供更多的信息，来一起看下这个问题。
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

你好 ，我补充了详细的日志信息 如下：
总共检查了两次 ，第一次是因为本地事务返回unkown进行检查回调check方法，在check方法里面继续返回unkown，确实也检查了一次 ，但是这次检查直接回滚了事务消息
------------------ 原始邮件 ------------------
发件人: "Heng Du"<notifications@github.com>;
发送时间: 2020年5月7日(星期四) 上午8:48
收件人: "apache/rocketmq"<rocketmq@noreply.github.com>;
抄送: "一页书"<407708958@qq.com>;"Mention"<mention@noreply.github.com>;
主题: Re: [apache/rocketmq] 事务消息回查 (#1974)
Closed #1974.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

总共检查了两次 ，第一次是因为本地事务返回unkown进行检查回调check方法，在check方法里面继续返回unkown，确实也检查了一次 ，但是这次检查直接回滚了事务消息

可以看下代码吗？或者使用不同group，确定一下在其他地方没有rollback的操作。

生产者
消费者
本地事务
本地事务回查
消息生产者和消费者都在同一个项目里面，生产者和消费者都只有一个
------------------ 原始邮件 ------------------
发件人: "Heng Du"<notifications@github.com>;
发送时间: 2020年5月7日(星期四) 上午10:21
收件人: "apache/rocketmq"<rocketmq@noreply.github.com>;
抄送: "一页书"<407708958@qq.com>;"Mention"<mention@noreply.github.com>;
主题: Re: [apache/rocketmq] 事务消息回查 (#1974)
总共检查了两次 ，第一次是因为本地事务返回unkown进行检查回调check方法，在check方法里面继续返回unkown，确实也检查了一次 ，但是这次检查直接回滚了事务消息
可以看下代码吗？或者使用不同group，确定一下在其他地方没有rollback的操作。
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

您好 ，我发现我修改了生产组的组名，以及topic名称之后，能达到我的预期结果了，这其中有什么关键因素吗？

您好 ，我发现我修改了生产组的组名，以及topic名称之后，能达到我的预期结果了，这其中有什么关键因素吗？

使用独立的组名，避免回查到其他非事务客户端

应该是没有重名的组名
------------------ 原始邮件 ------------------
发件人: "Heng Du"<notifications@github.com>;
发送时间: 2020年5月9日(星期六) 下午3:20
收件人: "apache/rocketmq"<rocketmq@noreply.github.com>;
抄送: "一页书"<407708958@qq.com>; "Mention"<mention@noreply.github.com>;
主题: Re: [apache/rocketmq] 事务消息回查 (#1974)
您好 ，我发现我修改了生产组的组名，以及topic名称之后，能达到我的预期结果了，这其中有什么关键因素吗？
使用独立的组名，避免回查到其他非事务客户端
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

@xiebin2015  感觉您如果可以复现之前的情况 push出transaction.log文件 这样更好分析您出现的问题了

🧐

我也遇到过类似的问题。分析源码后，发现其实是transientStorePoolEnable配置为true对事务消息的判断有影响。希望对你排查这个问题有帮助。

authors: Garlichead MnameHZJ duhenglucky xiebin2015

repo: apache/rocketmq


text:
修复事务消息过大，压缩导致的循环check

What is the purpose of the change
XXXXX
Brief changelog
XX
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage decreased (-0.07%) to 50.542% when pulling f8eb4d80a2d56d6147add795963f4931fa0fa424 on guyuezheng:master into 75fa7fb131ef11a5a3311a3d39dcb8204614ab9a on apache:develop.

authors: coveralls guyuezheng