text:
sem: don't clear waitobj when do sem recover

Summary
This commit fixed data abort issue when task restart in wait sem status.
If delete waitobj in the sem recover function, then we will get the wrong task list when remove the task from task list.
A new test case was added for this issue,  apply the following patch and run ostest can reproduce it.
https://github.com/apache/incubator-nuttx-apps/pull/1342
Impact
NA
Testing
ostest with sabre-6quad:nsh

This commit fixed data abort issue when task restart in wait sem status.

@zyfeier
How can we reproduce the data abort?

This commit fixed data abort issue when task restart in wait sem status.

@zyfeier How can we reproduce the data abort?

Hi, masayuki2009:
I added a test case for this issue, you can use this patch to reproduce.
https://github.com/apache/incubator-nuttx-apps/pull/1342

This commit fixed data abort issue when task restart in wait sem status.
If delete waitobj in the sem recover function, then we will get the wrong task list when remove the task from task list.

@zyfeier
Please add the above summary to the commit log.

This commit fixed data abort issue when task restart in wait sem status.
If delete waitobj in the sem recover function, then we will get the wrong task list when remove the task from task list.

@zyfeier Please add the above summary to the commit log.

Done

Done

@zyfeier
Hmm, I can not see the updates.

A new test case was added for this issue, apply the following patch and run ostest can reproduce it.
https://github.com/apache/incubator-nuttx-apps/pull/1342

@zyfeier
Hmm, I tried the above PR with sabre-6quad:nsh on qemu-6.2 but I can not reproduce the data abort.

A new test case was added for this issue, apply the following patch and run ostest can reproduce it.
apache/incubator-nuttx-apps#1342

@zyfeier Hmm, I tried the above PR with sabre-6quad:nsh on qemu-6.2 but I can not reproduce the data abort.
apps:
$ git log --pretty=oneline
c2912b33a1efb58491ee17ba0e91c12b963170f5 (HEAD -> ostest, origin/ostest) ostest: add semaphore wait case for task restart
ab7cb90f88c3207c6764e71a0fd3c108ad03d5e1 (origin/master, origin/HEAD, master) ftpd: Fixed compiler warnings.

nuttx:
$ git log --pretty=oneline
130b196876bbebc5286d8b68987258d99d36531e (HEAD -> master, origin/master, origin/HEAD) Refine how to specify iob and ramlog data section
9bc47ecdce7959bc50ce7ea8f3d05d4cf626c1ae Added lock in ifr ioctl calls.

@masayuki2009
Could you check  about your syslog configure, the default configure is RAM log device, you can disable ram log device support and choose default syslog device, then try it again.

Could you check about your syslog configure, the default configure is RAM log device, you can disable ram log device support and choose default syslog device, then try it again.

@zyfeier
Though I disabled the ramlog, I still can not reproduce the data abort.
Could you tell us your environments such as (host machine, host OS, gcc version, qemu version)?
Also, please update the commit log.

Could you check about your syslog configure, the default configure is RAM log device, you can disable ram log device support and choose default syslog device, then try it again.

@zyfeier Though I disabled the ramlog, I still can not reproduce the data abort. Could you tell us your environments such as (host machine, host OS, gcc version, qemu version)?
Also, please update the commit log.

@masayuki2009
This bug was introduced by #7159， it's not included in your branch.

@zyfeier

This bug was introduced by https://github.com/apache/incubator-nuttx/pull/7159，

If so, please add such important information to the commit log as well.

it's not included in your branch.

Hmm, I still do not understand what you want to say, because the above PR was merged to the upstream on 2022/09/22.

Hmm, I still do not understand what you want to say, because the above PR was merged to the upstream on 2022/09/22.

@masayuki2009
According to the log you provided, this PR is not included.
nuttx:
$ git log --pretty=oneline
130b196876bbebc5286d8b68987258d99d36531e (HEAD -> master, origin/master, origin/HEAD) Refine how to specify iob and ramlog data section
9bc47ecdce7959bc50ce7ea8f3d05d4cf626c1ae Added lock in ifr ioctl calls.

@zyfeier
OK, I understand your master branch is too old to test.
Now I can reproduce the data abort.
arm_dataabort: Data abort. PC: 10814e4c DFAR: 00000004 DFSR: 0000080d
up_assert: Assertion failed at file:armv7-a/arm_dataabort.c line: 161 task: ostest
arm_registerdump: R0: 10826550 R1: 00000000 R2: 00000000  R3: 00000004
arm_registerdump: R4: 10826550 R5: 10820a30 R6: 0000005f  FP: 10824556
arm_registerdump: R8: 10824528 SB: 10818016 SL: 00000000 R11: 00000000
arm_registerdump: IP: 00000000 SP: 10826498 LR: 10814e2d  PC: 10814e4c
arm_registerdump: CPSR: 800000ff
arm_dump_stack: IRQ Stack:
arm_dump_stack: sp:     10826320
arm_dump_stack:   base: 1081e5a8
arm_dump_stack:   size: 00000800
arm_dump_stack: ERROR: IRQ Stack pointer is not within the stack
arm_stackdump: 1081ed00: deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef 10822660 1080c85b
arm_stackdump: 1081ed20: 10822660 10804b57 00000000 000001df 10822660 00000001 00000006 1080ce77
arm_stackdump: 1081ed40: 00000000 1081e494 10824120 1080c85b 10824120 10804b57 1080d601 1081fefc
arm_stackdump: 1081ed60: 00000000 00000057 1081f7ac 1080d21f 02098000 00000001 1081f7a8 1080452d
arm_stackdump: 1081ed80: 10820ec0 108011eb 00a00000 10800cfd 00000057 1081f85c 1081f858 10800955
arm_dump_stack: User Stack:
arm_dump_stack: sp:     10826498
arm_dump_stack:   base: 10824560
arm_dump_stack:   size: 00001fb0
arm_stackdump: 10826480: 10818016 00000000 00000000 00000000 10814e4c 800000ff 00000005 10820a30
arm_stackdump: 108264a0: 108184ff 10814eb7 00000005 108104b5 10818c0c 00000000 00007fff 108264c8
arm_stackdump: 108264c0: 108205e4 1080f5a7 3f7dee80 00000002 0000003c 3f7d9ab0 000053b0 3f7d9ad0
arm_stackdump: 108264e0: 00000000 1080f49d 00000005 10824528 00000000 00000000 00000000 10802f9b
arm_showtasks:    PID    PRI     STACK      USED   FILLED    COMMAND
arm_showtasks:   ----   ----      2048       144     7.0%    irq
arm_dump_task:      0      0      1000       376    37.6%    Idle Task
arm_dump_task:      1    192      1992       408    20.4%    hpwork 0x1081fed8
arm_dump_task:      2    100      2000       776    38.8%    nsh_main
arm_dump_task:      3    100      2008       712    35.4%    ostest
arm_dump_task:      4    100      8112       932    11.4%    ostest Arg1 Arg2 Arg3 Arg4
arm_dump_task:      5    100      8080       696     8.6%    ostest This is argument 1 Argument 2 here Lastly, the 3rd argument

@zyfeier
This bug was introduced by https://github.com/apache/incubator-nuttx/pull/7159，
If so, please add such important information to the commit log as well.

@zyfeier
Please update the commit log.


text:
odd SSL error when passing large payloads

try calling ow.packges.get({name:"foo", junk: "SOME VERY LARGE PAYLOAD"})
if the payload is too big (in my case, 330k), you will get back an odd SSL error:
Error encountered calling OpenWhisk: Error: error:0906D06C:PEM routines:PEM_read_bio:no start line"

@starpit is still a problem?
Why are you passing large data on a get?
Maybe a different use case a post (50mb)  or put (1mb)?

source: gharchive/issue

authors: csantanapr starpit

repo: apache/incubator-openwhisk-client-js

license_type: permissive

license_source: github-api


text:
Add the installation script to install openwhisk

The source code to be released needs an installation script for the user to deploy the openwhisk environment.
One candidate of the script can be a vagrant script.

On the page of README.md of this project, you can check the status of each openwhisk project to be released: https://github.com/apache/incubator-openwhisk-release#project-status. We are instantly monitoring all the Travis builds including the deployment of openwhisk and the test of openwhisk We decided to close this issue for openwhisk-release.

repo: apache/incubator-openwhisk-release


text:
Remove pekko prefix from modules

This PR makes pekko-http reflect the same dir structure as every other pekko project (including pekko core as of https://github.com/apache/incubator-pekko/pull/71). Unlike pekko-core the changes are much more trivial, aside from the dir renames only paradox markdown doc links and build.sbt needed to be updated.
In regards to published modules with publishLocal
Akka ASL 2.0 with org com.typesafe.akka
akka-http-bom_2.13
akka-http-caching_2.13
akka-http-core_2.13
akka-http-jackson_2.13
akka-http-scalafix-rules_2.13
akka-http-spray-json_2.13
akka-http-testkit_2.13
akka-http-xml_2.13
akka-http2-support_2.13
akka-http_2.13
akka-parsing_2.13

Pekko-http (this branch/PR) with org org.apache.pekko
pekko-http-bom_2.13
pekko-http-caching_2.13
pekko-http-core_2.13
pekko-http-jackson_2.13
pekko-http-scalafix-rules_2.13
pekko-http-spray-json_2.13
pekko-http-testkit_2.13
pekko-http-xml_2.13
pekko-http2-support_2.13
pekko-http_2.13
pekko-parsing_2.13

So everything is working as expected, exceptions have been noted as comments below.

should we also remove the http prefix? Since the pekko-http prefix is implied, then don't we want core instead of http-core, caching instead of http-caching, etc.? In your current change it looks like we're doing this already with bill-of-materials and parsing.

For me the redundant part is the root project (i.e. pekko), not what the project is implementing. The changes to bill-of-materials was done just to make it consistent with pekko core and the change to parsing wasn't exceptional in any way (it was pekko-parsing before, not pekko-http-parsing.
While I think I understand the sentiment I believe this is going too far, most importantly since there is a relation of the name to the published artifact if we hypothetically rename http-core to core (for example), the artifact still needs to be called pekko-http-core and not pekko-core (because otherwise it can conflict with other pekko repos under the same org.apache.pekko). Even the current proposed changes is stretching it a bit for current committers and to me what is making it acceptable is that at least for the modules that create artifacts (which bill-of-materials/root/docs are not) there is a logical relation/computation to the artifact

Yeah, the fact that there are modules like parsing that aren't specifically for http functionality makes it weird to remove the http prefix everywhere, ok. I was thinking that the module name would always have akka-http- prefixed, but that won't work if we have modules that aren't http.

source: gharchive/pull-request

authors: gmethvin mdedetrich

repo: apache/incubator-pekko-http


text:
Adding a dedicated  stylesheet to the page layout

Adding a dedicated stylesheet option to the page layout to support dedicated styling for pages.

I don't quite get how this works. There is already material.custom.stylesheet. So far, the only examples I have seen of these props being set - sets them for the whole build. I'm not sure how they can be set on a per page basis.
What if page.properties has both stylesheet properties set - won't that lead to 2 <link rel="stylesheet" entries?
I still have a big preference for https://github.com/apache/incubator-pekko-sbt-paradox/issues/84 and gutting this project. It is far too complicated to vote on pekko-sbt-paradox releases and then have to uptake the releases in so many places.

I don't quite get how this works. There is already material.custom.stylesheet. So far, the only examples I have seen of these props being set - sets them for the whole build. I'm not sure how they can be set on a per page basis. What if page.properties has both stylesheet properties set - won't that lead to 2 <link rel="stylesheet" entries?
I still have a big preference for #84 and gutting this project. It is far too complicated to vote on pekko-sbt-paradox releases and then have to uptake the releases in so many places.

These props can be set also at markdown level using the following approach:
---
material.custom.stylesheet: ../assets/stylesheet/style.css
---

## Apache Pekko

That said, I agree that we need a more sustainable way of updating the static assets. I'm proceeding by commenting #84 with the next steps.

Thanks, I wasn't aware that you can set properties that way (in the markdown file itself).
With that approach, do we need material.custom.page-stylesheet ? I'm still concerned that can end up with 2 <link rel="stylesheet" entries in the HTML.

Thanks, I wasn't aware that you can set properties that way (in the markdown file itself).
With that approach, do we need material.custom.page-stylesheet ? I'm still concerned that can end up with 2 <link rel="stylesheet" entries in the HTML.

Yes, we gonna end up with +1 <link rel="stylesheet" in the HTML. At the moment, the page is rendered as:
...
<meta name="description" content="incubator-pekko-site">
<link rel="shortcut icon" href="assets/images/pekko_favicon.png">
<title>Apache Pekko</title>
<link rel="stylesheet" href="assets/stylesheets/application.451f80e5.css">
<link rel="stylesheet" href="assets/stylesheets/application-palette.22915126.css">
<link rel="stylesheet" href="lib/material__tabs/dist/mdc.tabs.min.css">
<link rel="stylesheet" href="lib/prettify/prettify.css">
<script src="assets/javascripts/modernizr.1aa3b519.js"></script>
<link rel="stylesheet" href="assets/fonts/font-awesome.css">
<link rel="stylesheet" href="assets/fonts/material-icons.css">
<link rel="stylesheet" href="assets/stylesheets/paradox-material-theme.css">
<link rel="stylesheet" href="assets/stylesheets/pekko-theme.css">
</head>
...

This change, in combination with a home.md specifying the following:
---
material.custom.page-stylesheet: ../assets/stylesheet/home.css
---

## Apache Pekko

The rendered result will be:
...
<meta name="description" content="incubator-pekko-site">
<link rel="shortcut icon" href="assets/images/pekko_favicon.png">
<title>Apache Pekko</title>
<link rel="stylesheet" href="assets/stylesheets/application.451f80e5.css">
<link rel="stylesheet" href="assets/stylesheets/application-palette.22915126.css">
<link rel="stylesheet" href="lib/material__tabs/dist/mdc.tabs.min.css">
<link rel="stylesheet" href="lib/prettify/prettify.css">
<script src="assets/javascripts/modernizr.1aa3b519.js"></script>
<link rel="stylesheet" href="assets/fonts/font-awesome.css">
<link rel="stylesheet" href="assets/fonts/material-icons.css">
<link rel="stylesheet" href="assets/stylesheets/paradox-material-theme.css">
<link rel="stylesheet" href="assets/stylesheets/pekko-theme.css">
<link rel="stylesheet" href="assets/stylesheet/home.css">
...

This would allow us to keep the pekko-theme.css as a common stylesheet for common elements such as footer,header, nav bar.
On top of that, we can optionally specify a <page_name>.css to be used for styles specific to a page i.e.: homepage
This approach might be benefcial because:

you keep styling across pages in pekko-theme.css
<page>.css for page-related sytling
Avoid duplication between pekko-theme.css and page.css styles.
Avoid to override a layout everytime you need page-specific style, see: https://github.com/apache/incubator-pekko-site/pull/47/files#diff-466bb92234fde71107d802b3aee2a64ecd1c40ec841da5176623f88f9d6a9f11R1

authors: pjfanning samueleresca

repo: apache/incubator-pekko-sbt-paradox


text:
Skip loading columns not in the schema

Description
Currently, when a column is added to the schema, server will generate default columns if the segment does not have the new column. But when a column is removed from the schema, the server still load all the columns within the segments. This could cause problem for select * queries because different segments might contain different columns.
With the change, server will skip loading the columns not in the schema, and all the segments should have the same set of columns loaded.

Codecov Report

Merging #6649 (92fbf09) into master (1beaab5) will decrease coverage by 22.35%.
The diff coverage is 42.93%.

@@             Coverage Diff             @@
##           master    #6649       +/-   ##
===========================================
- Coverage   66.44%   44.09%   -22.36%
===========================================
  Files        1075     1360      +285
  Lines       54773    66545    +11772
  Branches     8168     9701     +1533
===========================================
- Hits        36396    29345     -7051
- Misses      15700    34741    +19041
+ Partials     2677     2459      -218

Flag
Coverage Δ

integration
44.09% <42.93%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...ot/broker/broker/AllowAllAccessControlFactory.java
100.00% <ø> (ø)

...t/broker/broker/BasicAuthAccessControlFactory.java
0.00% <0.00%> (ø)

.../helix/BrokerUserDefinedMessageHandlerFactory.java
52.83% <0.00%> (-13.84%)
:arrow_down:

...org/apache/pinot/broker/queryquota/HitCounter.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...che/pinot/broker/queryquota/MaxHitRateTracker.java
0.00% <0.00%> (ø)

...ache/pinot/broker/queryquota/QueryQuotaEntity.java
0.00% <0.00%> (-50.00%)
:arrow_down:

...ker/routing/instanceselector/InstanceSelector.java
100.00% <ø> (ø)

...ceselector/StrictReplicaGroupInstanceSelector.java
0.00% <0.00%> (ø)

...roker/routing/segmentpruner/TimeSegmentPruner.java
0.00% <0.00%> (ø)

...roker/routing/segmentpruner/interval/Interval.java
0.00% <0.00%> (ø)

... and 1386 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b2d716d...92fbf09. Read the comment docs.

Does this imply that we now support dropping columns?

@snleee Not fully supported, but can reduce the inconsistency. Dropping columns won't take effect on consuming segments.

authors: Jackie-Jiang codecov-io snleee

repo: apache/incubator-pinot


text:
improve robustness of basic auth (tls) realtime integration tests

Description
Add checked wait to BasicAuth(Tls)RealtimeIntegrationTest to reduce flakiness on slow environments.
Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

[ ] Yes (Please label as backward-incompat, and complete the section below on Release Notes)

Does this PR fix a zero-downtime upgrade introduced earlier?

[ ] Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

Does this PR otherwise need attention when creating release notes?

[ ] Yes (Please label this PR as release-notes and complete the section on Release Notes)

Release Notes
none
Documentation
none

Codecov Report

Merging #7038 (9d0143b) into master (6aac7dd) will decrease coverage by 7.94%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #7038      +/-   ##
============================================
- Coverage     73.35%   65.41%   -7.95%
  Complexity       12       12
============================================
  Files          1453     1453
  Lines         72027    72027
  Branches      10428    10428
============================================
- Hits          52837    47115    -5722
- Misses        15663    21507    +5844
+ Partials       3527     3405     -122

Flag
Coverage Δ

integration
?

unittests
65.41% <ø> (-0.02%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...a/org/apache/pinot/minion/metrics/MinionMeter.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../apache/pinot/common/metrics/BrokerQueryPhase.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../apache/pinot/minion/metrics/MinionQueryPhase.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...pache/pinot/common/utils/grpc/GrpcQueryClient.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...pinot/minion/exception/TaskCancelledException.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...t/core/startree/plan/StarTreeDocIdSetPlanNode.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../core/startree/plan/StarTreeTransformPlanNode.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...core/startree/plan/StarTreeProjectionPlanNode.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...t/minion/executor/MinionTaskZkMetadataManager.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...startree/executor/StarTreeAggregationExecutor.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 338 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6aac7dd...9d0143b. Read the comment docs.

authors: apucher codecov-commenter


text:
[REEF-679] Made IContext immutable

This makes all fields in IContext immutable. This change is reflected in the implementation classes. Also, those are now internal, which prompted a change to (I)FailedEvaluator which wasn't coded against the context interfaces.
JIRA:
REEF-679

Done with a first pass.

@afchung Thanks! I have addressed the parts of your feedback I could. Please have another look.

@afchung Hmm, I have addressed REEF-666 as well. Seems OK, as we did changes in that file anyhow in here.

@markusweimer Left a few more minor comments. I think it may be worthwhile to investigate whether the Clr2Java objects can change in the unmanaged layer of the code such that we don't need to instantiate a new object on every get call, but this might be another item and may require a look from @beysims or @yingdachen.

@afchung Thanks. I filled REEF-772 to track down the Clr2Java properties. We won't close on this without input from @beysims or @yingdachen. Hence, I suggest we move forward with this PR as a net-positive and clean those up later.
I'll act on your other feedback later today.

@markusweimer sounds good.

@afchung I addressed your feedback, please have another look.

Looks good, will test and merge.

authors: afchung markusweimer

repo: apache/incubator-reef


text:
[REEF-828] Fix name typos of Class/Functions in REEF.NET IO

This PR renames the following functions and class.

GetByteSerilizerConfigString -> GetByteSerializerConfigString
GetRowSerilizerConfigString -> GetRowSerializerConfigString
FilePathsForPatitions -> FilePathsForPartitions

JIRA:
REEF-828
Pull Request:
This closes #

I'll test and merge

Thank you.

authors: dongjoon-hyun tcNickolas


text:
[SCB-656]edge service should return back the providers error code

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Format the pull request title like [SCB-XXX] Fixes bug in ApproximateQuantiles, where you replace SCB-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Run mvn clean install to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage increased (+0.02%) to 87.389% when pulling eea2b0ff6a1a62057ec0454b3d3428f0a72e63e0 on liubao68:edge-code into 8f420f1bab6f015062e772357669bdc7b248021f on apache:master.


text:
sharding-sphere is 4.0.0.M1-SNAPSHOT cann't configure  multiple config-sharding files.

The version using sharding-sphere is 4.0.0.M1-SNAPSHOT。
I configure a config-sharding file and the query is ok，but I configure multiple config-sharding files, execute sql query with java code will report an error：
Exception in thread "main" java.sql.SQLException: Unknown exception: null
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2531)
at com.mysql.jdbc.ConnectionImpl.configureClientCharacterSet(ConnectionImpl.java:1765)
at com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3364)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2283)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2069)
at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:794)
at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:44)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:389)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:399)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:325)
at java.sql.DriverManager.getConnection(DriverManager.java:579)
at java.sql.DriverManager.getConnection(DriverManager.java:221)
at com.zhongan.dmds.sharding_sphere.QueryTest.main(QueryTest.java:32)

I can't reproduce this error. Please provide more information like: config file name/config file.

first yaml config file :
schemaName: smp_gr_bcp_dev
dataSources:
  ds_smp_gr_bcp_dev_0:
	url: jdbc:mysql://url:3306/database_00?useSSL=false
	username: username
	password: password
	maxPoolSize: 5
	minPoolSize: 2
  ds_smp_gr_bcp_dev_1:
	url: jdbc:mysql://url:3306/database_01?useSSL=false
	username: username
	password: password
	maxPoolSize: 5
	minPoolSize: 2
shardingRule:
  tables:
	bcp_router:
	  actualDataNodes: ds_smp_gr_bcp_dev_0.bcp_router_000${0..1},ds_smp_gr_bcp_dev_1.bcp_router_000${2..3}
	  databaseStrategy:
		inline:
		  shardingColumn: src_id
		  algorithmExpression: ds_smp_gr_bcp_dev_${(src_id.hashCode().abs() % 4).intdiv(2)}
	  tableStrategy:
		inline:
		  shardingColumn: src_id
		  algorithmExpression: bcp_router_${"000" + (src_id.hashCode().abs() % 4)}
  defaultDataSourceName: ds_smp_gr_bcp_dev_0
  defaultDatabaseStrategy:
	none:
  defaultTableStrategy:
	none:

second yaml config file :
schemaName: smp_gr_customer_dev
dataSources:
  ds_smp_gr_customer_dev_0:
	url: jdbc:mysql://url:3306/database_00?useSSL=false
	username: username
	password: password
	maxPoolSize: 5
	minPoolSize: 2
  ds_smp_gr_customer_dev_1:
	url: jdbc:mysql://url:3306/database_01?useSSL=false
	username: username
	password: password
	maxPoolSize: 5
	minPoolSize: 2
shardingRule:
  tables:
	trans_router:
	  actualDataNodes: ds_smp_gr_customer_dev_0.trans_router_000${0..1},ds_smp_gr_customer_dev_1.trans_router_000${2..3}
	  databaseStrategy:
		inline:
		  shardingColumn: trans_no
		  algorithmExpression: ds_smp_gr_customer_dev_${(trans_no.hashCode().abs() % 4).intdiv(2)}
	  tableStrategy:
		inline:
		  shardingColumn: trans_no
		  algorithmExpression: trans_router_${"000" + (trans_no.hashCode().abs() % 4)}
  defaultDataSourceName: ds_smp_gr_customer_dev_0
  defaultDatabaseStrategy:
	none:
  defaultTableStrategy:
	none:

You didn't show the config file name. Can you explicitly show them?
Do you mean each config can work well separately?
Please set sql.show: true in server.yaml and show me the log.

first config filename is config-sharding-sompo-dev____smp_gr_bcp_dev.yaml.
second config filename is config-sharding-sompo-dev____smp_gr_customer_dev.yaml
each config can work well separately.
after setting  sql.show: true  in server.yaml, the log is as follows：

[INFO ] 16:56:12.895 [nioEventLoopGroup-2-1] i.n.handler.logging.LoggingHandler - [id: 0xf0b3b2e9, L:/0:0:0:0:0:0:0:0:3308] READ: [id: 0x2e621ddf, L:/127.0.0.1:3308 - R:/127.0.0.1:53944]
[INFO ] 16:56:12.897 [nioEventLoopGroup-2-1] i.n.handler.logging.LoggingHandler - [id: 0xf0b3b2e9, L:/0:0:0:0:0:0:0:0:3308] READ COMPLETE

I find that the error message of the client query is not display on the server. Why is this design?

I really can't reproduce it. Can you debug it with break point?

Here are some progress:
If I use CLI to connect Sharding-Proxy, will success in any condition.
But when I use HikariCP to connect Sharding-Proxy, exception fired no matter how many sharding congfigs there are.

Can you add my QQ ? My QQ : 491799734

I don't login QQ for months, my wechat is 15810310987. I can pull you to wechat group.

Resolved via https://github.com/apache/incubator-shardingsphere/issues/1830.

repo: apache/incubator-shardingsphere

license_source: bigquery


text:
How to config JNDI in datasource

No sample,below is error:
spring.shardingsphere.datasource.ds0.type=JNDI
spring.shardingsphere.datasource.ds0.jndiname=java:comp/env/jdbc/mymaster
What is correct?

type mean is your data source pool class name, JNDI is not  data source pool class name obviously.
You can use java API to generate datasource from JNDI and pass them to sharding rule config.

The above code is just a guess.
Hard code is not simple，we must bind datasource and all rules， even a bit cumbersome...
Additionally, Spring boot 2.0 supports JNDI now. So I hope Shardingsphere could support it.

authors: childewuque terrymanu


text:
Add caching at df level

Make sure these boxes are checked before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if any
[x] I have reproduced the issue with at least the latest released version of superset
[x] I have checked the issue tracker for the same issue and I haven't found one similar

Superset version
Master version
Expected results
The caching feature of superset currently is only used in in get_payload which is a function that is called to retrieve the HTTP response sent to the client when they are loading a vizualisation.
get_payload makes a get_df function call and passes the retrieved df to a get_data function which generates the data that is saved in the payload. This data is then cached using whichever caching system is used.
When you are downloading a csv file for a slice from a dashboard, the get_csv function is called which calls get_df, an intermediate function within get_payload.
If we were to cache at a dataframe level, instead of payload level, we can provide caching to more features such as downloading csv files.
I'm raising this issue as it is a common use case at our company. We often have multiple team members downloading the same slice from a shared dashboard, and all of them have to go through the full processing time, which could be improved.
Actual results
To implement caching at a data frame level so that the benefits of caching can be passed down to more features such as downloading slice csv files from dashboards. Any feedback or suggestions?
Steps to reproduce

I just tested the code changes to use cache for csv download, fairly simple and contained to get_csv. I'll look to contribute this.

I submitted #6202 to fix this.

repo: apache/incubator-superset


text:
SQLlab - Visualize the query results

Make sure these boxes are checked before submitting your issue - thank you!

[ ] I have checked the superset logs for python stacktraces and included it here as text if any
[ ] I have reproduced the issue with at least the latest released version of superset
[ ] I have checked the issue tracker for the same issue and I haven't found one similar

Superset version
0.24.0
Expected results
Able to see the report
Actual results
Incorrect report
Steps to reproduce
Execute query in SQL lab and Click on Visualize.
SELECT year AS year,
count() AS count
FROM
(SELECT date_part('year',created_date) as year,
count() as total_apps
from application
GROUP by 1
order by 1) AS expr_qry
GROUP BY year
ORDER BY count DESC
LIMIT 10000;
How do I get rid of outer grouping?  Otherwise, how do I visualize the query results?

I appreciate any help or pointer here.

For now the easiest path for your specific chart is to simply add application as a table and not go through SQL Lab. The SQL Lab "Visualize" flow allows using your query as a logical view to slice/dice on top of it.
In the near future we'll be working on charting/visualizations inside SQL Lab that do not require going to the "explore view".
We're well aware that the "Visualize" flow is confusing and far from what it needs to be.

@mistercrunch: Thanks for the information. Yes, I totally agreed, SQL Lab "Visualize" is very confusing flow. Since, I was able to create a dashboard through add table interface and thought it should be similar in SQL Lab interface. Only other option is to save results in a table and visualize it.
Is there a workaround via config or db to disable "Visualize" in SQL Lab?

@mistercrunch Is there any difference between about sql query process between sql lab and exploring chart, I find the time consumings of same sql statment in two places  are totally different, and some sql statment in sql lab is time-out.

Does anyone know where to find a simple tutorial on how to create a chart from a query using multiple tables with inner join?

authors: YHGui agabardo mistercrunch mmuru


text:
[TUBEMQ-468] add new feature - add one or more brokers in one cluster - online\offline\reload

[TUBEMQ-468] add new feature - add one or more brokers in one cluster - online\offline\reload

LGTM, pending on tests results

Codecov Report

Merging #367 (8b9f4d1) into TUBEMQ-421 (37739df) will decrease coverage by 0.40%.
The diff coverage is n/a.

@@               Coverage Diff               @@
##             TUBEMQ-421    #367      +/-   ##
===============================================
- Coverage          8.02%   7.62%   -0.41%
- Complexity          466     505      +39
===============================================
  Files               249     262      +13
  Lines             27115   29946    +2831
  Branches           4440    5521    +1081
===============================================
+ Hits               2176    2282     +106
- Misses            24468   27137    +2669
- Partials            471     527      +56

Impacted Files
Coverage Δ
Complexity Δ

...ava/org/apache/flink/connectors/tubemq/Tubemq.java
86.79% <0.00%> (-2.87%)
15.00% <0.00%> (+7.00%)
:arrow_down:

...rg/apache/tubemq/client/config/ConsumerConfig.java
20.00% <0.00%> (-2.86%)
6.00% <0.00%> (+2.00%)
:arrow_down:

...emq/client/consumer/SimplePullMessageConsumer.java
2.40% <0.00%> (-0.58%)
1.00% <0.00%> (ø%)

.../apache/tubemq/client/config/TubeClientConfig.java
12.63% <0.00%> (-0.39%)
3.00% <0.00%> (ø%)

...aster/nodemanage/nodebroker/BrokerConfManager.java
0.06% <0.00%> (-0.03%)
1.00% <0.00%> (ø%)

.../java/org/apache/tubemq/server/master/TMaster.java
0.00% <0.00%> (ø)
0.00% <0.00%> (ø%)

...a/org/apache/tubemq/corebase/utils/MixedUtils.java
0.00% <0.00%> (ø)
0.00% <0.00%> (ø%)

.../org/apache/tubemq/corebase/TErrCodeConstants.java
0.00% <0.00%> (ø)
0.00% <0.00%> (ø%)

...e/tubemq/server/broker/web/BrokerAdminServlet.java
0.00% <0.00%> (ø)
0.00% <0.00%> (ø%)

...tubemq/server/master/web/action/screen/Master.java
0.00% <0.00%> (ø)
0.00% <0.00%> (ø%)

... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 37739df...8b9f4d1. Read the comment docs.

authors: EMsnap codecov-io yuanboliu

repo: apache/incubator-tubemq


text:
[MINOR] Simplify creating buffer logic

What changes were proposed in this pull request?
[MINOR] Simplify creating buffer logic

Why are the changes needed?
Minor improvement to avoid duplicate code.

Does this PR introduce any user-facing change?
No

How was this patch tested?
No need.

Codecov Report

Merging #227 (5f0287a) into master (a775ea4) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master     #227      +/-   ##
============================================
- Coverage     59.13%   59.11%   -0.03%
+ Complexity     1327     1325       -2
============================================
  Files           160      160
  Lines          8727     8725       -2
  Branches        817      816       -1
============================================
- Hits           5161     5158       -3
- Misses         3301     3302       +1
  Partials        265      265

Impacted Files
Coverage Δ

.../org/apache/spark/shuffle/writer/WriterBuffer.java
93.18% <100.00%> (-0.30%)
:arrow_down:

...storage/handler/impl/DataSkippableReadHandler.java
81.25% <0.00%> (-3.13%)
:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter zuston

repo: apache/incubator-uniffle


text:
Remove incubating from pom files

What is this PR for?
Remove incubating term from pom files
Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

LGTM 👍

LGTM

LGTM

LGTM

Just one curious question.
When is the incubator- been removed from the github mirror ? So we can have github.com/apache/zeppelin ?

LGTM

LGTM

Merging this if no more discussion.

authors: AhyoungRyu astroshim cloverhearts felixcheung iemejia jongyoul minahlee prabhjyotsingh

repo: apache/incubator-zeppelin


text:
[HOTFIX] ZEPPELIN-931: fix interpreter listing bug

What is this PR for?
Currently available interpreter list is not shown in Creating New Interpreter section. It seems this bug was generated after #835 was merged. So I temporally deactivated 3 SerializedName code lines.
What type of PR is it?
Bug Fix
Todos

[x] - Fix interpreter listing bug when creating new interpreter

What is the Jira issue?
ZEPPELIN-931
How should this be tested?

Build latest master branch and browse Zeppelin home
Create new interpreter -> You can not see the available interpreter list in this step like below attached screenshot
Apply this patch
Build again and browse  -> You can see the available interpreter list as normal

Screenshots (if appropriate)

Before

After

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

@jongyoul I saw you are working on applying new mechanism to all of the existing interpreters in ZEPPELIN-804 as subtasks. I think this bug was caused since interpreterGroup , interpreterName and interpreterClassName  that I deactivated was not matched with group in here. So if you have a better way to solve this bug, just let me know :)

@AhyoungRyu I've tested and reproduced what you mentioned. Your change looks simple but it's not perfect because you cannot see spark interpreter in your interpreter list because interpreter-setting.json read interpreterGroup, interpreterName and interpreterClassName while initialisation. I'll submit a PR into your branch. Thanks for reporting this.

I don't know whether front-end uses RegisteredInterprerter. We need to find a way to indicate or mark which class is used in front and backend.

@jongyoul I see. That's why I can't see Spark interpreter in the list. Thanks ! Okay i'll check ASAP.

@AhyoungRyu Could you please add "[HOTFIX]" in a title?

@jongyoul Yeah I just added :)

@jongyoul I applied your patch and it works :)

@AhyoungRyu I've missed something. I submitted a PR, again.

@jongyoul That was also applied.

@AhyoungRyu Thanks.

@jongyoul Let's wait the travis then :)

@Leemoonsoo CI has been passed. Could you check this one ? :)

LGTM. I'll merge it asap.

@jongyoul Yeah sounds good.

authors: AhyoungRyu jongyoul


text:
Using openapi to generate rest framework (go and java) codes

Description
OpenAPI is a uniform DSL for describing the restful interface.
And, when developing a restful framework, we have to write many trivial codes (that will be also written in almost all restful services). So, generating such of codes is a good solution as we just need to develop the real logic and do not need to care to maintain the generated codes.
This PR defines several restful APIs for serving Grafana and Prometheus, and uses openapi-generator to generate the source codes of the restful framework in Go and Java. (The generator can also generate client-sdk, but I have not time to try it and we have no requirements).
So, just run mvn generate-sources in openapi' module, and then develop your logic in  openapi/java-rest/src/main/java/org/apache/iotdb/openapi/gen/handler/impl/V1ApiServiceImpl.java (for java) andopenapi/go-rest/src/iotdbrestimpl/api_service_impl.go`(for Go).
The only 3 things should to know are,

current openapi-generator has some bug for go-server codes, see: https://github.com/OpenAPITools/openapi-generator/issues/8448 and I have write how to fix it manually in go-rest/README.md

current openapi-generator can not generate java codes using native Netty. But it seems developing such a generator is not easy for openapi-generator.

I copy some generated codes to src folder.

The author of PR https://github.com/apache/iotdb/pull/2400 can consider to using this generated Go-server codes. (or, do some experiments to check whether the generated codes have an acceptable performance).

Does it support REST API for https?

has been merged into https://github.com/apache/iotdb/pull/3299

repo: apache/iotdb


text:
MINOR: Fix logging in ClusterControlManager

Out of order put/get is affecting the logging in ClusterControlManager.

If all quorum voters wrote a RegisterVoterRecord or similar to the metadata quorum with the node.id and  process.roles of that node, then controllers would be able to triage invalid clusters where brokers register with the same node.id as a controller.

@jsancio are you going to merge this?

@jsancio are you going to merge this?

Yes. I can merge it. @dielhennr can you resolve the conflicts?

This was already fixed in trunk. Closing the PR.

authors: dielhennr ijuma jsancio

repo: apache/kafka


text:
KAFKA-14108: Ensure both JUnit 4 and JUnit 5 tests run

This pull request addresses the problem reported in https://github.com/apache/kafka/pull/12285 and tracked in https://issues.apache.org/jira/browse/KAFKA-14108

@clolov

don't we need to remove "streams" from

  def shouldUseJUnit5 = !(["runtime", "streams"].contains(it.project.name))

as well?

please add the test report for "before" the change and for "after" the change. This would help us validate that number of tests run "after" this change is greater than or equal to "before" the change.

Thanks for the PR, @clolov !
The builds have errors:
* What went wrong:
[2022-07-26T12:41:05.041Z] A problem occurred evaluating root project 'Kafka_kafka-pr_PR-12441'.
[2022-07-26T12:41:05.041Z] > Cannot convert a null value to an object of type Dependency.
[2022-07-26T12:41:05.041Z]   The following types/formats are supported:
[2022-07-26T12:41:05.041Z]     - Instances of Dependency.
[2022-07-26T12:41:05.041Z]     - String or CharSequence values, for example 'org.gradle:gradle-core:1.0'.
[2022-07-26T12:41:05.041Z]     - Maps, for example [group: 'org.gradle', name: 'gradle-core', version: '1.0'].
[2022-07-26T12:41:05.041Z]     - FileCollections, for example files('some.jar', 'someOther.jar').
[2022-07-26T12:41:05.041Z]     - Projects, for example project(':some:project:path').
[2022-07-26T12:41:05.041Z]     - ClassPathNotation, for example gradleApi().

@clolov I also found a mistake in #12285:
In KTableSourceTopicRestartIntegrationTest line 103 should be
sourceTopic = SOURCE_TOPIC + "-" + testInfo.getTestMethod().map(Method::getName).orElse("");`

instead of
sourceTopic = SOURCE_TOPIC + "-" + testInfo.getTestMethod().map(Method::getName);

otherwise the test does not run wince the topic name contains illegal characters.

Thanks for the comments @cadonna @ijuma. Christo is on vacation right now (we are team mates) but I will take up this PR and file a revision addressing your comments tomorrow.

@divijvaidya That would be great, because I start feeling uncomfortable having some integration tests not running on the builds.
Please also make sure to compare test reports before and after the change as you proposed.
Thanks a lot for taking over this work for Christo when he is on vacation!

@divijvaidya Regarding your question

don't we need to remove "streams" from def shouldUseJUnit5 = !(["runtime", "streams"].contains(it.project.name)) as well?

Yes, we need.

Changes

Replace with @Category({IntegrationTest.class}) with @Tag("integration")
Unblock streams project from using JUnitPlatform.
Use junit-vintage and junit-jupiter engine for executing tests of streams project. Tests formatted as Junit4 will automatically use junit-vintage and tests formatted as JUnit5 will automatically use junit-jupiter.
Minor change required in KTableSourceTopicRestartIntegrationTest.java after using junit-jupiter

Results
Result of executing ./gradlew :streams:test before & after this change.
Before

After

Note that after this change we are running more number of tests. This is because tests which were migrated to JUnit 5 were not being executed earlier.
Comments
Addressed concerns from @ijuma:

regarding not affecting other projects with this change (note that junit-vintage is being used for streams only)
regarding removal of addition of unnecessary dependencies (removed the unnecessary dependency)

Addressed concerns from @cadonna

regarding adding junitJupiterParams dependency only when needed (removed the dependency)

Next steps (separate PRs, in order)

Migrate tests using PowerMock to use Mockito
Migrate all tests to Junit5
Remove the Junit4 dependency completely from build.gradle

@divijvaidya Thank you for the updates!
I am afraid you did a mistake when generating the test reports to test the changes. You changed task integrationTest and task unitTest but you generated the reports with streams:test. In task test no filtering is done on the tags integration and org.apache.kafka.test.IntegrationTest. Also the builds use integrationTest and unitTest to run the tests and not test. You should generate the reports with streams:integrationTest and streams:unitTest.
In my tests yesterday, I experienced that you cannot use @Tags("integration") (and filter on it) on integration tests that are written JUnit 4, but you need to leave @Category({IntegrationTest.class}) in the tests and use includeTags "org.apache.kafka.test.IntegrationTest" in the build file. For integration tests written in JUnit 5, you need to use @Tags("integration") in the tests and includeTags "integration" in the build file. See my comment https://github.com/apache/kafka/pull/12441#discussion_r932429697.

Updated the code. @cadonna should be ready for your review ones the test run is complete.
Result for ./gradlew :streams:integrationTest
Before

After

Result for ./gradlew :streams:unitTest
Before

After

I verified that all Streams' tests are run in the builds and I also verified that the tests of other modules are run in the builds.

Build failures are unrelated.

authors: cadonna clolov divijvaidya


text:
KAFKA-17347: Add missing client-metrics option to kafka-configs.sh

When KIP-714 was developed, the entity type of client-metrics was added to the kafka-configs.sh tool. The idea was to have two forms of specifying the name and type of a client metrics config resource, either --entity-type client-metrics --entity-name NAME or --client-metrics NAME. This style of alias is used for all of the entity types. Unfortunately, the --client-metrics form was not implemented. This PR corrects that and adds more tests.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Only 3 unit test failures, unrelated to this PR.

@DL1231 I wonder if you'd like to review this PR based on your similar work in kafka-configs.sh.

Thanks for the PR @AndrewJSchofield. Though it's good to have this functionality but I don't find --client-metrics NAME this way of usage in the KIP-714. And given we already have kafka-client-metircs.sh utility then do we need this change?

The kafka-configs.sh tool has aliases for all of the resource types with the exception of client metrics. So, this change is just filling in that gap and removing the inconsistency.

@chia7712 I wonder if you would be prepared to take a look at this small PR.

@AndrewJSchofield thanks for this patch
Should we add client-metrics to
https://github.com/apache/kafka/blob/0af75c0e41184e362efca25317b5e8dd89507cd2/core/src/main/scala/kafka/admin/ConfigCommand.scala#L704
?

Not strictly necessary because the value of ConfigType.CLIENT_METRICS is already "client-metrics".


text:
KAFKA-17821: the set of configs displayed by logAll could be invalid

Jira: https://issues.apache.org/jira/browse/KAFKA-17821
User can config different protocol in consumer config. We will print all config in log, but some configs has the default value and is unsupported for each protocol, thus we print these config will misdirect user.
We should improve this, and won't show the unsupported configs in log.
test in my local, if use CLASSIC protocol won't show group.remote.assignor
[2024-11-30 19:25:10,460] INFO ConsumerConfig values:
	metric.reporters = [org.apache.kafka.common.metrics.JmxReporter]
	sasl.oauthbearer.token.endpoint.url = null
	sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
	retry.backoff.max.ms = 1000
	reconnect.backoff.max.ms = 1000
	partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor, class org.apache.kafka.clients.consumer.CooperativeStickyAssignor]
	ssl.engine.factory.class = null
	sasl.oauthbearer.expected.audience = null
	ssl.keystore.type = JKS
	enable.auto.commit = false
	sasl.oauthbearer.header.urlencode = false
	interceptor.classes = []
	exclude.internal.topics = true
	ssl.truststore.password = null
	default.api.timeout.ms = 60000
	ssl.endpoint.identification.algorithm = https
	max.poll.records = 500
	check.crcs = true
	sasl.login.refresh.buffer.seconds = 300
	receive.buffer.bytes = 65536
	ssl.truststore.type = JKS
	sasl.oauthbearer.clock.skew.seconds = 30
	client.dns.lookup = use_all_dns_ips
	fetch.min.bytes = 1
	send.buffer.bytes = 131072
	sasl.oauthbearer.jwks.endpoint.url = null
	value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
	enable.metrics.push = true
	sasl.login.retry.backoff.ms = 100
	metadata.recovery.rebootstrap.trigger.ms = 300000
	ssl.secure.random.implementation = null
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	ssl.trustmanager.algorithm = PKIX
	sasl.jaas.config = null
	sasl.kerberos.min.time.before.relogin = 60000
	connections.max.idle.ms = 540000
	session.timeout.ms = 45000
	internal.leave.group.on.close = true
	ssl.keystore.certificate.chain = null
	socket.connection.setup.timeout.ms = 10000
	ssl.provider = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
	ssl.cipher.suites = null
	security.protocol = PLAINTEXT
	allow.auto.create.topics = true
	ssl.keymanager.algorithm = SunX509
	sasl.login.callback.handler.class = null
	auto.offset.reset = latest
	metadata.max.age.ms = 300000
	reconnect.backoff.ms = 50
	sasl.kerberos.ticket.renew.window.factor = 0.8
	max.partition.fetch.bytes = 1048576
	bootstrap.servers = []
	metrics.recording.level = INFO
	ssl.truststore.certificates = null
	security.providers = null
	sasl.mechanism = GSSAPI
	client.id = consumer-null-1
	request.timeout.ms = 30000
	sasl.login.retry.backoff.max.ms = 10000
	heartbeat.interval.ms = 3000
	auto.commit.interval.ms = 5000
	sasl.login.class = null
	ssl.truststore.location = null
	ssl.keystore.password = null
	fetch.max.bytes = 52428800
	max.poll.interval.ms = 300000
	group.protocol = classic
	sasl.login.connect.timeout.ms = null
	socket.connection.setup.timeout.max.ms = 30000
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.min.period.seconds = 60
	sasl.oauthbearer.scope.claim.name = scope
	group.id = null
	sasl.oauthbearer.expected.issuer = null
	sasl.login.read.timeout.ms = null
	retry.backoff.ms = 100
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	internal.throw.on.fetch.stable.offset.unsupported = false
	metadata.recovery.strategy = rebootstrap
	ssl.key.password = null
	fetch.max.wait.ms = 500
	ssl.keystore.key = null
	sasl.client.callback.handler.class = null
	metrics.num.samples = 2
	key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
	ssl.protocol = TLSv1.3
	group.instance.id = null
	client.rack =
	ssl.keystore.location = null
	sasl.oauthbearer.sub.claim.name = sub
	sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
	metrics.sample.window.ms = 30000
	isolation.level = read_uncommitted
	sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
	sasl.login.refresh.window.jitter = 0.05
 (org.apache.kafka.common.config.AbstractConfig:380)

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Thanks for @kirktrue review, addressed all comments.


text:
MINOR: Fix potential bug in LogConfig.getConfigValue and improve test coverage

LogConfig.getConfigValue would throw a NoSuchElementException if any log
config was defined without a server default mapping.
Added a unit test for getConfigValue and a sanity test for
toHtml/toRst/toEnrichedRst, which were previously not exercised during
the test suite.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

retest this please

One of the builds succeeded and the two others failed due to unrelated flaky tests. Started another build just in case.

Thanks @ijuma. LGTM

Two builds succeeded (although one reported a failure, it actually passed) and one failed with known flake (kafka.server.DescribeLogDirsRequestTest.testDescribeLogDirsRequest). Merging to master.

authors: ijuma jsancio


text:
HOTFIX: don't close or wipe out someone else's state

When it comes to actually closing a task we now treat all states exactly the same, and call StateManagerUtil#closeStateManager regardless of whether it's in CREATED or RESTORING or RUNNING
Unfortunately StateManagerUtil doesn't actually check to make sure that we actually own the lock for this task's state. During a dirty close with eos enabled, we wipe the state -- but in some cases, this means deleting the state out from under another StreamThread who is still in the process of revoking this task

Retest this please.

in StateManagerUtil#registerStateStores it is very possible that the lock(id) function would return false, which would be interpreted as fatal error

@guozhangwang Can you clarify? If the task fails to lock the directory in registerStateStores it throws a LockException, which is ultimately caught and handled by the TaskManager in tryToCompleteRestoration. It'll just stay in CREATED and try again on the next loop right?

test this please

test this please

test this please

test this please

test this

test this

test this!

test this

test this

@guozhangwang Can you clarify? If the task fails to lock the directory in registerStateStores it throws a LockException, which is ultimately caught and handled by the TaskManager in tryToCompleteRestoration. It'll just stay in CREATED and try again on the next loop right?

Ah yes, that's right.

Failures unrelated:
ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
SuppressionDurabilityIntegrationTest.shouldRecoverBufferAfterShutdown[exactly_once]

test this please

test this please

test this please

authors: ableegoldman guozhangwang mjsax


text:
KAFKA-9313: Set use_all_dns_ips as the new default for client.dns.lookup (KIP-602)

This applies to the producer, consumer, admin client, connect worker
and inter broker communication.
ClientDnsLookup.DEFAULT has been deprecated and a warning
will be logged if it's explicitly set in a client config.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

We should also add a note to upgrade.html.

Can you please rebase against master.

@ijuma Merged against trunk, fixed conflict in ConsumerConfig, and updated upgrade.html.

@ijuma LGTM. Much clearer. Thanks.

retest this please

Thanks for the contribution! Merged to trunk and 2.6 branches.

authors: badaiaqrandista ijuma


text:
KYLIN-4684 Fix web ui bug when create streamig table

Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
Types of changes
What types of changes does your code introduce to Kylin?
Put an x in the boxes that apply

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices apply)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have create an issue on Kylin's jira, and have described the bug/feature there in detail
[x] Commit messages in my PR start with the related jira ID, like "KYLIN-0000 Make Kylin project open-source"
[ ] Compiling and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] If this change need a document change, I will prepare another pr against the document branch
[ ] Any dependent changes have been merged

Further comments
If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc...

Pull Request Test Coverage Report for Build 6224

0 of 0   changed or added relevant lines in 0 files are covered.
6 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.002%) to 28.029%

Files with Coverage Reduction
New Missed Lines
%

core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java
2
80.23%

core-metrics/src/main/java/org/apache/kylin/metrics/lib/impl/MetricsSystem.java
4
68.33%

Totals

Change from base Build 6212:
-0.002%

Covered Lines:
26226

Relevant Lines:
93568

💛  - Coveralls

@zhangayqian  Do you have do a test/verify for this patch ?

Test
Reproduce in kylin3.1.0

Before this issue be fixed, set TS Partten to S when load streamingV2 table and save this table

Then check table schema of this table, found that the TS Partten is saved as MS actually.

Test in Fixed version

Set TS Partten to S when load streamingV2 table and save this table

Then check table schema of this table,  the TS Partten is saved as S.

Test Result

Issue be fixed by this PR.

Thank you @zhangayqian .

authors: coveralls hit-lacus zhangayqian

repo: apache/kylin


text:
Fix AppState when Engine connection is terminated

:mag: Description
Issue References 🔗

This issue was noticed a few times when the batch state was set to ERROR, but the appState kept the non-terminal state forever (e.g. RUNNING), even if the application was finished (in this case Yarn Application).
{
"id": "********",
"user": "****",
"batchType": "SPARK",
"name": "*********",
"appStartTime": 0,
"appId": "********",
"appUrl": "********",
"appState": "RUNNING",
"appDiagnostic": "",
"kyuubiInstance": "*********",
"state": "ERROR",
"createTime": 1725343207318,
"endTime": 1725343300986,
"batchInfo": {}
}

It seems that this happens when there is some intermittent failure during the monitoring step and the batch ends with ERROR, leaving the application metadata without an update. This can lead to some misinterpretation that the application is still running. We need to set this to UNKNOWN state to avoid errors.
Describe Your Solution 🔧
This is a simple fix that only checks if the batch state is ERROR and the appState is not in a terminal state and changes the appState to UNKNOWN, in these cases (during the batch metadata update).
Types of changes :bookmark:

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪
Behavior Without This Pull Request :coffin:
If there is some error between the Kyuubi and the Application request (e.g. YARN client), the batch is finished with ERROR state and the application keeps the last know state (e.g. RUNNING).
Behavior With This Pull Request :tada:
If there is some error between the Kyuubi and the Application request (e.g. YARN client), the batch is finished with ERROR state and the application has a non-terminal state, it is forced to UNKNOWN state.
Related Unit Tests
I've tried to implement a unit test to replicate this behavior but I didn't make it. We need to force an exception in the Engine Request (e.g. YarnClient.getApplication) but we need to wait for the application to be in the RUNNING state before raising this exception, or maybe block the connection between kyuubi and the engine.

Checklist 📝

[ ] This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

@joaopamaral please fix the code style. (simply run dev/reformat if you are using Linux or macOS)

thanks, merging to master

reopen to retest

Codecov Report
Attention: Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (2d64255) to head (da8c356).
Report is 57 commits behind head on master.

Files with missing lines
Patch %
Lines

...g/apache/kyuubi/operation/BatchJobSubmission.scala
0.00%
4 Missing :warning:

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #6722    +/-   ##
=======================================
  Coverage    0.00%   0.00%
=======================================
  Files         684     687     +3
  Lines       42282   42445   +163
  Branches     5767    5793    +26
=======================================
- Misses      42282   42445   +163

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests
JS Bundle Analysis - Avoid shipping oversized bundles

thanks, merged to master/1.10.1/1.9.3

authors: codecov-commenter joaopamaral pan3793 turboFei

repo: apache/kyuubi


text:
[INFRA] Update archive mailing list address

Why are the changes needed?

Replace the obsolete address with a new one.
How was this patch tested?

Review
Was this patch authored or co-authored using generative AI tooling?

No

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 0.00%. Comparing base (eb1b599) to head (858e6cc).

Additional details and impacted files
@@          Coverage Diff           @@
##           master   #6838   +/-   ##
======================================
  Coverage    0.00%   0.00%
======================================
  Files         687     687
  Lines       42441   42441
  Branches     5793    5793
======================================
  Misses      42441   42441

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Thanks, merged to all release branches, namely master/1.10/1.9


text:
Port Packed Producing Python Programs, #1008

[X] You've read the Contributor Guide and Code of Conduct.
[X] You've included unit or integration tests for your change, where applicable.
[X] You've included inline docs for your change, where applicable.
[X] There's an open issue for the PR that you are making. If you'd like to propose a change, please open an issue to discuss the change or find an existing issue.

Ports the Python scripts for generating the various Lucene.Net.Util.Packed files.
Fixes #1008
Description
This ports the four Python scripts in the Packed package/namespace that generate the various files there, and updates them to produce C# code instead, as well as updates them to be compatible with Python 3 (as Python 2, which was used at the time, is now very out of support). This results in some minor syntax differences in the generated code. The README.md file in this folder explains how to run the scripts (which is simply to run them with Python 3, no virtual environment or dependencies needed).
Having these scripts ported will help us ensure that the code is being generated the same way as in Java, as well as to assist any necessary future changes to how the files are generated. It also just helps complete our 1:1 port of the Lucene code, be it Java or Python.
This also corrects an out-of-date type name in the package.md documentation for this namespace.

I'm renaming the GitHub workflow file. I think we should keep the current names because it more closely matches the Java file names. This can always be revisited later in a separate PR if desired.

repo: apache/lucenenet


text:
[EXPERIMENT] Drop DELETE_ON_CLOSE flag

This removes concurrency on delete (that seems to affect Windows users) on the other hand will leave lock structure present after Maven done.
Change for https://github.com/apache/maven-mvnd/issues/728

Is there a possibility to clean it up during daemon idle time?

This experiment "works" at the cost of leaving $M2_REPO/.locks/ full of 0 byte lock files.

Superseded by https://github.com/apache/maven-resolver/pull/216

authors: cstamas fsmherrm

repo: apache/maven-resolver


text:
controller: use OS specific PRNG for RIOT builds

Currently the controller is using jrand48() to generate pseudo
random numbers. For RIOT builds this commit switches to the RIOT
build-in PRNG API, as this has two benefits: i) it saves 500bytes
of flash and ii) jrand48() internally uses malloc, which has been
source to runtime failures in the past.

All green -> go

repo: apache/mynewt-nimble


text:
MINIFI-124 The active_pid() function kill -s 0  returns 0 on success (PID i…

MINIFI-124.  In the active_pid() function kill -s 0  returns 0 on success (the PID is currently running), yet the if statement in the stop case statement is checking for 0 as a failure condition.

reviewing

Looks good here and thanks for catching!  Will merge.

authors: apiri codingogre

repo: apache/nifi-minifi-cpp


text:
Add table attributes to flow files

https://issues.apache.org/jira/browse/NIFI-3525
Adding table name to flowfile attributes

@ambud - Welcome to NiFi!  Thank you for contributing this, I will review.  In the future, please keep the PR boilerplate comment and fill out the checklist.  It has some helpful tips on the PR  and commit naming conventions and info for reviewers.

I also recommend running a full build with mvn clean install -Pcontrib-check.  It looks like several of the QueryDatabaseTableTest unit tests are failing:

Tests run: 11, Failures: 8, Errors: 0, Skipped: 0, Time elapsed: 1.006 sec <<< FAILURE! - in org.apache.nifi.processors.standard.QueryDatabaseTableTest
testWithRuntimeException(org.apache.nifi.processors.standard.QueryDatabaseTableTest)  Time elapsed: 0.388 sec  <<< FAILURE!
java.lang.AssertionError: org.apache.nifi.processor.exception.FlowFileHandlingException: Cannot commit session because the following FlowFiles have not been removed or transferred: [0]

My quick take is that the happy-path works fine with your code, but paths that do not result in a flowfile output are failing because the session is not cleaned up.  How about moving the call to session.putAttribute down to line ~300 by where RESULT_ROW_COUNT is added?  A viable output flowfile is being formatted here, and being in the same neighborhood with other attribute assignments might be easier to understand.  What do you think?

@jvwing thanks for the input, changed and fixed. The issue was i didn't reassign the flow file variable fromt the put attribute statement.

Thanks for the update @ambud, the tests are passing.  I have two other comments:
1.) Similar attributes, like RESULT_ROW_COUNT are defined as static final member variables, so they can be referenced consistently (mostly in tests).  I recommend we do the same for table.name.
2.) I looked for an established attribute naming convention that would apply to table.name.

The ListDatabaseTables processor uses db.table.name, but it has a consistent db.* pattern for all of its attributes that QueryDatabaseTable does not.
QueryDatabaseTable has querydbtable.row.count, which might suggest a querydbtable.* pattern like querydbtable.table.name, except that there is only one such attribute (the fragment attributes are idiomatic and might not apply, maxvalue.* is the only other existing attribute written).

Having done this research, I'm now more confused than when I started :), so I don't have the concrete recommendation I was hoping to arrive at.  Have you looked into this, and how did you arrive at table.name?

Agreed, I overlooked it and will add that

Ideally I would just do tablename (one word) without delimiters for simplicity of downstream processing. Let me know if there's another preferred convention here.

No, there doesn't seem to be a convention to guide us.  If you prefer tablename, I'm OK with it.

@jvwing changed the name, squashed commits.

Thanks, @ambud, we're getting closer.  There are a few items in the latest commit:

QueryDatabaseTable has class members TABLE_NAME for the input property descriptor, and TABLENAME for the output attribute name.  How about renaming TABLENAME to RESULT_TABLENAME to clarify?
The unit test testAddedRows is failing because it is still using "table.name" as a literal string.
There is a checkstyle error for a tab in QueryDatabaseTable.java:106.

You can see the test and checkstyle errors when you run mvn clean install -Pcontrib-check for the NiFi project.

Sorry about that, fixed.

Thanks again for contributing this, @ambud!

Thanks for your help in fixing the unit tests @jvwing

authors: ambud jvwing

repo: apache/nifi


text:
NIFI-5828: Documents behavior of ExecuteSQL attrs when Max Rows Per Flow File is enabled

For all changes:

[X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?
[X] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[X] Has your PR been rebased against the latest commit within the target branch (typically master)?
[X] Is your initial contribution a single, squashed commit?

For code changes:

[X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] ~Have you written or updated unit tests to verify your changes?~
[ ] ~If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?~
[ ] ~If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?~
[ ] ~If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?~
[ ] ~If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?~

For documentation related changes:

[X] Have you ensured that format looks appropriate for the output in which it is rendered?

LGTM +1, merging. Thanks @colindean !

@colindean mind closing this PR? Doesn't look like the comment in the commit caused it to auto-close (maybe the period after the PR number?) Please and thanks! Also I updated the Fix Version to be 1.9.0 as 1.8.0 was already released

authors: colindean ijokarumawak mattyb149


text:
NIFI-7890 - Added record support to ConsumeMQTT processor

This also adds the option to define a delimiter instead of record reader/writer so that multiple messages can be appended into a single flow file. This greatly increases the performances of the processor when the use case requires high throughput in terms of events per second.
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Enables X functionality; fixes bug NIFI-YYYY.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically main)?

[x] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[x] Have you verified that the full build is successful on JDK 8?
[x] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

Thanks for the improvement @pvillard31. I think this will give us MUCH better performance for any MQTT related use cases. @exceptionfactory had a few comments but otherwise I'm a +1.

Thanks for the improvement @pvillard31. I think this will give us MUCH better performance for any MQTT related use cases. @exceptionfactory had a few comments but otherwise I'm a +1.

Thanks for the review @exceptionfactory and @markap14 - I made some changes following your recommendations.

Thanks for the review @exceptionfactory and @markap14 - I made some changes following your recommendations.

@pvillard31 Thanks for the feedback and changes, looks good.

@pvillard31 Thanks for the feedback and changes, looks good.

Squashed and rebased against main to fix the conflicts caused by the 2 others PRs related to this processor being merged today.

Squashed and rebased against main to fix the conflicts caused by the 2 others PRs related to this processor being merged today.

Thanks for the review @turcsanyip - I think I addressed your comments, let me know if there is something else.

Thanks for the review @turcsanyip - I think I addressed your comments, let me know if there is something else.


text:
NIFI-10481 Add NiFi configuration to disable zk client ensembleTracker

Summary
NIFI-10481
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[x] Apache NiFi Jira issue created

Pull Request Tracking

[x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[x] Pull Request based on current revision of the main branch
[x] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[ ] Build completed using mvn clean install -P contrib-check

[ ] JDK 8
[ ] JDK 11
[ ] JDK 17

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

Will review

Thanks @nathluu, this looks good. I tested this out in a 3 node cluster with an external Zookeeper cluster and found things to be working normally. I was not specifically sure how to verify that the ensembleTracker is actually disabled. Is there something which indicates this is now disabled?

Thank @thenatog for helping review and test my PR. To verify the EnsembleTracker option, we need the zookeeper quorum deployed inside the K8S cluster same as with NiFi. Below are the steps for testing:
Setup:

Deploy ZK quorum in the AKS
Deploy NiFi to use K8S service FQDN of ZK as the connection string e.g. test-zookeeper.zk-ns.svc.cluster.local:2181
Case 1: EnsembleTracker enabled (default)
1/ Rollout restart all ZK pods (Not NiFi pods)
2/ When all ZK restarted and up again, access the NiFi canvas UI
3/ Verify that the UI will not be load
4/ Rollout restart all of the NiFi pods will resolve the issue
Case 2: EnsembleTracker disable (set nifi.zookeeper.client.ensembleTracker=false in nifi.properties file)
With a working NiFi (canvas is accessible)
1/ Rollout restart all ZK pods (Not NiFi pods)
2/ When all ZK restarted and up again, access the NiFi canvas UI
3/ Verify that the NiFi canvas can be access normally

authors: nathluu thenatog


text:
add tests for LIMIT/OFFSET in QueryRunnerTest [MINOR]

This just helps indicate/confirm that queries with LIMIT and OFFSET work properly in the multistage query engine.

Codecov Report

Merging #9496 (c2e3016) into master (3057712) will decrease coverage by 42.70%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master    #9496       +/-   ##
=============================================
- Coverage     68.66%   25.96%   -42.71%
+ Complexity     5136       44     -5092
=============================================
  Files          1916     1904       -12
  Lines        102069   101725      -344
  Branches      15481    15447       -34
=============================================
- Hits          70083    26409    -43674
- Misses        26980    72662    +45682
+ Partials       5006     2654     -2352

Flag
Coverage Δ

integration1
25.96% <ø> (-0.18%)
:arrow_down:

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...in/java/org/apache/pinot/spi/utils/BytesUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/spi/trace/BaseRecording.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/spi/trace/NoOpRecording.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/config/table/FSTType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/config/user/RoleType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/data/MetricFieldSpec.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/stream/StreamMessage.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/common/tier/TierFactory.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...a/org/apache/pinot/spi/config/table/TableType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../org/apache/pinot/spi/data/DimensionFieldSpec.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 1334 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: agavra codecov-commenter

repo: apache/pinot


text:
Upgrade Iceberg 1.7.0

Description
Please include a summary of the changes and the related issue. Please also include relevant motivation and context. List any dependencies that are required for this change.
Upgrade Apache Iceberg to 1.7.0
Additionally :

Removes PolarisS3FileIOFactory as it's not required anymore (if it's a public class, we might wanna see this deprecation)
Add Endpoint discovery support as newly added feature in 1.7
Fix Nested Namespace failure
TODO : check why this changed, working on tracking down the commit

Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Documentation update
[ X ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ X ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[ X] Existing Suite

Test Configuration:

Hardware:
Toolchain:
SDK:

Checklist:
Please delete options that are not relevant.

[ X ] I have performed a self-review of my code
[ X ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ X ] My changes generate no new warnings
[ ] If adding new functionality, I have discussed my implementation with the community using the linked GitHub issue

LGTM with a minor style issue:
> The following files had format violations:
      src/main/java/org/apache/polaris/service/catalog/BasePolarisCatalog.java
          @@ -47,7 +47,6 @@
           import·org.apache.iceberg.TableMetadata;
           import·org.apache.iceberg.TableMetadataParser;
           import·org.apache.iceberg.TableOperations;
          -import·org.apache.iceberg.aws.s3.S3FileIOProperties;
           import·org.apache.iceberg.catalog.Namespace;
           import·org.apache.iceberg.catalog.SupportsNamespaces;
           import·org.apache.iceberg.catalog.TableIdentifier;
  Run './gradlew :polaris-service:spotlessApply' to fix these violations.

checking test failure

on debugging it, looks like we need to handle this change cleanly [1], since polaris supports more endpoint than default impl,
let me fix this !
[1] https://github.com/apache/iceberg/commit/a2b8008da7bc26e03248a35eeee60d1cc7e8499d#diff-86450612dbe323d6d06cbc3846aa1913f042eaedadc0ca027c36bfbe08d3a46c

@flyrain @singhpk234 should we update iceberg 1.7.0 dependencies for "getting start" examples as well? Most of them are using 1.5.2 iceberg runtime. I can take this if not already done.

Note: Today I realized that there are two "special" changes in Iceberg/Java 1.7.0:

Object-storage layout - the generated paths have changed. This is relevant when IAM policies for S3 + GCS are generated. IIRC, IAM policies generated by Polaris are not yet adopted for object-storage layout (write.object-storage.enabled=true + data path). Before 1.7.0, the paths were generated like s3://bucket/<random-ish-base64>/..., since 1.7.0 it's s3://bucket/<random-ish-binary-part-1>/<random-ish-binary-part-2>/<random-ish-binary-part-3>/<random-ish-binary-part-4>/...)
Iceberg/Java 1.7.0 refuses to submit some requests like the multi-table-commit, if the service does not return ConfigResponse.endpoints() or that list is empty.

Object-storage layout - the generated paths have changed. This is relevant when IAM policies for S3 + GCS are generated. IIRC, IAM policies generated by Polaris are not yet adopted for object-storage layout (write.object-storage.enabled=true + data path). Before 1.7.0, the paths were generated like s3://bucket//..., since 1.7.0 it's s3://bucket/////...)

This is due to the new Object Storage v3 changes introduced in the 1.7 release. However, since Polaris doesn’t currently support the hash prefix before the table path, this update should not impact functionality as far as I understand. Let me know if I’m missing any nuances here!

Iceberg/Java 1.7.0 refuses to submit some requests like the multi-table-commit, if the service does not return ConfigResponse.endpoints() or that list is empty.

#383 will resolve this issue, so let's prioritize it. In the short term, it should be manageable since most REST clients haven’t yet adopted the latest 1.7 release.

Object-storage layout - the generated paths have changed. This is relevant when IAM policies for S3 + GCS are generated. IIRC, IAM policies generated by Polaris are not yet adopted for object-storage layout (write.object-storage.enabled=true + data path). Before 1.7.0, the paths were generated like s3://bucket//..., since 1.7.0 it's s3://bucket/////...)

This is due to the new Object Storage v3 changes introduced in the 1.7 release. However, since Polaris doesn’t currently support the hash prefix before the table path, this update should not impact functionality as far as I understand. Let me know if I’m missing any nuances here!

For Polaris it's the amount of combinations that need to be supported: no object-storage prefix, the old and the new one. For GCS it's easier, because Google accepts regular expressions - S3 however does not.

Iceberg/Java 1.7.0 refuses to submit some requests like the multi-table-commit, if the service does not return ConfigResponse.endpoints() or that list is empty.

#383 will resolve this issue, so let's prioritize it. In the short term, it should be manageable since most REST clients haven’t yet adopted the latest 1.7 release.

@flyrain @singhpk234 should we update iceberg 1.7.0 dependencies for "getting start" examples as well? Most of them are using 1.5.2 iceberg runtime. I can take this if not already done.

+1, @MonkeyCanCode! Thanks for picking it up as a follow-up.

@flyrain @singhpk234 should we update iceberg 1.7.0 dependencies for "getting start" examples as well? Most of them are using 1.5.2 iceberg runtime. I can take this if not already done.

+1, @MonkeyCanCode! Thanks for picking it up as a follow-up.

Anytime. Will PR later tonight then tag the PR here.

I will merge this by EOD if there is no objection.

Here is a PR for the 2 missing changes: https://github.com/apache/polaris/pull/447

Took this branch and ran with 1.7.1-RC1 of Iceberg. All tests passed

@singhpk234 1.7.1 is out now so we should be able to get this in

@singhpk234 1.7.1 is out now so we should be able to get this in

Will update my PR for the same.

@singhpk234 1.7.1 is out now so we should be able to get this in

Will update my PR for the same.

Here is the updated PR for 1.7.1 for the two files I changed earlier for 1.7.0: https://github.com/apache/polaris/pull/447/files#diff-a5126960d98d899570bfc55c394a55d80dcd83f6d53ddbf26954731137b9facb

authors: MonkeyCanCode RussellSpitzer flyrain singhpk234 snazy

repo: apache/polaris


text:
fix ttl expiry does not take effect

Fixes #12265
Motivation
fix ttl expiry does not take effect
Modifications
When ManagedLedgerImpl.getPositionAfterN

check whether the  currentLeder contains entry when count the entries of currentLeder
check the result and LAC first before return

Verifying this change
add test in ManagedLedgerTest.testGetPositionAfterN
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): ( no)
The public API: ( no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation
No-doc-need since this pull request is a inner bug fix.

@eolivelli PTAL

repo: apache/pulsar


text:
Added local filesystem backend for package manager

Motivation
Added new implementation for package manager that writes files into local filesystem.

@zymap Would you take a look at this PR?

authors: jiazhai merlimat


text:
[Authenticate] optimization getHostName correct when use broker ip in Kerberos authentication

Motivation
When Pulsar client use broker ip in Kerberos authentication, will result in an error，Expected to be correct broker hostname. Because Kerberos principal naming convention is servieceName/hostName@realms, so if client  use broker ip authent，Kerberos will not found the serverPrincipal.
Here PulsarSaslClient code:
String serverPrincipal = serverType.toLowerCase() + "/" + serverHostname;
Sasl.createSaslClient(mechs, clientPrincipalName, serviceName, serviceHostname, null,
    new ClientCallbackHandler());

produce example
sh pulsar-client --url pulsar://10.3.0.8:6650 produce
org.apache.pulsar.client.impl.auth.PulsarSaslClient - Using JAAS/SASL/GSSAPI auth to connect to server Principal broker/10.3.0.8
#KDC configure
Kerberos principal  broker/hostname@PULSAR.COM

Modifications
chang InetSocketAddresscreateUnresolved to
new InetSocketAddress(hostUri.getHost(), hostUri.getPort())
InetSocketAddress.createUnresolved  can not get correct hostname
/**
 *
 * Creates an unresolved socket address from a hostname and a port number.
 *

 * No attempt will be made to resolve the hostname into an InetAddress.
 * The address will be flagged as unresolved.
 *

 * A valid port value is between 0 and 65535.
 * A port number of {@code zero} will let the system pick up an
 * ephemeral port in a {@code bind} operation.
 *
 * @param   host    the Host name
 * @param   port    The port number
 * @throws IllegalArgumentException if the port parameter is outside
 *                  the range of valid port values, or if the hostname
 *                  parameter is {@code null}.
 * @see     #isUnresolved()
 * @return  an {@code InetSocketAddress} representing the unresolved
 *          socket address
 * @since 1.5
 */
public static InetSocketAddress createUnresolved(String host, int port)

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no )
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: ( no)
Anything that affects deployment: (no)

Documentation
Check the box below or label this PR directly.
Need to update docs?

[X] no-need-doc

@codelipenghui @jiazhai PTAL

Thanks, @liudezhi2098 for the help. Why this change will fix this issue? Would you please help provide more information in the PR description? such as the reason behind the original issue, and the mechanism behind this fix. So it would be better for the community to understand this issue.

Thanks, @liudezhi2098 for the help. Why this change will fix this issue? Would you please help provide more information in the PR description? such as the reason behind the original issue, and the mechanism behind this fix. So it would be better for the community to understand this issue.
OK

/pulsarbot rerun-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks


text:
[pulsar-broker] Fix delete empty namespace with partitioned-topic metadata

Motivation
Right now, broker gives below error while deleting empty namespace without any topic under the namespace but namespace was having partitioned-topic metadata.
pulsar-admin namespaces delete prop/us-west/my-ns
Cannot delete non empty namespace

Reason: Cannot delete non empty namespace

Modification
It's because broker was incorrectly assuming topic existence by checking partitioned-topic metadata. delete-admin API cleans up partitioned-metadata after successfully deleting namespace and it it should not incorrectly depend on partitioned-metadata check. this issue is already fixed with forcefully delete-ns admin-api and delete-admin also needs the same fix.
Result
User can delete namespace after successfully deleting all topics under the namespace.

@rdhabalia is this a bug fix and we do not need to update docs?

this PR doesn't need doc update.

@rdhabalia LGTM but the PR needs rebasing

authors: Anonymitaet dlg99 rdhabalia


text:
[fix][test] Fix flaky test ReplicatorTest.testDoNotReplicateSystemTopic

Fixes #19297
Motivation
ReplicatorTest.testDoNotReplicateSystemTopic is flaky, see #19297
Modifications

Move topic creation before creating the consumer.

Creating the consumer created a race for creating the topic via replication and that caused the flakiness

Fix invalid way to check for topic stats. The usage of Awaitility was invalid in this case and it caused a 10 second delay.

Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

/pulsarbot rerun-failure-checks


text:
[flink-consumer-source] fix up streaming to be consistent with batch

fix up streaming to be consistent with batch
Supplementary streaming README.me

@ambition119 Thanks for this PR. Overall LGTM. Just a few minor point.

@ambition119 Thanks for this PR. Overall LGTM. Just a few minor point.

Thank you for your comments.

@erenavsarogullari Could you help run C++ / Python Tests ? tks.

rerun cpp tests

@sijie conflict with the master, Please review it carefully

@sijie  Could you help run C++ / Python Tests ? tks.
[ERROR] Tests run: 27, Failures: 1, Errors: 0, Skipped: 19, Time elapsed: 50.22 s <<< FAILURE! - in org.apache.pulsar.client.impl.BrokerClientIntegrationTest
2018-12-31\T\06:43:33.971 [ERROR] setup(org.apache.pulsar.client.impl.BrokerClientIntegrationTest)  Time elapsed: 10.36 s  <<< FAILURE!
org.apache.pulsar.broker.PulsarServerException: java.io.IOException: Failed to bind Pulsar broker on /0.0.0.0:22138

@ambition119 at any time, you can comment with "run java8 tests" to trigger running CI.

authors: ambition119 erenavsarogullari sijie


text:
Update and cleanup test dependencies

Update testng ver
Update testcontainers ver
Remove arquillian references
Remove jmockit references

run cpp tests

run integration tests

run java8 tests

run cpp tests

run integration tests

@merlimat @sijie @srkukarni
please merge this

please merge this


text:
[Doc]--Update proxy config file for Pulsar proxy gateway improvement

(If this PR fixes a github issue, please add Fixes #<xyz>.)
Fixes #
(or if this PR is one task of a github issue, please add Master Issue: #<xyz> to link to the master issue.)
Master Issue: #
Motivation
Explain here the context, and why you're making that change. What is the problem you're trying to solve.
Modifications
Describe the modifications you've done.
Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API: (yes / no)
The schema: (yes / no / don't know)
The default values of configurations: (yes / no)
The wire protocol: (yes / no)
The rest endpoints: (yes / no)
The admin cli options: (yes / no)
Anything that affects deployment: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? ( docs)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

Update the reference > pulsar configuration > Pulsar proxy section for the Pulsar 2.4.0 released PR: https://github.com/apache/pulsar/pull/3915/files#diff-8149d86481529536e6153c07dae4d5a5
only docs in the master is updated. once the contents is reviewed and approved. will update the doc to previous releases since 2.4.0.

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

close the PR as CI fails and finds no method to troubleshoot it. will re-create a PR for updating the doc.

authors: Huanli-Meng jiazhai


text:
[Issue 7736][broker]Fix broker lookup return wrong url when specify advertised listener

Motivation
Fixes #7736
Modifications
In function searchForCandidateBroker(), return the url according to advertisedListenerName when the candidateBroker is not self.

/pulsarbot run-failure-checks

@wangjialing218 Can you write a simple test case?

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

@sijie @wolfstudy  Can this pr be cherry-picked to upcoming release 2.6.1, we also need this fix in our product env.

@sijie @wolfstudy Can this pr be cherry-picked to upcoming release 2.6.1, we also need this fix in our product env.

Sure, please make sure the Action CI pass, will cherry-pick this change to branch-2.6

/pulsarbot run-failure-checks

@sijie @jiazhai @codelipenghui PTAL thanks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

@wangjialing218 Can you merge the master code to fix CI error?

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks


text:
Support key_based batch builder for functions and sources

Signed-off-by: xiaolong.ran rxl@apache.org
Motivation
Currently, we support the Key_Shared subscription mode in Pulsar Function. In order to ensure that when batch is turned on, we can also ensure that messages are distributed to different consumers in the correct order, so we need to support the batch builder of KEY_BASED in Pulsar Functions.
Modifications

Add --batch-builder for Pulsar Functions
Add --batch-builder for Pulsar Sources
Add test case

/pulsarbot run-failure-checks

Overall looks good. However, I didn't see any test case. Can you add one?

The current unit tests can be covered. The change of this pull request is to add a batchBuilder field to ProducerConfig. The current unit test framework covers the Pojo object ProducerConfig, so we only need to add this field to the existing unit test framework. Refer to pulsar-functions/utils/src/test/java/org/apache/pulsar/functions/utils/SourceConfigUtilsTest.java and pulsar-functions/utils/src/test/java/org/apache/pulsar/functions/utils/FunctionConfigUtilsTest.java.

You mean, do we add another integration test on top of the unit test?

@sijie PTAL again thanks. I added the --batch-builder parameter to the existing integration test to confirm whether this parameter is correctly identified.