text:
[ISSUE #1283] update go server connector api

Motivation
#1283
update go server connector api:

add RequestReplyCallback as request argument of Producer.Request

Modifications
update go server connector api:

add RequestReplyCallback as request argument of Producer.Request

Documentation
None

Could you please combine the consumer plugin and producer pulgin together, such as standalone connector and rocketmq connector. There will be no mixed plugin mode, such as standalone producer with rocketmq consumer.

Could you please combine the consumer plugin and producer pulgin together, such as standalone connector and rocketmq connector. There will be no mixed plugin mode, such as standalone producer with rocketmq consumer.

ok, I will update the connector interface to combine consumer/producer into one plugin type,  and I will submit it in a new issue and pr.

Could you please combine the consumer plugin and producer pulgin together, such as standalone connector and rocketmq connector. There will be no mixed plugin mode, such as standalone producer with rocketmq consumer.

ok, I will update the connector interface to combine consumer/producer into one plugin type, and I will submit it in a new issue and pr. close it first.

Great


text:
[Improve][TubeMQ]  repHelperHost for master should be exposed in configuration

Description
Currently, repHelperHost in master.ini isn't exposed.
See https://github.com/apache/incubator-inlong/blob/master/inlong-tubemq/conf/master.ini?#L70
This is a key configuration for Tube master especially when you build a failover master service.
So, it should be exposed, also, we should add some guides for this configuration
InLong Component
InLong TubeMQ
Are you willing to submit PR?

[ ] Yes, I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

@aloyszhang, you are right, the parameters of the copy are very important, and each copy group requires a different parameter configuration; commenting it out will make the outside mistakenly think that only the default value is needed.

Can you contribute a PR?

Yeah, I'll send a pull request soon.

source: gharchive/issue

authors: aloyszhang gosonzhang

repo: apache/incubator-inlong

license_type: permissive

license_source: github-api


text:
[KYUUBI #3010] Bump Jetty from 9.4.41.v20210516 to 9.4.48.v20220622

Why are the changes needed?

close #3010
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

dev/dependencyList File should be updated

Thanks, merging to master

source: gharchive/pull-request

repo: apache/incubator-kyuubi


text:
Fix flaky test - AdminResourceSuite: delete engine - user share level

Why are the changes needed?

close #3751
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

UT passed one time, try again

retrying https://github.com/apache/incubator-kyuubi/actions/runs/3384044997

Passed again

try again

cc @cfmcgrady @wForget
I do not know why indeed.

I'm also confused about this, but this fix looks good.

cc @cfmcgrady @wForget
I do not know why indeed.

Is it possible that serviceNode: PersistentNode is automatically recreate after being deleted?

Is it possible that serviceNode: PersistentNode is automatically recreate after being deleted?

Not sure, but it did can be deleted finally.

- JpsApplicationOperation with jstat *** FAILED ***
  null equaled null (JpsApplicationOperationSuite.scala:56)
  S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT    CGC    CGCT     GCT
  0.00 100.00  92.04  78.54  96.79  87.17     45    0.786     0    0.000    22    0.470    1.255
  0.00 100.00  93.81  78.54  96.79  87.17     45    0.786     0    0.000    22    0.470    1.255

thanks, merging to master

authors: turboFei wForget


text:
[LIVY-11] Enable HA support

What changes were proposed in this pull request?
This pull request enables Active/Passive HA in Livy through the use of Zookeeper. The CuratorElectorService class coordinates leadership election between multiple Livy instances
https://issues.apache.org/jira/browse/LIVY-11
How was this patch tested?
unit tests
manual verification using 2 Livy instances

How does this pr related to pr #189 ?

Do you need a proxy-like thing to find the latest active node?

How do the clients know the latest active node? Do they need to ask apache curator? If so, do we need to update the client code?

Another question is if the current leader crash and a new leader is elected, should the session state in the old leader be restored in the new leader?

Another question is if the current leader crash and a new leader is elected, should the session state in the old leader be restored in the new leader?

The code does not do the state synchronization related to the Session State. Is it a good choice to put it in zk?

How do the clients know the latest active node? Do they need to ask apache curator? If so, do we need to update the client code?
When people use a restful client(e.g. curl) to communicate with Livy, should they query zk every time?

#189 implements Livy Server discovery to provide the ability to get current Livy Server address by client to use it in REST requests. There is Java/Scala API for it. If someone uses not Java world client then he should ask ZK to get the address.

When HA mode is enabled, the following happens.
Once new Livy Server starts, it uses Curator to connect to the Zookeeper state store to try to acquire leadership. The process will wait until it is able to acquire leadership. The leader latch library will automatically ensure that only 1 Livy Server among those that are connect to that state store will be elected Leader at any point.
Once it acquires leadership, it starts the Livy Web Server in the same way that it's currently handled, recovering session state if it is configured and all other associated configurations.
The client ideally doesn't need to do anything with respect to contacting the zookeeper server, as ideally you have a load balancer of some sort failover controller to eventually link the client to the actively running Livy instance. This solution is primarily concerned with ensuring that given a group of Livy Servers deployed in HA mode, exactly one of them will be active and running at any one time.

Another question is if the current leader crash and a new leader is elected, should the session state in the old leader be restored in the new leader?

The code does not do the state synchronization related to the Session State. Is it a good choice to put it in zk?

This is already implemented previously. You can set livy.server.recovery.state-store to "zookeeper" and livy.server.recovery.state-store.url to zookeeper quorum. This PR mainly concerns the leader election part for the active-passive Livy Server.

How do the clients know the latest active node? Do they need to ask apache curator? If so, do we need to update the client code?
When people use a restful client(e.g. curl) to communicate with Livy, should they query zk every time?

#189 implements Livy Server discovery to provide the ability to get current Livy Server address by client to use it in REST requests. There is Java/Scala API for it. If someone uses not Java world client then he should ask ZK to get the address.

Another solution is we can put all active/standby services behind a load balancer. The client only needs to communicate with that load balancer. Then there's no need to change the client-side code. The server switch should be transparent to the client.

Can someone help get this feature approved and checked in?
@jerryshao

How do you do state transfer from old server to newly elected  livy server? How do you do server discovery? How do you do session recovery? I was wondering such a simple solution is good enough. Also there's no UT and IT for the code, I don't think it is ready to merge.

State transfer and session recovery are handled by the pre-existing livy.server.recovery.mode and livy.server.recovery.state-store components, provided they are configured to a Zookeeper instance.
Server discovery is decentralized though, would it be better to have an explicit configuration? I feel like having that wouldn't give any user-side benefits, given that there is no external command line tool to check the status of Livy instances.

Livy HA with redirection:
In this pull request we are adding Active/Standby High Availability (HA) to Livy Server. The goal of HA is to ensure that amoung a group of Livy Servers, only 1 is Active at
any given time while the others are on Standby. Our changes are as following:
Leveraging Zookeeper to handle Leader Election:
In order to achieve HA, we need a service to handle keeping track of the current leader. To do this, we created a CuratorElectorService class which will register the Livy Server
instance into the Leader election group and keep track of the leader. This class leverages the Apache Curator LeaderLatch Library to keep track of the current leader.
This class is also responsible for restarting the Livy Webserver when it transitions from Standby state to Active. This restart is where the Livy Webserver performs session recovery.
Redirecting to the active webserver:
With the CuratorElectorService class, Livy Server can now determine whether it is Active or on Standby. If it is active, it proceeds as usual. However, if it is on standby, it will
use the configuration in livy.conf to determine the endpoint to redirect the request to.
Creating configurations to enable HA:
We added user configurations to enable HA. These include specifying whether or not HA is enabled, the endpoints of the Livy Server instances used for HA, and the zookeeper instances
used for leader election.
Splitting Livy's start() function to init() and start():
Currently, Livy's start() function is overloaded to both initalize all of its components (the webserver, its servlets, and session managers) and then safely start them. This PR
separates the start() function into an init() component and a start() components. This is to allow us to restart the session managers and trigger session recovery without having
to safely tear down and rebuild the Livy Webserver every time.
@jerryshao can I get some eyes for a code review/suggestions?

@jerryshao small ping for a review

@jerryshao Another ping

@jerryshao pinging again

@jerryshao another ping

@jerryshao

@jerryshao

@jerryshao

Taking over the pull request with @RogPodge 's permission. A bug involving query omission on failover was fixed, along with a bit of minor refactoring and renaming.
I'm not too familiar with the code review structure of this repo, but can someone take a look at the PR and its changes? (Pinging @jerryshao based on prior messages as well)

authors: OpenOpened RogPodge jameschen1519 jerryshao o-shevchenko shanyu yantzu yiheng

repo: apache/incubator-livy


text:
MADLIB-1133. TLP graduation - remove references to "incubating".

o Remove references to "(incubating)" scattered about.
o Remove DISCLAIMER file - not needed as Apache Top-Level Project (TLP)
o Remove incubation paragraphs:
Apache MADlib is an effort undergoing incubation at the Apache Software
Foundation (ASF), sponsored by the Apache Incubator PMC.

Incubation is required of all newly accepted projects until a further review
indicates that the infrastructure, communications, and decision making process
have stabilized in a manner consistent with other successful ASF projects.

While incubation status is not necessarily a reflection of the completeness or
stability of the code, it does indicate that the project has yet to be fully
endorsed by the ASF.

o Rename git repo references:
example: (from) https://github.com/apache/incubator-madlib.git
(to)   https://github.com/apache/madlib.git
o Rename mail archive references:
example: (from) https://mail-archives.apache.org/mod_mbox/incubator-madlib-dev/
(to)   https://mail-archives.apache.org/mod_mbox/madlib-dev/
o Rename references to git work directories:
example: (from) ${workdir}/incubator-madlib/.${extension}"
(to)   ${workdir}/madlib/.${extension}"
o The jenkins_build.sh & rat_check.sh scripts are partially
updated. They will need to be updated again (change value of
reponame variable) when the actual git repo name changes.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/136/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/137/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/138/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/139/

@njayaram2 great idea -- why don't you just reuse the file and put something like this there:
Be advised that the registered trademark for MADlib is in a process of being transferred to the Apache Software Foundation. Please refer to https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-125 for more details.

I have filed MADLIB-1144 to track the removal of the DISCLAIMER file once the MADlib registered trademark is fully transferred to ASF.

+1
LGTM

Jenkins please retest.

Jenkins ok to test

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/149/

authors: asfgit edespino njayaram2 rvs

repo: apache/incubator-madlib

license_source: bigquery


text:
about the IRHeader when using pack and unpack

#12544 I am packing a multi-label dataset, and use header.flag, but after pack and unpack, the flag will be always equal to the label length, is it by design or a bug?

I am looking at the code for IRHeader, pack, and unpack and it looks like it is by design https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/recordio.py#L340. The flag is used for label size throughout. If so, we should update the documentation to better explain this.
Any idea about this @Roshrini?
@mxnet-label-bot add [Data-loading]

yes, I have read it too, if we want to use it as we want, we can write a custom pack and unpack

authors: HaoLiuHust zachgk

repo: apache/incubator-mxnet


text:
most time cost in NDArray GetData() func

My predict code is like this， I have 32 worker thread, each has 1 openmp thread, the mxnet engine is NaiveEngine , using the openblas, disable the mkldnn, running the code on the Skylatex machine, using the avx512, but it is not faster than the avx2, I don't know why?
And what's more, the naiveEngine means that all the funcs are sequential？(not asynchronous?)
In this condition, the WaitAll() func should removed?
what's more , I found that the most time is the NDArray.GetData(), about 17ms, and the Forward() cost 6ms, why GetData() cost so much time? How to increase the proformance?
the Avx512 doesn't work.
    dnn_model->model_data["data"].SyncCopyFromCPU(batch_data.data(), batch_size * fea_num);
    mxnet::cpp::NDArray::WaitAll();
    dnn_model->exec->Forward(false);
    mxnet::cpp::NDArray::WaitAll();
    uint32_t output_num = dnn_model->exec->outputs.size();
    for (uint32_t j = 0; j < b_num; j++) {
        for (uint32_t q_idx = 0; q_idx < output_num; q_idx++) {
            //cvm_q_t* item = result + ins_idx;
            //item->q[q_idx]  = dnn_model->exec->outputs[q_idx].GetData()[j];
            _result[ins_index][q_idx] = dnn_model->exec->outputs[q_idx].GetData()[j];
            CVM_DEBUG_LOG("ins_index:%u  q_idx %u q:%f", ins_index, q_idx, _result[ins_index][q_idx]);
        }
		ins_index += 1;
    }

@mxnet-label-bot add [question, performance]

I am wondering the same thing. Can we omit the WaitAll if we are running the NaiveEngine?

authors: cyrusbehr frankfliu gxkevin


text:
[v1.4.x] add license to pom files

Description
Add License to XML
@szha @zachgk @aaronmarkham @piyushghai
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Errors:
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-gpu/detail/PR-14155/1/pipeline#step-97-log-1537 (test_stn)
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-14155/1/pipeline/1112#step-1838-log-1537 (test_stn)
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwebsite/detail/PR-14155/1/pipeline#step-114-log-1420 (clojure doc, also happened in another PR)
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-14155/1/pipeline/1121#step-1815-log-2420 (R covr)
No issue found for scala.


text:
Add -R option to ci/build.py to avoid rebuilding containers

There are a couple of problems which trigger continuous rebuild of the containers which have difficult solutions, this option enables a workaround.
https://github.com/docker/docker.github.io/issues/8886
@marcoabreu
In one ec2 machine:
Sending build context to Docker daemon  264.2kB                                                                                [73/1937]
Step 1/41 : FROM ubuntu:16.04
 ---> 13c9f1285025
Step 2/41 : WORKDIR /work/deps
 ---> Using cache
 ---> fd89693ec7d8
Step 3/41 : COPY install/ubuntu_core.sh /work/
 ---> Using cache
 ---> acc84b2e4ad2
Step 4/41 : RUN /work/ubuntu_core.sh
 ---> Using cache
 ---> 97d47815b79e
Step 5/41 : COPY install/deb_ubuntu_ccache.sh /work/
 ---> Using cache
 ---> 458bcdc2b9e2

Different machine:
build.py: 2019-07-02 01:05:05,098Z INFO Running command: 'docker build -f docker/Dockerfile.build.ubuntu_cpu --build-arg USER_ID=624416249 --build-arg GROUP_ID=600260513 --cache-from mxnetci/build.ubuntu_cpu -t mxnetci/build.ubuntu_cpu docker'
Sending build context to Docker daemon  264.2kB
Step 1/41 : FROM ubuntu:16.04
 ---> 13c9f1285025
Step 2/41 : WORKDIR /work/deps
 ---> Using cache
 ---> 44a97319ecd8
Step 3/41 : COPY install/ubuntu_core.sh /work/
 ---> d0284928a471
Step 4/41 : RUN /work/ubuntu_core.sh
 ---> Running in cc6d4d02a210
+ apt-get update
Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [109 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1558 kB]

I haven't found a solution for this and I think is related to different user id's and problems with docker caching.

@mxnet-label-bot add [CI]

Ye it's probably user id - for whatever reason docker cares about that...

Danke fuer die merge.

authors: larroy marcoabreu roywei


text:
add inference mode for op perf benchmark

As one of the efforts to address https://github.com/apache/incubator-mxnet/issues/15429
We need to run op perf in inference mode as well as in 1.4.1 where runtime_feature is not available.

Thanks for the improvement.
Could you paste the output of inference OP for both CPU and GPU?

@mxnet-label-bot add [performance, pr-work-in-progress]

authors: pengzhao-intel roywei


text:
[Numpy] add new ffi for np.linalg.norm

Description
add new ffi for np.linalg.norm
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

@hzfan

@mxnet-bot run ci [unix-gpu, centos-gpu]


text:
Fix LeakyRELU Bug

mx.symbol.LeakyReLU crashes when using vector inputs. For example, this crashes for the forward pass:
a = mx.sym.var('data')
s=mx.symbol.LeakyReLU(data=a, act_type="elu")
e=s.simple_bind(mx.cpu(), data=(2,))
e.forward()

This PR is a simple fix, along with a test for mx.sym.LeakyReLU(x, act_type="elu", slope=1.0) for vector inputs.

@piiswrong: the addition of a test for ELU is causing failures in test_operator.test_broadcast_binary_op:
Items are not equal:
Error 14.595687 exceeds tolerance rtol=0.010000, atol=0.001000.  Location of maximum error:(0, 0, 4), a=0.014596, b=0.000000
 NUMERICAL_a: array([[[ 1.72120333,  1.26115978,  1.4104917 ,  1.06704044,  0.01459569]],

       [[ 0.        ,  0.        ,  0.        ,  0.        ,  0.44905394]],...
 BACKWARD_a: array([[[ 1.72122073,  1.26117206,  1.41049075,  1.06703949,  0.        ]],

       [[ 0.        ,  0.        ,  0.        ,  0.        ,  0.4490537 ]],...

This PR makes no changes to any dependencies of this test, so the only explanation is that different random numbers are used, and this is causing a violation of the tolerance bound.
What should I do?

This doesn't look like a random number problem.
I think something really is broken.
Please try rebasing to master first

Ok, done.

This reliably fails on the same test. So I think these is a bug.

Yes. Going to close this and make a new PR.

authors: piiswrong sbodenstein


text:
Fix check of kvstore type

Description
kvstore type can be something like local as well as have device in it. For such kvstores as well gradient compression is supported
Checklist
Essentials

[ ] Passed code style checking (make lint)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage
[ ] For user-facing API changes, API doc string has been updated. For new C++ functions in header files, their functionalities and arguments are well-documented.
[ ] To my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

LGTM


text:
License fixes

Description
(Brief description on what this PR is about)
Checklist
Essentials

[ ] Passed code style checking (make lint)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage
[ ] For user-facing API changes, API doc string has been updated. For new C++ functions in header files, their functionalities and arguments are well-documented.
[ ] To my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

mshadow and nnvm are not apache projects. They don't need this header

So which ones do?  We got a downvote by Apache on the release due to problems with licensing headers in files in these modules.

It would help if some dmlc people got on general@ and disputed these issues rather than us Apache committers try to make a case for dmlc people's wishes and restrictions.

We should solve this by not including any 3rdparty source code in the release tar.
For example we can't add asf header to nvdia cub

Looks ok to me.
The submodule changes are not being merged, right?

Right, no submodule changes are being merged.

As suggested, I will move cub to 3rdparty in another PR

authors: cjolivier01 mbaijal piiswrong


text:
merge master branch into bluetooth5

some minor conflicts in controller code were resolved

lgtm

Looks good

authors: andrzej-kaczmarek rymanluk sjanc

repo: apache/incubator-mynewt-core


text:
Fix the minor style issue

Summary
Impact
Minor
Testing
Pass CI

Please ignore the style warning.

Please rebase

@xiaoxiang781216 could you please take a look on CI failures? I restarted it couple of time, but failures persist

Done.

repo: apache/incubator-nuttx-apps


text:
Fix CLE initial cursor position

Summary
As noted by @v01d in incubator-nuttx#2838, there was a missing whitespace after the NSH prompt.
It seems it was due to a bug in the CLE application, which was miscalculating the initial cursor position. At first, after printing the "nsh> " prompt the Host correctly reports the initial cursor position as ESC ] 39 ; 6 R, which means row:39 and column:6.
But, next the CLE app calculates an offset of 5 (column - 1) for the initial cursor position. But, this offset is sent to the Host later as if the first column was 0.
NuttShell (NSH) NuttX-10.0.1
nsh> Returning: .[1b]
Returning: [[5b]
Returning: 3[33]
Returning: 9[39]
Returning: ;[3b]
Returning: 6[36]
Returning: R[52]
row=39 column=6
row=39 column=6
row=39 column=0 offset=5
row=39 column=0 offset=5

According to the VT100 documentation, the column for the initial cursor position is 1:

Impact
Testing
Tested on:

esp32c3:nsh
esp32:nsh

I don't remember seeing this issue in other archs (but maybe I'm wrong). Do you think this bug could depend on arch somehow? Maybe it would be good to verify that other arch still works as expected.

I don't remember seeing this issue in other archs (but maybe I'm wrong). Do you think this bug could depend on arch somehow? Maybe it would be good to verify that other arch still works as expected.

I've tried to test it on sim:nsh, but I had some trouble building the simulator on OSX.
Unfortunately I don't have any other boards here with me at the moment =/

I don't remember seeing this issue in other archs (but maybe I'm wrong). Do you think this bug could depend on arch somehow? Maybe it would be good to verify that other arch still works as expected.

I can test in a few hours, if anyone else does not test it sooner.

I realize now that there aren't that many configs with SYSTEM_CLE, which would explain why it was only present in this case. In fact I remember seeing this issue when enabling CLE on some board.


text:
Cannot build Haskell binding on MacOS M2

Install toolchain with: brew install ghc cabal-install
$ ghc -V

The Glorious Glasgow Haskell Compilation System, version 9.8.1

$ cabal -V
cabal-install version 3.10.1.0
compiled using version 3.10.1.0 of the Cabal library

Gets:
$ cabal update
Downloading the latest package list from hackage.haskell.org
Package list of hackage.haskell.org has been updated.
The index-state is set to 2023-12-29T05:19:00Z.

$ cabal haddock
Resolving dependencies...
Error: cabal: Could not resolve dependencies:
[__0] trying: opendal-0.1.0.0 (user goal)
[__1] next goal: base (dependency of opendal)
[__1] rejecting: base-4.19.0.0/installed-inplace (conflict: opendal =>
base>=4.10 && <4.17)
[__1] skipping: base-4.19.0.0, base-4.18.1.0, base-4.18.0.0, base-4.17.2.1,
base-4.17.2.0, base-4.17.1.0, base-4.17.0.0 (has the same characteristics that
caused the previous version to fail: excluded by constraint '>=4.10 && <4.17'
from 'opendal')
[__1] rejecting: base-4.16.4.0, base-4.16.3.0, base-4.16.2.0, base-4.16.1.0,
base-4.16.0.0, base-4.15.1.0, base-4.15.0.0, base-4.14.3.0, base-4.14.2.0,
base-4.14.1.0, base-4.14.0.0, base-4.13.0.0, base-4.12.0.0, base-4.11.1.0,
base-4.11.0.0, base-4.10.1.0, base-4.10.0.0, base-4.9.1.0, base-4.9.0.0,
base-4.8.2.0, base-4.8.1.0, base-4.8.0.0, base-4.7.0.2, base-4.7.0.1,
base-4.7.0.0, base-4.6.0.1, base-4.6.0.0, base-4.5.1.0, base-4.5.0.0,
base-4.4.1.0, base-4.4.0.0, base-4.3.1.0, base-4.3.0.0, base-4.2.0.2,
base-4.2.0.1, base-4.2.0.0, base-4.1.0.0, base-4.0.0.0, base-3.0.3.2,
base-3.0.3.1 (constraint from non-upgradeable package requires installed
instance)
[__1] fail (backjumping, conflict set: base, opendal)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: base, opendal

$ cabal install
Error: cabal: Could not resolve dependencies:
[__0] trying: opendal-0.1.0.0 (user goal)
[__1] next goal: base (dependency of opendal)
[__1] rejecting: base-4.19.0.0/installed-inplace (conflict: opendal =>
base>=4.10 && <4.17)
[__1] skipping: base-4.19.0.0, base-4.18.1.0, base-4.18.0.0, base-4.17.2.1,
base-4.17.2.0, base-4.17.1.0, base-4.17.0.0 (has the same characteristics that
caused the previous version to fail: excluded by constraint '>=4.10 && <4.17'
from 'opendal')
[__1] rejecting: base-4.16.4.0, base-4.16.3.0, base-4.16.2.0, base-4.16.1.0,
base-4.16.0.0, base-4.15.1.0, base-4.15.0.0, base-4.14.3.0, base-4.14.2.0,
base-4.14.1.0, base-4.14.0.0, base-4.13.0.0, base-4.12.0.0, base-4.11.1.0,
base-4.11.0.0, base-4.10.1.0, base-4.10.0.0, base-4.9.1.0, base-4.9.0.0,
base-4.8.2.0, base-4.8.1.0, base-4.8.0.0, base-4.7.0.2, base-4.7.0.1,
base-4.7.0.0, base-4.6.0.1, base-4.6.0.0, base-4.5.1.0, base-4.5.0.0,
base-4.4.1.0, base-4.4.0.0, base-4.3.1.0, base-4.3.0.0, base-4.2.0.2,
base-4.2.0.1, base-4.2.0.0, base-4.1.0.0, base-4.0.0.0, base-3.0.3.2,
base-3.0.3.1 (constraint from non-upgradeable package requires installed
instance)
[__1] fail (backjumping, conflict set: base, opendal)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: base, opendal

cc @silver-ymz

This is because co-log package only supports ghc-9.2 before v0.6.0. You need to dnowgrade haskell version temporarily. Issue https://github.com/apache/incubator-opendal/issues/3853 is opened for it.

Thank you. Closed in favor of #3853.

authors: silver-ymz tisonkun

repo: apache/incubator-opendal


text:
adapt tests for provide-api-key annotation upstream

bumped gradle version due to SSL errors
copied changes from https://github.com/apache/incubator-openwhisk/pull/4423

This PR should be merged without squashing independent commits in case we need to roll back. Ideally orthogonal changes are in different PRs.

@matthiaskubik I'm going to bump the gradle version to 4.10.2 in this PR https://github.com/apache/incubator-openwhisk-cli/pull/395. This PR then will address just the test changes. Sorry for the confusion. I'm trying to reduce the work for you.

@matthiaskubik i rebased your PR to master to pick up the gradle bump and retitled your PR accordingly. Thanks for the contribution and fixing the tests.

Thanks @matthiaskubik for the contribution.

authors: matthiaskubik rabbah

repo: apache/incubator-openwhisk-cli


text:
Travis errors due to missing controller:latest image.

I'm seeing failures in Travis this morning for Apache OpenWhisk repos due to the controller:latest image being referenced. I'm guessing this is due to the move to use nightly in accordance with ASF guidelines.
https://travis-ci.org/apache/incubator-openwhisk-client-js/builds/554135693
https://travis-ci.org/apache/incubator-openwhisk-cli/builds/554119370
fatal: [controller0]: FAILED! => {"changed": false, "msg": "Error pulling image openwhisk/controller:latest - 404 Client Error: Not Found (\"manifest for openwhisk/controller:latest not found\")"}

This comes from the following task:
https://github.com/apache/incubator-openwhisk/blob/ef8b06f499764d38b29c1a8095fba09836d666a1/ansible/roles/controller/tasks/deploy.yml#L312-L331
The docker.image.tag uses latest I guess? Should that config value be changed to nightly?

I think change the default_image_tag to nightly in ansible/group_vars/all should be better comparing to change travis configuration in other projects like incubator-openwhisk-cli

This seems sensible to me but I'm not familiar enough with the configuration to make a judgement on whether this is the best approach. Will let someone more experienced decide....
cc @rabbah @dgrove-oss

With #4538, we need to change this to nightly as well to make Jenkins build work because it uses a private registry.
https://github.com/apache/incubator-openwhisk/blob/f8a1506b2edc5e61476e44b924962a567c283b22/gradle/docker.gradle#L43
But this will change the behavior of local build and deployment via a private registry.
I think it is correct to use latest if a user uses images built in a local environment with a private registry(deployed via OpenWhisk).
And for build jobs such as CLI and client-js, I think it is correct to use nightly built official images.
Because it just requires them as a dependency and requires a "fresh" version from the master branch.
So I am inclined to change CI configuration of CLI and client-js instead replacing default_image_tag.

I think the right thing to do is to change downstream repos travis scripts to be specific that they are pulling nightly where appropriate.   Trying it for the cli here https://github.com/apache/incubator-openwhisk-cli/pull/450.

I believe this is all sorted out now.

authors: dgrove-oss jiangpengcheng jthomas style95

repo: apache/incubator-openwhisk


text:
[Feature] Support integration with JuiceFS

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation
JuiceFS is a high-performance POSIX file system released under Apache License 2.0, particularly designed for the cloud-native environment. The data, stored via JuiceFS, will be persisted in object storage (e.g. Amazon S3), and the corresponding metadata can be persisted in various database engines such as Redis, MySQL, and TiKV based on the scenarios and requirements. Paimon could support integration with JuiceFS.
Solution
Paimon supports integration with JuiceFS.
Anything else?
No response
Are you willing to submit a PR?

[X] I'm willing to submit a PR!

I'm willing to submit a PR!

repo: apache/incubator-paimon


text:
[Feature] Add Procedure in flink for creating and deleting branch

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation
subtask of #1795 Add Procedure in flink to support creating and deleting branch
Solution
No response
Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

Please assign to me. Thanks @FangYongs . I will continue to improve the branch feature.

@TaoZex I have create a PR for it, you can help to review it, thanks

@FangYongs Thanks for your response. I have reviewed it.

authors: FangYongs TaoZex


text:
Fix KafkaPartitionLevelConsumerTest by using NetUtils to fetch the localhost ip

Also move NetUtils (renamed from NetUtil) to pinot-spi so that it can be accessed from the kafka package
Here are some explanations on why it works: https://stackoverflow.com/questions/9481865/getting-the-ip-address-of-the-current-machine-using-java

Codecov Report

Merging #6784 (d96966f) into master (3f93cfb) will decrease coverage by 30.42%.
The diff coverage is 45.45%.

@@              Coverage Diff              @@
##             master    #6784       +/-   ##
=============================================
- Coverage     74.14%   43.72%   -30.43%
+ Complexity       12        7        -5
=============================================
  Files          1412     1412
  Lines         68762    68763        +1
  Branches       9934     9934
=============================================
- Hits          50986    30069    -20917
- Misses        14461    36176    +21715
+ Partials       3315     2518      -797

Flag
Coverage Δ
Complexity Δ

integration
43.72% <45.45%> (-0.16%)
0.00 <0.00> (ø)

unittests
?
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ
Complexity Δ

.../pinot/broker/broker/helix/HelixBrokerStarter.java
66.11% <0.00%> (-2.23%)
0.00 <0.00> (ø)

...org/apache/pinot/controller/ControllerStarter.java
69.47% <0.00%> (-3.74%)
0.00 <0.00> (ø)

...main/java/org/apache/pinot/spi/utils/NetUtils.java
0.00% <0.00%> (ø)
0.00 <0.00> (?)

...in/java/org/apache/pinot/minion/MinionStarter.java
81.25% <50.00%> (ø)
0.00 <0.00> (ø)

...r/api/resources/PinotControllerResponseFilter.java
69.23% <100.00%> (ø)
0.00 <0.00> (ø)

...nt/virtualcolumn/VirtualColumnProviderFactory.java
62.50% <100.00%> (ø)
0.00 <0.00> (ø)

...pinot/server/starter/helix/HelixServerStarter.java
52.27% <100.00%> (ø)
0.00 <0.00> (ø)

...c/main/java/org/apache/pinot/common/tier/Tier.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

...ava/org/apache/pinot/spi/data/MetricFieldSpec.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

...ava/org/apache/pinot/core/util/fst/FSTBuilder.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

... and 882 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3f93cfb...d96966f. Read the comment docs.

Fixed in #6786

authors: Jackie-Jiang codecov-io

repo: apache/incubator-pinot


text:
Introduced typed property getters to SytemConfiguration.

Replaced multiple places getting a string value and converting to
boolean or int with calls to new typed property getters.
Removed API to clear out all system properties, as some are always
required.
Enhanced javadoc on SystemConfiguration.

+1

+1 to merge

authors: ellisonanne smarthi tellison

repo: apache/incubator-pirk


text:
RYA-325 Renamed rya.console to rya.shell.

Description
Renamed rya.console to rya.shell.
Tests
N/A
Links
Jira
Checklist

[ ] Code Review
[ ] Squash Commits

People To Reivew
@isper3at
@meiercaleb

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/348/

authors: asfgit jdasch

repo: apache/incubator-rya


text:
[ISSUE #2588] org.apache.shenyu.sync.data.etcd.EtcdClient method [getSubNodeKeyName] throw “StringIndexOutOfBoundsException”

org.apache.shenyu.sync.data.etcd.EtcdClient method [getSubNodeKeyName] throw “StringIndexOutOfBoundsException”
// Describe your PR here; eg. Fixes #issueNo
Add protection measures and return null.
Fixes #2588

Make sure that:

[ ] You have read the contribution guidelines.
[ ] You submit test cases (unit or integration tests) that back your changes.
[ ] Your local test passed mvn clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #2589 (115b5a7) into master (832f0ba) will decrease coverage by 0.35%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #2589      +/-   ##
============================================
- Coverage     60.80%   60.44%   -0.36%
+ Complexity     4213     4192      -21
============================================
  Files           654      652       -2
  Lines         18493    18509      +16
  Branches       1660     1651       -9
============================================
- Hits          11244    11188      -56
- Misses         6375     6417      +42
- Partials        874      904      +30

Impacted Files
Coverage Δ

...ache/shenyu/plugin/grpc/cache/GrpcClientCache.java
68.75% <0.00%> (-31.25%)
:arrow_down:

...n/alibaba/dubbo/cache/AlibabaDubboConfigCache.java
45.88% <0.00%> (-28.24%)
:arrow_down:

...gin/apache/dubbo/cache/ApacheDubboConfigCache.java
45.23% <0.00%> (-26.20%)
:arrow_down:

...pache/shenyu/admin/transfer/ConditionTransfer.java
56.09% <0.00%> (-21.96%)
:arrow_down:

...che/shenyu/sync/data/http/HttpSyncDataService.java
61.38% <0.00%> (-19.81%)
:arrow_down:

...henyu/plugin/grpc/resolver/ShenyuNameResolver.java
45.26% <0.00%> (-18.95%)
:arrow_down:

...che/shenyu/admin/service/impl/RuleServiceImpl.java
82.85% <0.00%> (-12.85%)
:arrow_down:

...a/org/apache/shenyu/common/utils/VersionUtils.java
67.85% <0.00%> (-10.72%)
:arrow_down:

...apache/shenyu/admin/utils/ShenyuDictH2Trigger.java
63.63% <0.00%> (-9.10%)
:arrow_down:

...ache/shenyu/admin/utils/PluginHandleH2Trigger.java
63.63% <0.00%> (-9.10%)
:arrow_down:

... and 53 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 832f0ba...115b5a7. Read the comment docs.

authors: codecov-commenter haibo-duan

repo: apache/incubator-shenyu


text:
[fix: bug] fix motan examples

// Describe your PR here; eg. Fixes #issueNo

Make sure that:

[ ] You have read the contribution guidelines.
[ ] You submit test cases (unit or integration tests) that back your changes.
[ ] Your local test passed mvn clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #2647 (d51d470) into master (f84d224) will decrease coverage by 0.61%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #2647      +/-   ##
============================================
- Coverage     61.20%   60.59%   -0.62%
+ Complexity     4330     4295      -35
============================================
  Files           660      660
  Lines         18981    18981
  Branches       1682     1682
============================================
- Hits          11618    11502     -116
- Misses         6407     6536     +129
+ Partials        956      943      -13

Impacted Files
Coverage Δ

...ache/shenyu/plugin/grpc/cache/GrpcClientCache.java
68.75% <0.00%> (-31.25%)
:arrow_down:

...n/alibaba/dubbo/cache/AlibabaDubboConfigCache.java
43.75% <0.00%> (-31.25%)
:arrow_down:

...gin/apache/dubbo/cache/ApacheDubboConfigCache.java
43.03% <0.00%> (-29.12%)
:arrow_down:

...henyu/plugin/grpc/resolver/ShenyuNameResolver.java
44.68% <0.00%> (-19.15%)
:arrow_down:

...che/shenyu/sync/data/http/HttpSyncDataService.java
61.38% <0.00%> (-11.89%)
:arrow_down:

...a/org/apache/shenyu/common/utils/VersionUtils.java
67.85% <0.00%> (-10.72%)
:arrow_down:

.../apache/shenyu/admin/model/dto/BatchCommonDTO.java
50.00% <0.00%> (-7.15%)
:arrow_down:

...apache/shenyu/admin/model/dto/AuthPathWarpDTO.java
57.14% <0.00%> (-7.15%)
:arrow_down:

...org/apache/shenyu/admin/model/dto/AuthPathDTO.java
58.82% <0.00%> (-5.89%)
:arrow_down:

...in/response/strategy/NettyClientMessageWriter.java
69.56% <0.00%> (-4.35%)
:arrow_down:

... and 24 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f84d224...d51d470. Read the comment docs.


text:
add instance id configuration item

Please answer these questions before submitting pull request

Why submit this pull request?
[ ] Bug fix
[x] New feature provided
[ ] Improve performance

Done!

repo: apache/incubator-skywalking


text:
[Bug] Fix the missing data of the menu and add the sql that removes t…

…he admin role in the upgraded 1.2.4.sql #1794

What problem does this PR solve?
Issue Number: close #1794
Problem Summary: Fix the missing data of the menu
What is changed and how it works?
Proposal: https://github.com/apache/incubator-streampark/issues/1794
Feel free to ping committers for the review!
Contribution Checklist

Make sure that the pull request corresponds to a GITHUB issue.

Name the pull request in the form "[Feature] [component] Title of the pull request", where Feature can be replaced by Hotfix, Bug, etc.

Minor fixes should be named following this pattern: [hotfix] [docs] Fix typo in README.md doc.

Purpose of this pull request
Fix the missing data of the menu. issue-1974

/cc @1996fanrui PTAL

authors: macksonmu wolfboys

repo: apache/incubator-streampark


text:
table view viz doesn't use time grain with Sqlalchemy table

Hi,
I want to create table view where counts are grouped by day.
I'm setting time grain to day but this has no effect.
I'm connecting to MySQL table.
Caravel version 0.11.0

Only time related visualization are affected by the time settings: table view is not one of them.

Said that on my pre 0.11 installation i can't use the time column as group by clause. @maver1ck do you see the same on 0.11?

@xrmx
I don't agree with you.
When I'm using druid I can use Time Granularity.

So you agree with me with something is broken :)

Yep.
Thanks for "bug" label.

We are using superset 0.13.2 with druid datasource, setting up a table viz with one day granularity since 7 days ago, but the actual query is with granularity: all, so no time column in the result table, is this a feature or a bug?
Thanks.

I think it's a new bug.

I have make a merge request,https://github.com/airbnb/superset/pull/1809

Notice: this issue has been closed because it has been inactive for 499 days. Feel free to comment and request for this issue to be reopened.

authors: fangyeqing maver1ck mistercrunch xrmx zhaown

repo: apache/incubator-superset


text:
fix: Make spacings in Settings menu more consistent

SUMMARY
Decreases margin for item group title, increases margin for the last element of item group, increases margin for "About" section.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

[x] Has associated issue: https://github.com/apache/incubator-superset/issues/11651
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

CC: @ktmud

Codecov Report

Merging #11709 (6232248) into master (2f036e9) will decrease coverage by 0.78%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #11709      +/-   ##
==========================================
- Coverage   62.85%   62.07%   -0.79%
==========================================
  Files         889      435     -454
  Lines       43053    26858   -16195
  Branches     4015        0    -4015
==========================================
- Hits        27063    16673   -10390
+ Misses      15812    10185    -5627
+ Partials      178        0     -178

Flag
Coverage Δ

javascript
?

python
62.07% <ø> (-0.81%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

superset/views/database/views.py
62.30% <0.00%> (-25.14%)
:arrow_down:

superset/db_engine_specs/mysql.py
79.59% <0.00%> (-12.25%)
:arrow_down:

superset/db_engine_specs/sqlite.py
65.62% <0.00%> (-9.38%)
:arrow_down:

superset/db_engine_specs/presto.py
73.65% <0.00%> (-8.64%)
:arrow_down:

superset/databases/commands/create.py
83.67% <0.00%> (-8.17%)
:arrow_down:

superset/databases/commands/update.py
85.71% <0.00%> (-8.17%)
:arrow_down:

superset/db_engine_specs/base.py
80.53% <0.00%> (-6.94%)
:arrow_down:

superset/sql_validators/base.py
93.33% <0.00%> (-6.67%)
:arrow_down:

superset/views/database/forms.py
83.33% <0.00%> (-5.56%)
:arrow_down:

superset/views/core.py
70.43% <0.00%> (-3.75%)
:arrow_down:

... and 468 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2f036e9...6232248. Read the comment docs.

authors: codecov-io kgabryje


text:
fix: remove unused .js module

found this unused file...

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@003e98c). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #8741   +/-   ##
=========================================
  Coverage          ?   65.87%
=========================================
  Files             ?      482
  Lines             ?    24143
  Branches          ?     2768
=========================================
  Hits              ?    15903
  Misses            ?     8064
  Partials          ?      176

Impacted Files
Coverage Δ

...et/assets/src/explore/components/controls/index.js
100% <ø> (ø)

superset/assets/src/featureFlags.ts
90.9% <ø> (ø)

superset/assets/src/explore/controlUtils.js
90.21% <ø> (ø)

superset/assets/src/explore/controls.jsx
40.14% <ø> (ø)

...rset/assets/src/explore/controlPanels/sections.jsx
100% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 003e98c...0bad2eb. Read the comment docs.

authors: codecov-io mistercrunch


text:
[SYSTEMML-445] [WIP] Initial version of GPU backend

This version calls to CuDNN and CuBLAS using JCuda wrappers. Please DO NOT merge until we add sparse matrix multiplication (in LibMatrixCUDA) and ensure that our integration test suite runs successfully with TEST_GPU flag turned on. Currently, the tensor/* tests and binary/matrix/MatrixMultiplicationTest runs successfully.
@mboehm7 @bertholdreinwald @shirisht @dusenberrymw @nakul02  Please review the PR and provide suggestions.

@nakul02 As we per our discussion, I included the dependency in POM rather than using mavenized JCuda. My guess is that included the jars might be throwing License errors (please see https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/391/):
17:28:04 [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (default) on project systemml: Too many files with unapproved license: 1 See RAT report in: /a/workspace/SystemML-PullRequestBuilder/target/rat.txt -> [Help 1]
If you have time, can you look into license related issues and provide suggestions regarding packaging of JCuda into SystemML ?

I had an offline discussion with @lresende and @deroneriksson.
There were some concerns about the way the pom file is being done.
I'll let them carry the discussion forward.

I would strongly recommend to hold off these major features until we actually made the 0.10 release; let's focus on robustness and fixes for now. Thanks.

@mboehm7 I agree. This PR should not be part of 0.10 release.
@nakul02 Sure, the PR is WIP and all suggestions are welcomed :)
As an FYI, if you want to follow the discussion on this topic, here are few links:

Link to mailing thread (May 3rd): https://www.mail-archive.com/dev@systemml.incubator.apache.org/msg00479.html
Link to JIRA (Jan 10th): https://issues.apache.org/jira/browse/SYSTEMML-445
Link to design document (Feb 26): https://docs.google.com/document/d/1eSBzP8dazetySZusH5cubqUurRH3KDjXJqdizLzVejA/edit

My primary concern is the addition of actual jars to the project and the use of system scope. The maven 'way' is typically to reference artifacts in a maven repository (not necessarily central). Would it be possible to build these jcuda artifacts and put them in a publicly accessibly maven repo and reference this repo in the pom?

Yes, since jcuda is open source, we can certainly create jcuda artifacts. Also, ideally I would prefer that too. However the problem is maintaining a public maven repo and hence I would defer that to Alan and/or Luciano.

We could add the jars on Maven central, or check if @akchinSTC can be of any help here. While first is preferable, latest might be easiest as we actually don't own jcuda.

Just to clarify, whether we go to mvn central or host repo at STC, there are two options:

Deploy both the jar and libs. See https://github.com/MysterionRise/mavenized-jcuda/
Deploy just the jar (i.e. the ones that are included). In this case, we expect users to install the libs (example: libJCublas-linux-x86_64.so or JCublas-windows-x86_64.dll).

@lresende @mboehm7 @deroneriksson @nakul02 I have removed the jars as well as system scope from our pom.xml. Also, I have updated the documentation to provide installation steps for JCuda. Please see https://github.com/niketanpansare/incubator-systemml/blob/1a52bd8908773593fb8b240b38ed0e81a33ebb3f/docs/devdocs/gpu-backend.md#setup-instructions-for-jcudacontext

Test this Jenkins

The tests passed: https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/410

great job for an initial drop. Minor comments:

the dependency on "mavenized-jcude-mvn-repo" is fragile, and only a snapshot. What is the release strategy for jcuda, and have proper maven coordinates?

update the help in DMLScript.java to output proper flags

for later, it would be good to see one example instruction with a cuda xt call.

The test passed: https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/423/

Closed by the commit https://github.com/apache/incubator-systemml/commit/f306b0b1ecd97fdb5755989f77728d1ca81358c5

authors: akchinSTC bertholdreinwald deroneriksson lresende mboehm7 nakul02 niketanpansare

repo: apache/incubator-systemml


text:
Update leveldb

Description
Update leveldb to v1.0.4 and move the code to third_party as a submodule for a clear view.
Fixes # (issue)
Type of change (select or add applied and delete the others)

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] API change with a documentation update
[ ] Additional test coverage
[X] Code cleanup or just sync with upstream third-party crates

How has this been tested?
Checklist

[X] Fork the repo and create your branch from master.
[ ] If you've added code that should be tested, add tests.
[ ] If you've changed APIs, update the documentation.
[X] Ensure the tests pass (see CI results).
[X] Make sure your code lints/format.

Hi @henrysun007, thanks for the PR.
The reason we choose to put in the common directory is that we have changed rusty_leveldb a lot, and we don't have any plan to upstream the changes. Therefore, we'd like to keep it in the Teaclave codebase as a part of Teaclave.
I saw you're using a submodule for the changes (which is not under the Apache organization). This may be not suitable for further maintenance. Can you PR the changes of the rusty_leveldb_sgx in the source code?
For the name of the common directory, if you have better suggestion. Please advise.

Hi mingshen,
Thanks for your comments. I am fine with the codebase and the name.
The reason for putting it as a submodule in this PR is to have a clear view on the modification to the original code. In one repo, it is not very clear what we have done to rusty-leveldb. Do you have any advice?

Yes, I still remember that Zhaofeng and I had a long discussion on this and we decided to put in the Teaclave repo. There are pros and cons. I think the most important reason is about the ownership. We want to maintain LevelDB for SGX as part of Teaclave. Definitely, this makes it not obvious to diff with the upstream, but we think it's worth to do that. The best why to upgrade LevelDB for SGX is to apply any patches from upstream to the LevelDB in Teaclave's repo. Also, to make it more clear, we can document that which version we are based on.

Thanks for your patience, @mssun . Let us own the code.

repo: apache/incubator-teaclave


text:
[TFLite] KeyError: 'conv2d/Kernel' on test_forward_mediapipe_hand_landmark()

After #5682, we're now seeing the error below on test_forward_mediapipe_hand_landmark. I think that it is not seen on upstream CI because of file caching. As I ran this test, master points at c7274fd3b0f693fc6214a450e13d5e99026337ae.
This is the error:
_____________________ test_forward_mediapipe_hand_landmark _____________________

    def test_forward_mediapipe_hand_landmark():
        """Test MediaPipe 2D hand landmark TF Lite model."""
        # MediaPipe 2D hand landmark TF
        tflite_model_file = download_testdata(
            "https://github.com/google/mediapipe/raw/master/mediapipe/models/hand_landmark.tflite",
            "hand_landmark.tflite")
        with open(tflite_model_file, "rb") as f:
            tflite_model_buf = f.read()
        data = np.random.uniform(size=(1, 256, 256, 3)).astype('float32')
        tflite_output = run_tflite_graph(tflite_model_buf, data)
>       tvm_output = run_tvm_graph(tflite_model_buf, data, 'input_1', num_output=2)

tests/python/frontend/tflite/test_forward.py:2451:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/python/frontend/tflite/test_forward.py:136: in run_tvm_graph
    dtype_dict=dtype_dict)
python/tvm/relay/frontend/tflite.py:2712: in from_tflite
    op_converter.convert_op_to_relay()
python/tvm/relay/frontend/tflite.py:185: in convert_op_to_relay
    ret = self.convert_map[op_code_str](op)
python/tvm/relay/frontend/tflite.py:2454: in convert_dequantize
    in_expr = self.get_expr(input_tensor.tensor_idx)
python/tvm/relay/frontend/tflite.py:2563: in get_expr
    return self.exp_tab.get_expr(get_tensor_name(self.subgraph, input_tensor_idx))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tvm.relay.frontend.common.ExprTable object at 0x7f244ac74f28>
name = 'conv2d/Kernel'

    def get_expr(self, name):
>       return self.exprs[name]
E       KeyError: 'conv2d/Kernel'

python/tvm/relay/frontend/common.py:294: KeyError
----------------------------- Captured stdout call -----------------------------

cc @FrozenGene

This model has been changed rapidly. Previous hand_landmark.tflite model is just FP32 model simply. Like this:

However, current master hand_landmark.tflite's weight is been quantized into FP16. Then it insert dequantize to FP32 execution, which is benefit for TFLite GPU execution. like this:

we doesn't support this kind of quantized model yet. I suggest we current lock the hand_landmark.tflite into this commit: https://github.com/google/mediapipe/blob/c27a7c1e1064e66427ad0656f776155ab456f90b/mediapipe/models/hand_landmark.tflite
@tqchen @leandron Could you accept this suggestion?

I think this is a good workaround.
I'd suggest to use a tag on that repository (rather than a commit hash), to make the URL a bit more meaningful: https://github.com/google/mediapipe/blob/v0.7.4/mediapipe/models/hand_landmark.tflite

I think this is a good workaround.
I'd suggest to use a tag on that repository (rather than a commit hash), to make the URL a bit more meaningful: https://github.com/google/mediapipe/blob/v0.7.4/mediapipe/models/hand_landmark.tflite

I think it is nice. @leandron  Could you help to make a pull request to lock it in the tag you mention?

I think it is nice. @leandron Could you help to make a pull request to lock it in the tag you mention?

Sure, will do.

Do we also need another issue to indicate that we don't support this kind of quantized networks and tag it as help needed ?

Do we also need another issue to indicate that we don't support this kind of quantized networks and tag it as help needed ?

I agree.

@u99127 Open the issue: https://github.com/apache/incubator-tvm/issues/5823

authors: FrozenGene leandron tqchen u99127

repo: apache/incubator-tvm


text:
relay.build not support size_var?

here is the test code
** 80 def test_dropout():
81     for dtype in ["float16", "float32"]:
82         #n, t, d = 1024, 1024, 1024
83         n, t, d = te.size_var("n"), 1024, 1024
84         input_ty = relay.TensorType((n, t, d), dtype)
85         x = relay.var("x", input_ty)
86         y = relay.nn.dropout(x, rate=0.75)
87         mod = tvm.IRModule.from_expr(relay.Function([x], y))
88         compiled_lib = relay.build(mod, tvm.target.create("cuda"))
89         assert "rate=" in y.astext()
90         yy = run_infer_type(y)
91         assert yy.checked_type == input_ty
**
the graph_plan_memory.cc:GetMemorySize check(dim) fail

Yes, you have to use Realy VM for dynamic input.

"Realy VM for dynamic input" can export a lib for "C++" runtime just as "relay.build" ?

authors: zhiics zhxfl


text:
[Release] Create a 1.11.0 branch to release

Description
No response
Use case
No response
Are you willing to submit PR?

[X] Yes, I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

see https://github.com/apache/inlong/tree/branch-1.11

see https://github.com/apache/inlong/tree/branch-1.11

repo: apache/inlong


text:
[INLONG-4854][Agent] Report metrics at inongGroupId and inlongStreamId

Prepare a Pull Request
(Change the title refer to the following example)

[INLONG-4854][Agent] Report metrics at inongGroupId and inlongStreamId

(The following XYZ should be replaced by the actual GitHub Issue number)

Fixes #4854

Motivation
Explain here the context, and why you're making that change. What is the problem you're trying to solve?
Modifications
Describe the modifications you've done.
Verifying this change
(Please pick either of the following options)

[ ] This change is a trivial rework/code cleanup without any test coverage.

[ ] This change is already covered by existing tests, such as:
(please describe tests)

[ ] This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation

@luchunliang @EMsnap PTAL

authors: dockerzhang pocozh


text:
[INLONG-7249][Sort] JDBC accurate dirty data archive and metric calculation

Prepare a Pull Request

Fixes #7249

Motivation
JDBC needs to archive dirty data. In this pr, I tried to explore to possibility of accurate dirty data archive for jdbc by modifying the executors. The modifications are well-tested and stable in most cases, and in other cases it will give a warning "jdbc enhance failed for class:{}", and won't affect normal code.
Modification
Support dirty data accurate archive by using reflection to replace some kinds of executors, and replace the tablesimplestatementexecutor with one that adds metrics.
To-do and to-improve
1.There are 8 total implementations of flink-cdc-executors.
The currently supported sink executor types are:
TableBufferedStatementExecutor
TableBufferReducedStatementExecutor
TableSimpleStatementExecutor
unsupported executor types: (will not archive dirty data, but will not break code)
KeyedBatchStatementExecutor
NoOPStatementExecutor
TableInsertOrUpdateStatementExecutor
no need to support:
InsertOrUpdateJdbcExecutor (TableInsertOrUpdateStatementExecutor is enough)
SimpleBatchStatementExecutor (TableSimpleStatementExecutor is enough)
2.This pr uses Java reflection depending on flink-cdc-connectors. This design can be improved by introducing the executors into Inlong-jdbc-connector, and modifying the builder to directly change the executor class, instead of doing all this reflection. Also, throughput can increase by refactoring too.
Even Though this pr still has imperfections, I think it can be merged for the following reasons:

This pr can support all the common cases of JDBC dirty data archive, and can support more in the future.
For cases where it does not support, it will not cause exceptions, but simply give a warning and go on.
This will not slow down the existing code and normal data integration.

@gong  @yunqingmoswu  Thanks for the review. I've addressed your comments.

@Yizhou-Yang pls resolve conflict

authors: Yizhou-Yang gong


text:
[Bug] v1.0.0 采用java原生接口insertAlignRecords批量插入时，regions可能为空List

Search before asking

[X] I searched in the issues and found nothing similar.

Version
OS: Centos 7.6
IoTDB: v1.0.0
Describe the bug and provide the minimal reproduce step
采用java原生接口insertAlignRecords批量插入时（每次写入5000个数据点），regions可能为空List。
datanode日志内容如下：

抛出异常的代码

应用程序控制台报错：

What did you expect to see?
异常出现的原因？
What did you see instead?
异常出现的原因？
Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

Hi @whsqjss, thanks for your reporting.
Could you please supply more details regarding your tests ?

how many confignodes/datanodes did you deployed in your tests.
Before writing data using insertAlignedRecords, did you create the schema manually ?
can you execute the cmd show regions by sql client ?

Thanks,
Jerry

I deployed 1 confignode and 3 datanodes

I created 1 schema for writing data

show regions

authors: whsqjss xingtanzjr

repo: apache/iotdb


text:
Add a boolean property to choose if we want to propagate the state of…

… the current element to the children of the tree
Description
Add a new jmeter property to choose if you want to propagate the state of the current element to the children of the tree OR NOT (gray for disable for all chidren in the tree).
Motivation and Context
The pull request https://github.com/apache/jmeter/pull/558  (Use gray icons for disabled elements in the tree) could be usefull in many cases.
But a script is not a simple tree from Thread Group but a graph when you use a Module Controler.

In the Thread Group 1, somes samplers are enabled (001 and 015) and somes are disabled (006, 008 and 014), you could see with the gray color or not.
The Thread Group 2 with module controler use a module controler that go to the Thread Group 1 > Recording Controller.
When i disable the "Thread Group 1" all the children samplers look disabled (gray), i can't see witch samplers are "really" enabled or disabled.
The Module Controler from Thead Group2 with module controler will call samplers children of Recording Controller but now i can't see witch samplers will be really call because all samplers look disabled.

I add a jmeter.property to choose if you want to propagate the state of the current element to the children of the tree or you do NOT want to propagate the state to children.
How Has This Been Tested?
Add log to see if new jmeter property jmeter.gui.propagates_parent_state_to_children=true if this property is correcly read from jmeter.properties file
Test with defaut value jmeter property jmeter.gui.propagates_parent_state_to_children=true
Test with jmeter.gui.propagates_parent_state_to_children=false
Create a new script with a tree and enabled ou disabled a element in the tree and see if children are also change color gray or not
Screenshots:
jmeter.gui.propagates_parent_state_to_children=false

jmeter.gui.propagates_parent_state_to_children=true (defaut value)

Types of changes

New feature (non-breaking change which adds functionality)

Checklist:

[x ] My code follows the code style of this project.
[x] I have updated the documentation : changes.xml and properties_reference.xml

Hello @vdaburon ,
Thanks for your PR.
I think there is a need to use your property also here:
https://github.com/apache/jmeter/pull/558/files#diff-603dd62ec85022de4001a09ab4de3d0fR56
Otherwise enabling an element which parent is disabled does nothing.
Regards

I think we should not introduce a gobal propery like that.
State propagation should be element-aware.

Hi,

I think we should not introduce a gobal propery like that.
State propagation should be element-aware.
The global property is a simple solution to select if you want the propagation of the parent state.

The behavior of the feature is as in version 5.1 or earlier and i like the old behavoir.
I think state propagation could be really complexe when you use multi-thread groups and module controlers and test fragments.
I could the time consuming for a complexe JMeter script to change the state.

Then I'm inclined to revert #558.

An alternative approach would be to use different styling for the items disabled by the user vs the ones that were disabled automatically.

Hello @vlsi ,
unless you have time to implement your proposal, I would suggest to revert #558 as you suggested.
Current behaviour is indeed problematic whenever you use some factorisation.
Regards

Frankly speaking, I don't know how to display "disabled item" vs "enabled but located under a disabled item", so let's revert 558 for now.

Reverted the problematic part of #558. Fixed by commit:
https://github.com/apache/jmeter/commit/4de138c40f8102bbfdf89e614a8b52581f56431e

authors: pmouawad vdaburon vlsi

repo: apache/jmeter


text:
[MINOR] Removing unused variables from StreamThreadTest

While working on https://github.com/apache/kafka/pull/13283 I found a few variables which aren't being used anymore but still exist. I believe this is happening due to some recent refactoring that was done. This PR removes those unused variables.

Thanks @lucasbru , oh I assumed it's due to recent refactoring. I should have checked before. Apologies

repo: apache/kafka


text:
KAFKA-15883: Implement RemoteCopyLagBytes

Summary
This pull request implements the first in the list of metrics in KIP-963: Additional metrics in Tiered Storage.
Since each partition of a topic will be serviced by its own RLMTask we need an aggregator object for a topic. The aggregator object in this pull request is BrokerTopicAggregatedMetric. Since the RemoteCopyLagBytes is a gauge I have introduced a new GaugeWrapper. The GaugeWrapper is used by the metrics collection system to interact with the BrokerTopicAggregatedMetric. The RemoteLogManager interacts with the BrokerTopicAggregatedMetric directly.

Heya @satishd and @showuon! I hope I have addressed your comments with the latest commits 😊

Heya @showuon! As always, your comments are spot on! I hope I have addressed them in the latest commits 😊

Will review the patch by today.

@clolov , could you address @kamalcph 's comments? I hope this KIP can be included into v3.7.0.

Heya, sorry, I will try to provide an updated version tomorrow

@clolov , there are checkstyle error, please correct them:
https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14832/6/pipeline
Thanks.

@clolov , I took the liberty to help fix the checksytle error. Let's see if the CI build passed. Thanks.

It failed 61 tests in the CI build. Re-triggering it:
https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-14832/8/

Failed tests are unrelated:
    Build / JDK 21 and Scala 2.13 / org.apache.kafka.connect.integration.OffsetsApiIntegrationTest.testAlterSinkConnectorOffsets
    Build / JDK 21 and Scala 2.13 / kafka.api.TransactionsBounceTest.testWithGroupMetadata()
    Build / JDK 21 and Scala 2.13 / org.apache.kafka.controller.QuorumControllerTest.testFenceMultipleBrokers()
    Build / JDK 8 and Scala 2.12 / kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures()
    Build / JDK 8 and Scala 2.12 / kafka.api.TransactionsBounceTest.testWithGroupMetadata()
    Build / JDK 8 and Scala 2.12 / kafka.server.DynamicBrokerReconfigurationTest.testTrustStoreAlter(String).quorum=kraft
    Build / JDK 8 and Scala 2.12 / org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManagerTest.testNewPartitionUpdates()
    Build / JDK 11 and Scala 2.13 / kafka.server.DelegationTokenRequestsTest.testDelegationTokenRequests(String).quorum=kraft
    Build / JDK 11 and Scala 2.13 / org.apache.kafka.tools.MetadataQuorumCommandTest.testDescribeQuorumStatusSuccessful [2] Type=Raft-Isolated, MetadataVersion=3.7-IV3, Security=PLAINTEXT
    Build / JDK 17 and Scala 2.13 / org.apache.kafka.common.security.authenticator.SaslAuthenticatorFailureNoDelayTest.testInvalidPasswordSaslScram()
    Build / JDK 17 and Scala 2.13 / kafka.api.PlaintextAdminIntegrationTest.testElectPreferredLeaders(String).quorum=kraft
    Build / JDK 17 and Scala 2.13 / org.apache.kafka.streams.integration.EosV2UpgradeIntegrationTest.shouldUpgradeFromEosAlphaToEosV2[true]
    Build / JDK 17 and Scala 2.13 / org.apache.kafka.tools.MetadataQuorumCommandTest.testDescribeQuorumReplicationSuccessful [2] Type=Raft-Isolated, MetadataVersion=3.7-IV3, Security=PLAINTEXT

authors: clolov kamalcph showuon


text:
KAFKA-15206: Fix the flaky RemoteIndexCacheTest.testClose test

It is possible that the Shutdownable thread never ran in the CI so the test is failing
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@divijvaidya @showuon
Would you please review this minor change?

Thanks for the change, @kamalcph! Just for my knowledge, why is there a possibility that the Shutdownable thread never ran in the CI for this test?

Thanks for the change, @kamalcph! Just for my knowledge, why is there a possibility that the Shutdownable thread never ran in the CI for this test?

Due to resource constraint, ShutdownableThread#run might be called later than the ShutdownableThread#close method.

Nice fix @kamalcph! A few questions:

Why did we create a new test file. Can't we add the tests to existing ShutdownableThreadTest https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/utils/ShutdownableThreadTest.scala ?
If the cleaner thread wasn't even started, are our really testing correct behaviour? For example, when we validate that cleaner shouldn't have deleted something, we would happily think that cleaner ran but didn't delete stuff whereas in reality, it may never have run at all. Should we instead add a wait loop in our setup for cache in the test to ensure that cleaner starts running?

Thanks for the review!

If the cleaner thread wasn't even started, are our really testing correct behaviour? For example, when we validate that cleaner shouldn't have deleted something, we would happily think that cleaner ran but didn't delete stuff whereas in reality, it may never have run at all. Should we instead add a wait loop in our setup for cache in the test to ensure that cleaner starts running?

The RemoteIndexCache (RIC) and RemoteIndexCacheTest (RICT) are in different packages, so we cannot introduce a new package-private method in the RIC to overwrite it in RICT to ensure that the cleaner-thread started. Also, the ShutdownableThread is widely used by various module, so not changing the behavior of the thread.

The ShutdownableThread source code is in Java, so thought to write the new unit tests in java.
In this test, we don't expect any entries to expire. The test fails 2% of time which mean the remaining 98% of time, the cleaner-thread ran. In the assertion, we ensure that the cleaner-thread is closing correctly for both the cases.

If the root cause of the flaky test is the ShutdownableThreadTest is not running, could we add TestUtils.waitUntilTrue(() => cache.cleanerThread.isStarted) before we invoke cache.close()?
I saw there's a method to check thread isRunning, but it's actually checking the thread is not closing. Maybe we can add a method to check the ShutdownableThread#isStarted value?

Failed tests are unrelated.

authors: divijvaidya kamalcph nikramakrishnan showuon


text:
KAFKA-16477: Detect thread leaked client-metrics-reaper in tests

related to KAFKA-16477,
After profiling the kafka tests, tons of client-metrics-reaper thread not cleanup after BrokerServer shutdown.
The thread client-metrics-reaper comes from ClientMetricsManager#expirationTimer, and BrokerServer#shudown doesn't close ClientMetricsManager which let the thread still runs in background.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@brandboat nice finding! Should we add the thread prefix to https://github.com/apache/kafka/blob/a2ee0855ee5e73f3a74555d52294bb4acfd28945/core/src/test/scala/unit/kafka/utils/TestUtils.scala#L2487 to avoid similar issue in the future?

@brandboat nice finding! Should we add the thread prefix to https://github.com/apache/kafka/blob/a2ee0855ee5e73f3a74555d52294bb4acfd28945/core/src/test/scala/unit/kafka/utils/TestUtils.scala#L2487
to avoid similar issue in the future?

Oh wow ! I didn't notice that we have this check before. Sure ! Already added the thread name in this check in commit https://github.com/apache/kafka/pull/15668/commits/8daf268f2c99b7a88dde150967c22301c88cc1d6. Many thanks !


text:
KAFKA-17978: Fix invalid topology on Task assignment

When we introduced "startup tasks" in #16922, we initialized them with
no input partitions, because they aren't known until assignment.
However, when we update them during assignment, it's possible that we
update the topology with the incorrect source topics for some internal
topics, due to a difference in the way internal topics are handled for
StandbyTasks.
To resolve this, we now initialize startup tasks with the correct input
partitions, by calculating them from the Topology.
When we assign our startup tasks, we now conditionally update their
input partitions only if they've actually changed, just as we do for
regular StandbyTasks.
With this, the E2E tests now pass, as expected.

Did we trigger a system test run to verify that the PR fixes the issue?

LGTM

Did we trigger a system test run to verify that the PR fixes the issue?

streams_upgrade_test.py pass on my local now. I will run e2e for all streams later

Did we trigger a system test run to verify that the PR fixes the issue?

I understood that @nicktelford verified that. He writes in the PR description:

With this, the E2E tests now pass, as expected.

Thanks both!
Running only the broker test is fine -- we usually trigger a semaphore run (Confluent setup) and post a link to the run to verify. As there was no link, I did ask :)

authors: cadonna chia7712 mjsax nicktelford


text:
KAFKA-17990: Flaky test improvements

Several of the ShareConsumerTest integration tests are a bit flaky. This PR tightens up the logic with the aim of eliminating the flakes. Annoyingly the tests seem rock solid locally so this might take some experimentation.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

We recently changed the default starting position for consumption from earliest to latest to match the KIP, which necessitated alter configs for the tests to "earliest". The point at which this takes effect is non-deterministic and I believe that's responsible for the increased flakiness.


text:
MINOR: Remove waitUntilTrue from TestUtils.scala.

Removed the waitUntilTrue function from TestUtils.scala and replaced it with waitForCondition.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

LGTM, except some improvement suggestions.

@dengziming
Thanks for the heads up, I've fixed the problem you mentioned.


text:
MINOR: Code cleanup of 'clients' module

Cleanup involves

Removing redundant / unnecessary code
Fixing typos
Simplifying code when possible
Refactoring to use lambda expressions in some places

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

retest this please

retest this please

@hachikuji @ijuma This PR touches several files but hopefully is trivial enough to merge. Please let me know if you see issues with it. Thanks!

retest this please

@ryannedolan @ijuma Thanks for reviewing the PR. I'll try to rebase and address the review comments soon.

@ijuma Thanks for additional improvements. They look good to me. There is just one unit test that's failing due to a change you made. Please see my note inline.

@vahidhashemian I reverted the TopicPartition constructor change since it's a widely used class and we should do it in a separate commit to discuss the potential compatibility implications. I left the TopicPartitionReplica change since its usage is much more limited and it's instantiated internally.

Test failure is unrelated, merging to trunk.

authors: ijuma vahidhashemian


text:
KAFKA-8331: stream static membership system test

As title suggested, we boost 3 stream instances stream job with one minute session timeout, and once the group is stable, doing couple of rolling bounces for the entire cluster. Every rejoin based on restart should have no generation bump on the client side.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@guozhangwang @mjsax @hachikuji Call for review!

Retest this please

Retest this please

Retest this please

Verified all affected system tests are passing:
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2692/
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2693/
https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2684/

LGTM. Ping @mjsax @vvcephei @ableegoldman for another review

@mjsax @ableegoldman @bbejeck thoughts?

Merged #6877 into trunk

@bbejeck Thank you Bill!

authors: abbccdda bbejeck guozhangwang


text:
Some parameters will be overwritten which was configured in consumer.config where running "ConsumerPerformance.scala". Linked  https://issues.apache.org/jira/browse/KAFKA-10043

Some parameters will be overwritten which was configured in consumer.config where running "ConsumerPerformance.scala"
linked see https://issues.apache.org/jira/browse/KAFKA-10043
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[x] Verify design and implementation
[x] Verify test coverage and CI build status
[x] Verify documentation (including upgrade notes)

Could you update the code (according to the reviews) and then push the changes to be a new commit. Also, please rebase your code (by trunk).
@chia7712  Can you take a look at this?

Could you add test for this issue?

Could you add test for this issue?

Do you mean to add unit tests？But i did not found unit tests for this class kafka.tools.ConsumerPerformance.scala

But i did not found unit tests for this class kafka.tools.ConsumerPerformance.scala

It would be great if you could be the first person to write the tests for  ConsumerPerformance :)

But i did not found unit tests for this class kafka.tools.ConsumerPerformance.scala

It would be great if you could be the first person to write the tests for ConsumerPerformance :)

I had committed the unit test in ConsumerPerformanceTest, What should I do to relate to that?

But i did not found unit tests for this class kafka.tools.ConsumerPerformance.scala

It would be great if you could be the first person to write the tests for ConsumerPerformance :)

I had committed the unit test in ConsumerPerformanceTest, What should I do to relate to that?

I found that it was automatically linked，but all checks had failed :( .
what should I do next :)

what should I do next :)

please take a look at the report and fix the error. https://github.com/apache/kafka/pull/9790/checks?check_run_id=1616222522

what should I do next :)

please take a look at the report and fix the error. https://github.com/apache/kafka/pull/9790/checks?check_run_id=1616222522

I fixed the error, but there were many errors in the second build. It seemed that the errors had nothing to do with what I changed. Please help me to have a look :)

@itantiger Thanks for your patch. Could you revise the title?

@itantiger Thanks for your patch. Could you revise the title?

Also thank you very much for your guidance :) .

@itantiger Thanks for your patch. Could you revise the title?

Finished.
When will this patch merge to trunk?

@itantiger The priority of loading props is a bit chaos in this case. It seems to me the property of loading properties is shown below.

command-lind
config file (consumer.config)
default value

The current priority is shown below.

command-lind
default value
config file (consumer.config)

The priority from this PR is shown below.

config file (consumer.config)
command-lind
default value

Of course, current priority has issue as properties from config file always get overwritten by default value. We need to fix it. However, I'd like to check the priority again. WDYT? Should config file has higher priority than command-lind?

command-lind
config file (consumer.config)
default value
I think this priority is more better, and this PR is to do this.

I think this priority is more better, and this PR is to do this.

We load the props from config file first. https://github.com/apache/kafka/blob/afa5423356d3d2a2135a51200573b45d097f6d60/core/src/main/scala/kafka/tools/ConsumerPerformance.scala#L275 and then call props.putIfAbsent to set args from command-line only if the prop is nonexistent in config file. Hence, the priority of config file is higher than command-line, right?

and then call props.putIfAbsent to set args from command-line only if the prop is nonexistent in config file. Hence, the priority of config file is higher than command-line, right?
You're right. I've ignored it. It should be modified here. :)

and then call props.putIfAbsent to set args from command-line only if the prop is nonexistent in config file. Hence, the priority of config file is higher than command-line, right?
You're right. I've ignored it. It should be modified here. :)

Current priority, some config params would be overwritten by default value, Do you think it needs to be revised?

Do you think it needs to be revised?

I guess you are talking about "title". yes, the title need to be revised.

I guess you are talking about "title". yes, the title need to be revised.
In addition, I think the trunk also needs to be modified.
Some config params would be overwritten by default value, WDYT?

I guess you are talking about "title". yes, the title need to be revised.

In addition, I think the trunk also needs to be modified.
Some config params would be overwritten by default value, WDYT?

I think the trunk also needs to be modified.

trunk?  pardon me that I can't catch your point.

Some config params would be overwritten by default value, WDYT?

yep, the default value CAN overwrite the configs from either command-line or config file.

trunk? pardon me that I can't catch your point.
That's my problem. hah
yep, the default value CAN overwrite the configs from either command-line or config file.
Let me optimize it again and commit. Do you think it's necessary?

trunk? pardon me that I can't catch your point.

That's my problem. hah

yep, the default value CAN overwrite the configs from either command-line or config file.

Let me optimize it again and commit. Do you think it's necessary?

Let me optimize it again and commit. Do you think it's necessary?

I'd like to address the following priority rule in this PR :)

command-lind
config file (consumer.config)
default value

I'd like to address the following priority rule in this PR :)

command-lind
config file (consumer.config)
default value

Of course, I also think this is more reasonable


text:
KYLIN-3392 Fix BufferUnderflowException read DecimalAggregator when it is null

Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
Types of changes
What types of changes does your code introduce to Kylin?
Put an x in the boxes that apply

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices apply)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have create an issue on Kylin's jira, and have described the bug/feature there in detail
[x] Commit messages in my PR start with the related jira ID, like "KYLIN-0000 Make Kylin project open-source"
[x] Compiling and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] If this change need a document change, I will prepare another pr against the document branch
[ ] Any dependent changes have been merged

Further comments
If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc...

Codecov Report

Merging #1524 (fe04ca8) into master (fe04ca8) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #1524   +/-   ##
=========================================
  Coverage     25.44%   25.44%
  Complexity     6765     6765
=========================================
  Files          1507     1507
  Lines         93784    93784
  Branches      13132    13132
=========================================
  Hits          23867    23867
  Misses        67533    67533
  Partials       2384     2384

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fe04ca8...16626be. Read the comment docs.

It's better to refine the generated testing data for IT to introduce nulls for more columns

authors: codecov-io hit-lacus kyotoYaho

repo: apache/kylin


text:
KYLIN-4273 Make cube planner works for real-time streaming job

Make cube planner works for real-time streaming job:
1, when no ready segment is ready for streaming cube, only 1 segment is allow to build, ensure that the 1st phrase cube planner will take effective
2, when there are any optimization job is running, stop new segment building for streaming cube

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Pull Request Test Coverage Report for Build 5302

7 of 48 (14.58%)  changed or added relevant lines in 2 files are covered.
6 unchanged lines in 4 files lost coverage.
Overall coverage decreased (-0.01%) to 27.928%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/coordinate/BuildJobSubmitter.java
7
25
28.0%

stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java
0
23
0.0%

Files with Coverage Reduction
New Missed Lines
%

server-base/src/main/java/org/apache/kylin/rest/util/QueryRequestLimits.java
1
47.62%

stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/Coordinator.java
1
26.37%

core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java
2
68.46%

core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java
2
80.23%

Totals

Change from base Build 5300:
-0.01%

Covered Lines:
23874

Relevant Lines:
85483

💛  - Coveralls

authors: allenma asf-ci coveralls


text:
[KYUUBI #4083] Flink returns duplicate results when executing query statement

Why are the changes needed?

Fix #4083
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

set kyuubi.session.engine.flink.max.rows to appropriate value (eg: 500).
whatever earliest kafkaTable or latest kafkaTable, when select * from yourtable,  you can get line number equal to kyuubi.session.engine.flink.max.rows.
https://kyuubi.readthedocs.io/en/v1.7.0/deployment/settings.html#session
Suggest kyuubi.session.engine.flink.max.rows default value change to 500.
And explanation: too large may trigger OOM

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

@felixzh2020 thanks for your contribution, do you have a chance to add unit tests to verify your fix?

Codecov Report

Merging #4701 (bd5a999) into master (f0615a9) will decrease coverage by 0.06%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #4701      +/-   ##
============================================
- Coverage     57.99%   57.94%   -0.06%
  Complexity       13       13
============================================
  Files           580      580
  Lines         32218    32220       +2
  Branches       4304     4304
============================================
- Hits          18684    18669      -15
- Misses        11749    11755       +6
- Partials       1785     1796      +11

see 13 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

@pan3793 I have added UT.

@pan3793 My UT is pass. Maybe rerun check?

@pan3793 Run  ./build/dist --tgz --spark-provided --flink-provided --hive-provided
Now scalastyle check is ok.

As far as I am concerned, master branch have change implementation plan, maybe this pr is not need. Closed.

repo: apache/kyuubi


text:
Fix tests with pytest 3.4

Fix tests with pytest 3.4
Description
The 3.4 pytest release introduced a breaking change: pytest no longer changes log levels: the default log level is now Python's default: WARNING. This commit ensures the test no longer depends on external log levels.
pytest 3.4 also adds a .pytest_cache folder: ignore it.
Status

done, ready for review

Checklist (tick everything that applies)

[x] Code linting (required, can be done after the PR checks)
[ ] Documentation
[x] Tests

Codecov Report

Merging #1170 into trunk will decrease coverage by <.01%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##            trunk    #1170      +/-   ##
==========================================
- Coverage   86.11%   86.11%   -0.01%
==========================================
  Files         348      348
  Lines       66890    66891       +1
  Branches     5938     5939       +1
==========================================
  Hits        57600    57600
  Misses       6861     6861
- Partials     2429     2430       +1

Impacted Files
Coverage Δ

libcloud/test/test_init.py
61.53% <0%> (-2.47%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e7429e9...fb622ac. Read the comment docs.

Note that the coverage is going down because we incorrectly compute coverage on only one Python 3 run instead of running codecov for every tox interpreter. But that should be fixed in another PR.

@pquentin pytest version should also be pinned in this PR

Thanks @tonybaloney, fixed. I'll update the pins of the other test requirements in another pull request if you don't mind.

:+1: Merging

authors: codecov-io pquentin tonybaloney

repo: apache/libcloud


text:
Add JUnit 5 extension for per test-class log directories

This PR adds a @TempLoggingDir extension, which is a reduced version of the standard @TempDir annotation.
It can be used on static fields and:

creates a directory specific for the test-class (right now in target/logs/<fully_qualified_class_name>,
injects the field with the value of the directory,
creates an entry in the TestProperties with key "logging.path",
deletes the directory if the test succeeds.

The TestProperties entry is available as both a Log4j2 system property and using the ${test:logging.path} lookup.

This testing infrastructure improvement is part of task #1776.


text:
Most links in the documentation are broken

This is where we are told the documentation is:
https://logging.apache.org/log4net/
All the top level links work, but as soon as you click through to a second level, the link is broken e.g.
https://logging.apache.org/log4net/release/sdk/index.html

@marcel-landman thanks for reporting.
I discovered the same yesterday 😉.
A fix (https://github.com/apache/logging-log4net/pull/121/commits/61e28326da40646161fe5167905f6c5f55e468a2) is under way and will be deployed next week (#121).

authors: FreeAndNil marcel-landman


text:
Remove direct dependency of NodeHash to FST

Description
Follow-up of https://github.com/apache/lucene/pull/12646. NodeHash still depends on both FSTCompiler and FST. With the current method signature, one can create the NodeHash with a FST and call addNode with different FSTCompiler. This PR simplifies the signature and allow a consistent behavior.

Actually, I'm nervous about my git merging skills!  Let's try to review/push the RAM limited FST building PR first?

That's ok for me too! I'll rebase after the other PR is merged

As the other PR has been merged, I have rebased and resolved the conflict

Thanks @dungba88 -- looks great, I'll merge soon!

Thanks @dungba88 -- I'll wait to backport this until after backporting https://github.com/apache/lucene/pull/12633

repo: apache/lucene