text:
Export symbols for checking cuda/cudnn versions

Fixes #15578
Two functions (CudaLibChecks and CuDNNLibChecks) were added in #15551 to fix #15548. These two functions need to be exported in our pip package such that downstream projects consuming mxnet library will not encounter undefined symbol error as shown below:
OSError: /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/horovod/mxnet/mpi_lib.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN5mxnet7Context13CudaLibChecksEv

@apeforest @szha @ptrendx @DickJC123 Please help review. Thanks.

@mxnet-label-bot add [pr-awaiting-review]

LGTM.
@yuxihu Thanks for stepping in quickly to provide this fix.  I would appreciate any additional insight you have that might help me from introducing this sort of problem in the future.  For example, I see that the wildcard pattern you introduced to ./make/config/libmxnet.{sym,ver} matches the missing entrypoints:
$ nm libmxnet.so | grep LibCheck | grep " T "
0000000003c889d0 T _ZN5mxnet7Context13CudaLibChecksEv
0000000003c89730 T _ZN5mxnet7Context14CuDNNLibChecksEv

However, I can also perform:
$ nm libmxnet.so | grep ResourceManager | grep Get | grep " T "
0000000003c77c00 T _ZN5mxnet15ResourceManager3GetEv

Why is no wildcard pattern similarly needed to cover ResourceManager::Get(), as created in ./include/mxnet/resource.h and ./src/resource.cc?

Why didn't the API follow existing naming convention?

Why didn't the API follow existing naming convention?

@szha What do you suggest to name these two functions CudaLibChecks and CuDNNLibChecks? Adding MX or similar as the prefix? If they can be named to match existing symbol wildcards, we do not need to export them explicitly.

+1 to renaming the API to follow the naming scheme.
While we're at it, can we please introduce a check to CI to make sure that all functions match the defined schemes?

Why is no wildcard pattern similarly needed to cover ResourceManager::Get(), as created in ./include/mxnet/resource.h and ./src/resource.cc?

@DickJC123 Are you using the mxnet pip package? The symbol is not exported in the pip package.
(mxnet_p27) ubuntu@ip-172-31-21-35:~$ pip install mxnet-cu100 --pre
Collecting mxnet-cu100
  Using cached https://files.pythonhosted.org/packages/b7/c4/1e0c4d21ed6dca0890a9db3e23f78c9d8df1651330547c7778aa1a974ea6/mxnet_cu100-1.5.0b20190717-py2.py3-none-manylinux1_x86_64.whl
Requirement already satisfied: graphviz<0.9.0,>=0.8.1 in ./anaconda3/envs/mxnet_p27/lib/python2.7/site-packages (from mxnet-cu100) (0.8.4)
Collecting numpy<2.0.0,>1.16.0 (from mxnet-cu100)
  Downloading https://files.pythonhosted.org/packages/1f/c7/198496417c9c2f6226616cff7dedf2115a4f4d0276613bab842ec8ac1e23/numpy-1.16.4-cp27-cp27mu-manylinux1_x86_64.whl (17.0MB)
    100% |████████████████████████████████| 17.0MB 2.6MB/s
Requirement already satisfied: requests<3,>=2.20.0 in ./anaconda3/envs/mxnet_p27/lib/python2.7/site-packages (from mxnet-cu100) (2.20.0)
Requirement already satisfied: idna<2.8,>=2.5 in ./anaconda3/envs/mxnet_p27/lib/python2.7/site-packages (from requests<3,>=2.20.0->mxnet-cu100) (2.6)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./anaconda3/envs/mxnet_p27/lib/python2.7/site-packages (from requests<3,>=2.20.0->mxnet-cu100) (3.0.4)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in ./anaconda3/envs/mxnet_p27/lib/python2.7/site-packages (from requests<3,>=2.20.0->mxnet-cu100) (1.23)
Requirement already satisfied: certifi>=2017.4.17 in ./anaconda3/envs/mxnet_p27/lib/python2.7/site-packages (from requests<3,>=2.20.0->mxnet-cu100) (2019.3.9)
Installing collected packages: numpy, mxnet-cu100
  Found existing installation: numpy 1.14.5
    Uninstalling numpy-1.14.5:
      Successfully uninstalled numpy-1.14.5
Successfully installed mxnet-cu100-1.5.0b20190717 numpy-1.16.4
You are using pip version 10.0.1, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

(mxnet_p27) ubuntu@ip-172-31-21-35:~$ nm ./anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet/libmxnet.so | grep ResourceManager | grep Get | grep " T "
nm: ./anaconda3/envs/mxnet_p27/lib/python2.7/site-packages/mxnet/libmxnet.so: no symbols

Actually no, I was examining the symbols of a locally built libmxnet.so.
I'm still trying to understand the recommended approach for the functionality I've introduced.  Any translation unit that includes include/mxnet/base.h and calls Context::Create(dev_type, dev_id) will need external linkage to the Context::CudaLibChecks() implementation from ./src/base.cc.  On the other hand, CudaLibChecks() is private and Create() is its only user.  What is the recommendation then for preceding the name with MX, implementing in c_api.h, etc.?

@szha @marcoabreu I just tried to add MX prefix to those two functions.
root@f905c57ff7f3:/build/mxnet-build# git diff
diff --git a/include/mxnet/base.h b/include/mxnet/base.h
index c1e2da7..e476734 100644
--- a/include/mxnet/base.h
+++ b/include/mxnet/base.h

-    static void CudaLibChecks();
+    static void MXCudaLibChecks();

-    static void CuDNNLibChecks();
+    static void MXCuDNNLibChecks();

-    CudaLibChecks();
+    MXCudaLibChecks();

-    CuDNNLibChecks();
+    MXCuDNNLibChecks();

diff --git a/src/base.cc b/src/base.cc
index 96d66ad..fc79c6a 100644
--- a/src/base.cc
+++ b/src/base.cc

-void Context::CudaLibChecks() {
+void Context::MXCudaLibChecks() {

-void Context::CuDNNLibChecks() {
+void Context::MXCuDNNLibChecks() {

However, the two functions are not exported with the existing wildcard pattern. Am I missing anything here? @szha
root@f905c57ff7f3:/build# nm -D mxnet-build/lib/libmxnet.so | grep MX | grep Lib
000000000252a480 T MXLibInfoFeatures

I suspect the problem is that the name must match MX* and either the mxnet namespace of the new functions or the Context class scope prevents a match.
I can reproduce the error locally.  Is horovod going around the official mxnet api by calling Context::create()?
nm -C /usr/local/lib/python3.6/dist-packages/horovod/mxnet/mpi_lib.cpython-36m-x86_64-linux-gnu.so | grep -i context | grep -i mxnet
                 U mxnet::Context::CudaLibChecks()
000000000007e510 t mxnet::Context::Create(mxnet::Context::DeviceType, int)

I can reproduce the error locally. Is horovod going around the official mxnet api by calling Context::create()?

Yes, in Horovod we call Context::CPU() and Context::GPU() which calls Create() internally. If we have a new C API defined for the Context::Create, we have to change to use it in Horovod which will introduce backward compatibility issue. Adding MX to those two functions does not simply work for us. Exporting symbols is a straightforward working solution, although it might not be the best practice as those two functions are private ones.

libmxnet.so shouldn't expose symbols that are not part of the API. if it's needed as API, it needs to be added to C-API. I think the former is more appropriate in this context (correction is welcome). so the linked issue indicates problem in horovod build rather than mxnet.

It seems horovod is going around the official mxnet API by refering directly to NDArray and Context. MXNet provides opaque pointers NDArrayHandle and ContextHandle, but I'm not sure the interface is fully functional, since it appears there is no API call to set a ContexHandle, as in (say) MXContextCreate(...,ContextHandle *out). Anyway, @yuxihu , here's a proposal for getting horovod back on the official API, at least in terms of this problem. Would the following change work for you in terms of compatibility issues?
Horovod now:
// Returns pointer to newly created NDArray
// If dev_id equal to CPU_DEVICE_ID, construct Tensor on CPU
// Otherwise construct on GPU
NDArray* TensorUtil::New(int device, int dtype) {
  if (device == CPU_DEVICE_ID) {
    NDArray* my_array = new NDArray(TShape(), Context::CPU(0), false, dtype);
    return my_array;
  } else {
    NDArray* my_array =
        new NDArray(TShape(), Context::GPU(device), false, dtype);
    return my_array;
  }
}

Horovod as potentially switched over to official MXNet API:
NDArray* TensorUtil::New(int device, int dtype) {
   NDArrayHandle my_array;
  if (device == CPU_DEVICE_ID)
    MXNDArrayCreateEx(nullptr, 0, cpu::kDevMask, 0, false, dtype, &my_array);
  else
    MXNDArrayCreateEx(nullptr, 0, gpu::kDevMask, device, false, dtype, &my_array);
  return static_cast<NDArray*>(my_array);
}

The compiles for me but has some runtime issues that may be unrelated to the change.  Maybe you could test?

Regarding compatibility, old horovod builds that still reference Context::Create() would not link against newer libmxnet.so (where Context::Create() needs the unexported CudaLibChecks()).

The above change was in the file tensor_util.cc.  Also needed is the following chane in mpi_ops.cc:
Was:
  static const auto MX_EXEC_CTX = Context::CPU();

Needs to be:
static const auto MX_EXEC_CTX = Context();

The default constructor of Context creates the equivalent CPU context on device 0, without calling the problematic Context::Create().
I have verified that the sequence of commands first posted in https://github.com/apache/incubator-mxnet/issues/15578.  My version of those commands:
pip install mxnet-cu101 --pre
HOROVOD_NCCL_INCLUDE=/usr/include HOROVOD_NCCL_LIB=/usr/lib/x86_64-linux-gnu HOROVOD_NCCL_LINK=SHARED HOROVOD_WITHOUT_PYTORCH=1 HOROVOD_WITHOUT_TENSORFLOW=1 HOROVOD_GPU_ALLREDUCE=NCCL pip install --no-cache-dir <my horovod clone dir>
mpirun -np 2 --allow-run-as-root -H localhost:2 pytest /home/dcarter/horovod2/test/test_mxnet.py

The default constructor of Context creates the equivalent CPU context on device 0, without calling the problematic Context::Create().

@DickJC123 This saves my day. I actually had the same idea of creating NDArray using C API but struggled on this part. I will create a PR in Horovod and verify. We can close this PR if all looks fine. Will keep you updated. Thanks.

Fixing the issue in Horovod. Close this PR in favor of https://github.com/horovod/horovod/pull/1244.

authors: DickJC123 marcoabreu szha yuxihu


text:
CI: Consolidate Dockerfiles

Description
This PR shows how to get rid of duplicate Dockerfiles and the install/X.sh code deduplication strategy on the example of CentOS7 Dockerfiles. Instead, we use a single templated Dockerfile with a multiple targets for different scenarios. The template arguments and respective targets are declared in docker-compose.yml.
This has multiple benefits:

Having a single Dockerfile instead of a number of scripts is easier to understand and maintain
Copying scripts into the Dockerfile as done in the previous approach prevents usage of the https://hub.docker.com/u/mxnetci/ cache on developer machines, as the COPY instruction invalidates the cache if the local repository does not match the repository used on CI (eg. development branch instead of master branch). If the cache is invalidated, developers must wait for long docker build time when attempting to reproduce certain CI builds (such as website build, cc @aaronmarkham) or using the docker image to perform the staticbuild.
Having fewer steps in the Dockerfile improves the build speed.

That looks really neat!

@mxnet-bot run ci [miscellaneous, website, centos-gpu, unix-gpu, edge, sanity, centos-cpu, clang, unix-cpu]

source: gharchive/pull-request

authors: leezu marcoabreu

repo: apache/incubator-mxnet

license_type: permissive

license_source: github-api


text:
[Website 2.0] Artifact URL Adjustment

Description
This PR officially transfers ownership of code version control and compressed artifact storage for the static artifacts for the MXNet website from Connor Goggins to the MXNet Website team. The new S3 bucket (owned by the AWS account for ai-mxnet-engineering@amazon.com) gives the team the ability to update the artifacts as needed. The specific purpose of this PR is to change the artifact download link in the Jenkinsfile to point to the compressed artifacts in the new S3 bucket instead of the previous bucket.
These changes will increase the accessibility of the MXNet Website 2.0 static artifacts to the team in case of future modifications. For additional details and supporting documentation, please read this doc.
Checklist
Essentials

[x] Changes are complete (i.e. I finished coding on this PR)
[x] Code is well-documented
[x] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

M docs/static_site/Makefile

Comments
These changes have been tested with a complete website build on Jenkins dev and local hosting. All functionality remains the same.
@sandeep-krishnamurthy @aaronmarkham @ys2843

@mxnet-label-bot add [pr-awaiting-review]

@mxnet-label-bot add [Website]


text:
[DOC] Update ubuntu_setup.md

Description
add libtiff which is a dependency
Checklist
Essentials

[X] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
[X] Changes are complete (i.e. I finished coding on this PR)
[X] All changes have test coverage
[X] Code is well-documented

Changes

[X] Add libtiff as a dependency for building R plugin to the Ubuntu documentation

Comments
May want to add version for Fedora as well, though I expect the assumption is that Ubuntu documentation should be general enough that it can be adapted to other linux distributions.

@bkmgit it may be easier to submit this PR to the v1.x branch

@leezu Ok, changed base branch to v1.x

Cleaner to merge to v1.x  in a new pull request, #19372

authors: bkmgit leezu


text:
[master] Unified oneDNN implementation calls for quantized operators

Description
In this PR I focused on unifying oneDNN implementation calls using DNNLRun method, so that all quantization operators would be consistent. Although some operators use  FStatefulComputeEx instead of FComputeEx which is not supported by the DNNLRun method and therefore they are unchanged.

@mxnet-bot run ci [sanity]

@mxnet-bot run ci [windows-gpu, unix-gpu, unix-cpu, unix-gpu, website]

@mxnet-bot run ci [centos-gpu]

@mxnet-bot run ci [centos-gpu, unix-cpu, unix-gpu, website]

@mxnet-bot run ci [unix-cpu]

@mxnet-bot run ci [unix-cpu]

authors: bgawrych piotrwolinski-intel


text:
net/tcp: fix devif callback list corruption on tcp_close()

Summary
net/tcp: fix devif callback list corruption on tcp_close()
devif_conn_event() will be called recursively in the psock_send_eventhandler(),
if the tcp event tcp_close_eventhandler() is marked as "next" in first devif_conn_event()
and released from sencond recursive call, the "next" event in the first devif_conn_event()
will become a wild pointer.
479 uint16_t devif_conn_event(FAR struct net_driver_s *dev, uint16_t flags,
480                           FAR struct devif_callback_s *list)
481 {
482   FAR struct devif_callback_s *next;
...
488   net_lock();
489   while (list && flags)
490     {
...
496       next = list->nxtconn;  <------------------  event tcp_close_eventhandler() on next
...
500       if (list->event != NULL && devif_event_trigger(flags, list->flags))
501         {
...
507           flags = list->event(dev, list->priv, flags);  <---------------- perform  psock_send_eventhandler(), event tcp_close_eventhandler() will be remove from tcp_lost_connection()
508         }
...
512       list = next;  <---------------- event tcp_close_eventhandler() has been released, wild pointer
513     }
514
515   net_unlock();
516   return flags;
517 }

The callstack as below:

Breakpoint 1, tcp_close_eventhandler (dev=0x56607d80 <g_sim_dev>, pvpriv=0x566084a0 <g_tcp_connections>, flags=65) at tcp/tcp_close.c:83
(gdb) bt
| #0  tcp_close_eventhandler (dev=0x56607d80 <g_sim_dev>, pvpriv=0x566084a0 <g_tcp_connections>, flags=65) at tcp/tcp_close.c:83
| #1  0x5658bb57 in devif_conn_event (dev=0x56607d80 <g_sim_dev>, flags=65, list=0x56609498 <g_cbprealloc+312>) at devif/devif_callback.c:507
                    ----------------> devif_conn_event() recursively
| #2  0x56589f8c in tcp_callback (dev=0x56607d80 <g_sim_dev>, conn=0x566084a0 <g_tcp_connections>, flags=65) at tcp/tcp_callback.c:169
| #3  0x565c55e4 in tcp_shutdown_monitor (conn=0x566084a0 <g_tcp_connections>, flags=65) at tcp/tcp_monitor.c:211
| #4  0x565c584b in tcp_lost_connection (conn=0x566084a0 <g_tcp_connections>, cb=0x566094b0 <g_cbprealloc+336>, flags=65) at tcp/tcp_monitor.c:391
| #5  0x565c028a in psock_send_eventhandler (dev=0x56607d80 <g_sim_dev>, pvpriv=0x566084a0 <g_tcp_connections>, flags=65) at tcp/tcp_send_buffered.c:544
                    ----------------> call psock_send_eventhandler() before tcp_close_eventhandler()
| #6  0x5658bb57 in devif_conn_event (dev=0x56607d80 <g_sim_dev>, flags=65, list=0x566094b0 <g_cbprealloc+336>) at devif/devif_callback.c:507
| #7  0x56589f8c in tcp_callback (dev=0x56607d80 <g_sim_dev>, conn=0x566084a0 <g_tcp_connections>, flags=65) at tcp/tcp_callback.c:169
| #8  0x5658e8cc in tcp_input (dev=0x56607d80 <g_sim_dev>, domain=2 '\002', iplen=20) at tcp/tcp_input.c:1059
| #9  0x5658ed77 in tcp_ipv4_input (dev=0x56607d80 <g_sim_dev>) at tcp/tcp_input.c:1355
| #10 0x5658c0a2 in ipv4_input (dev=0x56607d80 <g_sim_dev>) at devif/ipv4_input.c:358
| #11 0x56577017 in netdriver_recv_work (arg=0x56607d80 <g_sim_dev>) at sim/up_netdriver.c:182
| #12 0x5655999e in work_thread (argc=2, argv=0xf3db5dd0) at wqueue/kwork_thread.c:178
| #13 0x5655983f in nxtask_start () at task/task_start.c:129
(gdb) c
Continuing.
Breakpoint 1, tcp_close_eventhandler (dev=0x56607d80 <g_sim_dev>, pvpriv=0x566084a0 <g_tcp_connections>, flags=65) at tcp/tcp_close.c:83
(gdb) bt
| #0  tcp_close_eventhandler (dev=0x56607d80 <g_sim_dev>, pvpriv=0x566084a0 <g_tcp_connections>, flags=65) at tcp/tcp_close.c:83
      ----------------------> "next" corrupted, invaild call tcp_close_eventhandler()
| #1  0x5658bb57 in devif_conn_event (dev=0x56607d80 <g_sim_dev>, flags=65, list=0x56609498 <g_cbprealloc+312>) at devif/devif_callback.c:507
| #2  0x56589f8c in tcp_callback (dev=0x56607d80 <g_sim_dev>, conn=0x566084a0 <g_tcp_connections>, flags=65) at tcp/tcp_callback.c:169
| #3  0x5658e8cc in tcp_input (dev=0x56607d80 <g_sim_dev>, domain=2 '\002', iplen=20) at tcp/tcp_input.c:1059
| #4  0x5658ed77 in tcp_ipv4_input (dev=0x56607d80 <g_sim_dev>) at tcp/tcp_input.c:1355
| #5  0x5658c0a2 in ipv4_input (dev=0x56607d80 <g_sim_dev>) at devif/ipv4_input.c:358
| #6  0x56577017 in netdriver_recv_work (arg=0x56607d80 <g_sim_dev>) at sim/up_netdriver.c:182
| #7  0x5655999e in work_thread (argc=2, argv=0xf3db5dd0) at wqueue/kwork_thread.c:178
| #8  0x5655983f in nxtask_start () at task/task_start.c:129
(gdb) c
Continuing.
[    2.680000] up_assert: Assertion failed at file:devif/devif_callback.c line: 85 task: lpwork

Signed-off-by: chao.an anchao@xiaomi.com
Impact
Testing

@fjpanag ,related issue:
https://github.com/apache/incubator-nuttx/pull/6923#issuecomment-1228726044

@fjpanag could you try this patch with your stability test?

@fjpanag could you try this patch with your stability test?

Yes, I will!
Today it is a bit rough, we are deploying.
At most by tomorrow morning, the tests will have been started.
I will report back.

@fjpanag could you try this patch with your stability test?

Yes, I will!
Today it is a bit rough, we are deploying. At most by tomorrow morning, the tests will have been started.
I will report back.

Ok. So let's wait with merge till tomorrow

I did lot's of testing on this, both manually and stress tests. I tested various error conditions.
No crashes anymore.
I noticed that tcp_close_eventhandler() is not called in all cases, but in my limited time I couldn't determine whether this is intentional, so I concentrated on stability tests. Since the issue is identified on #6956, I think we can ignore it here.
I propose to merge this, everything seems fine.
The system seems very stable now.

authors: anchao fjpanag pkarashchenko xiaoxiang781216

repo: apache/incubator-nuttx


text:
WIP(service/box): add box service support

related issee: #2773

Very Sorry for the delay, I was exhausted from preparing for various intern job opportunities in the past few months, Now things have done, Let me try
to fix this pr🥺

Very Sorry for the delay, I was exhausted from preparing for various intern job opportunities in the past few months

There is no sorry needed :heart:

Now things have done, Let me try to fix this pr🥺

Our core changed a lot since your last update, it's better to update branch first.

Hi, @A-Stupid-Sun, we've made numerous updates to our Rust core, rendering this PR quite outdated. I'll be closing it but feel free to open a new one. I'm here to offer any help you might need.

authors: A-Stupid-Sun Xuanwo

repo: apache/incubator-opendal


text:
Documentation requirements for LogBack (default logger)

With the noted library use in LICENSE of "logback: which uses LGPL, have we done enough? After reading the advice in URL below, we address #2, but can we address #1?
https://softwareengineering.stackexchange.com/questions/86142/what-exactly-do-i-need-to-do-if-i-use-a-lgpl-licenced-library/86158
as noted logback is default logger as per docs:
https://github.com/apache/incubator-openwhisk/blob/925500cf8d34bb75bebd077ea057af236eba0b01/docs/logging.md
Default Logging Provider
OpenWhisk uses Logback as default logging provider via slf4j.
Logback can be configured in the configuration file logback.xml.
Besides other things this configuration file defines the default log level for OpenWhisk. Akka Logging inherits the log level defined here.
If we can show another logger over the same Interface this could satisfy #1

I found out a nice video, giving instructions regarding how to create the LICENSE for a project:
https://vimeo.com/171210141
When the project needs to depend on another project with dual license, we can pick up one of the licenses, which is more friendly to our need.
LogBack is under two licenses: Eclipse 1.0 and GNU LPGL 2.1, so we can select Eclipse 1.0 as a friendly choice for open source project.

Revisit this issue for further binary releases.

source: gharchive/issue

authors: houshengbo mrutkows

repo: apache/incubator-openwhisk-release


text:
Openwhisk native deployment, API host connection issue

vagrant@vagrant-ubuntu-trusty-64:~/openwhisk/bin$ ./wsk -i list
error: Unable to obtain the list of entities for namespace 'default': The connection failed, or timed out. (HTTP status code 502).

Can you try with -v: wsk -i -v list. Also consider joining http://slack.openwhisk.org (you're welcome to direct message me for help).

Sure. Thank you

repo: apache/incubator-openwhisk


text:
Support reading param.json from stdin #2837

For #2837. Adds support for -P - that reads the param json file from the stdin, as a common practice in *nix tools.
Disclaimer: My first shot at programming in go. Thus I blindly cargo-cult copied from the existing code base :)
I deliberately added the feature only for -P / --param-file. Initially I tried to do it generically inside readFile() from util.go, but it seems it's not a good idea to enable this for all callers of readFile(), which apparently include configuration files (i.e. file name is not passed as argument on the command line).
Example usage after the change:
cat params.json | envsubst | wsk -i action invoke myaction -P -

I think this needs a documentation/usage output change.

Usage now says for all cases of --param-file:
  -P, --param-file FILE   FILE containing parameter values in JSON format, use '-' to read from stdin

Also added a full example to docs/actions.md (scroll down to 4.).

@rabbah How do I run the WskBasicUsageTests.scala easily and isolated?

@alexkli, locally you can run ./gradlew tests:test -Dtest.single=WskBasicUsageTests from the OpenWhisk root directory.

Ah, but that gives this error (I think I saw it before when trying to run tests):
* Where:
Build file '/Users/alex/Work/code/openwhisk/incubator-openwhisk/tests/build.gradle' line: 74

* What went wrong:
Execution failed for task ':tests:createKeystore'.
> java.io.FileNotFoundException: /Users/alex/Work/code/openwhisk/incubator-openwhisk/whisk.properties (No such file or directory)

Do I need to run a full local OpenWhisk and have the connection details in the whisk.properties before being able to run the tests?

@alexkli, you do need a full OpenWhisk deployment setup locally.

@dubeejw Thanks. I have been using devtools docker-compose, I was told that's easiest to setup (and it works fine). How do I get the whisk.properties file for it? What needs to go into it?
In the meantime I found tools/build/redo props which says it will build whisk.properties file (required for tests). Had to install ansible using brew install ansible (which installed a lot of dependencies...).
However running tools/build/redo props does not create a whisk.properties, it outputs:
ansible-playbook -i environments/docker-machine properties.yml
 [WARNING]: Unable to parse /Users/alex/Work/code/openwhisk/incubator-openwhisk/ansible/environments/docker-machine as an inventory source

 [WARNING]: No inventory was parsed, only implicit localhost is available

 [WARNING]: Could not match supplied host pattern, ignoring: all

 [WARNING]: provided hosts list is empty, only localhost is available
...
skipping: no hosts matched

AFAICS, the wsk cli tests don't (all?) require a server? Is there no way to run them without?

While I have to wait for help with getting the tests to run, I had a look at WskBasicUsageTests.scala.
It seems there is not any positive test for -P, only failure tests if no filename is given or if it's an empty file. I feel like I am not familiar enough with scala and the code base here to know how to add such a positive test.

@akrabat, check here for the valid -P tests.
https://github.com/apache/incubator-openwhisk/blob/master/tests/src/test/scala/whisk/core/cli/test/WskBasicUsageTests.scala#L193
https://github.com/apache/incubator-openwhisk/blob/master/tests/src/test/scala/whisk/core/cli/test/WskBasicUsageTests.scala#L920
https://github.com/apache/incubator-openwhisk/blob/master/tests/src/test/scala/whisk/core/cli/test/WskBasicUsageTests.scala#L920

@alexkli, I'm not familiar with the devtools project. Perhaps @rabbah has some suggestions.

Try wskdev setup props
If you have not run ansible before the hosts file may not exist and the error suggests this (i guess we have a missing depedendencd).

wskdev setup props fails similar to tools/build/redo props, as they seem to run the same and expect a running local ansible based OpenWhisk.
However, after some fiddling around, I was able to successfully run at least the WskBasicUsageTests
./gradlew tests:test -Dtest.single=WskBasicUsageTests

against my local OpenWhisk from the docker-compose setup using this minimal whisk.properties file in the root of the checkout:
openwhisk.home=$OPENWHISK_CHECKOUT_DIR

testing.auth=$USER_HOME/tmp/openwhisk/src/ansible/files/auth.guest
whisk.ssl.cert=$USER_HOME/tmp/openwhisk/apigateway/ssl/openwhisk-server-cert.pem

edge.host=$LOCAL_IP:443
whisk.api.host.proto=https
whisk.api.host.name=$LOCAL_IP
whisk.api.host.port=443

vcap.services.file=
use.cli.download=false

Replace the variables in there:

$OPENWHISK_CHECKOUT_DIR: absolute path to the openwhisk git checkout
$USER_HOME: absolute path to your user home (e.g. /Users/alex), tilde ~ does not work
$LOCAL_IP: local IP (localhost might work too)

@dubeejw @rabbah Thanks for the help. Once I got the tests running and the project in IntelliJ with Scala support, it was fairly straightforward.
PR updated with

tests for -P - option for: action/package/trigger create, and action invoke with file and stdin cases
updated the i18n json file

PG3 1328 ⏳

Unfortunately this will require some git surgery. The go changes need to go in the cli repo now. The rest can stay here I think.

@rabbah are you asking something from my side? I am not familiar with the repo split.

I’ll try to spare you the overhead.

Closing this one as the CLI has moved to incubator-openwhisk-cli and incubator-openwhisk-client-go.

authors: alexkli dubee dubeejw rabbah


text:
Make stemcells configurable by deployment

Description
This PR replaces #3232 and makes stem cell containers configurable from the runtime manifest. See the changes to runtimes.json for an example (or the new tests). It is possible to have multiple stem cells per runtime kind. For example:
{ "runtimes": {
    "node": [
      {
        "kind": "nodejs:6",
        "image": {
          "name": "nodejsaction"
        },
        "stemCells": [{
          "count": 1,
          "memory": "128 MB"
        }]
      }, {
        "kind": "nodejs:8",
        "default": true,
        "image": {
          "name": "nodejsaction"
        },
        "stemCells": [{
          "count": 1,
          "memory": "128 MB"
        }, {
          "count": 1,
          "memory": "256 MB"
        }]
      }
    ]...

This PR improves #3232 with stricter types on the StemCell configuration, filters configrations for runtimes for the ones that exists to make it easier to work with stem cells in the invoker.
Closes #2401.

Related issue and scope

[x] There is an open an issue to propose and discuss this change (#2401)

My changes affect the following components

[ ] API
[x] Controller
[ ] Message Bus (e.g., Kafka)
[ ] Loadbalancer
[x] Invoker
[ ] Intrinsic actions (e.g., sequences, conductors)
[ ] Data stores (e.g., CouchDB)
[x] Tests
[x] Deployment
[ ] CLI
[ ] General tooling
[ ] Documentation

Types of changes

[ ] Bug fix (generally a non-breaking change which closes an issue).
[x] Enhancement or new feature (adds new functionality).
[x] Breaking change (a bug fix or enhancement which changes existing behavior).

Labelled breaking change because it will require downstream changes to enable stem cells.
It's plausible we should warn or fail startup if no stemcells are defined.
Checklist:

[x] I signed an Apache CLA.
[x] I reviewed the style guides and followed the recommendations (Travis CI will check :).
[x] I added tests to cover my changes.
[ ] My changes require further changes to the documentation.
[x] I updated the documentation where necessary.

Codecov Report

Merging #3669 into master will increase coverage by 0.09%.
The diff coverage is 90.9%.

@@            Coverage Diff             @@
##           master    #3669      +/-   ##
==========================================
+ Coverage   74.49%   74.58%   +0.09%
==========================================
  Files         126      126
  Lines        5990     6004      +14
  Branches      390      393       +3
==========================================
+ Hits         4462     4478      +16
+ Misses       1528     1526       -2

Impacted Files
Coverage Δ

...ain/scala/whisk/core/invoker/InvokerReactive.scala
0% <0%> (ø)
:arrow_up:

...rc/main/scala/whisk/core/entity/ExecManifest.scala
96.62% <100%> (+0.52%)
:arrow_up:

.../scala/src/main/scala/whisk/core/entity/Size.scala
97.82% <100%> (+0.2%)
:arrow_up:

...scala/whisk/core/containerpool/ContainerPool.scala
85.88% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed71ee0...d78d599. Read the comment docs.

@tysonnorris I haven't had a chance to open PR against the downstream repos which will be affected (no stem cells by default).
Also, it appears the commit message possibly because of subsequent commits whose messages were amended to the message did not properly credit @Himavanth.

authors: codecov-io rabbah


text:
Add wait for condition for toggle test

This PR:

adds wait for condition for toggle test in setup method.
changed the APIs to get the list of instance configs in cached data accessor.

Codecov Report

Merging #4662 into master will increase coverage by 21.71%.
The diff coverage is 100%.

@@              Coverage Diff              @@
##             master    #4662       +/-   ##
=============================================
+ Coverage     42.97%   64.69%   +21.71%
- Complexity        0       16       +16
=============================================
  Files          1056     1056
  Lines         56200    56200
  Branches       8330     8330
=============================================
+ Hits          24152    36357    +12205
+ Misses        29917    17136    -12781
- Partials       2131     2707      +576

Impacted Files
Coverage Δ
Complexity Δ

...ntroller/helix/core/PinotHelixResourceManager.java
61.55% <100%> (+24.44%)
0 <0> (ø)
:arrow_down:

...n/java/org/apache/pinot/core/transport/Server.java
59.25% <0%> (-7.41%)
0% <0%> (ø)

...thandler/SingleConnectionBrokerRequestHandler.java
84.09% <0%> (-4.55%)
0% <0%> (ø)

...pache/pinot/core/util/SortedRangeIntersection.java
83.82% <0%> (-4.42%)
0% <0%> (ø)

...org/apache/pinot/client/DynamicBrokerSelector.java
69.69% <0%> (-3.04%)
0% <0%> (ø)

...der/HighLevelConsumerBasedRoutingTableBuilder.java
90.9% <0%> (-3.04%)
0% <0%> (ø)

...e/impl/dictionary/LongOnHeapMutableDictionary.java
64.63% <0%> (-2.44%)
0% <0%> (ø)

...pinot/core/operator/docidsets/OrBlockDocIdSet.java
84.9% <0%> (-1.89%)
0% <0%> (ø)

...er/routing/builder/RoutingTableInstancePruner.java
84.21% <0%> (ø)
0% <0%> (ø)
:arrow_down:

...r/validation/RealtimeSegmentValidationManager.java
81.96% <0%> (ø)
0% <0%> (ø)
:arrow_down:

... and 575 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8a80760...05a3778. Read the comment docs.

Don't change the test. We need to fix the code

This PR isn't needed any more. Closing this PR.

authors: Jackie-Jiang codecov-io jackjlli

repo: apache/incubator-pinot


text:
shading dependencies in pulsar client

Minimizing the number of dependencies in Pulsar client by shading them

retest this please

retest this please

retest this please

retest this please

repo: apache/incubator-pulsar


text:
[REEF-971] Create test for FailedTaskHandler

This addressed the issue by

Create test for FailedTaskHandler.
Added TestCleanup for TestFailedEvaluatorEventHandler.

JIRA:
REEF-971

It looks good to me. I'll test and merge this.

authors: afchung dongjoon-hyun

repo: apache/incubator-reef


text:
The ASF comments are showing up in cli help logs.

The rather interesting way comments are logged does not include
stripping out any comments.
This is a hasty way, although there could be better alternatives
in the future.

All, and @TimBarham thoughts on including this in first release? Seems to work good.

Seems a reasonable enough approach to me. Only suggestion I would make is to strip the left over leading line breaks?

Oh, and regarding including this in the release - I'm fine with that, though we would have to bump the version again and rebuild the package, and restart the vote :).

@TimBarham thanks for the thoughts. I went ahead and implemented that. Forced pushed this commit (you can see spaces are removed): https://github.com/brentlintner/incubator-ripple/commit/9d1b0d148f40d8262038446a3b14c00d0d807acb#diff-9669607edc603c1f4cd67195aa47db35R25
Seems to work properly.
As for the release, I'll leave it to you to decide. You are the one doing the work. :-). I will bump the version, regardless, when I merge this. :-)

authors: TimBarham brentlintner

repo: apache/incubator-ripple


text:
[Bug] [seatunnel-core-base] Error when tar directory 'plugins' to file 'plugins.tar.gz'

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
when run seatunnel job ,i want to know  which version can we use 2.1.0 or 2.1.1 . like this simple error
Exception in thread "main" java.nio.file.AccessDeniedException: plugins.tar.gz
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
at java.nio.file.Files.newOutputStream(Files.java:216)
at org.apache.seatunnel.utils.CompressionUtils.tarGzip(CompressionUtils.java:62)
at org.apache.seatunnel.SparkStarter$ClusterModeSparkStarter.buildCommands(SparkStarter.java:381)
at org.apache.seatunnel.SparkStarter.main(SparkStarter.java:98)
0 [main] INFO org.apache.seatunnel.utils.CompressionUtils - Tar directory 'plugins' to file 'plugins.tar.gz'. 1 [main] ERROR org.apache.seatunnel.utils.CompressionUtils - Error when tar directory 'plugins' to file 'plugins.tar.gz'.
SeaTunnel Version
2.1.1
SeaTunnel Config
..............

Running Command
....................

Error Exception
Exception in thread "main" java.nio.file.AccessDeniedException: plugins.tar.gz
		at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
		at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
		at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
		at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
		at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
		at java.nio.file.Files.newOutputStream(Files.java:216)
		at org.apache.seatunnel.utils.CompressionUtils.tarGzip(CompressionUtils.java:62)
		at org.apache.seatunnel.SparkStarter$ClusterModeSparkStarter.buildCommands(SparkStarter.java:381)
		at org.apache.seatunnel.SparkStarter.main(SparkStarter.java:98)
	0 [main] INFO org.apache.seatunnel.utils.CompressionUtils - Tar directory 'plugins' to file 'plugins.tar.gz'. 1 [main] ERROR org.apache.seatunnel.utils.CompressionUtils - Error when tar directory 'plugins' to file 'plugins.tar.gz'.

Flink or Spark Version
2.4.0-cdh6.3.2
Java or Scala Version
1.8
Screenshots
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Please provide your environment, like Spark deployment mode (yarn or standalone), and your submit shell command.

And please make sure your SeaTunenl package permission is can be access with your start shell command's user.

cluster mode,  why use shell to tar resource ? Don't be so different between versions. Don't refactor stable modules and focus on other directions

cluster mode, why use shell to tar resource ? Don't be so different between versions. Don't refactor stable modules and focus on other directions

Compress resource is in Java code. Use shell to call java program. And can you provide permission for plugins.tar.gz and project root path?

In the user directory(/home), and I can tar resource under the shell, 1.5 X and 2.1.0 are OK. The changes in this part are so big that the test work is not done well. users can't run job  with root

cluster mode, why use shell to tar resource ? Don't be so different between versions. Don't refactor stable modules and focus on other directions

Compress resource is in Java code. Use shell to call java program. And can you provide permission for plugins.tar.gz and project root path?

Did you develop this code?, Can we go back to 2.1.0? Don't be so fancy on these simple functions that we can't be used

plugins.tar.gz have a long history, not develop on 2.1.1

mark

repo: apache/incubator-seatunnel


text:
[Connector-V2][JDBC-connector] Add more databases that support JDBC

Search before asking

[X] I had searched in the feature and found no similar feature requirement.

Description
Currently the JDBC connector supports mysql and Postgres, and we plan to support more.
as follows:

NO
Database
Source/Sink
Contributer
Status

1
Mysql
sink
@ic4y
Done

2
Mysql
source
@ic4y
Done

3
Postgres
sink
@ic4y
Done

4
Postgres
source
@ic4y
Done

3
SQLServer
sink

4
SQLServer
source

5
Orecal
sink

6
Orecal
source

7
GaussDB
sink

8
GaussDB
source

9
DM达梦
sink

10
DM达梦l
source

11
DB2
sink

12
DB2l
source

Additions and contributions are welcome.
Usage Scenario
No response
Related issues
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I want try it for dm first

Could you assigne the DB2 to me ，and I am willing to do the weekend.
@Hisoka-X

@lhyundeadsoul
Are you interested in implementing TiDB under JDBC Connector?

@lhyundeadsoul Are you interested in implementing TiDB under JDBC Connector?
TiDB is as same as MySQL.
This has been discussed:
https://github.com/apache/incubator-seatunnel/issues/1946#issuecomment-1211797586

You are right, thanks for the reminder. TiDB uses the mysql driver.

authors: ic4y laglangyue lhyundeadsoul


text:
[Feature][Flink-SQL-connector] add flink sql connector elasticsearch 6&7 and docs

Purpose of this pull request
issue link https://github.com/apache/incubator-seatunnel/issues/1922
Check list

[ ] Code changed are covered with tests, or it does not need tests for reason:
[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

Hi, @ruanwenjun , would you help review this PR?
Thanks.

@legendtkl I see the code at Executor, can the CustomClassLoader load the plugin jar? it seems you don't add the jar path into the classloader before you execute below code.
Iterator<Factory> factories = ServiceLoader.load(Factory.class, CLASSLOADER).iterator();

@legendtkl I see the code at Executor, can the CustomClassLoader load the plugin jar? it seems you don't add the jar path into the classloader before you execute below code.
Iterator<Factory> factories = ServiceLoader.load(Factory.class, CLASSLOADER).iterator();

Hi, @ruanwenjun , Flink SQL connector jars will not be added to the JVM classpath by default. We will add it to the classpath when we parse the SQL and find the connector Factory not in Classload.
This is what Executor.loadConnector method does.

@legendtkl I see the code at Executor, can the CustomClassLoader load the plugin jar? it seems you don't add the jar path into the classloader before you execute below code.
Iterator<Factory> factories = ServiceLoader.load(Factory.class, CLASSLOADER).iterator();

Hi, @ruanwenjun , Flink SQL connector jars will not be added to the JVM classpath by default. We will add it to the classpath when we parse the SQL and find the connector Factory not in Classloader.
This is what Executor.loadConnector method does.

I understand. When the connector has not been loaded, you will find the connector jar and add to classoader. There is no problem.
while (factories.hasNext()) {
    Factory factory = factories.next();

    /**
     * Handle for two cases:
     * 1. Flink built-in connectors.
     * 2. Connectors have been placed in classpath.
     */
    if (factory.factoryIdentifier().equals(connectorType)) {
        return;
    }
}

authors: legendtkl ruanwenjun


text:
Upgrade from 3.0.0 to 3.1.0.M1 error reporting，java.lang.NullPointerException: Cannot invoke method mod() on null object

sharding code:

sharding database column name : traceId
sharding strategy :ds${traceId%4}

      // 配置真实数据源
        Map<String, DataSource> dataSourceMap = new HashMap<>();
        dataSourceMap.put("ds0",ds0DataSource());
        dataSourceMap.put("ds1",ds1DataSource());
        dataSourceMap.put("ds2",ds2DataSource());
        dataSourceMap.put("ds3",ds3DataSource());
        dataSourceMap.put("mainDataSource",mainDataSource());

        // 配置Order表规则
        TableRuleConfiguration orderTableRuleConfig = new TableRuleConfiguration();
        orderTableRuleConfig.setLogicTable("push_message");
        orderTableRuleConfig.setActualDataNodes("ds${0..3}.push_message");

        //分布式主键
        orderTableRuleConfig.setKeyGeneratorColumnName("id");
        ShardingDefaultKeyGenerator shardingDefaultKeyGenerator = new ShardingDefaultKeyGenerator();
        orderTableRuleConfig.setKeyGenerator(shardingDefaultKeyGenerator);

        // 配置分库策略
        orderTableRuleConfig.setDatabaseShardingStrategyConfig(
                new InlineShardingStrategyConfiguration("traceId", "ds${traceId%4}"));

        // 配置分片规则
        ShardingRuleConfiguration shardingRuleConfig = new ShardingRuleConfiguration();
        shardingRuleConfig.getTableRuleConfigs().add(orderTableRuleConfig);
        shardingRuleConfig.setDefaultDataSourceName("mainDataSource");

        Properties properties = new Properties();
        properties.setProperty("sql.show","true");
        properties.setProperty("max.connections.size.per.query","200");
        properties.setProperty("executor.size",String.valueOf(Runtime.getRuntime().availableProcessors() * 2));

        DataSource dataSource = ShardingDataSourceFactory.createDataSource(
                dataSourceMap, shardingRuleConfig, new ConcurrentHashMap(), properties);
        return dataSource

error info:
java.lang.NullPointerException: Cannot invoke method mod() on null object

	at org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:91)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:35)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at Script2$_run_closure1.doCall(Script2.groovy:1)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
	at groovy.lang.Closure.call(Closure.java:426)
	at groovy.lang.Closure.call(Closure.java:420)
	at com.fly.ApplicationTests.testInline(ApplicationTests.java:98)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
	at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
	at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
	at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

sharding key is hump naming，calculating rote Error,
    //InlineShardingStrategy.java
    private String execute(final PreciseShardingValue shardingValue) {
        Closure<?> result = closure.rehydrate(new Expando(), null, null);
        result.setResolveStrategy(Closure.DELEGATE_ONLY);
        //method toLowerCase()
        result.setProperty(shardingValue.getColumnName().toLowerCase(), shardingValue.getValue());
        return result.call().toString();
    }

unit test :

test fail

@Test
    public void testInline(){
        Closure<?> closure = new InlineExpressionParser("ds${traceId%4}").evaluateClosure();
        Closure<?> result = closure.rehydrate(new Expando(), null, null);
        result.setResolveStrategy(Closure.DELEGATE_ONLY);
        PreciseShardingValue shardingValue = new PreciseShardingValue("push_message","traceId",342342);
        result.setProperty(shardingValue.getColumnName().toLowerCase(), shardingValue.getValue());
        System.out.println("--------:"+result.call().toString());
    }

test pass

@Test
    public void testInline(){
        Closure<?> closure = new InlineExpressionParser("ds${traceid%4}").evaluateClosure();
        Closure<?> result = closure.rehydrate(new Expando(), null, null);
        result.setResolveStrategy(Closure.DELEGATE_ONLY);
        PreciseShardingValue shardingValue = new PreciseShardingValue("push_message","traceId",342342);
        result.setProperty(shardingValue.getColumnName().toLowerCase(), shardingValue.getValue());
        System.out.println("--------:"+result.call().toString());
    }

For the two test cases, what's the standard of pass or fail? I didn't find what you expect.

Fixed.
You are right, algorithmExpression will be transform to groovy script which is case sensitive. So columnName should not cast to lower case.

Caused by: java.lang.NullPointerException: Cannot invoke method mod() on null object
at org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:91)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:43)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:34)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:55)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
at Script2$_run_closure1.doCall(Script2.groovy:1)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
at groovy.lang.Closure.call(Closure.java:405)
at groovy.lang.Closure.call(Closure.java:399)
at org.apache.shardingsphere.core.strategy.route.inline.InlineShardingStrategy.execute(InlineShardingStrategy.java:86)
at org.apache.shardingsphere.core.strategy.route.inline.InlineShardingStrategy.doSharding(InlineShardingStrategy.java:68)
at org.apache.shardingsphere.core.strategy.route.inline.InlineShardingStrategy.doSharding(InlineShardingStrategy.java:59)
at org.apache.shardingsphere.core.route.type.standard.StandardRoutingEngine.routeDataSources(StandardRoutingEngine.java:191)
at org.apache.shardingsphere.core.route.type.standard.StandardRoutingEngine.route(StandardRoutingEngine.java:178)

repo: apache/incubator-shardingsphere

license_source: bigquery


text:
order by not support string column when cross database query

Bug Report
For English only, other languages will not accept.
Before report a bug, make sure you have:

Searched open and closed GitHub issues.
Read documentation: ShardingSphere Doc.

Please pay attention on issues you submitted, because we maybe need more details.
If no response more than 7 days and we cannot reproduce it on current information, we will close it.
Please answer these questions before submitting your issue. Thanks!
Which version of ShardingSphere did you use?
4.0.0
Which project did you use? Sharding-JDBC or Sharding-Proxy?
Sharding-Proxy
Expected behavior
cross database query support order by use string column, query sql is always executing, cann't get result or error
Actual behavior
not support, only support when route one database, cross database support i test is int、datetime etc
Reason analyze (If you can)
Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.
create table sql
CREATE TABLE t_cluster(
    clusterId VARCHAR(64) NOT NULL   COMMENT '数据库集群ID' ,
    clusterName VARCHAR(128) NOT NULL   COMMENT '数据库集群名称' ,
    projectId VARCHAR(64) NOT NULL   COMMENT '项目ID' ,
    createTime DATETIME NOT NULL   COMMENT '创建时间' ,
    updateTime DATETIME NOT NULL   COMMENT '更新时间' ,
    PRIMARY KEY (clusterId)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT = '数据库集群信息表';
ALTER TABLE t_cluster ADD UNIQUE index_cluster_unique_name(clusterName);

server.yaml
authentication:
  users:
    root:
      password: xxxxx
    migu:
      password: xxxxx
      authorizedSchemas: databasemanager
props:
  sql.show: true

config-databasemanager.yaml
schemaName: databasemanager

dataSources:
  ds_0:
    url: jdbc:mysql://192.168.137.102:3306/databasemanager?serverTimezone=UTC&useSSL=false
    username: root
    password: xxxxx
    connectionTimeoutMilliseconds: 30000
    idleTimeoutMilliseconds: 60000
    maxLifetimeMilliseconds: 1800000
    maxPoolSize: 50
  ds_1:
    url: jdbc:mysql://192.168.137.102:3307/databasemanager?serverTimezone=UTC&useSSL=false
    username: root
    password: xxxx
    connectionTimeoutMilliseconds: 30000
    idleTimeoutMilliseconds: 60000
    maxLifetimeMilliseconds: 1800000
    maxPoolSize: 50

shardingRule:
  tables:
    t_cluster:
      actualDataNodes: ds_${0..1}.t_cluster
      tableStrategy:
        none:
      databaseStrategy:
        inline:
          shardingColumn: clusterId
          algorithmExpression: ds_${clusterId.hashCode().abs() % 2}
    t_cluster_instance:
      actualDataNodes: ds_${0..1}.t_cluster_instance
      tableStrategy:
        none:
      databaseStrategy:
        inline:
          shardingColumn: clusterId
          algorithmExpression: ds_${clusterId.hashCode().abs() % 2}
  bindingTables:
    - t_cluster,t_cluster_instance
  defaultDatabaseStrategy:
    none:
  defaultTableStrategy:
    none:

Example codes for reproduce this issue (such as a github link).
test sql
select * from t_cluster order by clusterId;

could you show your actual result and expected result, actual sql of all tables.

could you show your actual result and expected result, actual sql of all tables.

table t_cluster no data
expected result
return no data
actual result
cann't get result, query is executing, no response

log
sharding-proxy

mysql

Cannot support cross database query for any SQL

Cannot support cross database query for any SQL

what you mean query sql must route one database, cann't query more than one database data?

authors: kimmking terrymanu xiaoxiong581


text:
Is the 2.4.0-release branch the latest stable version ?

I want to know,Is the 2.4.0-release branch the latest stable version ?
if it is , I will update my code.

Not yet. The new release is in progress.

repo: apache/incubator-shenyu


text:
Sync 6.0.0-alpha release and version up to beta.

Even if we failed in alpha release, we will use origin/6.0.0-alpha-release branch to do alpha2 release, I intend up master branch version into beta

Coverage decreased (-0.09%) to 12.1% when pulling 823d2fcdd0f11dcac880f9f2306e47191c1d403d on release into 7537af05981958686a99d1ec2e1b9a9eeb0d1d4f on master.

authors: coveralls wu-sheng

repo: apache/incubator-skywalking


text:
refactor: Remove usages of reactable from TableLoader

SUMMARY
The goal is to replace all uses of reactable-arc library with DataTable from superset-ui (which uses react-table under the hood) and keep the UI/UX mostly unchanged. The reason for that is that reactable-arc has been unsupported for 2 years now, uses deprecated lifecycle methods and is incompatible with Typescript.
This PR replaces reactable-arc in TableLoader component.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Before (nothing renders here, even though the data is correctly fetched. Bug perhaps?):

After:

TEST PLAN

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

@rusackas Can you please take a look?

Codecov Report

Merging #10999 into master will decrease coverage by 4.37%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master   #10999      +/-   ##
==========================================
- Coverage   65.76%   61.38%   -4.38%
==========================================
  Files         816      816
  Lines       38399    38413      +14
  Branches     3607     3616       +9
==========================================
- Hits        25254    23581    -1673
- Misses      13037    14644    +1607
- Partials      108      188      +80

Flag
Coverage Δ

#cypress
?

#javascript
61.65% <0.00%> (-0.08%)
:arrow_down:

#python
61.22% <ø> (-0.20%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...rset-frontend/src/components/ListView/ListView.tsx
95.60% <ø> (-3.30%)
:arrow_down:

superset-frontend/src/components/TableLoader.jsx
4.25% <0.00%> (-2.42%)
:arrow_down:

superset-frontend/src/SqlLab/App.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/explore/App.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/dashboard/App.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/explore/index.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/dashboard/index.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/setup/setupColors.js
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/chart/ChartContainer.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/setup/setupFormatters.js
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 171 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bd140e0...25b77a8. Read the comment docs.

@rusackas Thanks for review. However, due to this issue, which is related to changes made here, what do you think about postponing merging this PR until the issue is resolved? So that we don't risk another regression.

authors: codecov-commenter kgabryje

repo: apache/incubator-superset


text:
failed to install

no such directory 'static'

Coverage remained the same at 69.535% when pulling 283ddcb27c0bc22de85c9cf9ceb7beab76a8b146 on joriewong:patch-1 into 9af34ba51cdb23a2970e968d0cbdee82778c6799 on apache:master.

Are you installing on windows?

centos

The there's something wrong with your installation as the path should be there.

Yeah that's better than going through the symlink, merging.

Broke the pypi build

authors: coveralls joriewong mistercrunch xrmx


text:
Separate vis-specific controls from centralized controls

Summary
Allow each visualization to define its own controls, instead of storing all controls in centralized controls.jsx

Test plan
Verified that the rose and paired t-test charts that are modified in this PR are working correctly.

@mistercrunch @williaster @michellethomas @graceguo-supercat @conglei @khtruong @xtinec

Codecov Report

Merging #6827 into master will decrease coverage by 0.03%.
The diff coverage is 64.06%.

@@            Coverage Diff             @@
##           master    #6827      +/-   ##
==========================================
- Coverage   64.94%   64.91%   -0.04%
==========================================
  Files         424      424
  Lines       20598    20613      +15
  Branches     2281     2292      +11
==========================================
+ Hits        13378    13381       +3
- Misses       7097     7110      +13
+ Partials      123      122       -1

Impacted Files
Coverage Δ

superset/assets/src/explore/controls.jsx
42.4% <ø> (ø)
:arrow_up:

superset/assets/src/explore/store.js
61.64% <43.33%> (-0.2%)
:arrow_down:

.../src/explore/components/ControlPanelsContainer.jsx
67.94% <82.35%> (-12.06%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 07d9dbd...4fe63c0. Read the comment docs.

@mistercrunch Saw that you are doing a heavy refactor on the control panel so I am not sure when should I land this one?

related to https://github.com/apache/incubator-superset/pull/7350
gonna have huge conflict.

@kristw on my PR I'm mostly just touching/refactoring store.js, and I think most of your changes on that file here are around naming (except for getControlItems where you go a bit deeper...). I could add flow + flatMap on my side.
Maybe you can leave your changes out on that file so I can get my PR in? Otherwise it looks like it will conflict pretty hard.
My PR addresses some of the bad names and breaks down the big nasty functions into smaller chunks and introduces decent test coverage.

authors: codecov-io kristw mistercrunch


text:
TO go: go get prior to running unit tests for the golang.org/x dependencies

unit tests in jenkins started failing when golang.org/x dependencies were removed from the vendor dir.   This gets those dependencies in the docker container prior to running the tests.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-trafficcontrol-PR/1386/
Test PASSed.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-trafficcontrol-PR/1405/
Test FAILed.

retest this please

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-trafficcontrol-PR/1407/
Test PASSed.

authors: asfgit dangogh

repo: apache/incubator-trafficcontrol


text:
[RFC] Benchmark Performance Log Format

Workload record storage proposal
This is a migration and summation of the discussion https://discuss.tvm.ai/t/rfc-benchmark-performance-log-format/4610/15. Please head over there to put in your two cents. This is a WIP issue maintained for visibility.
We propose a json-based data storage format that unifies how TVM results are tracked. For now the only metric we suggest tracking is execution time, with a focus on reproducibility. We foresees the need for a standard record format to pinpoint the most crucial target areas for future development in TVM.
Use cases and motivation

Easier cross-reference to other frameworks/runs
Experiment reproduction
Building nightly perf regression
Standardization will separate logging and visualization efforts

Key points to consider

Maintain reproducibility of experiments based solely on an output record
Granularity of metrics -- e.g. per-layer vs end-to-end
config/misc json objects -- enforce prescriptive keys or keep it flexible for now?
Extensibility of this schema towards desired features in the future

Proposed format

header
examples
category
notes/justification

timestamp
1572282699.6
metadata
indicates when this record is logged

schema_version
0.1
metadata
ensure reproducibility as we iterate on this schema

metadata
{ "docker_tag":"gcr.io/.../0a680", ... }
metadata
docker_tag is optional

workload
resnet-18
workload
name of workload

workload_args
{“input_name”: "Input3", “input_shape”: [list_of_shape], “data_layout”: NHCW}
workload

workload_metadata
{"class": "vision","doc_url": "https://github.com/.../README.md","opset": 7,"type": "body_analysis","url": "https://onnxzoo...ferplus.tar.gz""md5": "07fc7..."}
workload
source of workload

engine
tvm / onnx-runtime
compiler

engine_version
"1.0.0"
compiler
version in semvar format

engine_config
{“llvm”: “llvm-8”, “nvcc”: 10.1, "accelerator": "MLAS"}
compiler
fields are optionally specified

compilation_config
{"opt_level": 3, "layer_schedules": [...]}
compiler
optional

software_config
{"os": "ubuntu:18.04","pip": { "docker": "4.1.0", "gitpython": "3.0.4", "numpy": "1.17.4", "onnx": "1.6.0"}, “cudnn”: “cudnn-8”, "cuda_driver”: “480.10.1”}
backend
env dependency list

runtime_config
{"num_cpu_threads": 3, ...}
runtime
info on non-hardware, non-software metadata

hardware
"gcp-c2-standard-16"
hardware
hardware target summary

hardware_config
{"cpu_count": 16,"cloud_machine_type":"c2-standard-16""memory_GB":64}
hardware
more detailed descriptor of target hardware environment

execution_config
{“number”: 1, “repeat”: 10, “min_repeat_ms”, 0}
statistics
workload execution parameters

metrics
{"accuracy": 34.1, "compilation_time_ms":24}
statistics
other metrics

runtime_raw
[{"runtime_ms": 12, ...}, {"runtime_ms":13,...},...]
statistics
optional raw metrics array

runtime_ms_mean
12
statistics

runtime_ms_std
3
statistics

Please share your thoughts about the RFC. this RFC will open for a week, if there are no further comments, we will harden it by bringing it as a part of tvm official document.

I just updated the initial comment with modifications -- this is the solidification of schema version 0.1.

Awesome, let us bring it to the official document by sending a pr to docs/dev/benchmark.rst

@anwang2009 I'd like to suggest one last modification: the removal of the runtime_config since many of the fields overlap with the software_config.

And if possible the addition of an md5 field under workload_metadata to attest the integrity of the model.

Other than that looks great 👍

markdown has been updated to reflect removal of runtime_config.
num_cpu_threads can be optionally specified in hardware_config, which is optionally json. Optional placement of cudnn and cuda_driver has been moved to software_config.
md5 has been added to workload_metadata
These changes will be reflected in the tvm PR.

Some further comments wrt to the recently hardware_config change

I still think it is important to keep the runtime_config, which includes fields like num_cpu_threads. Now that software config mainly specifies the environment dependencies, maybe it makes sense to move num_cpu_threads into runtime_config, as it is something that can be specified through runtime env variable.
It is important to keep hardware_config simple, because this is the key that we use to compare among targets.
docker_tag should not be a required field, as not all experiments have to ran through a docker container. I would even consider move docker_tag as an optional field of engine_config, as it is something related to the software engine.

@tqchen docker_tag is under metadata; therefore it's mostly seen as optional (and is here for example purposes)
Indeed num_cpu_threads is a tricky one and I thought long and hard about this; it makes sense not to place it under software_config, and instead could be considered a hardware metric. But generally thinking, when do we want to reduce the number of CPU threads that are made usable by the hardware target? For instance with n1-standard-16 we have 16CPUs. In what situations do we want to reduce the number of available virtual threads? And how would that be enforced? There are multiple ways to do so; one example is while doing docker run we can pass the number of CPUs the container can utilize.

I think docker_tag under metadata as long as it is optional (the RFC said the field was required which confuses me).
There are certainly cases when people want to limit the number of threads, e.g. use only the little cores on the phone to save power. While these could have been a nested field in the hardware_config, it makes the field quite complicated.
One of the key property of we should push for the log format is its simplicity. The three fields(engine, workload, hardware_config) are three fields that the user will mostly visit. e.g. ask how can "tvm" do for "resnet-18" on "rasp3b"
Of course the information was incomplete, but the in many cases the three fields already gives a quite complete picture, assuming the default implied target and maximum usage of the resources.
Given that number of threads and other runtime configurations(thread pool setups) are indeed runtime properties that was different from software_config. I think we should keep  the original runtime_config field for these information.

That's a sound argument; we should keep runtime config for the num_cpu_threads then to store information on non-hardware, and non-software metadata

I think docker_tag under metadata as long as it is optional (the RFC said the field was required which confuses me).
There are certainly cases when people want to limit the number of threads, e.g. use only the little cores on the phone to save power. While these could have been a nested field in the hardware_config, it makes the field quite complicated.

One of the key property of we should push for the log format is its simplicity. The three fields(engine, workload, hardware_config) are three fields that the user will mostly visit. e.g. ask how can "tvm" do for "resnet-18" on "rasp3b"

Specifically Rasp3b is fine because it's not big Little . It only has Cortex-A53 cores.
If the run of the benchmarking is able to pin tasks on specific cores for b.L systems, then I don't see how you can avoid that - unless the TVM stack can only run on the big or only on the little.

Of course the information was incomplete, but the in many cases the three fields already gives a quite complete picture, assuming the default implied target and maximum usage of the resources.

Sorry, that  puzzles me again - missing hardware configuration will cause confusion, it certainly  misses some information.
Not capturing that information would result in user confusion and comparison of apples and oranges. I appreciate the push towards simplicity but it feels that it's at the cost of accuracy. If I were looking at it and I wanted to know the difference then I would prefer to look at it. It's quite likely that if we were looking for performance monitoring over time, then figuring out a way of distinguishing between pinning tasks on big vs pinning tasks on little vs using EAS across the board and therefore note pinning would need to be capture.

Given that number of threads and other runtime configurations(thread pool setups) are indeed runtime properties that was different from software_config. I think we should keep the original runtime_config field for these information.

To clarify, by in-complete i mean the three fields(model, engine, and hardware) are incomplete. The additional information(engine_config, version, runtime_config) are meant to fill in that gap and provide a complete information.

Cool, thanks @tqchen

To pick up on a couple of topics from the Pull Request.

I've found that keeping the raw data allows one to process it in other ways in terms of { iteration: 1 runtime: .... } instead of only storing the statistics ?

Different folks would want to do different statistics with the data and we may not be able to specify all the statistics needed on day 1 and would need to keep adding. By keeping raw data we are able to do the aggregation on a later date across history.

This is a bit minor - but why are we refering to the tvm stack as "engine" rather than just tvm ? Can we rename engine to tvm , engine_version to tvm_version, engine_configuration to tvm_build_configuration or some such ?

Ramana

After looking at the schema for a bit. I feel that it would be helpful to categorize things into two categories:

timestamp, schema, workload, engine, hardware, runtime_ms_mean, runtime_ms_std (I think the runtime_mean_ms and std are important enough for most cases that should be bought into the top-level)
additional config info and statistics that are meant to supplement the setups.

re @u99127 's comment about raw statistics, given that some executions have to be run over multiple iterations to get a meaningful measurement, individual run stats may not be as meaningful and accurate. We can still provide a runtime_raw array which might be useful for future statistics calculation, but I do think providing mean and std would be what most folks will look for.
the term "engine". the schema was also meant to store baseline benchmarks(e.g. TFLite) and allow cross comparisons between tvm results and baselines, that is why we adopt the term engine.

Thanks @tqchen , Ok, now the term "engine" makes sense.
My experience has been different. Some prefer median over mean, others min / max and some other a geometric mean.
regards
Ramana

@u99127 do you lean more towards storing the list of measurements (e.g. over 10 runs) and letting the user implement their aggregation function? Or list multiple ones? Or just agree to standardize towards one (e.g. mean).

@tmoreau89  -  I lean more towards storing the list of raw measurements.

To address @u99127 's concern, we could add an optional runtime_raw field. Note we may not necessarily be able to get accurate measurement of exact runs, as cases like GPU executions needs async execution and average over multiple runs, so the statistics will be wrt to the number runs instead of an iteration, see https://docs.tvm.ai/api/python/module.html?highlight=time_evaluator#tvm.module.Module.time_evaluator

Thanks for everyone's comments! I have made the following amendments in the original post and the pr:

runtime_raw is an optional top-level field
runtime_ms_mean and runtime_ms_std are back at the top level
metrics is a top-level json object that can contain fields like accuracy or compilation_time
runtime_config is a top-level field again

Thanks everyone for great discussion so far, if there is no further comments, we will merge the current proposal by the end of the week

Thanks for everyone's participation. the first version of the benchmark logging format is now merged.

repo: apache/incubator-tvm


text:
[CI] Update actions miniconda

cc @comaniac

Thanks @tqchen @zhiics

authors: comaniac tqchen


text:
[POC][Feature] Introduce the taskId in the spark client side log

What changes were proposed in this pull request?
[Feature] Introduce the taskId in the spark client side log

Why are the changes needed?
When multiple tasks are running in the same executor at the same time, it will be hard to analysis the rss log belonging to the specified task. To solve this, it's better to make log show task id in the rss client codebase.
https://user-images.githubusercontent.com/8609142/189322463-94bfc55b-0082-40aa-828b-88617f43cb28.png

Does this PR introduce any user-facing change?
Yes. Need to update the spark log4j.properties. like this
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}%X{taskId}: %m%n

How was this patch tested?

Manual tests.

PTAL @jerqi POC

Codecov Report

Merging #246 (df3ba98) into master (f1cb43f) will increase coverage by 0.03%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #246      +/-   ##
============================================
+ Coverage     59.31%   59.35%   +0.03%
- Complexity     1345     1346       +1
============================================
  Files           161      161
  Lines          8780     8780
  Branches        828      828
============================================
+ Hits           5208     5211       +3
+ Misses         3304     3302       -2
+ Partials        268      267       -1

Impacted Files
Coverage Δ

...rg/apache/uniffle/storage/common/LocalStorage.java
44.82% <0.00%> (+2.06%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Maybe we just need to change the information that we want to log.

Maybe we just need to change the information that we want to log.

This is not enough general and duplicate params scattered enerywhere

authors: codecov-commenter jerqi zuston

repo: apache/incubator-uniffle


text:
Rebase postgres branch on master

NO MERGE

08:03:05,640 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.13.0:compile (default-compile) on project blob-postgres: Compilation failure
08:03:05,640 [ERROR] /home/jenkins/jenkins-agent/workspace/james_ApacheJames_PR-2398/server/blob/blob-postgres/src/main/java/org/apache/james/blob/postgres/PostgresBlobStoreDAO.java:[157,41] cannot find symbol
08:03:05,640 [ERROR]   symbol:   method from(java.lang.String)
08:03:05,640 [ERROR]   location: variable blobIdFactory of type org.apache.james.blob.api.BlobId.Factory

I will continue this work at pr: https://github.com/apache/james-project/pull/2467

db5f2765db37363c0a252e132941d93be9785f02

authors: Arsnael hungphan227 vttranlina

repo: apache/james-project


text:
DOCTYPE and HTML lang="en" attribute addition

No functional change as this is primarily an accessibility enhancement for screen reader software such as JAWS.   DOCTYPEs added to improve validation.

merged on 2.11.0-M2-git-03
I've just seen I've messed with the incorrect merge button, so your commits are not shown on the master branch, my apologies :-( Your PR is still referenced on the commit history though, and your contribution is listed on the ChangeLog file, so at least public attribution is still preserved.

authors: juanpablo-santos skotfred

repo: apache/jspwiki


text:
KAFKA-17194: Don't create cluster for MetadataQuorumCommandTest#testCommandConfig

Did testCommandConfig a unit test instead of an integration test.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Hi @chia7712
There are failed tests MetadataQuorumCommandTest.testDescribeQuorumReplicationSuccessful in the report: https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-16682/2/testReport/
Locally they pass.
Can you take a look?

https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-16682/3/testReport/org.apache.kafka.tools/MetadataQuorumCommandTest/

Thanks for this patch. I am traveling. Will take a look next Monday 😄

repo: apache/kafka


text:
KAFKA-17248 - KIP 1076 implementation

Implementation of KIP-1076 to allow for adding client application metrics to the KIP-714 framework
Integration and unit tests forthcoming
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Thanks for the comments @apoorvmittal10 - I've addressed your comments.

should we also verify state store metrics (or are they automatically verified as part of task metrics)?

They are covered as part of task metrics, but I expanded the test to explicitly verify state store metrics are correct as well

@mjsax thanks for the second review, I've addressed your comments

@mjsax @apoorvmittal10 comments addressed

@AndrewJSchofield, thanks for the review; I've addressed your comment.

@mjsax

Did not look at test code yet.

I've updated the integration test and I consider it completed at this point

should we extend the test, to not only verify task metrics, but also thread metrics? What about DEBUG and TRACE level?

Sure, but I don't think that should block the PR. Given that TRACE will include all metrics I'd say the end-to-end test should simply extend to set that level to ensure all expected metrics make it through.

@bbejeck Can you rebase this PR to resolve merge conflicts?

@mjsax rebased on trunk - ready for final review

Merged #17021 into trunk

authors: bbejeck mjsax


text:
KAFKA-3999: Record raw size of fetch responses as part of consumer metrics

Currently, only the decompressed size of fetch responses is recorded. This PR adds a sensor to keep track of the raw size as well.

I thought I'd comment, since I've worked in this area of the code as well. LGTM.

@hachikuji, thanks for the feedback. I drafted a short KIP. This metric still sounds useful to me as it differs from the existing Selector metric, as you mentioned.

authors: vahidhashemian wushujames


text:
MINOR: Upgrade zk to 3.5.6

It includes an important fix for people running on k8s:

ZOOKEEPER-3320: Leader election port stop listen when
hostname unresolvable for some time

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Two builds passed, one failed with unrelated flaky test failure:

kafka.admin.ReassignPartitionsClusterTest.shouldPerformMultipleReassignmentOperationsOverVariousTopics

Given that this is a minor dependency bump, I'll go ahead and merge.

Merged to trunk and cherry-picked to 2.4 branch.


text:
KAFKA-10393: messages for fetch snapshot and fetch

Adds the changes to Fetch and FetchSnapshot as describe in KIP-630.
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Thanks for the review @chia7712 and sorry for the delay but I have decided to merge this PR with https://github.com/apache/kafka/pull/9553.
I will take care of your comments in that PR!


text:
MINOR: Include connector name in error message

These log messages aren't triggered very frequently, but when they are it can indicate a serious problem with the connector, and it'd be nice to know exactly which connector is having that problem without having to dig through other log messages and try to correlate the stack trace here with existing connector configs.

@C0urante Thanks for your patch. Merge to trunk!

Thanks @chia7712!


text:
KAFKA-3201: Added rolling upgrade system tests from 0.8 and 0.9 to 0.10

Three main tests:

Setup: Producer (0.8) → Kafka Cluster → Consumer (0.8)
First rolling bounce: Set inter.broker.protocol.version = 0.8 and message.format.version = 0.8
Second rolling bonus, use latest (default) inter.broker.protocol.version and message.format.version
Setup: Producer (0.9) → Kafka Cluster → Consumer (0.9)
First rolling bounce: Set inter.broker.protocol.version = 0.9 and message.format.version = 0.9
Second rolling bonus, use latest (default) inter.broker.protocol.version and message.format.version
Setup: Producer (0.9) → Kafka Cluster → Consumer (0.9)
First rolling bounce: Set inter.broker.protocol.version = 0.9 and message.format.version = 0.9
Second rolling bonus: use inter.broker.protocol.version = 0.10 and message.format.version = 0.9

Plus couple of variations of these tests using old/new consumer and no compression / snappy compression.

@apovzner, who will be reviewing this? @granders and/or @ewencp?

LGTM

authors: apovzner ewencp ijuma


text:
Reduce number of calculations in FSTCompiler

Description

@mikemccand You are welcome. I am a ream human. no an AI. -_-

@mikemccand You are welcome. I am a ream human. no an AI. -_-

Ahh thanks for the reply.  Could you describe how you got involved / interested in open source development?  How about specifically Apache Lucene?

I had use elasticsearch in our sofeware.Elasticsearch is so faster to search data. I know elasticsearch use lucence to make it faster.I am very interested in this and enjoy read lucence source code.
---Original---
From: "Michael @.>
Date: Fri, Sep 20, 2024 07:26 AM
To: @.>;
Cc: "Jason @.@.>;
Subject: Re: [apache/lucene] Reduce number of calculations in FSTCompiler (PR#13788)
@mikemccand You are welcome. I am a ream human. no an AI. -_-
Ahh thanks for the reply.  Could you describe how you got involved / interested in open source development?  How about specifically Apache Lucene?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

authors: mikemccand mrhbj

repo: apache/lucene


text:
porting/FreeRTOS: fix IRQ priority

FreeRTOS assumes a max irq priority for IRQs that are allowed to do system calls. This is set to 5 by default, so we use the highest allowed for the Radio.

Sure, changed as you suggested.

repo: apache/mynewt-nimble


text:
Inconsistent with the expected result when calling cmake module

https://github.com/apache/nuttx/blob/bc181f3f9c98ac527df84cdf0af449c6454f3367/cmake/nuttx_add_library.cmake#L64-L80
I'm migrating some apps libraries to cmake build system. when calling the nuttx_add_user_library module function , like this
  # In lib-need-to-archive-in-apps/CMakeLists.txt
  nuttx_add_user_library(lib-need-to-archive-in-apps)

  target_sources(lib-need-to-archive-in-apps ${SRCS} )
  target_compile_options(lib-need-to-archive-in-apps ${FLAGS})
  target_include_directories(lib-need-to-archive-in-apps ${INCDIR})

what I expect is that all output files are archived in libapps, as the description by this function.
But it turns out that it is neither archived in libapps nor involved in the linking of nuttx.
I guess it may be caused by the inheritance of INTERFACE, it is not correctly assigned to target apps.
should we change INTERFACE in line 76 and line 79 to  PRIVATE ？

Yes, seems make sense, INTERFACE just appended to the link interface and are not used for linking to target, you can change it to PRIVATE if you want to use this interface to link with libapps

Yes, seems make sense, INTERFACE just appended to the link interface and are not used for linking to target, you can change it to PRIVATE if you want to use this interface to link with libapps

thanks a lot🙏 , I'm planning  to fix the nuttx_add_user_library  function to the expected behavior, which is  archived to libapps.
and also planning to enhance  those independent static libraries  that generated by nuttx_add_libray(foo STATIC), make those static libraries have a unified export path target.

repo: apache/nuttx


text:
merge some nrf52 configurations into one

Summary

boards/arm/nrf52/nrf52832-dk: replace some configs with jumbo config
boards/arm/nrf52/nrf52840-dk: replace some configs with jumbo config

There is no reason in keeping them as separate configurations, we can easily fit them in single jumbo configuration
Impact
remove some configurations and replace them with one jumbo config
Testing
jumbo tested on nrf52832-dk and nrf52840-dk

[Experimental Bot, please feedback here]
Fill In The Commit Message: This PR contains a Commit with an Empty Commit Message. Please fill in the Commit Message with the PR Summary.
The PR summary is too short and doesn't explain what the change actually does or why it's beneficial.  Here's a breakdown of what needs improvement and how to address it:
Summary:

Insufficient Detail: Simply stating "replace some configs with jumbo config" doesn't provide enough context.

What configs are being replaced? Be specific about the configurations being removed.
Why is this change beneficial? Explain the advantages of using a jumbo config in this case (e.g., reduces redundancy, simplifies maintenance, etc.).

Impact:

Too Vague:  "remove some configurations and replace them with one jumbo config" is just a restatement of the summary. You need to be much more specific about the potential impact:

Build: Will this affect build times? Configuration options available?
Hardware: Are there any boards or peripherals that might be impacted by this consolidation?
Documentation:  Does any documentation need to be updated to reflect the new configuration structure?

Testing:

Lack of Specificity: "jumbo tested on nrf52832-dk and nrf52840-dk" is not sufficient.

What tests were run? Provide specific examples of tests you performed to validate the jumbo configuration on both boards.
Show Evidence (Logs):  Include relevant snippets from the testing logs (before and after the change) to demonstrate that the functionality is equivalent or improved.

Example of an Improved Summary:

This PR consolidates several configurations for the nrf52832-dk and nrf52840-dk boards into a single, unified "jumbo" configuration. The previous configurations were largely redundant, leading to unnecessary code duplication and maintenance overhead. This change simplifies the build system, reduces code size, and makes it easier for developers to configure these boards.

Key Takeaways:

Be Thorough:  Provide sufficient detail in all sections to allow reviewers to fully understand the purpose, impact, and validation of your changes.
Provide Evidence:  Back up your testing claims with concrete examples and log snippets.
Focus on Clarity:  Write your PR description in a clear and concise manner that is easy for others to follow.

FYI the Build Rules arch.yml should have classified this as a Simple Arm32 PR, instead of a Complex PR (because of the doc). This patch will fix it: https://github.com/apache/nuttx/pull/14282

authors: lupyuen nuttxpr raiden00pl


text:
Allow decision about action result inclusion in logs on a per call basis

Add the possibility to decide whether results should be included in the action log on a per-call basis for each activation individually.
Description
Action results may contain sensible data that should not be logged. There is a system configuration flag (writeResultToFile) that can be used to switch on or off the result inclusion in logs. But this is a global switch that holds for all activations. In our case we want to able to decide per activation whether or not the result should be included in the log or not. In our special case we want results to be included in case of errors (in other words, our predicate function for the decision is: 'Does the result contain an error field?'). But also other decision logic may be applicable.
My changes affect the following components

Controller (for sequences)
Invoker

Types of changes
Small, low risk change: add one optional, boolean parameter to the method activationToFileExtended of org.apache.openwhisk.core.database.ActivationFileStorage that determines the decision whether or not the result of the current activation should be included in the log. The default value of the new parameter is the value of the system configuration flag writeResultToFile. The method activationToFile is unchanged and works as before. The private method transcribeActivation now takes the flag from activationToFileExtended.

Codecov Report

Merging #4604 into master will decrease coverage by 5.77%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #4604      +/-   ##
==========================================
- Coverage   84.55%   78.77%   -5.78%
==========================================
  Files         183      183
  Lines        8305     8305
  Branches      581      581
==========================================
- Hits         7022     6542     -480
- Misses       1283     1763     +480

Impacted Files
Coverage Δ

...penwhisk/core/database/ActivationFileStorage.scala
95.34% <100%> (ø)
:arrow_up:

...core/database/cosmosdb/RxObservableImplicits.scala
0% <0%> (-100%)
:arrow_down:

...ore/database/cosmosdb/cache/CacheInvalidator.scala
0% <0%> (-100%)
:arrow_down:

...core/database/cosmosdb/CosmosDBArtifactStore.scala
0% <0%> (-95.89%)
:arrow_down:

...tabase/cosmosdb/cache/CacheInvalidatorConfig.scala
0% <0%> (-94.74%)
:arrow_down:

...sk/core/database/cosmosdb/CosmosDBViewMapper.scala
0% <0%> (-92.6%)
:arrow_down:

...e/database/cosmosdb/cache/ChangeFeedListener.scala
0% <0%> (-86.67%)
:arrow_down:

...e/database/cosmosdb/cache/KafkaEventProducer.scala
0% <0%> (-76.48%)
:arrow_down:

...whisk/core/database/cosmosdb/CosmosDBSupport.scala
0% <0%> (-74.08%)
:arrow_down:

...a/org/apache/openwhisk/common/ExecutorCloser.scala
0% <0%> (-66.67%)
:arrow_down:

... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ee07500...a099dc7. Read the comment docs.

Should this be tested in ArtifactWithFileStorageActivationStoreTests?

Should this be tested in ArtifactWithFileStorageActivationStoreTests?

Yes, I added tests for the two states of the flag. Individual sets of additional fields for logs and activation are also tested. The default settings for the flag are indirectly tested by the existing previous tests (activationToFile calls activationToFileExtended).

authors: codecov-io ruediger-maass tysonnorris

repo: apache/openwhisk