text:
Add junit5 test code for module [linkis-storage]

Add junit5 test code for module [linkis-storage]

Module: linkis-storage
Location:linkis-commons/linkis-storage
Task status:✗(unfinished)/ ✓(finished)
Test code guide:https://linkis.apache.org/zh-CN/community/how-to-write-unit-test-code/
Test code example: https://github.com/apache/incubator-linkis/tree/master/linkis-public-enhancements/linkis-publicservice/linkis-jobhistory/src/test

模块名: linkis-storage
代码位置:linkis-commons/linkis-storage
任务状态:✗(unfinished)/ ✓(finished)
单测编写指引:https://linkis.apache.org/zh-CN/community/how-to-write-unit-test-code/
参考样例: https://github.com/apache/incubator-linkis/tree/master/linkis-public-enhancements/linkis-publicservice/linkis-jobhistory/src/test

Task status
Class
Type
Level of Difficulty
Note

✗
org.apache.linkis.storage.excel.*
utils
normal

✗
org.apache.linkis.storage.exception.*
Exceptions
normal

✗
org.apache.linkis.storage.factory.impl.BuildHDFSFileSystem
Factories
normal

✗
org.apache.linkis.storage.factory.impl.BuildLocalFileSystem
Factories
normal

✗
org.apache.linkis.storage.fs.impl.HDFSFileSystem
Factories
normal

✗
org.apache.linkis.storage.fs.impl.LocalFileSystem
Factories
normal

✗
org.apache.linkis.storage.pipeline.PipelineReader
Factories
normal

✗
org.apache.linkis.storage.pipeline.PipelineWriter
Factories
normal

✗
org.apache.linkis.storage.csv.*
Cvs
normal

✗
org.apache.linkis.storage.domain.DataType
Domain
normal

✗
org.apache.linkis.storage.domain.Dolphin
Domain
normal

✗
org.apache.linkis.storage.domain.MethodEntity
Domain
normal

✗
org.apache.linkis.storage.excel.*
Utils
normal
scala

✗
org.apache.linkis.storage.io.IOClient
Utils
normal

✗
org.apache.linkis.storage.io.IOMethodInterceptorCreator
Utils
normal

✗
org.apache.linkis.storage.resultset.html.*
ResultSet
normal

✗
org.apache.linkis.storage.resultset.io.*
ResultSet
normal

✗
org.apache.linkis.storage.resultset.picture.*
ResultSet
normal

✗
org.apache.linkis.storage.resultset.table.*
ResultSet
normal

✗
org.apache.linkis.storage.resultset.txt.*
ResultSet
normal

✗
org.apache.linkis.storage.resultset.*
ResultSet
normal

✗
org.apache.linkis.storage.script.compaction.*
Script
normal

✗
org.apache.linkis.storage.script.parser.*
Script
normal

✗
org.apache.linkis.storage.script.reader.*
Script
normal

✗
org.apache.linkis.storage.script.writer.*
Script
normal

✗
org.apache.linkis.storage.script.*
Script
normal

✗
org.apache.linkis.storage.source.*
Source
normal

✗
org.apache.linkis.storage.utils.*
Utils
normal

✗
org.apache.linkis.storage.FSFactory
Utils
normal

✗
org.apache.linkis.storage.LineMetaData
Utils
normal

✗
org.apache.linkis.storage.LineRecord
Utils
normal

Completed in dev-1.3.1

repo: apache/incubator-linkis


text:
Minor. Fix deprecated conf warning log issue

./livy-sshao-server.out.5:17/05/08 16:50:44 WARN LivyConf: The configuration key livy.spark.deployMode has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.spark.deploy-mode instead.
./livy-sshao-server.out.5:17/05/08 16:50:45 WARN LivyConf: The configuration key livy.spark.scalaVersion has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.spark.scala-version instead.
./livy-sshao-server.out.5:17/05/08 16:51:04 WARN RSCConf: The configuration key livy.rsc.driver_class has been deprecated as of Livy 0.4 and may be removed in the future. Please use the new key livy.rsc.driver-class instead.

This log is incorrect even if we use new configuration key. This is mainly because the logic in logDeprecationWarning to check alternative configurations is not correct.

Codecov Report

Merging #13 into master will increase coverage by 0.18%.
The diff coverage is 94.73%.

@@             Coverage Diff              @@
##             master      #13      +/-   ##
============================================
+ Coverage     70.47%   70.65%   +0.18%
- Complexity      729      733       +4
============================================
  Files            96       96
  Lines          5161     5177      +16
  Branches        779      781       +2
============================================
+ Hits           3637     3658      +21
+ Misses         1006      996      -10
- Partials        518      523       +5

Impacted Files
Coverage Δ
Complexity Δ

...java/org/apache/livy/client/common/ClientConf.java
99% <94.73%> (-1%)
44 <6> (+3)

rsc/src/main/java/org/apache/livy/rsc/RSCConf.java
86.73% <0%> (-1.03%)
7% <0%> (ø)

...in/java/org/apache/livy/rsc/driver/JobWrapper.java
80.64% <0%> (ø)
8% <0%> (+1%)
:arrow_up:

rsc/src/main/java/org/apache/livy/rsc/rpc/Rpc.java
78.61% <0%> (+0.62%)
12% <0%> (ø)
:arrow_down:

...ain/java/org/apache/livy/rsc/driver/RSCDriver.java
78.44% <0%> (+0.86%)
40% <0%> (-1%)
:arrow_down:

rsc/src/main/java/org/apache/livy/rsc/Utils.java
85.71% <0%> (+2.38%)
16% <0%> (ø)
:arrow_down:

...in/java/org/apache/livy/rsc/rpc/RpcDispatcher.java
69.56% <0%> (+3.26%)
20% <0%> (+1%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 412ccc8...e3a584d. Read the comment docs.

Thanks @ajbozarth , merging to master.

source: gharchive/pull-request

authors: codecov-io jerryshao

repo: apache/incubator-livy

license_type: permissive

license_source: github-api


text:
Something wrong of form .rec of my own data

I try to form .rec and .idx of my own data.
First, run im2rec.py to generate list :
python im2rec.py --list --recursive --train-ratio 1 [list folder] [images folder]

Second, run im2rec.py to generate .rec and .idx.
python im2rec.py --num-thread 4 [list folder] [images folder]

But when I read index,
imgrec = mx.recordio.MXIndexedRecordIO(args.idx_path, args.bin_path, 'r')
s = imgrec.read_idx(0)
header, _ = mx.recordio.unpack(s)

The "header" has no label value.

Where is the problem?
Thanks.

@mxnet-label-bot [Question]

Thanks a lot for reporting this issue. I tried it out and you are right, that the header does not have the correct values.  Using imgrec.read instead of imgrec.read_idx  seems to solve this problem.

Thanks a lot for reporting this issue. Is every label in the file incorrect or just a few?

source: gharchive/issue

authors: NRauschmayr frankfliu larsonwu0220

repo: apache/incubator-mxnet


text:
OSError: libopenblas.so.0: cannot open shared object file: No such file or directory in mxnet 1.8.0

Description
mxnet 1.8.0 emits the following error when running import mxnet:
OSError: libopenblas.so.0: cannot open shared object file: No such file or directory

Error Message
(generated from the dockerfile attached in the to-reproduce section)
+ Step 1/3 : FROM python:3.7
 ---> 7fefbebd95b5
+ Step 2/3 : RUN pip install mxnet
 ---> Running in fc634966f9aa
Collecting mxnet
  Downloading mxnet-1.8.0-py2.py3-none-manylinux2014_x86_64.whl (38.7 MB)
Collecting graphviz<0.9.0,>=0.8.1
  Downloading graphviz-0.8.4-py2.py3-none-any.whl (16 kB)
Collecting numpy<2.0.0,>1.16.0
  Downloading numpy-1.20.1-cp37-cp37m-manylinux2010_x86_64.whl (15.3 MB)
Collecting requests<3,>=2.20.0
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
Collecting certifi>=2017.4.17
  Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
Installing collected packages: urllib3, idna, chardet, certifi, requests, numpy, graphviz, mxnet
+ Successfully installed certifi-2020.12.5 chardet-4.0.0 graphviz-0.8.4 idna-2.10 mxnet-1.8.0 numpy-1.20.1 requests-2.25.1 urllib3-1.26.4
WARNING: You are using pip version 20.3.1; however, version 21.0.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
Removing intermediate container fc634966f9aa
 ---> b1c12d9f4376
+ Step 3/3 : RUN python -c "import mxnet"
 ---> Running in 8362fc58b280
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/mxnet/__init__.py", line 23, in <module>
    from .context import Context, current_context, cpu, gpu, cpu_pinned
  File "/usr/local/lib/python3.7/site-packages/mxnet/context.py", line 23, in <module>
    from .base import classproperty, with_metaclass, _MXClassPropertyMetaClass
  File "/usr/local/lib/python3.7/site-packages/mxnet/base.py", line 351, in <module>
    _LIB = _load_lib()
  File "/usr/local/lib/python3.7/site-packages/mxnet/base.py", line 342, in _load_lib
    lib = ctypes.CDLL(lib_path[0], ctypes.RTLD_LOCAL)
  File "/usr/local/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
+ OSError: libopenblas.so.0: cannot open shared object file: No such file or directory
The command '/bin/sh -c python -c "import mxnet"' returned a non-zero code: 1

(key lines are colored green)
To Reproduce
Steps to reproduce

Prepare the following dockerfile:

FROM python:3.7

RUN pip install mxnet
RUN python -c "import mxnet"

Then, run docker build .

What have you tried to solve it?

Environment
We recommend using our script for collecting the diagnostic information with the following command
curl --retry 10 -s https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py | python3

Environment Information
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       39 bits physical, 48 bits virtual
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           8
Vendor ID:           GenuineIntel
CPU family:          6
Model:               158
Model name:          Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Stepping:            13
CPU MHz:             2400.000
BogoMIPS:            4800.00
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            16384K
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht pbe syscall nx pdpe1gb lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq dtes64 ds_cpl ssse3 sdbg fma cx16 xtpr pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch pti fsgsbase bmi1 avx2 bmi2 erms xsaveopt arat
----------Python Info----------
Version      : 3.7.9
Compiler     : GCC 8.3.0
Build        : ('default', 'Nov 18 2020 14:10:47')
Arch         : ('64bit', 'ELF')
------------Pip Info-----------
Version      : 20.3.1
Directory    : /usr/local/lib/python3.7/site-packages/pip
----------MXNet Info-----------
No MXNet installed.
----------System Info----------
Platform     : Linux-4.19.76-linuxkit-x86_64-with-debian-10.6
system       : Linux
node         : 9bcb86c4d6cf
release      : 4.19.76-linuxkit
version      : #1 SMP Tue May 26 11:42:35 UTC 2020
----------Hardware Info----------
machine      : x86_64
processor    :
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0155 sec, LOAD: 0.9034 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.1894 sec, LOAD: 0.2668 sec.
Error open Gluon Tutorial(cn): https://zh.gluon.ai, <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1091)>, DNS finished in 0.3720698356628418 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0193 sec, LOAD: 0.5068 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0474 sec, LOAD: 0.5751 sec.
Error open Conda: https://repo.continuum.io/pkgs/free/, HTTP Error 403: Forbidden, DNS finished in 0.01907634735107422 sec.
----------Environment----------
Removing intermediate container 9bcb86c4d6cf

I think this is due to a change in CD that openblas is no longer statically linked in libmxnet. For now you can install openblas separately.
cc @mseth10 @leezu

@harupy it's unclear what you are doing. You need to provide more details.

We observe the same issue when trying to use pip-installed MXNet 1.8 (ubuntu, CPU): https://github.com/awslabs/sockeye/runs/2161521442?check_suite_focus=true

@mseth10 @access2rohit please take a look why the CD didn't package the libopenblas.so
tools/pip/setup.py includes instructions for copying libopenblas.so in v1.8.x, v1.x, and master. But apparently that didn't work in v1.8.x, potentially due to some missing library file names in the jenkins files?
v1.8.x https://github.com/apache/incubator-mxnet/blob/a0535ddfb0246f53f7b851baf861fc06d3ff48c3/tools/pip/setup.py#L170-L172
v1.x https://github.com/apache/incubator-mxnet/blob/cfa1c890a7ecb8b5e29ff4e90d6784141f09c4cd/tools/pip/setup.py#L164-L166
master https://github.com/apache/incubator-mxnet/blob/4d706e8c19b3354878eda9467b149c0ce1fd6d47/tools/pip/setup.py#L165-L167
However, I noted that v1.8.x also attempts to copy libquadmath, which MUST NOT happen due to license of libquadmath. That should be fixed. Fortunately that code didn't run due to the current bug.

The problem is that https://github.com/apache/incubator-mxnet/pull/19514 is missing on v1.8.x

My container suddenly started failing to build and upon looking, this was the error. I started using previous version which is 1.7.0.post2 and works perfectly

@praneethkv we are working on patching the wheels to fix the problem

authors: fhieber harupy leezu praneethkv szha


text:
Improve row_sparse tutorial

Description
@haojin2
Checklist
Essentials

[ ] Passed code style checking (make lint)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
[ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

LGTM!

can you associate this with a jira ?

I don’t think Tiny PR like this needs one but yeah I can create a JIRA item..


text:
Add dtype visualization to plot_network

Description
Add possibility to print type information alongside shape in mx.vis.plot_network.
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[x] Changes are complete (i.e. I finished coding on this PR)
[x] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
[x] Code is well-documented:
For user-facing API changes, API doc string has been updated.
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

@mxnet-label-bot add [pr-awaiting-review, Visualization]

@szha Does anything else need to be done with this PR?

Merged. Thanks for your contribution!

authors: ptrendx vandanavk wkcn


text:
Silence excessive mkldnn logging output on tests.

Description
Silenced excessive logging output:
http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/mxnet-validation/pipelines/unix-cpu/branches/PR-14940/runs/1/nodes/283/steps/749/log/?start=0
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[x] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[x] Changes are complete (i.e. I finished coding on this PR)
[x] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[x] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

@pengzhao-intel I saw this excessive log output and had a 3h timeout on tests, wanted to see if the PR validation time goes down because of this, probably not related.

@larroy Sorry just notice that you're trying to save time of CI? I remember MXNET_MKLDNN_DEBUG is turned on explicitly in CI. So my suggestion might not help for this case.

@larroy Sorry just notice that you're trying to save time of CI? I remember MXNET_MKLDNN_DEBUG is turned on explicitly in CI. So my suggestion might not help for this case.

How about turning it off because we have enough test cases in the CI for MKLDNN now?

Thank you for your improvement.  Merge now.

authors: TaoLv larroy pengzhao-intel


text:
fix row_sparse demo tutorials doc

Description
row sparse demo tutorials, it has a mistake that x should be 12 and w should be 23.

The CI is a bit brittle. Do you mind sync with master and trigger the CI again?

Ok! I will try @eric-haibin-lin

why is there no change?

Looks like this is already merged in. Closing

authors: burness eric-haibin-lin piiswrong


text:
[NETBEANS-54] Module Review defaults

no external library

checked Rat report; unrecognized license headers manually changed

skimmed through the module, did not notice additional problems

Merged into master with 1 positive review. Thank you for reviewing.

repo: apache/incubator-netbeans


text:
net: tcp: Fix compile error in tcp.h

During build testing for spresense with the latest master, I encountered the following compile error.
tcp/tcp_netpoll.c: In function 'tcp_pollsetup':
./tcp/tcp.h:1191:42: error: expected ')' before ';' token
 #  define tcp_backlogavailable(c) (false);
                                          ^
tcp/tcp_netpoll.c:308:40: note: in expansion of macro 'tcp_backlogavailable'
   if (!IOB_QEMPTY(&conn->readahead) || tcp_backlogavailable(conn))
                                        ^~~~~~~~~~~~~~~~~~~~
Makefile:102: recipe for target 'tcp_netpoll.o' failed

Actually, the line 1191 in tcp/tcp.h was added in 2014-07-06 but not used so far.
And the line 308 in tcp/tcp_netpoll.c was not called on my environment before merging the following commit
commit 90c52e6f8f7efce97ac718c0f98addc13ec880d2
Author: Xiang Xiao <xiaoxiang@xiaomi.com>
Date:   Tue Dec 31 09:26:14 2019 -0600

The fix in this PR is very trivial and I think @xiaoxiang781216 enabled CONFIG_NET_TCPBACKLOG on his environments because he did not encounter the error.

BTW: CONFIG_NET_TCPBACKLOG probably should always be enabled.  Otherwise, you will miss connection requests.  That configuration is a candidate to be removed and just have connection backlog support enabled at all times.

BTW: CONFIG_NET_TCPBACKLOG probably should always be enabled. Otherwise, you will miss connection requests. That configuration is a candidate to be removed and just have connection backlog support enabled at all times.

@patacongo Thanks for the comment. I will modify our defconfigs in separate PR later.

@masayuki2009 Removing support for CONFIG_NET_TCPBACKLOG altogether might be a better solution?  Anyone else have an opinion to the contrary?
The TCP backlog was conditioned originally only to support support super tiny networking.  But with all of the growth in networking, I think super-tiny networking is not really an option and is certainly not advised in this case since the consequences of disabling backlog are so bad.  NuttX now really only supports "small" networking, not super-tiny networking.

@masayuki2009 Removing support for CONFIG_NET_TCPBACKLOG altogether might be a better solution? Anyone else have an opinion to the contrary?
The TCP backlog was conditioned originally only to support super-tiny networking. But with all of the growth in networking, I think super-tiny networking is not really an option and is certainly not advised in this case since the consequences of disabling backlog are so severe. NuttX now really only supports "small" networking, not super-tiny networking.

@patacongo I think that's a good idea, because we will not use NuttX networking with super-tiny environment. If nobody has an objection on it, please remove support for CONFIG_NET_TCPBACKLOG.

repo: apache/incubator-nuttx


text:
Refactor Whisk Deploy errors into new package to avoid cyclic dependencies

currently, wskdeployerror.go (and it unit test file) are in "utils" package; however, if we want to do better unit testing we need to be able to test errors against manifests and the YAML parser.  This would mean importing "parsers" into "utils" which leads to cyclic dependency errors in GoLang.
To overcome this cyclic error, we must refactor the error modules into a new "wskdeplyerror" package.

Fixed with wskderrors at https://github.com/apache/incubator-openwhisk-wskdeploy/blob/master/wskderrors/wskdeployerror.go

authors: mrutkows pritidesai

repo: apache/incubator-openwhisk-wskdeploy


text:
support config for jackson buffer recycler pool

the buffer recycler is an important performance feature in Jackson
Jackson 2.17 also changes the default pool implementation and this has proved an issue - see https://github.com/FasterXML/jackson-module-scala/issues/672
my plan for Pekko is to keep the existing ThreadLocal implementation as the default even if Jackson has a different default

It looks good.
Is it necessary to supplement the documentation?
https://pekko.apache.org/docs/pekko/current/serialization-jackson.html#additional-features

authors: laglangyue pjfanning

repo: apache/incubator-pekko


text:
[REEF-624] Replace all use of DriverBridgeConfiguration with DriverConfiguration

This addressed the issue by

replacing all usages of DriverBridgeConfiguration in examples and tests
with DriverConfiguration
adding IObservable and necessary methods to test drivers

JIRA:
REEF-624
Pull Request:
Closes #

I'll test and merge

There are test failures:
BroadcastReduceTest.TestBroadcastAndReduceOnLocalRuntime:
Cleaning up test.
Org.Apache.REEF.Driver.Bridge.ClrHandlerHelper Start: 0 : 2015-08-31T17:42:23.2005420-07:00 0016
START: 2015-08-31 17:42:23 ClrHandlerHelper::CopyDllsToAppDirectory
Org.Apache.REEF.Driver.Bridge.ClrHandlerHelper Stop: 0 : 2015-08-31T17:42:23.2070308-07:00 0016
EXIT: 2015-08-31 17:42:23 ClrHandlerHelper::CopyDllsToAppDirectory. Duration: [00:00:00.0062184].
Org.Apache.REEF.Driver.Bridge.ClrHandlerHelper Start: 0 : 2015-08-31T17:42:23.2100389-07:00 0016
START: 2015-08-31 17:42:23 ClrHandlerHelper::GetAssembliesListForReefDriverApp
Org.Apache.REEF.Driver.Bridge.ClrHandlerHelper Stop: 0 : 2015-08-31T17:42:23.3015332-07:00 0016
EXIT: 2015-08-31 17:42:23 ClrHandlerHelper::GetAssembliesListForReefDriverApp. Duration: [00:00:00.0916312].
GetLogFile, driverContainerDirectory:C:\src\reef\lang\cs\TestResults\Deploy_mweimer 2015-08-31 17_42_20\Out\REEF_LOCAL_RUNTIME\ReefClrBridge-1441068147372\driver
Lines read from log file : 181
Cleaning up test.
Assert.IsTrue failed.
   at Org.Apache.REEF.Tests.Functional.ReefFunctionalTest.ValidateSuccessForLocalRuntime(Int32 numberOfEvaluatorsToClose, String testFolder) in ReefFunctionalTest.cs: line 161
Assert.IsTrue failed.
   at Org.Apache.REEF.Tests.Functional.ReefFunctionalTest.ValidateSuccessForLocalRuntime(Int32 numberOfEvaluatorsToClose, String testFolder) in ReefFunctionalTest.cs: line 161
Assert.IsTrue failed.
   at Org.Apache.REEF.Tests.Functional.ReefFunctionalTest.ValidateSuccessForLocalRuntime(Int32 numberOfEvaluatorsToClose, String testFolder) in ReefFunctionalTest.cs: line 161

PipelinedBroadcastReduceTest.TestPipelinedBroadcastAndReduceOnLocalRuntime:
Cleaning up test.
GetLogFile, driverContainerDirectory:C:\src\reef\lang\cs\TestResults\Deploy_mweimer 2015-08-31 17_42_20\Out\REEF_LOCAL_RUNTIME\ReefClrBridge-1441068245737\driver
Lines read from log file : 183
Cleaning up test.
Assert.IsTrue failed.
   at Org.Apache.REEF.Tests.Functional.ReefFunctionalTest.ValidateSuccessForLocalRuntime(Int32 numberOfEvaluatorsToClose, String testFolder) in ReefFunctionalTest.cs: line 161

ScatterReduceTest.TestScatterAndReduceOnLocalRuntime:
Cleaning up test.
GetLogFile, driverContainerDirectory:C:\src\reef\lang\cs\TestResults\Deploy_mweimer 2015-08-31 17_42_20\Out\REEF_LOCAL_RUNTIME\ReefClrBridge-1441068234174\driver
Lines read from log file : 181
Cleaning up test.
Assert.IsTrue failed.
   at Org.Apache.REEF.Tests.Functional.ReefFunctionalTest.ValidateSuccessForLocalRuntime(Int32 numberOfEvaluatorsToClose, String testFolder) in ReefFunctionalTest.cs: line 161

TestTaskMessage.TestSendTaskMessage():
Cleaning up test.
Running test edf0b053. If failed AND log uploaded is enabled, log can be find in 2015-08-31\edf0b053
GetLogFile, driverContainerDirectory:C:\src\reef\lang\cs\TestResults\Deploy_mweimer 2015-08-31 17_42_20\Out\REEF_LOCAL_RUNTIME\ReefClrBridge-1441068183300\driver
Lines read from log file : 165
Cleaning up test.
Assert.IsTrue failed.
   at Org.Apache.REEF.Tests.Functional.ReefFunctionalTest.ValidateSuccessForLocalRuntime(Int32 numberOfEvaluatorsToClose, String testFolder) in ReefFunctionalTest.cs: line 161
Assert.IsTrue failed.
   at Org.Apache.REEF.Tests.Functional.ReefFunctionalTest.ValidateSuccessForLocalRuntime(Int32 numberOfEvaluatorsToClose, String testFolder) in ReefFunctionalTest.cs: line 161

This might be a problem with our master branch, not just with my change. I'm trying to run tests on various recent commits, and I'm getting these test failures starting with https://github.com/apache/incubator-reef/commit/e374351a957c437f29fe9b61f3cf9f703e40ee33 (and not getting them in earlier commits). Could you please double-check?

I've fixed the tests

@tcNickolas Thanks, I will test again.

authors: markusweimer tcNickolas

repo: apache/incubator-reef


text:
【功能对齐】梳理 Seata 1.6-2.x 的 seata Java 和 seata-go 待对齐的工作

What would you like to be added:
https://github.com/apache/incubator-seata/issues?q=label%3Amultilingual+is%3Aclosed 梳理了1.6-2.x(最新开发分支)的所有feature和部分需要同步的optimize，待梳理成 seat-go 中需要支持的工作
Why is this needed:

已完成梳理

repo: apache/incubator-seata-go


text:
optimize:compatible with integration-tx-api module and spring module

[x] I have registered the PR changes.

Ⅰ. Describe what this PR did
Ⅱ. Does this pull request fix one issue?

fixes #6334
fixes #6335
Ⅲ. Why don't you add test cases (unit test/integration test)?
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

some checkstyle failed.

ok

Codecov Report
Attention: 33 lines in your changes are missing coverage. Please review.

Comparison is base (74785d2) 51.95% compared to head (45b667b) 51.21%.
Report is 6 commits behind head on 2.x.

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x    #6342      +/-   ##
============================================
- Coverage     51.95%   51.21%   -0.75%
+ Complexity     5171     5117      -54
============================================
  Files           918      921       +3
  Lines         32039    32166     +127
  Branches       3866     3874       +8
============================================
- Hits          16647    16474     -173
- Misses        13768    14102     +334
+ Partials       1624     1590      -34

Files
Coverage Δ

...in/java/org/apache/seata/common/DefaultValues.java
0.00% <ø> (ø)

...che/seata/common/exception/JsonParseException.java
100.00% <ø> (ø)

.../org/apache/seata/common/metadata/ClusterRole.java
100.00% <ø> (ø)

...ava/org/apache/seata/common/metadata/Metadata.java
100.00% <ø> (ø)

...apache/seata/common/metadata/MetadataResponse.java
80.00% <ø> (ø)

...in/java/org/apache/seata/common/metadata/Node.java
100.00% <ø> (ø)

...java/org/apache/seata/common/util/ConfigTools.java
100.00% <ø> (ø)

...ava/org/apache/seata/common/util/DurationUtil.java
83.33% <ø> (ø)

...a/org/apache/seata/common/util/HttpClientUtil.java
52.45% <ø> (+4.91%)
:arrow_up:

...main/java/org/apache/seata/common/util/IOUtil.java
70.00% <ø> (ø)

... and 91 more

... and 165 files with indirect coverage changes

Done. @slievrly

authors: codecov-commenter xingfudeshi

repo: apache/incubator-seata


text:
Split integrated test cases to a new project

[x] Create new repo named sharding-sphere-integrated-test
[ ] Move JDBC integrated test cases to sharding-sphere-integrated-test
[ ] Add MySQL & PostgreSQL docker image for integrated test cases
[ ] Comb JDBC test cases

[ ] Comb JDBC test cases for sharding-rule(sharding, masterslave) with raw bootstrap
[ ] Comb JDBC test cases for other bootstrap(spring namespace, springboot)
[ ] Comb JDBC test cases for database pool(DBCP, HikariCP, C3P0)
[ ] Comb JDBC test cases for ORM(Spring JDBC Template, JPA, Mybatis, Hibernate)

[ ] Comb Proxy test cases

[ ] Add Sharding-Proxy docker image for integrated test cases
[ ] Comb Proxy test cases for sharding-rule(sharding, masterslave) with raw bootstrap
[ ] Comb Proxy test cases for other bootstrap(spring namespace, springboot)
[ ] Comb Proxy test cases for database pool(DBCP, HikariCP, C3P0)
[ ] Comb Proxy test cases for ORM(Spring JDBC Template, JPA, Mybatis, Hibernate)

[ ] Comb Orchestration test cases

[ ] Add Zookeeper docker image for integrated test cases
[ ] Add Etcd docker image for integrated test cases
[ ] Comb Orchestration test cases for jdbc and proxy

expired

repo: apache/incubator-shardingsphere

license_source: bigquery


text:
apm-spring-annotation-plugin grpc exception

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
[ ] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?
when not use everything is ok. After using this plugin apm-spring-annotation-plugin-5.0.0-RC-SNAPSHOT.jar,  exception.

Bug

Which version of SkyWalking, OS and JRE?
SkyWalking: 5.0.0.RC
CentOS 7

Which company or project?

What happen?
After using this plugin apm-spring-annotation-plugin-5.0.0-RC-SNAPSHOT.jar:

ERROR 2018-08-06 15:39:25 TraceSegmentServiceClient :  Send UpstreamSegment to collector fail with a grpc internal exception.
org.apache.skywalking.apm.dependencies.io.grpc.StatusRuntimeException: UNKNOWN
        at org.apache.skywalking.apm.dependencies.io.grpc.Status.asRuntimeException(Status.java:526)
        at org.apache.skywalking.apm.dependencies.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:419)
        at org.apache.skywalking.apm.dependencies.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41)
        at org.apache.skywalking.apm.dependencies.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
        at org.apache.skywalking.apm.dependencies.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41)
        at org.apache.skywalking.apm.dependencies.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:391)
        at org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:475)
        at org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:557)
        at org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:478)
        at org.apache.skywalking.apm.dependencies.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:590)
        at org.apache.skywalking.apm.dependencies.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at org.apache.skywalking.apm.dependencies.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Requirement or improvement

Please describe about your requirements or improvement suggestions.

This is nothing about the apm-spring-annotation-plugin. This exception means agent can't connect or uplink data to backend/collector.

may be apm-spring-annotation-plugin  create much more spans. I have tried it many times，remove this  plugin everything is ok,    after adding  exceptions will occur

may be apm-spring-annotation-plugin create much more spans. I have tried it many times，remove this plugin everything is ok, after adding exceptions will occur

No. I am pretty sure, this error is nothing about the plugin. If spans trigger something wrong, should have collector side's log or memory related error.
gRPC transports bytes to backend, even don't know the span concept.

adding this plugin there is another exception when  start  app with agent:
WARN  TracingContext :  More than 300 spans required to create java.lang.RuntimeException: Shadow tracing context. Thread dump         at org.apache.skywalking.apm.agent.core.context.TracingContext.isLimitMechanismWorking(TracingContext.java:515)         at org.apache.skywalking.apm.agent.core.context.TracingContext.createLocalSpan(TracingContext.java:290)         at org.apache.skywalking.apm.agent.core.context.ContextManager.createLocalSpan(ContextManager.java:110)         at org.apache.skywalking.apm.plugin.spring.annotations.SpringAnnotationInterceptor.beforeMethod(SpringAnnotationInterceptor.java:32)         at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:89)         at springfox.documentation.schema.property.ObjectMapperBeanPropertyNamingStrategy.nameForDeserialization(ObjectMapperBeanPropertyNamingStrategy.java)         at springfox.documentation.schema.property.BeanPropertyDefinitions.name(BeanPropertyDefinitions.java:48)         at springfox.documentation.schema.property.OptimizedModelPropertiesProvider.beanModelProperty(OptimizedModelPropertiesProvider.java:281)         at springfox.documentation.schema.property.OptimizedModelPropertiesProvider.access$200(OptimizedModelPropertiesProvider.java:79)         at springfox.documentation.schema.property.OptimizedModelPropertiesProvider$2.apply(OptimizedModelPropertiesProvider.java:163)         at springfox.documentation.schema.property.OptimizedModelPropertiesProvider$2.apply(OptimizedModelPropertiesProvider.java:155)         at com.google.common.base.Present.transform(Present.java:79)

That [warn] means you are trying to create over 300 spans in a single segment. I wonder this is your real purpose. It is very dangerous for your memory.

These are automatically generated. what should I do?

These are automatically generated. what should I do?

I don't know what does your application do, and why do you have so many spans in a single request. Only you or your team can say this is reasonable or some incompatible bug. I don't know, sorry.

authors: wu-sheng ylywyn

repo: apache/incubator-skywalking


text:
Redshift could not connect to the server

I tried connect to my Redshift database. There are millions of rows. When query time is longer than
one minute, Superset response is "could not connect to the server". I checked Allow Run Async, but there is another message "Failed to start remote query on worker. Tell your administrator to verify the availability of the message queue." Could you help me with this issue?

You can start by reading the documentation https://superset.incubator.apache.org/installation.html#sql-lab

thanks, i missed it

I installed Redis
pip3 install redis

dependencies
pip install -U "celery[redis]"

run Redis
redise-server

Paste this code to /usr/local/lib/python3.4/dist-packages/superset/config.py
Configure the class "CeleryConfig", by adding the URL of your Redis installation (in my case, localhost:6379):
class CeleryConfig(object):
  BROKER_URL =  'redis://localhost:6379/'
  CELERY_IMPORTS = ('superset.sql_lab', )
  CELERY_RESULT_BACKEND =  'redis://localhost:6379/'
  CELERY_ANNOTATIONS = {'tasks.add': {'rate_limit': '10/s'}}
CELERY_CONFIG = CeleryConfig
HTTP_HEADERS = {
    'super': 'header!'
}

# comment the current RESULTS_BACKEND value
#RESULTS_BACKEND = None

# assign a new value to RESULTS_BACKEND
RESULTS_BACKEND = FileSystemCache('/tmp/sqllab_cache', default_timeout=60*24*7)

When i try superset worker
Starting SQL Celery worker. Traceback (most recent call last):   File "/home/rko/venv/bin/superset", line 15, in <module>     manager.run()   File "/home/rko/venv/lib/python3.4/site-packages/flask_script/__init__.py", line 412, in run     result = self.handle(sys.argv[0], sys.argv[1:])   File "/home/rko/venv/lib/python3.4/site-packages/flask_script/__init__.py", line 383, in handle     res = handle(*args, **config)   File "/home/rko/venv/lib/python3.4/site-packages/flask_script/commands.py", line 216, in __call__     return self.run(*args, **kwargs)   File "/home/rko/venv/lib/python3.4/site-packages/superset/cli.py", line 189, in worker     'broker': config.get('CELERY_CONFIG').BROKER_URL, AttributeError: 'NoneType' object has no attribute 'BROKER_URL'
Can you give me any tips?

i've installed Redis, run superset worker and run superset server..
SQLlab gives me "Failed to start remote query on worker. Tell your administrator to verify the availability of the message queue."
and console:
/usr/local/lib/python3.4/dist-packages/sqlalchemy/sql/compiler.py:624: SAWarning: Can't resolve label reference 'changed_on desc'; converting to text() (this warning may be suppressed after 10 occurrences)
util.ellipses_string(element.element))
/usr/local/lib/python3.4/dist-packages/sqlalchemy/sql/compiler.py:624: SAWarning: Can't resolve label reference 'database_name asc'; converting to text() (this warning may be suppressed after 10 occurrences)
util.ellipses_string(element.element))
2017-07-20 14:15:42,256:INFO:root:Parsing with sqlparse statement SELECT count(uid) as pocet
FROM events2
where event = 'pageview' and time like '%2017-06-30%'
2017-07-20 14:15:42,300:INFO:root:Triggering query_id: 43
/usr/local/lib/python3.4/dist-packages/sqlalchemy/sql/sqltypes.py:596: SAWarning: Dialect sqlite+pysqlite does not support Decimal objects natively, and SQLAlchemy must convert from floating point - rounding errors and other issues may occur. Please consider storing Decimal numbers as strings or integers on this platform for lossless storage.
'storage.' % (dialect.name, dialect.driver))
2017-07-20 14:15:42,980:ERROR:root:[Errno 111] Connection refused
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/kombu/utils/functional.py", line 36, in call
return self.value
AttributeError: 'ChannelPromise' object has no attribute 'value'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 494, in _ensured
return fun(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/kombu/messaging.py", line 187, in _publish
channel = self.channel
File "/usr/local/lib/python3.4/dist-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/usr/local/lib/python3.4/dist-packages/kombu/utils/functional.py", line 38, in call
value = self.value = self.contract()
File "/usr/local/lib/python3.4/dist-packages/kombu/messaging.py", line 224, in
channel = ChannelPromise(lambda: connection.default_channel)
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 819, in default_channel
self.connection
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 802, in connection
self._connection = self._establish_connection()
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 757, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/local/lib/python3.4/dist-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/usr/local/lib/python3.4/dist-packages/amqp/connection.py", line 296, in connect
self.transport.connect()
File "/usr/local/lib/python3.4/dist-packages/amqp/transport.py", line 123, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/usr/local/lib/python3.4/dist-packages/amqp/transport.py", line 164, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 414, in _reraise_as_library_errors
yield
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 515, in _ensured
reraise_as_library_errors=False,
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 405, in ensure_connection
callback)
File "/usr/local/lib/python3.4/dist-packages/kombu/utils/functional.py", line 333, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 261, in connect
return self.connection
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 802, in connection
self._connection = self._establish_connection()
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 757, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/local/lib/python3.4/dist-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/usr/local/lib/python3.4/dist-packages/amqp/connection.py", line 296, in connect
self.transport.connect()
File "/usr/local/lib/python3.4/dist-packages/amqp/transport.py", line 123, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/usr/local/lib/python3.4/dist-packages/amqp/transport.py", line 164, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/superset/views/core.py", line 2011, in sql_json
store_results=not query.select_as_cta)
File "/usr/local/lib/python3.4/dist-packages/celery/app/task.py", line 412, in delay
return self.apply_async(args, kwargs)
File "/usr/local/lib/python3.4/dist-packages/celery/app/task.py", line 535, in apply_async
**options
File "/usr/local/lib/python3.4/dist-packages/celery/app/base.py", line 737, in send_task
amqp.send_task_message(P, name, message, **options)
File "/usr/local/lib/python3.4/dist-packages/celery/app/amqp.py", line 558, in send_task_message
**properties
File "/usr/local/lib/python3.4/dist-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 527, in _ensured
errback and errback(exc, 0)
File "/usr/lib/python3.4/contextlib.py", line 77, in exit
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 419, in _reraise_as_library_errors
sys.exc_info()[2])
File "/usr/local/lib/python3.4/dist-packages/vine/five.py", line 178, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 414, in _reraise_as_library_errors
yield
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 515, in _ensured
reraise_as_library_errors=False,
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 405, in ensure_connection
callback)
File "/usr/local/lib/python3.4/dist-packages/kombu/utils/functional.py", line 333, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 261, in connect
return self.connection
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 802, in connection
self._connection = self._establish_connection()
File "/usr/local/lib/python3.4/dist-packages/kombu/connection.py", line 757, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/local/lib/python3.4/dist-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/usr/local/lib/python3.4/dist-packages/amqp/connection.py", line 296, in connect
self.transport.connect()
File "/usr/local/lib/python3.4/dist-packages/amqp/transport.py", line 123, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/usr/local/lib/python3.4/dist-packages/amqp/transport.py", line 164, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 111] Connection refused
[2017-07-20 14:15:51 +0200] [1746] [INFO] Handling signal: winch
[2017-07-20 14:15:51 +0200] [1746] [INFO] Handling signal: winch

You have misconfigured celery, it's looking for an amqp broker while you said you want to use redis.

i changed that and worker console:
[2017-07-20 14:35:59,067: ERROR/ForkPoolWorker-29] Task superset.sql_lab.get_sql_results[dc953f93-5cf6-4de6-9432-0d09a354ca2e] raised unexpected: Exception("Results backend isn't configured.",)
Traceback (most recent call last):
File "/home/rko/venv/lib/python3.4/site-packages/celery/app/trace.py", line 367, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/rko/venv/lib/python3.4/site-packages/celery/app/trace.py", line 622, in protected_call
return self.run(*args, **kwargs)
File "/home/rko/venv/lib/python3.4/site-packages/superset/sql_lab.py", line 81, in get_sql_results
handle_error("Results backend isn't configured.")
File "/home/rko/venv/lib/python3.4/site-packages/superset/sql_lab.py", line 78, in handle_error
raise Exception(query.error_message)
Exception: Results backend isn't configured.

i have this config:
/superset/config.py
Configure the class "CeleryConfig", by adding the URL of your Redis installation (in my case, localhost:6379):
class CeleryConfig(object):
BROKER_URL =  'redis://localhost:6379/'
CELERY_IMPORTS = ('superset.sql_lab', )
CELERY_RESULT_BACKEND =  'redis://localhost:6379/'
CELERY_ANNOTATIONS = {'tasks.add': {'rate_limit': '10/s'}}
CELERY_CONFIG = CeleryConfig
from werkzeug.contrib.cache import RedisCache
RESULTS_BACKEND = RedisCache(
host='localhost', port=6379, key_prefix='superset_results')

You have to add to add proper quoting to your code excerpts otherwise it's impossible to help you.

i changed that and worker console:
[2017-07-20 14:35:59,067: ERROR/ForkPoolWorker-29] Task superset.sql_lab.get_sql_results[dc953f93-5cf6-4de6-9432-0d09a354ca2e] raised unexpected: Exception("Results backend isn't configured.",)
Traceback (most recent call last):
File "/home/rko/venv/lib/python3.4/site-packages/celery/app/trace.py", line 367, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/rko/venv/lib/python3.4/site-packages/celery/app/trace.py", line 622, in protected_call
return self.run(*args, **kwargs)
File "/home/rko/venv/lib/python3.4/site-packages/superset/sql_lab.py", line 81, in get_sql_results
handle_error("Results backend isn't configured.")
File "/home/rko/venv/lib/python3.4/site-packages/superset/sql_lab.py", line 78, in handle_error
raise Exception(query.error_message)
Exception: Results backend isn't configured.

i have this config:
/superset/config.py
Configure the class "CeleryConfig", by adding the URL of your Redis installation (in my case, localhost:6379):
class CeleryConfig(object):
  BROKER_URL =  'redis://localhost:6379/'
  CELERY_IMPORTS = ('superset.sql_lab', )
  CELERY_RESULT_BACKEND =  'redis://localhost:6379/'
  CELERY_ANNOTATIONS = {'tasks.add': {'rate_limit': '10/s'}}
CELERY_CONFIG = CeleryConfig

from werkzeug.contrib.cache import RedisCache
RESULTS_BACKEND = RedisCache(
    host='localhost', port=6379, key_prefix='superset_results')

is this better?

i changed that and worker console:
[2017-07-20 14:35:59,067: ERROR/ForkPoolWorker-29] Task superset.sql_lab.get_sql_results[dc953f93-5cf6-4de6-9432-0d09a354ca2e] raised unexpected: Exception("Results backend isn't configured.",)
Traceback (most recent call last):
File "/home/rko/venv/lib/python3.4/site-packages/celery/app/trace.py", line 367, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/rko/venv/lib/python3.4/site-packages/celery/app/trace.py", line 622, in protected_call
return self.run(*args, **kwargs)
File "/home/rko/venv/lib/python3.4/site-packages/superset/sql_lab.py", line 81, in get_sql_results
handle_error("Results backend isn't configured.")
File "/home/rko/venv/lib/python3.4/site-packages/superset/sql_lab.py", line 78, in handle_error
raise Exception(query.error_message)
Exception: Results backend isn't configured.

i have this config:
/superset/config.py
Configure the class "CeleryConfig", by adding the URL of your Redis installation (in my case, localhost:6379):
class CeleryConfig(object):
  BROKER_URL =  'redis://localhost:6379/'
  CELERY_IMPORTS = ('superset.sql_lab', )
  CELERY_RESULT_BACKEND =  'redis://localhost:6379/'
  CELERY_ANNOTATIONS = {'tasks.add': {'rate_limit': '10/s'}}
CELERY_CONFIG = CeleryConfig

from werkzeug.contrib.cache import RedisCache
RESULTS_BACKEND = RedisCache(
    host='localhost', port=6379, key_prefix='superset_results')

is this better?
What mean? Results backend isn't configured.

repo: apache/incubator-superset


text:
Refactor treemap

Decouple the visualization code from slice and formData
Test
Ran a development instance with the code above and verified with production instance that they produce the same results.
@williaster @conglei @graceguo-supercat

Codecov Report

Merging #5670 into master will decrease coverage by 0.03%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #5670      +/-   ##
==========================================
- Coverage   63.51%   63.48%   -0.04%
==========================================
  Files         360      360
  Lines       22904    22915      +11
  Branches     2551     2551
==========================================
  Hits        14548    14548
- Misses       8341     8352      +11
  Partials       15       15

Impacted Files
Coverage Δ

superset/assets/src/visualizations/treemap.js
0% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cdd348a...723f82b. Read the comment docs.

authors: codecov-io kristw


text:
[fix] view results in sql lab

click view results from sql lab will see JS excetions:

exception is from this line:
https://github.com/apache/incubator-superset/blob/69e8df404d46e35bf686cc92992d6e0415172d90/superset/assets/src/SqlLab/components/ExploreResultsButton.jsx#L171
@mistercrunch @michellethomas @kristw

Codecov Report

Merging #6405 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #6405   +/-   ##
=======================================
  Coverage   77.31%   77.31%
=======================================
  Files          67       67
  Lines        9581     9581
=======================================
  Hits         7408     7408
  Misses       2173     2173

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c42bcf8...7cebf7b. Read the comment docs.

LGTM


text:
[epoch] Remove non-UTC epoch logic

CATEGORY
Choose one

[x] Bug Fix
[ ] Enhancement (new features, refinement)
[ ] Refactor
[ ] Add tests
[ ] Build / Development Environment
[ ] Documentation

SUMMARY
As @agrawaldevesh correctly identified in https://github.com/apache/incubator-superset/pull/6721 previously we were computing the Unix timestamp for the right-hand-side (RHS) of the temporal filter condition using the local time zone as opposed to UTC which is the definition of Unix (or epoch) time.
@agrawaldevesh's change was behind a feature flag and disabled by default however this clearly is a bug and I sense we should remedy the problem by merely replacing the previously incorrect logic. Note I strongly believe users were probably unaware of the issue as Unix timestamps aren't human readable.
TEST PLAN
CI.
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

REVIEWERS
to: @agrawaldevesh @betodealmeida @michellethomas @mistercrunch @villebro

https://github.com/apache/incubator-superset/issues/7656

Codecov Report

Merging #7667 into master will increase coverage by <.01%.
The diff coverage is 75%.

@@            Coverage Diff             @@
##           master    #7667      +/-   ##
==========================================
+ Coverage   65.57%   65.58%   +<.01%
==========================================
  Files         435      435
  Lines       21754    21749       -5
  Branches     2394     2394
==========================================
- Hits        14266    14264       -2
+ Misses       7367     7364       -3
  Partials      121      121

Impacted Files
Coverage Δ

superset/config.py
93.97% <ø> (-0.04%)
:arrow_down:

superset/connectors/sqla/models.py
82.39% <75%> (+0.41%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d62c37b...e93f33b. Read the comment docs.

@agrawaldevesh are you onboard with this change?

Go for it ! I only introduced the flag since I did not want to break
existing use cases. I have no issues with making this be default
On Mon, Jun 10, 2019 at 5:02 PM John Bodley notifications@github.com
wrote:

@agrawaldevesh
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_agrawaldevesh&d=DwMCaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=Z3aBmCzO3AGueh3ZEPa3n4ujmbgsJJDE-x0U4W8t_Us&m=O_sBQHDGT4uYRBuD0MeJgRcwajjdTrjVL114CzrCW6A&s=jbT8jeF_xUkzsr1n0uEA-H-vfQW3Vu6ryUdUELUU7vQ&e=
are you onboard with this change?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dsuperset_pull_7667-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAE44R77I3WBGYPB5XWLSMQLPZ3TRVA5CNFSM4HVJ6VW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXLRTFI-23issuecomment-2D500636053&d=DwMCaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=Z3aBmCzO3AGueh3ZEPa3n4ujmbgsJJDE-x0U4W8t_Us&m=O_sBQHDGT4uYRBuD0MeJgRcwajjdTrjVL114CzrCW6A&s=3Z1HFovjiRBXcX0cRTa1BCbVGz5q-yVMM9B9u7jB_yM&e=,
or mute the thread
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AE44R77CTI7IHEM57QUNF2LPZ3TRVANCNFSM4HVJ6VWQ&d=DwMCaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=Z3aBmCzO3AGueh3ZEPa3n4ujmbgsJJDE-x0U4W8t_Us&m=O_sBQHDGT4uYRBuD0MeJgRcwajjdTrjVL114CzrCW6A&s=S3v_GuSJtAWrrOdirnt1nz9dQNSS9KGhn1jIfdjd24I&e=
.

authors: agrawaldevesh codecov-io john-bodley


text:
[SYSTEMML-1554] IPA Scalar Transient Read Replacement

Currently, during IPA we collect all variables (scalars & matrices)
eligible for propagation across blocks (i.e. not updated in block), and
then propagate the only the matrix sizes across the blocks. It seems
plausible that we could also replace all eligible scalar transient reads
with literals based on the variables that have already been collected.
The benefit is that many ops will be able to determine their respective
output sizes during regular compilation, instead of having to wait until
dynamic recompilation, and thus we can reduce the pressure on dynamic
recompilation.
Are there drawbacks to this approach?  The use case is that I was seeing a large number of memory warnings while training a convolutional net due to the sizes being unknown during regular compilation, yet the engine only having CP versions of the ops.  Additionally, I was running into actual heap space OOM errors for situations that should not run out of memory, and thus I started exploring.
I've attached an example script and the explain plan (hops & runtime) w/ and w/o the IPA scalar replacement to the associated JIRA issue.
cc @mboehm7

thanks @dusenberrymw I gave it a try on our ARIMA application testcase which historically was challenging for scalar propagation. Unfortunately, it failed due to an - probably unrelated - issue in replaceLiteralFullUnaryAggregate. Once I resolved this, I'll play around with it a bit more.
Once this PR is in, we should also think about the problem of propagating scalars into functions if functions are called once or with consistent scalar inputs.

@mboehm7 Great, interested to see what else is needed for this to be generally applicable.  Also, definitely +1 for propagating scalars into functions.  In particular, we should allow for the case of functions for which any subset of the inputs are consistent scalars.  I.e., a function may have an unknown matrix size as an input, but then have several other scalar inputs that are always consistent.

Thanks, @mboehm7.  Looks like it is failing a test still -- org.apache.sysml.test.integration.functions.misc.DataTypeChangeTest#testDataTypeChangeValidate4c.  Looking into it, it fails due to trying to cast a Matrix to a Scalar object.  At a deeper level, it looks like the propagated variable map is holding onto the "matrix" X, rather than dropping it as it should, since X is turned into a scalar by the call X = foo(X).  Interestingly, the FunctionOp for the foo function is marked as having an Unknown datatype and valuetype.  That to me seems to be  big issue, but I'm not sure exactly where that is failing.  Thoughts?  Overall, this seems like a bug that was just hidden before, rather than being newly introduced.

yes this is almost certainly a bug - originally we allowed data type changes in conditional control flow (e.g., if branch assigns a scalar and else branch a matrix), in which case we assign UNKNOWN for subsequent references. However, I modified this years ago because SystemML could not compile valid instructions for these scenarios unless we extend the recompiler to actually update the data type and block sizes there.

By the way, aside from that test, everything else passed.

cc @mboehm7 Can you review this fix for the datatype conversion issue?  I'm also waiting for the full testing with Jenkins.

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1437/

Thanks, @mboehm7.  I'll update the docs and merge.

Refer to this link for build results (access rights to CI server needed):
https://sparktc.ibmcloud.com/jenkins/job/SystemML-PullRequestBuilder/1442/

authors: akchinSTC dusenberrymw mboehm7

repo: apache/incubator-systemml


text:
[iOS] PR #2394 covered by PR #2520 so release v0.26.0 still have thread issue

#2394
#2520

Try release 0.28
If this still bothers you and you find a solution, you could give us a PR if you can, I am very happy to talk with you implemenation detail or review you PR in mailing list.
I have a busy schedule and I can't read Github issue every day, but I check mailing list every day. I am sorry if this bothers you.

authors: YorkShen darkThanBlack

repo: apache/incubator-weex


text:
[ZEPPELIN-757] Ordering dropdown menu items alphabetically.

What is this PR for?
Fixing documentation.
What type of PR is it?
Documentation
Todos
N/A
What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-757
How should this be tested?
Follow the steps in https://github.com/apache/incubator-zeppelin/blob/master/docs/README.md to build the documentation.
Screenshots (if appropriate)

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

Ready for review.

Thanks @jsimsa for the fix. LGTM and merge if there're no more discussions.

authors: Leemoonsoo jsimsa

repo: apache/incubator-zeppelin


text:
Constant High CPU Usage

Describe the bug
I have a small IOTDB instance running on Ubuntu. Over time the CPU utilisation is very high even though there are a low number of clients and transactions. The CPU usage appears to climb over time. This system has been running for several weeks without a restart. If I run "ps -ef" I can see that it is IOTDB that is using the CPU constantly.
Below is the CPU usage before and after I ran stop-server/start-server. The CPU dropped from > 80% down to < 10%.

To Reproduce
Steps to reproduce the behavior:

Run IOTDB 11.0 with default settings

Expected behavior
The CPU usage should not be so high.
Screenshots
See above.
Desktop (please complete the following information):

OS: Ubuntu 20.04.1 LTS
Browser Not Applicable
Version 11.0

Additional context
None

I have upgraded to 11.2 so I will monitor and see if the issue persists.

Step 1: Could you upload the logs during the high cpu usage?
Step 2: If possible, could you please use JProfiler to record the CPU when the CPU usage is high, and save it as a .jps snapshot file, then we can see what happens.
Step 3: Some config that may solve the problem:

iotdb-engine.properties

enable_unseq_compaction=false

I have been monitoring the CPU over the past week or so. The image below shows a weeks worth of CPU data. Notice that after a few days the CPU is back up high.

There are no errors in the logs - most logging is info level with query reponse time.
I am going to restart the database with "enable_unseq_compaction=false" and see how that goes.

I am still having the same problem even with "enable_unseq_compaction=false"

I have restarted to captured the logs.

authors: ope-nz qiaojialin

repo: apache/iotdb


text:
KAFKA-13116: Fix message_format_change_test and compatibility_test_new_broker_test failures

These failures were caused by a46b82bea9abbd08e5. Details for each test:

message_format_change_test: use IBP 2.8 so that we can write in older message
formats.
compatibility_test_new_broker_test_failures: fix down-conversion path to handle
empty record batches correctly. The record scan in the old code ensured that
empty record batches were never down-converted, which hid this bug.

Verified with ducker that some variants of these tests failed without these changes
and passed with them.
Note that the upgrade_test is still failing. It looks like there are multiple causes,
so I left that for another PR.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@hachikuji I addressed your comment, this is ready for another review.

To double-check the local results, I started the branch builder here too https://jenkins.confluent.io/job/system-test-kafka-branch-builder/4620/

Failures are unrelated. Merging to master and cherry-picking to 3.0.

The branch builder system tests passed btw.

repo: apache/kafka


text:
KAFKA-13971: Fix atomicity violations caused by improper usage of ConcurrentHashMap - part2

## Problem #1 in DelegatingClassLoader.java
Atomicity violation in example such as:
Consider thread T1 reaches line 228, but before executing context switches to thread T2 which also reaches line 228. Again context switches to T1 which reaches line 232 and adds a value to the map. T2 will execute line 228 and creates a new map which overwrites the value written by T1, hence change done by T1 would be lost. This code change ensures that two threads cannot initiate the TreeMap, instead only one of them will.
Problem #2 in RocksDBMetricsRecordingTrigger.java
Atomicity violation in example such as:
Consider thread T1 reaches line 40 but before executing it context switches to thread T2 which also reaches line 40. In a serialized execution order, thread T2 should have thrown the exception but it won't in this case. The code change fixes that.
Note that some other problems associated with use of concurrent hashmap has been fixed in https://github.com/apache/kafka/pull/12277

Is the relevant code specified as thread safe?

Is the relevant code specified as thread safe?

Thank you for your review @ijuma. I appreciate it. Though, I am afraid I don't understand your question.
Are you asking whether the existing code is supposed to be thread safe?
If yes, for DelegatingClassLoader.java the javadoc for the class mentioned that it is supposed to be thread safe (but it isn't due to the bug that is fixed in this review). For the RocksDBMetricsRecordingTrigger.java, we run a thread periodically from a metric trigger thread pool which reads from the map maintained in the class. At the same time it is possible that another thread is mutating the map during startup/shutdown of rocksDB which may leave the map in inconsistent state. Hence, it's important for this class to be thread safe as well.
Also, note that both the classes in this review use ConcurrentHashMap (albeit incorrectly) to ensure thread safe mutation over the map.
Are you asking whether the changed code is thread safe?
If yes, the change uses atomic operations provided by ConcurrentHashMap to ensure thread safety.

@C0urante please review when you get a chance.

authors: divijvaidya ijuma


text:
KAFKA-14260: add synchronized to prefixScan method

As a result of "14260: InMemoryKeyValueStore iterator still throws ConcurrentModificationException", I'm adding synchronized to prefixScan as an alternative to going back to the ConcurrentSkipList.
I've read up on testing multi-threaded behavior and I believe it's best to leave the testing as it is for now as testing whether synchronized works doesn't always work. I did make sure ./gradlew test was green on my branch. Happy to be corrected here.
This is my first PR. As of the guidelines, I  that the contribution is my original work and that I license the work to the project under the project's open source license. I see that I also need to make a build trigger request, @ableegoldman I would appreciate one please :)
I do not believe this requires a documentation update as it is just bringing a method up to standard. Again, happy to help out if it turns out otherwise.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

I see that I also need to make a build trigger request

By the way, this is thankfully no longer the case (it used to be really annoying and only worked like half the time) -- these days the build will run on any PR that's opened, and will rerun each time you push a new commit. I guess the contributing guidelines are out of date so thanks for bringing that up 🙂 I'll update them

Merged to trunk and cherrypicked to 3.4

authors: Cerchie ableegoldman


text:
KAFKA-14909: check zkMigrationReady tag before migration

add ZkMigrationReady in apiVersionsResponse
check all nodes if ZkMigrationReady are ready before moving to next migration state

TODO: add more tests
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Failed tests are unrelated and also failed in trunk build:
    Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.DedicatedMirrorIntegrationTest.testSingleNodeCluster()
    Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.DedicatedMirrorIntegrationTest.testMultiNodeCluster()
    Build / JDK 11 and Scala 2.13 / kafka.server.KRaftClusterTest.testLegacyAlterConfigs()
    Build / JDK 11 and Scala 2.13 / kafka.server.RaftClusterSnapshotTest.testSnapshotsGenerated()
    Build / JDK 11 and Scala 2.13 / kafka.server.RaftClusterSnapshotTest.testSnapshotsGenerated()
    Build / JDK 11 and Scala 2.13 / org.apache.kafka.controller.QuorumControllerTest.testConfigurationOperations()
    Build / JDK 11 and Scala 2.13 / org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskRequestWithOldStartMsGetsUpdated()
    Build / JDK 17 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.DedicatedMirrorIntegrationTest.testMultiNodeCluster()
    Build / JDK 17 and Scala 2.13 / kafka.server.KRaftClusterTest.testCreateClusterAndPerformReassignment()
    Build / JDK 17 and Scala 2.13 / kafka.server.KRaftClusterTest.testUnregisterBroker()
    Build / JDK 17 and Scala 2.13 / kafka.server.KRaftClusterTest.testCreateClusterAndCreateAndManyTopics()
    Build / JDK 17 and Scala 2.13 / org.apache.kafka.controller.QuorumControllerTest.testUpgradeMigrationStateFrom34()
    Build / JDK 17 and Scala 2.13 / org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskRequestWithOldStartMsGetsUpdated()
    Build / JDK 8 and Scala 2.12 / org.apache.kafka.connect.mirror.integration.DedicatedMirrorIntegrationTest.testSingleNodeCluster()
    Build / JDK 8 and Scala 2.12 / org.apache.kafka.connect.mirror.integration.DedicatedMirrorIntegrationTest.testMultiNodeCluster()
    Build / JDK 8 and Scala 2.12 / org.apache.kafka.connect.mirror.integration.IdentityReplicationIntegrationTest.testSyncTopicConfigs()
    Build / JDK 8 and Scala 2.12 / org.apache.kafka.connect.integration.OffsetsApiIntegrationTest.testGetSinkConnectorOffsetsDifferentKafkaClusterTargeted
    Build / JDK 8 and Scala 2.12 / kafka.server.KRaftClusterTest.testSetLog4jConfigurations()
    Build / JDK 8 and Scala 2.12 / kafka.server.KRaftClusterTest.testCreateClusterAndPerformReassignment()
    Build / JDK 8 and Scala 2.12 / kafka.server.KRaftClusterTest.testCreateClusterAndPerformReassignment()
    Build / JDK 8 and Scala 2.12 / org.apache.kafka.controller.QuorumControllerTest.testCreateAndClose()
    Build / JDK 8 and Scala 2.12 / org.apache.kafka.controller.QuorumControllerTest.testCreateAndClose()

I'd like to merge it after the test failure fix completed in this PR: https://github.com/apache/kafka/pull/13647 to make sure we don't introduce more failed tests.

Failed tests are unrelated:
    Build / JDK 11 and Scala 2.13 / org.apache.kafka.connect.runtime.distributed.DistributedHerderTest.testExternalZombieFencingRequestAsynchronousFailure
    Build / JDK 17 and Scala 2.13 / integration.kafka.server.FetchFromFollowerIntegrationTest.testRackAwareRangeAssignor()
    Build / JDK 17 and Scala 2.13 / kafka.server.CreateTopicsRequestTest.testErrorCreateTopicsRequests(String).quorum=kraft
    Build / JDK 17 and Scala 2.13 / org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskRequestWithOldStartMsGetsUpdated()


text:
KAFKA-16707: Kafka Kraft : using Principal Type in StandardACL in order to defined ACL with a notion of group without rewriting KafkaPrincipal of client by rules

Default StandardAuthorizer in Kraft mode is defining a KafkaPrincpal as type=User and a name, and a special wildcard eventually.
The difficulty with this solution is that we can't define ACL by group of KafkaPrincipal.
There is a way for the moment to do so by defining RULE to rewrite the KafkaPrincipal name field, BUT, to introduce this way the notion of group, you have to set rules which will make you loose the uniq part of the KafkaPrincipal name of the connected client.
The concept here, in the StandardAuthorizer of Kafka Kraft, is to add  the management of KafkaPrincipal type:
Regex
StartsWith
EndsWith
Contains
(User is still available and keep working as before to avoid any regression/issue with current configurations)

This would be done in the StandardAcl class of metadata/authorizer, and the findresult method of StandardAuthorizerData will delegate the match to the StandardAcl class (for performance reason: precompile regex in ACL).
*I added tests in metadat, and run ./gradlew test from kafak:trunk and my fork: no more failed test on my branch than kafka:trunk
Committer Checklist (excluded from commit message)

[ x ] Verify design and implementation => thanks to spell checker in gradle process
[ x ] Verify test coverage and CI build status => adding few tests in metadata, an run gradlew test without more failed test thant kafka:trunk
[ x ] Verify documentation (including upgrade notes) : added few lines in doc, no upgrade info as the previous behaviour should still work as before.

Link to the JIRA-16707

Hello, When I'm running "./gradlew test" on my side from apache/kafka/trunk clone, I have failed tests.
So Is there a way to know (an easy one?) to know which failed test in "continuous-integration/jenkins/pr-merge" I have to look ?


text:
MINOR: Cache topic resolution in TopicIds set

Looking up topics in a TopicsImage is relatively slow. Cache the results
in TopicIds to improve assignor performance. In benchmarks, we see a
noticeable improvement in performance in the heterogeneous case.
Before
Benchmark                                       (assignmentType)  (assignorType)  (isRackAware)  (memberCount)  (partitionsToMemberRatio)  (subscriptionType)  (topicCount)  Mode  Cnt    Score   Error  Units
ServerSideAssignorBenchmark.doAssignment             INCREMENTAL           RANGE          false          10000                         10         HOMOGENEOUS          1000  avgt    5   36.400 ± 3.004  ms/op
ServerSideAssignorBenchmark.doAssignment             INCREMENTAL           RANGE          false          10000                         10       HETEROGENEOUS          1000  avgt    5  158.340 ± 0.825  ms/op
ServerSideAssignorBenchmark.doAssignment             INCREMENTAL         UNIFORM          false          10000                         10         HOMOGENEOUS          1000  avgt    5    1.329 ± 0.041  ms/op
ServerSideAssignorBenchmark.doAssignment             INCREMENTAL         UNIFORM          false          10000                         10       HETEROGENEOUS          1000  avgt    5  382.901 ± 6.203  ms/op

After
Benchmark                                       (assignmentType)  (assignorType)  (isRackAware)  (memberCount)  (partitionsToMemberRatio)  (subscriptionType)  (topicCount)  Mode  Cnt    Score   Error  Units
ServerSideAssignorBenchmark.doAssignment             INCREMENTAL           RANGE          false          10000                         10         HOMOGENEOUS          1000  avgt    5   36.465 ± 1.954  ms/op
ServerSideAssignorBenchmark.doAssignment             INCREMENTAL           RANGE          false          10000                         10       HETEROGENEOUS          1000  avgt    5  114.043 ± 1.424  ms/op
ServerSideAssignorBenchmark.doAssignment             INCREMENTAL         UNIFORM          false          10000                         10         HOMOGENEOUS          1000  avgt    5    1.454 ± 0.019  ms/op
ServerSideAssignorBenchmark.doAssignment             INCREMENTAL         UNIFORM          false          10000                         10       HETEROGENEOUS          1000  avgt    5  342.840 ± 2.744  ms/op

Based heavily on https://github.com/apache/kafka/pull/16527.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@mumrah

Another thing to consider is the lifetime of the cache. Do we really need the ID + name mappings kept in memory forever?

The lifetime of the cache is bound to the call. It is not kept forever.

Does this come down to performance differences between HashMap and PCollectionsImmutableMap?

Yes.

If we decide we really need faster topic ID to name lookups, I would consider adding it to TopicsImage. Managing a cache outside of the image will be a bit difficult.

We could consider this separately. At the moment, we don't really have the time to do it. The current strategy seems to be a good tradeoff at the moment given that it is only bound to the call and not kept forever.

@dajac thanks for the explanation, makes sense. Can we include a javadoc on the class describing the expected lifetime of this class?

@mumrah

Seeing that the cache is not actually used outside of tests and benchmarks, I'm guessing this is still WIP.
It's used in TargetAssignmentBuilder.build(), which is used by the new group coordinator.

I've updated the javadoc to describe the lifetime of the cache.

@mumrah I will merge it. If you have further comments, @squah-confluent can address separately.

authors: dajac mumrah squah-confluent


text:
kafka-4767: KafkaProducer is not joining its IO thread properly

KafkaProducer#close swallows the InterruptedException which might be acceptable when it's invoked from within the main thread or user is extending Thread and therefore controls all the code higher up on the call stack. For other cases, it'd better retstore the interupted status after capturing the exception.

@ijuma Please have a review. Thanks.

@ijuma Please have a reivew on this PR. Thanks.

LGTM

Thanks for the PR. I looked this in more detail and it looks like we eventually throw KafkaException for this case. In the consumer, we throw InterruptException (which is a non-checked version of InterruptedException that inherits from KafkaException). Seems like we should do the same here. That class sets the interrupt in the constructor.

@ijuma Followed the same pattern as what KafkaConsumer#close treats interruption but also explicitly added an if clause to check InterruptedException since firstException would be set to it explicitly in  KafkaProducer#close. Please have a review on that. Thanks.

@ijuma Well, already removed those dead code and also added the code to reserve the interruption status. Looks good now?

authors: amethystic ijuma omkreddy


text:
KAFKA-5534: offsetForTimes result should include partitions with no offset

For topics that support timestamp search, if no offset is found for a partition, the partition should still be included in the result with a null offset value. This KafkaConsumer method currently excludes such partitions from the result.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5806/
Test PASSed (JDK 8 and Scala 2.12).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5820/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/5937/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/5922/
Test PASSed (JDK 8 and Scala 2.12).

@hachikuji, is this what you had in mind?

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6154/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6138/
Test PASSed (JDK 8 and Scala 2.12).

We should add a note to the upgrade notes and I think we can only merge this in trunk as it does change the behaviour.

Thanks @ijuma. I'll update to upgrade notes with this change. I assume it's this file that needs to be updated.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6180/
Test PASSed (JDK 7 and Scala 2.11).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6164/
Test PASSed (JDK 8 and Scala 2.12).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6210/
Test PASSed (JDK 8 and Scala 2.12).

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6226/
Test PASSed (JDK 7 and Scala 2.11).

authors: asfgit ijuma vahidhashemian


text:
MINOR: add window store range query in simple benchmark

There are a couple minor additions in this PR:

add a new test for window store, to range query upon receiving each record.
in the non-windowed state store case, add a get call before the put call.
Enable caching by default to be consistent with other Join / Aggregate cases, where caching is enabled by default.

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@mjsax @bbejeck @vvcephei


text:
Upgrade ZooKeeper to 3.4.12 and Scala to 2.12.6

ZK 3.4.12 fixes the regression that forced us to go back to
3.4.10. Release notes:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310801&version=12342040
Scala 2.12.6 fixes the issue that prevented us from upgrading
to 2.12.5.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@junrao trying the ZK upgrade again.


text:
MINOR: more info in error msg

@guozhangwang

got it.

LGTM

authors: ewencp ymatsuda


text:
[WIP] KNOX-2095 - Adding in DefaultDispatch code and tests to handle 504 errors

What changes were proposed in this pull request?
Currently, Knox masks all connection errors as 500 errors, when they may be more accurately described using other error codes, especially 504. A change has been made to display a 504 error in the event of a socket timeout..
How was this patch tested?
Ran ant verify under Knox 1.2.0. The patch was generated against the Knox 1.2.0 branch, and subsequently applied to Knox 1.4.0. Currently ant verify is failing, but these are most likely transient errors as the same errors appear in master. Still running tests.

Hmm. It looks like the check at https://api.travis-ci.org/v3/job/607855611/log.txt failed, but the logs suggest that it may be running plugins that aren't thread safe. Assuming that this is a threading issue, is there any way to rerun the tests, possibly without parallelism?

The error was
[ERROR] Failures:
[ERROR]   GatewayCorrelationIdTest.testTestService:209
Expected: is <46>
     but: was <45>

Not an error I've seen before. I retriggered the JDK 11 job. We have some flaky tests related to ZK but not that test.

Hi Kevin, sorry for the late followup; was out yesterday. Seems like it failed on gateway-service-remoteconfig again; was it still the testTestService test? I don't have JDK 11 installed on this machine and I'd like to confirm before trying to pursue the error.
(On that note, is there any good way to check the Surefire reports/view build artifacts?)

Grasping at straws here, but looking through the test case at https://github.com/apache/knox/blob/89caa5feeed706abc8d7ce1407830ae00d97d405/gateway-test/src/test/java/org/apache/knox/gateway/GatewayCorrelationIdTest.java, is it possible that the reduced timeout might be causing the issue? I'm not completely sure how the test works, but with the change in this PR, all connection attempts that experience a socket timeout are automatically given a 403, whereas without the change, there would at least be an attempt to contact the failover nodes.
...then again, I suppose this wouldn't explain the successes in JDK8. It's a bit difficult to tell without looking at the reports unfortunately.

@jameschen1519 I don't think the test failures are related to your change.

(On that note, is there any good way to check the Surefire reports/view build artifacts?)

The Travis build details are linked in the pr and then you go to the specific build and build log. That has the same output if you were to run locally.

This PR has not been touched for over a year ago; closing it.

repo: apache/knox


text:
[WIP][KYUUBI #6031] Add CollectMetricsPrettyDisplayListener

:mag: Description
Issue References 🔗

This pull request fixes #6031
Describe Your Solution 🔧

Types of changes :bookmark:

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪
Behavior Without This Pull Request :coffin:
Behavior With This Pull Request :tada:
Related Unit Tests

Checklist 📝

[ ] This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

cc @zhouyifan279

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (208354c) 61.17% compared to head (cd23093) 61.03%.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #6035      +/-   ##
============================================
- Coverage     61.17%   61.03%   -0.14%
  Complexity       23       23
============================================
  Files           623      623
  Lines         37144    37144
  Branches       5032     5032
============================================
- Hits          22721    22669      -52
- Misses        11979    12018      +39
- Partials       2444     2457      +13

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter ulysses-you wForget

repo: apache/kyuubi


text:
Upgrade kotlin and log4j versions

Hello,
It seems that the Log4j version of the Kotlin API is out of date.
On top of that, the Kotlin version hasn't been updated for the past 2 years.
I have did a code analysis scan and found no incompatibles, tests are passing, and everything seems to be working as excepted.
Reasons to upgrade to 1.8.0 that are relevant:

1.8.0 - Improved kotlin-reflect performance
1.6.0 - Further Improvements to type inference for recursive generic types
1.5.30 - Improvements to type inference for recursive generic types
1.5.0 - Inline classes are released as Stable

And lots of other performance improvements overall...
Changes

Upgrade log4j to 2.19.0
Upgrade Kotlin to 1.8.0
kotlinx.coroutines to 1.6.4

Trivial changes - Feel free to amend as needed. :)

@jvz Can you have a look please?

The Kotlin API is a minimum version requirement. I've been using this library with Kotlin 1.4.x, 1.7.x, and 1.8.x. The Log4j version update is good, though!

I see, that makes sense, thanks for letting me know, I am closing this one and opening a more appropriate PR for Log4j upgrade only.

authors: jvz u-ways


text:
Make FST BytesStore grow smoothly

Description

Too bad we don't have a writer that uses tiny (like 8 bytes) block at first, but doubles size for each new block (16 bytes, 32 bytes next, etc.). Then we would naturally use log(size) number of blocks without over-allocating.
But then reading bytes is a bit tricky because we'd need to take discrete log (base 2) of the address. Maybe it wouldn't be so bad -- we could do this with Long.numberOfLeadingZeros maybe? But that's a bigger change ... we can do this separately/later.

From https://github.com/apache/lucene/pull/12604#discussion_r1344639608

Note that oal.store.ByteBuffersDataOutput takes a different and neat approach to gracefully growing: it picks an initial block size, and appends new blocks as you write bytes, but then if it reaches 100 blocks, it "resizes" itself by doubling the block size and copying over, so that now you have 50 blocks.
So it's still O(N) amortized cost of that doubling/copying with time, and at any given moment you will not be wasting too many %tg of the bytes you've written, except at the start 1 KB block size.

In https://github.com/apache/lucene/pull/12624, I moved the main FST body out of BytesStore into ByteBuffersDataOutput, and BytesStore becomes only a single byte[] for the currently written node so maybe we don't need to do this?

repo: apache/lucene


text:
Add new option "failOnWarning".

This option causes the maven-compiler-plugin to treat warnings as errors
and fail accordingly.
Simply adds "-Werror" to the compiler command line. It may be nice to
add this to the plexus-compiler-api proper but as the sonatype repo only
has tags to 2.4 and the current plugin references 2.6 (and I have no
idea where that comes from), I went the easy route. Happy to refactor
it if wanted.

Is this PR still relevant. If so, is there any reason not to pass to compilerArgs?

Hm. No comment for almost a year and then one comment and closed within 8 days.
Yes, it is still relevant. As an option, I can expose this as a property with
...
   <failOnWarning>${failWarningSwitch}</failOnWarning>
...

so this can be overridden from the command line. If it gets added to , there is no way to control this dynamically from the command line.

plexus compiler api 2.8.1 supports failOnWarning directly. I may simply redo this patch to leverage this.

Let's open, Can you reopen, can you provide a PR?

authors: hgschmie michael-o

repo: apache/maven-plugins


text:
P-NUCLEO-WB55

Slinky works, SPI/I2C/TIM under test...

@kasjer Could you take a look at this again, I think most review issues were tackled, apart from the int vs unsigned for I2c pins and flash suggestions (FLASH_PAGE_SIZE and removing _ prefix). I am not sure what your suggestion is for the first one, but the issue spans across families so I think it would makes more sense to do it in another PR that covers them all. For the flash suggestions, I agree, but would rather send a new PR that changes it on every family. Is that OK?

@kasjer All issues addressed.

repo: apache/mynewt-core


text:
NIFI-2887 NumberFormatException in HL7ExtractAttributes for repeating segments with order control codes

Fixes NIFI-2887 and adds more test cases

+1 LGTM, ran the unit tests and on a full NiFi, verified the NFE is not presented and the flow file is parsed successfully. Merging to master, thanks!

+1 LGTM, ran the unit tests and with a full NiFi, verified the NFE is not presented and messages are parsed successfully. Merging to master, thanks!

repo: apache/nifi


text:
NIFI-4371 - add support for query timeout in Hive processors

Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit?

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

The "unit tests" for TestSelectHiveQL use Derby as the database, only to test the functionality of getting the "HiveQL" statement to the database and parsing its results. In that vein, Derby supports setQueryTimeout (DERBY-31), so can we add a unit test that sets the value, to exercise that part of the code?

Thanks for the review @mattyb149 and @joewitt. I updated the property description based on your comments. Regarding the unit test, since I'm using a HiveStatement object in the custom validate method, I'm not sure I can easily test the property (it'll always fail in a default build even with the Derby backend). And, if adding a unit test where I expect the validation to raise an error, this test may not have the expected result if using different profiles in the maven build.

I'm getting NPEs in the unit tests, something weird with MockPropertyValue getting created without "expectExpressions" being set to anything, causing isExpressionLanguagePresent() to throw the NPE

I already noticed this error while working on others PRs (I'm a bit surprised I didn't notice the NPE on this PR...). It's because we're checking if the processor is valid before enabling expression validation (https://github.com/apache/nifi/blob/master/nifi-mock/src/main/java/org/apache/nifi/util/StandardProcessorTestRunner.java#L169). We can't really do it the other way around without changing a lot of things.
I updated the PR to just check if expectExpressions is null and, if yes, return false. This way we can use isExpressionLanguagePresent() in a custom validate method.

I talked to @markap14 about it, perhaps this fix is fine or we can just change it to a boolean, but I'll let him take a look too.

@pvillard31 Mind doing a rebase here, and updating the QUERY_TIMEOUT property to use FlowFile Attribute scope? I pushed up a rebased branch with the additional commit (https://github.com/mattyb149/nifi/commit/40b9d1db89168fac08f343be772516132f1f67c0) but I don't know if you can cherry-pick from there or if you have to do your own rebase, then cherry-pick my additional commit (if you want to use it of course).

Done @mattyb149 - thanks!

Hey @mattyb149 - I believe we added this one for Hive 3 processors but forgot this PR. I know you're not available at the moment, but just a reminder for when you're back ;) (or if someone else wants to merge it in)

finally got time to get back on this one... if you want to have another look @mattyb149


text:
NIFI-4445 Add support for ListS3Version2 API

Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit?

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

@aburkard can you confirm the JIRA you meant this for?  NIFI-4445 looks like a typo.
Thanks

@joewitt yep my bad, NIFI-4628 is the right issue.

authors: aburkard joewitt