text:
Test Apache GitBox

This is a test of Apache GitBox, the issue should be synced to the mailing list.

Looks good. Closed.

source: gharchive/issue

repo: apache/incubator-mesatee

license_type: permissive

license_source: github-api


text:
METRON-466: Full Dev Platform build sometimes fails

Tested on Full Dev and Quick Dev.
Ambari seems to have changed their response in 2.4.0.1-1, if you request "START ALL" on a cluster where all services are already started, it doesn't actually make a request, so there is no request ID reported back.
I also suppressed starting of the Yaf probe (because we don't run the parser topology for Yaf) and bumped java_home so Snort would work.
I needed to separate the inventory directories for quick and full dev certain properties (in this case java_home) can lag between quick and full dev.
Note- Full dev still doesn't come up cleanly after the build successfully completes due to what looks like insufficient memory. I've opened METRON-467.

So, just recapping and making sure I understand; the issue is as follows: Yaf being started tips us over the edge memory-wise in full-dev, correct?
What we're debating here is whether we should not install yaf in quick-dev and full-dev or whether we should install it, but not start it, right?

What is our preferred way to manually add sensors that were not installed by default?  How would I add YAF back if I needed it?

@james-sirota - a couple of ways:
Pre-deploy:

Decide which parser/probe pair you wish to trade.
Toggle it off by setting install_(probe): false and add it to ansible-skip-tags in run.sh in place of 'yaf'.

Post-Deploy

Decide which parser/probe pair you wish to trade.
Kill the parsing topology and stop the associated sensor.
Toggle install_yaf to true.
Add yaf-parser to services to start.
./run_ansible_role "sensors,enrichment,monit,start"

Ok, just to clarify a bit; I think what is happening is that the yaf topology (which does exist) does not run by default (this was changed a while back because of resource constraints), but that does not prevent it from installing the yaf probe that pushes data to kafka.  It's the yaf probe that @dlyle65535 is stopping by not installing it.
In general, just my $0.02, but I tend think the proper behavior is as follows:

All sensors get installed by default; bro, snort, yaf, etc.
None of them get started by default in either full-dev or quick-dev.  If you want them to start, turn them on via monit

I'm not sure this should be within the scope of this PR, which is why @dlyle65535 made METRON-467.  As for what we do in the interrim, I'd say that either:

we either are ok with not installing yaf
we make the sensors in the services_to_start not start, not just the topologies.

Thoughts and corrections would be greatly appreciated. :)

just by the way, the above comments are just my opinions.  More discussion about the proper way to deal with the resource constraints long-term can and should happen either on the dev list or on METRON-467

+1 to this.  Here is why.  If fulldev can only support 2 running parser topologies I think it makes sense to let a user pick which 2 they want.  The intention of vagrant dev images is to spin them up to test a PR and then destroy them.  We generally only need a few topologies for that and it is not the intention to use vagrant images to simulate a full running system.  By having only 2 topologies started by default we will avoid 90% of problems people have when they spin up the vagrant images and run out of memory.  With that said I will also document what needs to be done to spin up the image with 4 topologies, but put a disclaimer to start at their own risk.

@james-sirota sounds good to me.  If we're going down this road, we might consider providing an argument to run.sh to specify up to 2 topologies as a follow-on JIRA.

@nickwallen - I think the discussion has died down. Are you good with me committing this?

Yep, I'm good.  We just need to be sure to update all docs around this at some point.

Thanks!

source: gharchive/pull-request

authors: cestella dlyle65535 james-sirota nickwallen

repo: apache/incubator-metron

license_source: bigquery


text:
Clip_gradient setting in gluon.trainer doesn’t work

Setting clip gradients in Trainer doesn’t work.
mxnet version: 1.2.0
Below code is used to clip gradients to [-2.0, 2.0]. With this setting, I printed the weights gradients of the last convolutional layer, and it turns out that some gradients are still much larger than 2.0.
trainer = gluon.Trainer(ssh_net.collect_params(), ‘sgd’, {‘learning_rate’:1e-4, ‘wd’:5e-4, ‘clip_gradient’: 2., ‘lr_scheduler’:lr_sch, ‘momentum’:0.9})

hi, I also meet this problem. Do you have sovled this problem?

Same here!

the gradient clipping happens inside optimizer update and is not updated back to the gradient buffers.

@szha Thank you very much for the answer; how do you use the clip_global_norm with a hybridized network?

@szha Also, clip_global_norm function does a rescaling and not a clipping. We just want want to ignore the gradients which are larger than some threshold.

@szha Also, clip_global_norm function does a rescaling and not a clipping. We just want want to ignore the gradients which are larger than some threshold.

I see. So you want to clip the absolute value of each dimension in the gradients instead of rescale based on global norm. You can do the following in this case:
for p in net.collect_params().values():
    if p.grad_req != 'null':
        for g in p.list_grad():
            mx.nd.clip(g, MIN, MAX, out=g)

Thanks a lot for the help @szha ; I've tried your proposed solution and done many different simulations but in each case, the problem persisted (though the gradient explosion is deferred to later stages of the learning)

authors: evi-Genius onurcanbektas szha zhaolili

repo: apache/incubator-mxnet


text:
undefined reference to `MXPredSetInput'

/home/lch/project/face/face_c/retinaface/./src/insightface_api.cpp:85: undefined reference toMXPredSetInput'
/home/lch/project/face/face_c/retinaface/./src/insightface_api.cpp:86: undefined reference to MXPredForward' /home/lch/project/face/face_c/retinaface/./src/insightface_api.cpp:91: undefined reference to MXPredGetOutputShape'
/home/lch/project/face/face_c/retinaface/./src/insightface_api.cpp:98: undefined reference to MXPredGetOutput'
I compile the C++ API of mxnet, and I -l the include path correctly.
I #include <mxnet/c_predict_api.h>
I've tried:

LD_LIBRARY_PATH=/home/lch/project/mxnet/lib/:$LD_LIBRARY_PATH
I compile mxnet using make USE_CPP_PACKAGE=1

I am facing the same issue. Did anyone solve it?


text:
[RFC] Build with MKL-DNN (or DNNL)

From https://github.com/apache/incubator-mxnet/issues/19610:

Intel MKL-DNN was renamed with DNNL in its v1.1 release. Since then, the MXNet community has been working on the transition to DNNL to leverage the latest features and optimizations from the library. That includes using the string “DNNL” or “dnnl” for future development and communication. We propose to promote the flag “USE_DNNL” since MXNet 2.0 and start deprecating “USE_MKLDNN” at the same time.
DNNL source code resides in the 3rdparty/mkldnn folder of the MXNet repository and is released and distributed along with MXNet source code. If one wants to build MXNet with DNNL to accelerate the execution on Intel CPU, she/he needs to enable -DUSE_DNNL=ON in CMake. However, this flag has been set to ON by default for all platforms except edge devices. On the contrary, to disable the DNNL acceleration, one needs to set -DUSE_DNNL=OFF explicitly in the CMake command line or the CMake configuration file.
As both MXNet and DNNL are under quick development with different release cadence, we decide to link the DNNL library into MXNet statically to avoid mis-linking in the user's environment. Given this, we need to set DNNL_LIBRARY_TYPE to STATIC when building DNNL.
Some additional flags to build DNNL:
DNNl_CPU_RUNTIME: Need set it to SEQ explicitly when USE_OPENMP=OFF;
DNNL_ARCH_OPT_FLAGS: Need pass compiler options to this build flag in string. Eg. -march or -mtune for GCC.
MKLDNN_BUILD_TESTS and MKLDNN_BUILD_EXAMPLES: We set these two flags to OFF to speed up the compilation.
One thing that needs to be taken care of is that the header dnnl_config.h and dnnl_version.h will be generated dynamically during compilation and will be copied to the installation destination when calling make install. That means these two headers are not distributed with DNNL source code. For downstream projects which are including these headers need to find them in the installation path rather than the source code path.

I prepared three commits regarding this point of main RFC:

changing USE_MKLDNN flag name to USE_ONEDNN to make it consistent with actual library name
I believe that this commit is complete and if there is such a will can be merged into master.

changing MXNET_USE_MKLDNN* flags names to MXNET_USE_ONEDNN* also for consistency reasons
This commit regards changing inner MXNET flag so that it will be consistent with the actual lib name. To avoid creating even bigger mixture of mkldnn/dnnl/onednn acronyms I believe it should be accompanied with another commit changing acronyms used in mxnet function names and comments regarding this particular lib to oneDNN.

changing the 3rdparty/mkldnn folder name to 3rdparty/onednn for consistency.

CC: @leezu, @TaoLv, @anko-intel, @szha, @akarbown

@mxnet-bot run ci [centos-gpu, windows-cpu]

@mxnet-bot run ci [windows-gpu]

There is also another idea to change the 3rdparty/mkldnn folder name to 3rdparty/onednn for consistency. The problem is that it will create unnecessary difference in directories names between branches.

since this is the master branch, and we're moving from 1.x to 2.0 shouldnt this be the time to make this change? Having a directory named "mkldnn" but using "onednn" everywhere will only cause confusion in the future for those who do not have the historical background.

Added commit regarding changing name of the submodule directory.

I think that the commits regarding updating onednns old acronyms used in mxnet in different contexts can be added to separate PR and if there is no objection this one can be already merged into master to avoid creating code with deprecated versions of the flags.

I think that the commits regarding updating onednns old acronyms used in mxnet in different contexts can be added to separate PR and if there is no objection this one can be already merged into master to avoid creating code with deprecated versions of the flags.

Can you give an example, which acronyms are you referring to?

I think that the commits regarding updating onednns old acronyms used in mxnet in different contexts can be added to separate PR and if there is no objection this one can be already merged into master to avoid creating code with deprecated versions of the flags.

Can you give an example, which acronyms are you referring to?

Before v1.1 of oneDNN it was called MKLDNN, then dnnl up until some time ago. Due to that we have a mixture of different acronyms used in the code and unifying them using the most recent one can help avoid some unnecessary confusion.

I adjusted the change of USE_MKLDNN flags to USE_ONEDNN by changing the library name in comments around it.
The changes regarding the USE_MKLDNN compilation flag, submodule directory name and the MXNET_USE_(MKL)ONEDNN seem to be complete.
After team discussion we agreed that as having all oneDNNs acronyms unified to one is desirable, it is impossible to be achieved as long as there is no update of oneDNNs functions/flags names from dnnl*/mkldnn* nomenclature. Therefore we believe that further efforts to unify oneDNNs acronyms should be postponed at least till we get an updated api.

Will this cause any incompatible issue for downstream projects? I'm not sure if there is any using the master branch and the alpha release is already in process.

@TaoLv it might if somebody is trying to build mxnet without mkldnn, as the old compilation flag will not work.

@leezu, do we claim any API compatibility on an alpha release or is it possible to include this into the alpha release?
@bartekkuncer, do you think we can keep the old build flag and raise a warning when it's set?

@leezu, do we claim any API compatibility on an alpha release or is it possible to include this into the alpha release?
@bartekkuncer, do you think we can keep the old build flag and raise a warning when it's set?

@TaoLv I believe we might keep it for some time. Will add it by eod.

@bartekkuncer, do you think we can keep the old build flag and raise a warning when it's set?

This is a good idea, but we can break things. Alpha is more of a preview than something that requires API compatibility. We can merge this change now and it can show up in the Beta or the final release.

@bartekkuncer, do you think we can keep the old build flag and raise a warning when it's set?

This is a good idea, but we can break things. Alpha is more of a preview than something that requires API compatibility. We can merge this change now and it can show up in the Beta or the final release.

@samskalicky I did not comment on that but already added back the support for an old flag: https://github.com/apache/incubator-mxnet/pull/19944/commits/020c892e9509e0a8a3ca5298d802fbf8e371812a.

@mxnet-bot run ci [all]

Fixed merge conflicts.

@mxnet-bot run ci [all]

@mxnet-bot run ci [sanity]

@mxnet-bot run ci [sanity,unix-cpu,windows-gpu]

@mxnet-bot run ci [sanity,unix-cpu]

@mxnet-bot run ci [sanity,unix-cpu]

@mxnet-bot run ci [sanity]

@mxnet-bot run ci [sanity]

Does anyone know why sanity check is stuck in "Expected — Waiting for status to be reported" and "@mxnet-bot run ci [sanity]" does nothing?
@samskalicky @leezu @szha Do you have any concerns regarding this change? If not I believe we can merge it into master.

@bartekkuncer no idea. Because sanity check is a required check, committers can't click on merge button if it's not green.

It complains that the commit is not mergeable: https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fsanity/detail/PR-19944/15/pipeline#log-42
@bartekkuncer would you mind doing a rebase to the latest master branch and do a force push?

It complains that the commit is not mergeable: https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fsanity/detail/PR-19944/15/pipeline#log-42
@bartekkuncer would you mind doing a rebase to the latest master branch and do a force push?

@szha Sure, thanks for help. It is quite strange that the merging worked for all other checks.

@mxnet-bot run ci [sanity]

@mxnet-bot run ci [unix-cpu]

@mxnet-bot run ci [unix-cpu]

@szha Sure, thanks for help. It is quite strange that the merging worked for all other checks.

I suspect the status for the other checks was outdated. For cost savings, the Sanity check is triggered automatically by Github if needed and the other checks are only retriggered if the Sanity check passed.

@mxnet-bot run ci [website,windows-gpu]

@mxnet-bot run ci [website,windows-gpu]

@mxnet-bot run ci [website,windows-gpu]

@mxnet-bot run ci [unix-cpu]

@mxnet-bot run ci [unix-cpu, centos-cpu]

@mxnet-bot run ci [unix-cpu, centos-cpu]

@mxnet-bot run ci [unix-gpu]

@mxnet-bot run ci [unix-gpu]

@szha @leezu @samskalicky Please help with the merge.

authors: TaoLv bartekkuncer leezu samskalicky szha


text:
Add support for PolarFire SoC and icicle board

Summary
Support for PolarFire RISC-V SoC and Icicle evaluation board
Impact
Only new features no changes to old code
Testing
Tested with Icicle board.
Nsh works and ostest runs fine.

Thanks everyone! 🎇

authors: jlaitine jrosberg

repo: apache/incubator-nuttx


text:
Use actionproxy from dockerSkeleton do not clown and own is the motto FTW!

Closes #3

No more clowning around!

@rabbah bump

authors: csantanapr rabbah


text:
The Load balancer can't schedule the last invoker when exist more than one invokers

After installed multiple invokers, access http://controller:10001/invokers, all the invoker's status is up, but when execute wsk -i action invoke --result <action> multiple times, the last invoker can't be scheduled by controller's loadbalancer to deal the action request.
I do several tests as below(in centos7, memory:16G, cpu:8 physical cpu total, 8 logic cores total)

test case one: only deploy two invokers
status for invokers: {"invoker0":"up","invoker1":"up"}
only invoker0 deals the request,  invoker1 seems don't recevie the request

test case one: deploy three invokers
status for invokers: {"invoker0":"up","invoker1":"up", "invoker2":"up"}
only invoker0 and invoker1 deal the request, invoker2 don't recevie the request.

test case one: deploy four invokers
status for invokers: {"invoker0":"up","invoker1":"up", "invoker2":"up","invoker3":"up"}
only invoker0 and invoker1  and invoker2 deal the request, invoker3 don't recevie the request.

test case one: deploy five invokers
status for invokers: {"invoker0":"up","invoker1":"up", "invoker2":"up","invoker3":"up","invoker4":"up"}
only invoker0 and invoker1  and invoker2 and invoker3 deal the request, invoker4 don't recevie the request.

So my test conclusion is that: since all invokers's status is ok, may be there has some bug on controller's load balancer‘s scheduling strategy, it leads to the last invoker can't be sheduled by load balancer to deal the action request

The last invoker is per default only used for blackbox actions. The fraction of invokers used for blackbox invocations is configurable but it's always at least one invoker (today). Arguably, we can adjust that logic to be disablable.

What's mean for  The blackbox actions, i do not know what role he has.
Where the configuration for the fraction of invokers?
regarding Arguably, we can adjust that logic to be disablable., you mean remove the blackbox actions used the last invoker logic?

Blackbox actions are actions based on user-provided docker images. Configuration is per environment variable and the code you posted.
We can adjust that logic, so you can completely disable that blackbox actions get their own invokers and are rather shared with all the other invokers.

This is working as designed.

Does the change to make any invoker pick up blackbox image require changes in source to "ShardingContainerPoolBalancer.scala" or is this configurable in the env variable somewhere?

@hsane2001 this PR https://github.com/apache/incubator-openwhisk/pull/3751 makes it possible to use all the invokers (for small N). Otherwise we don't have a way to disable the blackbox isolation when your N is high enougn (10 invokers for a 10% fraction).

authors: hsane2001 markusthoemmes ningyougang rabbah

repo: apache/incubator-openwhisk


text:
Refactor gcs

Description

Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

[ ] Yes (Please label as backward-incompat, and complete the section below on Release Notes)

Does this PR fix a zero-downtime upgrade introduced earlier?

[ ] Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

Does this PR otherwise need attention when creating release notes? Things to consider:

New configuration options
Deprecation of configurations
Signature changes to public methods/interfaces
New plugins added or old plugins removed

[ ] Yes (Please label this PR as release-notes and complete the section on Release Notes)

Release Notes

Documentation

Codecov Report

Merging #6841 (5bf20e8) into master (ac5aee7) will decrease coverage by 0.00%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master    #6841      +/-   ##
============================================
- Coverage     43.52%   43.51%   -0.01%
  Complexity        7        7
============================================
  Files          1421     1421
  Lines         69142    69145       +3
  Branches       9987     9988       +1
============================================
- Hits          30093    30091       -2
- Misses        36504    36516      +12
+ Partials       2545     2538       -7

Flag
Coverage Δ
Complexity Δ

integration
43.51% <0.00%> (-0.01%)
7.00 <0.00> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ
Complexity Δ

.../java/org/apache/pinot/spi/filesystem/PinotFS.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

...ion/groupby/AggregationGroupByTrimmingService.java
66.66% <0.00%> (-28.99%)
0.00% <0.00%> (ø%)

...readers/constant/ConstantMVForwardIndexReader.java
14.28% <0.00%> (-28.58%)
0.00% <0.00%> (ø%)

...ot/segment/local/customobject/MinMaxRangePair.java
75.86% <0.00%> (-24.14%)
0.00% <0.00%> (ø%)

...pache/pinot/core/util/SortedRangeIntersection.java
73.52% <0.00%> (-11.77%)
0.00% <0.00%> (ø%)

.../filter/predicate/InPredicateEvaluatorFactory.java
55.17% <0.00%> (-11.50%)
0.00% <0.00%> (ø%)

...lter/predicate/NotInPredicateEvaluatorFactory.java
48.95% <0.00%> (-10.42%)
0.00% <0.00%> (ø%)

...nction/DistinctCountBitmapAggregationFunction.java
42.78% <0.00%> (-10.31%)
0.00% <0.00%> (ø%)

...t/local/segment/index/readers/FloatDictionary.java
70.00% <0.00%> (-10.00%)
0.00% <0.00%> (ø%)

.../local/segment/index/readers/DoubleDictionary.java
60.00% <0.00%> (-10.00%)
0.00% <0.00%> (ø%)

... and 41 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ac5aee7...5bf20e8. Read the comment docs.

This is an awesome fix! Many thanks @elonazoulay !

authors: codecov-commenter elonazoulay xiangfu0

repo: apache/incubator-pinot


text:
Make all cli arguments follow consistent - notation instead of camelcase

Motivation
Explain here the context, and why you're making that change.
What is the problem you're trying to solve.
Modifications
Describe the modifications you've done.
Result
After your change, what will change.

run java8 tests

run java8 tests

repo: apache/incubator-pulsar


text:
Support setting private key content in athenz client plugin

Motivation
We can't set private key content, but it can only be set file path in athenz client plugin.
Modifications
Supports setting private key content in athenz client plugin.
Result
We can set private key content instead of file path.

Next, we plan to modify c++ athenz client plugin.

I don't know much about athenz but why is this needed? What would be the benefits?
If we provide multiple ways to achieve something,  we should provide their pros and cons too so that users can make a right decision. Otherwise it would introduce confusion.

In case of PCF, we should not deploy private key as a file in terms of security and auto scale, we should set private key content to Environment variable.
so we added this modification and we think this is useful in the similar case.

👍 this also makes it easier to integrate with a CKMS. the file name interface is the one that should go away, but that would be an API breaking change.

what about getting rid of the privateKeyPath field, and during init load the content of the file if privateKeyPath is in the config?  then getZtsClient can lose the branch.

OK, that makes sense. Since it's just a plugin I don't object introducing its own unique way, however, it sounds like not athenz specific issue. The same issue will appear if somebody tried to use TLS auth, and they would need to re-implement the logic to read secrets from somewhere. Also, all users who read it from environment variables need to implement the logic even though it's not an unique way.
How about adding support for URL schemes (e.g. file:, data:, env:, exec:) to privateKeyPath config param? If we provided the reader and plugins used it, the plugins would get the same benefit when we support new scheme (new data source), plus it can be applicable for any configuration parameters.

How about adding support for URL schemes (e.g. file:, data:, env:, exec:) to privateKeyPath config param? If we provided the reader and plugins used it, the plugins would get the same benefit when we support new scheme (new data source), plus it can be applicable for any configuration parameters.

+1 for this. In container deployements line Kubernetes, is common to have private key in files that are mounted as "secrets" and seen as files by the application.

OK. We will consider it.

We discussed support of URL scheme and we will implement it by the following:
Add new field(privateKey) to Athenz client plugin's authParams and this fields support URL scheme(file: and data:).
following three patterns are supported:
file:/path/to/file
file:///path/to/file

// application/x-pem-file;base64, is fixed.
data:application/x-pem-file;base64,<base64-encoded value>

Also, We will deprecate privateKeyPath field.
If there is anything that concerns you, please tell me.

PTAL

👍

@maskit
Thank you for your checking.
I created issue( https://github.com/apache/incubator-pulsar/issues/713 ) about document and in the future we will consider changes such as providing as part of pulgin api so that the specification will not be different.

authors: hrsakai maskit merlimat msb-at-yahoo nkurihar


text:
[License Issue] Please update your source codes' license

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
I accidentally noticed this PR, https://github.com/apache/incubator-seatunnel/pull/912/files#diff-a0275ef336fd0cf14c3d4c026ce3e918301dc0cf7b2663fbf484d9cff5288bd2, which carried the dependency license check script into your source codes. And the same style of known-dependencies.txt file, which was created by SkyWalking.
But you should know, this script is coming(copied) from Apache SkyWalking, which was written by one PMC member. And I noticed this script was not submitted by himself(he still owns this file). This file's owner has reviewed this PR, but I think he missed this. Unless he is the one to submit, the License should be taken care of.
So, you should update the source codes' LICENSE to indicate you have copied it from another project.
FYI @CalvinKirs @zhongjiajie
SeaTunnel Version
The dev branch
SeaTunnel Config
No need.

Running Command
No need.

Error Exception
No need.

Flink or Spark Version
No response
Java or Scala Version
No response
Screenshots
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Thanks for the reminder, I'll deal with this asap.

authors: CalvinKirs wu-sheng

repo: apache/incubator-seatunnel


text:
How to develop a new Connector？

Discussed in https://github.com/apache/incubator-seatunnel/discussions/4723

Originally posted by YuanjunXu May  9, 2023
I want to develop a new Connector.
What should I do or what interfaces should I implement？

You can see https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-connectors-v2/README.zh.md

thx


text:
[Feature][Connector-V2] Add oss sink

Purpose of this pull request
#1946 Oss sink connector

Check list

[x] Code changed are covered with tests, or it does not need tests for reason:
[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

LGTM


text:
Create python-vector-osm.md

I've found it difficult to find more complex examples at this documentation so I made this one, that involves spark slaves, osm data and hdfs. I hope it can be helpfull to someone else in the future.
Is this PR related to a proposed Issue?
No
What changes were proposed in this PR?
A new documentation example to guide other developers, in a more complex environment, closest to reality of some needs they may have as well
How was this patch tested?
The examples were tested at jupyter, connected to a spark infrastruture, with 4 slaves machines
Did this PR include necessary documentation updates?
Yes

@carolgit Thanks for your contribution! I think your tutorial is more suitable for a standalone Jupyter notebook example.
Can you make it a runnable example on binder and add it to our binder examples? https://github.com/apache/incubator-sedona/tree/master/binder

@jiayuasu ,
I can take that if it's OK.

@netanel246 Please go ahead :-)

@netanel246 I will merge this PR and move the notebook to binder/contrib. You can still work on the notebook if you want.

authors: carolgit jiayuasu netanel246

repo: apache/incubator-sedona


text:
SpringCloud plugin register center init

Now we used Springcloud plugin like this

add register center in soul-boostrap

in yaml
eureka:
client:
serviceUrl:
defaultZone: http://localhost:8761/eureka/
instance:
prefer-ip-address: true
in pom and add this
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-commons</artifactId>
        <version>2.2.0.RELEASE</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-ribbon</artifactId>
        <version>2.2.0.RELEASE</version>
    </dependency>

Fetrue:
condition1 ：user config register and enabled to spring-cloud plugin  in the soul-admin
condition2： this config we sync soul-boostrap
at this time we init springcloud register center。
we support Nacos and eureka and other registry

pls see https://shenyu.apache.org/docs/design/register-center-design

repo: apache/incubator-shenyu


text:
Add a FAQ about SkyWalking upgrade

Many people (ex #1029) face the SkyWalking upgrade, but miss an important point, which is ES types should be created, all data need to remove.
I hope @peng-yongsheng could provide a document to description this.

Create a new file named Upgrade.md, describe all the upgrade FAQ in this file. Create a new section with the title of our project version and use subsection to separate UI, agent, collector when this version have some notices about upgrading.

Done

authors: peng-yongsheng wu-sheng

repo: apache/incubator-skywalking


text:
Markdown 3 doesn't seem to work

Installed superset from pypi today on Ubuntu and the version of Markdown it pulled (3.x) was incompatible with Superset and it refused to run db install commands etc on the command line.
Superset version 0.27 I believe

pip install markdown==2.6.11

repo: apache/incubator-superset


text:
Permission Issue With Admin User

Problem
With a clean (first time) installation without docker, I got an error popup at the bottom of the landing pages (daskboards) -- "You don't have the necessary permissions to load dashboards. Please contact your administrator".  Clicking on "profile" gives same error message.
Expected
No such error message
Screenshot:

How to reproduce the issue
pip3.6 install apache-superset --user --upgrade
superset db upgrade
export FLASK_APP=supersetexport FLASK_APP=superset
superset fab create-admin
superset load_examples
superset run -p 8088 --with-threads --reload --debugger

superset version: 0.36.0
python version: 2.7.5 (I have python 3.6 installed as well but not sure which one superset uses)
node.js version: not installed as it is not listed as required here nor as optioal
npm version: not installed.

Checklist
Make sure these boxes are checked before submitting your issue - thank you!

[X] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

stdout:
logging was configured successfully
INFO:superset.utils.logging_configurator:logging was configured successfully
/home/hc/.local/lib/python3.6/site-packages/flask_caching/__init__.py:189: UserWarning: Flask-Caching: CACHE_TYPE is set to null, caching is effectively disabled.
  "Flask-Caching: CACHE_TYPE is set to null, "
 * Serving Flask app "superset" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit)
INFO:werkzeug: * Restarting with stat
logging was configured successfully
INFO:superset.utils.logging_configurator:logging was configured successfully
/home/hc/.local/lib/python3.6/site-packages/flask_caching/__init__.py:189: UserWarning: Flask-Caching: CACHE_TYPE is set to null, caching is effectively disabled.
  "Flask-Caching: CACHE_TYPE is set to null, "
WARNING:werkzeug: * Debugger is active!
INFO:werkzeug: * Debugger PIN: 271-279-629
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET / HTTP/1.1" 302 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /superset/welcome HTTP/1.1" 302 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /login/ HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/appbuilder/css/bootstrap.min.css HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/appbuilder/datepicker/bootstrap-datepicker.css HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/appbuilder/css/flags/flags16.css HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/appbuilder/css/font-awesome.min.css HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/appbuilder/css/ab.css HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/appbuilder/select2/select2.css HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/appbuilder/js/ab_filters.js HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/appbuilder/js/ab_actions.js HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/assets/theme.e76619c6a440fc07fa28.entry.css HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/assets/theme.e76619c6a440fc07fa28.entry.js HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/appbuilder/js/jquery-latest.js HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/appbuilder/js/bootstrap.min.js HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/appbuilder/datepicker/bootstrap-datepicker.js HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/appbuilder/js/ab.js HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/appbuilder/select2/select2.js HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/assets/images/superset-logo@2x.png HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/assets/vendors-addSlice-preamble-profile-showSavedQuery-sqllab-welcome.7e152e5dd5d1e28b7651.chunk.js HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/appbuilder/img/flags/flags16.png HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/assets/preamble.e22606bd0dac72848a12.entry.js HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/assets/1e081edc16d92d42aeccec760174fbf4.woff2 HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/assets/e7ae98681edfa1df7f1e3ebba0d4fb88.woff2 HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/appbuilder/fonts/fontawesome-webfont.woff2?v=4.7.0 HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/assets/vendors-major.26c6716120c96d10daf2.chunk.js HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:24] "GET /static/assets/images/favicon.png HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:30] "POST /login/ HTTP/1.1" 302 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:30] "GET / HTTP/1.1" 302 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:30] "GET /superset/welcome HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:30] "GET /static/assets/welcome.cb9ef677374cffc1955d.entry.css HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:30] "GET /static/assets/images/loading.gif HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:30] "GET /static/assets/welcome.cb9ef677374cffc1955d.entry.js HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:30] "GET /api/v1/dashboard/?q={%22order_column%22:%22changed_on%22,%22order_direction%22:%22desc%22,%22page%22:0,%22page_size%22:25} HTTP/1.1" 401 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:30] "GET /superset/fave_slices/1/ HTTP/1.1" 401 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:30] "GET /superset/fave_dashboards/1/ HTTP/1.1" 401 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:30] "GET /superset/recent_activity/1/?limit=50 HTTP/1.1" 401 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:53] "GET /users/userinfo/ HTTP/1.1" 302 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:53] "GET /login/?next=http%3A%2F%2Flocalhost%3A8888%2Fusers%2Fuserinfo%2F HTTP/1.1" 302 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:53] "GET / HTTP/1.1" 302 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:53] "GET /superset/welcome HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:53] "GET /superset/fave_slices/1/ HTTP/1.1" 401 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:53] "GET /api/v1/dashboard/?q={%22order_column%22:%22changed_on%22,%22order_direction%22:%22desc%22,%22page%22:0,%22page_size%22:25} HTTP/1.1" 401 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:53] "GET /superset/fave_dashboards/1/ HTTP/1.1" 401 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:37:53] "GET /superset/recent_activity/1/?limit=50 HTTP/1.1" 401 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:04] "GET /users/userinfo/ HTTP/1.1" 302 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:04] "GET /login/?next=http%3A%2F%2Flocalhost%3A8888%2Fusers%2Fuserinfo%2F HTTP/1.1" 302 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:04] "GET / HTTP/1.1" 302 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:04] "GET /superset/welcome HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:05] "GET /api/v1/dashboard/?q={%22order_column%22:%22changed_on%22,%22order_direction%22:%22desc%22,%22page%22:0,%22page_size%22:25} HTTP/1.1" 401 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:05] "GET /superset/recent_activity/1/?limit=50 HTTP/1.1" 401 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:05] "GET /superset/fave_slices/1/ HTTP/1.1" 401 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:05] "GET /superset/fave_dashboards/1/ HTTP/1.1" 401 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:54] "GET /chart/add HTTP/1.1" 302 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:54] "GET /login/?next=http%3A%2F%2Flocalhost%3A8888%2Fchart%2Fadd HTTP/1.1" 302 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:54] "GET / HTTP/1.1" 302 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:54] "GET /superset/welcome HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:54] "GET /superset/recent_activity/1/?limit=50 HTTP/1.1" 401 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:54] "GET /superset/fave_dashboards/1/ HTTP/1.1" 401 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:54] "GET /api/v1/dashboard/?q={%22order_column%22:%22changed_on%22,%22order_direction%22:%22desc%22,%22page%22:0,%22page_size%22:25} HTTP/1.1" 401 -
INFO:werkzeug:127.0.0.1 - - [06/May/2020 15:38:54] "GET /superset/fave_slices/1/ HTTP/1.1" 401 -

Closest, similar issue is this

@hc128168 you also need to run superset init

Good catch @villebro !


text:
[docs] Add Deepomatic in organization list

CATEGORY

[x] Documentation

SUMMARY
Add Deepomatic in organization list.

Codecov Report

Merging #8347 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #8347   +/-   ##
=======================================
  Coverage   67.69%   67.69%
=======================================
  Files         447      447
  Lines       22653    22653
  Branches     2363     2363
=======================================
  Hits        15336    15336
  Misses       7179     7179
  Partials      138      138

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9596935...7435c46. Read the comment docs.

authors: Zanoellia codecov-io


text:
Add c++ and python local deploy example

Issue:
Currently there is no an example to show how to deploy VTA with c++ and no example for python local deployment.
Solution:
This patch is the example to show how to deploy vta with c++ and python in local.

Very exciting development @huajsj ! I'm looking forward to the C++ example!

@tmoreau89 , thanks for the kindly follow up, the code& test done, if you have time, could you help to have a review?  Thanks.

@tmoreau89 , all review comments get addressed, please kindly check.

@tmoreau89 , could you help to review this patch again? Thanks
Regards
Hua

Thanks so much @huajsj the PR has been approved and merged.

repo: apache/incubator-tvm-vta


text:
Add support for quant. mul operator in tflite frontend

A test for qnn_mul has to be added when the qnn elemwise tests (#4282) get merged.

@kevinthesun @FrozenGene can you please take a look at this?

Hi @inadob
Should we add the test case for quant.mul operator in tflite frontend?

@cchung100m
I can add the tests on my own as soon as we agree on the #4282 implementation and merge it. I believe we can merge this patch now as there are a few other qnn operators that are already added relying on end-to-end testing by running the qnn models at the end of test_forward.py

LGTM, but you need an approver @anijain2305 , @FrozenGene ?

Hi @inadob , as your other PR is merged, will it be ok to add the test for mul op?
Sorry for the delay.

Thanks @u99127 @inadob @cchung100m

repo: apache/incubator-tvm


text:
[REFACTOR][IR] Unified IR Primitive Op and Registry

This PR migrates relay's Op into the ir folder.
Op and its registry provides an useful mechanism to
store any attribute meta-data of an operator include
function signatures, lowering rules, side effect etc.
These features are not only useful for Relay, but also needed in the low-level IR.
At the current moment, intrinsic functions in the low-level IR are simply
represented by a string. This means we cannot type-check the low-level IR
when the type does not meet the constraint, nor can we obtain further
information such as side-effect and read write relation of these intrinsics
wrt to arguments.
Op will be used as the way to handle primitive ops(in DL terminology)
(builtin intrinsics or in compiler terminology).
We will perform follow-up refactors to make low-level CallNode
take Op as the function argument.

depends on #4678 , cc @jroesch @zhiics @wweic @icemelon9 @yzhliu @ZihengJiang @Hzfengsy @merrymercy


text:
Minimize apache-rat excluded files

What changes were proposed in this pull request?
Add license header to files so we can minimize files to exclude in apache-rat check.
Why are the changes needed?
Asf podding issues
Does this PR introduce any user-facing change?
License header is visible in docs, just like spark, for example:
https://github.com/kaijchen/incubator-uniffle/blob/rat-exclude/docs/index.md
How was this patch tested?
CI: https://github.com/kaijchen/incubator-uniffle/runs/7176341386?check_suite_focus=true

Thanks @jerqi for the review.

repo: apache/incubator-uniffle


text:
[YUNIKORN-397] Makefile use go version from go.mod

Instead of hard coding a go version in the Makefile use the version
from the go.mod file. The build will fail if the installed tools version
is lower than the version from the go.mod file.

Pushing changes to build in all 3 repos


text:
Minor tweaks to the tutorial notebook to support Spark 1.3.0

Also a couple of minor grammar tweaks in the first paragraph.

Thanks for recreating PR. LGTM.

authors: Leemoonsoo langley

repo: apache/incubator-zeppelin


text:
[Feature][SDK] InlongSDK support retry sending when failed

Description
When InlongSDK fails to send data to DataProxy, it returns the failure to the user directly. Add the capability to retry sending.

Location:
inlong-sdk/dataproxy-sdk/
Requirements:

Choose another dataproxy node to resend data when the send fails.
Able to configure max retry times.
Check failures and only retry those retriable, discard those not.

This issue is a dedicated issue for the 2024 Tencent Rhino-bird Open Source Talent Training event and is only available for college students to claim
【Claim the issue】Sign up for the corresponding issue in columns M~R of the Research and Study Base "issue Camp" to be considered as successfully claimed.
【Complete the issue】For students who have claimed the issue, please also reply "Successfully claimed this issue" in the comment section of this issue; if there is no progress submitted within 7 days, including but not limited to comments, commits, or Pull Requests, it will be considered as voluntarily giving up the issue and the committee will release the issue to the next person in line.
Use case
No response
Are you willing to submit PR?

[ ] Yes, I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Successfully claimed this issue

authors: Hackerismydream vernedeng

repo: apache/inlong


text:
[Bug]

What happened

sort don't support tubemq?
What you expected to happen
use tubemq
How to reproduce
Hive Example
Environment
InLong 1.2.0
InLong version
1.2.0
InLong Component
InLong Sort
Are you willing to submit PR?

[X] Yes, I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Could you  use the master code?  It is not to support  in this release. @948843289


text:
Compact specified

Description
Currently, "merge" command can not specify a storage group.
this pr support the following command:
merge root.sg
Content1 ...
Content2 ...
Content3 ...

This PR has:

[ ] been self-reviewed.

[ ] concurrent read
[ ] concurrent write
[ ] concurrent read and write

[ ] added documentation for new or modified features or behaviors.
[ ] added Javadocs for most classes and all non-trivial methods.
[ ] added or updated version, license, or notice information
[ ] added comments explaining the "why" and the intent of the code wherever would not be obvious
for an unfamiliar reader.
[ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold
for code coverage.
[ ] added integration tests.
[ ] been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

Coverage increased (+0.1%) to 67.49% when pulling 43ce461d4012d61183427189deae0aef5bc58b53 on lyssom:compact_specified into 3333f3b2a6a5035b7f42a33dd23b3e841b0768b1 on apache:master.

authors: coveralls lyssom

repo: apache/iotdb


text:
[SONARLINT] Fixing a bunch of sonarlint wanrings across the project

Bug fixes
Performance enhancements
Code style refactorings

Rebase to solve conflicts

authors: Arsnael chibenwa

repo: apache/james-project


text:
MINOR: Add __cluster_metadata topic to list of internal topics

Details

Adds the new KRaft __cluster_metadata topic to INTERNAL_TOPICS which tracks internal topics

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Since we are going to make a potentially breaking change, I would be in favour of enforcing that any topic starting with __ is considered as internal topic for Kafka. It would prevent possibility of a similar situation in future when we introduce a new reserved topic for some reason.

repo: apache/kafka


text:
MINOR: change Streams topic-level metrics tag from 'topic-name' to 'topic'

Changes the tag name from topic-name to just topic to conform to the way this tag is named elsewhere (ie in the clients)
Also addresses the followup feedback from @cadonna on the original PR:

fixes a comment about dynamic topic routing
fixes some indentation in MockRecordCollector
Undoes the changes to KStreamSplitTest.scala and TestTopicsTest which are no longer necessary after this hotfix

Build failures are unrelated:
Build / JDK 17 and Scala 2.13 / org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationSSLTest.testReplication()

authors: ableegoldman cadonna


text:
KAFKA-16139: Fix StreamsUpgradeTest

Adds version 3.6 to the possible values for config upgrade_from.

This PR must be cherry-picked to 3.7 branch.

Merged to trunk and cherry-picked to 3.7 branch.


text:
KAFKA-16115: Adding missing heartbeat metrics

Add a HeartbeatMetrics module to measure various metrics related to the heartbeat. Here is the highlight of the changes:

HeartMetrics encapsulate a heartbeat sensor
Add metrics suffix types to the AbstractConsumerMetrics so that it can be extended to different metric groups

Non-related refactor:

Rename the metric classes to "MetricManager" the reason being "Metrics" seems to relate to the Metric class however these managers are merely containers holding sensor references for recording purposes.
Created OffsetCommitMetricsManager so that we are more consistent with the metrics management
Extended KafkaConsumerMetrics to the AbstractConsumerMetrics so that the groupName is consistently created (without random concatenation)

Some follow ups:

Refactor commit sensor by introducing a CommitMetrics module so that we can keep most of the metrics in one place and let it be a bit more consistent stylistically
Possibly refactor fetch manager metrics

Hi @lucasbru - This PR is ready for review.  I did a bit of refactor in this PR so not everything here is in the scope.  Let me know what do you think!

Thanks, @philipnee. I will not get to it today, but I will review it tomorrow.

I have one last comment, otherwise the PR looks good to me.

@lucasbru - Thank you for spending time reviewing the PR.  I responded to your comment about resource closing. It is indeed a bit strange why we need to do this.
A weird thing I found while reviewing sensor code is that - there's a cyclic relationship between sensor and metrics.  Sensor holds on to Metric's reference and Metrics holds on to sensor lookup map (which has Sensor as the val). See this
public class Metrics implements Closeable {

    private final MetricConfig config;
    private final ConcurrentMap<MetricName, KafkaMetric> metrics;
    private final ConcurrentMap<String, Sensor> sensors;

and
public final class Sensor {

    private final Metrics registry;

I wonder if this is why we should remove these sensors upon closing.

Hello @philipnee @lucasbru
Why you remove GroupState class in this commit?
This class will be used in java version of ConsoleGroupCommand when https://github.com/apache/kafka/pull/14471 will be fully merged
Class was merged to trunk in https://github.com/apache/kafka/commit/ff25c350a799f372ce6c56e14bbfe3bc7e088218 to simplify further reviews.

@nizhikov Sorry, during review I thought @philipnee removed a left-over class from an earlier PR also called GroupState. https://github.com/apache/kafka/blob/6e164bb9ace3ea7a1a9542904d1a01c9fd3a1b48/clients/src/main/java/org/apache/kafka/clients/consumer/internals/GroupState.java but instead he deleted a different class. I should have checked. Please, bring the class back and sorry for the confusion.

@lucasbru No worries. I wanted to make it clear that class removed by mistake and didn't broke something.
Will return it in the next PR regarding Console Group Command.

authors: lucasbru nizhikov philipnee


text:
[REST] Return more fields within session/batch data

:mag: Description
Issue References 🔗

This pull request fixes #
Describe Your Solution 🔧
Return sessionName and totalOperations within SessionData.
Return appStartTime with list batches.
Types of changes :bookmark:

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪
Behavior Without This Pull Request :coffin:
Behavior With This Pull Request :tada:
Related Unit Tests

Checklist 📝

[x] This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 58.44%. Comparing base (dd0aced) to head (5fb3fb8).
Report is 9 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #6359      +/-   ##
============================================
- Coverage     58.49%   58.44%   -0.06%
  Complexity       24       24
============================================
  Files           653      653
  Lines         39865    39894      +29
  Branches       5481     5481
============================================
- Hits          23319    23316       -3
- Misses        14055    14082      +27
- Partials       2491     2496       +5

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

thanks, merged to master

authors: codecov-commenter turboFei

repo: apache/kyuubi


text:
[Bug] linkis-cs-listener pom - wrong parent relative path

Search before asking

[X] I searched the issues and found no similar issues.

Linkis Component
linkis-public-enhancements
What happened + What you expected to happen
The pom is pointing to the wrong relative parent path, ending up in linkis-public-enhancements instead of linkis:
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>linkis</artifactId>
        <groupId>org.apache.linkis</groupId>
        <version>1.1.3</version>
        <relativePath>../../pom.xml</relativePath>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>linkis-cs-listener</artifactId>

should be
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>linkis</artifactId>
        <groupId>org.apache.linkis</groupId>
        <version>1.1.3</version>
        <relativePath>../../../pom.xml</relativePath>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>linkis-cs-listener</artifactId>

Relevent platform
IntelliJ, maven, macOs
Reproduction script
build with maven/intellij
Anything else
No response
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Thank you, you can submit PR to fix it, it is recommended to submit it to the branch dev-1.2.0

Without further feedback, I closed this issue. You can reopen if necessary

authors: casionone idoflax peacewong

repo: apache/linkis


text:
[Feature][Web] Refactoring of Linkis Management Console

当前主要优化方向

文件目录太乱（名字不规范/冗余文件夹），增加了开源用户参与的门槛
界面风格不统一，有不少比较原生的用法，存在不同页面风格不一致问题，考虑使用开源框架统一全部风格
使用iframe的页面，是否考虑使用微应用替换
国际化适配问题

规范相关

文件目录划分以及命名规范
国际化规范
图标license问题: 确定能使用的图标库，后续优先使用，如google 的icon库 https://marella.me/material-icons/demo/
无关代码的移除
前端代码规范和一些准则 如关键逻辑的注解
公共工具/组件的抽象 如加解密 单位换算等  需要补充对应的文档

新增的功能点
框架选型

基于已有框架的选型，需要给出对比参考
不少数据表格的条目会比较多，页面只会展示部分关键信息， 如历史任务页面，要考虑如何方便的展示其他不重要信息字段
设计风格上和wds的其他模块是否统一？还是优先考虑开源的组件

功能上

新增 菜单支持通过配置方式控制，后端接口返回的菜单的列表
支持简单的配置角色与菜单的关系
规划有面板指标页面 展示任务资源的图形面板

阶段
主要任务
参与人员
时间点

重构需求的收集
收集社区以及行内前/后端开发人员的一些想法和建议

选型方案的确认
输出对比方案要考虑开源情况，主流技术工作量 主要的工作工作内容社区同学能否比较好的上手参与 和dss当前的兼容性是否存在问题

UI设计
基于已有管理台，设计一套新的UI 主要是布局和风格  icon图库的确认

Demo版本
提供几个页面的demo版本，最好能覆盖主要的组件功能， 初始代码库以及规范文档，公共工具/组件的筹抽离

全量重构
进行前端代码的全量重构

意见收集
周鑫

vue版本，是否切换到vue3
菜单栏权限问题，显示隐藏由后台接口返回的菜单的控制。
同时前端项目的结构感觉没必要分为几个部分。统一为一个模块。
部分组件的可以使用已有的ui组件： 比如左侧菜单栏等。没必要自己重新实现。
权限，管理员和常用字典等数据需要单独接口返回，前端使用vuex统一管理
设计风格上和行内的其他模块是否统一
使用iframe的页面，能否使用微应用替换

BeaconTown

form-create灵活性很差，有个bug一直不知道咋处理，点新增的时候命名数据传了为空但是还是有数据填写好了
element ui我个人觉得好看的，目前用的iview感觉美观度一般，element ui官方有现成的表单组件

陆小龙

建议UI组件iview换一下哈哈   上ts

The current main optimization direction

The file directory is too messy (irregular names/redundant folders), which increases the threshold for open source user participation
The interface style is not uniform, there are many native usages, and there are inconsistencies in the styles of different pages. Consider using an open source framework to unify all styles
For pages that use iframe, whether to consider replacing them with micro-apps
Internationalization adaptation issues

specification related

File directory division and naming convention
Internationalization specification
Icon license problem: Determine the icon library that can be used, and use it first, such as Google's icon library [https://marella.me/material-icons/demo/](https://marella.me/material-icons/ demo/)
Removal of extraneous code
Front-end code specification and some guidelines such as annotations of key logic
The abstraction of public tools/components such as encryption and decryption, unit conversion, etc. need to supplement the corresponding documents

Added function points
Frame selection

Based on the selection of the existing framework, it is necessary to give a comparison reference
Many data tables will have many entries, and the page will only display some key information, such as the historical task page, it is necessary to consider how to conveniently display other unimportant information fields
Is the design style consistent with other modules in the industry? Still give priority to open source components

functionally

Added menu support to be controlled by configuration, and the list of menus returned by the backend interface
Support simple configuration of the relationship between roles and menus
There is a panel indicator page planned to display the graphic panel of task resources

Phase
Main Tasks
Participants
Time

Collection of Refactoring Requirements
Collect some ideas and suggestions from the community and front-end/back-end developers in the industry

Confirmation of selection plan
Output comparison planConsider open source situation, mainstream technologyMain work content of workloadCommunity students can Is it better to get startedIs there any problem with the current compatibility with dss

UI Design
Based on the existing management platform, design a new UI mainly layout and style confirmation of icon gallery

Demo version
Provide a demo version of several pages, preferably covering the main component functions, initial code base and specification documents, departure of public tools/components

Full Refactoring
Full refactoring of the front-end code

Comment collection
Zhou Xin

vue version, whether to switch to vue3
Menu bar permission problem, display and hide the control of the menu returned by the background interface.
At the same time, the structure of the front-end project does not feel necessary to be divided into several parts. unified into one module.
Some components can use existing ui components: such as the left menu bar, etc. There's no need to reimplement it yourself.
Permissions, administrators and commonly used dictionaries and other data need to be returned through a separate interface, and the front end uses vuex for unified management
Is the design style consistent with other modules in the line
Can the page using iframe be replaced by micro-app

Beacon Town

The flexibility of form-create is very poor. There is a bug that I don’t know how to deal with. When I click Add, the named data is passed as empty but there is still data to fill in.
I personally think that element ui is good-looking. The current iview is not as good-looking. Element ui officially has ready-made form components

Lu Xiaolong

It is recommended to change the UI component iview haha on ts

提供最新版DataSphereStudio全家桶安装服务。
包括（DataSphereStudio, Linkis, Exchangeis(Datax, Sqoop), Scriptis, Streamis(Flink), Visualis, DolphinScheduler)

另外包含 元数据，数据安全，数据质量，数据血缘。（这些都是DataSphereStudio不自带的服务）
包含hadoop集群搭建（hive, spark, flink, kafak, kylin）
提供上述所有系统二次定制开发服务
提供系统搭建运维疑难问题解决

添加备注，DSS大数据
持续为DataSphereStudio开源贡献力量

authors: YinDonglai casionone


text:
Springboot - not replace spring propreties in log4j.xml

Description
I use the log4j in the springBoot and i add the artifactId "log4j-spring-boot" for enable the use the springboot properties in the log4j.xml.
When i use inside the File, it's work very well, but when i try use in the File header, the log4j send the error.
In summary, below i pasted the code, but when i try use the ${spring:log4j.solver.service.path}, the system throw the error (log)
Configuration
Version: 2.19.0
Operating system: Windows 10
JDK: Java 17 (Adoptium)
application.properties file:
log4j.solver.service.name=${LOG4J_SOLVER_SERVICE_NAME:solver-service}
log4j.solver.service.version=${LOG4J_SOLVER_SERVICE_VERSION:1}
log4j.solver.service.enviroment=${LOG4J_SOLVER_SERVICE_ENVIROMENT:dev}
log4j.solver.service.path=${LOG4J_SOLVER_SERVICE_PATH:solver-service.log}`

LOG4J.XML file
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <Appenders>
        <File name="solver-service" fileName="${spring:log4j.solver.service.path}" append="false">
            <EcsLayout serviceName="${spring:log4j.solver.service.name}" serviceVersion="${spring:log4j.solver.service.version}"  serviceEnvironment="${spring:log4j.solver.service.enviroment}"/>
        </File>
        <Console name="STDOUT" target="SYSTEM_OUT">
            <PatternLayout pattern="%level - %m%n"/>
        </Console>
    </Appenders>
    <Loggers>
        <Logger name="propagate-logger" level="INFO" additivity="false">
            <AppenderRef ref="solver-service"/>
        </Logger>
        <Root level="INFO">
            <AppenderRef ref="STDOUT"/>
        </Root>
    </Loggers>
</Configuration>

Logs
Connected to the target VM, address: '127.0.0.1:52527', transport: 'socket'
2023-02-06 09:38:15,994 main ERROR FileManager (${spring:log4j.solver.service.path}) java.io.IOException: A sintaxe do nome do arquivo, do nome do diretório ou do rótulo do volume está incorreta java.io.IOException: A sintaxe do nome do arquivo, do nome do diretório ou do rótulo do volume está incorreta
	at java.base/java.io.WinNTFileSystem.canonicalize0(Native Method)
	at java.base/java.io.WinNTFileSystem.canonicalize(WinNTFileSystem.java:462)
	at java.base/java.io.File.getCanonicalPath(File.java:626)
	at java.base/java.io.File.getCanonicalFile(File.java:651)
	at org.apache.logging.log4j.core.util.FileUtils.makeParentDirs(FileUtils.java:139)
	at org.apache.logging.log4j.core.appender.FileManager$FileManagerFactory.createManager(FileManager.java:436)
	at org.apache.logging.log4j.core.appender.FileManager$FileManagerFactory.createManager(FileManager.java:423)
	at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:144)
	at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:100)
	at org.apache.logging.log4j.core.appender.FileManager.getFileManager(FileManager.java:183)
[application.properties.txt](https://github.com/apache/logging-log4j2/files/10663203/application.properties.txt)

	at org.apache.logging.log4j.core.appender.FileAppender$Builder.build(FileAppender.java:99)
	at org.apache.logging.log4j.core.appender.FileAppender$Builder.build(FileAppender.java:52)
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:124)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1138)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1063)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1055)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:664)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:258)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:304)
	at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:621)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:694)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711)
	at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
	at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
	at org.apache.commons.logging.LogAdapter$Log4jLog.<clinit>(LogAdapter.java:155)
	at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:122)
	at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:89)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:67)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:59)
	at org.springframework.boot.SpringApplication.<clinit>(SpringApplication.java:179)
	at net.weg.solverservice.SolverServiceApplication.main(SolverServiceApplication.java:18)

2023-02-06 09:38:15,997 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.FileAppender for element File: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.FileManager$FileManagerFactory@2b95e48b] unable to create manager for [${spring:log4j.solver.service.path}] with data [org.apache.logging.log4j.core.appender.FileManager$FactoryData@4a3329b9] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.FileManager$FileManagerFactory@2b95e48b] unable to create manager for [${spring:log4j.solver.service.path}] with data [org.apache.logging.log4j.core.appender.FileManager$FactoryData@4a3329b9]
	at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:146)
	at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:100)
	at org.apache.logging.log4j.core.appender.FileManager.getFileManager(FileManager.java:183)
	at org.apache.logging.log4j.core.appender.FileAppender$Builder.build(FileAppender.java:99)
	at org.apache.logging.log4j.core.appender.FileAppender$Builder.build(FileAppender.java:52)
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:124)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1138)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1063)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1055)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:664)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:258)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:304)
	at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:621)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:694)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711)
	at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
	at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
	at org.apache.commons.logging.LogAdapter$Log4jLog.<clinit>(LogAdapter.java:155)
	at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:122)
	at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:89)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:67)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:59)
	at org.springframework.boot.SpringApplication.<clinit>(SpringApplication.java:179)
	at net.weg.solverservice.SolverServiceApplication.main(SolverServiceApplication.java:18)

2023-02-06 09:38:15,998 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.FileAppender for element File: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.FileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.FileAppender
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:260)
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1138)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1063)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1055)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:664)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:258)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:304)
	at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:621)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:694)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711)
	at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
	at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
	at org.apache.commons.logging.LogAdapter$Log4jLog.<clinit>(LogAdapter.java:155)
	at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:122)
	at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:89)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:67)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:59)
	at org.springframework.boot.SpringApplication.<clinit>(SpringApplication.java:179)
	at net.weg.solverservice.SolverServiceApplication.main(SolverServiceApplication.java:18)

2023-02-06 09:38:16,003 main ERROR Null object returned for File in Appenders.
2023-02-06 09:38:16,008 main ERROR Unable to locate appender "solver-service" for logger config "propagate-logger"

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.7.3)

INFO - HV000001: Hibernate Validator 6.2.4.Final
INFO - Starting SolverServiceApplication using Java 17.0.4.1 on BRJGSD357212 with PID 73264 (C:\Projetos\solver-service\target\classes started by schotten in C:\Projetos\solver-service)
INFO - The following 1 profile is active: "dev"
INFO - Multiple Spring Data modules found, entering strict repository configuration mode
INFO - Bootstrapping Spring Data JPA repositories in DEFAULT mode.
INFO - Finished Spring Data repository scanning in 16 ms. Found 0 JPA repository interfaces.
INFO - Tomcat initialized with port(s): 8010 (http)
INFO - Initializing ProtocolHandler ["http-nio-8010"]

Reproduction
Configure the springboot app and use the Log4j.xml and applicatio.proprerties informed in this message and reproduce the error.

@schotten,
Logging in a Spring Boot application is configured at least twice:

when the SpringApplication class is loaded (that is what you see in your stacktrace). This process is described in automatic configuration and does not have access to Spring's environment (that didn't start yet),
after the ApplicationEnvironmentPreparedEvent is fired (cf. lifecycle of a Spring Boot application). This process is specific to Spring Boot, has access to Spring's environment and uses the log4j2-spring.xml file (falling back on log4j2.xml if such a file does not exist).

If you want to use Spring's enviroment to configure your logging system you have to options:

Create two files: log4j2.xml and log4j2-spring.xml. The first one will be used during the bootstrap phase of your application and can not use Spring's environment. The second one will be used, when your application is up and running.
Stick to a single configuration file and use arbiters to provide multiple versions of your "solver-service" appender:

<Select>
    <SpringProfile name="dev">
        <File name="solver-service" fileName="${spring:log4j.solver.service.path}" append="false">
            <EcsLayout serviceName="${spring:log4j.solver.service.name}" serviceVersion="${spring:log4j.solver.service.version}"  serviceEnvironment="${spring:log4j.solver.service.enviroment}"/>
        </File>
    </SpringProfile>
    <DefaultArbiter>
        <Null name="solver-service"/>
    </DefaultArbiter>
</Select>

@ppkarwasz  I have the same problem since I upgraded from 2.17.2 to 2.19 (Spring boot 2.3.9)
I don't need two log4j files or arbiters to make it work in 2.17.2
But since 2.18 it stopped working because of the new way to load LoggingSystemFactory (spring.factories)
I still don't understand why spring.factories are not loaded

I am closing this for lack of activity. I think that:

my previous comment solves @schotten's problem,
@SebHeuze's problem was discussed in #1652.

Fill free to reopen the issue.

authors: SebHeuze ppkarwasz schotten


text:
JsonLayout support for custom key-value pairs being inserted into JSON

Note: This is not ready PR, there are no unit tests for new feature - please review before i'll invest more time into it.

Changed AbstractJacksonLayout.convertMutableToLog4jEvent to protected Object wrapLogEvent(LogEvent event)
Added Extras as KeyValuePair[] to JsonLayout
When extras as specified, LogEvent is wrapped in new LogEventWithAdditionalFields class, which merges LogEvent serialized data with provided extras

This is generally wanted feature, for example see here:
https://github.com/ggrandes/log4j2-simplejson
https://github.com/majikthys/log4j2-logstash-jsonevent-layout
How it works:
<JsonLayout>
  <KeyValuePair key="hostName" value="${hostName}"/>
</JsonLayout>

Inserts into each generated JSON hostName key:
{
"timeMillis":1505228896690,
"thread":"main",
"level":"INFO",
"loggerName":"org.springframework....",
"message":"Refreshing ...",
"endOfBatch":false,
"loggerFqcn":"org....",
"contextMap":{},
"threadId":1,
"threadPriority":5,
"hostName":"wp21257b"
}

This is accomplished by tricks with Jackon json serializer:
public static class LogEventWithAdditionalFields {
    // ....
    @JsonUnwrapped // Deserializes actual LogEvent into root object
    public Object getLogEvent() { return logEvent; }

    @JsonAnyGetter // Adds everything from the map into root object
    public Map<String, Object> getAdditionalFields() { return additionalFields; }
}

I wonder if the KeyValuePair[] should be pulled up from GelfLayout up to its superclass AbstractStringLayout. This would let other layouts like AbstractJacksonLayout use it since it also subclasses AbstractStringLayout.

@garydgregory You are right, there is already AdditionalField, i'll rename it.
There is however problem with pulling it up to AbstractStringLayout, since its used by many layouts where would be problematic (or impossible) to use it. Thats why I chose to implement it directly to JsonLayout.

You're right, we do not want to say that all subclasses can or should implement extra fields.

Note that in the GelfLayout, the field is called "AdditionalField". It seems like a good idea to use the same name. Also, don't forget to update /src/site/xdoc/manual/layouts.xml.vm

You should not change the createLayout() method which is already deprecated. Adding new features is one of the main reasons we switched to the builder pattern.

Rolled back change of createLayout in eeebf4e, you are indeed correct about that one.

Linking to JIRA issue: https://issues.apache.org/jira/browse/LOG4J2-1694

It would be nice to add this to XmlLayout and YamlLayout also.

Have a look at the unfinished branch https://github.com/apache/logging-log4j2/tree/LOG4J2-1694 you can probably pick suitable unit tests from there.

@mikaelstaldal Thanks, I looked for existing issue but somehow missed this one. I'll merge the branches and we'll see whether it will work for all cases. I did not know how to make it work for XML, did the ObjectMapper.withAttributes work in XML? Thanks

@mikaelstaldal I've looked at GelfLayout, and it resolves ${} expressions during serialization. Is this really nescessary? Looks like pretty big overhead to me. I've tested only XML config, but expressions are resolved automatically when parsed. Won't there be a possible conflict? I don't see so far.

@mikaelstaldal I've merged your tests, thanks for that! That saved some time.
Anyway, I've pulled up my changes in JsonLayout to AbstractJacksonLayout - annotations, despite being named @Json* works both for XML and YAML as well.
Please take a look, it looks promising, hopefully it won't break anything (I don't fully trust the @JsonUnwrapped annotation, but seems to be working atm).

Link to JIRA issue:
https://issues.apache.org/jira/browse/LOG4J2-1694

Hi,
Thank you for the PR. Did you run 'mvn clean install' successfully?

Partially, log4j-core completed successfully including all tests, whole project failed somewhere on Cassandra with some JNI exception. I did not look further into that yet. Does not seem related.

@mdvorak Resolving ${} expressions during serialization is necessary to support dynamic lookups which depend on the current log event, such as ContextMapLookup.

@mikaelstaldal Yeah i figured after digging deeper. Implemented it with correct support.
Only thing that bugs me is that dynamically created LinkedHashMap - thats wasteful and expensive (I know Jackson alone is not exactly garbage-free, but still).
I'm thinking about using ThreadLocals, but I understand there are both potentional memory leaks, and Log4j alone has special policies about that (log4j2.enable.threadlocals), and I don't have atm time to dive into that. Any ideas? Or do you consider it fine as it is?

@mdvorak I think that we should not care so much about garbage and micro optimizations in the Jackson based layouts, since they are far from garbage-free or optimal anyway.
Keep it as is, and don't bother with ThreadLocals.
If we want to make JsonLayout garbage-free, we would have to stop using Jackson and do something similar to GelfLayout.

@mdvorak The code looks fine to me. But we will wait a while before merging, since we plan to do a 2.9.1 bugfix release soon. This will go into 2.10.0.

This is build log, it hangs after that.. But i'm not sure why I'm seeing this exception.
https://gist.github.com/anonymous/9909ea92d10ac4ebe36f27a0610e7c2d

@mikaelstaldal is there any estimation on 2.10 release date? Obviously I need this :)

We are in the process of releasing 2.9.1, anything beyond that does not have a hard set date yet.

2.9.1 is now released, and the next release will (most likely) be 2.10.0 which will include this. We have no date set for that release, but historically there has been at least three months between minor releases of Log4j 2.x, and 2.9.0 was released one month ago.

This is a needed feature.  Can a release go out soon that will include it?

Does this support variable replacement.
i.e. in PatternLayout one can do something like this:
<Pattern>%d{ISO8601}{UTC} %d{Z}{UTC} [CID:%X{conversation-id}] [%t] %-5p %c{1.} - %m%n</Pattern>
But it seems that
<KeyValuePair key="CID" value="%X{conversation-id}"/>
or even
<KeyValuePair key="CID" value="${conversation-id}"/>
doesn't work.

authors: dankirkd garydgregory mdvorak mikaelstaldal


text:
Improve set deletions percentage javadoc

As a follow item of Discussion issue, improve the java doc to make the formula of deletion percentage clear to developers.

It looks like this PR is in a bad state somehow?

authors: jpountz yugushihuang

repo: apache/lucene


text:
Added queued flush check optimization to changes.txt

Description
Adding changes in #13115 to changes.txt

Thank you @CaptainDredge and @jpountz!

authors: CaptainDredge mikemccand


text:
Add Query for reranking KnnFloatVectorQuery

Description
fixes #13564
Added a new Query which wraps around KnnFloatVectorQuery and does re-ranking using full precision vectors.
Questions:

Should we expose the target inside KnnFloatVectorQuery so that users don't need to pass the target twice? Currently it only exposes the getTargetCopy() which requires array copy so it's inefficient, but I assume the intention is to encapsulate the array so that it won't be modified from outside?

The build fails with The import org.apache.lucene.codecs.lucene100 cannot be resolved, I thought this is already in mainline. Will check.

I have a preliminary benchmark here (top-k=100, fanout=0) using Cohere 768 dataset.

Anyhow I can see these 2 things that should be addressed:

If we access the full-sized vectors, it will swap the memory that is allocated (either through preloading, or through mmap) for quantized vectors (main search phase) if there's not enough memory. Eventually, some % part of the quantized index will be swapped out which will slower the search. If we have to load all full-precision vectors to memory, then that kinda defeats the purpose of quantization. I'm wondering if there could be a way we can access full-precision vectors without interfering with the space of quantized vectors.
The latency could be better. With oversample=1.5 (second dot) for 4_bit, we have around the same latency and recall as baseline. Although one can argue that we can save memory compared to baseline, with new access pattern of two-phase search that saving might be diminished. Otherwise it seems to have little benefit over just using plain HNSW.

I reran the benchmark, this time comparing the two-phase search against no-reranking as well. It seems the re-ranking phase only adds a trivial amount of latency so that's a good thing.

Benchmark parameters:

topk = 100
fanout = 0
Datasets: Cohere 768

Also this is the luceneutil branch I used for benchmarking: https://github.com/dungba88/luceneutil/tree/dungba88/two-phase-search, which incorporates the test for BQ implementation by @benwtrent and the two-phase search.


text:
regress: Update linear regression docs

As part of commit b0991965f7d4007d47902afcf7b319a78d766613 , we updated
num_rows_processed column to be of type BIGINT.  This commit makes the
doc changes for that code change
Authored-by: Ekta Khanna ekhanna@vmware.com

[ ] Add the module name, JIRA# to PR/commit and description.
[ ] Add tests for the change.

LGTM

LGTM

repo: apache/madlib


text:
[MENFORCER-314] - Include entire exception if message is missing

This should help to find out why DependencyConvergence
sometimes fails without providing a message.
Following this checklist to help us incorporate your
contribution quickly and easily:

[x] Make sure there is a JIRA issue filed
for the change (usually before you start working on it).  Trivial changes like typos do not
require a JIRA issue.  Your pull request should address just this issue, without
pulling in other changes.
[x] Each commit in the pull request should have a meaningful subject line and body.
[x] Format the pull request title like [MENFORCER-XXX] - Fixes bug in ApproximateQuantiles,
where you replace MENFORCER-XXX with the appropriate JIRA issue. Best practice
is to use the JIRA issue title in the pull request title and in the first line of the
commit message.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Run mvn clean verify to make sure basic checks pass. A more thorough check will
be performed on your pull request automatically.

If your pull request is about ~20 lines of code you don't need to sign an
Individual Contributor License Agreement if you are unsure
please ask on the developers list.
To make clear that you license your contribution under
the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

[x] I hereby declare this contribution to be licenced under the Apache License Version 2.0, January 2004

[ ] In any other case, please file an Apache Individual Contributor License Agreement.

Alternative approach to #43 which I actually like better but I am not sure.
Please merge whichever PR you like best.

repo: apache/maven-enforcer


text:
METRON-1641: Enable Pcap jobs to be submitted asynchronously

Contributor Comments
https://issues.apache.org/jira/browse/METRON-1641
This enables Pcap Jobs to be run asynchronously. The PcapJob class itself is now a Statusable implementation that allows clients to poll for current JobStatus. This implementation exposes the new functionality on the job class but keeps the existing PcapCli functionality intact and unchanged. The tests for this will be in a comment below, taken from https://github.com/apache/metron/pull/256.
This validation should check that the current pcap functionality does not break. Follow on PR's will leverage the new asynchronous capabilities.
Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:

[x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
[x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

[x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?

[x] Have you included steps or a guide to how the change may be verified and tested manually?

[ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh

[x] Have you written or updated unit tests and or integration tests to verify your changes?

[ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:
cd site-book
mvn site

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

Looks good!  One thing I'm trying to wrap my head around is how we get status if we only have a job id or unique identifier for a job?  JobStatus doesn't have an id so I'm assuming resultPath is the unique identifier here.
As far as I can tell an instance of org.apache.hadoop.mapreduce.Job is kept in memory and is responsible for reporting status.  I can think of a couple scenarios where this might be problematic.
One is if I ran a query from the CLI but then wanted to get status from REST.  How would that work?  That's probably not a likely use case so maybe not an issue there.  What happens if I submit a query through REST and REST is restarted while jobs are running?  Do we lose job status information?

I think for a first cut, it's ok to have the restrictions that:

the REST API controls only the jobs it creates.  Otherwise, we would need more refactoring in the CLI to drop the output in the same HDFS directory rather than it be user specifiable and output locally.  Ultimately, while they use the same mechanism, the UX is different between the two approaches (e.g. the CLI entirely cleans up after itself and outputs to the local directory whereas the REST approach stores the results in HDFS until manual cleanup).
If a job is running while the REST API dies, we should consider that job to be runaway and needs to be killed by the admin or left to complete without the result being published.  One thing that we might consider doing is enabling the job naming to have a prefix of METRON_REST_PCAP so, upon REST start, it can kill existing jobs.  I think for THIS PR, we should just have REST pcap jobs have that prefix and leave it to a follow-on PR to do the actual killing.

One thing I didn't see.  Can we make sure we pass along the yarn queue to the job?

One more comment about restartability, I think we could potentially support this with this architecture in the future.  You can recover the Job object from MR via the JobClient
Configuration conf = new Configuration();
JobClient jobClient = new JobClient(new JobConf(conf)); // deprecation WARN
JobID jobID = JobID.forName("job_201107011451_0001");   // deprecation WARN
RunningJob runningJob = jobClient.getJob(jobID);

We could look for jobs which are completed but not in the HDFS structure and recover them on REST start.  I would suggest doing that as a follow-on though.

Perfect.  That addresses my concern.  Doing this in a follow on is fine since it's not necessary when using the CLI.

I spun this up in full dev, ran a fixed query, and got this error:
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: file://./, expected: hdfs://node1:8020
	at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:666)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:214)
	at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1181)
	at org.apache.hadoop.hdfs.DistributedFileSystem$25.doCall(DistributedFileSystem.java:1177)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1195)
	at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1169)
	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1925)
	at org.apache.metron.common.utils.HDFSUtils.write(HDFSUtils.java:71)
	at org.apache.metron.pcap.writer.ResultsWriter.write(ResultsWriter.java:38)
	at org.apache.metron.pcap.mr.PcapJob.writeResults(PcapJob.java:270)
	at org.apache.metron.pcap.query.PcapCli.run(PcapCli.java:155)
	at org.apache.metron.pcap.query.PcapCli.main(PcapCli.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

Looks like the MR jobs succeeded but partitioning the files to the local FS did not work.

Good catch on the FS @merrimanr - also finding that via manual testing. I believe I have a workaround that degrades nicely to the configuration default and also allows you to pass in the scheme in the path.
FileSystem fs = FileSystem.newInstance(outPath.toUri(), config);

Also @merrimanr agreed about the points you made regarding restart-ability, etc. in the long run. In the short term, @cestella has done a rather good job of summarizing my thoughts on a v1 pass at this feature set.
I will:

add Job ID in the return status (was going to add that when I do the job manager follow-on to this PR, but I'll just do it here)
add the ability to pass a job name, ie job.setJobName(name) - I'll handle the actual job naming in the pcapservice and pass that as a parameter. I think that's a natural place for that logic.
Add ability to pass the queue name

How's that sound to you both?

Heads up, hadoop config class is where you set queue name iirc. We already pass that in as an arg. This would simply need to be provided via the calling job manager class. config.setProperty("mapreduce.job.queuename", "somequeue");

That sounds good to me.

I tested this and it's working for me in full dev.  I think it's good enough to go into the feature branch.  +1

Can we merge this?  Any other items you would like addressed @cestella?

+1, lgtm

authors: cestella merrimanr mmiklavc

repo: apache/metron


text:
Error status 0x0181 sent back in BLE write call-back.  What does it mean?

My app is getting back the error status 0x0181 (385 decimal) in the BLE write callback when it tries to write to the FTMS's Fitness Machine Control Point characteristic 0x2AD9.  Browsing the NimBLE code, I can't figure out what this status code means.  The BLE device rejecting this write request is a ZwiftHub indoor bike trainer.  No idea what BLE stack this device uses.  Does anyone know?

Hi,
In Nimble error codes have ranges so that one can figure out from which layer error originated from.
0x181 is 0x100 + 0x81 which means it is ATT (0x100) error code with value 0x81.
0x81 is from ATT application error code range which means that its meaning is defined in FTMS specification.
More info on errors can be found in documentation: https://mynewt.apache.org/v1_12_0/network/ble_hs/ble_hs_return_codes.html

Thanks for your clear explanation @sjanc !
For NimBLE neophytes (such as me) it would be helpful to have a comment with your clear explanation in the definition of the 'struct ble_gatt_error' inside ble_gatt.h.

authors: elfrances sjanc

repo: apache/mynewt-nimble


text:
porting: cleanup porting_init()

code simplification
add some more comments
call ble_hci_ram_init() before ble_ll_init() to make sure the
buffers are intialized before they are potentially used

rebased

and retested, everything fine. -> go