text:
[MKLDNN] Add LSTMP to v1.6.x

Description
This PR aims to support projection feature for LSTM towards the CPU platform.
Checklist
Essentials

[x] Upgrade dnnl(mkldnn) to its v1.4 version
[x] Change some statement of mkldnn to dnnl
[x] Implement LSTMP from both native and DNNL pass
[x] Add unit test for LSTMP
[x] Add env var to control the initializing behavior of RNN
[x] Remain input dim as int type in dnnl pass
[x] Add logging message for MXNET_RNN_USE_WEIGHT_CACHE=1

Comments

This feature requires dnnl-v1.4rc porting to MXNet v1.6.x branch. Perhaps, there are several unexpected failures of unit tests. We need to have a test of CI in advance. And then switch to the formal release of dnnl-v1.4rc.
Performance data will be added later.

@ciyongch @TaoLv @pengzhao-intel

@sandeep-krishnamurthy @szha

Please note that all the changes in this PR are ready for review except a minor pending item: to upgrade mkldnn to v1.4rc version to finalize the PR.

@mxnet-bot run ci [windows-cpu]

@mxnet-bot run ci [centos-gpu]

@eric-haibin-lin @apeforest @ChaiBapchya - Can you please help review this PR?

Seems there are problems in CI @zixuanweeei could you point out the failure cases and I think AWS colleagues can help double-check.

Seems there are problems in CI @zixuanweeei could you point out the failure cases and I think AWS colleagues can help double-check.

I created #17964 to track the CI status for v1.6.x branch. Basically, we can see several failures which appear in both sides:

[unix-cpu] - R: CPU
[unix-cpu] - R: MKLDNN-CPU
[unix-gpu] - R: GPU
[unix-gpu] - Static build GPU 14.04 Python
[website] - R Docs

There are lots of failures with R. Besides, I believe test_operator_gpu.test_parallel_random_seed_setting_for_context is a flaky test (#15856).
@sandeep-krishnamurthy Could you help us to take a look for the CI failures?

@mxnet-bot run ci [windows-gpu]

After upgrading the MKLDNN (oneDNN) to the formal v1.4-rc, same failures appeared as mentioned above. I thinks it may be caused by the CI issues rather than the functionality issues of MKLDNN(oneDNN). Please take review @sandeep-krishnamurthy , thanks.

@eric-haibin-lin - Can you take a look at this PR please?
@ChaiBapchya - Any pointers on why R tests are failing here?

The issue related to R package test is fixed with importing the function first, using library(methods)
I've made the change to v1.6.x branch [testing it locally now].
Referring the doc for Reproducing the tests : https://cwiki.apache.org/confluence/display/MXNET/Reproducing+test+results

Requirements

pip3 install -r ci/requirements.txt --user

Build

ci/build.py --docker-registry mxnetci --platform ubuntu_cpu --docker-build-retries 3 --shm-size 500m /work/runtime_functions.sh build_ubuntu_cpu_openblas

Test

ci/build.py --docker-registry mxnetci --platform ubuntu_cpu --docker-build-retries 3 --shm-size 500m /work/runtime_functions.sh unittest_ubuntu_cpu_R

The issue related to R package test is fixed with importing the function first, using library(methods)
I've made the change to v1.6.x branch [testing it locally now].
Referring the doc for Reproducing the tests : https://cwiki.apache.org/confluence/display/MXNET/Reproducing+test+results

Requirements

pip3 install -r ci/requirements.txt --user

Build

ci/build.py --docker-registry mxnetci --platform ubuntu_cpu --docker-build-retries 3 --shm-size 500m /work/runtime_functions.sh build_ubuntu_cpu_openblas

Test

ci/build.py --docker-registry mxnetci --platform ubuntu_cpu --docker-build-retries 3 --shm-size 500m /work/runtime_functions.sh unittest_ubuntu_cpu_R

R docs are broken on 1.6.x so maybe this fix can be applied to that pipeline too?
https://github.com/apache/incubator-mxnet/issues/17920

Thanks for your quick response @aaronmarkham @ChaiBapchya. Hope it will work soon.
Besides, [unix-gpu] - Static build GPU 14.04 Python looks like being broken by some network issues. And it also appeared in #17964.

Performance comparison between the native and DNNL(oneDNN) implementation (tested on 28-core, 2.70GHz Intel CPU).
batch_size=64, sequence_len=50, input embedding_size=512

directions
layer_num
hidden_size
proj_size
DNNL
Native
Throughput Speedup

fwd time
fwd samples/sec
fwd time
fwd samples/sec

1
1
512
256
0.01
8312.23
0.04
1539.50
5.40

2
1
512
256
0.01
4416.66
0.06
1060.28
4.17

1
1
1024
512
0.02
3892.16
0.06
1048.26
3.71

2
1
1024
512
0.03
1979.32
0.10
626.18
3.16

1
3
512
256
0.02
3027.49
0.08
832.71
3.64

2
3
512
256
0.04
1537.22
0.18
361.05
4.26

1
3
1024
512
0.05
1346.44
0.18
349.79
3.85

2
3
1024
512
0.12
534.99
0.33
195.29
2.74

@ChaiBapchya @aaronmarkham Thanks for looking into those failed R tests, I'm wondering whether https://github.com/apache/incubator-mxnet/issues/17920 can fix the CI failure or not?

I've a PR #17993 it should fix the R related errors

I've a PR #17993 it should fix the R related errors

Thanks @ChaiBapchya , hope this fix will be merged into v1.6.x soon to move this PR forward:)
Ping @eric-haibin-lin for a code review.

Yes, waiting on the builds to pass and will get it merged soon!

@zixuanweeei PR #17993 merged
Please rebase. Let's get this merged!
Thanks!

@ChaiBapchya Thanks for your prompt fix on the CI issues, now all the CI tests are passed:)

Since this a new feature and it is not included in 1.6.0 pypi
should we cherry-pick this to v1.7.x in time for release of v1.7.0?
@zixuanweeei @ciyongch @pengzhao-intel wdyt?

I suggest to leave this new feature to the next release version (which requires upgrade oneDNN (ex: DNNL)), as the current version 1.7.0 is code freeze and suppose to only accept bug fixes, besides this feature is also included in master branch. The reason for applying it in 1.6.x branch is it's required by customer.
@ChaiBapchya is there any special requirement for this feature in 1.7.0?

@ciyongch While the customer request is resolved with 1.6.x backport, it made sense to include it in upcoming minor release. Backport to 1.6 should be included by default in 1.7 right?

Backport to 1.6 should be included by default in 1.7 right?

Hi @ChaiBapchya , not sure if it's a common practice to do so. As there're several new features including (oneDNN) FP32 LSTMP/INT8 LSTM/INT8 LSTMP introduced in v1.6.x branch, are they all needed to go into 1.7.0 or only FP32 LSTMP is good enough? As currently INT8 LSTMP relies on non-release version of oneDNN which might be involved till it's matured and included in a formal release version. Previously MXNet is always pin to a formal release version of oneDNN.
cc @TaoLv to see if there's any other comments.
Thanks,
Ciyong.

@ChaiBapchya @ciyongch As far as I know, the feature lives on 1.6.x branch but is not released in any release yet. Given the code of 1.7.0 has been frozen for a while and to pay respect to the effort from release manager, I would suggest to not include this feature in the coming rc0. We can reconsider the request if we will have another rc1. The pined version of oneDNN is also a valid concern.

@TaoLv Makes sense. Thanks for the clarification. Let's circle back to this point if rc1 is required.

Thanks for your understand and help making it clear @TaoLv @ChaiBapchya .

authors: ChaiBapchya TaoLv aaronmarkham ciyongch pengzhao-intel sandeep-krishnamurthy zixuanweeei


text:
[v1.7] Cherry pick Increase staggered build timeout (#18568)

Increase staggered build timeout to 180 min, since sanity build has 180 min timeout.

Decrease timeout so everyone is happy.

Co-authored-by: Joe Evans joeev@amazon.com

@mxnet-bot run ci [unix-gpu]

Hi @josephevans , please help to re-trigger the unix-gpu job as it's stuck.
The modification increases the full time from 30min to 60 min, while decrease the sanity time from 180min to 60min, seems not align to the description?

@mxnet-bot run ci [unix-gpu]

Hi @josephevans , please help to re-trigger the unix-gpu job as it's stuck.
The modification increases the full time from 30min to 60 min, while decrease the sanity time from 180min to 60min, seems not align to the description?

Hi @ciyongch, we actually increased the overall "full" job which first triggers sanity build, then if it passes we run the rest of the jobs. But since it doesn't make sense to have a longer timeout for sanity build than the "full" build, we made them the same. If the sanity build timed out, the full build would have already timed out of the timeout is longer.

Thanks for the update @josephevans . Ping @leezu @marcoabreu @aaronmarkham to have a review and help merge. Thanks!

Hi @leeze @ChaiBapchya @marcoabreu @aaronmarkham , can you help to take a review and merged. The corresponding one https://github.com/apache/incubator-mxnet/pull/18568 to v1.x branch is already merged.

source: gharchive/pull-request

authors: ciyongch josephevans

repo: apache/incubator-mxnet

license_type: permissive

license_source: github-api


text:
Maven first in New Project Wizard

There's a discussion in the netbeans-dev mailing list about changing the layout of the "New project" wizard.
Currently the "Java" Category creates an ANT-based project. As ANT is old and only few projects use it, this Category should use Maven. This can be done by simply rearranging the projects a bit, mostly via branding.

Not sure this was intentionally closed, but could we please not go down the "Ant(ic)" route? It is not funny. People choose ant for a reason and you don't need to belittle them.

Thanks for the comment, I was in a hurry to get on the road and accidentally closed it.
Agreed, I can change the name tomorrow morning, what would you suggest?

I updated the PR with a more agreeable categrory name:

The original/current names in the wizard are Java and Maven, why not now Java Maven and Java Ant? So everyone knows on the very first view what is meant here.
Followed the discussion, future wish could be only using Java maybe with sub categories or wizard enhancements. But for now I would prefer to see Java Maven and Java Ant, IMHO.

First I thought, it's only a small change, and I should do it, but actually I think the current naming is better, as it serves a purpose:
"Java Maven" and "Java Ant" does not really fix the problem of making Maven the default and "nudging" new developers to stay away from Ant. It only changes the visibility of Maven a little bit, and instead forces users to make a decision.
Imagine a new developer who wants to create his first Java project. If we present him the options "Java" and "Java Ant", he'll very likely choose "Java", which would be great. If we present him with "Java Maven" and "Java Ant", he'll be lost and in the best case resort to google. Like Tim said on the mailing list, we force a user into a choice they don't (shouldn't) care about For reference: Joel Spolsky's Blog Post.
I'd prefer to keep the names like this, as a tool to point users in the right direction.

I would concur with @Chris2011 and go with "Java Ant" and "Java Maven" or "Java (ant)" and "Java (maven)". If the label is not present and I want to create a maven project, I have to guess how to get there (you are hindering my experience in that case).
Yes, as a developer you have to know some terms and if you don't know them, you should better learn.

If you don't say, that "Java" means "Maven" how should an experienced user know hat netbeans does when he clicks "Next"? You still ask the user to make a decision and then you hide the important information. So if I can choose between:

Java
Ant Java

how should I know which is the right one, given that I want to create a maven project? Behind "Java" everything could hide (gradle, maven, make, $newCoolBuildsystem).
For me (yes, I'm also a user) hiding the build system makes the situation worse.

Are we ready to merge this?

Who's "are"?Me? Yes, I support merging.Toni? Ye.s.Community? Likely.Matthias? No.-jtOdesláno z mého chytrého telefonu Samsung Galaxy.
-------- Původní zpráva --------Od: Geertjan Wielenga notifications@github.com Datum: 27.11.18  16:07  (GMT+01:00) Komu: apache/incubator-netbeans incubator-netbeans@noreply.github.com Cc: Jaroslav Tulach jaroslav.tulach@oracle.com, Review requested review_requested@noreply.github.com Předmět: Re: [apache/incubator-netbeans] Maven first in New Project Wizard (#1038) Are we ready to merge this?
—You are receiving this because your review was requested.Reply to this email directly, view it on GitHub, or mute the thread.
{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/apache/incubator-netbeans","title":"apache/incubator-netbeans","subtitle":"GitHub repository","main_image_url":"https://assets-cdn.github.com/images/email/message_cards/header.png","avatar_image_url":"https://assets-cdn.github.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/apache/incubator-netbeans"}},"updates":{"snippets":[{"icon":"PERSON","message":"@geertjanw in #1038: Are we ready to merge this?"}],"action":{"name":"View Pull Request","url":"https://github.com/apache/incubator-netbeans/pull/1038#issuecomment-442091771"}}}
[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/apache/incubator-netbeans/pull/1038#issuecomment-442091771",
"url": "https://github.com/apache/incubator-netbeans/pull/1038#issuecomment-442091771",
"name": "View Pull Request"
},
"description": "View this Pull Request on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
},
{
"@type": "MessageCard",
"@context": "http://schema.org/extensions",
"hideOriginalBody": "false",
"originator": "AF6C5A86-E920-430C-9C59-A73278B5EFEB",
"title": "Re: [apache/incubator-netbeans] Maven first in New Project Wizard (#1038)",
"sections": [
{
"text": "",
"activityTitle": "Geertjan Wielenga",
"activityImage": "https://assets-cdn.github.com/images/email/message_cards/avatar.png",
"activitySubtitle": "@geertjanw",
"facts": [
]
}
],
"potentialAction": [
{
"name": "Add a comment",
"@type": "ActionCard",
"inputs": [
{
"isMultiLine": true,
"@type": "TextInput",
"id": "IssueComment",
"isRequired": false
}
],
"actions": [
{
"name": "Comment",
"@type": "HttpPOST",
"target": "https://api.github.com",
"body": "{\n"commandName": "IssueComment",\n"repositoryFullName": "apache/incubator-netbeans",\n"issueId": 1038,\n"IssueComment": "{{IssueComment.value}}"\n}"
}
]
},
{
"name": "Close pull request",
"@type": "HttpPOST",
"target": "https://api.github.com",
"body": "{\n"commandName": "PullRequestClose",\n"repositoryFullName": "apache/incubator-netbeans",\n"pullRequestId": 1038\n}"
},
{
"targets": [
{
"os": "default",
"uri": "https://github.com/apache/incubator-netbeans/pull/1038#issuecomment-442091771"
}
],
"@type": "OpenUri",
"name": "View on GitHub"
},
{
"name": "Unsubscribe",
"@type": "HttpPOST",
"target": "https://api.github.com",
"body": "{\n"commandName": "MuteNotification",\n"threadId": 415723158\n}"
}
],
"themeColor": "26292E"
}
]

If you don't say, that "Java" means "Maven" how should an experienced user know hat netbeans does when he clicks "Next"?

Just like it worked for last 20 years: If you select a project, a description is displayed, which also says how this project is built. You can see that in the 2nd and 3rd screenshot "A simple Java Application using Maven."...

Can this PR be merged now? Or is there someone who wants Toni to organize a vote on the mailing list?
My feeling is that there exists a majority that is either +1 or +0 on the "Maven 1st UI change" and this PR is good enough to begin our move towards that direction. A formal vote would of course verify/reject my feeling properly.
Speakup if you want Toni to initiate such vote.

There is always the option to cancel the PR and rethink. My opinion stands, I want our users to make informed decisions and that means he/she should know what he/she chooses. You plan to take away that information . For me this is a step back.
The quoted article Joel on Software - Choices does not even apply here. It argues, that useless decisions should not be presented to the user, which this PR does not do, it leaves all options accessible. It explicitly does not state, that information should be hidden from the user, which hinders the decision process.

Certainly not eligible for cherry picking into NetBeans 10. We need more time to stabilize and polish the change.
@matthiasblaesing and/or everyone else, please explicitly state you insist on a vote. Otherwise I'll push forward merging of this PR on the basis of lazy consensus.

I don't insist on a vote, I want a better solution. I veto the merge of this PR.

Hi all,
I have to agree with @matthiasblaesing and I think this deserves some further consideration/thought, preferably in the mailing list. Matthias is putting the user first, and that's my motto as well, so I have to agree with his veto.
Trying to be positive, I'd rather see a "Java" main section with the following options:

Ant based project.
Maven based project.

Also let's try to be positive about ant. Apache Ant is a companion project in the ASF and deserves our consideration. The fact that it's an "old technology" deserves a respect as well. Ant has served NetBeans well during many years, and we still depend on it to build our own food.

@viero I think everyone here tries to put the user first. In my personal opinion, the PR does that, and the other solution would be confusing and bad for the user. You have a different opinion. That's OK and I respect that. But please don't assume that I don't care about usability. If you read my replies you should be able to see that my whole argument is about usability, and I'd be glad if you could respect that too, even if you don't agree.
@matthiasblaesing There was a "Java" category before, which defaulted to ANT, now it defaults to Maven. How is this a step back? It is a change of the default for Java, no more no less. And regarding the "Choices" blog, that Tim Boudreau mentioned on the mailing list: For a Java newbie it's exactly the same as the "optimize database" example in the blog: We ask the user for a decision he very likely doesn't care about and doesn't have enough information to make (What is ANT or Maven?). And for a more experienced user it's not hard to find the information, as it is in the description of the wizard.

@eppleton the referenced article argues, to not ask the user at all, if the developer should know better and the user can be expected not to care/know. We can't take that road, as there are different pros and cons to choose the build system and the option needs to be there. So the user needs to choose and needs information to base its decision on. Given that the difference between the two options is the build system I don't see why that fact should be hidden and why that would help the developer at all.
Before the the change there was:

Java: An ant based project with netbeans specific project files
Maven: A maven based project without any netbeans specifics

No we have:

Java
Ant Java

So what tells me what "Java" does? If my peer group says "gradle" is the build system, I would expect that to be created. Bummer - does not happen - there is some file "pom.xml" generated, but no  "build.gradle". You can replace "gradle" with $featured build system. So why not:

Java (maven based)
Java (ant based)

Putting maven first, indicates to order "we" suggest and lets the user choose.

@eppleton Oh, of course we all want the best for the user. Apologies for my bad explanation. I like best your proposal of explaining the build system "Java (ant based)" and "Java (maven based)".

What shall be the next steps with this PR? Can those who object to current state (@matthiasblaesing, @vieiro) modify the PR so it is acceptable for them?
Btw. recently I was teaching my kids Java and found a nice course that uses NetBeans! Having courses like this is an excellent value that we don't want to loose. It has to be admitted that for existing materials like this, it is important to keep the Ant based projects around and deprecate them carefully.

Since this is going to impact one of the very first steps in all of our tutorials, and could have an adverse effect on the starting point of each and every newbie of Apache NetBeans, I think we should together take a look at the impact of this on the documentation -- i.e., if someone can't do step 1 because they cannot find the project template described in step 1 of a tutorial, there's a real chance they'll not go to step 2 at all.

Here is where our documentation at Apache is found:
https://github.com/vieiro/incubator-netbeans-website/tree/feature/third-donation-tutorials/netbeans.apache.org/src/content/kb/docs
Here's the quick start, i.e., the starting point for most users, for example, though note some broken images:
https://github.com/vieiro/incubator-netbeans-website/blob/feature/third-donation-tutorials/netbeans.apache.org/src/content/kb/docs/java/quickstart.asciidoc

I think we should only do a vote on a unified wizard once we have a commitment from someone to actually implement that. And it will not be trivial to implement. :-) (Plus, as @neilcsmith-net points out, it may end up being a lot less intuitive than the current situation for end users. Don't forget that all changes have unintended side effects, so it's better to make changes as small as possible.)
Today, I will create a PR for the proposal I suggested above and that seems to have everyone's approval -- "Java with Maven", "Java with Gradle", and "Java with Ant", in that order, with all the project templates related to Java organized into those three categories.

Here is the issue with full description:

Here is the PR: https://github.com/apache/incubator-netbeans/pull/1115
Here is the blog with pic: https://blogs.apache.org/netbeans/entry/restructuring-of-project-templates-in

My PR is merged. As a consequence, since Maven is now first, as requested via this PR, I am closing this PR.

authors: Chris2011 JaroslavTulach dukescript eppleton geertjanw jtulach matthiasblaesing vieiro

repo: apache/incubator-netbeans


text:
[NETBEANS-2160] Start on jdk11.0.2

Updates Felix to version 6.0.2 that supports the jrt:// protocol.

+1

authors: JaroslavTulach jlahoda


text:
examples: Fix the parallel build break

1.Remove build target to avoid build and install run parallelly
2.Invoke the sub Makefile just before the parent source code compile
Signed-off-by: Xiang Xiao xiaoxiang@xiaomi.com

@masayuki2009 @patacongo this PR can fix the parallel build break, please merge it.

@patacongo this PR can fix the parallel build break, please merge it.

Yes, of course.  I don't normally merge build system changes only because I don't understand them completely.

@masayuki2009 @patacongo this PR can fix the parallel build break, please merge it.

@xiaoxiang781216
Hmm, I've just tried parallel-build-aging.sh which I did before but it failed.
Anway, let's see what will happen at today's nightly build.
make[4]: Entering directory '/mnt/m2ssd/opensource/github_masayuki2009/apache-nuttx-sync/apps/examples/elf/tests/helloxx'
LD: hello++1.o
riscv64-unknown-elf-ld: hello++1.o: file not recognized: file truncated
Makefile:126: recipe for target 'hello++1' failed
make[4]: *** [hello++1] Error 1
make[4]: Leaving directory '/mnt/m2ssd/opensource/github_masayuki2009/apache-nuttx-sync/apps/examples/elf/tests/helloxx'
Makefile:86: recipe for target 'helloxx_install' failed
make[3]: *** [helloxx_install] Error 2
make[3]: Leaving directory '/mnt/m2ssd/opensource/github_masayuki2009/apache-nuttx-sync/apps/examples/elf/tests'
Makefile:59: recipe for target 'build' failed
make[2]: *** [build] Error 2
make[2]: Leaving directory '/mnt/m2ssd/opensource/github_masayuki2009/apache-nuttx-sync/apps/examples/elf'
CC:  environ/env_removevar.c
Makefile:64: recipe for target '/mnt/m2ssd/opensource/github_masayuki2009/apache-nuttx-sync/apps/examples/elf_all' failed
make[1]: *** [/mnt/m2ssd/opensource/github_masayuki2009/apache-nuttx-sync/apps/examples/elf_all] Error 2
make[1]: Leaving directory '/mnt/m2ssd/opensource/github_masayuki2009/apache-nuttx-sync/apps'

@masayuki2009 @patacongo this PR can fix the parallel build break, please merge it.

@xiaoxiang781216
Hmm, I've just tried parallel-build-aging.sh which I did before but it failed.
Anway, let's see what will happen at today's nightly build.

Yes, there are other place has the similar issue need to fix too.

@masayuki2009 @patacongo this PR can fix the parallel build break, please merge it.

@xiaoxiang781216
Hmm, I've just tried parallel-build-aging.sh which I did before but it failed.
Anway, let's see what will happen at today's nightly build.

The key fix is avoid invoke sub Makefile parallelly, like below:

Because populate depends on build and install, build and install will run concurrently in parallel build, then the same object file may be written by multiple thread at the same time.
There are other place has the similar issue we need to find and fix.

@xiaoxiang781216
Hmm, however, it seems that the latest build results (https://builds.apache.org/job/NuttX-Nightly-Build/74/consoleText) show many errors (not only link errors) in the following targets
samv71-xult/module, lc823450-xgevk/elf, lc823450-xgevk/posix_spawn, stm32f4discovery/elf,
spresense/elf, maix-bit/elf, maix-bit/posix_spawn and maix-bit/module.
In my understanding, the above targets were going to be built correctly by this PR.
However, the results show worse than before.

Because populate depends on build and install, build and install will run concurrently in parallel build, then the same object file may be written by multiple thread at the same time.
There are other place has the similar issue we need to find and fix.

@xiaoxiang781216
Hmm, however, it seems that the latest build results (https://builds.apache.org/job/NuttX-Nightly-Build/74/consoleText) show many errors (not only link errors) in the following targets
samv71-xult/module, lc823450-xgevk/elf, lc823450-xgevk/posix_spawn, stm32f4discovery/elf,
spresense/elf, maix-bit/elf, maix-bit/posix_spawn and maix-bit/module.
In my understanding, the above targets were going to be built correctly by this PR.
However, the results show worse than before.

The high level Makefile likely inovke examples/elf/Makefile.. for different targets concurrently. We need fix other location too, with this command:
make -j8 --debug=basic V=1
We need review the log to fix the location wich trigger the same file compile multiple.

repo: apache/incubator-nuttx-apps


text:
fs/littlefs: correct some code format in lfs.h by running checkpatch.sh

Signed-off-by: liuhaitao liuhaitao@xiaomi.com

one config of arm07 build breaks, it should be fixed to pass check build : (
====================================================================================
Configuration/Tool: sama5d3x-ek/nsh,CONFIG_ARMV7M_TOOLCHAIN_GNU_EABIL
Configuring...
Copy files
Select CONFIG_HOST_LINUX=y
Refreshing...
Disabling CONFIG_ARMV7A_TOOLCHAIN_GNU_EABIL
Enabling CONFIG_ARMV7M_TOOLCHAIN_GNU_EABIL
Refreshing...
Building NuttX...
chip/sam_memories.c: In function 'periphb_physregaddr':
chip/sam_memories.c:86:5: warning: "SAM_PERIPHB_PSECTION" is not defined, evaluates to 0 [-Wundef]
86 | #if SAM_PERIPHB_PSECTION != SAM_PERIPHB_VSECTION
|     ^~~~~~~~~~~~~~~~~~~~
chip/sam_memories.c:95:10: error: 'SAM_PERIPHB_PSECTION' undeclared (first use in this function); did you mean 'SAM_PERIPHA_PSECTION'?
95 |   return SAM_PERIPHB_PSECTION + sectoffset;
|          ^~~~~~~~~~~~~~~~~~~~
|          SAM_PERIPHA_PSECTION
chip/sam_memories.c:95:10: note: each undeclared identifier is reported only once for each function it appears in
chip/sam_memories.c: In function 'sysc_physregaddr':
chip/sam_memories.c:149:5: warning: "SAM_SYSC_PSECTION" is not defined, evaluates to 0 [-Wundef]
149 | #if SAM_SYSC_PSECTION != SAM_SYSC_VSECTION
|     ^~~~~~~~~~~~~~~~~
chip/sam_memories.c:158:10: error: 'SAM_SYSC_PSECTION' undeclared (first use in this function); did you mean 'SAM_SYSC_VSECTION'?

one config of arm07 build breaks, it should be fixed to pass check build : (

That is not relevant to this change, is it?  That is a SAMA5Dx and has nothing to do with LittleFS.  I will continue with the merge.

ok, thanks

repo: apache/incubator-nuttx


text:
board/Board.mk: Make romfs_img 4 byte aligned

Summary
Impact
None
Testing
CI and custom board

@no1wudi shouldn't https://github.com/apache/incubator-nuttx/blob/master/tools/mkromfsimg.sh he updated as well?


text:
test(services/dropbox): enable CI test for dropbox

Add test for service Dropbox.
Because of one specific limit, we should use cargo test to make tests run in sequence.
https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/Too-many-write-operations-when-moving-a-folder/td-p/266160
cc @Zheaoli

Seems that we need to add stat for the Dropbox service.

Seems that we need to add stat for the Dropbox service.
It might also want to check if the root is in effect.

SGTM, I will make a new PR to update it

Seems that we need to add stat for the Dropbox service. It might also want to check if the root is in effect.

Please creata seperate issues for dropbox with label services/dropbox so that others can take part in.

Please creata seperate issues for dropbox with label services/dropbox so that others can take part in.

Created with #2510

Let's close this PR until we can make some progress.

authors: Xuanwo Zheaoli suyanhanx

repo: apache/incubator-opendal


text:
Fix awk command in init.sh generating resolvers.conf file

Fixes #312

@baxtersa Thanks for identifying this problem and addressing the script command. Out of curiosity, have you signed an Apache ICLA?  If you have not, and wish to contribute more extensive code fixes, please consider it.

@mrutkows No problem! I submitted an Apache ICLA last week. I think I should be set on that front.

authors: baxtersa mrutkows

repo: apache/incubator-openwhisk-apigateway


text:
Use git commit hash instead of HEAD in rcverify instructions.

@dgrove-oss this will replace HEAD in the instructions with the result of git show-ref --hash --abbrev -- refs/remotes/upstream/master  (which requires that upstream is set in the remotes). I could replace this with refs/heads/master but it could be brittle (without further sanity checks).

Seems like a reasonable tradeoff to me.  Just another pre-req to document in how we expect a release manager to setup their environment that the local clone of this repo should have a remote called upstream

authors: dgrove-oss rabbah

repo: apache/incubator-openwhisk-release


text:
Update docs. to relect latest runtime language versions/support

Describe updates, for NodeJS, Java and Python runtimes with new versions/features.

Added Ballerina runtime to home/docs pages.

repo: apache/incubator-openwhisk-website


text:
Add documentation on API Gateway deployment

The programming guide should include a section on the API Gateway to demonstrate the basics of how to configure it.

We have API Gateway doc here:
https://github.com/apache/incubator-openwhisk-wskdeploy/blob/master/docs/wskdeploy_apigateway_helloworld.md
Closing this issue.

source: gharchive/issue

authors: brunogirin pritidesai

repo: apache/incubator-openwhisk-wskdeploy


text:
Add Test to Ensure Preservation of Annotations for Action Web Flag

Using the --web flag during an action update call does not preserve the action annotations.

PG4 371

@mdeuser, please review.

PG approved.

@mdeuser, please review again. Now this PR contains only a test as the code change was merged elsewhere.

@mdeuser, I made changes made based on your review comments.

LGTM

@bjustin-ibm, can you merge this one?

authors: dubeejw mdeuser

repo: apache/incubator-openwhisk


text:
[Feature] provide a flink option to chose which log to read from

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation
For now, a paimon table can keep log both in file store and external log system at the same time , by setting log.system=kafka and changelog-producer=input/full-compaction at the same time.
But at the source side, if log.system=kafka is set, the continues read is bound to kafka, and file store log will be ignored.
Since log of file store can support some feature that external log system doesn't support, like full-changelog, I think it's reasonable to provide a option for user to chose which log to read from.
Solution
Provide a flink option scan.log.from=external/internal, or sth like that to expose both choices.
Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

I have interest to work for this issue. Please assign this issue to me.

authors: Alibaba-HZY nonggialiang

repo: apache/incubator-paimon


text:
[Flink] Avoid unexpeted Num of  L0 File

When writing Paimon in Flink, Even if compaction fails, there will still be new L0 files commited successfully. The motivation behind this PR is to reduce the number of L0 files .
Waiting for the merge to complete during preCommit if there are too many Sorted Runs.

fix #1391

@JingsongLi cc

related #1375


text:
correct values for project-info.conf jdk-versions

currently have:
  shared-info {
    jdk-versions: ["Adopt OpenJDK 8", "Adopt OpenJDK 11"]

Adopt JDK is no longer supported. Maybe Temurin or Adoptium.
Generally need to work out what we want to appear in the docs.

#293 merged

repo: apache/incubator-pekko


text:
Add QueryContext to replace BrokerRequest in the query engine

Motivation:
For historic reason, BrokerRequest does not support expressions natively.
In order to support expressions (transform), the current solution is to
store the expressions as String and compile them when constructing the
operators.
The problem with this approach is that the same compilation is performed
multiple times on EACH segment. This could cause performance degradation
when each server is hosting lots of segments.
In this PR, we introduced a new Object - QueryContext to encapsulate all
the query related information to be used by the query engine. It is
already fully compiled and can be shared by all the segments.
The reason of introducing the new QueryContext over using the PinotQuery
is that with QueryContext:

It is not used in the wiring layer so that execution layer can be
decoupled from the wiring layer, and changes for one layer won't affect
the other layer.
It is very hard to change wiring Object (PinotQuery) because it involves
protocol change, so we should make it as generic as possible to support
future features. Instead, QueryContext can be upgraded along with the
new future support in query engine as needed, and we don't have to make
it very generic, which can help save the overhead of handling generic
Objects.
In case we need to change the wiring Object (e.g. switch from Thrift to
Protobuf), we don't need to change the whole query engine.
We can add some helper variables or methods in the context classes which
can be shared for all segments to reduce the repetitive work for each
segment.

This PR only introduces the new QueryContext, the related context classes
and a Util class to get QueryContext from BrokerRequest. The following PRs
will make incremental changes to move the query engine to use the new
QueryContext.

Should we solve the issues in BrokerRequest, or move to PinotQuery? Latter would be ideal, except in organizations with large Pinot deployments, moving to SQL may not happen immediately.  This implies, that we would need to build PinotQuery out of brokerRequest (instead of the other way round).

@kishoreg @mayankshriv Updated the description on why I prefer adding this context Object over using PinotQuery directly, and revised the code. QueryContext can be extracted from PinotQuery if exists, or BrokerRequest if PinotQuery is not available. It serves as the query context to isolate the wiring layer and the transport layer, and all the segment independent information can be stored within it to save the repetitive work currently happening inside each segment (e.g. creating Predicate).

@kishoreg @mayankshriv @siddharthteotia This PR is ready for review.

It seems like we are adding a some duplicate code (like Predicate, FilterInfo). We already have them.
I agree that on server we can move to QueryRequest and build all the execution related information once (segment independent). But that can also be done while we continue to leverage existing interfaces. We already have Expression, FilterQueryTree, Predicate etc. Why do we need a new implementation

It seems like we are adding some duplicate code (like Predicate, FilterInfo). We already have them.
I agree that on server we can move to QueryRequest and build all the execution related information once (segment independent). But that can also be done while we continue to leverage existing interfaces and POJOs. We already have Expression, FilterQueryTree, Predicate etc. Why do we need a new implementation?

This PR introduces all the classes needed to replace the BrokerRequest. The reason for introducing new classes instead of changing the existing ones is that we can make incremental changes to replace them layer by layer (aggregation, selection, transform, projection, filter, etc.) without touching other layers. After deprecating BrokerRequest completely, the old classes will be removed.
TransformExpressionTree -> Expression
FilterQueryTree -> FilterInfo
Predicate -> new Predicate with compiled expressions

I missed reviewing the converter earlier. Will do another pass today.

In general this looks fine to me. I am just worried about the confusion that having so many classes with duplicate names will cause.

@mayankshriv I thought about it and actually already tried with multiple names, and finally decide to go with the most simple and intuitive ones, although they are the same as the thrift ones for the following reasons:

The thrift and context classes will only co-exist in the converter, on the engine side only the context ones will be used; on the wiring side, only the thrift ones will be used.
I tried to append Info after each class, but we already have a class named FunctionInfo, and IMO it is easier to distinguish thrift class from context class comparing to another POJO class.
With the most simple and intuitive class name, it can improve the readability of the code, as long as the import is correct (IDE should be able to handle this properly)

Approving this since overall I am onboard with the change. My only request is to please try and make one more attempt at keeping separate names unless absolutely impossible. Even if the new name is ugly, it is fine IMHO.

@siddharthteotia Appended Context to these classes to differentiate them from the thrift classes.

Approving this since overall I am onboard with the change. My only request is to please try and make one more attempt at keeping separate names unless absolutely impossible. Even if the new name is ugly, it is fine IMHO.

@siddharthteotia Appended Context to these classes to differentiate them from the thrift classes.

Thanks @Jackie-Jiang . Let's merge as soon as we get a clean build.

repo: apache/incubator-pinot


text:
java.io.NotSerializableException: java.io.StringReader

When I insert data into a field of text type in the database, I report the following error. It seems that sharding has rewritten this field. How to solve this problem?
java.io.NotSerializableException: java.io.StringReader
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at com.mysql.jdbc.PreparedStatement.setSerializableObject(PreparedStatement.java:3824)
at io.shardingsphere.shardingjdbc.jdbc.adapter.invocation.JdbcMethodInvocation.invoke(JdbcMethodInvocation.java:47)
at io.shardingsphere.shardingjdbc.jdbc.adapter.AbstractShardingPreparedStatementAdapter.replaySetParameter(AbstractShardingPreparedStatementAdapter.java:264)
at io.shardingsphere.shardingjdbc.jdbc.core.statement.ShardingPreparedStatement.setParametersForStatements(ShardingPreparedStatement.java:194)
at io.shardingsphere.shardingjdbc.jdbc.core.statement.ShardingPreparedStatement.initPreparedStatementExecutor(ShardingPreparedStatement.java:189)
at io.shardingsphere.shardingjdbc.jdbc.core.statement.ShardingPreparedStatement.execute(ShardingPreparedStatement.java:136)

Duplicate with https://github.com/apache/incubator-shardingsphere/issues/1567.
Will be fixed together.

Fixed via https://github.com/apache/incubator-shardingsphere/pull/2388.

repo: apache/incubator-shardingsphere

license_source: bigquery


text:
Table modification of sub-table data error reporting

Tables with separate tables to modify data errors，My approach uses a Distributed Annotation @TxcTransaction

Bad SQL grammar, please check you SQL first

错误的SQL语法，请先检查SQL
No problem with sql. I can put it directly into the library and execute it. Is there any configuration missing?

Can you fill necessary information? the template is below:

Bug Report
For English only, other languages will not accept.
Before report a bug, make sure you have:

Searched open and closed GitHub issues.
Read documentation: ShardingSphere Doc.

Please pay attention on issues you submitted, because we maybe need more details.
If no response more than 7 days and we cannot reproduce it on current information, we will close it.
Please answer these questions before submitting your issue. Thanks!
Which version of ShardingSphere did you use?
Which project did you use? Sharding-JDBC or Sharding-Proxy?
Expected behavior
Actual behavior
Reason analyze (If you can)
Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.
Example codes for reproduce this issue (such as a github link).

@wanggang931111 Can you explain the reason for issue closed?

@wanggang931111你能解释一下问题的原因吗？
It's the problem of distributed transaction in LCN, change this annotation it will be all right,but I don't know if it has compatibility with middleware.


text:
SINGA-61 Support user defined classes

To make SINGA extensible, polymorphism is used for many classes, e.g., Layer, Updater. Users configure a type field to decide which subclass to use at run time.
E.g., the LayerProto has a type field. But this type field can only be set to built-in layer types.
This commit adds a user_type field for configuring user defined layers. E.g.,
LayerProto {
  optional string user_type = 10;
}

A helper function Layer::Create(const LayerProto&) is added to create a Layer object based on user configuration.
Similar updates are applied for other classes, namely, Param, Worker, Updater.
Driver class in declared in driver.h.
singa.h now has only include directions.

Please re-create data shards for mnist and cifar10 examples, as the protocol buffer message for Record has been changed.

repo: apache/incubator-singa


text:
Feature: update spans display

1.Add top childrens and top slow span.
2.fix windowresize bug.
3.fix time and span tiptip connection.
4.Optimized tree rendering algorithm.

I have two suggestions

TopSlow should be 'Top 5 of slow span', also in this mode, span duration could show itself only, not include children spans. Because you calculate the list by this way.
TopSpan should be 'Top 5 of children span number'. And show the child spans number, only if possible, this is not as important as span duration.

Cool. Like this

I will write an English and Chinese document for this new page.

😄thanks！

Please send a pull request to sync main repo. @TinyAllen

authors: TinyAllen wu-sheng

repo: apache/incubator-skywalking-ui


text:
URLFrontier spout doesn't take into account crawl Id

The URLFrontier Spout
( https://github.com/apache/incubator-stormcrawler/blob/main/external/urlfrontier/src/main/java/org/apache/stormcrawler/urlfrontier/Spout.java ) doesn't take into account the crawl Id that can be specified in the configuration parameters (URLFRONTIER_CRAWL_ID_KEY = "urlfrontier.crawlid" defined in  org.apache.stormcrawler.urlfrontier.Constants)
This results in a mix of URLs coming from distinct frontiers in URLFrontier.

Would you Like to submit a PR with a fix?

Would you Like to submit a PR with a fix?

Yes, I will but  won't be able to do before 2 weeks.

repo: apache/incubator-stormcrawler


text:
[Improve] Update How-to-release.md

update release step and fix link

cc @caicancai

repo: apache/incubator-streampark-website


text:
Feature: Dashboard filters: datepicker and time granularity

Make sure these boxes are checked before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if any
[x] I have reproduced the issue with at least the latest released version of superset
[x] I have checked the issue tracker for the same issue and I haven't found one similar

When checking "Date" under options in the dashboard filter explore view, only the default time options for since and until are available, with no ability to set time granularity.

The time granularity option and datepicker should be available for dashboard filters.
Superset version
0.19.3
Expected results
Dashboard date filters should have time granularity and datepicker.
Actual results
Dashboard date filters only allow the default time options

Why did you close this item? I have the same problem.

Huh that's odd, I must've hit it by accident. Anyway I did make a PR for this https://github.com/apache/incubator-superset/pull/3508

Notice: this issue has been closed because it has been inactive for 206 days. Feel free to comment and request for this issue to be reopened.

Not stale

authors: Mogball gsemyonov mistercrunch sh0gg0th

repo: apache/incubator-superset


text:
Superset Issue #4323

Adding configurable poll interval for Hive Engine

I'm pretty sure that test failure is unrelated to changes I made.


text:
Fix time filter in dashboard v2

@mistercrunch:

Codecov Report

Merging #5412 into master will not change coverage.
The diff coverage is 0%.

@@           Coverage Diff           @@
##           master    #5412   +/-   ##
=======================================
  Coverage   61.21%   61.21%
=======================================
  Files         373      373
  Lines       23709    23709
  Branches     2750     2750
=======================================
  Hits        14513    14513
  Misses       9181     9181
  Partials       15       15

Impacted Files
Coverage Δ

...et/assets/src/dashboard/reducers/dashboardState.js
94.59% <ø> (ø)
:arrow_up:

.../explore/components/controls/DateFilterControl.jsx
60.55% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b4e6c7...40e5788. Read the comment docs.

authors: betodealmeida codecov-io


text:
[SYSTEMML-655] Utilizes java from Yarn environment

Patch provided by Greg Senia

PR Merged

repo: apache/incubator-systemml


text:
[IOTDB-4779] Fix remove metrics in Metric Module

See: https://issues.apache.org/jira/browse/IOTDB-4779

LGTM

authors: SpriCoder xinzhongtianxia

repo: apache/iotdb


text:
Bug58980

Proposed fix for  bug58980, allowing JMS Subscriber to check and fail if not enough messages are received.

Hi,
Thanks for PR.
Do you think 404 is a good response code ?
Why not a 500 ?
Regards

I was not sure exactly what to put, 404/500. Leaning against 404 because some messages were found, but against 500 because we know what the error is. If we are using 404 to mean no message was found, then it should be changed to 500.

authors: Htermotto pmouawad

repo: apache/jmeter


text:
KAFKA-13512: topicIdsToNames and topicNamesToIds allocate unnecessary maps

We are creating a new map unnecessarily for these methods. Remove the extra map creation and simply wrap in unmodifiable map.
I've also added a benchmark for the map method.
Here are some results when I limited partitions to 20 only.
Before change:
Benchmark                                 (partitionCount)  (topicCount)  Mode  Cnt   Score   Error  Units
MetadataRequestBenchmark.testTopicIdInfo                20           500  avgt   15  16.942 ± 0.306  ns/op
MetadataRequestBenchmark.testTopicIdInfo                20          1000  avgt   15  19.476 ± 0.339  ns/op
MetadataRequestBenchmark.testTopicIdInfo                20          5000  avgt   15  18.989 ± 0.482  ns/op

After change:
Benchmark                                 (partitionCount)  (topicCount)  Mode  Cnt   Score   Error  Units
MetadataRequestBenchmark.testTopicIdInfo                20           500  avgt   15  11.120 ± 0.336  ns/op
MetadataRequestBenchmark.testTopicIdInfo                20          1000  avgt   15  11.173 ± 0.489  ns/op
MetadataRequestBenchmark.testTopicIdInfo                20          5000  avgt   15  11.003 ± 0.042  ns/op

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

The MetadataSnapshot is immutable so this seems fine to me as well.

@cmccabe This is a blocker for 3.1. Could also merge it to the 3.1 branch please?

authors: dajac jolshan

repo: apache/kafka


text:
MINOR; Update scalafmt to latest version

Updates scalafmt to the latest version. This not only means migrating
the checkstyle/.scalafmt.conf file but updating spotless as well. Due
to improvements in the new version of scalafmt, it now also
alphabetically sorts imports.
The diff to the actual .scala source files is due to the alphabetical sorting of imports. I verified that this PR passes spotless/scalacheck by running  ./gradlew :spotlessScalaCheck locally.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@divijvaidya This is one of the first PR's in adding the linting/formatting using scalafmt/scalafix for the Scala sources within Kafka if you want to have a look. @cadonna Do you want to have a look as well since its part of kafka scala streams?

The scala 2.12 build failed https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-12475/

@ijuma Thanks for the pickup.
I have investigated the root cause of the problem, basically when you specify the scala version with -PscalaVersion=2.12 it also happens to override the Scala version that is used by scalafmt and the latest version of scalafmt (3.5.8) only runs with Scala 2.13.x. This is normally a non issue because scalafmt is designed to be used like a cli tool, i.e. its meant to be run independently of your project.
I created an issue upstream at https://github.com/diffplug/spotless/issues/1273

@ijuma So I have added a commit which adds a workaround for the Scala runtime conflict issues. The CI is failing but it appears it's not using the updated Jenkinsfile in the commit so I suspect that after a merge the Apache CI should use the updated Jenkinsfile?
@mimaison For visibility as well.

Actually lets wait on merging this PR, it may be possible to fix the underlying problem in spotless (see https://github.com/diffplug/spotless/issues/1273#issuecomment-1222221116)

@ijuma So I have some good news, I managed to completely the solve the underlying problem upstream (see https://github.com/diffplug/spotless/issues/1273#issuecomment-1224978741). The PR has been updated/rebased and as you can see spotlessScalaCheck even works when Scala 2.12 is set, i.e.
./gradlew -PscalaVersion=2.12 :spotlessScalaCheck

Also tagging @mimaison for visibility

Tagging @C0urante for visibility

I've kicked off another Jenkins run to see if we can get at least one green build with Scala 2.12.

So Jenkins did in fact pass this PR as green build. I have just pushed to this PR so another build has been triggered but https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-12475/8/pipeline is the build that passed.

@vvcephei @cadonna would either of you have time to double-check this quick Scalafmt bump?

authors: C0urante ijuma mdedetrich


text:
KAFKA-14869: Ignore unknown record types for coordinators (KIP-915, P…

…art-1) (#13511)
This patch implemented the first part of KIP-915. It updates the group coordinator and the transaction coordinator to ignores unknown record types while loading their respective state from the partitions. This allows downgrades from future versions that will include new record types.
Reviewers: Alexandre Dupriez alexandre.dupriez@gmail.com, David Jacot djacot@confluent.io
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@jeffkbkim This PR has failures related to the change. Could you take a look?
Build / JDK 11 and Scala 2.13 / testLoadGroupsAndOffsetsWillIgnoreUnknownMessage() – kafka.coordinator.group.GroupMetadataManagerTest
<1s
Build / JDK 11 and Scala 2.13 / testLoadGroupsAndOffsetsWillIgnoreUnknownMessage() – kafka.coordinator.group.GroupMetadataManagerTest
<1s
Build / JDK 16 and Scala 2.13 / testLoadGroupsAndOffsetsWillIgnoreUnknownMessage() – kafka.coordinator.group.GroupMetadataManagerTest
<1s
Build / JDK 16 and Scala 2.13 / testLoadGroupsAndOffsetsWillIgnoreUnknownMessage() – kafka.coordinator.group.GroupMetadataManagerTest
<1s
Build / JDK 8 and Scala 2.12 / testLoadGroupsAndOffsetsWillIgnoreUnknownMessage() – kafka.coordinator.group.GroupMetadataManagerTest
<1s
Build / JDK 8 and Scala 2.12 / testLoadGroupsAndOffsetsWillIgnoreUnknownMessage() – kafka.coordinator.group.GroupMetadataManagerTest

authors: dajac jeffkbkim


text:
KAFKA-15290: Handle topic-level dynamic remote storage enable configuration

To onboard existing topics to tiered storage, bootstrap the remote-log-components when updating the dynamic remote.storage.enable config on the topic.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Do we need to implement dynamically configurable remote storage related things in a DynamicRemoteLogManagerConfig class similar to https://github.com/apache/kafka/pull/14243/files ? (I haven't looked into the code in detail yet so asking for your view point)

I took a pass on #14243, there we are changing the broker level configs that are related to remote storage. We can extend the DynamicRemoteLogManagerConfig with dynamic remote log read/write quota, thread-pool size and so on.
In this PR, we are changing the topic-level configs. remote.storage.enable config is applicable only at the topic level.

@clolov @divijvaidya @showuon @satishd
Call for review. PTAL.

Integration test for this patch was covered in RemoteTopicCrudTest#testEnableRemoteLogOnExistingTopicTest.

Filed KAFKA-15400 to fix the flaky RemoteIndexCacheTest

There are a few unrelated test failures. Merging to trunk.

Merging to 3.6 branch.

authors: kamalcph satishd


text:
KAFKA-15428: Cluster-wide dynamic log adjustments for Connect

Jira, KIP-976
Adds a new scope URL query parameter to the existing PUT /admin/loggers/{logger} endpoint, which accepts values of worker (default, modifies logging levels for the worker that receives the request) and cluster (modifies logging levels for all workers in the cluster).
In addition, adds a new last_modified field to the responses from the GET /admin/loggers and GET /admin/loggers/{logger} endpoints, which contains the Unix epoch (in milliseconds) for when the logger was last modified (or null, if no dynamic modifications have taken place).
Unit, integration, and system tests are included, as detailed in the KIP's test plan.
This PR is divided into three commits:

Functional changes and unit tests
Integration tests (which includes adding support for running embedded standalone workers)
System tests

Each can be reviewed separately, but they are included together in this PR in order to simplify iteration as it's possible (likely, even) that addressing review comments made on one commit will require changes to be made for files touched on in another commit. But if it makes life easier for reviewers, I'm happy to break this down into three separate PRs.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@gharris1727, @yashmayya, @mimaison would any of you be able to take a look?

Ah, never mind--I realize that some static fields for timeouts from the EmbeddedConnectClusterAssertions class may be used in downstream projects. I've pushed a new commit with the suggested deprecated shim. Thanks again for catching this, Greg!

Thanks for the reviews, all!


text:
KAFKA-15704: Set missing ZkMigrationReady field on ControllerRegistrationRequest

This field was missed by the initial KIP-919 PR(s). The result is that migrations can't begin since the controllers will never become ready. This patch fixes that as well as pulls over some fixes from the 3.6 branch.

committed, thanks!

authors: cmccabe mumrah


text:
KAFKA-15853: Move ClientQuotaManagerConfig outside of core

blocker for #15158
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@mimaison
@ijuma
This PR are ready for merge for me. Please, take a look.

Hello @OmniaGM
Can you, please, omit force push to the PR's? :)
This breaks commit history and reviewers (like I am) don't understand what have changed with force push.
PR will be squashed to the single nice looking commit during the merge, anyway.

@OmniaGM This needs rebasing too. Thanks

Hello @OmniaGM
Can you, please, omit force push to the PR's? :) This breaks commit history and reviewers (like I am) don't understand what have changed with force push.
PR will be squashed to the single nice looking commit during the merge, anyway.

I do this usually with medium/big prs but as this one small and we have a chain of prs relays on each others for this works this makes it easier to me to cherry-picking kinda for me.

@OmniaGM This needs rebasing too. Thanks

done

authors: OmniaGM mimaison nizhikov


text:
KAFKA-16629: Add broker-related tests to ConfigCommandIntegrationTest

wip, add new KRAFT test for ConfigCommandIntegrationTest

Because Old Test Only have tests with Zookeeper, thus I change old test for Zookeeper and Kraft, and add new tests for KRaft, I mainly test the the config update mode for configs

Hello @chia7712 please review, Thank you

@m1a2st please fix the build error

@chia7712 , I have been add zookeeper --bootstrap-server tests, please review, thank you

错误
org.apache.kafka.common.errors.InvalidConfigurationException: All sensitive broker config entries must be specified for --alter, missing entries: Set(message.max.size)
栈跟踪
org.apache.kafka.common.errors.InvalidConfigurationException: All sensitive broker config entries must be specified for --alter, missing entries: Set(message.max.size)

@m1a2st Could you please take a look at above failures?

@chia7712, Please review, Thanks for your comments.

@m1a2st Could you please fix conflicts?

@chia7712 , I have been resolve th conflict.

@m1a2st please fix the build error


text:
KAFKA-17277: [1/2] Add version mapping command to the storage tool and feature command tool

As a part of KIP-1022 the following has been implemented in this patch:

A version-mapping command to to look up the corresponding features for a given metadata version. Using the command with no --release-version argument will return the mapping for the latest stable metadata version.
This command has been added to the FeatureCommand Tool and the Storage Tool.
The storage tools parsing method has been made more modular similar to the feature command tool

Can we also check the java 8 build? Seems like compilation failed.


text:
KAFKA-4254: Update producers metadata before failing on non-existent …

…partition

Also rebased over latest trunk.

Thanks again for the comments. Mostly addressed. Additional test is pending.

Code coverage has increased by extending the initial mocking-based test and by adding a new test  testing end-to-end record sends before and after partitions are added. Let me know if you find the coverage sufficient.

Thank you @ijuma. The PR has been updated and I'd appreciate if both you and @hachikuji could take a look.

Minor comment on the javadoc, but the rest looks good.

LGTM. Thanks for the patch! Merging to trunk and 0.10.1.

Thank you for your comments!

authors: hachikuji kkonstantine


text:
KAFKA-4372: Kafka Connect REST API does not handle DELETE of connector with slashes in their names

Kafka Connect REST API does not handle in many places connectors with slashes in their names because it expects PathParams, this PR intends to :

Reject as bad requests API calls trying to create connectors with slashes in their names
Add support for connector with slashes in their names in the DELETE part of the API to allow users to cleanup their connectors without dropping everything.

This PR adds as well the Unit Test needed for the creation part and was tested manually for the DELETE part.

The test failures do not seem to be related to the PR

Aside from open Q about whether or not to disallow and when, LGTM
cc @ewencp or @hachikuji for thoughts & review

Fixed code styling in tests too

LGTM, thanks!

thx

authors: ewencp ogirardot shikhar


text:
MINOR: Arrays.toList replaced with Collections.singletonList() where possible.

This is something I did after my working hours, I would ask people reviewing this do the same, don't take time for this during your work hours. It's not actual functionality, it is an internal rewrite based on suggestions provided by the static code analysis built into the Intellij IDE.
I try to keep such a PR as limited as possible, for clarity of reading.
==========
In places where Arrays.asList() is given exactly 1 argument, replace it with Collections.singletonList().
Internally, this is a smaller object, so it uses a bit less memory at runtime.
An important thing to note is that you cannot add anything to such a list. So if it is returned through a method into a random List<> object, performing add() on it will throw an exception.
I checked every usage of all replaced instances, they're only ever returned to be read, so this should not occur. A
One might say this is a micro-optimization, I would say that's true, it is.
It's up to the maintainers whether or not they want this in. Personally I checked all code usages and did not find a point where an exception would be caused because of these changes. So it should not break anything and provide a tiny improvement in terms of memory footprint at runtime.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

FAILURE
No test results found.
--none--

FAILURE
No test results found.
--none--

FAILURE
No test results found.
--none--

Oh dear, never seen that before. What causes this?

retest this please

Oh, I totally forgot about this and the other one. Looks like the "No test results found." stuff happened once again.

There are unused imports that are causing checkstyle to fail.

Merged in trunk and cleaned up the 2 unused imports.

authors: KoenDG asfgit ijuma


text:
MINOR: Simplify Node.hashCode and equals

This can be a bottleneck in RecordAccumulator.ready if the number of
brokers and partitions is large enough. We just use the id since it
should be unique.
Originally reported in https://github.com/apache/kafka/pull/4350:
"Faced with the producer performance degradation in case of high
load and large number of brokers (100), topics (150) and partitions (350).
Made several diagnostic records with the java flight recorder and found that the method HashSet::contains in RecordAccumulator::ready takes about 40% of the whole time of the application. It is caused by re-calculating a hash code of a leader
(Node instance) for every batch entry."
That PR cached the hashCode, which still requires computing the
complex hashCode once and does not make equals cheap.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Any reason why we should not do this?

The only thing I was wondering is whether this would make usage a bit trickier since we wouldn't be able to compare Node objects retrieved from separate metadata requests.
Note that the test failure looks relevant to this patch.

retest this please

Closing this as we merged the alternative that caches the hash code.

authors: hachikuji ijuma


text:
KAFKA-10449: Add some important parameter desc in connect-distributed.properties

…rties.
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

In WokerConfig.java we found that REST_HOST_NAME_CONFIG(rest.host.name) and REST_PORT_CONFIG(rest.port) were deprecated. And some new configuration parameters are introduced such as LISTENERS_CONFIG(listeners), REST_ADVERTISED_LISTENER_CONFIG(rest.advertised.listener),ADMIN_LISTENERS_CONFIG(admin.listeners) but not list in the sample conf file.

Thanks for the PR @tinawenqiao.
It would be nice to also mention rest.port and rest.host.name are deprecated in their descriptions. Could you add something like:
DEPRECATED: only used when listeners is not set. Use listeners instead.  to their descriptions in WorkerConfigs?
Thanks

Thanks for the PR @tinawenqiao.
It would be nice to also mention rest.port and rest.host.name are deprecated in their descriptions. Could you add something like:
DEPRECATED: only used when listeners is not set. Use listeners instead.  to their descriptions in WorkerConfigs?
Thanks

Thanks for your notice. A new patch is ready.

@tinawenqiao It looks like you made a mistake when rebasing your branch, can you ensure the PR only contains your diff? Thanks

@tinawenqiao It looks like you made a mistake when rebasing your branch, can you ensure the PR only contains your diff? Thanks

Sorry for the mistake. A clean patch is submit. Is it ok now?

Thanks for the update, it looks much better.
I should have been clearer, I meant to update the documentation text of these settings, not just the comments in WorkerConfigs.
For example in https://github.com/apache/kafka/pull/9235/files#diff-5664ddd985065623bab1870e031ce4f3fcfee0bc4951e41b6b0e1584cd32fa2dR153, we could do something like:
private static final String REST_HOST_NAME_DOC
    = "Hostname for the REST API. If this is set, it will only bind to this interface. Deprecated, only used when listeners is not set. Use listeners instead.";

Thanks for the update, it looks much better.
I should have been clearer, I meant to update the documentation text of these settings, not just the comments in WorkerConfigs.
For example in https://github.com/apache/kafka/pull/9235/files#diff-5664ddd985065623bab1870e031ce4f3fcfee0bc4951e41b6b0e1584cd32fa2dR153, we could do something like:
private static final String REST_HOST_NAME_DOC
    = "Hostname for the REST API. If this is set, it will only bind to this interface. Deprecated, only used when listeners is not set. Use listeners instead.";

Doc desc has been added.

Thanks for the update, it looks much better.
I should have been clearer, I meant to update the documentation text of these settings, not just the comments in WorkerConfigs.
For example in https://github.com/apache/kafka/pull/9235/files#diff-5664ddd985065623bab1870e031ce4f3fcfee0bc4951e41b6b0e1584cd32fa2dR153, we could do something like:
private static final String REST_HOST_NAME_DOC
    = "Hostname for the REST API. If this is set, it will only bind to this interface. Deprecated, only used when listeners is not set. Use listeners instead.";

Doc desc has been added.

@mimaison is it ok now?

authors: mimaison tinawenqiao


text:
KYLIN-5292 broadcast the kill event when discard job. on the other ha…

…nd, JobServer listens to the event to destroy SparkSubmit.
Proposed changes
The destroy process can only be carried out on the node where SparkSubmit runs, so need to broadcast the kill event when discard job.
After the patch:

Branch to commit

[ ] Branch kylin3 for v2.x to v3.x
[x] Branch kylin4 for v4.x
[ ] Branch kylin5 for v5.x

Types of changes
What types of changes does your code introduce to Kylin?
Put an x in the boxes that apply

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices apply)

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have created an issue on Kylin's jira, and have described the bug/feature there in detail
[x] Commit messages in my PR start with the related jira ID, like "KYLIN-0000 Make Kylin project open-source"
[ ] Compiling and unit tests pass locally with my changes
[x] I have added tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation (if appropriate)
[ ] Any dependent changes have been merged

Further comments
If this is a relatively large or complex change, kick off the discussion at user@kylin.apache.org or dev@kylin.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

retest this please.

retest this please

retest this please

retest this please

retest this please

retest this please

retest this please

@Mukvin , Thanks for your suggestion, I will refine this

repo: apache/kylin


text:
Align ScalaTest Plus plugin versions and bump ScalaTest from 3.2.9 to 3.2.15

Why are the changes needed?

bump ScalaTest version from 3.2.9 to 3.2.15, updated to use same scala version 2.12.17 in Kyuubi. (Release notes: https://github.com/scalatest/scalatest/releases/tag/release-3.2.15)
bump scalatest-maven-plugin from 2.0.2 to 2.2.0 (https://github.com/scalatest/scalatest-maven-plugin/releases/tag/release-2.2.0)
align scalatestplus versions to the version above, removing the misleading scalacheck.version property, (ScalaTest + ScalaCheck Version: https://www.scalatest.org/plus/scalacheck/versions)
bump scalatestplus plugins to 3.2.15.0 with bumping dependency

scalatestplus-scalacheck (https://github.com/scalatest/scalatestplus-scalacheck/releases/tag/release-3.2.15.0-for-scalacheck-1.17)
scalatestplus-mockito (https://github.com/scalatest/scalatestplus-mockito/releases/tag/release-3.2.15.0-for-mockito-4.6)
mockito from 3.4 to 4.6 (https://github.com/mockito/mockito/releases/tag/v4.6.0)
scalacheck from 1.15 to 1.17 (https://github.com/typelevel/scalacheck/releases/tag/v1.17.0)

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

cc @pan3793
All CI checks passed. Should be good to go.

Thanks, merged to master

repo: apache/kyuubi


text:
[LIBCLOUD-1002] Add instance template support to GCE create_node

Instance template support for GCE create_node
Allows Nodes to be created from GCE Instance Templates
This change allows Nodes to be created from predefined Instance Templates in GCE.
This is useful in a number of scenarios:

Organizational standards are implemented with Templates.
etc.

The attributes specified in the call to create_node will take precedence over those defined in the template (per provider documentation).
Provider docs:

https://cloud.google.com/compute/docs/reference/rest/v1/instances/insert
https://cloud.google.com/compute/docs/instances/create-vm-from-instance-template

Status

done, ready for review

Checklist (tick everything that applies)

[x] Code linting (required, can be done after the PR checks)
[x] Documentation
[x] Tests
[ ] ICLA (required for bigger changes)

Perhaps the signature for create_node should be modified to accept a generic template parameter rather than the provider specific ex_source_instancetemplate and ex_launch_template_* ( #1224 ).  It would actually be helpful for the specific project I'm working on, but I'm not sure how this project feels about changes to signatures like that.

Perhaps the signature for create_node should be modified to accept a generic template parameter rather than the provider specific ex_source_instancetemplate and ex_launch_template_* ( #1224 ). It would actually be helpful for the specific project I'm working on, but I'm not sure how this project feels about changes to signatures like that.

If we can abstract template functionality in a standard base class, I would be fine with that.

authors: Kami rojopolis

repo: apache/libcloud


text:
Support deletes in IndexRearranger

Description
IndexRearranger applies deletes found in the original index, so the rearranged index will not contain any deleted docs. While deleted docs don't show up in search results, they still impact search performance. If we want the rearranged index to be as similar as possible to the original index, we should also include the docs marked for deletion.

IndexRearranger now supports selecting docs for deletion from the original index and applying the deletes to the rearranged index.

repo: apache/lucene


text:
METRON-1850: Stellar REST function

Contributor Comments
This PR adds a Stellar REST function that can be used to enrich messages with data from 3rd party REST services.  This function leverages the Apache HttpComponents library for Http requests.
The function call follows this format:  REST_GET(rest uri, optional rest settings)
There are a handful of settings including basic authentication credentials, proxy support, and timeouts.  These settings are included in the RestConfig class.  Any of these settings can be defined in the global config under the stellar.rest.settings property and will override any default values.  The global config settings can also be overridden by passing in a config object to the expression.  This will allow support for multiple REST services that may have different requirements.
Responses are expected to be in JSON format.  Successful requests will return a MAP object in Stellar.  Errors will be logged and null will be returned by default.  There are ways to override this behavior by configuring a list of acceptable status codes and/or values to be returned on error or empty responses.
Considering this will be used on streaming data, how we handle timeouts is important.  This function exposes HttpClient timeout settings but those are not enough to keep unwanted latency from being introduced.  A hard timeout is implemented in the function to abort a request if the timeout is exceeded.  The idea is to guarantee the total request time will not exceed a configured value.
Changes Included

HttpClient capability added to Stellar
HttpClient setup added to the various bolts and Stellar REPL
Utility added for setting up a pooling HttpClient (and possibly other types of clients in the future)
Configuration mechanism added for Stellar REST function including settings for authentication, proxy support, timeouts and other settings
Function implementation and appropriate unit/integration tests (both unit tests and integration tests are included)

Testing
There are several different ways to test this feature and I would encourage reviewers to get creative and look for cases I may not have thought of.  For my testing, I used an online Http service that provides simple endpoints for simulating different use cases:  http://httpbin.org/#/.  Feel free to try your own or use this one.
I tested this in full dev using the Stellar REPL and the parser and enrichment topologies.  First you need to perform a couple setup steps:

Spin up full dev and ensure everything comes up and data is being indexed
Ssh to full dev and install the Squid proxy server:

yum -y install squid

Create a password file that Squid can use for basic authentication

yum -y install httpd-tools
touch /etc/squid/passwd && chown squid /etc/squid/passwd
htpasswd /etc/squid/passwd user # (Will prompt for a password)

Configure Squid for basic authentication by adding these lines to /etc/squid/squid.conf, under the lines with acl Safe_ports*:

auth_param basic program /usr/lib64/squid/ncsa_auth /etc/squid/passwd
auth_param basic children 5
auth_param basic realm Squid Basic Authentication
auth_param basic credentialsttl 2 hours
acl auth_users proxy_auth REQUIRED
http_access allow auth_users

Start Squid and verify it is working correctly:

service squid restart
curl --proxy-user user:password -x node1:3128 http://www.google.com/

Next create password files in HDFS:

su hdfs
cd ~
echo passwd > basicPassword.txt
hdfs dfs -put basicPassword.txt /apps/metron
echo password > proxyPassword.txt
hdfs dfs -put proxyPassword.txt /apps/metron
exit

To test with the Stellar REPL, follow these steps:

Start the Stellar REPL and verify the REST_GET function is available:

/usr/metron/0.6.1/bin/stellar --zookeeper node1:2181
[Stellar]>>> %functions REST
REST_GET

Test a simple get request:

[Stellar]>>> REST_GET('http://httpbin.org/get')
{args={}, headers={Accept=application/json, Accept-Encoding=gzip,deflate, Cache-Control=max-age=259200, Connection=close, Host=httpbin.org, User-Agent=Apache-HttpClient/4.3.2 (java 1.5)}, origin=127.0.0.1, 136.62.241.236, url=http://httpbin.org/get}

Test a get request with basic authentication:

[Stellar]>>> config := {'basic.auth.user':'user','basic.auth.password.path':'/apps/metron/basicPassword.txt'}
{basic.auth.user=user, basic.auth.password.path=/apps/metron/basicPassword.txt}
[Stellar]>>> REST_GET('http://httpbin.org/basic-auth/user/passwd', config)
{authenticated=true, user=user}

Try the same request without passing in the config.  You should get an authentication error:

[Stellar]>>> REST_GET('http://httpbin.org/basic-auth/user/passwd')
2018-10-28 00:32:20 ERROR RestFunctions:161 - Stellar REST request to http://httpbin.org/basic-auth/user/passwd expected status code to be one of [200] but failed with http status code 401:
java.io.IOException: Stellar REST request to http://httpbin.org/basic-auth/user/passwd expected status code to be one of [200] but failed with http status code 401:
	at org.apache.metron.stellar.dsl.functions.RestFunctions$RestGet.doGet(RestFunctions.java:209)
	at org.apache.metron.stellar.dsl.functions.RestFunctions$RestGet.apply(RestFunctions.java:157)
	at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:652)
	at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:250)
	at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
	at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:409)
	at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:260)
	at org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:357)
	at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

You should also be able to set the basic authentication settings through the global config:

[Stellar]>>> %define stellar.rest.settings := {'basic.auth.user':'user','basic.auth.password.path':'/apps/metron/basicPassword.txt'}
{basic.auth.user=user, basic.auth.password.path=/apps/metron/basicPassword.txt}
[Stellar]>>> REST_GET('http://httpbin.org/basic-auth/user/passwd')
{authenticated=true, user=user}

Now verify you can send a request through the proxy:

[Stellar]>>> config := {'proxy.host':'node1','proxy.port':3128,'proxy.basic.auth.user':'user','proxy.basic.auth.password.path':'/apps/metron/proxyPassword.txt'}
{proxy.basic.auth.password.path=/apps/metron/proxyPassword.txt, proxy.port=3128, proxy.host=node1, proxy.basic.auth.user=user}
[Stellar]>>> REST_GET('http://httpbin.org/get', config)
{args={}, headers={Accept=application/json, Accept-Encoding=gzip,deflate, Cache-Control=max-age=259200, Connection=close, Host=httpbin.org, User-Agent=Apache-HttpClient/4.3.2 (java 1.5)}, origin=127.0.0.1, 136.62.241.236, url=http://httpbin.org/get}

Leave out the proxy credentials, you should get a proxy error:

[Stellar]>>> config := {'proxy.host':'node1','proxy.port':3128}
{proxy.port=3128, proxy.host=node1}
[Stellar]>>> REST_GET('http://httpbin.org/get', config)
2018-10-28 00:43:48 ERROR RestFunctions:161 - Stellar REST request to http://httpbin.org/get expected status code to be one of [200] but failed with http status code 407: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>ERROR: Cache Access Denied</title>

Timeout is 1000 milliseconds by default.  Test the timeout by setting it to 1 or a value where a request won't finish in time.  You should get an error:

[Stellar]>>> REST_GET('http://httpbin.org/get', config)
2018-10-28 00:53:07 ERROR RestFunctions:161 - Total Stellar REST request time to http://httpbin.org/get exceeded the configured timeout of 1 ms.
java.io.IOException: Total Stellar REST request time to http://httpbin.org/get exceeded the configured timeout of 1 ms.
	at org.apache.metron.stellar.dsl.functions.RestFunctions$RestGet.doGet(RestFunctions.java:188)
	at org.apache.metron.stellar.dsl.functions.RestFunctions$RestGet.apply(RestFunctions.java:157)
	at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:652)
	at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:250)
	at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
	at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:409)
	at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:260)
	at org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:357)
	at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

You can also configure which status codes should be handled as errors.  A 404 is considered an error by default:

[Stellar]>>> REST_GET('http://httpbin.org/status/404', config)
2018-10-28 00:56:08 ERROR RestFunctions:161 - Stellar REST request to http://httpbin.org/status/404 expected status code to be one of [200] but failed with http status code 404:
java.io.IOException: Stellar REST request to http://httpbin.org/status/404 expected status code to be one of [200] but failed with http status code 404:
	at org.apache.metron.stellar.dsl.functions.RestFunctions$RestGet.doGet(RestFunctions.java:209)
	at org.apache.metron.stellar.dsl.functions.RestFunctions$RestGet.apply(RestFunctions.java:157)
	at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:652)
	at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:250)
	at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
	at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:409)
	at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:260)
	at org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:357)
	at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

This behavior can be changed by configuring a 404 to be an acceptable status code and returning an empty object instead of null:
{response.codes.allowed=[200, 404], empty.content.override={}}
[Stellar]>>> REST_GET('http://httpbin.org/status/404', config)
{}

The value returned on an error can also be changed from null:

config := {'error.value.override':'got an error'}
[Stellar]>>> result := REST_GET('http://httpbin.org/status/500', config)
2018-10-28 00:59:41 ERROR RestFunctions:161 - Stellar REST request to http://httpbin.org/status/500 expected status code to be one of [200] but failed with http status code 500:
java.io.IOException: Stellar REST request to http://httpbin.org/status/500 expected status code to be one of [200] but failed with http status code 500:
	at org.apache.metron.stellar.dsl.functions.RestFunctions$RestGet.doGet(RestFunctions.java:209)
	at org.apache.metron.stellar.dsl.functions.RestFunctions$RestGet.apply(RestFunctions.java:157)
	at org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:652)
	at org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:250)
	at org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
	at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:409)
	at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:260)
	at org.apache.metron.stellar.common.shell.specials.AssignmentCommand.execute(AssignmentCommand.java:66)
	at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:255)
	at org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:357)
	at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
got an error

To test with the parser and enrichment topologies follow these steps:

Make sure the topologies are running and data is flowing through.  It's easier to test if you restart the parser topology with only a single sensor running.
Add a Stellar field transformation to the parser that is still running:

"fieldTransformations": [
    {
      "input": [],
      "output": [
        "parser_rest_result"
      ],
      "transformation": "STELLAR",
      "config": {
        "parser_rest_result": "REST_GET('http://httpbin.org/get?type=parser')"
      }
    }
  ],

Listen on the enrichments Kafka topic.  The parser_rest_result field should now be present.
Add a Stellar enrichment to the sensor:

"fieldMap": {
      "geo": [
        "ip_dst_addr",
        "ip_src_addr"
      ],
      "host": [
        "host"
      ],
      "stellar": {
        "config": {
          "enrichment_rest_result": "REST_GET('http://httpbin.org/get?type=enrichment')"
        }
      }
    }

Listen on the indexing Kafka topic.  The enrichment_rest_result field should now be present.

Outstanding Issues

Caching was briefly discussed in the discuss thread for this feature.  Stellar provides a caching mechanism but we may need to be more selective about what is cached right now.  I believe this should be a follow on.
There was a comment in the Jira related to adding a circuit breaker.  Does that need to be done in this PR or can it be a follow on?  Should we also explore/discuss a retry strategy?
It was also suggested that we create an abstraction for higher latency enrichments such as this in the discuss thread.  I would prefer we create a few of these higher latency functions first so that we have a better understanding of how this abstraction would look.  Do we want to take that on here?

Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:

[x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
[x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

[x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?

[x] Have you included steps or a guide to how the change may be verified and tested manually?

[x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh

[x] Have you written or updated unit tests and or integration tests to verify your changes?

[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

[x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:
cd site-book
mvn site

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

@merrimanr - about the need for a close/shutdown hook - why is it needed in the first place? I don't believe we shutdown Zookeeper connections explicitly. They end when the topology spins down. Why should this be any different?

We do shut down Zookeeper connections here:  https://github.com/apache/metron/blob/master/metron-platform/metron-common/src/main/java/org/apache/metron/common/bolt/ConfiguredBolt.java#L129.  The Apache HttpComponents docs recommend closing clients:  http://hc.apache.org/httpcomponents-client-4.5.x/tutorial/html/fundamentals.html#d5e217.

I'm just not sure we need to shut them down. It's not like there's writing or state-related concerns that push the need to finish with a formal shutdown.  The only time we would ever shutdown a client connection would be when we're killing the topology, right?
I really think we need to get this client code out of the bolts. I looked a bit through the Stellar code and if we really think we need this, I think it can be accomplished through a change to the FunctionResolver classes. I'm going to noodle on this a bit.

I think we should close it, docs are pretty clear about that.  I think it's possible connections could be left open on the server side.
Moving the code out of the bolt is an organization/style issue.  I think there should be a functional justification like there is for closing a client.

Throwing this out there for consideration - https://github.com/apache/metron/pull/1251

The latest commit incorporates the changes from https://github.com/apache/metron/pull/1251 and moves the httpclient inside the function.  I was able to verify proper closing in the REPL and the EnrichmentIntegrationTest.  I was not able to verify in the Storm topologies because Storm does not guarantee these methods will be called in production.  I added code comments in the bolts where we are calling StellarFunctions.close().
I spun this up in full dev again and ran through the test plan.   Everything continued to work as expected.

Travis failure does not appear related here

@merrimanr Pending a successful Travis run, I'm +1 by inspection. Thanks!

+1, thanks!

authors: justinleet merrimanr mmiklavc

repo: apache/metron


text:
NimBLE sample build error with apache-mynewt-core

Hi,
I'm trying to build the BLE sample following the Set up a bare bones NimBLE application. I'm working on Ubuntu 20.04 with WSL2.
There is one step in the doc to add following configuration in syscfg.yml:
syscfg.vals:
    BLE_HCI_TRANSPORT: builtin

When I ran newt build ble_tgt, the following error appeared:
Building target targets/ble_tgt
Error: Override of defunct settings detected:
    BLE_HCI_TRANSPORT: see doc/transport.md

Setting history (newest -> oldest):
    BLE_HCI_TRANSPORT: [apps/ble_app:builtin, @apache-mynewt-nimble/nimble/transport:]

It seems that the BLE_HCI_TRANSPORT has been removed(the tutorial is outdated).
So I temporarily removed the config in syscfg.yml, but then I got another error while building:
Error: repos/apache-mynewt-core/encoding/tinycbor/src/cborpretty.c: In function 'value_to_pretty':
repos/apache-mynewt-core/encoding/tinycbor/src/cborpretty.c:334:33: error: expected ')' before 'PRIu64'
  334 |             if (fprintf(out, "%" PRIu64, val) < 0)
      |                                 ^~~~~~~
      |                                 )
repos/apache-mynewt-core/encoding/tinycbor/src/cborpretty.c:334:31: error: spurious trailing '%' in format [-Werror=format=]
  334 |             if (fprintf(out, "%" PRIu64, val) < 0)
      |                               ^
repos/apache-mynewt-core/encoding/tinycbor/src/cborpretty.c:343:38: error: expected ')' before 'PRIu64'
  343 |                 if (fprintf(out, "-%" PRIu64, val) < 0)
      |                                      ^~~~~~~
      |                                      )
repos/apache-mynewt-core/encoding/tinycbor/src/cborpretty.c:343:36: error: spurious trailing '%' in format [-Werror=format=]
  343 |                 if (fprintf(out, "-%" PRIu64, val) < 0)
      |                                    ^
repos/apache-mynewt-core/encoding/tinycbor/src/cborpretty.c:390:29: error: expected ')' before 'PRIu64'
  390 |         if (fprintf(out, "%" PRIu64 "(", tag) < 0)
      |                             ^~~~~~~
      |                             )
repos/apache-mynewt-core/encoding/tinycbor/src/cborpretty.c:390:27: error: spurious trailing '%' in format [-Werror=format=]
  390 |         if (fprintf(out, "%" PRIu64 "(", tag) < 0)
      |                           ^
cc1: all warnings being treated as errors

Solutions  I can find all suggest that this is a C99 standard problem.
I have added below code to repos/apache-mynewt-core/encoding/tinycbor/src/cborpretty.c and repos/apache-mynewt-core/encoding/tinycbor/src/cbortojson.c:
#define __STDC_FORMAT_MACROS 1
...
#include <inttypes.h>

But it doesn't solve the problem. My compiler version is arm-none-eabi-g++ (15:9-2019-q4-0ubuntu1) 9.2.1 20191025 (release) [ARM/arm-9-branch revision 277599]. What is the suggesting version of gcc to build MynewtOS?
I find no __STDC_FORMAT_MACROS in the entire Mynewt repo. Does it mean thiserror has never occurred before?

This is an Ubuntu package problem. Try to install the ARM GNU toolchains from the official site.

repo: apache/mynewt-nimble