text:
NIFI-5479 Upgraded Jetty. Moved where we unpack bundled deps to so we…

… can avoid a new jetty bug with META-INF loading logic.   WIP for testing/eval.  Not ready for merge
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit?

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

2018-08-02 16:20:07,347 WARN [NiFi Web Server-21] o.e.jetty.annotations.AnnotationParser javax.inject.Inject scanned from multiple locations: jar:file:///Users/jwitt/Development/joewitt-nifi.git/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-SNAPSHOT/work/jetty/nifi-update-attribute-ui-1.8.0-SNAPSHOT.war/webapp/WEB-INF/lib/javax.inject-1.jar!/javax/inject/Inject.class, jar:file:///Users/jwitt/Development/joewitt-nifi.git/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-SNAPSHOT/work/jetty/nifi-update-attribute-ui-1.8.0-SNAPSHOT.war/webapp/WEB-INF/lib/javax.inject-2.5.0-b42.jar!/javax/inject/Inject.class
2018-08-02 16:20:07,348 WARN [NiFi Web Server-21] o.e.jetty.annotations.AnnotationParser javax.inject.Named scanned from multiple locations: jar:file:///Users/jwitt/Development/joewitt-nifi.git/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-SNAPSHOT/work/jetty/nifi-update-attribute-ui-1.8.0-SNAPSHOT.war/webapp/WEB-INF/lib/javax.inject-1.jar!/javax/inject/Named.class, jar:file:///Users/jwitt/Development/joewitt-nifi.git/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-SNAPSHOT/work/jetty/nifi-update-attribute-ui-1.8.0-SNAPSHOT.war/webapp/WEB-INF/lib/javax.inject-2.5.0-b42.jar!/javax/inject/Named.class
2018-08-02 16:20:07,348 WARN [NiFi Web Server-21] o.e.jetty.annotations.AnnotationParser javax.inject.Provider scanned from multiple locations: jar:file:///Users/jwitt/Development/joewitt-nifi.git/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-SNAPSHOT/work/jetty/nifi-update-attribute-ui-1.8.0-SNAPSHOT.war/webapp/WEB-INF/lib/javax.inject-1.jar!/javax/inject/Provider.class, jar:file:///Users/jwitt/Development/joewitt-nifi.git/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-S
2018-08-02 16:20:16,335 WARN [main] o.e.j.webapp.StandardDescriptorProcessor Duplicate mapping from / to default
2018-08-02 16:20:16,429 WARN [main] o.e.jetty.annotations.AnnotationParser Unknown asm implementation version, assuming version 393216
Todo:

identify source of and cleanup warnings that now show up on application startup
test/what if one just copied the contents of a new lib dir on top of their old lib dir as an upgrade process...will our work dirs get cleaned and restart properly
add docs in the unpack method to explain why we move META-INF/bundled-dependencies to NAR-INF/bundled-dependencies

this is just a technique to work with older/current nar creation approach.
this is done because jetty's code assumes that META-INF is only in a directory path once or else it fails to find some tlds.
but we want to keep META-INF for things like META-INF/MANIFEST.mf and maven bits.
we might want to just move META-INF/bundled-dependencies to bundled-dependencies.  The 'NAR-INF' part is not value add since the nar metadata is in META-INF/MANIFEST.mf and not easily moved due to jar/manifest loading code

file a JIRA to change where we write them in the nar plugin to NAR-INF/bundled-dependencies directly
Test secure/non-secure clusters/etc..

big thanks to @mcgilman  for finding the needed dep change in nifi-web-ui and identifying why we needed a workaround for how we extract working dir nar deps due to recent jetty change

Would it help if you would load NAR's without unpacking them to disk?

that would not enable us to work around this issue and does not bring the benefits that led to unpacking in the first place

@joewitt Thanks for the PR! When starting up in secure mode using a configuration that works with current master branch, I received some stack traces regarding the initialization of the SSLContext. There appears to be a runtime difference introduced here that affects the loading of providers.

1305 Caused by: java.security.NoSuchAlgorithmException: no such algorithm: JKS for provider BC
1306     at sun.security.jca.GetInstance.getService(GetInstance.java:87)
1307     at sun.security.jca.GetInstance.getInstance(GetInstance.java:206)
1308     at java.security.Security.getImpl(Security.java:698)
1309     at java.security.KeyStore.getInstance(KeyStore.java:896)
1310     ... 21 common frames omitted

authors: joewitt mcgilman ottobackwards


text:
NIFI-6323 Changed URLs in XML files to use https:// where possible

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
This PR changes existing URLs (project description, mailing lists, dependency repositories, and schema references) to use the https:// protocol when possible. It also standardizes the location of the Maven 4.0.0 XML schema descriptor.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

Will review...

did full clean build w/contrib check.  all looks good and nifi itself still seems good.  +1 (assuming gilman also is)

Also +1. Successful build with cleaned mvn repo. Verified standalone and clustered functionality. Will merge.

source: gharchive/pull-request

authors: alopresto joewitt mcgilman

repo: apache/nifi

license_type: permissive

license_source: github-api


text:
NIFI-7240: Fixed out-of-order Table Map events in CaptureChangeMySQL

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
When using triggers to update tables in MySQL, the Table Map events may be out of order with the corresponding Write Rows events for those tables. This PR keeps a temporary map of table IDs to cache keys in order to retrieve the correct table information during Write Rows processing.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[x] Have you written or updated unit tests to verify your changes?
[ ] Have you verified that the full build is successful on both JDK 8 and JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

I've successfully compiled it on JDK 8 and I can confirm that it does fix the issue, it would be great if this change could make it into the next stable version.

Hey @mattyb149 - can you rebase against the main branch? Happy to get this in. I recently reviewed another PR related to this processor.

@mattyb149 Are u still following this thread?

There are two more issues opened about this now:

https://issues.apache.org/jira/browse/NIFI-6914
https://issues.apache.org/jira/browse/NIFI-7252

This issue is drastically impacting my organization's ability to utilize NiFi for synchronizing changes from an older legacy system into our new one. Any way this PR can be re-opened or reviewed? @mattyb149 @pvillard31


text:
NIFI-7486 Make InvokeHttp authentication properties able to read from variables.

Description of PR

InvokeHTTP Basic HTTP credentials support variable registry

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[X] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[X] Has your PR been rebased against the latest commit within the target branch (typically master)?

[X] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[X] Have you written or updated unit tests to verify your changes?
[ ] Have you verified that the full build is successful on JDK 8?
[ ] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

Hey @adarmiento - does it really makes sense now that we have the parameters concept in NiFi? Besides variables should not be used for sensitive properties so I'd definitely not recommend using a variable on the password property.

Hey @adarmiento - does it really makes sense now that we have the parameters concept in NiFi? Besides variables should not be used for sensitive properties so I'd definitely not recommend using a variable on the password property.

Hello @pvillard31, I did not think about it, probably because I was still using old 1.9.2 until now.
Probably could be a security bad choice then (I noted that also the proxy credentials allow variables, maybe we should then update those two properties for consistency)
Thanks for the tips anyway, I'll keep this in mind for the future

I think it's hard to remove variable support after the fact because of backward compatibility concerns (even though I'd be a +1 for removing variable support on sensitive properties). I think that (to be discussed though) when the community will start working around Apache NiFi 2.x, we would possibly remove the variables to only support parameters.
If you think that a change would still make sense, let me know and we can have a look.

In my case, I´m running NIFI under Kubernets and the usage of variables in sensitive properties is needed to use environment variables. These type of properties are injected in the container during the execution and then can be used in NIFI.
If it´s not supported, I need to write in each processor the user and password.
Maybe, in version 2.x, NIFI can have a different expression language scopes. One for environment variable and other for variables.

@axdmoraes - thanks for the feedback. How is the flow published in NiFi? As part of the Docker image? via a volume? or from a NiFi Registry instance?

Using Nifi registry instance. We use the same registry for different environments.

In that case, would it be an option to use the CLI or REST API to set the parameters values after the flow has been deployed in NiFi from the NiFi Registry. On my side, with my k8s deployments, I'm doing something looking like the below
# add the Registry client in NiFi (to adapt for your secured NiFi instances)
curl 'http://nifi:8080/nifi-api/controller/registry-clients' -H 'Content-Type: application/json' --data-binary '{"revision":{"version":0},"component":{"name":"NiFi Registry","uri":"http://nifi-registry:18080/nifi-registry"}}'

# Deploy the flow in NiFi (add the logic to retrieve the bucket/flow/version)
/opt/nifi/nifi-toolkit-1.11.4/bin/cli.sh nifi pg-import -u http://nifi:8080 --bucketIdentifier $bucketID --flowIdentifier $flow --flowVersion $version

# Get the parameter context ID
paramContextID=`/opt/nifi/nifi-toolkit-1.11.4/bin/cli.sh nifi list-param-contexts -u http://nifi:8080 -ot json | grep -v cli.sh | jq -r '.parameterContexts[].id'`

# Set the parameters values (you can do something dynamic based on your needs)
/opt/nifi/nifi-toolkit-1.11.4/bin/cli.sh nifi set-param -u http://nifi:8080 --paramContextId $paramContextID --paramName MY_PARAMETER --paramValue MY_VALUE

# Start the controller services (add your logic to retrieve the PG ID)
/opt/nifi/nifi-toolkit-1.11.4/bin/cli.sh nifi pg-enable-services -u http://nifi:8080 --processGroupId $pgid

# Start the process group (add your logic to retrieve the PG ID)
/opt/nifi/nifi-toolkit-1.11.4/bin/cli.sh nifi pg-start -u http://nifi:8080 --processGroupId $pgid

Thanks for the suggestion. I will try. We were using version 1.9.2 and I didn't know about parameters context.

authors: adarmiento axdmoraes pvillard31


text:
NIFI-10549: Remove group name wildcard from assembly for nifi-ranger-resources

Summary
NIFI-10549
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[x] Apache NiFi Jira issue created

Pull Request Tracking

[x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[x] Pull Request based on current revision of the main branch
[x] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[ ] Build completed using mvn clean install -P contrib-check

[ ] JDK 8
[x] JDK 11
[ ] JDK 17

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

LGTM

Looks good, will merge

authors: bbende mattyb149 tamas-horvath


text:
nshlib: Add startup process buried points

Summary

nshlib: Add startup process buried points
trace: fix trace dump crash

Impact
Testing
sim

[Experimental Bot, please feedback here]
No, this PR does not fully meet the NuttX requirements.
Missing Information:

Summary:

Why are these changes necessary?  Bug fixes? Enhancements?
How do the changes work? Provide specifics about the buried points added and the cause of the trace dump crash and how it was fixed.
Issue References: Link any relevant NuttX or NuttX Apps issues.

Impact:

Address all impact categories (user, build, hardware, documentation, security, compatibility). Even if there's no impact, state "NO" explicitly for each.

Testing:

Insufficient Detail: "sim" isn't descriptive enough. Specify the simulator (e.g., qemu-rv32) and the configuration used.
Missing Logs: Provide actual testing logs from before and after the changes.

Recommendations:

Expand the Summary: Explain the "why" and "how" for each change.
Complete the Impact Assessment:  Go through each category and provide a clear "YES" or "NO" along with a brief explanation if necessary.
Provide Detailed Testing Information: Specify the exact simulator, configurations, and include the relevant testing logs.

My only remark is this is very useful feature, probably will be developed further in future, but as for quite a big change better description in the PR and some additional comments in the commit message would help understand the change better by others that want to use it too :-)

Is existing API impacted anyhow? Will old code work the same way or needs an update?
Is documentation update required  / necessary? Maybe it would be good to provide documentation on how to use new functionalities? Newcomers tend to start at documentation so share your inventions there too :-)
If the buffer is too small, sure we can increase the buffer, but also overflow checks are necessary?

authors: Gary-Hobson cederom nuttxpr

repo: apache/nuttx-apps


text:
LPC17_40 CAN driver SocketCAN enforce TX fifo behaviour

Summary
The SocketCAN driver expects a FIFO behaviour yet LPC17_40 didn't enforce this.
This changes enables prioritization on transmit function to enforce this.
Impact
Fix transmit behaviour
Testing
Tested on a LPC1768

Could you please squash commits?

Could you please squash commits?

And now we have a documentation to explain how to do it:
https://nuttx.apache.org/docs/latest/contributing/making-changes.html#how-to-include-the-suggestions-on-your-pull-request
Of course, @PetervdPerk is not a New Kind on the Block, the idea was to help new contributors

Let' ignore the macOS ci broken.

authors: PetervdPerk acassis pkarashchenko xiaoxiang781216

repo: apache/nuttx


text:
ORC-1021: Add -fno-omit-frame-pointer in DEBUG and RELWITHDEBINFO builds

What changes were proposed in this pull request?

This PR adds -fno-omit-frame-pointer gcc option in DEBUG and RELWITHDEBINFO builds, which helps to generate stacktrace in debugging and profiling. Refs:
https://www.brendangregg.com/perf.html#StackTraces
https://issues.apache.org/jira/browse/IMPALA-4132
Why are the changes needed?

Described as above.
How was this patch tested?

Built in ubuntu16.04 with gcc 8.4.0.

+1 LGTM

I backported this to branch-1.7.

authors: dongjoon-hyun guiyanakuang stiga-huang

repo: apache/orc

license_source: bigquery


text:
HDDS-4549. Fix typos in documents

What changes were proposed in this pull request?
Fix typos in documents
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4549
How was this patch tested?
No need

@cku328 done, thanks

Thanks @lamber-ken for working on this.
I will merge it later.

repo: apache/ozone


text:
HDDS-5084. Include HISTORY.md/SECURITY.md/CONTRIBUTING.md in the release artifacts.

JIRA: https://issues.apache.org/jira/browse/HDDS-5084
What changes were proposed in this pull request?
During the ozone-1.1.0 vote I realized that HISTORY.md/SECURITY.md/CONTRIBUTING.md files are missing from the bin and src artifacts. I think they include very useful information and would be better to include them in the release artifacts.
How was this patch tested?
mvn clean install -Dmaven.javadoc.skip=true -DskipTests -Psign,dist,src -Dtar -Dgpg.keyname=$CODESIGNINGKEY
  cd hadoop-ozone/dist/target/
tar tzf hadoop-ozone-1.1.0-SNAPSHOT.tar.gz
tar tzf hadoop-ozone-1.1.0-src-SNAPSHOT.tar.gz

We should update the History.md, it is quite old right now.

@mukul1987 This is a good suggestion, but I disagree with the other statement:

Including it in the current form in the release doesn't makes sense.

It is not going to be part of the 1.1.0 release, so there is plenty of time to update it until the next one.
It's not only about HISTORY.md, but other two docs as well.
Writing prose for the history doc is quite distinct from updating a script to copy some files.  It may very well be updated by other people, not necessarily @elek.

So I think this change is fine in its scope.

Fair point @adoroszlai. Can we please create a followup jira and mark it as a blocker for 1.2.0 release? I feel if we will update the history by the next release, then we should be good.

Thanks the suggestion @mukul1987, very good point.
It seems to be a small update, so I created the patch itself (please see #2149). And agree: as we have PRs for both problems in our radar we can merge the two PRs in any order.

Thanks Marton. +1 for this patch as well.
I have already added +1 to the other patch.
Thanks for updating the file.

Thanks the review @mukul1987 @ayushtkn and @adoroszlai
I am merging it after the green build.

authors: adoroszlai elek mukul1987


text:
Bump Thrift to 0.16.0

Thrift 0.16.0 has been released https://github.com/apache/thrift/releases/tag/v0.16.0
Reporter: Vinoo Ganesh / @vinooganesh
Assignee: Vinoo Ganesh / @vinooganesh
Related issues:

Release 1.12.3 (is depended upon by)

Note: This issue was originally created as PARQUET-2128. Please see the migration documentation for further details.

Vinoo Ganesh / @vinooganesh:
Fixed in https://github.com/apache/parquet-mr/pull/948

Steve Loughran / @steveloughran:
homebrew doesn't have anything < 0..18.0, which is java11+ only, so not something parquet can switch to.
which means that we have to stop using homebrew here and take control of our build dependencies ourselves. I've already done that with maven and openjdk as brew is too enthusiastic about breaking my workflow.
*none of us can rely on homebrew or use "homebrew doesn't have this" as a reason for reverting a change.
All old thrift releases can be found at https://archive.apache.org/dist/thrift/

source: gharchive/issue

repo: apache/parquet-java


text:
optimize queries where lhs and rhs of predicate are equal

This is a minor performance bugfix.

this fixes NullPointerExceptions in existing optimizers when performing WHERE 1=1 queries. These would fail because the filter expression had no function call
I noticed that WHERE 1=1 was no simplified, but WHERE col1>0 AND 1=1 was actually being simplified in the NumericalFilterOptimizer. So I put that part in a separate class to be used more generally for future cases like this

it does a little more work than expected once it sees and AND/OR/NOT expression
something else is converting 1=1 to literal TRUE, but I'm not sure where that is

This adds a IdenticalPredicateFilterOptimizer class that converts WHERE 1=1 or WHERE "colA"!="colA" to TRUE/FALSE respectively

I've added a bunch more test cases, and I've tested manually in the Quickstart app. This is my first contribution to the query parsing part of the code base, so I don't have a great sense what test coverage looks like. But I imagine between unit and integration tests, this should catch any glaring breaks?

Codecov Report

Merging #10444 (c7c578f) into master (d9c4315) will decrease coverage by 50.31%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master   #10444       +/-   ##
=============================================
- Coverage     64.21%   13.90%   -50.31%
+ Complexity     6089      237     -5852
=============================================
  Files          2007     2009        +2
  Lines        109281   109337       +56
  Branches      16692    16708       +16
=============================================
- Hits          70177    15208    -54969
- Misses        33993    92897    +58904
+ Partials       5111     1232     -3879

Flag
Coverage Δ

unittests1
?

unittests2
13.90% <0.00%> (-0.02%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

.../pinot/controller/recommender/io/InputManager.java
93.22% <ø> (ø)

...che/pinot/core/query/optimizer/QueryOptimizer.java
0.00% <ø> (-100.00%)
:arrow_down:

...imizer/filter/BaseAndOrBooleanFilterOptimizer.java
0.00% <0.00%> (ø)

.../optimizer/filter/FlattenAndOrFilterOptimizer.java
0.00% <0.00%> (-77.78%)
:arrow_down:

...izer/filter/IdenticalPredicateFilterOptimizer.java
0.00% <0.00%> (ø)

...ery/optimizer/filter/MergeEqInFilterOptimizer.java
0.00% <0.00%> (-92.60%)
:arrow_down:

...ery/optimizer/filter/NumericalFilterOptimizer.java
0.00% <0.00%> (-80.90%)
:arrow_down:

... and 1351 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Looks good in general, great job!

thank you! i see all checks passed. let me know if you have further comments, though

repo: apache/pinot


text:
separate tags with commas as indicated in action doc

@erichgess found that https://github.com/apache/pinot/pull/10528 broke the codecov coverate.
In fact coverage was uploaded, but tags were incorrectly configured and therefore they are not uploaded with the expected metadata.
As indicated in https://github.com/codecov/codecov-action, different tags should be separated by commas.

Codecov Report

Merging #11303 (7b23f93) into master (6fa4268) will increase coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master   #11303     +/-   ##
=========================================
  Coverage    0.11%    0.11%
=========================================
  Files        2231     2157     -74
  Lines      120139   116982   -3157
  Branches    18218    17772    -446
=========================================
  Hits          137      137
+ Misses     119982   116825   -3157
  Partials       20       20

Flag
Coverage Δ

integration1temurin11
?

integration1temurin17
?

integration1temurin20
?

integration2temurin11
?

integration2temurin17
?

integration2temurin20
?

java-20
0.11% <ø> (?)

temurin
0.11% <ø> (?)

unittests1temurin11
?

unittests1temurin17
?

unittests1temurin20
?

unittests2
0.11% <ø> (?)

unittests2temurin11
?

unittests2temurin17
?

unittests2temurin20
?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 76 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter gortiz


text:
Added UTs for null handling in CaseTransform function.

Added unit test cases for null handling in  CaseTransformFunction
Also, added test case for isNullLiteralTransformation function asked in the PR

cc: @shenyu0127 @Jackie-Jiang

Codecov Report

Merging #11721 (6a96f1a) into master (ae16812) will decrease coverage by 48.69%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master   #11721       +/-   ##
=============================================
- Coverage     63.11%   14.42%   -48.69%
+ Complexity     1117      201      -916
=============================================
  Files          2342     2342
  Lines        125802   125800        -2
  Branches      19336    19336
=============================================
- Hits          79395    18150    -61245
- Misses        40745   106116    +65371
+ Partials       5662     1534     -4128

Flag
Coverage Δ

integration
?

integration1
?

integration2
?

java-11
14.42% <ø> (-48.64%)
:arrow_down:

java-17
?

java-20
?

temurin
14.42% <ø> (-48.69%)
:arrow_down:

unittests
14.42% <ø> (-48.68%)
:arrow_down:

unittests1
?

unittests2
14.42% <ø> (-0.06%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files
Coverage Δ

...ator/transform/function/CaseTransformFunction.java
0.00% <ø> (-57.98%)
:arrow_down:

... and 1521 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: abhioncbr codecov-commenter


text:
Enhance optimizeDictionary to optionally optimize var-width type cols

Changes

Add noDictionaryCardinalityRatioThreshold config.

If populated, and optimizeDictionary is true, then Pinot will override dictionary encoding with raw encoding based on the condition cardinality / numDocs > noDictionaryCardinalityRatioThreshold.
If the new config is omitted, optimizeDictionary behavior is unchanged

Motivation
When storing log data, often columns will contain many repeated values. It's useful to take advantage of Pinot's dictionary encoding which usually provides better storage/query performance for these columns. Dictionary encoding high cardinality columns is cost/storage prohibitive, so we'd like to avoid applying dictionary encoding unless it is safe. Since column cardinality/values can change rapidly we'd like to make these decisions within Pinot itself.
In our experience, cardinality is a good indicator of whether to dictionary or raw encode a col. With a 0.10 threshold (10%), we see roughly 40-60% improvement in storage compared to raw encoding everything.

Codecov Report
Attention: Patch coverage is 0% with 26 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (59551e4) to head (8c368f9).
Report is 1029 commits behind head on master.

Files with missing lines
Patch %
Lines

.../segment/index/dictionary/DictionaryIndexType.java
0.00%
12 Missing :warning:

...ocal/segment/index/loader/ForwardIndexHandler.java
0.00%
4 Missing :warning:

...ot/segment/spi/creator/SegmentGeneratorConfig.java
0.00%
4 Missing :warning:

...ment/creator/impl/SegmentColumnarIndexCreator.java
0.00%
3 Missing :warning:

.../apache/pinot/spi/config/table/IndexingConfig.java
0.00%
3 Missing :warning:

:exclamation:  There is a different number of reports uploaded between BASE (59551e4) and HEAD (8c368f9). Click for more details.
HEAD has 48 uploads less than BASE

Flag
BASE (59551e4)
HEAD (8c368f9)

integration
7
2

integration2
3
2

temurin
12
2

java-21
7
2

skip-bytebuffers-true
3
1

skip-bytebuffers-false
7
1

unittests
5
0

unittests1
2
0

java-11
5
0

unittests2
3
0

integration1
2
0

custom-integration1
2
0

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #13994       +/-   ##
=============================================
- Coverage     61.75%    0.00%   -61.76%
=============================================
  Files          2436     2514       +78
  Lines        133233   139046     +5813
  Branches      20636    21371      +735
=============================================
- Hits          82274        0    -82274
- Misses        44911   139046    +94135
+ Partials       6048        0     -6048

Flag
Coverage Δ

custom-integration1
?

integration
0.00% <0.00%> (-0.01%)
:arrow_down:

integration1
?

integration2
0.00% <0.00%> (ø)

java-11
?

java-21
0.00% <0.00%> (-61.63%)
:arrow_down:

skip-bytebuffers-false
0.00% <0.00%> (-61.75%)
:arrow_down:

skip-bytebuffers-true
0.00% <0.00%> (-27.73%)
:arrow_down:

temurin
0.00% <0.00%> (-61.76%)
:arrow_down:

unittests
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Certain settings are impossible, e.g. only apply cardinality based optimization and skip optimization for fixed-length type. I'm not too worried about this limitation, but it would be good if we make it possible

If I understand the concern correctly, I think users can set noDictionaryCardinalityRatioThreshold = 0 to effectively skip optimization for fixed-length type?
I'd prefer to merge as is, since providing a way to use the cardinality ratio threshold instead of size ratio threshold means making the old size ratio threshold config optional, which is backwards compatible and could be done in the future if the need is found

Basically we specify:

Size based only for fixed length type
Cardinality based only for var-length type

Let's document this behavior so that user doesn't expect wrong type being applied


text:
simplify segment pruning

Segment pruning feels over-engineered:

DataSchemaSegmentPruner and ValidSegmentPruner are just one liners which can be applied when necessary
ColumnValueSegmentPruner and SelectionQuerySegmentPruner are mutually exclusive so never both need to run, and these cases can be identified easily by examining the QueryContext
No new segment pruners have been in a very long time

This leads to inefficiencies like

None of the pruners inline
Lots of lists are created unnecessarily
We end up tracing one liner checks

This PR removes the two trivial pruners and applies them inline within the two remaining pruners. It adds a new method to identify based on the query context whether the pruner should run at all.

Codecov Report

Merging #8790 (853cdc9) into master (c4549e2) will decrease coverage by 48.69%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master    #8790       +/-   ##
=============================================
- Coverage     62.86%   14.17%   -48.70%
+ Complexity     4601      168     -4433
=============================================
  Files          1690     1688        -2
  Lines         89212    89211        -1
  Branches      13411    13415        +4
=============================================
- Hits          56082    12642    -43440
- Misses        29079    75623    +46544
+ Partials       4051      946     -3105

Flag
Coverage Δ

unittests1
?

unittests2
14.17% <0.00%> (-0.03%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...ot/core/query/pruner/ColumnValueSegmentPruner.java
0.00% <0.00%> (-72.17%)
:arrow_down:

.../apache/pinot/core/query/pruner/SegmentPruner.java
0.00% <0.00%> (-88.89%)
:arrow_down:

...pinot/core/query/pruner/SegmentPrunerProvider.java
0.00% <0.00%> (-66.67%)
:arrow_down:

.../pinot/core/query/pruner/SegmentPrunerService.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...core/query/pruner/SelectionQuerySegmentPruner.java
0.00% <0.00%> (-86.37%)
:arrow_down:

...src/main/java/org/apache/pinot/sql/FilterKind.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ain/java/org/apache/pinot/core/data/table/Key.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...in/java/org/apache/pinot/spi/utils/StringUtil.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/org/apache/pinot/spi/utils/BooleanUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/org/apache/pinot/core/data/table/Record.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 1137 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4549e2...853cdc9. Read the comment docs.

LGTM

authors: codecov-commenter gortiz richardstartin


text:
[PIO-187] Livedoc with Docker Installation Update

After #462, we have Docker support in the repo.
This PR will modify the corresponding updates in Livedoc.
@marevol, please help to check if I make any mistake.

Thanks for writing this up! One small request: we have to declare that any Docker container image published to Docker Hub be not official ASF releases. Those can only be referred to as convenience binaries. We can say that the Docker build files in our Git repo as official though.
For more information please refer to http://www.apache.org/legal/release-policy.html.

@dszeto, I will modify it for sure. thanks!

LGTM. @marevol do you want to take a second look?

Going to merge this now. Thanks @Wei-1 !
@marevol if you see issues please open a separate ticket.

repo: apache/predictionio


text:
[Broker] Fix producerFuture not completed in ServerCnx#handleProducer

Motivation
producerFuture should be completed and removed from producers when exception occurs.
Modifications
Add producerFuture.completeExceptionally
Verifying this change

[ ] Make sure that the change passes the CI checks.

This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?

[x] no-need-doc
bug fix.

@codelipenghui - it'd be great to include this in 2.10.0 rc 2, if possible.

@michaeljmarshall Yes, I have cherry-picked to branch-2.10

authors: Jason918 codelipenghui michaeljmarshall

repo: apache/pulsar


text:
[improve][build] Move docker-push profile to submodule

Motivation
The profile refactoring breaks the pulsar release in the #23091.
Modifications

Move docker-push profile to docker/pulsar and docker/pulsar-all modules

Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 73.44%. Comparing base (bbc6224) to head (6022c9e).
Report is 479 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23093      +/-   ##
============================================
- Coverage     73.57%   73.44%   -0.14%
- Complexity    32624    33524     +900
============================================
  Files          1877     1919      +42
  Lines        139502   144087    +4585
  Branches      15299    15745     +446
============================================
+ Hits         102638   105824    +3186
- Misses        28908    30145    +1237
- Partials       7956     8118     +162

Flag
Coverage Δ

inttests
27.58% <ø> (+2.99%)
:arrow_up:

systests
24.76% <ø> (+0.43%)
:arrow_up:

unittests
72.51% <ø> (-0.34%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.
see 516 files with indirect coverage changes

authors: codecov-commenter nodece


text:
[Conf] Add controller start config file

Add the configuration file for startup controller to the configuration file directory. when controller Independent Deployment.

Well, thanks for you attention, could you please submit a pr to add it?

Well, thanks for you attention, could you please submit a pr to add it?

yes I will submit a PR for this

repo: apache/rocketmq


text:
[ISSUE #1770]Add a query message trace command in mqadmin.

What is the purpose of the change
Add a query message trace command in mqadmin.
Brief changelog
Add a query message trace command in mqadmin.

Coverage decreased (-0.09%) to 50.829% when pulling 4fca91fa6f3c91ba0b6c8f13f4fdfcfffb2401d7 on zhangjidi2016:add_query_trace_command into 3974677f04815609951c17059d85d3795eb51247 on apache:develop.

The command result in console，@zongtanghu @duhenglucky ，please help to review it，thanks！


text:
Fix 2046

fix selectOneMessageQueue must select lastFailBroker
As you say, if lastBrokerName is null, it will select one MessageQueue, assuming MessageQueue-a. If it occurs some exception while producing to MessageQueue-a, it will always select MessageQueue-a since then.

Coverage increased (+0.2%) to 51.023% when pulling aea101b6e26f05d3677e5842702d42312537d921 on HaoTianZhao:fix-2046 into 8ef01a6c635f6972847c40d5540b1945180d7cbd on apache:master.

authors: HaoTianZhao coveralls


text:
[ISSUE #6576] Fix pop lmq message

Make sure set the target branch to develop
What is the purpose of the change

fix #6576
Brief changelog
If use pop to consume a LMQ message, recode message, set LMQ's POP_CK, topic, queueId and queueOffset
Verifying this change

Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Codecov Report

Merging #6577 (b4d9cdf) into develop (f44a1c3) will increase coverage by 0.03%.
The diff coverage is 77.77%.

@@              Coverage Diff              @@
##             develop    #6577      +/-   ##
=============================================
+ Coverage      43.08%   43.11%   +0.03%
- Complexity      8994     8998       +4
=============================================
  Files           1107     1107
  Lines          78257    78278      +21
  Branches       10201    10203       +2
=============================================
+ Hits           33716    33750      +34
+ Misses         40318    40305      -13
  Partials        4223     4223

Impacted Files
Coverage Δ

...rocketmq/broker/processor/PopMessageProcessor.java
39.67% <77.77%> (+2.06%)
:arrow_up:

... and 16 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: HScarb codecov-commenter


text:
[Feature][Transformer] Supporting fake data generation in transformer for sensitive data masking options

Search before asking

[X] I had searched in the feature and found no similar feature requirement.

Description
Hi All Following to the short discussion I create this issue.
https://github.com/apache/seatunnel/discussions/7746
So there is an idea and goal to source and sink completely full postgres (or etc) to another postgres (source) with data masking or generation fake data for sensitive attributes. Good to know that there are a lot of fakesource available with random generators but at this moment I don't know is it working in transformer or not. Also some good news that there is dynamic compilation available for some completely custom cases.
What do you think?
Usage Scenario
Some maybe will try to use Transformer in case of masking and fake generation.
The real case is to make data synchronization from prod to test environment with some predefined option by user request

[ ] Support fake data generation in transformer for sensitive data masking options in full DB sync case or partial

Related issues
Supporting fake data generation in transformer
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

How about support join with dimension table (fake source is one type of dimension table)?
I think we can extend this requirement to any source.
eg:
join with jdbc
transform {

    JoinWithSource {
        join_on = "source.id = type_bin.item_id"
        source = [
            Jdbc {
                url = "jdbc:mysql://localhost/test?serverTimezone=GMT%2b8"
                driver = "com.mysql.cj.jdbc.Driver"
                connection_check_timeout_sec = 100
                user = "root"
                password = "123456"
                query = "select * from type_bin"
            }
        ]
    }
}

or join with fake source
transform {
    JoinWithSource {
        join_on = "source.id = fake.c_int"
        source = [
            FakeSource {
                row.num = 5
                schema {
                    fields {
                    c_string = string
                    c_tinyint = tinyint
                    c_smallint = smallint
                    c_int = int
                    c_bigint = bigint
                    c_float = float
                    c_double = double
                    }
                }
            }
        ]
    }
}

Then we can use SQL transform to filter data you want.

Or join with sql transform
env {
  parallelism = 10
  job.mode = "BATCH"
}
source {
    Jdbc {
        url = "jdbc:mysql://localhost/test?serverTimezone=GMT%2b8"
        driver = "com.mysql.cj.jdbc.Driver"
        connection_check_timeout_sec = 100
        user = "root"
        password = "123456"
        table_path = "testdb.table1"
        query = "select * from testdb.table1"
        split.size = 10000
    }
    FakeSource {
                row.num = 5
                schema {
                    fields {
                    c_string = string
                    c_tinyint = tinyint
                    c_smallint = smallint
                    c_int = int
                    c_bigint = bigint
                    c_float = float
                    c_double = double
                    }
                }
            }
}

transform {
    sql {
        query = "select * from table1 join table2 on table1.id = table2.id"
    }
}

sink {
  Console {}
}

authors: Hisoka-X YuriyGavrilov

repo: apache/seatunnel


text:
[BUG] Shenyu-Admin > BasicConfig > Plugin list page 2 cannot display

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
I do not know when to start, the second page of  BasicConfig > Plugin cannot be displayed（empty white page），but the first and third page is OK（Page size is 12）
F12 of Chrome show this:

Expected Behavior
No response
Steps To Reproduce
No response
Environment
ShenYu version(s):2.5.0

Debug logs
No response
Anything else?
No response

And when I open Request plugin，all page of Shenyu-Admin can not display，so I have to close the Request plugin by modifing Database table (plugin.enable => 0）.
The Request plugin is in Plug in list Page 2，maybe it cause the problem?
The stacktrace of Shenyu-Admin is below:

can you execute  right SQL ？ about some plugin~

repo: apache/shenyu


text:
[type:refactor] Put the creation logic of HttpClient in a separate class

The creation logic of HttpClient is too complicated. A better way than putting it in the HttpClientPluginConfiguration is to write it into a separate class.

Codecov Report

Merging #5107 (d9a6391) into master (74cc301) will decrease coverage by 0.16%.
Report is 1 commits behind head on master.
The diff coverage is 48.05%.

:exclamation: Current head d9a6391 differs from pull request most recent head cb4230e. Consider uploading reports for the commit cb4230e to get more accurate results

@@             Coverage Diff              @@
##             master    #5107      +/-   ##
============================================
- Coverage     61.81%   61.65%   -0.16%
+ Complexity     8497     8482      -15
============================================
  Files          1227     1228       +1
  Lines         36963    36958       -5
  Branches       3514     3511       -3
============================================
- Hits          22849    22787      -62
- Misses        12156    12216      +60
+ Partials       1958     1955       -3

Files Changed
Coverage Δ

...t/starter/plugin/httpclient/HttpClientFactory.java
47.36% <47.36%> (ø)

...ugin/httpclient/HttpClientPluginConfiguration.java
81.25% <100.00%> (+30.67%)
:arrow_up:

... and 33 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter xuziyang


text:
Fix/sql temporal

@desruisseaux  I'd like your review please (still cannot assign pull request to you).
PR content:

added tests upon date/time conversion from sql to java types.
Changed output API of date conversion from java.sql.Date to java.time.LocalDate
Fix timezone management for TIME WITH TIMEZONE and TIMESTAMP WITH TIMEZONE sql types.

Will apply this pull request with one amendment. This pull request changes the TIMESTAMP_WITH_TIMEZONE mapping from java.time.OffsetDateTime to java.time.Instant. I propose to keep the previous OffsetDataTime. Some searches on internet suggest that this mapping is part of JDBC 4.2 specification:

JDBC Maintenance Release 4.2
Using Java 8 Date and Time classes in PostgreSQL
Mapping between PostgreSQL and Java date/time types

authors: alexismanin desruisseaux

repo: apache/sis


text:
在UI管理页面的日志页面，无法查看client js上报的error page字段和调用栈

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
[x] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?

Bug

Which version of SkyWalking, OS, and JRE?
SkyWalking client js
SkyWalking  v8.3.0 for H2/MySQL/TiDB/InfluxDB/ElasticSearch 7

Which company or project?

What happened?
If possible, provide a way to reproduce the error. e.g. demo application, component version.

vue开发的前端控制台报了错误，但在UI管理页面貌似没有正确展示字段和调用栈
能否展示更多字段，以及展示调用栈。

Requirement or improvement

Please describe your requirements or improvement suggestions.

Please use English on Github.

Please use English on Github.

authors: withyanni wu-sheng

repo: apache/skywalking


text:
add debug info when type not match in instrumentation process

[x] If this is non-trivial feature, paste the links/URLs to the design doc.

[x] Update the documentation to include this new feature.

[x] Tests(including UT, IT, E2E) are added to verify the new feature.

[x] If it's UI related, attach the screenshots below.

[x] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.

[x] Update the CHANGES log.

Principle, we should not add log for helping developing. The logs are for product env debug only.

Principle, we should not add log for helping developing. The logs are for product env debug only.

I still do not think the log is just for product env. It's a worth as long as it can help any people resolve their problems and save their time. :)

Principle, we should not add log for helping developing. The logs are for product env debug only.

I still do not think the log is just for product env. It's a worth as long as it can help any people resolve their problems and save their time. :) And I think adding logs here is cheap.

Then a project has to face countless PR to add logs, because any line of codes could have potential risk, people could ask to log out anything. Then your system breaks.
All internal systems are being added logs randomly, one way, it is a bad thing, and also, you only face limited developers, so damages are controllable. But in the open source, especially like SkyWalking, we have 400+ code contributors, and more in potential. We cant afford to argue with every one, this log is acceptable, and others are not.
The easy, clear, and affordable principle could be only, you will need this in the runtime, or are facing the requirements.

Then a project has to face countless PR to add logs, because any line of codes could have potential risk, people could ask to log out anything. Then your system breaks.
All internal systems are being added logs randomly, one way, it is a bad thing, and also, you only face limited developers, so damages are controllable. But in the open source, especially like SkyWalking, we have 400+ code contributors, and more in potential. We cant afford to argue with every one, this log is acceptable, and others are not.
The easy, clear, and affordable principle could be only, you will need this in the runtime, or are facing the requirements.

How many contributors is agent contributors of the 400+ code contributors?
I think this log is important because I feel it. Actually I merely log anything in my project. I am not a crazy logger.

The easy, clear, and affordable principle could be only, you will need this in the runtime, or are facing the requirements.

This is a acceptable reason but still good for me. :)
Anyway, thank you for spending you energy on this pr. :)

How many contributors is agent contributors of the 400+ code contributors?

Over 70% focused on or worked on agent, AFAIK. Agent side clearly has more plugins than the server side, and easier.

I think this log is important because I feel it. Actually I merely log anything in my project. I am not a crazy logger.

You may not, but how the community should answer the question, when another people want to add logs and quote this PR? How could I prove this is useful than another log? :) It is hard for the community, and hard to understand to the new contributors.

I know where I was wrong.
I should requset a more valuable pr, add this log in passing.

Should we add a troubleshoutting at the plugin develop doc?

Should we add a troubleshoutting at the plugin develop doc?

That depends, how it could be written. It is not easy to provide such kind of documentation. Usually it is a presentation of showcases, but documentation will require it more like a book.

authors: libinglong wu-sheng


text:
Adding support to dynamic value in values.yaml

First of all this project is amazing!
I encountered an issue when i tried to use this chart as an dependency to my chart.
in my use case i want to be able to give a dynamic value in values yaml.
for an example i want that the configmap name to be dynamic because i want to deploy multiple solr cluters in the same k8s envirement.
my chart values.yaml
global:
  configmap: idolaman

solr:
  podOptions:
    volumes:
      - name: my-configmap
        source:
          configMap:
            name: "{{ .Values.global.configmap }}"

...

my Chart.yaml
name: idolaman-solr
dependencies:
  - name: solr
    ....

an solution might be change the solr chart to evaluate using tpl every value it gets

Is this common for other Helm charts?
I understand how it could be useful, but it would add a lot of complexity to the already fairly complex helm chart.

@HoustonPutman I also think that it would be nice and actually I think it's quite common. I've found this article that helps with this issue. In this example, the tpl function is used in the Helm chart templates to allow for dynamic referencing of values. Useful for scenarios where you want to deploy multiple instances of an application (like multiple Solr clusters).
This example aligns with this scenario where we want to dynamically set the configmap name for deploying multiple Solr clusters.
To show the need for the feature in general I'm referring you to a SO question the requests the same.

authors: HoustonPutman almogtavor idolaman

repo: apache/solr-operator


text:
SOLR-16179 : Updated the documentation as reported in the ticket , along with the examples

https://issues.apache.org/jira/browse/SOLR-16179
Description
terms.fl can be specified in the query multiple times, which is not clear from the documentation.
Solution
Added the respective documentation with examples in JSON and XML
Tests
Reviewed the created doc with asciidoctor and verified the changes.
Checklist
Please review the following and check all that apply:

[x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
[x] I have created a Jira issue and added the issue ID to my pull request title.
[x] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
[x] I have developed this patch against the main branch.
[x] I have run ./gradlew check.
[ ] I have added tests for my changes.
[x ] I have added documentation for the Reference Guide

The rest of the Terms page ONLY has xml...    And since the xml output and the json output are really jsut the same, I don't think having both formats helps the readability.   I could see a case for just using JSON everywhere????

authors: atarora epugh

repo: apache/solr


text:
[SPARK-10359][PROJECT-INFRA] Use more random number in dev/test-dependencies.sh; fix version switching

This patch aims to fix another potential source of flakiness in the dev/test-dependencies.sh script.
@pwendell's original patch and my version used $(date +%s | tail -c6) to generate a suffix to use when installing temporary Spark versions into the local Maven cache, but this value only changes once per second and thus is highly collision-prone when concurrent builds launch on AMPLab Jenkins. In order to reduce the potential for conflicts, this patch updates the script to call Python's random number generator instead.
I also fixed a bug in how we captured the original project version; the bug was causing the exit handler code to fail.

/cc @rxin

Test build #48589 has started for PR 10558 at commit 8e86e9c.

Test build #48589 has finished for PR 10558 at commit 8e86e9c.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48589/
Test FAILed.

Test failure was due to Python 3.

Test build #48591 has started for PR 10558 at commit 77a23bf.

Test build #48591 has finished for PR 10558 at commit 77a23bf.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48591/
Test FAILed.

Merged build finished. Test FAILed.

Test build #48594 has started for PR 10558 at commit 0a6b120.

Test build #2298 has started for PR 10558 at commit 0a6b120.

Test build #48594 has finished for PR 10558 at commit 0a6b120.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48594/
Test PASSed.

Test build #2298 has finished for PR 10558 at commit 0a6b120.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

lgtm
On Saturday, January 2, 2016, Apache Spark QA notifications@github.com
wrote:

Test build #2298 has finished
https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2298/consoleFull
for PR 10558 at commit 0a6b120
https://github.com/apache/spark/commit/0a6b120b13cca8b4c4264bbda6ceb7c3ec5b7135
.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/10558#issuecomment-168449633.

Jenkins, retest this please.

Test build #48595 has started for PR 10558 at commit 0a6b120.

Test build #48595 has finished for PR 10558 at commit 0a6b120.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48595/
Test FAILed.

Aha! It looks like the code for resetting the version has a problem:
+ build/mvn --force -q versions:set '-DnewVersion=
``

`OLD_VERSION` isn't being set properly:

OLD_VERSION='
[WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, no dependency information available

[WARNING] Failed to retrieve plugin descriptor for org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not be resolved: Could not find artifact org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in central (https://repo1.maven.org/maven2)
2.0.0-SNAPSHOT'

Pushed a fix for the version issue, so I'm going to run this a few more times then will merge if it's passing.

Test build #48596 has started for PR 10558 at commit ae3d7a3.

Test build #48596 has finished for PR 10558 at commit ae3d7a3.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48596/
Test FAILed.

Test build #48599 has started for PR 10558 at commit a2d59e5.

Test build #48599 has finished for PR 10558 at commit a2d59e5.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48599/
Test FAILed.

Merged build finished. Test FAILed.

Jenkins, retest this please.

Test build #48615 has started for PR 10558 at commit a2d59e5.

Test build #48615 has finished for PR 10558 at commit a2d59e5.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48615/
Test FAILed.

Jenkins, retest this please.

Test build #48640 has started for PR 10558 at commit a2d59e5.

Test build #48640 has finished for PR 10558 at commit a2d59e5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48640/
Test PASSed.

Merged build finished. Test PASSed.

Merging now.

authors: AmplabJenkins JoshRosen SparkQA rxin

repo: apache/spark


text:
[SPARK-15030] [ML] [SparkR] Support formula in spark.kmeans in SparkR

What changes were proposed in this pull request?

RFormula supports empty response variable like ~ x + y.
Support formula in spark.kmeans in SparkR.
Fix some outdated docs for SparkR.

How was this patch tested?
Unit tests.

Test build #57439 has started for PR 12813 at commit f1ba442.

Test build #57439 has finished for PR 12813 at commit f1ba442.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57439/
Test FAILed.

Merged build finished. Test FAILed.

Jenkins, test this please.

Test build #57442 has started for PR 12813 at commit f1ba442.

Test build #57442 has finished for PR 12813 at commit f1ba442.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57442/
Test FAILed.

Test build #57445 has started for PR 12813 at commit 79d1be4.

Test build #57446 has started for PR 12813 at commit 5bdce92.

Test build #57445 has finished for PR 12813 at commit 79d1be4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57445/
Test PASSed.

Merged build finished. Test PASSed.

Test build #57446 has finished for PR 12813 at commit 5bdce92.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57446/
Test PASSed.

LGTM. Merged into master. Thanks!

authors: AmplabJenkins SparkQA mengxr yanboliang


text:
[SPARK-20453] Bump master branch version to 2.3.0-SNAPSHOT

This patch bumps the master branch version to 2.3.0-SNAPSHOT.

Test build #76122 has started for PR 17753 at commit 983f746.

Test build #76122 has finished for PR 17753 at commit 983f746.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76122/
Test PASSed.

Merging in master.


text:
[SPARK-19507][SPARK-21296][PYTHON] Avoid per-record type dispatch in schema verification and improve exception message

What changes were proposed in this pull request?
Context
While reviewing https://github.com/apache/spark/pull/17227, I realised here we type-dispatch per record. The PR itself is fine in terms of performance as is but this prints a prefix, "obj" in exception message as below:
from pyspark.sql.types import *
schema = StructType([StructField('s', IntegerType(), nullable=False)])
spark.createDataFrame([["1"]], schema)
...
TypeError: obj.s: IntegerType can not accept object '1' in type <type 'str'>

I suggested to get rid of this but during investigating this, I realised my approach might bring a performance regression as it is a hot path.
Only for SPARK-19507 and https://github.com/apache/spark/pull/17227, It needs more changes to cleanly get rid of the prefix and I rather decided to fix both issues together.
Propersal
This PR tried to

get rid of per-record type dispatch as we do in many code paths in Scala  so that it improves the performance (roughly ~25% improvement) - SPARK-21296
This was tested with a simple code spark.createDataFrame(range(1000000), "int"). However, I am quite sure the actual improvement in practice is larger than this, in particular, when the schema is complicated.

improve error message in exception describing field information as prose - SPARK-19507

How was this patch tested?
Manually tested and unit tests were added in python/pyspark/sql/tests.py.
Benchmark - codes: https://gist.github.com/HyukjinKwon/c3397469c56cb26c2d7dd521ed0bc5a3
Error message - codes: https://gist.github.com/HyukjinKwon/b1b2c7f65865444c4a8836435100e398
Before
Benchmark:

Results: https://gist.github.com/HyukjinKwon/4a291dab45542106301a0c1abcdca924

Error message

Results: https://gist.github.com/HyukjinKwon/57b1916395794ce924faa32b14a3fe19

After
Benchmark

Results: https://gist.github.com/HyukjinKwon/21496feecc4a920e50c4e455f836266e

Error message

Results: https://gist.github.com/HyukjinKwon/7a494e4557fe32a652ce1236e504a395

Closes #17227

Test build #79116 has started for PR 18521 at commit d7f6778.

Test build #79116 has finished for PR 18521 at commit d7f6778.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79116/
Test PASSed.

cc @ueshin and @holdenk who were reviewing it and, @dgingrich the author of that PR.

cc @cloud-fan who I believe reviewed my related few PRs before and @davies who I believe is used to this code path.

Test build #79128 has started for PR 18521 at commit 5b80a8b.

Test build #79128 has finished for PR 18521 at commit 5b80a8b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79128/
Test PASSed.

Merged build finished. Test PASSed.

Test build #79131 has started for PR 18521 at commit 9ee8d03.

Test build #79134 has started for PR 18521 at commit 420b4bf.

Test build #79131 has finished for PR 18521 at commit 9ee8d03.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class DataTypeVerificationTests(unittest.TestCase):

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79131/
Test PASSed.

Test build #79134 has finished for PR 18521 at commit 420b4bf.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79134/
Test FAILed.

Merged build finished. Test FAILed.

Test build #79140 has started for PR 18521 at commit 15c575f.

Test build #79141 has started for PR 18521 at commit 826dcfd.

Test build #79140 has finished for PR 18521 at commit 15c575f.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79140/
Test FAILed.

Test build #79141 has finished for PR 18521 at commit 826dcfd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79141/
Test PASSed.

@cloud-fan, I believe it is ready for another look.

LGTM, merging to master!

authors: AmplabJenkins HyukjinKwon SparkQA cloud-fan


text:
[SPARK-21513][SQL][FOLLOWUP] Allow UDF to_json support converting MapType to json for PySpark and SparkR

What changes were proposed in this pull request?
In previous work SPARK-21513, we has allowed MapType and ArrayType of MapTypes convert to a json string but only for Scala API. In this follow-up PR, we will make SparkSQL support it for PySpark and SparkR, too. We also fix some little bugs and comments of the previous work in this follow-up PR.
For PySpark
>>> data = [(1, {"name": "Alice"})]
>>> df = spark.createDataFrame(data, ("key", "value"))
>>> df.select(to_json(df.value).alias("json")).collect()
[Row(json=u'{"name":"Alice")']
>>> data = [(1, [{"name": "Alice"}, {"name": "Bob"}])]
>>> df = spark.createDataFrame(data, ("key", "value"))
>>> df.select(to_json(df.value).alias("json")).collect()
[Row(json=u'[{"name":"Alice"},{"name":"Bob"}]')]

For SparkR
# Converts a map into a JSON object
df2 <- sql("SELECT map('name', 'Bob')) as people")
df2 <- mutate(df2, people_json = to_json(df2$people))
# Converts an array of maps into a JSON array
df2 <- sql("SELECT array(map('name', 'Bob'), map('name', 'Alice')) as people")
df2 <- mutate(df2, people_json = to_json(df2$people))

How was this patch tested?
Add unit test cases.
cc @viirya @HyukjinKwon

Can one of the admins verify this patch?

ok to test

Test build #81739 has started for PR 19223 at commit 29e7323.

Test build #81739 has finished for PR 19223 at commit 29e7323.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81739/
Test FAILed.

Test build #81761 has started for PR 19223 at commit 158140e.

LGTM except for one comment left.

Test build #81766 has started for PR 19223 at commit af8d941.

Test build #81766 has finished for PR 19223 at commit af8d941.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #81761 has finished for PR 19223 at commit 158140e.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81766/
Test FAILed.

Merged build finished. Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81761/
Test FAILed.

retest this please.

Test build #81769 has started for PR 19223 at commit af8d941.

Test build #81769 has finished for PR 19223 at commit af8d941.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81769/
Test PASSed.

@HyukjinKwon @felixcheung @viirya
I has finished those change at your suggestions for this PR and it also passed all tests. Please take a look when you are available. Thanks :)

Test build #81780 has started for PR 19223 at commit 8a3a068.

Test build #81780 has finished for PR 19223 at commit 8a3a068.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81780/
Test FAILed.

Test build #81781 has started for PR 19223 at commit 66bc5b7.

Test build #81781 has finished for PR 19223 at commit 66bc5b7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81781/
Test PASSed.

LGTM

AppVeyor didn't run on this?

D'oh, yes. I wonder why it was not triggered. I manually triggered via my account:
Build started: [SparkR] ALL
Diff: https://github.com/apache/spark/compare/master...spark-test:8981A5F1-E2DC-4015-8266-12E9ADEE189B

@HyukjinKwon Thanks for triggering AppVeyor. In normal case, will AppVeyor be triggered automatically?

Yes, when there are some changes in:
https://github.com/apache/spark/blob/828fab03567ecc245a65c4d295a677ce0ba26c19/appveyor.yml#L29-L35
It should run the R tests on Windows via AppVeyor.

ok. I got it. Thanks :)

Looks passed fine. Let me merge this one.

Thanks @felixcheung @HyukjinKwon

Merged to master.

Thanks @HyukjinKwon @felixcheung @viirya

authors: AmplabJenkins HyukjinKwon SparkQA felixcheung goldmedal viirya


text:
[SPARK-22585][Core] Path in addJar is not url encoded

What changes were proposed in this pull request?
This updates a behavior of addJar method of sparkContext class. If path without any scheme is passed as input it is used literally without url encoding/decoding it.
How was this patch tested?
A unit test is added for this.

Test build #84262 has started for PR 19834 at commit 1fc5db3.

@srowen Let's continue our discussion here.

So I have removed those three commented lines. Is there anything else to do before merge?

Test build #84266 has started for PR 19834 at commit bd667d9.

Test build #84262 has finished for PR 19834 at commit 1fc5db3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84262/
Test PASSed.

Test build #84266 has finished for PR 19834 at commit bd667d9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84266/
Test FAILed.

LGTM.

Another LGTM

retest this please

Test build #84295 has started for PR 19834 at commit bd667d9.

Test build #84295 has finished for PR 19834 at commit bd667d9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84295/
Test FAILed.

retest this please

Test build #84302 has started for PR 19834 at commit bd667d9.

Test build #84302 has finished for PR 19834 at commit bd667d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84302/
Test PASSed.

Merged to master.

Thanks!

authors: AmplabJenkins HyukjinKwon SparkQA james64 jerryshao jiangxb1987


text:
[SPARK-25041][build] upgrade genJavaDoc-plugin from 0.10 to 0.11

What changes were proposed in this pull request?
This PR fixes a build error with sbt using Scala-2.12. Since [genJavaDoc-plugin] (https://mvnrepository.com/artifact/com.typesafe.genjavadoc/genjavadoc-plugin) 0.10 is not prepared for Scala-2.12.6, the recent version of genJavaDoc-plugin is necessary.
The version 0.11 of genJavaDoc-plugin is also prepared for Scala-2.11.12.
genJavaDoc-0.10
genJavaDoc-0.11
How was this patch tested?
Manually tested for Scala-2.12.

cc @ueshin @HyukjinKwon @srowen

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1899/
Test PASSed.

Test build #94356 has started for PR 22020 at commit 1b41ce4.

Test build #94356 has finished for PR 22020 at commit 1b41ce4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94356/
Test PASSed.

Merged to master

authors: AmplabJenkins SparkQA kiszk srowen


text:
[SPARK-26449][PYTHON] add a transform method to the Dataframe class

What changes were proposed in this pull request?
added a transform method to the Dataframe class, see https://issues.apache.org/jira/browse/SPARK-26449
How was this patch tested?
Tested manually by injecting the proposed method to the current spark version dataframe class.
I've tried to compile spark from scratch and test using ./build/mvn test. However, unrelated tests fails before my change.
Please review http://spark.apache.org/contributing.html before opening a pull request.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

ok to test

Test build #100560 has started for PR 23414 at commit def5b2c.

Test build #100560 has finished for PR 23414 at commit def5b2c.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100560/
Test FAILed.

Test build #100562 has started for PR 23414 at commit def5b2c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6494/
Test PASSed.

Test build #100562 has finished for PR 23414 at commit def5b2c.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100562/
Test FAILed.

Merged build finished. Test FAILed.

Test build #100592 has finished for PR 23414 at commit b370363.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100592/
Test FAILed.

@HyukjinKwon I get the following errors:
[error] running /home/jenkins/workspace/SparkPullRequestBuilder@2/dev/lint-python ; received return code 1
Attempting to post to Github...
 > Post successful.
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100592/
Test FAILed.
Finished: FAILURE

Can you please help?

Looks it's failed for below reasons.
pycodestyle checks failed:
./python/pyspark/sql/dataframe.py:2048:1: W293 blank line contains whitespace
./python/pyspark/sql/dataframe.py:2064:1: W293 blank line contains whitespace

added doctest and removed more empty line with spaces. please re-test

Test build #100594 has started for PR 23414 at commit f5aaa1a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6523/
Test PASSed.

Test build #100594 has finished for PR 23414 at commit f5aaa1a.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100594/
Test FAILed.

removed *args **kwargs (albeit I think they're useful). Please re-test

Test build #100595 has started for PR 23414 at commit 0b1f562.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6524/
Test FAILed.

Test build #100595 has finished for PR 23414 at commit 0b1f562.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100595/
Test FAILed.

@HyukjinKwon I am sorry for being newbie but I don't understand the fail reason:
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress https://github.com/apache/spark.git +refs/pull/23414/*:refs/remotes/origin/pr/23414/*" returned status code 128:
stdout:
stderr: error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: The remote end hung up unexpectedly

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6525/
Test PASSed.

@HyukjinKwon what do you mean when you say the Scala impl has this? I'm missing it.
I don't see the value in this. From the blog post at https://medium.com/@mrpowers/chaining-custom-pyspark-transformations-4f38a8c7ae55 why is ...
actual_df = (source_df
    .transform(lambda df: with_greeting(df))
    .transform(lambda df: with_something(df, "crazy")))

better than just
actual_df = with_greeting(source_df)
actual_df = with_something(actual_df, "crazy")

The idea is to be able to chain function easily when you have 10 stages. no need for keeping temporary variables.

You can also...
actual_df = source_df
for f in [...]:
    actual_df = f(actual_df)

Unless I'm really missing something this doesn't exist for Scala (?) and I can't see adding an API method for this. The small additional maintenance and user cognitive load just doesn't seem to buy much at all.

@srowen the motivation is from this blogpost https://medium.com/@mrpowers/chaining-custom-pyspark-transformations-4f38a8c7ae55

I was referring:
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L2497
If it were new API, I won't encourage to add but it's existing. I think we should rather deprecate Scala side one if we don't see some values on that. Otherwise, I thought matching it is fine.

Oh hm I had never seen that! Yah seems fine for consistency then.

@chanansh, also please fix the PR title to [SPARK-26449][PYTHON] ...  so that it automatically links your PR to the JIRA.

Test build #100610 has started for PR 23414 at commit 9919e28.

Test build #100610 has finished for PR 23414 at commit 9919e28.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100610/
Test FAILed.

Test build #100611 has started for PR 23414 at commit e54d2f7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6535/
Test PASSed.

Test build #100611 has finished for PR 23414 at commit e54d2f7.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100611/
Test FAILed.

Test build #100612 has started for PR 23414 at commit 3d9a751.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/6536/
Test PASSed.

Test build #100612 has finished for PR 23414 at commit 3d9a751.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/100612/
Test PASSed.

Looks fine except https://github.com/apache/spark/pull/23414/files#r244654162

Closing this due to author's inactivity.

sorry, please reopen I will do it.
HS
On Mon, Feb 11, 2019 at 12:10 PM Hyukjin Kwon notifications@github.com
wrote:

Closed #23414 https://github.com/apache/spark/pull/23414.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/23414#event-2130116400, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFtFzGK1zXfQq-EhAgPgm1uYJcDhZCMjks5vMUGsgaJpZM4Zk9No
.

Just push more commits; I think that reopens it.

is this one still open? I would want to PR basically the same thing. Should I commit here or create a new PR?

You can pick up commits and create new PR. Looks the author is inactive.

authors: AmplabJenkins Hellsen83 HyukjinKwon SparkQA chanansh srowen


text:
[MINOR][DOCS] Clarify that Spark apps should mark Spark as a 'provided' dependency,  not package it

What changes were proposed in this pull request?
Spark apps do not need to package Spark. In fact it can cause problems in some cases. Our examples should show depending on Spark as a 'provided' dependency.
How was this patch tested?
Doc build

Test build #102943 has started for PR 23938 at commit f8fcc52.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/8412/
Test PASSed.

Test build #102943 has finished for PR 23938 at commit f8fcc52.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/102943/
Test PASSed.

@dongjoon-hyun yeah, that one I wasn't sure about as it's some support code that sounded like it was meant to be bundled in an app. @steveloughran is that correct -- hadoop-cloud should be a compile scope dependency, not provided by the cluster?

you should compile with hadoop-cloud and add those JARs it pulls in to the spark tarball placed on the shared cluster FS for YARN to pick up. Don't know about other deployment engines I'm afraid. The build also adds it to the SPARK_HOME/lib, which gives it to you for spark-standalone during spark submit, either for anything related to JAR upload, or for any store which implements delegation tokens (HADOOP-14456, HADOOP-16068, etc), so it collects the tokens for all stores listed in spark.yarn.hadoopFilesystems.

@steveloughran to be clear do you compile your app, or Spark, with this dependency? it sounds like "Spark" not the app. If so I'll update this further.

sorry, yeah, spark.
Even if the spark team doesn't redist those JARs, it'd be really useful if the release process published the POM. that way, if you want your build to pick up the exact set of dependencies which are in sync with spark, excluding all the stuff which will cause grief, you'd just add it as a dependency.

Ah OK on further review @steveloughran , the docs here are saying to include the dependency in your app, which would be the right thing if not bundled by Spark, and that's the current state of things for a default cluster. I think that much of the doc is then OK, and shouldn't change to mentioned provided.

Merged to master/2.4/2.3

authors: AmplabJenkins SparkQA srowen steveloughran


text:
[SPARK-28656][SQL] Support millennium, century and decade at extract()

What changes were proposed in this pull request?
In the PR, I propose new expressions Millennium, Century and Decade, and support additional parameters of extract() for feature parity with PostgreSQL (https://www.postgresql.org/docs/11/functions-datetime.html#FUNCTIONS-DATETIME-EXTRACT):

millennium - the current millennium for given date (or a timestamp implicitly casted to a date). For example, years in the 1900s are in the second millennium. The third millennium started January 1, 2001.
century - the current millennium for given date (or timestamp). The first century starts at 0001-01-01 AD.
decade - the current decade for given date (or timestamp). Actually, this is the year field divided by 10.

Here are examples:
spark-sql> SELECT EXTRACT(MILLENNIUM FROM DATE '1981-01-19');
2
spark-sql> SELECT EXTRACT(CENTURY FROM DATE '1981-01-19');
20
spark-sql> SELECT EXTRACT(DECADE FROM DATE '1981-01-19');
198

Also the expressions are registered as functions - millennium, century and decade. For example:
spark-sql> SELECT MILLENNIUM('2019-08-08');
3
spark-sql> SELECT CENTURY('2019-08-08');
21
spark-sql> SELECT DECADE('2019-08-08');
201

How was this patch tested?
Added new tests to DateExpressionsSuite, DateFunctionsSuite, and uncommented existing tests in pgSQL/date.sql.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13887/
Test PASSed.

Test build #108806 has started for PR 25388 at commit 6755bce.

Can one of the admins verify this patch?

Test build #108806 has finished for PR 25388 at commit 6755bce.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108806/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13906/
Test PASSed.

Test build #108829 has started for PR 25388 at commit 381f214.

Test build #108829 has finished for PR 25388 at commit 381f214.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108829/
Test PASSed.

Hi, @MaxGekk .

Supporting Extract seems to be enough for PostgreSQL feature parity.
Supporting the followings are easy, but I'm not sure about these. In general, I'd recommend not to register as functions. PMC may have different opinions.

spark-sql> SELECT MILLENNIUM('2019-08-08');
3
spark-sql> SELECT CENTURY('2019-08-08');
21
spark-sql> SELECT DECADE('2019-08-08');
201

How do you think about (2) which adds these new functions , @gatorsmile and @cloud-fan ?

Let's not add builtin functions that only exist in Spark.

Thank you for the decision, @cloud-fan !

Test build #108867 has started for PR 25388 at commit 9d9a0ad.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13940/
Test PASSed.

Test build #108867 has finished for PR 25388 at commit 9d9a0ad.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
sealed trait RewritableTransform extends Transform
case class ArrayForAll(
case class DescribeTable(table: NamedRelation, isExtended: Boolean) extends Command
trait V2CreateTablePlan extends LogicalPlan
case class DescribeColumnStatement(
case class DescribeTableStatement(
case class InsertAdaptiveSparkPlan(
case class DescribeTableExec(table: Table, isExtended: Boolean) extends LeafExecNode

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108867/
Test FAILed.

jenkins, retest this, please

Test build #108868 has started for PR 25388 at commit 9d9a0ad.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13941/
Test PASSed.

Test build #108868 has finished for PR 25388 at commit 9d9a0ad.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
sealed trait RewritableTransform extends Transform
case class ArrayForAll(
case class DescribeTable(table: NamedRelation, isExtended: Boolean) extends Command
trait V2CreateTablePlan extends LogicalPlan
case class DescribeColumnStatement(
case class DescribeTableStatement(
case class InsertAdaptiveSparkPlan(
case class DescribeTableExec(table: Table, isExtended: Boolean) extends LeafExecNode

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108868/
Test PASSed.

authors: AmplabJenkins MaxGekk SparkQA cloud-fan dongjoon-hyun