text:
porting: Update targets and updating script

We need to update pkg names to represent at path to the targets.
Also update the update_generated_files.sh script to
refer to the targets using a path. This is required
after recent change to Newt.

The change in newt that caused this: https://github.com/apache/mynewt-newt/pull/381

On paper the changes look valid to me. However, I am having trouble to get them running...
So from what I know about newt by now, I expected to do the following:

create a new (empty) project, newt new foo && cd foo
change nimble repo version to 0.0.0 (master) and update repos  newt upgrade
run ./repos/apache-mynewt-nimble/porting/update_generated_files.sh
or trigger selected porting targets manually -> newt build riot? or newt build @apache-mynewt-nimble/porting/targets/riot?

But:
a) how to set the NimBLE repo to use the remote branch from this commit?
b) when calling the target manually from within the foo project (newt build "@apache-mynewt-nimble/porting/targets/riot") I get Error: Could not resolve target name: @apache-mynewt-nimble/porting/targets/riot
So after all, I seem to be unable to build those targets included in the nimble repo...

@haukepetersen
a) not sure if you can do that using newt upgrade but you can go into nimble repo and either:

add my fork as a remote and then fetch the branch or
fetch the PR branch directly git fetch origin pull/ID/head:BRANCHNAME

b) That's because the target names need to be changed too. If you open pkg.yml in riot target and change pkg.name to porting/targets/riot, then you should be able to build using newt build @apache-mynewt-nimble/porting/targets/riot

of course, setting the branch manually inside of my newt project is something I didn't think of... I actually spend some time figuring out, how I can configure my project to use my local nimble repository I normally use for development, which is located outside of the newt project...
Anyway, now it works and I can confirm that the changes in this PR are working as expected. Except different values for
MYNEWT_VAL_REPO_HASH_APACHE_MYNEWT_CORE, MYNEWT_VAL_REPO_HASH_MCUBOOT,
MYNEWT_VAL_REPO_HASH_APACHE_MYNEWT_NIMBLE, MYNEWT_VAL_REPO_HASH_APACHE_MYNEWT_MCUMGR, MYNEWT_VAL_REPO_HASH_APACHE_MYNEWT_CORE, and MYNEWT_VAL_REPO_VERSION_APACHE_MYNEWT_CORE, my generated files are identical to the ones in this PR.

Actually now that you mentioned it, these MYNEWT_VAL_REPO_HASH_* defines kind of defeat the purpose of this script as they will change all the time. I want to use this script in a CI to automatically remind contributors to update these generated files. So as a workaround I will filter out these lines using a script.

@haukepetersen I updated the script and generated files.

Works like a charm, the change in #828 now only triggers that one change in the generated syscfg.h file, perfect :-)
Before merging, you might want to move the latest change from the 2nd into the 1st commit, should be easier to track.

authors: haukepetersen michal-narajowski


text:
NIFI-3052 Update screenshots in Admin Guide and Getting Starting guid…

…e for additional colors added to UI.  Also updated Global Menu (text alignment has been fixed) and lineage graph (download icon has been added).

Reviewing...

@andrewmlim I dont think you can just change the start and stop icons since the context menu and the operate pallet still use the blue icons...

@scottyaslan Thanks for catching that.  I think we can go through with this PR because for the Admin Guide and Getting Started Guide, we do want to use the colored icons.  So iconRun, iconStop, iconAlert will be the screenshots that show status.   I will create a separate Jira to capture the blue icons needed for the operate pallete and context menu buttons.

Filed as:  https://issues.apache.org/jira/browse/NIFI-3056

@andrewmlim  another item to be addressed here would be the enabled/disabled and the transmitting/not-transmitting icons as the color of these icons have changed recently as well....

Thanks @scottyaslan.  I modified https://issues.apache.org/jira/browse/NIFI-3056 to be more generic to capture all the screenshots that still need to be updated as a a result of https://issues.apache.org/jira/browse/NIFI-2603.   But I think this PR should go through since all the screenshots for the Admin Guide and Getting Started guide do not need any additional changes.

Thanks @andrewmlim this has been merged to master.

source: gharchive/pull-request

authors: andrewmlim scottyaslan

repo: apache/nifi

license_type: permissive

license_source: github-api


text:
NIFI-4137: Add support for UTF-8 in GetFTP and PutFTP

I couldn't add unit tests as the FakeFtpServer doesn't have a mode where UTF-8 is not supported. However it should be easy to test with a local FTP server.
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit?

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[x] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

@mattyb149 will review

Ran tests using FileZilla on Windows 10 Server with flow containing GetFTP -> UpdateAttributes (for filename append) -> PutFTP.  Worked as expected with the below use cases:
Below demonstrates cases where UTF-8 was disabled then enabled

Files were stored as expected on FTP server:

+1  will merge shortly


text:
NIFI-5906 Removed FILE_EXISTS_VALIDATOR for LivySessionController in order to allow HDFS and remote file path

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Removed FILE_EXISTS_VALIDATOR for LivySessionController in order to allow HDFS path and remote file path
Fixes bug NIFI-5906
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically main)?

[ ] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] Have you verified that the full build is successful on JDK 8?
[ ] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

Hi @celsomarques - Can you please rebase your pull request and make sure it contains only a single commit with your changes?


text:
NIFI-9080 Converted nifi-commons to use JUnit 5.

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Enables X functionality; fixes bug NIFI-YYYY.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically main)?

[ ] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] Have you verified that the full build is successful on JDK 8?
[ ] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

@exceptionfactory I went back and did a number on this one, particularly in converting fail() to the appropriate throws/doesnothrow logic. Builds locally.

@exceptionfactory should be ready now

Thanks the updates @MikeThomsen, can you rebase from the main branch to resolve conflicts from recent adjustments to unit tests?  I also noticed a handful of tests marked Disabled, some of which appear to be performance-related, others which might be potential candidates for removal.

@exceptionfactory ready to go

@exceptionfactory should be good now.

authors: MikeThomsen exceptionfactory


text:
NIFI-12333 MiNiFi property override through env and system variables

Summary
NIFI-12333
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[x] Apache NiFi Jira issue created

Pull Request Tracking

[x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[x] Pull Request based on current revision of the main branch
[ ] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[x] Build completed using mvn clean install -P contrib-check

[x] JDK 21

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

Thanks for the comments, I've applied them. Also needed to rename / create a new minifi-commons-utils module, hope it doesn't make the change too noisy.


text:
mcuboot: update mcuboot to support swap using move

Summary
Older versions of mcuboot did not support swap using move and therefore the build would have failed CONFIG_MCUBOOT_SWAP_USING_MOVE=y.
Newer mcuboot contains initial support of swap using move for NuttX. Also Makefile was updated to as some bootutil files were removed from mcuboot.
Impact
Updates MCUboot and because of deletion of some files older versions will not compile.
Testing
Tested on SAMv7 MCU.

Also updated cmake format

repo: apache/nuttx-apps


text:
riscv/rv64ilp32: revise fpucmp

Summary
This revises up_fpucmp() to support rv64ilp32.
Impact
riscv
Testing

local test with rv-virt target
CI checks

@acassis @xiaoxiang781216, sorry I only meant to send 1 patch but accidently sent 3. Now I've dropped

@acassis and @xiaoxiang781216, Sorry for my bad, I only meant to send 1 patch. now it is cleaned.

repo: apache/nuttx


text:
drivers/sensors: add support of InvenSense MPU-9250 sensor

Summary
Implement drivers  of MPU-9250 in new sensor framework, include Accelerometer, Gyroscope, Magnetometer
Impact
nothing
Testing
Custom board and mpu9250 on SPI bus
sensortest accel0
sensortest gyro0
sensortest mag0
uorb_listener

Very nice @zouboan! I suggest you to include later a board usage example for some known, low cost and easy to find board like stm32f4discovery, raspberrypi-pico or esp32-devkit.

@acassis Thanks, i will explorer which board have MPU9250

authors: acassis zouboan


text:
Fix deprecated override warning in ResourceLeakDetector.

Description
This class yields an annoying compiler warning like:
Note: /Users/mthoemme/dev/openwhisk/tests/src/test/scala/org/apache/openwhisk/core/database/cosmosdb/RecordingLeakDetectorFactory.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

Sadly, we need to override the method as it is abstract. Suppressing the warning should be fine in that case.
Checklist:

[X] I signed an Apache CLA.
[X] I reviewed the style guides and followed the recommendations (Travis CI will check :).

uses or overrides a deprecated API.

what's the replacement? can we migrate?

@rabbah The suggested replacement is only for people using the base class. For overriding (at least if my Java ain't too rusty) we need to override the abstract member with an explicit implementation (which sadly is also the deprecated thingy here).

authors: markusthoemmes rabbah

repo: apache/openwhisk


text:
HDDS-10699. Refactor ContainerBalancerTask and tests in TestContainerBalancerTask

In PR for HDDS-9889 we discussed with Siddhant Sangwan that tests form org.apache.hadoop.hdds.scm.container.balancer.TestContainerBalancerTask could be refactored using MockedSCM class (introduced in HDDS-9889)
What changes were proposed in this pull request?

Refactor some tests from org.apache.hadoop.hdds.scm.container.balancer.TestContainerBalancerTask
Made some cleanups in org.apache.hadoop.hdds.scm.container.balancer.ContainerBalancerTask related to changes in refactored test

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10699
How was this patch tested?
Use standalone tests

Thanks @Montura for the patch, @sumitagrawl, @Tejaskriya for the review.

authors: Montura adoroszlai

repo: apache/ozone


text:
[core][spark] Support adding and dropping nested columns in Spark

Purpose
Spark SQL supports adding and dropping nested columns, but Paimon currently does not support it. This PR adds support for adding and dropping nested columns.
Tests
Unit tests and IT cases.
API and Format
No format changes.
Documentation
This feature does not need a document.

Try to support this in Flink SQL too:
Flink SQL> create table s1 (a int, nc row<f1 int, f2 string> ) with ('connector'='datagen', 'number-of-rows'='10');
[INFO] Execute statement succeed.

Flink SQL> alter table s1 modify nc row<f1 int, f2 string, f3 string>;
[INFO] Execute statement succeed.

Try to support this in Flink SQL too:
Flink SQL> create table s1 (a int, nc row<f1 int, f2 string> ) with ('connector'='datagen', 'number-of-rows'='10');
[INFO] Execute statement succeed.

Flink SQL> alter table s1 modify nc row<f1 int, f2 string, f3 string>;
[INFO] Execute statement succeed.

Let's implement this in another PR.

authors: JingsongLi tsreaper

repo: apache/paimon


text:
IDs are dropped in metadata conversion

Reporter: Ryan Blue / @rdblue
Assignee: Ryan Blue / @rdblue
PRs and other links:

PR #428

Note: This issue was originally created as PARQUET-1141. Please see the migration documentation for further details.

Ryan Blue / @rdblue:
Merged #428.

source: gharchive/issue

repo: apache/parquet-java


text:
ThriftParquetWriter converts thrift byte to int32 without adding logical type

The current implementation of Parquet serialisation from Thrift Definitions results in the incorrect conversion of Thrift byte fields into INT32 without preserving the required LogicalType Metadata in the Parquet file. This behaviour leads to a loss of information and is inconsistent with the expected behaviour. The correct conversion should result in INT32 with LogicalType metadata indicating a bit width of 8 and signed as true.

Thrift Definition

struct TestLogicalType {
1: required i16 test_i16,
2: required byte test_i8
}

Current Parquet Schema Generated

message ParquetSchema {
  required int32 test_i16 (INTEGER(16,true)) = 1;
  required int32 test_i8 = 2;
}

Expected Parquet Schema

message ParquetSchema {
  required int32 test_i16 (INTEGER(16,true)) = 1;
  required int32 test_i8 (INTEGER(8,true)) = 2;
}

Reporter: Shreyas B / @shreyas-dview
Assignee: Shreyas B / @shreyas-dview
PRs and other links:

GitHub Pull Request #1254

Note: This issue was originally created as PARQUET-2420. Please see the migration documentation for further details.

Shreyas B / @shreyas-dview:
PR Fix: https://github.com/apache/parquet-mr/pull/1254


text:
Updating Staging

This PR does a few things:

Moves from the docsy git submodule to hugo modules
Pulls in the changes from production to bring staging up to the same point
Updates the github actions versions
Modes from _index.html -> _index.md for the site root
Updates the 404 page

IIUC, the staging size is no longer maintained. cc @gszadovszky @shangxinli

Yeah haha @wgtmac, that's mostly me being negligent. I'm going to get everything up to date on both staging and production now

+1

FYI in case someone finds this later, the URL of staging is https://parquet.staged.apache.org/

Is it possible to retire the staging site? It does not seem necessary to me.

@wgtmac - that's a valid point. It's good for situations like this where we have to debug something strange before going to prod (https://github.com/apache/parquet-site/actions/runs/8149845126/job/22275151185).
But, overall, it's pretty common for folks to have a staging site. For example: https://github.com/apache/dubbo-website/blob/master/.asf.yaml#L31. ASF docs have it as an option too: https://github.com/apache/infrastructure-asfyaml/blob/main/README.md#staging.

Ugh I think staging is hitting this: https://github.com/golang/go/issues/62278
go: download go1.23 for linux/amd64: toolchain not available

authors: shangxinli vinooganesh wgtmac

repo: apache/parquet-site


text:
PHOENIX-6874 Support older HBase versions with broken ShortCircuitCon…

…nection

The changes look good. I was thinking, can we make hbasexy.compat.version as ${hbase-x.y.runtime.version} for each compat module?
Like hbase23.compat.version as ${hbase-2.3.runtime.version}?

nvm, this is not required to be handled here.

authors: stoty virajjasani

repo: apache/phoenix

license_source: bigquery


text:
Broker Query Timeout Metric

Exposing a new metric to trace broker query timeouts.

Codecov Report

Merging #11892 (5139ef3) into master (5d58102) will decrease coverage by 26.75%.
The diff coverage is 12.50%.

@@              Coverage Diff              @@
##             master   #11892       +/-   ##
=============================================
- Coverage     61.45%   34.71%   -26.75%
+ Complexity     1147      945      -202
=============================================
  Files          2375     2299       -76
  Lines        128500   124770     -3730
  Branches      19846    19292      -554
=============================================
- Hits          78974    43308    -35666
- Misses        43815    78419    +34604
+ Partials       5711     3043     -2668

Flag
Coverage Δ

custom-integration1
?

integration
<0.01% <0.00%> (ø)

integration1
<0.01% <0.00%> (ø)

integration2
?

java-11
?

java-21
34.71% <12.50%> (-26.60%)
:arrow_down:

skip-bytebuffers-false
<0.01% <0.00%> (-61.42%)
:arrow_down:

skip-bytebuffers-true
46.45% <100.00%> (-14.84%)
:arrow_down:

temurin
34.71% <12.50%> (-26.75%)
:arrow_down:

unittests
46.45% <100.00%> (-15.00%)
:arrow_down:

unittests1
46.45% <100.00%> (-0.20%)
:arrow_down:

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files
Coverage Δ

...a/org/apache/pinot/common/metrics/BrokerMeter.java
94.00% <100.00%> (+0.12%)
:arrow_up:

...thandler/SingleConnectionBrokerRequestHandler.java
0.00% <0.00%> (-18.19%)
:arrow_down:

...requesthandler/MultiStageBrokerRequestHandler.java
0.00% <0.00%> (-18.71%)
:arrow_down:

... and 835 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

authors: codecov-commenter suddendust

repo: apache/pinot


text:
Gzip compression, ensure uncompressed size can be calculated from compressed buffer

this is a follow up on PR https://github.com/apache/pinot/pull/12668 addressing originally https://github.com/apache/pinot/issues/11434 .
This dcument https://www.rfc-editor.org/rfc/rfc1952 claims that a field ISIZE is apended at the end of a block (block == compressed segment). This does not seem to be the case when using the Deflater, so we simply append it. The Inflater is cleaver enough to ignore these extra 4 bytes at the end of the compressed buffer, while we can use them to compute
ChunkDecompressor :: int decompressedLength(ByteBuffer compressedInput) throws IOException;

Codecov Report
Attention: Patch coverage is 0% with 8 lines in your changes are missing coverage. Please review.

Project coverage is 34.81%. Comparing base (59551e4) to head (0127e0b).
Report is 215 commits behind head on master.

Files
Patch %
Lines

...t/segment/local/io/compression/GzipCompressor.java
0.00%
4 Missing :warning:

...segment/local/io/compression/GzipDecompressor.java
0.00%
4 Missing :warning:

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #12802       +/-   ##
=============================================
- Coverage     61.75%   34.81%   -26.94%
+ Complexity      207        6      -201
=============================================
  Files          2436     2390       -46
  Lines        133233   131208     -2025
  Branches      20636    20285      -351
=============================================
- Hits          82274    45683    -36591
- Misses        44911    82241    +37330
+ Partials       6048     3284     -2764

Flag
Coverage Δ

custom-integration1
?

integration
<0.01% <0.00%> (-0.01%)
:arrow_down:

integration1
<0.01% <0.00%> (-0.01%)
:arrow_down:

integration2
0.00% <0.00%> (ø)

java-11
0.00% <0.00%> (-61.71%)
:arrow_down:

java-21
34.81% <0.00%> (-26.81%)
:arrow_down:

skip-bytebuffers-false
34.80% <0.00%> (-26.95%)
:arrow_down:

skip-bytebuffers-true
34.79% <0.00%> (+7.06%)
:arrow_up:

temurin
34.81% <0.00%> (-26.94%)
:arrow_down:

unittests
46.54% <0.00%> (-15.21%)
:arrow_down:

unittests1
46.54% <0.00%> (-0.35%)
:arrow_down:

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter marregui


text:
Fix default column handling when forward index is disabled

This was introduced in #14150 where _forwardIndexDisabledColumns is no longer extracted in IndexLoadingConfig
Fix the behavior in BaseDefaultColumnHandler to use the standard way to create forward index config

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 0.00%. Comparing base (59551e4) to head (92da172).
Report is 1177 commits behind head on master.

:exclamation:  There is a different number of reports uploaded between BASE (59551e4) and HEAD (92da172). Click for more details.
HEAD has 53 uploads less than BASE

Flag
BASE (59551e4)
HEAD (92da172)

integration
7
1

integration2
3
1

temurin
12
1

java-21
7
1

skip-bytebuffers-true
3
0

skip-bytebuffers-false
7
1

unittests
5
0

unittests1
2
0

java-11
5
0

unittests2
3
0

integration1
2
0

custom-integration1
2
0

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #14215       +/-   ##
=============================================
- Coverage     61.75%    0.00%   -61.76%
=============================================
  Files          2436        3     -2433
  Lines        133233        6   -133227
  Branches      20636        0    -20636
=============================================
- Hits          82274        0    -82274
+ Misses        44911        6    -44905
+ Partials       6048        0     -6048

Flag
Coverage Δ

custom-integration1
?

integration
0.00% <ø> (-0.01%)
:arrow_down:

integration1
?

integration2
0.00% <ø> (ø)

java-11
?

java-21
0.00% <ø> (-61.63%)
:arrow_down:

skip-bytebuffers-false
0.00% <ø> (-61.75%)
:arrow_down:

skip-bytebuffers-true
?

temurin
0.00% <ø> (-61.76%)
:arrow_down:

unittests
?

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: Jackie-Jiang codecov-commenter


text:
Skip creating null value vector when there is no null value

Also clean up the unused hasNull() in the stats collection. A column contains null values when the null value vector exists and not empty.

Codecov Report

Merging #8756 (38d376d) into master (6ad7973) will decrease coverage by 40.61%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master    #8756       +/-   ##
=============================================
- Coverage     69.61%   28.99%   -40.62%
+ Complexity     4613       45     -4568
=============================================
  Files          1730     1718       -12
  Lines         90404    90075      -329
  Branches      13444    13412       -32
=============================================
- Hits          62934    26117    -36817
- Misses        23090    61547    +38457
+ Partials       4380     2411     -1969

Flag
Coverage Δ

integration1
26.83% <0.00%> (+0.04%)
:arrow_up:

integration2
25.27% <0.00%> (+0.02%)
:arrow_up:

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...ltime/converter/stats/MutableColumnStatistics.java
0.00% <ø> (-59.68%)
:arrow_down:

...verter/stats/MutableNoDictionaryColStatistics.java
0.00% <ø> (-40.91%)
:arrow_down:

...creator/impl/nullvalue/NullValueVectorCreator.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../stats/BigDecimalColumnPreIndexStatsCollector.java
0.00% <ø> (-68.30%)
:arrow_down:

...impl/stats/BytesColumnPredIndexStatsCollector.java
0.00% <ø> (-79.25%)
:arrow_down:

...impl/stats/DoubleColumnPreIndexStatsCollector.java
0.00% <ø> (-78.05%)
:arrow_down:

.../impl/stats/FloatColumnPreIndexStatsCollector.java
0.00% <ø> (-78.05%)
:arrow_down:

...or/impl/stats/IntColumnPreIndexStatsCollector.java
0.00% <ø> (-78.05%)
:arrow_down:

...r/impl/stats/LongColumnPreIndexStatsCollector.java
0.00% <ø> (-78.05%)
:arrow_down:

...impl/stats/StringColumnPreIndexStatsCollector.java
0.00% <ø> (-78.85%)
:arrow_down:

... and 1173 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ad7973...38d376d. Read the comment docs.


text:
Receiving "Connection Already Closed Error on Creating Producer"

Expected behavior
I would expect the client to connect and build, or give a more useful error.
Actual behavior
When attempting to create the producer, I receive this error:
Exception in thread "main" org.apache.pulsar.client.api.PulsarClientException: Connection already closed
	at org.apache.pulsar.client.impl.ClientCnx.channelInactive(ClientCnx.java:169)
	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377)
	at org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342)
	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
	at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1409)
	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
	at org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
	at org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:927)
	at org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822)
	at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
	at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

Steps to reproduce
We are not able to reproduce this issue with the python library. Here is the java code:
messageBus = PulsarClient.builder()
                .serviceUrl(messageBusUrl)
                .tlsTrustCertsFilePath("/Users/josh/Documents/myriadmobile/kernel/translator-mono/core/src/main/resources/pulsar-ca-cert.pem")
                .authentication("org.apache.pulsar.client.impl.auth.AuthenticationTls",
                        "tlsCertFile:/Users/josh/Documents/myriadmobile/kernel/translator-mono/core/src/main/resources/pulsar-public.pem,tlsKeyFile:/Users/josh/Documents/myriadmobile/kernel/translator-mono/core/src/main/resources/pulsar-private.pem")
                .build()

System configuration
This was originally encountered with 2.0.1-incubating (as referenced for the import in the java client import), and also persists in 2.1.0 and 2.1.1

@MrThreepwood I think you need to set enableTls to true.
 messageBus = PulsarClient.builder()
                 .serviceUrl(messageBusUrl)
+                .enableTls(true)
                 .tlsTrustCertsFilePath("/Users/josh/Documents/myriadmobile/kernel/translator-mono/core/src/main/resources/pulsar-ca-cert.pem")
                 .authentication("org.apache.pulsar.client.impl.auth.AuthenticationTls",
                         "tlsCertFile:/Users/josh/Documents/myriadmobile/kernel/translator-mono/core/src/main/resources/pulsar-public.pem,tlsKeyFile:/Users/josh/Documents/myriadmobile/kernel/translator-mono/core/src/main/resources/pulsar-private.pem")
                 .build()

This configuration will be unnecessary in 2.2.0 or later.
cf. https://github.com/apache/pulsar/pull/2315

The message is still misleading. It would be good to have something a bit more explicit to say that the client was not able to connect to the remote endpoint.

This appears to be working for me, thanks. It is not mentioned as part of the documentation here however:
https://pulsar.apache.org/docs/latest/security/tls/
I also agree with ivankelly that this is very misleading. Clearly I'm encountering an auth error, so simply closing the connection and throwing the first time I attempt to create a producer or consumer isn't behavior that I would expect.

@MrThreepwood I think the documentation in the latest website is correct. You were checking the old website.
Closing this issue for now. If you need more help from us, feel free to reopen it.

I have the same issue running for Python version 2, pulsar version 2.4.0.
Ubuntu 18.04.2 LTS (Bionic Beaver) running under Virtual Box in Windows 10
import pulsar
client = pulsar.Client('pulsar://localhost:6650')
producer = client.create_producer('my-topic')
producer.send("testing")
client.close()
2019-07-16 18:19:47.192 INFO ConnectionPool:72 | Created connection for pulsar://localhost:6650
2019-07-16 18:19:47.193 ERROR ClientConnection:360 | [ -> pulsar://localhost:6650] Failed to establish connection: Connection refused
2019-07-16 18:19:47.193 INFO ClientConnection:1297 | [ -> pulsar://localhost:6650] Connection closed
2019-07-16 18:19:47.193 ERROR ClientImpl:182 | Error Checking/Getting Partition Metadata while creating producer on persistent://public/default/my-topic -- 5
2019-07-16 18:19:47.193 INFO ClientImpl:482 | Closing Pulsar client
Traceback (most recent call last):
producer = client.create_producer('my-topic')
File "/usr/local/lib/python2.7/dist-packages/pulsar/init.py", line 476, in create_producer
p._producer = self._client.create_producer(topic, conf)
Exception: Pulsar error: ConnectError
2019-07-16 18:19:47.200 INFO ClientConnection:222 | [ -> pulsar://localhost:6650] Destroyed connection
changing 'pulsar://localhost:6650' -> 'http://<local_ip>:6650' doesn't work either.
I am really confused with this library.

hi, is the issue above solved? i am confused with that error "Error Checking/Getting Partition Metadata while creating producer" too @gurgen2727

authors: MrThreepwood a1onzo gurgen2727 ivankelly massakam sijie

repo: apache/pulsar


text:
[PIP-124] Create init subscription before sending message to DLQ

Fix #13408
Motivation
If we enable the DLQ when consuming messages. For some messages that can't be processed successfully, the messages will be moved to the DLQ, but if we do not specify the data retention for the namespace or create a subscription for the DLQ to retain the data, the data of the DLQ will be removed automatically. Therefore, we need to create the initial subscription before sending messages to the DLQ.
Modifications
When the deadLetterProducer is initialized, the consumer will set the initial subscription of the deadLetterProducer according to the DeadLetterPolicy.
When the broker creates a producer(after receiving the CommandProducer), if it finds that the topic does not exist, it will not only create the topic(if the topic automatically creation is enabled) but also create the initial subscription(if the initial subscription name is set). When creating an initial subscription, the user needs to have the canConsume permission and the allowAutoSubscriptionCreation needs to be enabled.
Verifying this change
This change is already covered by existing tests, such as
testPermissionForProducerCreateInitialSubscription
testDeadLetterTopicWithInitialSubscription

testDeadLetterTopicWithInitialSubscriptionAndMultiConsumers* and so on.

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (yes)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?

[x] doc-required

@merlimat Thanks for your review. I have created a proposal for this fix: https://github.com/apache/pulsar/issues/13408. PTAL

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

What was the purpose of commit 0f202b5a0ca690041451f2c57bf68205fe607c2b "Make initialSubscriptionName internal"?

What was the purpose of commit 0f202b5 "Make initialSubscriptionName internal"?

Exposing the initialSubscriptionName property to the producer is a bit wired. From the user's perspective, the producer should not care about the subscription. So I changed it to internal.

authors: RobertIndie hangc0276 lhotari


text:
[C++] PIP 37: Support large message size

Motivation
This PR is a C++ catch up of https://github.com/apache/pulsar/pull/4400, which implements PIP 37.
Modifications
Changes of the interfaces:

Add the chunkingEnabled config to producer so that producer can enable chunking messages by enabling chunkingEnabled and disabling batchingEnabled.
Add the maxPendingChunkedMessage and autoOldestChunkedMessageOnQueueFull configs to consumer to control the behavior to handle chunks.

Implementations of producer side:

Refactor the sendAsync method to follow the similar logic of Java client. If canAddToBatch returns false and the message size exceeds the limit, split the large message into chunks and send them one by one.
Fix the bug of Commands::newSend that it didn't use readableBytes() as the buffer's size, which could lead to a wrong check sum result when sending chunks because each chunk is a reference of the original large buffer and only the reader index is different.

Implementations of consumer side:

Add a MapCache class to wrap a hash map with a double ended queue that records the UUIDs in order of time. Because when we removes an UUID, we need to remove the related cache from both the map and the queue. This class only exposes the necessary methods to avoid one of the two data structures is not cleaned. In addition, it makes test easier to verify the cache is cleared after chunks are merged.
For chunks, call processMessageChunk to cache these chunks and merge the chunks with the same UUID into the completed message once the last chunks arrived.

Tests:

Configure maxMessageSize=10240 in test-conf/standalone-ssl.conf so that we don't need to create a too large message in tests.
Add MapCacheTest to verify the public methods of MapCache.
Add MessageChunkingTest to verify the basic end to end case with all compression types.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API: (yes)
The schema: (yes / no / don't know)
The default values of configurations: (yes / no)
The wire protocol: (yes / no)
The rest endpoints: (yes / no)
The admin cli options: (yes / no)
Anything that affects deployment: (yes / no / don't know)

It looks like some code is not compatible with the test env. I'll fix it soon.

Hi @merlimat , could you review this PR?

ping @merlimat Please help review this PR.

Hi @momo-jun could you please take a look at this PR? Similar to the Java client, do we need to add chunking docs for C++ client? Thanks

Hi @momo-jun could you please take a look at this PR? Similar to the Java client, do we need to add chunking docs for C++ client? Thanks

Sure. Will look into this.

authors: Anonymitaet BewareMyPower codelipenghui momo-jun


text:
[Doc] [Do-not-Merge] TableView java client: Add a new section and change code example

Modifications

Add a new section "Registering listeners"
Change code example for "Configure TableView"

Preview

Documentation

[x] doc

Hi @nlu90 @Anonymitaet, can you please review this doc PR? Thank you.


text:
doc: en to zh-cn

pageId: sql-getting-started
English to Chinese

cc @momo-jun @DaveDuggins @D-2-Ed

@AdamTyn Hi, welcome to contribute your Chinese document translation in Crowdin. You can take this document as a reference.

This PR can be closed now.

authors: AdamTyn Anonymitaet tuhaihe


text:
Update swagger documentation for query param: authoritative

Motivation
Applies the changes from https://github.com/apache/pulsar/pull/16222 to the generated swagger for older versions. Since this is only updating a comment, I don't think this presents any issues for backwards compatibility. However, if something thinks it does, please let me know.
Modifications

Replace Is authentication required to perform this operation with Whether leader broker redirected this call to this broker. For internal use.
Update several of the isGlobal blocks so that we remove the authoritative description and put the correct description in the authoritative block.

Verifying this change
This is a trivial rework of the swagger docs.
Does this pull request potentially affect one of the following parts:
This updates the already generated swagger. That could be an issue, but I don't think it will be.
Documentation

[x] doc

After thinking about this a bit more, I don't think we should update the existing swagger documentation since it is generated from the source code for the specific tag.


text:
[feature][compact] Enhanced topic compression

The ospp project.
Pulsar hierarchical storage is used to compress the data.

@azhsmesos Please check the details in "Pulsar CI / Preconditions" to trigger CI in your fork.

@azhsmesos Please check the details in "Pulsar CI / Preconditions" to trigger CI in your fork.

Please give me the relevant links for me to understand

@azhsmesos Thanks for the contribution.
Please check the relevant document and discussion blow
The PR template: https://raw.githubusercontent.com/apache/pulsar/master/.github/PULL_REQUEST_TEMPLATE.md
Mailing list discussions:
https://lists.apache.org/thread/gwfmxmxlhtsjn17sxxc367jcs4pcwofz
https://lists.apache.org/thread/p7rb04vf1mt0kk3v2r7xl9dvb3zkhtxf

Since we will start the RC version of 3.0.0 on 2023-04-11, I will change the label/milestone of PR who have not been merged.

The PR of type feature is deferred to 3.1.0
The PR of type fix is deferred to 3.0.1

So drag this PR to 3.1.0

authors: Jason918 azhsmesos codelipenghui poorbarcode


text:
[improve][build] Use slf4j-bom to align slf4j versions

Motivation
slf4j 2.0.x comes with slf4j-bom which can be used to align slf4j library versions.
Modifications
use slf4j-bom to align slf4j versions
Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 72.70%. Comparing base (bbc6224) to head (75b66e1).
Report is 218 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #22646      +/-   ##
============================================
- Coverage     73.57%   72.70%   -0.88%
+ Complexity    32624    32554      -70
============================================
  Files          1877     1887      +10
  Lines        139502   141006    +1504
  Branches      15299    15478     +179
============================================
- Hits         102638   102516     -122
- Misses        28908    30618    +1710
+ Partials       7956     7872      -84

Flag
Coverage Δ

inttests
27.39% <ø> (+2.80%)
:arrow_up:

systests
24.58% <ø> (+0.26%)
:arrow_up:

unittests
71.47% <ø> (-1.38%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.
see 344 files with indirect coverage changes

authors: codecov-commenter lhotari


text:
[fix][test] Fix running ClusterMetadataSetupTest in IDE

Motivation
Running ClusterMetadataSetupTest fails in IntelliJ since the tests is invalid.
Modifications
Fix the test
Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 74.29%. Comparing base (bbc6224) to head (b192d7e).
Report is 693 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23492      +/-   ##
============================================
+ Coverage     73.57%   74.29%   +0.71%
- Complexity    32624    34384    +1760
============================================
  Files          1877     1943      +66
  Lines        139502   146974    +7472
  Branches      15299    16191     +892
============================================
+ Hits         102638   109191    +6553
- Misses        28908    29356     +448
- Partials       7956     8427     +471

Flag
Coverage Δ

inttests
27.31% <ø> (+2.73%)
:arrow_up:

systests
24.37% <ø> (+0.05%)
:arrow_up:

unittests
73.66% <ø> (+0.82%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.
see 653 files with indirect coverage changes


text:
[Issue 5676][C++ client] Expose redelivery count

Fixes #5676
Motivation
Expose the redelivery count in the C++ client.
Modifications
Exposed the redelivery count from the broker in Message and MessageImpl. Set the counter when receiving messages. Added test.
Verifying this change

[ ] Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

Added test to check the counter increases on NAck

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): no
The public API: no
The schema: no
The default values of configurations: no
The wire protocol: no
The rest endpoints: no
The admin cli options: no
Anything that affects deployment: don't know

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? not documented yet
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

retest this please

run java8 tests
run cpp tests

@merlimat is there a way to debug why Jenkins is breaking at testPartitionedReceiveAsyncFailedConsumer? it works on my machine ™.

run cpp tests

is there a way to debug why Jenkins is breaking at testPartitionedReceiveAsyncFailedConsumer? it works on my machine ™.

Running the tests inside a Docker container (especially from MacOS) is a good way to slow it down make it similar to ASF Jenkins environment: pulsar-client-cpp/docker-tests.sh

@frejonb ^^

run cpp tests

@merlimat it looks like Jenkins is failing in different tests in the BasicEndToEnd suite. I tried running the tests in docker but I'm getting
➜  pulsar-client-cpp/docker-tests.sh
---- Testing Pulsar C++ client using image apachepulsar/pulsar-build:ubuntu-16.04 (type --help for more options)
ubuntu-16.04: Pulling from apachepulsar/pulsar-build
Digest: sha256:d03f66e312923a5e84936d92cfa0af6c95977e5cd4ecd4edb42aa808d543eeab
Status: Image is up to date for apachepulsar/pulsar-build:ubuntu-16.04
doing start standalone ...
starting standalone, logging to /tmp/pulsar-test-dist/logs/pulsar-standalone-c0edc72f1b2c.log
[AppClassLoader@18b4aac2] info AspectJ Weaver Version 1.9.2 built on Wednesday Oct 24, 2018 at 15:43:33 GMT
[AppClassLoader@18b4aac2] info register classloader sun.misc.Launcher$AppClassLoader@18b4aac2
[AppClassLoader@18b4aac2] info using configuration file:/tmp/pulsar-test-dist/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.5.0-SNAPSHOT.jar!/META-INF/aop.xml
[AppClassLoader@18b4aac2] info using configuration file:/tmp/pulsar-test-dist/lib/org.apache.pulsar-pulsar-zookeeper-2.5.0-SNAPSHOT.jar!/META-INF/aop.xml
[AppClassLoader@18b4aac2] info register aspect org.apache.pulsar.broker.zookeeper.aspectj.ClientCnxnAspect
[AppClassLoader@18b4aac2] info register aspect org.apache.pulsar.zookeeper.FinalRequestProcessorAspect
[AppClassLoader@18b4aac2] info register aspect org.apache.pulsar.zookeeper.ZooKeeperServerAspect
-- Wait for Pulsar service to be ready
-- Pulsar service is ready -- Configure permissions
-- Ready to start tests
/pulsar/pulsar-client-cpp/tests /pulsar/pulsar-client-cpp
---- Run unit tests in parallel
Traceback (most recent call last):
  File "/gtest-parallel/gtest-parallel", line 18, in <module>
    sys.exit(gtest_parallel.main())
  File "/gtest-parallel/gtest_parallel.py", line 801, in main
    tasks = find_tests(binaries, additional_args, options, times)
  File "/gtest-parallel/gtest_parallel.py", line 582, in find_tests
    stderr=subprocess.STDOUT)
  File "/usr/lib/python2.7/subprocess.py", line 567, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

run cpp tests

run cpp tests.

run cpp tests

@sijie I had to explicitly set subscriptionRedeliveryTrackerEnabled=true in order to get the redelivery counts. Could this be causing the issues? sadly I cannot do local testing since the docker container doesn't seem to be working https://github.com/apache/pulsar/pull/5677#issuecomment-555940892

@sijie I managed to get docker tests running, I'll debug the problem.

run cpp tests

run integration tests

authors: codelipenghui frejonb jiazhai merlimat sijie


text:
Update the package management module parent version

Motivation
When the PR is waiting for merging, we upgrade the pulsar version
and get it merged into master. So we have a build failed now at
the master branch.

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks


text:
Consider whether to add query message by topic and tags

Consider whether to add a method that  query message by topic and tags in message query module.
FEATURE REQUEST
Consider whether to add a method that  query message by topic and tags in message query module.

Thanks for your request but we may not add this feature.
Message tags is designed for message filter Message filtering
If you want to query message by index, you should use key but not tag (a message can contains multiple keys)
Message msg = new Message("TopicTest",
    "TagA",
    "Hello world".getBytes(RemotingHelper.DEFAULT_CHARSET));
msg.setKeys(Arrays.asList("key1","key2"));   // you by query by both key1 and key2
SendResult sendResult = producer.send(msg);```

Hope this solved your problem.

authors: StyleTang joewee

repo: apache/rocketmq-dashboard


text:
Useless class file  MessageFilterImpl

see: MessageFilterImpl
It is a useless file, it can be removed.

hello @duhenglucky,
may i work on this one

@TakshakRamteke Welcome，go ahead

authors: TakshakRamteke duhenglucky ni-ze

repo: apache/rocketmq


text:
fix iterator.remove() bug

Brief Description
bugfix: CopyOnWriteArray#listIterator do not support remove action when iterating

Hi @EvanMi Could you create an issue related the pr？

Codecov Report
Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (9dec4cf) 43.12% compared to head (7bff5a7) 43.19%.
Report is 1 commits behind head on develop.

Files
Patch %
Lines

...ava/org/apache/rocketmq/store/MappedFileQueue.java
75.00%
0 Missing and 1 partial :warning:

Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #7682      +/-   ##
=============================================
+ Coverage      43.12%   43.19%   +0.07%
- Complexity      9793     9806      +13
=============================================
  Files           1165     1165
  Lines          84482    84485       +3
  Branches       10968    10969       +1
=============================================
+ Hits           36429    36492      +63
+ Misses         43523    43465      -58
+ Partials        4530     4528       -2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Hi @EvanMi Could you create an issue related the pr？

see https://github.com/apache/rocketmq/issues/7684

Hi @EvanMi Could you create an issue related the pr？

see https://github.com/apache/rocketmq/issues/7684

authors: EvanMi RongtongJin codecov-commenter


text:
[ISSUE #8365] add remoting client non-oneway updateConsumerOffset function

Which Issue(s) This PR Fixes
Enhancement https://github.com/apache/rocketmq/issues/8365
Brief Description
add remoting client non-oneway updateConsumerOffset function
How Did You Test This Change?
add some new unit test

Codecov Report
Attention: Patch coverage is 14.28571% with 18 lines in your changes missing coverage. Please review.

Project coverage is 44.31%. Comparing base (6f77f05) to head (072bdb1).

Files
Patch %
Lines

...he/rocketmq/proxy/processor/ConsumerProcessor.java
12.50%
14 Missing :warning:

...q/proxy/service/message/ClusterMessageService.java
0.00%
2 Missing :warning:

...tmq/proxy/processor/DefaultMessagingProcessor.java
50.00%
1 Missing :warning:

...tmq/proxy/service/message/LocalMessageService.java
0.00%
1 Missing :warning:

Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #8391      +/-   ##
=============================================
- Coverage      44.36%   44.31%   -0.05%
+ Complexity     10782    10769      -13
=============================================
  Files           1274     1274
  Lines          88951    88968      +17
  Branches       11433    11433
=============================================
- Hits           39463    39427      -36
- Misses         44535    44577      +42
- Partials        4953     4964      +11

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


text:
dubbo demo运行问题

1、jar冲突：log4j的两个桥接包冲突，guava16.1和20.0版本冲突，多个版本的netty冲突
2、dubbo2.6.1存在服务注册不到zookeeper上的bug

@endy0114 你的问题解决了吗？
版本冲突可能是你的工程的问题，我们在集成测试的过程中没有发现这样的问题。

repo: apache/servicecomb-pack


text:
[Bug]

Search before asking

[X] I had searched in the issues and found no similar issues.

Apache SkyWalking Component
OAP server (apache/skywalking)
What happened
I have used node_exporter in my backend node js service for metrics. Its reporting CPU properly but not memory. I am on mac m1 system.
What you expected to happen
It should report memory usage as well. Its always reporting 0.
How to reproduce
Install node_exporter on localhost. Create a express app. Install skywalking backend nodejs agent. Configure otel-collector and run that along woth OAP server.
In native UI you should be able to see CPU usage but not memory usage in Linux root dashboard
Anything else
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

This issue is more seems to be realted to prometheus node exporter than to skywalking, so clong this one

repo: apache/skywalking


text:
how to push tracing data to skywalking with istio

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ✅] Question or discussion
[ ] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question
how to push tracing data to skywalking?
java: 1.8.0_212
os: centos7.6
skywalking: 8.1
istio config:
    accessLogFile: /dev/stdout
    defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      envoyAccessLogService:
        address: skywalking-oap.log.svc.cluster.local:11800
      envoyMetricsService:
        address: skywalking-oap.log.svc.cluster.local:11800
      proxyMetadata: {}
      tracing:
        zipkin:
          address: zipkin.istio-system:9411
    enableEnvoyAccessLogService: true
    enablePrometheusMerge: true
    rootNamespace: istio-system
    trustDomain: cluster.local
  meshNetworks: 'networks: {}'

skywalking  config:
helm install my-release skywalking -f skywalking/values.yaml --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh --set fullnameOverride=skywalking --set ui.image.tag=8.1.0 --set oap.image.tag=8.1.0-es6 --set oap.storageType=elasticsearch  --set oap.envoy.als.enabled=true  -n log

i queried with istio example(http://ingressgateway:port/product), but thers is no data in skywalking

Istio doesn't support it yet. The Envoy tracer is there.
API has been submitted by @Shikugawa  through https://github.com/istio/api/pull/1909

@dbaicss Yes. I'm working on Istio and SkyWalking integration right now.

authors: Shikugawa dbaicss wu-sheng


text:
[Bug] [Nodejs] install plugin error if mongodb driver is 4.1.3

Search before asking

[X] I had searched in the issues and found no similar issues.

Apache SkyWalking Component
NodeJS Server Side Agent (apache/skywalking-nodejs)
What happened
In file lib/plgins/MongoDBPlugin.js there is a line contain this code :

this.Cursor = installer.require('mongodb/lib/cursor');

this will cause en error : Uncaught Error Error: Cannot find module 'mongodb/lib/cursor' if version of mongodb driver is ^4.1.3. But it is fine if mongodb driver is 3.

What you expected to happen
should support high version Mongodb driver
How to reproduce
use skywalking-backend-js and version of mongodb is ^4.1.3.
Anything else
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

this will cause en error : Uncaught Error Error: Cannot find module 'mongodb/lib/cursor' if version of mongodb driver is ^4.1.3. But it is fine if mongodb driver is 3.

This means we don't support mongodb 4.x, please consider contributing if you need.

yarn add skywalking-backend-js-zy


text:
ResourceDectorator and Filter (to capture the requesting path)

Initial PR to make it easier to see the changes

With the upcoming migration to git this repository will be deleted. If the pull request contains data that should be re-submitted I suggest that you save the changes locally and resubmit them after the migration, which will likely happen on Monday the 9th of October.

authors: heervisscher rombert

repo: apache/sling


text:
Update the default solr.xml to include maxBooleanClauses

The default solr.xml does not contain a default entry for maxBooleanClauses.
This prevents the value to be set via javaOpts and forces uses to implement a custom config map with a custom solr.xml;
Adding <int name="maxBooleanClauses">${solr.max.booleanClauses:1024}</int> to the const DefaultSolrXML would not change the default values but would allow a simple override via the javaOpts
basically the change is solr_util.go
const DefaultSolrXML = `<?xml version="1.0" encoding="UTF-8" ?>
<solr>
  %s
  <int name="maxBooleanClauses">${solr.max.booleanClauses:1024}</int>
  <solrcloud>
    <str name="host">${host:}</str>
    <int name="hostPort">${hostPort:80}</int>
    <str name="hostContext">${hostContext:solr}</str>
    <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>
    <int name="zkClientTimeout">${zkClientTimeout:30000}</int>
    <int name="distribUpdateSoTimeout">${distribUpdateSoTimeout:600000}</int>
    <int name="distribUpdateConnTimeout">${distribUpdateConnTimeout:60000}</int>
    <str name="zkCredentialsProvider">${zkCredentialsProvider:org.apache.solr.common.cloud.DefaultZkCredentialsProvider}</str>
    <str name="zkACLProvider">${zkACLProvider:org.apache.solr.common.cloud.DefaultZkACLProvider}</str>
  </solrcloud>
  <shardHandlerFactory name="shardHandlerFactory"
    class="HttpShardHandlerFactory">
    <int name="socketTimeout">${socketTimeout:600000}</int>
    <int name="connTimeout">${connTimeout:60000}</int>
  </shardHandlerFactory>
  %s
</solr>
`

Probably a future version of the operator won't need to maintain a default solr.xml, if it can require a minimum solr version of 9.0. Since Solr v9.0 Solr will use a built-in default if no solr.xml is present in SOLR_HOME.

The Solr Operator currently supports Solr 7.7 (and 8.0), which do not have this option unfortunately, so we cannot add it by default.
We will soon release the next version of the Solr Operator and then we can increase the minimum version of Solr to 8.x (for some version of x), and this can be included in the next release. I'll get the ball rolling on this.

Since Solr v9.0 Solr will use a built-in default if no solr.xml is present in SOLR_HOME.

I am unbelievably excited for this when we have Solr 9.0 as a minimum version.

issue is solved in https://github.com/apache/solr-operator/issues/630

authors: HoustonPutman coolstim janhoy

repo: apache/solr-operator


text:
Added support in solrj client to add query fields/params through java 8 consumer

https://issues.apache.org/jira/browse/SOLR-XXXXX

Description
Added support in solrj client to add query fields/params through java 8 consumer. I have added new method in solr query class which will take solr query consumer as parameter and return the solr query in order to make it easy for chaining.
Solution
Please provide a short description of the approach taken to implement your solution.
Tests
Please describe the tests you've developed or run to confirm this patch implements the feature or solves the problem.
Checklist
Please review the following and check all that apply:

[ ] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
[ ] I have created a Jira issue and added the issue ID to my pull request title.
[ ] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
[ ] I have developed this patch against the main branch.
[ ] I have run ./gradlew check.
[ ] I have added tests for my changes.
[ ] I have added documentation for the Reference Guide

Please create a JIRA and fill in the PR template. Also, you have to explain the feature you are proposing much more in detail. Give examples of what is difficult to achieve today, then explain what a Consumer is and how to use it and why you think it justifies expanding the API. Then there may be some discussion in JIRA or here in the PR.
If there is agreement that this is useful, you'd need to also add CHANGES.txt entry and Reference Guide documentation (if applicable)

Any followup on this?

authors: janhoy venkatrockz

repo: apache/solr


text:
[SPARK-13621] [CORE] TestExecutor.scala needs to be moved to test package

Moved TestExecutor.scala from src to test package and removed the unused file TestClient.scala.

Can one of the admins verify this patch?

Jenkins test this please

Test build #52328 has started for PR 11474 at commit 894638b.

Test build #52328 has finished for PR 11474 at commit 894638b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52328/
Test PASSed.

Thanks - I've merged this in master.

@devaraj-kavali next time please use the pull request template.

authors: AmplabJenkins SparkQA devaraj-kavali rxin srowen

repo: apache/spark


text:
New exprcode

What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

(created accidentally; closing)

Test build #56153 has started for PR 12482 at commit bae08b1.

Test build #56153 has finished for PR 12482 at commit bae08b1.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56153/
Test FAILed.

Merged build finished. Test FAILed.

authors: AmplabJenkins SparkQA sameeragarwal


text:
[SPARK-14884][SQL][Streaming][WebUI] Fix call site for continuous queries

What changes were proposed in this pull request?
Since we've been processing continuous queries in separate threads, the call sites are then run at <unknown>:0. It's not wrong but provides very little information; in addition, we can not distinguish two queries only from their call sites.
This patch fixes this.
Before
[Jobs Tab]

[SQL Tab]

After
[Jobs Tab]

[SQL Tab]

How was this patch tested?
Manually checks - see screenshots above.

Test build #56839 has started for PR 12650 at commit 44d93c7.

Test build #56839 has finished for PR 12650 at commit 44d93c7.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56839/
Test FAILed.

Test build #56842 has started for PR 12650 at commit 70c3f4e.

Test build #56842 has finished for PR 12650 at commit 70c3f4e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56842/
Test PASSed.

@andrewor14 @zsxwing would you mind taking a look? Thanks! :-)

Test build #56975 has started for PR 12650 at commit 061237c.

Test build #56975 has finished for PR 12650 at commit 061237c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56975/
Test PASSed.

Merged build finished. Test PASSed.

Test build #57586 has started for PR 12650 at commit f7d8460.

@zsxwing I've made updates per your comments; would you take a another look? Thanks!

Test build #57586 has finished for PR 12650 at commit f7d8460.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57586/
Test PASSed.

LGTM. Merging to master and 2.0, thanks!

authors: AmplabJenkins SparkQA lw-lin zsxwing


text:
[SPARK-25691][SQL] Use semantic equality in Optimizer in order to compare attributes

What changes were proposed in this pull request?
When we compare attributes, in general, we should always refer to semantic equality, as the default equal method can return false when there are "cosmetic" differences between them, but still they are the same thing; at least we have to consider them so when analyzing/optimizing queries.
The PR focuses on the usage and comparison of the output of a LogicalPlan, which is a Seq[Attribute]. In this case, using equality implicitly fails to check the semantic equality. In analyzer and optimizer, the occurrences of such a comparison have been replaced with proper comparison using semantic equality.
How was this patch tested?
added UT + running the tests with the patch provided by @maryannxue in https://github.com/apache/spark/pull/22060

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3947/
Test PASSed.

Could we just hold this PR until we can ensure the expression ID are unique in the whole tree?

@gatorsmile thanks for your comment. Honestly I don't see much value in holding, as we are (nearly always) using semantic equality throughout the Analyzer/Optimizer. So different expressions with the same ID are already causing troubles. Here we are changing/introducing no news about that. Usually, instead, not using semantic equality results in a bug (that's also the reason why we have AttributeSet for instance).
I agree on enforcing that exprIds are unique on the plan, otherwise we do can have issues, but we would already have regardless of this PR.
cc also @cloud-fan for his thoughts.

Test build #97344 has finished for PR 22713 at commit 87fc62a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97344/
Test PASSed.

Merged build finished. Test PASSed.

I do not want to take any risk unless you can have a test case to show the benefit of each change.

Thanks @gatorsmile, I agree. Then I reduced the scope to only the analyzer rules and added the UT for the only rule changed (other than the AliasViewChild one which is tested with the code provided by @maryannxue).
I removed the changes in the Analyzer, because those cases are hard to test, in the sense that I have not found operators causing problems for them, as they rely on working on operators which bring the very same output (eg. in one case the very same plan is used, in the other all the operators I have seen just do def output = child.output, so the output is really the same). So despite I think the changes there should be done, I am ok leaving them out of scope here as they are of no harm.
Thanks.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3960/
Test FAILed.

Test build #97361 has started for PR 22713 at commit 64aafc5.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3962/
Test PASSed.

Test build #97361 has finished for PR 22713 at commit 64aafc5.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
sealed class BaseSimpleAnalyzer(caseSensitive: Boolean) extends Analyzer(

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97361/
Test PASSed.

Test build #97380 has started for PR 22713 at commit c4aaa8d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3973/
Test PASSed.

We do need a test case here anyway. Ideally it would be just as simple as #22701 but the difficulty is in declaring a view.

Test build #97380 has finished for PR 22713 at commit c4aaa8d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97380/
Test PASSed.

Test build #97450 has started for PR 22713 at commit 0d334e3.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4032/
Test PASSed.

Test build #97450 has finished for PR 22713 at commit 0d334e3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97450/
Test PASSed.

anymore comments on this @cloud-fan @maryannxue @viirya ?

LGTM. To confirm, this is a potential bug, currently end-users can't hit it, right?

LGTM

@cloud-fan I would answer yes. At least, I haven't found a real case when this can cause a bug visible to end users, but I am not 100% that there is no weird case where someone could hit it.

Test build #98268 has started for PR 22713 at commit 0e49e91.

Merged build finished. Test PASSed.

Test build #98268 has finished for PR 22713 at commit 0e49e91.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98268/
Test PASSed.

thanks, merging to master!

authors: AmplabJenkins SparkQA cloud-fan gatorsmile maryannxue mgaido91 viirya


text:
[SPARK-29928][SQL][TESTS] Check parsing timestamps up to microsecond precision by JSON/CSV datasource

What changes were proposed in this pull request?
In the PR, I propose to add tests from the commit https://github.com/apache/spark/commit/9c7e8be1dca8285296f3052c41f35043699d7d10 for Spark 2.4 that check parsing of timestamp strings for various seconds fractions.
Why are the changes needed?
To make sure that current behavior is the same as in Spark 2.4
Does this PR introduce any user-facing change?
No
How was this patch tested?
By running CSVSuite, JsonFunctionsSuite and TimestampFormatterSuite.

@cloud-fan @dongjoon-hyun May I ask you to review this PR.

Test build #113940 has started for PR 26558 at commit a692f66.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18808/
Test PASSed.

Test build #113940 has finished for PR 26558 at commit a692f66.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113940/
Test PASSed.

authors: AmplabJenkins MaxGekk SparkQA


text:
[SPARK-32998] Add ability to override default remote repos with inter…

What changes were proposed in this pull request?

Building spark internally in orgs where access to outside internet is not allowed takes a long time because unsuccessful attempts are made to download artifacts from repositories which are not accessible. The unsuccessful attempts unnecessarily add significant amount of time to the build. I have seen a difference of up-to 1hr for some runs.
Adding 3 environment variables that should be present that the start of the build and if they exist, override the default repos defined in the code and scripts.
envVariables:
- TYPESAFE_SBT_MIRROR=https://artifacts.internal.com/typesafe-ivy-releases-cache
- TYPESAFE_MIRROR=https://artifacts.internal.com/lightbend-dist
- INTERNAL_ARTIFACTORY=https://artifacts.internal.com/libs-release/

Why are the changes needed?

To allow orgs to build spark internally without relying on external repositories for artifact downloads.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Multiple builds with and without env variables set.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

okay to test.

ok to test

Test build #129146 has started for PR 29874 at commit de4c4d7.

Test build #129146 has finished for PR 29874 at commit de4c4d7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129146/
Test FAILed.

Test build #129369 has started for PR 29874 at commit 1b0ba28.

Test build #129369 has finished for PR 29874 at commit 1b0ba28.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129369/
Test FAILed.

Test build #129371 has started for PR 29874 at commit 12a06c0.

Test build #129371 has finished for PR 29874 at commit 12a06c0.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129371/
Test FAILed.

Test build #129478 has started for PR 29874 at commit fba96a5.

Test build #129478 has finished for PR 29874 at commit fba96a5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129478/
Test FAILed.

Hi, @ankits . The master branch is upgraded to SBT 1.3.13.
Could you resolve the conflicts?

Test build #129590 has started for PR 29874 at commit 8da6fd5.

Test build #129590 has finished for PR 29874 at commit 8da6fd5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129590/
Test FAILed.

Retest this please

Unfortunately, this PR doesn't pass both Jenkins and Github Action until now.
@ankits . Is this tested in your environment with unit tests and integration tests before?

Test build #129859 has started for PR 29874 at commit 8da6fd5.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34465/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34465/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34465/
Test FAILed.

Test build #129859 has finished for PR 29874 at commit 8da6fd5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129859/
Test FAILed.

Test build #130023 has started for PR 29874 at commit 11a9a67.

Test build #130023 has finished for PR 29874 at commit 11a9a67.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130023/
Test FAILed.

Test build #130028 has started for PR 29874 at commit ff644eb.

Test build #130028 has finished for PR 29874 at commit ff644eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130028/
Test PASSed.

Retest this please.

Test build #130108 has started for PR 29874 at commit ff644eb.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34717/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34717/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34717/
Test PASSed.

Test build #130108 has finished for PR 29874 at commit ff644eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130108/
Test PASSed.

I tested master internally just now and didn't need to update pom.xml or project/SparkBuild.scala.

Test build #130171 has started for PR 29874 at commit b58631c.

Test build #130171 has finished for PR 29874 at commit b58631c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130171/
Test PASSed.

Test build #130177 has started for PR 29874 at commit 5a90eb7.

Test build #130178 has started for PR 29874 at commit 91d0d4a.

Test build #130179 has started for PR 29874 at commit f0ee8b8.

Test build #130180 has started for PR 29874 at commit cfb86c3.

Thank you, @ankits and @srowen .
Merged to master for Apache Spark 3.1.0.

Thank you for your first contribution ,@ankits.
You are added to the Apache Spark contributor group and SPARK-32998 is assigned to you.
Welcome to the Apache Spark community!

Test build #130177 has finished for PR 29874 at commit 5a90eb7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130177/
Test PASSed.

Test build #130178 has finished for PR 29874 at commit 91d0d4a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130178/
Test PASSed.

Test build #130179 has finished for PR 29874 at commit f0ee8b8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130179/
Test PASSed.

Test build #130180 has finished for PR 29874 at commit cfb86c3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Building spark internally ...

Do we need to change pom.xml and project/SparkBuild.scala?

I tested master internally just now and didn't need to update pom.xml or project/SparkBuild.scala.

I could not follow why we don't need to update pom.xml and project/SparkBuild.scala. Isn't this PR to provide an option for a mirror that should be used during building? Or does this PR target to set a mirror to use during runtime e.g. Spark summit and configuration?

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130180/
Test PASSed.

For maven we can setup  a settings.xml in $M2_HOME that points to internal artifact repository. Similar setup can be done for sbt where you can define the -Dsbt.repository.config to fetch artifacts from the internal artifact repository.
The main purpose of the PR was to address test execution and download of artifacts which wasn't supported with Dsbt.repository.config.

Ah, right. That makes sense.

authors: AmplabJenkins HyukjinKwon SparkQA ankits dbtsai dongjoon-hyun


text:
Quote arguments with special characters (e.g. "&") on Windows

This fixes issue #35124. Note that it is quite unlikely that this issue would be mitigated or fixed in the Python standard library itself code due to the history of "subprocess.list2cmdline" (see https://bugs.python.org/issue8972 which was created in 2010).

What changes were proposed in this pull request?
This adds support for a number of special characters (such as "&") appearing in Spark configuration values (common in cloud resource URIs).

Why are the changes needed?
If one of the relevant characters appears in a configuration value, the Windows submit scripts do not work correctly, interpreting the character literally. For example, an ampersand marks the beginning of a new command (leading Windows to actually start multiple processes which would typically fail with a rather strange and unexpected error message).

Does this PR introduce any user-facing change?
No.

How was this patch tested?
This was tested manually, providing a configuration value with a "&" character in it.

Jenkins ok to test

Test build #140340 has started for PR 32237 at commit 5ed9037.

Test build #140340 has finished for PR 32237 at commit 5ed9037.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140340/

@holdenk I'm not really a Windows user either so I had to piece together a solution based on snippets and assorted help pages from Microsoft – really I just ended up experimenting to get it right.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44871/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44871/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44871/

authors: AmplabJenkins SparkQA holdenk malthe


text:
[SPARK-35618][SQL] Resolve star expressions in subqueries using outer query plans

What changes were proposed in this pull request?
This PR supports resolving star expressions in subqueries using outer query plans.
Why are the changes needed?
Currently, Spark can only resolve star expressions using the inner query plan when resolving subqueries. Instead, it should also be able to resolve star expressions using the outer query plans.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Unit tests

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43878/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43878/

Test build #139356 has finished for PR 32787 at commit 6150cc4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139356/

Test build #139437 has started for PR 32787 at commit 05b0613.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43960/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43960/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43960/

Test build #139437 has finished for PR 32787 at commit 05b0613.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139437/

cc @cloud-fan

Test build #139601 has started for PR 32787 at commit 1704a9a.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44128/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44128/

Test build #139601 has finished for PR 32787 at commit 1704a9a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139601/

Test build #139855 has started for PR 32787 at commit e0460c5.

Test build #139855 has finished for PR 32787 at commit e0460c5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139855/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44384/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44384/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44384/

Test build #140221 has started for PR 32787 at commit 8f1f42a.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44749/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44749/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44749/

Test build #140221 has finished for PR 32787 at commit 8f1f42a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140221/

Test build #140339 has started for PR 32787 at commit b093b97.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44870/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44870/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44870/

Test build #140339 has finished for PR 32787 at commit b093b97.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140339/

Test build #140381 has started for PR 32787 at commit fe5f89a.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140381/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44904/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44904/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44904/

Looks fine otherwise except for the @cloud-fan comment.

Test build #140491 has started for PR 32787 at commit 0346d3b.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45003/

thanks, merging to master!

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45003/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45003/

Test build #140491 has finished for PR 32787 at commit 0346d3b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140491/

authors: AmplabJenkins SparkQA allisonwang-db cloud-fan maropu


text:
[SPARK-36379][SQL][3.1] Null at root level of a JSON array should not fail w/ permissive mode

This PR backports https://github.com/apache/spark/pull/33608 to branch-3.1

What changes were proposed in this pull request?
This PR proposes to fail properly so JSON parser can proceed and parse the input with the permissive mode.
Previously, we passed nulls as are, the root InternalRows became nulls, and it causes the query fails even with permissive mode on.
Now, we fail explicitly if null is passed when the input array contains null.
Note that this is consistent with non-array JSON input:
Permissive mode:
spark.read.json(Seq("""{"a": "str"}""", """null""").toDS).collect()

res0: Array[org.apache.spark.sql.Row] = Array([str], [null])

Failfast mode:
spark.read.option("mode", "failfast").json(Seq("""{"a": "str"}""", """null""").toDS).collect()

org.apache.spark.SparkException: Malformed records are detected in record parsing. Parse Mode: FAILFAST. To process malformed records as null result, try setting the option 'mode' as 'PERMISSIVE'.
	at org.apache.spark.sql.catalyst.util.FailureSafeParser.parse(FailureSafeParser.scala:70)
	at org.apache.spark.sql.DataFrameReader.$anonfun$json$7(DataFrameReader.scala:540)
	at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)

Why are the changes needed?
To make the permissive mode to proceed and parse without throwing an exception.
Does this PR introduce any user-facing change?
Permissive mode:
spark.read.json(Seq("""[{"a": "str"}, null]""").toDS).collect()

Before:
java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)

After:
res0: Array[org.apache.spark.sql.Row] = Array([null])

NOTE that this behaviour is consistent when JSON object is malformed:
spark.read.schema("a int").json(Seq("""[{"a": 123}, {123123}, {"a": 123}]""").toDS).collect()

res0: Array[org.apache.spark.sql.Row] = Array([null])

Since we're parsing one JSON array, related records all fail together.
Failfast mode:
spark.read.option("mode", "failfast").json(Seq("""[{"a": "str"}, null]""").toDS).collect()

Before:
java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)

After:
org.apache.spark.SparkException: Malformed records are detected in record parsing. Parse Mode: FAILFAST. To process malformed records as null result, try setting the option 'mode' as 'PERMISSIVE'.
	at org.apache.spark.sql.catalyst.util.FailureSafeParser.parse(FailureSafeParser.scala:70)
	at org.apache.spark.sql.DataFrameReader.$anonfun$json$7(DataFrameReader.scala:540)
	at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)

How was this patch tested?
Manually tested, and unit test was added.

cc @cloud-fan @dongjoon-hyun @MaxGekk FYI

Test build #142551 has started for PR 33762 at commit 41d8254.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47053/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47053/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47053/

Merged to branch-3.1.

Test build #142551 has finished for PR 33762 at commit 41d8254.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142551/

authors: AmplabJenkins HyukjinKwon SparkQA