text:
build failed on AArch64, Fedora 33

Apache Thrift 0.12.0 is required. Building it reports unsupported .NET, etc.
Installing 0.13.0 using yum results in an error on mvn package.
Reporter: Lutz Weischer
Note: This issue was originally created as PARQUET-2000. Please see the migration documentation for further details.

Lutz Weischer:
I restarted from scratch and get this:
LC_ALL=C mvn install
...
Testing PARQUET_1_0/FIXED_LEN_BYTE_ARRAY/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/FIXED_LEN_BYTE_ARRAY/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1,100.501 sec
Results :
Tests in error:
testInteropReadEncryptedParquetFiles(org.apache.parquet.hadoop.ITTestEncryptionOptions): github.com: Name or service not known
Tests run: 17, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO] — maven-failsafe-plugin:2.10:verify (default) @ parquet-hadoop —
[INFO] Failsafe report directory: /data/jw/parquet/parquet-hadoop/target/failsafe-reports
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Parquet MR 1.12.0-SNAPSHOT:
[INFO]
[INFO] Apache Parquet MR .................................. SUCCESS [  8.688 s]
[INFO] Apache Parquet Format Structures ................... SUCCESS [  8.891 s]
[INFO] Apache Parquet Generator ........................... SUCCESS [  3.289 s]
[INFO] Apache Parquet Common .............................. SUCCESS [ 10.041 s]
[INFO] Apache Parquet Encodings ........................... SUCCESS [ 33.835 s]
[INFO] Apache Parquet Column .............................. SUCCESS [01:32 min]
[INFO] Apache Parquet Arrow ............................... SUCCESS [  3.288 s]
[INFO] Apache Parquet Jackson ............................. SUCCESS [  4.596 s]
[INFO] Apache Parquet Hadoop .............................. FAILURE [22:59 min]
[INFO] Apache Parquet Avro ................................ SKIPPED
[INFO] Apache Parquet Benchmarks .......................... SKIPPED
[INFO] Apache Parquet Pig ................................. SKIPPED
[INFO] Apache Parquet Thrift .............................. SKIPPED
[INFO] Apache Parquet Cascading [Deprecated, will be removed after 1.12] SKIPPED
[INFO] Apache Parquet Cascading (for Cascading 3.0 onwards) [Deprecated, will be removed after 1.12] SKIPPED
[INFO] Apache Parquet Command-line ........................ SKIPPED
[INFO] Apache Parquet Pig Bundle .......................... SKIPPED
[INFO] Apache Parquet Protobuf ............................ SKIPPED
[INFO] Apache Parquet Scala ............................... SKIPPED
[INFO] Apache Parquet Scrooge [Deprecated, will be removed after 1.12] SKIPPED
[INFO] Apache Parquet Hadoop Bundle ....................... SKIPPED
[INFO] Apache Parquet Tools; [Deprecated, will be removed after 1.12; Use parquet-cli instead] SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  25:45 min
[INFO] Finished at: 2021-03-24T15:06:55+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.10:verify (default) on project parquet-hadoop: There are test failures.
[ERROR]
[ERROR] Please refer to /data/jw/parquet/parquet-hadoop/target/failsafe-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :parquet-hadoop
(base) [jw@cn06 parquet]$
I can access github.com, because I git clone'd from it.
(base) [jw@cn06 parquet]$ ls /data/jw/parquet/parquet-hadoop/target/failsafe-reports
TEST-org.apache.parquet.encodings.FileEncodingsIT.xml       org.apache.parquet.encodings.FileEncodingsIT.txt
TEST-org.apache.parquet.hadoop.ITTestEncryptionOptions.xml  org.apache.parquet.hadoop.ITTestEncryptionOptions.txt
failsafe-summary.xml
(base) [jw@cn06 parquet]$ nano /data/jw/parquet/parquet-hadoop/target/failsafe-reports/failsafe-summary.xml
shows

Gabor Szadovszky / @gszadovszky:
Could you try to get the file https://github.com/apache/parquet-testing/raw/40379b3/data/encrypt_columns_and_footer.parquet.encrypted manually (e.g. with wget or curl)?
It would also help if you could dig up the whole stack trace from failsafe-reports.
@andersonm-ibm, do you have an idea why it is happening and why we get the error message github.com: Name or service not known?

Lutz Weischer:
Switched to parquet-mr.
[jw@cn05 parquet]$ LC_ALL=C mvn install
...
[INFO]
[INFO] — maven-jar-plugin:2.4:jar (default-jar) @ parquet-hadoop —
[INFO] Building jar: /home/jw/apache/parquet/parquet-hadoop/target/parquet-hadoop-1.12.0-SNAPSHOT.jar
[INFO]
[INFO] — maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ parquet-hadoop —
[INFO]
[INFO] — maven-jar-plugin:2.4:test-jar (default) @ parquet-hadoop —
[INFO] Building jar: /home/jw/apache/parquet/parquet-hadoop/target/parquet-hadoop-1.12.0-SNAPSHOT-tests.jar
[INFO]
[INFO] — maven-shade-plugin:3.2.1:shade (default) @ parquet-hadoop —
[INFO] Excluding org.apache.parquet:parquet-column:jar:1.12.0-SNAPSHOT from the shaded jar.
[INFO] Excluding org.apache.parquet:parquet-common:jar:1.12.0-SNAPSHOT from the shaded jar.
[INFO] Excluding org.apache.yetus:audience-annotations:jar:0.12.0 from the shaded jar.
[INFO] Excluding org.apache.parquet:parquet-encoding:jar:1.12.0-SNAPSHOT from the shaded jar.
[INFO] Including net.openhft:zero-allocation-hashing:jar:0.9 in the shaded jar.
[INFO] Excluding org.apache.parquet:parquet-format-structures:jar:1.12.0-SNAPSHOT from the shaded jar.
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.22 from the shaded jar.
[INFO] Excluding javax.annotation:javax.annotation-api:jar:1.3.2 from the shaded jar.
[INFO] Excluding org.apache.parquet:parquet-jackson:jar:1.12.0-SNAPSHOT from the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.12.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.12.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.12.2 in the shaded jar.
[INFO] Excluding org.xerial.snappy:snappy-java:jar:1.1.8 from the shaded jar.
[INFO] Excluding commons-pool:commons-pool:jar:1.6 from the shaded jar.
[INFO] Including it.unimi.dsi:fastutil:jar:8.4.2 in the shaded jar.
[INFO] Excluding com.github.luben:zstd-jni:jar:1.4.9-1 from the shaded jar.
[INFO] Minimizing jar org.apache.parquet:parquet-hadoop:jar:1.12.0-SNAPSHOT
[INFO] Minimized 11128 -> 491 (4%)
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing /home/jw/apache/parquet/parquet-hadoop/target/parquet-hadoop-1.12.0-SNAPSHOT.jar with /home/jw/apache/parquet/parquet-hadoop/target/parquet-hadoop-1.12.0-SNAPSHOT-shaded.jar
[INFO] Dependency-reduced POM written at: /home/jw/apache/parquet/parquet-hadoop/dependency-reduced-pom.xml
[INFO]
[INFO] — maven-failsafe-plugin:2.10:integration-test (default) @ parquet-hadoop —
[INFO] Failsafe report directory: /home/jw/apache/parquet/parquet-hadoop/target/failsafe-reports

T E S T S
Running org.apache.parquet.hadoop.ITTestEncryptionOptions
[main] INFO org.apache.parquet.hadoop.TestEncryptionOptions - ======== testInteropReadEncryptedParquetFiles target/parquet-testing/data ========
[main] INFO org.apache.parquet.hadoop.TestEncryptionOptions - Download interop files if needed
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/home/jw/.m2/repository/org/apache/hadoop/hadoop-auth/2.10.1/hadoop-auth-2.10.1.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
10:20:52,298 DEBUG                         Tracer: 106 - sampler.classes = ; loaded no samplers
10:20:52,318 DEBUG                         Tracer: 128 - span.receiver.classes = ; loaded no span receivers
[main] INFO org.apache.parquet.hadoop.TestEncryptionOptions - target/parquet-testing/data exists?: true
[main] INFO org.apache.parquet.hadoop.TestEncryptionOptions - Download interop file: https://github.com/apache/parquet-testing/raw/40379b3/data/uniform_encryption.parquet.encrypted
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 8.554 sec <<< FAILURE!
Running org.apache.parquet.encodings.FileEncodingsIT
Testing codecs: [UNCOMPRESSED]
Testing PARQUET_1_0/BOOLEAN/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/BOOLEAN/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_1_0/BOOLEAN/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/BOOLEAN/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_1_0/INT32/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/INT32/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_1_0/INT32/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/INT32/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_1_0/INT64/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/INT64/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_1_0/INT64/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/INT64/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_1_0/INT96/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/INT96/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_1_0/INT96/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/INT96/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_1_0/FLOAT/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/FLOAT/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_1_0/FLOAT/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/FLOAT/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_1_0/DOUBLE/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/DOUBLE/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_1_0/DOUBLE/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/DOUBLE/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_1_0/BINARY/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/BINARY/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_1_0/BINARY/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/BINARY/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_1_0/FIXED_LEN_BYTE_ARRAY/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/FIXED_LEN_BYTE_ARRAY/UNCOMPRESSED encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_1_0/FIXED_LEN_BYTE_ARRAY/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Testing PARQUET_2_0/FIXED_LEN_BYTE_ARRAY/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1,108.336 sec
Results :
Tests in error:
testInteropReadEncryptedParquetFiles(org.apache.parquet.hadoop.ITTestEncryptionOptions): github.com: Name or service not known
Tests run: 17, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO] — maven-failsafe-plugin:2.10:verify (default) @ parquet-hadoop —
[INFO] Failsafe report directory: /home/jw/apache/parquet/parquet-hadoop/target/failsafe-reports
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Parquet MR 1.12.0-SNAPSHOT:
[INFO]
[INFO] Apache Parquet MR .................................. SUCCESS [ 10.510 s]
[INFO] Apache Parquet Format Structures ................... SUCCESS [  9.290 s]
[INFO] Apache Parquet Generator ........................... SUCCESS [  3.472 s]
[INFO] Apache Parquet Common .............................. SUCCESS [ 10.361 s]
[INFO] Apache Parquet Encodings ........................... SUCCESS [ 37.394 s]
[INFO] Apache Parquet Column .............................. SUCCESS [01:41 min]
[INFO] Apache Parquet Arrow ............................... SUCCESS [  3.497 s]
[INFO] Apache Parquet Jackson ............................. SUCCESS [  5.068 s]
[INFO] Apache Parquet Hadoop .............................. FAILURE [23:20 min]
[INFO] Apache Parquet Avro ................................ SKIPPED
[INFO] Apache Parquet Benchmarks .......................... SKIPPED
[INFO] Apache Parquet Pig ................................. SKIPPED
[INFO] Apache Parquet Thrift .............................. SKIPPED
[INFO] Apache Parquet Cascading [Deprecated, will be removed after 1.12] SKIPPED
[INFO] Apache Parquet Cascading (for Cascading 3.0 onwards) [Deprecated, will be removed after 1.12] SKIPPED
[INFO] Apache Parquet Command-line ........................ SKIPPED
[INFO] Apache Parquet Pig Bundle .......................... SKIPPED
[INFO] Apache Parquet Protobuf ............................ SKIPPED
[INFO] Apache Parquet Scala ............................... SKIPPED
[INFO] Apache Parquet Scrooge [Deprecated, will be removed after 1.12] SKIPPED
[INFO] Apache Parquet Hadoop Bundle ....................... SKIPPED
[INFO] Apache Parquet Tools; [Deprecated, will be removed after 1.12; Use parquet-cli instead] SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  26:21 min
[INFO] Finished at: 2021-03-22T10:39:29+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.10:verify (default) on project parquet-hadoop: There are test failures.
[ERROR]
[ERROR] Please refer to /home/jw/apache/parquet/parquet-hadoop/target/failsafe-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :parquet-hadoop
[jw@cn05 parquet]$

Gabor Szadovszky / @gszadovszky:
The current master release requires 0.13.0. The build would fail with any other version of the thrift binary. Could you please explain what mvn error do you see when you use 0.13.0?

Gabor Szadovszky / @gszadovszky:
The failing test tries to download a file from github directly. The exact link is listed in the log. It seems github.com is not accessible from your environment.

Lutz Weischer:
[jw@cn06 parquet]$ mvn package
[INFO] Scanning for projects...
[INFO]
[INFO] -----------------< org.apache.parquet:parquet-format >------------------
[INFO] Building Apache Parquet Format 2.9.0-SNAPSHOT
[INFO] -------------------------------[ jar ]--------------------------------
[INFO]
[INFO] — thrift-maven-plugin:0.10.0:compile (thrift-sources) @ parquet-format —
[INFO]
[INFO] — exec-maven-plugin:1.2.1:exec (check-thrift-version) @ parquet-format —
========== [FATAL] Build is configured to require Thrift version 0.12.0 ==========
========== Currently installed: Thrift version 0.13.0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.682 s
[INFO] Finished at: 2021-03-17T08:54:58+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (check-thrift-version) on project parquet-format: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[jw@cn06 parquet]$ thrift -version
Thrift version 0.13.0
[jw@cn06 parquet]$ mvn install -DskipTests
[INFO] Scanning for projects...
[INFO]
[INFO] -----------------< org.apache.parquet:parquet-format >------------------
[INFO] Building Apache Parquet Format 2.9.0-SNAPSHOT
[INFO] -------------------------------[ jar ]--------------------------------
[INFO]
[INFO] — thrift-maven-plugin:0.10.0:compile (thrift-sources) @ parquet-format —
[INFO]
[INFO] — exec-maven-plugin:1.2.1:exec (check-thrift-version) @ parquet-format —
========== [FATAL] Build is configured to require Thrift version 0.12.0 ==========
========== Currently installed: Thrift version 0.13.0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.428 s
[INFO] Finished at: 2021-03-17T08:56:36+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (check-thrift-version) on project parquet-format: Command execution failed.: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[jw@cn06 parquet]$

Gabor Szadovszky / @gszadovszky:
So, you are trying to build parquet-format (and not parquet-mr). parquet-format still configured for thrift 0.12.0 while it does not really means anything because the parquet-format thrift compile is only for testing purposes. The production code for parquet-format is generated inside parquet-mr (in the module parquet-format-structures).
Why are you trying to build parquet-format? If you are interested in the java implementation of parquet I would suggest looking into parquet-mr.

Lutz Weischer:
I got a fresh installation and have now this result:
(base) [jw@cn05 parquet]$ LC_ALL=C mvn install -DskipTests
...
[INFO] Apache Parquet Pig ................................. SUCCESS [  9.340 s]
[INFO] Apache Parquet Thrift .............................. SUCCESS [ 11.328 s]
[INFO] Apache Parquet Cascading [Deprecated, will be removed after 1.12] FAILURE [  1.496 s]
[INFO] Apache Parquet Cascading (for Cascading 3.0 onwards) [Deprecated, will be removed after 1.12] SKIPPED
[INFO] Apache Parquet Command-line ........................ SKIPPED
[INFO] Apache Parquet Pig Bundle .......................... SKIPPED
[INFO] Apache Parquet Protobuf ............................ SKIPPED
[INFO] Apache Parquet Scala ............................... SKIPPED
[INFO] Apache Parquet Scrooge [Deprecated, will be removed after 1.12] SKIPPED
[INFO] Apache Parquet Hadoop Bundle ....................... SKIPPED
[INFO] Apache Parquet Tools; [Deprecated, will be removed after 1.12; Use parquet-cli instead] SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:21 min
[INFO] Finished at: 2021-03-26T14:08:18+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.siom79.japicmp:japicmp-maven-plugin:0.14.2:cmp (default) on project parquet-cascading-deprecated: Could not resolve org.apache.parquet:parquet-cascading:jar:1.12.0 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :parquet-cascading-deprecated
(base) [jw@cn05 parquet]$
Trying differently:
(base) [jw@cn05 parquet]$ mvn install -DskipTests
...
[INFO] Apache Parquet Pig ................................. SUCCESS [  8.357 s]
[INFO] Apache Parquet Thrift .............................. SUCCESS [ 10.033 s]
[INFO] Apache Parquet Cascading [Deprecated, will be removed after 1.12] FAILURE [  0.686 s]
[INFO] Apache Parquet Cascading (for Cascading 3.0 onwards) [Deprecated, will be removed after 1.12] SKIPPED
[INFO] Apache Parquet Command-line ........................ SKIPPED
[INFO] Apache Parquet Pig Bundle .......................... SKIPPED
[INFO] Apache Parquet Protobuf ............................ SKIPPED
[INFO] Apache Parquet Scala ............................... SKIPPED
[INFO] Apache Parquet Scrooge [Deprecated, will be removed after 1.12] SKIPPED
[INFO] Apache Parquet Hadoop Bundle ....................... SKIPPED
[INFO] Apache Parquet Tools; [Deprecated, will be removed after 1.12; Use parquet-cli instead] SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:06 min
[INFO] Finished at: 2021-03-26T14:33:17+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.12:check (default) on project parquet-cascading-deprecated: Too many files with unapproved license: 1 See RAT report in: /data/jw/apache/metron/parquet/parquet-cascading-deprecated/target/rat.txt -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :parquet-cascading-deprecated
(base) [jw@cn05 parquet]$
And:
(base) [jw@cn05 parquet]$ mvn install -DskipTests -Drat.skip=true
...
[INFO] Apache Parquet Pig ................................. SUCCESS [  8.106 s]
[INFO] Apache Parquet Thrift .............................. SUCCESS [  9.405 s]
[INFO] Apache Parquet Cascading [Deprecated, will be removed after 1.12] FAILURE [  0.566 s]
[INFO] Apache Parquet Cascading (for Cascading 3.0 onwards) [Deprecated, will be removed after 1.12] SKIPPED
[INFO] Apache Parquet Command-line ........................ SKIPPED
[INFO] Apache Parquet Pig Bundle .......................... SKIPPED
[INFO] Apache Parquet Protobuf ............................ SKIPPED
[INFO] Apache Parquet Scala ............................... SKIPPED
[INFO] Apache Parquet Scrooge [Deprecated, will be removed after 1.12] SKIPPED
[INFO] Apache Parquet Hadoop Bundle ....................... SKIPPED
[INFO] Apache Parquet Tools; [Deprecated, will be removed after 1.12; Use parquet-cli instead] SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:03 min
[INFO] Finished at: 2021-03-26T14:42:35+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.siom79.japicmp:japicmp-maven-plugin:0.14.2:cmp (default) on project parquet-cascading-deprecated: Could not resolve org.apache.parquet:parquet-cascading:jar:1.12.0 -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :parquet-cascading-deprecated
(base) [jw@cn05 parquet]$

Gabor Szadovszky / @gszadovszky:
[~LutzWeischerFujitsu], this is the previous issue you've already posted. Please, check my [previous comment|#comment-17307872].

Lutz Weischer:
Looks like we've progress but we're not there yet, due to failing tests. Using Java 11, setting ANT_OPTS with proxy:
(base) [jw@cn06 parquet]$ LC_ALL=C mvn install
...
Testing PARQUET_2_0/FIXED_LEN_BYTE_ARRAY/UNCOMPRESSED + DICTIONARY encodings using ROW_GROUP_SIZE=131072 PAGE_SIZE=16384
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1,062.679 sec
Results :
Tests in error:
testInteropReadEncryptedParquetFiles(org.apache.parquet.hadoop.ITTestEncryptionOptions): github.com: Name or service not known
Tests run: 17, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO] — maven-failsafe-plugin:2.10:verify (default) @ parquet-hadoop —
[INFO] Failsafe report directory: /data/jw/apache/parquet/parquet-hadoop/target/failsafe-reports
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Parquet MR 1.13.0-SNAPSHOT:
[INFO]
[INFO] Apache Parquet MR .................................. SUCCESS [  8.110 s]
[INFO] Apache Parquet Format Structures ................... SUCCESS [  8.766 s]
[INFO] Apache Parquet Generator ........................... SUCCESS [  3.380 s]
[INFO] Apache Parquet Common .............................. SUCCESS [  9.466 s]
[INFO] Apache Parquet Encodings ........................... SUCCESS [ 35.689 s]
[INFO] Apache Parquet Column .............................. SUCCESS [01:33 min]
[INFO] Apache Parquet Arrow ............................... SUCCESS [  3.252 s]
[INFO] Apache Parquet Jackson ............................. SUCCESS [  4.634 s]
[INFO] Apache Parquet Hadoop .............................. FAILURE [22:20 min]
[INFO] Apache Parquet Avro ................................ SKIPPED
[INFO] Apache Parquet Benchmarks .......................... SKIPPED
[INFO] Apache Parquet Pig ................................. SKIPPED
[INFO] Apache Parquet Thrift .............................. SKIPPED
[INFO] Apache Parquet Cascading [Deprecated, will be removed after 1.12] SKIPPED
[INFO] Apache Parquet Cascading (for Cascading 3.0 onwards) [Deprecated, will be removed after 1.12] SKIPPED
[INFO] Apache Parquet Command-line ........................ SKIPPED
[INFO] Apache Parquet Pig Bundle .......................... SKIPPED
[INFO] Apache Parquet Protobuf ............................ SKIPPED
[INFO] Apache Parquet Scala ............................... SKIPPED
[INFO] Apache Parquet Scrooge [Deprecated, will be removed after 1.12] SKIPPED
[INFO] Apache Parquet Hadoop Bundle ....................... SKIPPED
[INFO] Apache Parquet Tools; [Deprecated, will be removed after 1.12; Use parquet-cli instead] SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  25:08 min
[INFO] Finished at: 2021-04-01T12:04:30+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.10:verify (default) on project parquet-hadoop: There are test failures.
[ERROR]
[ERROR] Please refer to /data/jw/apache/parquet/parquet-hadoop/target/failsafe-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :parquet-hadoop
(base) [jw@cn06 parquet]$

Lutz Weischer:
wget https://github.com/apache/parquet-testing/raw/40379b3/data/encrypt_columns_and_footer.parquet.encrypted
works.
LC_ALL=C mvn clean install
which includes the tests, works, using Java 16.

Maya Anderson / @andersonm-ibm:
@gszadovszky, just like you suggested, I expect the same error when trying to get the file [https://github.com/apache/parquet-testing/raw/40379b3/data/encrypt_columns_and_footer.parquet.encrypted] manually (e.g. with wget or curl), since this looks like a DNS resolution issue.
[~LutzWeischerFujitsu]  , to isolate the problem - does adding github.com directly to /etc/hosts  make any difference?

Gabor Szadovszky / @gszadovszky:
[~LutzWeischerFujitsu], this is a known issue. See PARQUET-2010 for details. You may either cherry-pick my change in the related PR or step back to the previous release in the history to go forward. I hope my change will get in soon so everyone would be able to continue working on the new release.

Gabor Szadovszky / @gszadovszky:
[~LutzWeischerFujitsu], PARQUET-2010 is already in master so it should work now.

Lutz Weischer:
Please be so kind to inform when the new (master) release is available and should not have that issue any more. I will than re-try. Thanks.

repo: apache/parquet-java


text:
Extensible PinotDataBuffer SPI

This is a PEP-Request. I propose to create a extensible SPI that can be used to change the PinotDataBuffer implementation used at runtime. Treat this issue as a clone of https://github.com/apache/pinot/issues/9162, but more formal.
What needs to be done?
Right now PinotDataBuffer instances are obtained by calling static methods:

PinotDataBuffer allocateDirect(long size, ByteOrder byteOrder, String description)
PinotDataBuffer loadFile(File file, long offset, long size, ByteOrder byteOrder, String description)
PinotDataBuffer mapFile(File file, boolean readOnly, long offset, long size, ByteOrder byteOrder, String description)

These methods have a static implementation that return a PinotDataBuffer that is backed by a ByteBuffer or by a LArray, depending on whether the requested size is greater than 2GBs or not. ByteBuffers are faster and more reliable than LArray, but they cannot be larger than Integer.MAX_INT (aka 2GB - 1byte).
Here I propose to change that implementation. Instead of having a static implementation, the algorithm used to instantiate the PinotDataBuffer will be delegated to an interface. The specific implementation will be set at Pinot startup time by reading the property pinot.offheap.buffer.factory, which will be an optional property whose value is a String. In case it exists, it should the qualified class name of a Class that implements PinotDataBufferFactory. There will be also another optional property called pinot.offheap.prioritize.bytebuffer which will be a boolean. If it is true, then the ByteBuffer implementation will be used when less than 2GB buffers are requested.
Why the feature is needed
The reason to be able to have new PinotDataBuffer implementations is that LArray is not compatible with Java >= 16, as detected in #8529. It also seems that our LArray implementation may have some bugs (see https://github.com/apache/pinot/pull/10774) and it doesn't seem that LArray will be updated (See https://github.com/xerial/larray/issues/75), so we need to move on.
Initial idea/proposal
Adding a SPI is something we have done several times in the last months/years. Most of the times we use one or more configuration properties to decide which instance we need to use and set that instance in a static attribute. That system works, but it is problematic when we want to change the implementation in an isolated way (for example, when doing tests).
That is why I propose to extend this system with a thread local on top of that. Something like:
  /**
   * The default {@link PinotBufferFactory} used by all threads that do not define their own factory.
   */
  private static PinotBufferFactory _defaultFactory = createDefaultFactory();
  /**
   * A thread local variable that can be used to customize the {@link PinotBufferFactory} used on tests. This is mostly
   * useful in tests.
   */
  private static final ThreadLocal<PinotBufferFactory> _FACTORY = new ThreadLocal<>();

  /**
   * Creates the default factory depending on the JVM version
   */
  public static PinotBufferFactory createDefaultFactory() {...}

  /**
   * Changes the default factory
   */
  public static void setDefaultFactory(PinotBufferFactory) {...}

  /**
   * Change the {@link PinotBufferFactory} used by the current thread.
   *
   * @see #loadDefaultFactory(PinotConfiguration)
   */
  public static void useFactory(PinotBufferFactory factory) {
    _FACTORY.set(factory);
  }

  /**
   * Returns the factory the current thread should use.
   */
  public static PinotBufferFactory getFactory() {
    PinotBufferFactory pinotBufferFactory = _FACTORY.get();
    if (pinotBufferFactory == null) {
      pinotBufferFactory = _defaultFactory;
    }
    return pinotBufferFactory;
  }

Then the static methods PinotDataBuffer.allocateDirect (and others) should delegate on PinotDataBuffer.getFactory(). By doing that, tests can call PinotDataBuffer.useFactory() in order to use, on that thread, the buffer library they want to test. Exploratory draft https://github.com/apache/pinot/pull/10528 implements all of these and modifies tests to use this API.
Actual buffer implementations
This is an optional part of the PEP. What we request is to have the PinotBufferFactory SPI. This is what we would like to do with it, but it could be part of another PEP if necessary.
Given that we cannot use LArray in modern versions of Java, we have three alternatives :

Use other third party libraries
Implement our own library on top of Unsafe
Use Foreign Memory API

I tried to use Chronicle Bytes in the past with partial success, see https://github.com/apache/pinot/pull/9842. Given that I don't know more third party buffer libraries that use long offsets (Netty and Agrona use ints), I explored the last two alternatives in https://github.com/apache/pinot/pull/10528.
Foreign Memory API is clearly the future, as it provides exactly what we need: A high performance buffer API that is maintained (given that will be included in the JVM) and uses longs as offsets and sizes. But it has two issues:

It is not included in Java 17
It is still in preview mode and it will continue like that in Java 21 (see this post where one of the author explains the reasons and the changes since Java 20).

Therefore we have three options:

Wait until it is stable (possible in Java 22 or 23, which won't be LTS, so maybe we need to wait 2 years until Java 25).
Use it as a preview in Java 21. This would imply that we could either run with Java 11+LArray or Java 21+Foreign, but do not support Java 17. Also, in Java 21 we would need to compile and start with --enable-preview.
Create our own library.

Previously mentioned https://github.com/apache/pinot/pull/10528 does create two buffer implementations: One on top of Unsafe and another on top of Foreign Memory API. The latter is trivial to implement, but it is very difficult to include in the current Apache Pinot CI. The problems I found are:

Some Maven plugins don't work in Java 21-ea (at least spotless doesn't work)
GitHub Actions cannot be configured to use Java 21-ea (see https://github.com/actions/setup-java/issues/492#issuecomment-1551508335).
In general, it is difficult to maintain in the same compilation unit code that has to be compiled with Java 11 and code that has to be compiled with Java 21
Presto still requires to use Java 8, which makes it even more difficult.

In case we decide to use my draft in the actual implementation of the PEP, I would suggest to remove the Foreign Memory API implementation from the branch and optionally move it to another github repo.
I've also modified BenchmarkPinotDataBuffer in order to run with the Unsafe based implementation. The modifications are in the draft.
I've run the benchmark in a M1 Pro and in a Ryzen 9 3900X with Ubuntu 22.10. The exact results of the benchmark can be found here but the following chars should be good enough:
Please note that LArray only runs in Java 11, so there are no data with LArray in Java 17 and 21.
This one shows the cost of executing batch writes (aka call PinotDataBuffer.readFrom) and batch read (aka call PinotDataBuffer.copyTo) with a byte array of 1024 elements (units are ns/op in Linux):

It is important to note that the implementation used in both LArray and Unsafe when dealing with batch reads and writes is to create a temporal direct byte buffer that points to the same address of the current buffer, so we can assume that the performance difference is due to the new instance creation.
This one shows the cost of executing non batch writes (calling PinotDataBuffer.putByte()) and reads (calling PinotDataBuffer.getByte) in a loop with 1024 consecutive offsets starting from a random one (units are ns/op in Linux):

In my opinion the latest is the most interesting, as it shows the improvements introduced in Java 17 and 21 JIT when dealing with loops.
I've also tried to add the Foreign Memory API implementation in the benchmark, but I found several problems so I gave up.

This is amazing @gortiz . I am +1

+1 from me as well.

+1, awesome work!

+1. Thanks @gortiz
So I am guessing the next plan of action is to:

Review the SPI and Unsafe based implementation in https://github.com/apache/pinot/pull/10528

Also, Foreign Memory is not ruled out right. Later on we should move to Foreign Memory API when stable support is available ?

Review the SPI and Unsafe based implementation in https://github.com/apache/pinot/pull/10528

I have some issues there. Some tests always get out of memory in GitHub Actions, while my amd pc and my M1 laptop run them without problems, even when running with mvn -P github-actions. Given that I cannot reproduce the problem locally, it is being very difficult to fix the problem. Any help there would be helpful.

Later on we should move to Foreign Memory when stable support is available ?

I guess we should decide what we want to do with that. We can either:

Wait until Foreign Memory API is production ready and stay in Java 11 meanwhile.

Versions on which we could run Pinot: [11, 15] and [22, ...).

Assuming that Foreign Memory API is released in Java 22.

Advantage: we will only use proven and stable apis.
Disadvantages: But it implies that we would need to either run with Java 11 until Java 23 or we would need to use non LTS versions.

Support preview of Foreign Memory API:

Versions on which we could run Pinot: [11, 15] and [21, ...).
Advantage: We would support Java 21 if running with --enable-preview
Disadvantage:

If OpenJDK decides to add more breaking changes in Java 22, we would need to maintain different versions of the code.
We would need to run with --preview-enable, which is not intended to be executed in production.

Use our own library on top of Unsafe

Versions on which we could run Pinot: [11, ...).
Advantages:

In Java [11, 15] we can choose to use LArray or Unsafe
In Java [16, 20] we can only use Unsafe
In Java 21 we can use Unsafe or run with --enable-preview and use Foreign Memory API
Once Foreign Memory API is released, we could always use Foreign Memory API

Disadvantages:

We need to maintain the Unsafe library.

Errors there may kill the JVM with segmentation faults. Details may depend on CPU architectures and OS. For example, mmap needs to be page aligned in Linux (otherwise segmentation fault kills the JVM).
Unsafe methods do change between major JVM versions.

I was able to find the reason why some tests were failing in https://github.com/apache/pinot/pull/10528. It was a Maven config problem related on how I was configuring the surefire argLine property.
Now I have green lights in Java 11 and Java 20. Last execution failed in Java 17 and I'm trying to verify if it is a flaky test or something actually related to these changes.
I'm thinking on removing the draft tag from the PR. We should discuss whether to include the Foreign Memory API or not in the PR. I think it is better to just drop that code and add them later once is it stable (or if we want to invest resources into trying with --preview-enabled)

We may want to close this issue

source: gharchive/issue

authors: gortiz kishoreg mayankshriv siddharthteotia xiangfu0

repo: apache/pinot

license_type: permissive

license_source: github-api


text:
Add instance id in response

Description
This PR adds instance id at the error message in the response. Currently the response is like below, which is hard to identify the target server:
        "errors": [
            {
                "message": "ServerTableMissing:\nFailed to find table: testTable_OFFLINE",
                "type": "UNKNOWN"
            }
         ]

Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

[ ] Yes (Please label as backward-incompat, and complete the section below on Release Notes)

Does this PR fix a zero-downtime upgrade introduced earlier?

[ ] Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

Does this PR otherwise need attention when creating release notes? Things to consider:

New configuration options
Deprecation of configurations
Signature changes to public methods/interfaces
New plugins added or old plugins removed

[ ] Yes (Please label this PR as release-notes and complete the section on Release Notes)

Release Notes

Documentation

Codecov Report

Merging #7937 (dfaf9e9) into master (aa2da07) will decrease coverage by 57.00%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master    #7937       +/-   ##
=============================================
- Coverage     71.32%   14.32%   -57.01%
+ Complexity     4092       80     -4012
=============================================
  Files          1589     1548       -41
  Lines         82139    80502     -1637
  Branches      12270    12067      -203
=============================================
- Hits          58589    11532    -47057
- Misses        19578    68109    +48531
+ Partials       3972      861     -3111

Flag
Coverage Δ

integration1
?

integration2
?

unittests1
?

unittests2
14.32% <0.00%> (-0.04%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...core/query/executor/ServerQueryExecutorV1Impl.java
0.00% <0.00%> (-83.34%)
:arrow_down:

...ain/java/org/apache/pinot/core/data/table/Key.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/org/apache/pinot/spi/utils/BooleanUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/org/apache/pinot/core/data/table/Record.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../java/org/apache/pinot/core/util/GroupByUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/config/table/FSTType.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ava/org/apache/pinot/spi/data/MetricFieldSpec.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...va/org/apache/pinot/spi/utils/BigDecimalUtils.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/common/tier/TierFactory.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...java/org/apache/pinot/common/utils/StringUtil.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 1264 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aa2da07...dfaf9e9. Read the comment docs.

source: gharchive/pull-request

authors: codecov-commenter jackjlli


text:
[WIP] Adaptive Server Selection to improve Query Processing Resiliency

label = feature
OSS issue https://github.com/apache/pinot/issues/8618
Design Doc
PR Description
Based on the segment assignment strategies used, we use one of the following instance selectors today:

BalancedInstanceSelector
ReplicaGroupInstanceSelector
StrictReplicaGroupInstanceSelector

Irrespective of which instance selection mechanism is used, we use a round robin approach. The round-robin approach is not sensitive to changes in the system like server slowness, GC on servers, etc. Hence we could end up overloading some servers especially in cases where they are underperforming thereby leading to higher latencies for queries.
This PR provides the implementation for the Adaptive Server Selection feature at brokers. When a query is received, we could use one of the implemented Adaptive Selectors (NumInFlightRequests, Latency, Hybrid) to efficiently route queries to the best server instead of using a naive round robin approach.
Test Results
TBD

Codecov Report

Merging #9311 (d975dbf) into master (1c9528c) will decrease coverage by 41.46%.
The diff coverage is 15.98%.

@@              Coverage Diff              @@
##             master    #9311       +/-   ##
=============================================
- Coverage     69.73%   28.27%   -41.47%
+ Complexity     5017       53     -4964
=============================================
  Files          1873     1868        -5
  Lines         99598    99613       +15
  Branches      15163    15169        +6
=============================================
- Hits          69453    28163    -41290
- Misses        25217    68713    +43496
+ Partials       4928     2737     -2191

Flag
Coverage Δ

integration1
26.05% <12.69%> (-0.09%)
:arrow_down:

integration2
24.81% <14.97%> (+0.02%)
:arrow_up:

unittests1
?

unittests2
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...e/pinot/broker/api/resources/PinotBrokerDebug.java
82.75% <0.00%> (-2.96%)
:arrow_down:

...roker/requesthandler/BaseBrokerRequestHandler.java
60.47% <ø> (-9.78%)
:arrow_down:

...routing/adaptiveserverselector/HybridSelector.java
0.00% <0.00%> (ø)

...outing/adaptiveserverselector/LatencySelector.java
0.00% <0.00%> (ø)

...adaptiveserverselector/NumInFlightReqSelector.java
0.00% <0.00%> (ø)

...e/pinot/common/utils/ExponentialMovingAverage.java
0.00% <0.00%> (ø)

...org/apache/pinot/core/transport/QueryResponse.java
100.00% <ø> (ø)

.../server/routing/stats/ServerRoutingStatsEntry.java
0.00% <0.00%> (ø)

...va/org/apache/pinot/spi/utils/CommonConstants.java
0.00% <0.00%> (-27.70%)
:arrow_down:

...erver/routing/stats/ServerRoutingStatsManager.java
6.04% <6.04%> (ø)

... and 1299 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

@Jackie-Jiang  Thanks for the comments. I've addressed them in #9462. Please review.
I'll add the following soon.

Release-note in PR description.
Example of configs to use this feature.
Pinot docs.

Merged user docs - https://docs.pinot.apache.org/operators/operating-pinot/tuning/query-routing-using-adaptive-server-selection

authors: codecov-commenter vvivekiyer


text:
Exclusive subscription mode for partitioned topics not seeing consistent order among different consumers

Describe the bug
In the official documentation of Apache Pulsar it says that an exclusive subscription sees a consistent order for a single consumer. Suppose we have multiple consumers each of them with its own exclusive subscription and they are reading from a partitioned topic. As presented in the docs:
"Decisions about routing and subscription modes can be made separately in most cases. In general, throughput concerns should guide partitioning/routing decisions while subscription decisions should be guided by application semantics.
There is no difference between partitioned topics and normal topics in terms of how subscription modes work, as partitioning only determines what happens between when a message is published by a producer and processed and acknowledged by a consumer."

Those statements lead us to infer that the readers would get a consistent global ordering among topic partitions. But that is not the case in my tests so far:

The code I have used to test it (Scala):
Producer:
  package pulsar

      import org.apache.pulsar.client.api.PulsarClient
      import java.util.UUID

      object Producer {

        def main(args: Array[String]): Unit = {

          val client = PulsarClient.builder()
            .serviceUrl(SERVICE_URL)
            .allowTlsInsecureConnection(true)
            .build()

          val producer = client.newProducer()
            .topic(TOPIC)
            .enableBatching(true)
            //.accessMode(ProducerAccessMode.Exclusive)
            .create()

          for(i<-0 until 100){
            val key = UUID.randomUUID().toString.getBytes()
            //val key = s"Hello-${i}".getBytes()
            producer.newMessage().orderingKey("k0".getBytes()).value(key).send()

            println(s"produced msg: ${i.toString}")
          }

          producer.flush()

          producer.close()
          client.close()
        }

      }

Consumer:
import org.apache.pulsar.client.api.{PulsarClient, SubscriptionInitialPosition, SubscriptionType}

 object Consumer  {

       def main(args: Array[String]): Unit = {

         val client = PulsarClient.builder()
           .serviceUrl(SERVICE_URL)
           .allowTlsInsecureConnection(true)
           .build()

         var l1 = Seq.empty[String]
         var l2 = Seq.empty[String]

         val c1 = client.newConsumer()
           .topic(TOPIC)
           .subscriptionType(SubscriptionType.Exclusive)
           .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
           .subscriptionName("c1")
           .subscribe()

         // To stop the consuption I put a limit (100) - this limit is known
         while(l1.length < 100){
           val msg = c1.receive()
           val str = new String(msg.getData)

           println(s"${Console.MAGENTA_B}$str${Console.RESET}")

           l1 = l1 :+ str

           //c1.acknowledge(msg.getMessageId)
         }

         val c2 = client.newConsumer()
           .topic(TOPIC)
           .subscriptionType(SubscriptionType.Exclusive)
           .subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)
           .subscriptionName("c2")
           .subscribe()

         while(l2.length < 100){
           val msg = c2.receive()
           val str = new String(msg.getData)

           println(s"${Console.GREEN_B}$str${Console.RESET}")

           l2 = l2 :+ str

           //c2.acknowledge(msg.getMessageId)
         }

         println()
         println(l1)
         println()

         println()
         println(l2)
         println()

         try {
           assert(l1 == l2)
         } finally {
           c1.close()
           c2.close()
           client.close()
         }
       }

 }

Am I wrong about it or Pulsar does not support the described behavior I expect?
**OBS.: I've tried every configuration for that. I've set Retention policies for the namespace as infinite both in size and number of messages. It does not work :(
I also tried:

SinglePartitionRouting for producer (it does not matter tho)
Setting an ordering key for the messages**

@lucasrpb
I think pulsar ensures the orderly consumer of one partition-topic. If multiple partition-topics are consumed in subscription mode, the a consistent global order cannot be guaranteed.

Just to make sure @shibd : Using the combo of SinglePartition config for the producer and ExclusiveSubscription for the consumer would not guarantee ordering?

Just to make sure @shibd : Using the combo of SinglePartition config for the producer and ExclusiveSubscription for the consumer would not guarantee ordering?

This can guarantee ordering.

Just to make sure @shibd : Using the combo of SinglePartition config for the producer and ExclusiveSubscription for the consumer would not guarantee ordering?

This can guarantee ordering.

https://stackoverflow.com/questions/68988929/does-pulsar-partitioned-topics-support-global-ordering-when-consuming-from-diffe/69032885#69032885

Thanks guys for the explanations! Unfortunately with the current implementation I cannot get the behavior I want! But it will be very easy to achieve it using an approach I have in mind already ! :) Maybe in the near future I can propose a pull request on it! :)

The issue had no activity for 30 days, mark with Stale label.

authors: codelipenghui lucasrpb miguelemosreverte shibd

repo: apache/pulsar


text:
Add offload ledger info for admin topics stats

Motivation
Currently, we don't have offload stats when getting the topic stats bin/pulsar-admin topics stats.
We should add metrics in topic stats on last offloaded ledger id, last successful offloaded timestamp, and last offload failure timestamp.
Modifications
Add lastOffloadedLedgerId, lastOffloadSuccessTimestamp, lastOffloadFailureTimestamp for ManagedLedgerImpl and TopicStatsImpl.

Thanks for your contribution. For this PR, do we need to update docs?
(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

Thanks for your contribution. For this PR, do we need to update docs?
(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

Yes, we need the "REST APIs => PERSISTENT TOPIC => Get the stats for the topic"  api return result updated.

authors: Anonymitaet frankxieke


text:
bundle metrics would overwrite loadbalance metrics

Motivation
#12366  added bundle metrics, But, If we enabled expose these metrics, It would overwrite previous load balancing metrics.
Modifications
seperate bundle metrics and load balance metrics
Documentation
Check the box below or label this PR directly (if you have committer privilege).
Need to update docs?

[ ] doc-required
(If you need help on updating docs, create a doc issue)

[x] no-need-doc
(Please explain why)

[ ] doc
(If this PR contains doc changes)

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

@hangc0276  PTAL at this.

Great catch. The code looks good to me. Before we merge this, can you please update the documentation? The doc is located here: site2/docs/reference-metrics.md.

@michaeljmarshall  Done.

@michaeljmarshall docs are done in https://github.com/apache/pulsar/pull/12825/files, so I label this PR w/ doc-added and remove doc-required.

@Anonymitaet - thank you. I forgot to switch labels once the docs were added.


text:
[fix][doc] Migrate config docs to pulsar-site

Motivation
Pulsar configuration docs are going to be migrated to https://pulsar.apache.org/reference (Not available currently, will be after apache/pulsar-site#169 is merged. Current version is deployed here). This PR aims to adjust the docs so that everything looks fine in the new site.
Modifications

Remove config docs that are auto-generated with code. Manually maintained ones are uneffected.
Adjust the format of admonition.
Correct the invalid links.
Adjust sidebar structure.

Verifying this change

[x] Make sure that the change passes the CI checks.

This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API: no
The schema: no
The default values of configurations: no
The wire protocol: no
The rest endpoints: no
The admin cli options: no
Anything that affects deployment: no

Documentation
Check the box below or label this PR directly.
Need to update docs?

[ ] doc-required
(Your PR needs to update docs and you will update later)

[] doc-not-needed
(Please explain why)

[x] doc
(Your PR contains doc changes)

[ ] doc-complete
(Docs have been already added)

@Anonymitaet PTAL

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

/pulsarbot run-failure-checks

authors: Anonymitaet SignorMercurio


text:
[fix][build] Dump Jacoco coverage data to file with JMX interface in TestNG listener

Fixes #19931
Motivation
See #19931. Sometimes the default Jacoco shutdown hook doesn't run and there's no coverage data. This causes the Codecov upload to fail since there's no Jacoco coverage report available.
Modifications

enable Jacoco agent's JMX interface
Dump Jacoco coverage to file with JMX interface in TestNG listener

Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository: https://github.com/lhotari/pulsar/pull/145

Yes, that's the case. Are your going to rerun the workflow in your fork and notify here once it passed, or we tag this PR as ready-to-test and rerun here?

@tisonkun re-running it. in progress at https://github.com/lhotari/pulsar/actions/runs/4540821034/jobs/8006055251?pr=145 . This time build step completed properly. Waiting for the whole build job to complete.

/pulsarbot rerun-failure-checks

Hmmm...Now we can pass the flaky tests job, but it seems the integration tests upload job is still suffering:

@lhotari the error info says "not_found" and I read it as file not created. But it seems different from the previous one.
Do we now always dump the file, but the upload action itself is still unstable? IIRC there is an upstream issue for the latter one.

@lhotari the error info says "not_found" and I read it as file not created. But it seems different from the previous one.
Do we now always dump the file, but the upload action itself is still unstable? IIRC there is an upstream issue for the latter one.

Exactly. For the upload issue, it seems that Codecov suggests to not consider the upload token as a secret:

Public repositories that rely on PRs via forks will find that they cannot effectively use Codecov if the token is stored as a GitHub secret. The scope of the Codecov token is only to confirm that the coverage uploaded comes from a specific repository, not to pull down source code or make any code changes.
For this reason, we recommend that teams with public repositories that rely on PRs via forks consider the security ramifications of making the Codecov token available as opposed to being in a secret.

authors: lhotari tisonkun


text:
Fixed NPE when closing batch during a reconnection

Motivation
There is a race condition when closing a batch and the producer is getting disconnected. This has been seen in some of the tests run and can be reproduced by running this particular test many times in a loop.
2019-05-31\T\16:32:49.647 [ERROR] testProducerReconnect(org.apache.pulsar.client.api.ClientErrorsTest)  Time elapsed: 0.034 s  <<< FAILURE!
org.apache.pulsar.client.api.PulsarClientException: java.lang.NullPointerException
	at org.apache.pulsar.client.impl.ProducerImpl.batchMessageAndSend(ProducerImpl.java:1354)
	at org.apache.pulsar.client.impl.ProducerImpl.triggerFlush(ProducerImpl.java:1282)
	at org.apache.pulsar.client.impl.ProducerBase.send(ProducerBase.java:82)
	at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.send(TypedMessageBuilderImpl.java:61)
	at org.apache.pulsar.client.impl.ProducerBase.send(ProducerBase.java:52)
	at org.apache.pulsar.client.api.ClientErrorsTest.testProducerReconnect(ClientErrorsTest.java:600)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:54)
	at org.testng.internal.InvokeMethodRunnable.run(InvokeMethodRunnable.java:44)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
	at org.apache.pulsar.client.impl.ProducerImpl.batchMessageAndSend(ProducerImpl.java:1329)
	... 17 more

The problem is that we're checking isConnected() and then using the cnx() which can return null if the producer is disconnected. Rather, we should first cache the reference to the ClientCnx instance and then double check if we're connected.
The same pattern is already applied for the non-batching scenario:
https://github.com/apache/pulsar/blob/ba24d73b5f77c42f832cd9d2720dc0aba388f909/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java#L399-L405

run java8 tests

run java8 tests

run java8 tests


text:
[Doc] Add a note about sink-type and source-type value of the built-in connector.

Fixes #6109
Motivation
Tell user the available values of "--sink-type" "--source-type" parameter of the Cassandra built-in connector.
Modifications
Update the following docs:

Pulsar IO > get started
Connector Admin CLI

@tuteng , please help review the doc update.

@jiazhai @Jennifer88huang , cloud you please help review the PR? Thanks.


text:
make pulsar executor pool size configurable

Motivation
The pulsar executor pool size number is hard code to 20 when pulsar service start, it should be configurable in broker.conf.
private final ScheduledExecutorService executor = Executors.newScheduledThreadPool(20,
            new DefaultThreadFactory("pulsar"));

Changes
make the executor pool size configurable in broker.conf

As a followup to these changes in thread pool sizes, it would be nice to reduce the size of these pools when running the unit tests

As a followup to these changes in thread pool sizes, it would be nice to reduce the size of these pools when running the unit tests

Yes, for the unit tests framework, i will reduce the pool size.

As a followup to these changes in thread pool sizes, it would be nice to reduce the size of these pools when running the unit tests

Yes, for the unit tests framework, i will reduce the pool size.

@merlimat @jiazhai  I have changed the default executor pool size to Runtime.getRuntime().availableProcessors() and update unit test executor pool size to 5, please take a look, thanks.

/pulsarbot run-failure-checks


text:
can not run in Windows

when i  run mvn clean package -Dmaven.test.skip=true
my os alert this software can not run on your platform
it seems that the version of yarn is incorretly!
Windows10
maven3.2.X
jdk 8

Could you provide the Error message?
By the way, If yarn doesn't work on your environment, you can remove the frontend-maven-plugin from pom.xml.
link

thank u for your reply
this is the screenshot of the error

could i use local yarn and node to build this project?

It seems node and yarn were installed succeed, but no permission when running yarn install, maybe we should fix the permission issue.
Another work around is that you can remove the frontend-maven-plugin from pom.xml as I mentioned before.

repo: apache/rocketmq-dashboard


text:
Build Fails on ARM MAC

When building on ARM MAC required to add -Dos.arch=x86_64 to bypass protobuff build issues
When this is passed the next issue is related to this https://github.com/apache/rocketmq-mqtt/issues/160 having PostConstruct deps are missing from POM.xml of related projects.
Any particular reason the javax.annotation deps are missing they are not part of ANY JDK it looks like these somehow on your build/classpath in project authors set-up and the required build instructions/set-up is then missing.

                   <dependency>
                        <groupId>javax.annotation</groupId>
                        <artifactId>javax.annotation-api</artifactId>
                        <version>1.3.2</version>
                    </dependency>

Also going back to item 2 pointing to issue is this project tied to https://github.com/apache/rocketmq-mqtt/issues/160#issuecomment-1329993294     RocketMQ 4.9.4 and must be build in Docker if yes any hint in setting this up is appreciated?

Thank you

The Java Common Annotations Module java.xml.ws.annotation was deprecated in Java version 9 and was removed in java version 11. If this leads to a problem you could try to add javax.annotation.

No these are two different deps javax.annotation is not the same as java.xml.ws.annotation my point is that  your project is either missing proper build instructions in your readme.   or pom.xml needs to have correct dependency added since other folks were able build this only while in they have javax.annotation deps on classpath https://github.com/apache/rocketmq-mqtt/issues/160#issuecomment-1329993294

authors: meshpaul pingww

repo: apache/rocketmq-mqtt


text:
LitePullConsumerSubscribe testcase not work [4.6.0]

BUG REPORT

Please describe the issue you observed:

I run the example below, but it doesn't work when the consumer group subscribe the topic the first time(I mean the consumer group have never consumed the topic). The consumer can not consume any message, include old messages and new messages.
testcase file:
https://github.com/apache/rocketmq/blob/rocketmq-all-4.6.0/example/src/main/java/org/apache/rocketmq/example/simple/LitePullConsumerSubscribe.java
If the consumer group have consumed some messages before , (may be consumed by DefaultPushConsumer), then the testcase works.

Please tell us about your environment:
rocketmq 4.6.0

Can you try if the same issue still exists in the latest version ？

4.7.0  still not work

authors: RongtongJin henrypoter

repo: apache/rocketmq


text:
为什么源代码都没加注释

至少关键部分应该加

@safph The issue tracker is only used for bug report and feature request,  you can subscribe mailing list for help , community friends will answer your questions about the design or even the code.
Of course, we will also improve our documentation to help everyone become familiar with RocketMQ.

authors: duhengforever safph


text:
One Swc depended on another Swc Causes Index.html Blank

I am compiling my own library DBIZLib.swc
Console : https://paste.apache.org/h4c5d
what should i do to remove these warnings?
because if i ignore these warnings and compile any project index.html is blank
Errors:
Uncaught ReferenceError: applications_mx_managers_SystemManager is not defined at index.html:962
Uncaught Error: Bad dependency path or symbol: org.apache.royale.effects.IEffect
at visit (base.js:2796)
at visit (base.js:2808)
at visit (base.js:2808)
at goog.DebugLoader_.load_ (base.js:2815)
at Object.goog.require (base.js:846)
at index.html:957
https://user-images.githubusercontent.com/42200979/61864583-1ab30c00-aeeb-11e9-9a8f-690798fec1cd.png

It looks like you're missing playerglobal.swc as an external dependency.
It would help to see what build configuration you are using.

Recently Josh made changes to how libraries are linked. I'm not using ANT, but I think ANT solves this with config.xml files in projects. Basically instead of listing all in library-path node, you must use "external-library-path" node in libraries to ensure all the code in the libraries linked are not compiled into that library. Then in Application, you must continue using library-path, since you want all code used be compiled in the application. In this part your application maybe is missing playerglobal or js swcs as Harbs comment.

authors: Harbs carlosrovira pashminakazi

repo: apache/royale-asjs


text:
Add test cases for service.manager.impl

Make sure that:

[√ ] You have read the contribution guidelines.
[√ ] You submit test cases (unit or integration tests) that back your changes.
[ √] Your local test passed ./mvnw clean install -Dmaven.javadoc.skip=true.

hi， pls checked checkstyle and  build in you local env.

so much checkstyle error

hi， pls checked checkstyle and build in you local env.

so much checkstyle error

thx，After this inspection, I learned a lot and found my own shortcomings. At present, I have made a modification

Codecov Report

Merging #4610 (e6a4129) into master (d6067f4) will increase coverage by 0.56%.
The diff coverage is 62.75%.

@@             Coverage Diff              @@
##             master    #4610      +/-   ##
============================================
+ Coverage     67.62%   68.18%   +0.56%
- Complexity     7845     7919      +74
============================================
  Files          1070     1078       +8
  Lines         30650    30775     +125
  Branches       2790     2807      +17
============================================
+ Hits          20726    20983     +257
+ Misses         8303     8139     -164
- Partials       1621     1653      +32

Impacted Files
Coverage Δ

...e/shenyu/plugin/base/fallback/FallbackHandler.java
36.84% <0.00%> (-17.01%)
:arrow_down:

.../apache/shenyu/plugin/hystrix/command/Command.java
7.69% <0.00%> (+2.69%)
:arrow_up:

.../shenyu/plugin/hystrix/command/HystrixCommand.java
55.55% <0.00%> (ø)

...plugin/hystrix/command/HystrixCommandOnThread.java
61.11% <0.00%> (ø)

.../springmvc/ShenyuSpringMvcClientConfiguration.java
100.00% <ø> (ø)

...henyuSpringMvcClientInfoRegisterConfiguration.java
0.00% <0.00%> (ø)

...pringboot/starter/gateway/ShenyuConfiguration.java
88.00% <0.00%> (-3.67%)
:arrow_down:

...shenyu/web/fallback/DefaultFallbackController.java
75.00% <0.00%> (-25.00%)
:arrow_down:

...pache/shenyu/web/filter/CollapseSlashesFilter.java
0.00% <0.00%> (ø)

...ava/org/apache/shenyu/web/filter/HealthFilter.java
62.50% <60.00%> (-37.50%)
:arrow_down:

... and 6 more

... and 7 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

@lijay7674  hi， can you updata  master  to you dev，then summit pull request ？

not merged

sorry,Can I initiate a new pull request

@lijay7674  you can add my wechat: xixy199195, i wiil invite you join contributor group

repo: apache/shenyu


text:
login interface

Please answer these questions before submitting your issue.

Why do you submit this issue?
Question

Question

What do you want to know?
Hi Mr.Wu , I was wondering if skywalking8.1.0 has a login interface . If there is , then which config file is to set up the user and password ? Thanks.

Requirement or improvement

I just upgrade the skw 6.1.0 to 8.1.0 and I can not find my log in interface . Neither could I set up in webapp.yml

No, there isn't. The old login is not real.

Use APISIX, Nginx or other proxy tech to set up a login page.

roger that

authors: sodared wu-sheng

repo: apache/skywalking

license_source: bigquery


text:
持久化到es后，造成es占用内存持续增加

jdk11环境，docker部署es7，使用SkyWalking8.6持久化到es后，会让es的内存逐渐增大，直到100%造成OOM，请问如何解决？

持久化到mysql8，也会使mysql8占用内存持续增长到100%造成oom

Github discussion is required to be English for global users.

Your description is totally wrong. We push data into a storage, then database fails, try to make the database more powerful.

authors: ZaneGeng wu-sheng


text:
[CI] Run build jobs on different nodes parallelly

In this patch, I've made it to run multiple jobs parallelly on different Jenkins slave nodes, thus, our build jobs can be distributed more evenly to our VM nodes, and the VM with lower resources can also be scheduled fairly.
In this PR, I also rewrote the Jenkisfile with scripted Pipeline syntax, our previous syntax is DSL, it's much more human readable, but lack of flexibility in our complex scenarios, now we're using scripted syntax, based on Groovy, it provides a programmable ability, and flexibility, to simplify the Jenkinsfile, (from ~200 lines to ~100 lines)

We don't need to group the jobs anymore, because the jobs are scheduled on different nodes in a smaller granularity (test case), meaning that every test case will take an executor, if there is no more available executor, the test case should be in queue automatically

Please move this as a record in WIKI. We may need this someday.


text:
doc: Add Self Observability service discovery (k8s)

[X] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #7259.
[X] Update the CHANGES log.

And for docs updates, I think it's better to loop @wingwong-knh in to review when the docs are updated in PR rather than review the whole docs after they're merged

I am not sure whether @wingwong-knh is available. Wing, if you are, please take a look.

And for docs updates, I think it's better to loop @wingwong-knh in to review when the docs are updated in PR rather than review the whole docs after they're merged

I am not sure whether @wingwong-knh is available. Wing, if you are, please take a look.

Indeed, for now I'm not sure if I can keep up with the daily updates! In the future, once all docs have been gone through once, I can start reviewing the updates in PR. For now, I'm only halfway through the backend doc (will up my pace!) For this particular doc, I will need to review from scratch.

And for docs updates, I think it's better to loop @wingwong-knh in to review when the docs are updated in PR rather than review the whole docs after they're merged

I am not sure whether @wingwong-knh is available. Wing, if you are, please take a look.

Indeed, for now I'm not sure if I can keep up with the daily updates! In the future, once all docs have been gone through once, I can start reviewing the updates in PR. For now, I'm only halfway through the backend doc (will up my pace!) For this particular doc, I will need to review from scratch.

Take your time. I know you are busy. We are going to continue, you could be back and polish the doc.

authors: wankai123 wingwong-knh wu-sheng


text:
[SPARK-10605][SQL] Add structs to collect_list/collect_set

We currently use the Hive implementations for the collect_list/collect_set aggregate functions. This has a few major drawbacks, the use of HiveUDAF (which has quite a bit of overhead) and the lack of support for struct datatypes. This PR adds native implementation of these functions to Spark.
The size of the collected list/set vary, this means we cannot use the fast, Tungsten, aggregation path to perform the aggregation, and that we fallback to the slower sort based path. Another big problem with these operators is that when the size of the collected list/set grows too large, we can start experiencing large GC pauzes and OOMEs.
This PR implements these UDAFs as ImperativeAggregates and uses a custom mutable ArrayData structure in order to efficiently collect the items. The current PR allows for partial aggregates, but I am not sure yet if we should support this.
Comments and feedback are appreciated.
cc @yhuai

Test build #50490 has started for PR 11004 at commit 8247d8e.

Test build #50490 has finished for PR 11004 at commit 8247d8e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50490/
Test FAILed.

Test build #50493 has started for PR 11004 at commit 326a213.

Test build #50493 has finished for PR 11004 at commit 326a213.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50493/
Test PASSed.

Merged build finished. Test PASSed.

closing this one for a better approach.

authors: AmplabJenkins SparkQA hvanhovell

repo: apache/spark


text:
[SPARK-14262] correct appStatus after master leader changed

What changes were proposed in this pull request?
Suppose master leadership changes, even if recovery is completed, those privious apps in RUNNING state are now in WAITING state from master's WebUI, although they are really at work. Now correct the state.
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

Can one of the admins verify this patch?

I think I have a PR #10506 which already covered this issue.

Can one of the admins verify this patch?

authors: AmplabJenkins GavinGavinNo1 jerryshao


text:
[SPARK-11249][launcher] Throw error if app resource is not provided.

Without this, the code would build an invalid spark-submit command line,
and a more cryptic error would be presented to the user. Also, expose
a constant that allows users to set a dummy resource in cases where
they don't need an actual resource file; for backwards compatibility,
that uses the same "spark-internal" resource that Spark itself uses.
Tested via unit tests, run-example, spark-shell, and running the
thrift server with mixed spark and hive command line arguments.

Test build #57807 has started for PR 12909 at commit 0ab4a28.

Test build #57807 has finished for PR 12909 at commit 0ab4a28.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57807/
Test FAILed.

retest this please

Test build #57900 has started for PR 12909 at commit 0ab4a28.

Test build #57900 has finished for PR 12909 at commit 0ab4a28.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57900/
Test FAILed.

Hmm, pyspark seems to be getting stuck, let me see if I broke it...

Test build #58030 has started for PR 12909 at commit 46664ba.

Test build #58030 has finished for PR 12909 at commit 46664ba.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58030/
Test FAILed.

retest this please

Test build #58039 has started for PR 12909 at commit 46664ba.

Test build #58039 has finished for PR 12909 at commit 46664ba.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58039/
Test PASSed.

@andrewor14

Merging to master / 2.0.

authors: AmplabJenkins SparkQA vanzin


text:
SPARK-17424: Fix unsound substitution bug in ScalaReflection.

What changes were proposed in this pull request?
This method gets a type's primary constructor and fills in type parameters with concrete types. For example, MapPartitions[T, U] -> MapPartitions[Int, String]. This Substitution fails when the actual type args are empty because they are still unknown. Instead, when there are no resolved types to subsitute, this returns the original args with unresolved type parameters.
How was this patch tested?
This doesn't affect substitutions where the type args are determined. This fixes our case where the actual type args are empty and our job runs successfully.

Test build #65269 has started for PR 15062 at commit 931f156.

Test build #65269 has finished for PR 15062 at commit 931f156.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65269/
Test PASSed.

Could you add a regression test?

@hvanhovell, I'm not sure what's causing this to build the test. I can step through with a debugger to see that the problem is caused by an empty actualTypeArgs list, but I don't know why it is that way. Maybe someone with more knowledge of Scala types can help fill in the details. Until then, I'm not sure how to construct a test case for this.

I don't quite understand what use-case this patch fixes. Can you provide an example (in the form of a test) that reproduces the issue?

Oh, I was too quick to comment, I see a complete description is in the JIRA.
It would still be good if you could add a test though.

@jodersky, as I said above, I'm not really sure how to build a test for this because I'm not too familiar with the Scala internals that are misused here. However, since I can confirm that it works in practice, I think it's reasonable to commit it anyway, before committers that know this code well can have a look to make sure this method is tested properly.
Another way to go is to revert PR #10970. That is a work-around because another test was already hitting this problem and the solution was to avoid the parameter substitution.

Jenkins retest this please

Test build #66190 has started for PR 15062 at commit 931f156.

I'm just unclear of why this is needed . According to the linked issue, some tests were failing and this fixed them. Everything just seems very vague to me. It would be very helpful if you could provide an example (it doesn't have to be a unit-test) that reproduces the error.
It would greatly help me understand what's going on and also distill things down to a regression test.

I think the best way to reproduce this is to revert #10970. I don't have a case that reproduces it that I can share since this was in a fairly large job.

Test build #66190 has finished for PR 15062 at commit 931f156.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66190/
Test PASSed.

Hi @rdblue can you revert #10970 in this PR? Then other reviewers can test this PR locally and verify if it works.

I tried to reproduce with #10970 reverted, but I didn't hit the issue in testing. I still think it's fine to move forward on this, even if it is hard to reproduce because we know the code is wrong and this fixes it.

retest this please

I'm fine with this change, cc @yhuai @liancheng

Test build #68939 has started for PR 15062 at commit 931f156.

Test build #68939 has finished for PR 15062 at commit 931f156.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68939/
Test PASSed.

any chance that is the same issue as https://issues.apache.org/jira/browse/SPARK-17109?
@rdblue When you were debugging this issue, which version of scala did you use? Scala 2.10 or Scala 2.11? If you were using scala 2.10, is it possible to try scala 2.11? Thanks!

We were seeing the problem when using the datasets API in our 1.6.1 build, which is based on Scala 2.10. I recently tried to reproduce this on master with 2.11 and #10970 reverted, but I didn't get a case that failed. Either way, I think the fix here makes sense: if there are no types to substitute, don't do it.

This does look the same as SPARK-17109. Does this fix that issue?

retest this please

Test build #76863 has started for PR 15062 at commit 931f156.

Test build #76863 has finished for PR 15062 at commit 931f156.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76863/
Test PASSed.

Merged build finished. Test PASSed.

the change looks reasonable, merging to master/2.2/2.1/2.0!

authors: AmplabJenkins SparkQA cloud-fan hvanhovell jodersky rdblue srowen yhuai


text:
[SPARK-17618] Guard against invalid comparisons between UnsafeRow and other formats

This patch ports changes from #15185 to Spark 2.x. In that patch, a  correctness bug in Spark 1.6.x which was caused by an invalid equals() comparison between an UnsafeRow and another row of a different format. Spark 2.x is not affected by that specific correctness bug but it can still reap the error-prevention benefits of that patch's changes, which modify  ``UnsafeRow.equals()to throw an IllegalArgumentException if it is called with an object that is not anUnsafeRow`.

Test build #65991 has started for PR 15265 at commit f4de93f.

LGTM - pending jenkins.

Test build #65991 has finished for PR 15265 at commit f4de93f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65991/
Test PASSed.

Merging to master and branch-2.0. Thanks!

authors: AmplabJenkins JoshRosen SparkQA hvanhovell


text:
[SPARK-19182][DStream] Optimize the lock in StreamingJobProgressListener to not block UI when generating Streaming jobs

What changes were proposed in this pull request?
When DStreamGraph is generating a job, it will hold a lock and block other APIs. Because StreamingJobProgressListener (numInactiveReceivers, streamName(streamId: Int), streamIds) needs to call DStreamGraph's methods to access some information, the UI may hang if generating a job is very slow (e.g., talking to the slow Kafka cluster to fetch metadata).
It's better to optimize the locks in DStreamGraph and StreamingJobProgressListener to make the UI not block by job generation.
How was this patch tested?
existing ut
cc @zsxwing

Test build #71441 has started for PR 16601 at commit 46036bf.

Test build #71441 has finished for PR 16601 at commit 46036bf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71441/
Test PASSed.

also cc @tdas

Test build #71492 has started for PR 16601 at commit eaa7b15.

Test build #71492 has finished for PR 16601 at commit eaa7b15.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71492/
Test PASSed.

@zsxwing Take a review please!

Test build #71553 has started for PR 16601 at commit e51623c.

Test build #71553 has finished for PR 16601 at commit e51623c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71553/
Test PASSed.

LGTM. Merging to master. Thanks!

authors: AmplabJenkins SparkQA uncleGen zsxwing


text:
[SPARK-19246][SQL]CataLogTable's partitionSchema order and exist check

What changes were proposed in this pull request?
CataLogTable's partitionSchema should check if each column name in partitionColumnNames must match one and only one field in schema, if not we should throw an exception
and CataLogTable's partitionSchema should keep order with partitionColumnNames
How was this patch tested?

Test build #71451 has started for PR 16606 at commit eaf18ce.

Test build #71451 has finished for PR 16606 at commit eaf18ce.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71451/
Test FAILed.

Test build #71453 has started for PR 16606 at commit 9296624.

Test build #71453 has finished for PR 16606 at commit 9296624.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71453/
Test FAILed.

Test build #71483 has started for PR 16606 at commit 4260f84.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71483/
Test FAILed.

retest this please

Test build #71494 has started for PR 16606 at commit 4260f84.

Test build #71497 has started for PR 16606 at commit 6f2816e.

Test build #71494 has finished for PR 16606 at commit 4260f84.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71494/
Test PASSed.

Test build #71497 has finished for PR 16606 at commit 6f2816e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71497/
Test PASSed.

Test build #71519 has started for PR 16606 at commit c08e1c9.

Test build #71519 has finished for PR 16606 at commit c08e1c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71519/
Test PASSed.

Test build #71552 has started for PR 16606 at commit 8cbee32.

Test build #71552 has finished for PR 16606 at commit 8cbee32.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71552/
Test FAILed.

Merged build finished. Test FAILed.

Test build #71554 has started for PR 16606 at commit 79e2e3f.

Test build #71554 has finished for PR 16606 at commit 79e2e3f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71554/
Test PASSed.

Test build #71580 has started for PR 16606 at commit 5e60f14.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71580/
Test FAILed.

Merged build finished. Test FAILed.

Test build #71853 has started for PR 16606 at commit 206b232.

how about we just add an assert? assert(schema.takeRight(partitionColumnNames.length) == partitionColumnNames)

Test build #71853 has finished for PR 16606 at commit 206b232.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71853/
Test PASSed.

Merged build finished. Test PASSed.

Test build #71889 has started for PR 16606 at commit 7e30cc7.

Test build #71889 has finished for PR 16606 at commit 7e30cc7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71889/
Test PASSed.

Test build #71902 has started for PR 16606 at commit 04d3940.

LGTM pending test

Test build #71902 has finished for PR 16606 at commit 04d3940.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71902/
Test PASSed.

Test build #71912 has started for PR 16606 at commit 72164eb.

LGTM, pending tests

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71912/
Test FAILed.

retest this please

Test build #71922 has started for PR 16606 at commit 72164eb.

Test build #71922 has finished for PR 16606 at commit 72164eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71922/
Test PASSed.

thanks, merging to master!

authors: AmplabJenkins SparkQA cloud-fan gatorsmile windpiger


text:
[SPARK-20543][SPARKR][WIP] skip tests when running on CRAN

What changes were proposed in this pull request?
General rule on skip or not:
skip if

RDD tests
tests could run long or complicated (streaming, hivecontext)
tests on error conditions
tests won't likely change/break

How was this patch tested?
unit tests, R CMD check --as-cran, R CMD check

Test build #76342 has started for PR 17817 at commit fc4965a.

Test build #76342 has finished for PR 17817 at commit fc4965a.

This patch fails R style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76342/
Test FAILed.

Test build #76343 has started for PR 17817 at commit 2c5e267.

Test build #76343 has finished for PR 17817 at commit 2c5e267.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76343/
Test PASSed.

Merged build finished. Test PASSed.

Test build #76346 has started for PR 17817 at commit bd3295a.

Test build #76346 has finished for PR 17817 at commit bd3295a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76346/
Test PASSed.

Skipping 153 tests ON CRAN
Jenkins, all tests: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76346/console
CRAN: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76343/console

merged to master/2.2

authors: AmplabJenkins SparkQA felixcheung


text:
[SPARK-21444] Be more defensive when removing broadcasts in MapOutputTracker

What changes were proposed in this pull request?
In SPARK-21444, @sitalkedia reported an issue where the Broadcast.destroy() call in MapOutputTracker's ShuffleStatus.invalidateSerializedMapOutputStatusCache() was failing with an IOException, causing the DAGScheduler to crash and bring down the entire driver.
This is a bug introduced by #17955. In the old code, we removed a broadcast variable by calling BroadcastManager.unbroadcast with blocking=false, but the new code simply calls Broadcast.destroy() which is capable of failing with an IOException in case certain blocking RPCs time out.
The fix implemented here is to replace this with a call to destroy(blocking = false) and to wrap the entire operation in Utils.tryLogNonFatalError.
How was this patch tested?
I haven't written regression tests for this because it's really hard to inject mocks to simulate RPC failures here. Instead, this class of issue is probably best uncovered with more generalized error injection / network unreliability / fuzz testing tools.

Test build #79691 has started for PR 18662 at commit a5ebcac.

cc @jiangxb1987

@JoshRosen - Thanks for the lightning fast response. The change looks reasonable to me.

LGTM

Test build #79691 has finished for PR 18662 at commit a5ebcac.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79691/
Test PASSed.

Merged build finished. Test PASSed.

Merged to master. Thanks for the quick reviews.

authors: AmplabJenkins JoshRosen SparkQA cloud-fan sameeragarwal sitalkedia