text:
HDDS-3787. Make CSI command configurable.

What changes were proposed in this pull request?
It doesn't have to use goofys as the mountable tools to access Ozone. With this PR, we can config an other command to mount file system which could be a fuse program.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3787
How was this patch tested?
With default configuration, CI checks should be passed.

Codecov Report

Merging #1067 into master will increase coverage by 0.94%.
The diff coverage is 47.36%.

@@             Coverage Diff              @@
##             master    #1067      +/-   ##
============================================
+ Coverage     69.46%   70.41%   +0.94%
- Complexity     9121     9241     +120
============================================
  Files           961      961
  Lines         48151    48132      -19
  Branches       4679     4676       -3
============================================
+ Hits          33448    33891     +443
+ Misses        12489    11988     -501
- Partials       2214     2253      +39

Impacted Files
Coverage Δ
Complexity Δ

...main/java/org/apache/hadoop/ozone/OzoneConsts.java
84.21% <ø> (ø)
1.00 <0.00> (ø)

...apache/hadoop/hdds/utils/db/RocksDBCheckpoint.java
90.90% <ø> (+0.90%)
5.00 <0.00> (-3.00)
:arrow_up:

.../java/org/apache/hadoop/ozone/om/OMConfigKeys.java
100.00% <ø> (ø)
1.00 <0.00> (ø)

...in/java/org/apache/hadoop/ozone/csi/CsiServer.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

.../java/org/apache/hadoop/ozone/csi/NodeService.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

.../apache/hadoop/ozone/om/OMDBCheckpointServlet.java
66.26% <ø> (-4.27%)
8.00 <0.00> (-2.00)

...a/org/apache/hadoop/ozone/om/ha/OMNodeDetails.java
86.66% <ø> (ø)
12.00 <0.00> (ø)

...p/ozone/om/ratis/utils/OzoneManagerRatisUtils.java
67.44% <0.00%> (-19.13%)
39.00 <0.00> (ø)

.../java/org/apache/hadoop/ozone/om/OzoneManager.java
63.90% <16.00%> (-0.69%)
185.00 <2.00> (-1.00)

...adoop/ozone/om/ratis/OzoneManagerStateMachine.java
58.24% <90.47%> (+2.50%)
27.00 <4.00> (+1.00)

... and 39 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78255dd...b595af4. Read the comment docs.

@elek 😃 Thanks for your "Big +1", with this PR, we can continue to test whether another tools can bring performance improvement.

authors: codecov-commenter maobaolong

repo: apache/hadoop-ozone


text:
HDDS-1776. Fix image name in some ozone docker-compose files. Contrib…

…uted by Xiaoyu Yao.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
32
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

0
yamllint
0
yamllint was not available.

+1
@author
0
The patch does not contain any @author tags.

-1
test4tests
0
The patch doesn't appear to include any new or modified tests.  Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1
mvninstall
489
trunk passed

+1
compile
249
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
1529
branch has no errors when building and testing our client artifacts.

+1
javadoc
154
trunk passed

_ Patch Compile Tests _

+1
mvninstall
427
the patch passed

+1
compile
250
the patch passed

+1
javac
250
the patch passed

+1
mvnsite
0
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
620
patch has no errors when building and testing our client artifacts.

+1
javadoc
144
the patch passed

_ Other Tests _

+1
unit
268
hadoop-hdds in the patch passed.

-1
unit
2536
hadoop-ozone in the patch failed.

+1
asflicense
38
The patch does not generate ASF License warnings.

6177

Reason
Tests

Failed junit tests
hadoop.ozone.client.rpc.TestContainerStateMachine

hadoop.ozone.client.rpc.TestCommitWatcher

hadoop.ozone.client.rpc.TestSecureOzoneRpcClient

hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline

hadoop.ozone.client.rpc.TestContainerStateMachineFailures

hadoop.ozone.client.rpc.TestOzoneAtRestEncryption

hadoop.hdds.scm.pipeline.TestSCMPipelineManager

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1066/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1066

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint

uname
Linux 248d2dc01ce8 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 96d0555

Default Java
1.8.0_212

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1066/1/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1066/1/testReport/

Max. process+thread count
4265 (vs. ulimit of 5500)

modules
C: hadoop-ozone/dist U: hadoop-ozone/dist

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1066/1/console

versions
git=2.7.4 maven=3.3.9

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

source: gharchive/pull-request

authors: hadoop-yetus xiaoyuyao

repo: apache/hadoop

license_type: permissive

license_source: github-api


text:
HADOOP-16732. S3Guard to support encrypted DynamoDB table

https://issues.apache.org/jira/browse/HADOOP-16732

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m  0s
Docker mode activated.

-1 :x:
docker
22m 45s
Docker failed to build yetus/hadoop:104ccca9169.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/1752

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/1/console

versions
git=2.17.1

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

I tested against us-west-2 region, and did not have all tests passing.
The most relevant test ITestDynamoDBMetadataStore passed with different config settings in auth-keys.xml including default SSE config (AWS owned CMK), AWS managed CMK and customer managed AWS key. Specially, with customer managed CMK, the timeout of the test rule should be up to 60s. Does this imply performance degradation when using customer managed CMK?
The failing ones include:

TestStagingCommitter is compaining about method not found java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager;
ITestDelegatedMRJob, ITestS3AMiniYarnCluster and ITestTerasortOnS3A are timing out w or w/o this change on my side. Not sure if this is related to the above method not found exception.

I will have a look at those failures before claiming this is tested.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m  0s
Docker mode activated.

-1 :x:
docker
0m  7s
Docker failed to build yetus/hadoop:104ccca9169.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hadoop/pull/1752

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/2/console

versions
git=2.17.1

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

The recent commit added SSE support for S3Guard init command. I'll update the info command in a new commit.

Is there any reason you wouldn't want to encrypt? Presumably any KMS billing and throttling will get in the way.

Yes, exactly. So default value might work just fine for most S3Guard users. AWS is saying:

Encryption at rest using the AWS owned CMK is offered at no additional charge. However, AWS KMS charges apply for an AWS managed CMK and for a customer managed CMK.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
3m  1s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
markdownlint
0m  1s
markdownlint was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
1m 10s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
18m 36s
trunk passed

+1 :green_heart:
compile
17m  8s
trunk passed

+1 :green_heart:
checkstyle
2m 41s
trunk passed

+1 :green_heart:
mvnsite
2m 15s
trunk passed

+1 :green_heart:
shadedclient
18m 30s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
2m 13s
trunk passed

+0 :ok:
spotbugs
1m 11s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 14s
trunk passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 24s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 19s
the patch passed

+1 :green_heart:
compile
16m 26s
the patch passed

+1 :green_heart:
javac
16m 26s
the patch passed

-0 :warning:
checkstyle
2m 38s
root: The patch generated 3 new + 20 unchanged - 0 fixed = 23 total (was 20)

+1 :green_heart:
mvnsite
2m 18s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
12m 56s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
2m 13s
the patch passed

+1 :green_heart:
findbugs
3m 27s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
9m  9s
hadoop-common in the patch passed.

+1 :green_heart:
unit
1m 35s
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 56s
The patch does not generate ASF License warnings.

131m 25s

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1752

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint

uname
Linux 523ff6a3bf80 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 65c4660

Default Java
1.8.0_222

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/3/artifact/out/diff-checkstyle-root.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/3/testReport/

Max. process+thread count
1347 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/3/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
2m 50s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
markdownlint
0m  0s
markdownlint was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
1m  6s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
18m 19s
trunk passed

+1 :green_heart:
compile
17m  1s
trunk passed

+1 :green_heart:
checkstyle
2m 41s
trunk passed

+1 :green_heart:
mvnsite
2m 17s
trunk passed

+1 :green_heart:
shadedclient
18m 54s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
2m 11s
trunk passed

+0 :ok:
spotbugs
1m 10s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m  9s
trunk passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 24s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 20s
the patch passed

+1 :green_heart:
compile
16m 25s
the patch passed

+1 :green_heart:
javac
16m 25s
the patch passed

-0 :warning:
checkstyle
2m 38s
root: The patch generated 3 new + 20 unchanged - 0 fixed = 23 total (was 20)

+1 :green_heart:
mvnsite
2m 15s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
12m 51s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
2m 15s
the patch passed

+1 :green_heart:
findbugs
3m 33s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
9m  7s
hadoop-common in the patch passed.

+1 :green_heart:
unit
1m 34s
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 55s
The patch does not generate ASF License warnings.

122m  8s

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1752

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint

uname
Linux 0ab4c7b0435b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 65c4660

Default Java
1.8.0_222

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/4/artifact/out/diff-checkstyle-root.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/4/testReport/

Max. process+thread count
1593 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/4/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 34s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
markdownlint
0m  0s
markdownlint was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s
The patch appears to include 2 new or modified test files.

_ trunk Compile Tests _

+0 :ok:
mvndep
0m 25s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
18m  6s
trunk passed

+1 :green_heart:
compile
17m 13s
trunk passed

+1 :green_heart:
checkstyle
2m 42s
trunk passed

+1 :green_heart:
mvnsite
2m 15s
trunk passed

+1 :green_heart:
shadedclient
18m 43s
branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
2m  9s
trunk passed

+0 :ok:
spotbugs
1m 12s
Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
3m 13s
trunk passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 24s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
1m 19s
the patch passed

+1 :green_heart:
compile
16m 34s
the patch passed

+1 :green_heart:
javac
16m 34s
the patch passed

-0 :warning:
checkstyle
2m 36s
root: The patch generated 3 new + 20 unchanged - 0 fixed = 23 total (was 20)

+1 :green_heart:
mvnsite
2m 15s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
12m 47s
patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
2m 13s
the patch passed

+1 :green_heart:
findbugs
3m 30s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
9m 10s
hadoop-common in the patch passed.

+1 :green_heart:
unit
1m 36s
hadoop-aws in the patch passed.

+1 :green_heart:
asflicense
0m 53s
The patch does not generate ASF License warnings.

119m  3s

Subsystem
Report/Notes

Docker
Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1752

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle markdownlint

uname
Linux 4a31fc63641e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / d12ad9e

Default Java
1.8.0_222

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/5/artifact/out/diff-checkstyle-root.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/5/testReport/

Max. process+thread count
1550 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1752/5/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Thanks @bgaborg for testing and providing update.
I can draft a release notes later, but to answer the question of what should we change to enable this:

We don't need to do anything (provide any configuration or change application code) if we don't want to enable server side encryption. The existing tables and configuration values will work as before because default values of this patch will keep existing behavior. To enable server side encryption, users can set fs.s3a.s3guard.ddb.table.sse.enabled as true. To specify a custom CMK, users can set her own CMK in configfs.s3a.s3guard.ddb.table.sse.cmk when fs.s3a.s3guard.ddb.table.sse.enabled is true.

I'll address Steve's other comments in a new commit, and post testing results with different config settings in auth-keys.xml including default SSE config (AWS owned CMK), AWS managed CMK and customer managed AWS key.


text:
HDDS-1151. Propagate the tracing id in ScmBlockLocationProtocol

The tracing propagation is not yet enabled for the ScmBlockLocationProtocol. We can't see the internal calls between OM and SCM. We need to propagate it at least for the allocateBlock call.
See: https://issues.apache.org/jira/browse/HDDS-1151

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
535
Docker mode activated.

_ Prechecks _

+1
@author
1
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

0
mvndep
79
Maven dependency ordering for branch

+1
mvninstall
1244
trunk passed

-1
compile
211
root in trunk failed.

+1
checkstyle
212
trunk passed

-1
mvnsite
20
server-scm in trunk failed.

-1
mvnsite
18
objectstore-service in trunk failed.

-1
mvnsite
16
ozone-manager in trunk failed.

+1
shadedclient
1076
branch has no errors when building and testing our client artifacts.

-1
findbugs
14
server-scm in trunk failed.

-1
findbugs
14
objectstore-service in trunk failed.

-1
findbugs
14
ozone-manager in trunk failed.

-1
javadoc
13
server-scm in trunk failed.

-1
javadoc
14
objectstore-service in trunk failed.

-1
javadoc
14
ozone-manager in trunk failed.

_ Patch Compile Tests _

0
mvndep
23
Maven dependency ordering for patch

-1
mvninstall
10
server-scm in the patch failed.

-1
mvninstall
11
objectstore-service in the patch failed.

-1
mvninstall
10
ozone-manager in the patch failed.

+1
compile
988
the patch passed

-1
cc
988
root generated 7 new + 2 unchanged - 0 fixed = 9 total (was 2)

-1
javac
988
root generated 660 new + 831 unchanged - 0 fixed = 1491 total (was 831)

+1
checkstyle
203
the patch passed

-1
mvnsite
23
server-scm in the patch failed.

-1
mvnsite
22
objectstore-service in the patch failed.

-1
mvnsite
19
ozone-manager in the patch failed.

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
711
patch has no errors when building and testing our client artifacts.

-1
findbugs
23
server-scm in the patch failed.

-1
findbugs
22
objectstore-service in the patch failed.

-1
findbugs
22
ozone-manager in the patch failed.

-1
javadoc
23
server-scm in the patch failed.

-1
javadoc
22
objectstore-service in the patch failed.

-1
javadoc
23
ozone-manager in the patch failed.

_ Other Tests _

-1
unit
86
common in the patch failed.

-1
unit
22
server-scm in the patch failed.

-1
unit
22
objectstore-service in the patch failed.

-1
unit
23
ozone-manager in the patch failed.

+1
asflicense
40
The patch does not generate ASF License warnings.

6161

Reason
Tests

Failed junit tests
hadoop.hdds.security.x509.certificate.client.TestDefaultCertificateClient

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/508

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle  cc

uname
Linux ad263b04c014 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 6cec906

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/branch-compile-root.txt

mvnsite
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/branch-mvnsite-hadoop-hdds_server-scm.txt

mvnsite
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/branch-mvnsite-hadoop-ozone_objectstore-service.txt

mvnsite
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/branch-mvnsite-hadoop-ozone_ozone-manager.txt

findbugs
v3.1.0-RC1

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/branch-javadoc-hadoop-hdds_server-scm.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/branch-javadoc-hadoop-ozone_objectstore-service.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/branch-javadoc-hadoop-ozone_ozone-manager.txt

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-mvninstall-hadoop-hdds_server-scm.txt

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-mvninstall-hadoop-ozone_objectstore-service.txt

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-mvninstall-hadoop-ozone_ozone-manager.txt

cc
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/diff-compile-cc-root.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/diff-compile-javac-root.txt

mvnsite
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-mvnsite-hadoop-hdds_server-scm.txt

mvnsite
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-mvnsite-hadoop-ozone_objectstore-service.txt

mvnsite
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-mvnsite-hadoop-ozone_ozone-manager.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-findbugs-hadoop-hdds_server-scm.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-findbugs-hadoop-ozone_objectstore-service.txt

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-findbugs-hadoop-ozone_ozone-manager.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-javadoc-hadoop-hdds_server-scm.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-javadoc-hadoop-ozone_objectstore-service.txt

javadoc
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-javadoc-hadoop-ozone_ozone-manager.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-unit-hadoop-hdds_common.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-unit-hadoop-hdds_server-scm.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-unit-hadoop-ozone_objectstore-service.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/artifact/out/patch-unit-hadoop-ozone_ozone-manager.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/testReport/

Max. process+thread count
305 (vs. ulimit of 5500)

modules
C: hadoop-hdds/common hadoop-hdds/server-scm hadoop-ozone/objectstore-service hadoop-ozone/ozone-manager U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-508/1/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

authors: elek hadoop-yetus


text:
HADOOP-18560. AvroFSInput opens a stream twice and discards the second one without closing

Description of PR
removes a needless line in AvroFSInput(final FileContext fc, final Path p which opens the path a second time and discards the result.
on input streams which immediately set up a connection with the store, this can leak connections.
Note, this is through the FileContext constructor, which is only used in production in org.apache.hadoop.fs.shell.Display, which being used from the command line, shouldn't be a long-lived process
How was this patch tested?
going to see what yetus says.
For code changes:

[X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

thanks chris. was my fault in the first place though -don't thank me too much


text:
fix comment word spelling error in  HRegionServer.java

Hello,
I found a word spelling error in HRegionServer . and this pr is used to fix it .
please review it.
thank you

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
3m 27s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m  3s
master passed

+1 :green_heart:
compile
2m 27s
master passed

+1 :green_heart:
checkstyle
0m 38s
master passed

+1 :green_heart:
spotless
0m 45s
branch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
1m 32s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 46s
the patch passed

+1 :green_heart:
compile
2m 26s
the patch passed

+1 :green_heart:
javac
2m 26s
the patch passed

+1 :green_heart:
checkstyle
0m 34s
the patch passed

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
4m 55s
Patch does not cause any errors with Hadoop 3.3.6.

+1 :green_heart:
spotless
0m 42s
patch has no errors when running spotless:check.

+1 :green_heart:
spotbugs
1m 47s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m  9s
The patch does not generate ASF License warnings.

31m 28s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5634/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5634

Optional Tests
dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile

uname
Linux 72c565e17f2e 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 6b0ce08c82

Default Java
Eclipse Adoptium-11.0.17+8

Max. process+thread count
81 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5634/1/console

versions
git=2.34.1 maven=3.8.6 spotbugs=4.7.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
4m 13s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m  3s
master passed

+1 :green_heart:
compile
0m 50s
master passed

+1 :green_heart:
shadedjars
5m  8s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 27s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 43s
the patch passed

+1 :green_heart:
compile
0m 53s
the patch passed

+1 :green_heart:
javac
0m 53s
the patch passed

+1 :green_heart:
shadedjars
5m  6s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 25s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
231m 28s
hbase-server in the patch passed.

259m 31s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5634/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5634

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 451c5d18d947 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 6b0ce08c82

Default Java
Eclipse Adoptium-11.0.17+8

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5634/1/testReport/

Max. process+thread count
4647 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5634/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Thanks for opening the PR.
HBase uses jira for tracking issues, please follow the guide on our readme page to register a jira account and file an issue for this PR.
https://github.com/apache/hbase?tab=readme-ov-file#how-to-contribute
Thanks.

authors: Apache-HBase Apache9 ZhongJinHacker

repo: apache/hbase


text:
HIVE-24192: Properly log TaskExecutorService eviction details

Change-Id: Ia552ea08ba61831f60e94f5a18913a9d87bb3385

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

cc: @abstractdog

repo: apache/hive


text:
[SUPPORT] org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem

Tips before filing an issue

Have you gone through our FAQs?

Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced
A clear and concise description of the problem.
To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.
Environment Description

Hudi version : 0.11.0

Spark version :3.2.1

Hive version :3.1.2

Hadoop version : 3.2.2

Storage (HDFS/S3/GCS..) : hdfs

Running on Docker? (yes/no) : no

Additional context
Add any other context about the problem here.
Stacktrace
Add the stacktrace of the error.
2022-04-28 17:19:05,955 INFO  org.apache.hadoop.io.compress.CodecPool                      [] - Got brand-new compressor [.gz]
2022-04-28 17:19:05,955 INFO  org.apache.flink.streaming.runtime.tasks.RegularOperatorChain [] - Could not complete snapshot 66 for operator stream_write -> Sink: clean_commits (1/1)#63. Failure reason: Checkpoint was declined.
org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 66 for operator stream_write -> Sink: clean_commits (1/1)#63. Failure reason: Checkpoint was declined.
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:265) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:170) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:348) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:233) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:206) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:186) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:605) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:315) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$14(StreamTask.java:1329) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1315) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1258) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:147) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.triggerCheckpoint(SingleCheckpointBarrierHandler.java:287) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.access$100(SingleCheckpointBarrierHandler.java:64) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler$ControllerImpl.triggerGlobalCheckpoint(SingleCheckpointBarrierHandler.java:493) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.triggerGlobalCheckpoint(AbstractAlignedBarrierHandlerState.java:74) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.barrierReceived(AbstractAlignedBarrierHandlerState.java:66) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.lambda$processBarrier$2(SingleCheckpointBarrierHandler.java:234) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.markCheckpointAlignedAndTransformState(SingleCheckpointBarrierHandler.java:262) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:231) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:496) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) [flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937) [flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766) [flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) [flink-dist_2.12-1.14.3.jar:1.14.3]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: java.lang.ClassCastException: org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
at org.apache.hudi.io.storage.HoodieParquetWriter.(HoodieParquetWriter.java:72) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.io.storage.HoodieFileWriterFactory.newParquetFileWriter(HoodieFileWriterFactory.java:84) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.io.storage.HoodieFileWriterFactory.newParquetFileWriter(HoodieFileWriterFactory.java:70) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.io.storage.HoodieFileWriterFactory.getFileWriter(HoodieFileWriterFactory.java:54) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.io.HoodieCreateHandle.(HoodieCreateHandle.java:101) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.io.HoodieCreateHandle.(HoodieCreateHandle.java:80) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.io.FlinkCreateHandle.(FlinkCreateHandle.java:67) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.io.FlinkCreateHandle.(FlinkCreateHandle.java:60) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.client.HoodieFlinkWriteClient.getOrCreateWriteHandle(HoodieFlinkWriteClient.java:479) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.client.HoodieFlinkWriteClient.upsert(HoodieFlinkWriteClient.java:143) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$1(StreamWriteFunction.java:184) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:461) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608) ~[?:1.8.0_282]
at org.apache.hudi.sink.StreamWriteFunction.flushRemaining(StreamWriteFunction.java:454) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.sink.StreamWriteFunction.snapshotState(StreamWriteFunction.java:131) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.hudi.sink.common.AbstractStreamWriteFunction.snapshotState(AbstractStreamWriteFunction.java:157) ~[blob_p-5b6e5de2921399080316dc7896ff5b9642acc0aa-120d6f609c46b048256bf770aa358086:0.11.0-rc1]
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:87) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:219) ~[flink-dist_2.12-1.14.3.jar:1.14.3]
... 33 more
2022-04-28 17:19:05,956 INFO  org.apache.hudi.async.AsyncCleanerService                    [] - Shutting down async clean service...
2022-04-28 17:19:05,956 WARN  org.apache.flink.runtime.taskmanager.Task                    [] - stream_write -> Sink: clean_commits (1/1)#63 (9a1c1c788f0890ec8536895ab4d266c0) switched from RUNNING to FAILED with failure cause: java.io.IOException: Could not perform checkpoint 66 for operator stream_write -> Sink: clean_commits (1/1)#63.
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1274)
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:147)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.triggerCheckpoint(SingleCheckpointBarrierHandler.java:287)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.access$100(SingleCheckpointBarrierHandler.java:64)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler$ControllerImpl.triggerGlobalCheckpoint(SingleCheckpointBarrierHandler.java:493)
at org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.triggerGlobalCheckpoint(AbstractAlignedBarrierHandlerState.java:74)
at org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.barrierReceived(AbstractAlignedBarrierHandlerState.java:66)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.lambda$processBarrier$2(SingleCheckpointBarrierHandler.java:234)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.markCheckpointAlignedAndTransformState(SingleCheckpointBarrierHandler.java:262)
at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:231)
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181)
at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159)
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:496)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 66 for operator stream_write -> Sink: clean_commits (1/1)#63. Failure reason: Checkpoint was declined.
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:265)
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:170)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:348)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:233)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:206)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:186)
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:605)
at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:315)
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$14(StreamTask.java:1329)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1315)
at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1258)
... 22 more
Caused by: java.lang.ClassCastException: org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to org.apache.hudi.common.fs.HoodieWrapperFileSystem
at org.apache.hudi.io.storage.HoodieParquetWriter.(HoodieParquetWriter.java:72)
at org.apache.hudi.io.storage.HoodieFileWriterFactory.newParquetFileWriter(HoodieFileWriterFactory.java:84)
at org.apache.hudi.io.storage.HoodieFileWriterFactory.newParquetFileWriter(HoodieFileWriterFactory.java:70)
at org.apache.hudi.io.storage.HoodieFileWriterFactory.getFileWriter(HoodieFileWriterFactory.java:54)
at org.apache.hudi.io.HoodieCreateHandle.(HoodieCreateHandle.java:101)
at org.apache.hudi.io.HoodieCreateHandle.(HoodieCreateHandle.java:80)
at org.apache.hudi.io.FlinkCreateHandle.(FlinkCreateHandle.java:67)
at org.apache.hudi.io.FlinkCreateHandle.(FlinkCreateHandle.java:60)
at org.apache.hudi.client.HoodieFlinkWriteClient.getOrCreateWriteHandle(HoodieFlinkWriteClient.java:479)
at org.apache.hudi.client.HoodieFlinkWriteClient.upsert(HoodieFlinkWriteClient.java:143)
at org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$1(StreamWriteFunction.java:184)
at org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:461)
at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
at org.apache.hudi.sink.StreamWriteFunction.flushRemaining(StreamWriteFunction.java:454)
at org.apache.hudi.sink.StreamWriteFunction.snapshotState(StreamWriteFunction.java:131)
at org.apache.hudi.sink.common.AbstractStreamWriteFunction.snapshotState(AbstractStreamWriteFunction.java:157)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:87)
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:219)
... 33 more

@Chosen123Wang To understand the issue, could you share the Hudi write configs and the steps and commands to reproduce the problem?  It looks like you're using Flink instead of Spark to write the data.

I have encountered such errors，this is because different classloader loads a same class，you need check you library

@Chosen123Wang After some investigation, this might be related to the class loading configuration in Java, as @YuangZhang pointed out.  I'll put up a patch for you to try out.

@Chosen123Wang Could you try this fix to see if it solves your issue: #5471?

@Chosen123Wang Could you try this fix to see if it solves your issue: #5471?

In my tests, some jobs can succeed and others can't. I would like to ask which class loader is loaded by the houdiewrapperfilesystem class. We want to specify it directly in the code

Hi, I also encountered the same problem. My approach here is

Modify the code in org.apache.hudi.io.storage.HoodieParquetWriter of hudi-client-common:
Comment out this variable directly HoodieWrapperFileSystem fs
[Before]
private final HoodieWrapperFileSystem fs;
[after]
//private final HoodieWrapperFileSystem fs;

Continue to modify the above 'fs' variable association method code in this class:

Comment out the 'fs' object reference code in the constructor of org.apache.hudi.io.storage.HoodieParquetWriter directly
[Before]
this.fs = (HoodieWrapperFileSystem) this.file.getFileSystem(FSUtils.registerFileSystem(file, parquetConfig.getHadoopConf()));
[after]
//this.fs = (HoodieWrapperFileSystem) this.file.getFileSystem(FSUtils.registerFileSystem(file, parquetConfig.getHadoopConf()));
Because this variable is not used for the time being, commenting it out will solve my problem, you can also try this method, I hope it can help you.

Hi, I also encountered the same problem. My approach here is

Modify the code in org.apache.hudi.io.storage.HoodieParquetWriter of hudi-client-common:
Comment out this variable directly HoodieWrapperFileSystem fs
[Before]
private final HoodieWrapperFileSystem fs;
[after]
//private final HoodieWrapperFileSystem fs;
Continue to modify the above 'fs' variable association method code in this class:

Comment out the 'fs' object reference code in the constructor of org.apache.hudi.io.storage.HoodieParquetWriter directly [Before] this.fs = (HoodieWrapperFileSystem) this.file.getFileSystem(FSUtils.registerFileSystem(file, parquetConfig.getHadoopConf())); [after] //this.fs = (HoodieWrapperFileSystem) this.file.getFileSystem(FSUtils.registerFileSystem(file, parquetConfig.getHadoopConf()));
Because this variable is not used for the time being, commenting it out will solve my problem, you can also try this method, I hope it can help you.

Hi, I also encountered the same problem. My approach here is

Modify the code in org.apache.hudi.io.storage.HoodieParquetWriter of hudi-client-common:
Comment out this variable directly HoodieWrapperFileSystem fs
[Before]
private final HoodieWrapperFileSystem fs;
[after]
//private final HoodieWrapperFileSystem fs;
Continue to modify the above 'fs' variable association method code in this class:

Comment out the 'fs' object reference code in the constructor of org.apache.hudi.io.storage.HoodieParquetWriter directly [Before] this.fs = (HoodieWrapperFileSystem) this.file.getFileSystem(FSUtils.registerFileSystem(file, parquetConfig.getHadoopConf())); [after] //this.fs = (HoodieWrapperFileSystem) this.file.getFileSystem(FSUtils.registerFileSystem(file, parquetConfig.getHadoopConf()));
Because this variable is not used for the time being, commenting it out will solve my problem, you can also try this method, I hope it can help you.

According to your solution, I tested it, the job is running normally, thank you for solving my problem。
What does this variable do? @yihua

It seems in latest master code, we already comment out the variable in HoodieBaseParquetWriter ?  cc @yihua

@Chosen123Wang I need to check the usage.  If it's not used, we should remove it.  The writer is used on the executor side so likely that HoodieWrapperFileSystem is somehow not on the classpath.

hi, The problem also appeared in my work,I use flink 1.15,hudi0.13.2,cloud you help me ?

It's not that easy to resolve, see a detaied analysis from Ken: https://github.com/apache/hudi/issues/8148, kind of flink classloader + hadoop fs bug here, what mode do you use for flink, the session cluster?

application

application

What classloader did you config? child-first or parent-first ?

+1, Using flink bulk_insert action, this exception still happen in Hudi 0.13.1 + flink1.14

Still catch this exception in Hudi 0.13.1 + flink 1.14 using bulk_insert operation

Thanks for the feedback @zhangyue19921010 , how you solve this issue finally?

Yeap, Will raise a PR to fix this issue today

Should be fixed via: https://github.com/apache/hudi/pull/9868

@danny0405 I am still seeing this exception when I try writing a COW type of table
flink 1.15.2
hudi 1.14.1
This is the stack trace
Caused by: java.lang.ClassCastException: class org.apache.hudi.common.fs.HoodieWrapperFileSystem cannot be cast to class org.apache.hudi.common.fs.HoodieWrapperFileSystem (org.apache.hudi.common.fs.HoodieWrapperFileSystem is in unnamed module of loader org.apache.flink.util.ChildFirstClassLoader @2d3c998e; org.apache.hudi.common.fs.HoodieWrapperFileSystem is in unnamed module of loader org.apache.flink.util.ChildFirstClassLoader @6f77e5b8)
	at org.apache.hudi.io.storage.row.HoodieRowDataParquetWriter.<init>(HoodieRowDataParquetWriter.java:51)
	at org.apache.hudi.io.storage.row.HoodieRowDataFileWriterFactory.newParquetInternalRowFileWriter(HoodieRowDataFileWriterFactory.java:79)
	at org.apache.hudi.io.storage.row.HoodieRowDataFileWriterFactory.getRowDataFileWriter(HoodieRowDataFileWriterFactory.java:55)
	at org.apache.hudi.io.storage.row.HoodieRowDataCreateHandle.createNewFileWriter(HoodieRowDataCreateHandle.java:220)
	at org.apache.hudi.io.storage.row.HoodieRowDataCreateHandle.<init>(HoodieRowDataCreateHandle.java:110)
	at org.apache.hudi.sink.bulk.BulkInsertWriterHelper.getRowCreateHandle(BulkInsertWriterHelper.java:129)
	at org.apache.hudi.sink.bulk.BulkInsertWriterHelper.write(BulkInsertWriterHelper.java:110)
	... 49 more

@ankit0811 How did you load the hudi flink bundle jar from the classpath then?

this is part of the pom (compile time)
Should this be added explicitly to the class path?

Then how do you submit the Flink application, the way Flink App jar loaded is very relevent with the classloader loadings.

We use helm charts to deploy flink standalone cluster. Post that for hudi POC we are uploading our custom hudi application jar manually
Is there any thing specific that you are looking for or we should take care of while deploying the cluster?

did you try the -j param or just putting the hudi-flink-bundle in the Flink lib dir.

Looks like it might just be flaky. Without changing a bit, now when I run the exact same pipeline, it succeeds :)
However, will definitely try what you suggested.
Thanks

source: gharchive/issue

repo: apache/hudi


text:
[HUDI-7297] Fix ambiguous error message when field type defined in sc…

…hema mismatches that in parquet file.
Change Logs
Fix ambiguous error message when field type defined in schema mismatches that in parquet file.
Impact
No impact for public API.
Risk level (write none, low medium or high below)
Low.
Documentation Update
There is no need to update the document.
Contributor's checklist

[x] Read through contributor's guide
[x] Change Logs and Impact were stated clearly
[x] Adequate tests were added if applicable
[ ] CI passed

@hudi-bot run azure


text:
[HUDI-875] Introduce a new pom module named hudi-common-sync

Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request
Introduce a new pom module named hudi-common-sync, before start abstract common meta sync module support multiple meta service. Design is in https://cwiki.apache.org/confluence/display/HUDI/RFC+-+17+Abstract+common+meta+sync+module+support+multiple+meta+service
Brief change log
Introduce a new pom module named hudi-common-sync, before start abstract common meta sync module support multiple meta service.
Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Codecov Report

Merging #1716 into master will decrease coverage by 54.04%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master    #1716       +/-   ##
=============================================
- Coverage     72.25%   18.21%   -54.05%
- Complexity      294      859      +565
=============================================
  Files           374      348       -26
  Lines         16371    15356     -1015
  Branches       1654     1523      -131
=============================================
- Hits          11829     2797     -9032
- Misses         3806    12201     +8395
+ Partials        736      358      -378

Impacted Files
Coverage Δ
Complexity Δ

.../java/org/apache/hudi/client/HoodieReadClient.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

.../java/org/apache/hudi/metrics/MetricsReporter.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

.../java/org/apache/hudi/common/model/ActionType.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

...java/org/apache/hudi/io/HoodieRangeInfoHandle.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

.../java/org/apache/hudi/hadoop/InputPathHandler.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

...a/org/apache/hudi/exception/HoodieIOException.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

...org/apache/hudi/table/action/commit/SmallFile.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

...rg/apache/hudi/index/bloom/KeyRangeLookupTree.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

...g/apache/hudi/exception/HoodieInsertException.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

...g/apache/hudi/exception/HoodieUpsertException.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

... and 335 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update acb1ada...48f17bc. Read the comment docs.

Good with the module.. Few thoughts/suggestions

Can we nest this under hudi-sync as hudi-sync-common, hudi-sync-hive and you can then introduce the aliyun metastore as hudi-sync-aliyun?
We need to just ensure there are no upgrades/backwards incompatible changes for existing users .

@vinothchandar What do you think about the updated PR

@umehrot2 @zhedoubushishi please weigh in with your thoughts as well

is the plan to eventually also move hudi-hive-sync under hudi-sync?

yes，when /hudi-sync/hudi-sync-common is ready, will move hudi-hive-sync under hudi-sync

Sorry for the delay. Was spending time on another large review.
Can we make those changes in this pr itself? Would love to see end-end functionality working. IIRC your ultimate goal is to plug in alliyun metastore?

s

Sorry for the delay. Was spending time on another large review.
Can we make those changes in this pr itself? Would love to see end-end functionality working. IIRC your ultimate goal is to plug in alliyun metastore?

ok , then i add the changes in this pr. aliyun metastore can plug is valuable for users.Now  users already use kafka -> spark -> hudi + aliyun metastore ->  presto + spark program.

Now users already use kafka -> spark -> hudi + aliyun metastore -> presto + spark program.

Thanks for letting me know.. Thats great! I am a fan of doing the refactor with a precise goal in mind.. so please feel free to reopen if you feel we need to restructure for something else.. AWS Glue also works with the current abstractions..


text:
[HUDI-3446][HUDI-FLINK] support batch Reader in BootstrapOperator#loadRecords

…dRecords
Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.

What is the purpose of the pull request
(For example: This pull request adds quick-start document.)
Brief change log
(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

hi @danny0405, pls review the pr :)

thx for your review @danny0405

HUDI-3446.patch.zip
Thanks, I have created a path for you to save review time, please validate the logic the test the correctness locally.

HUDI-3446.patch.zip Thanks, I have created a path for you to save review time, please validate the logic the test the correctness locally.

The PR is correct by TestParquetUtils#testFetchRecordKeyPartitionPathFromParquet

Did you notice that i have attached a patch for this pr? Please apply it then


text:
[HUDI-6606] Use record level index with SQL equality queries

Change Logs
With Record level support in HUDI, the sql queries related to record keys can leverage the Record Index. The Jira aims to add support for equality matches with record keys.
Impact
This should help improve performance for some sql equalTo queries on record keys.
Risk level (write none, low medium or high below)
low
Documentation Update
NA
Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

if I am not wrong, we have few follow ups on this regard right.
1: Get query predicates work w/ multi fields. if there are more than 1 query predicate where one of the column refers to primary key column, for primary key, we should look up in RLI, for others we can look up in col stats.
2: In addition to equality, we need to get "in" also supported.

and this works only for simple key gen right?
for any other key gen, unless we run through key generator, we can't look up in RLI as is. So, we do have some limitation here.

Created https://issues.apache.org/jira/browse/HUDI-6631 to track the followup items.


text:
Convert the Logical to Physical map to a visitor

I noticed that the FixedType was missing.

Thanks @rdblue for the quick review! 🙌

repo: apache/iceberg-python


text:
chore: Update reader api status

File based planning api is done, update the status in README.

cc @Fokko @Xuanwo @ZENOTME PTAL

repo: apache/iceberg-rust


text:
IGNITE-21967: Extend test coverage for SQL E091-06 (Set functions. ALL quantifier)

Adds references of this SQL feature to tests.
https://issues.apache.org/jira/browse/IGNITE-21967

Thank you for submitting the pull request.
To streamline the review process of the patch and ensure better code quality
we ask both an author and a reviewer to verify the following:
The Review Checklist

[ ] Formal criteria: TC status, codestyle, mandatory documentation. Also make sure to complete the following:
- There is a single JIRA ticket related to the pull request.
- The web-link to the pull request is attached to the JIRA ticket.
- The JIRA ticket has the Patch Available state.
- The description of the JIRA ticket explains WHAT was made, WHY and HOW.
- The pull request title is treated as the final commit message. The following pattern must be used: IGNITE-XXXX Change summary where XXXX - number of JIRA issue.
[ ] Design: new code conforms with the design principles of the components it is added to.
[ ] Patch quality: patch cannot be split into smaller pieces, its size must be reasonable.
[ ] Code quality: code is clean and readable, necessary developer documentation is added if needed.
[ ] Tests code quality: test set covers positive/negative scenarios, happy/edge cases. Tests are effective in terms of execution time and resources.

Notes

Apache Ignite Coding Guidelines

seems we need also to cover : SELECT sum(ALL s1), SELECT count(ALL ...

authors: lowka zstan


text:
Add Stackspace to airflow users list

Dear Airflow Maintainers,
Please accept this PR that addresses the following issues:

Add Stackspace to airflow users list

Per Apache guidelines you need to create a Jira issue.
Testing Done:

Unittests are required, if you do not include new unit tests please
specify why you think this is not required. We like to improve our
coverage so a non existing test is even a better reason to include one.

Reminders for contributors (REQUIRED!):

Your PR's title must reference an issue on
Airflow's JIRA.
For example, a PR called "[AIRFLOW-1] My Amazing PR" would close JIRA
issue #1. Please open a new issue if required!

For all PRs with UI changes, you must provide screenshots. If the UI changes are not obvious, either annotate the images or provide before/after screenshots.

Please squash your commits when possible and follow the How to write a good git commit message.
Summarized as follows:

Separate subject from body with a blank line
Limit the subject line to 50 characters
Do not end the subject line with a period
Use the imperative mood in the subject line (add, not adding)
Wrap the body at 72 characters
Use the body to explain what and why vs. how

Welcome to the community!

Current coverage is 65.95% (diff: 100%)

Merging #1884 into master will decrease coverage by 0.01%

@@             master      #1884   diff @@
==========================================
  Files           129        129
  Lines          9898       9898
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
- Hits           6529       6528     -1
- Misses         3369       3370     +1
  Partials          0          0

Powered by Codecov. Last update 7d9295a...a4661f3

Also, for future commits, pls follow the instructions on https://cwiki.apache.org/confluence/display/AIRFLOW/Contributors'+Guide

authors: codecov-io dbkegley r39132

repo: apache/incubator-airflow

license_source: bigquery


text:
Airflow 1639 Fix Fernet error handling

Dear Airflow maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
When the encrypted string cannot be decryted using Fernet for some reason, an error will be thrown. In Python 3.6.2 the .message attr is not available. By casting the ValueError to a string, the message will be extracted from the Error.
JIRA

[x] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-1639

Description

[x] Here are some details about my PR, including screenshots of any UI changes:

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Codecov Report

Merging #2629 into master will increase coverage by 0.09%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #2629      +/-   ##
==========================================
+ Coverage   71.14%   71.23%   +0.09%
==========================================
  Files         150      150
  Lines       11742    11742
==========================================
+ Hits         8354     8365      +11
+ Misses       3388     3377      -11

Impacted Files
Coverage Δ

airflow/models.py
87.15% <0%> (ø)
:arrow_up:

airflow/jobs.py
78.07% <0%> (+0.45%)
:arrow_up:

airflow/utils/helpers.py
56.32% <0%> (+2.87%)
:arrow_up:

airflow/task_runner/bash_task_runner.py
100% <0%> (+6.66%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d8c0a1f...60ed2e4. Read the comment docs.

authors: Fokko codecov-io


text:
[AIRFLOW-1908] Fix celery broker options config load

Make sure you have checked all steps below.
JIRA

[X] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-1908

Description

[X] Here are some details about my PR, including screenshots of any UI changes:

Tests

[X] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[X] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

[X] Passes git diff upstream/master -u -- "*.py" | flake8 --diff

cc: @FrankNagel @Fokko

Codecov Report

Merging #2867 into master will increase coverage by 0.02%.
The diff coverage is 93.33%.

@@            Coverage Diff            @@
##           master   #2867      +/-   ##
=========================================
+ Coverage   73.98%     74%   +0.02%
=========================================
  Files         160     160
  Lines       12153   12167      +14
=========================================
+ Hits         8991    9004      +13
- Misses       3162    3163       +1

Impacted Files
Coverage Δ

airflow/config_templates/default_celery.py
68% <ø> (ø)
:arrow_up:

airflow/configuration.py
84.54% <93.33%> (+0.53%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 02ff8ae...d100d1e. Read the comment docs.

Nice change, a good addition to the configuration framework. Personally I would only accept true instead of t to avoid ambiguity, but this is matter of taste I would say.

t/f is congruent with earlier (getboolean)

authors: Fokko bolkedebruin codecov-io


text:
[AIRFLOW-1191] Simplify override of spark submit command

This will better support distros which ship spark 1 & 2 ( and eventually 3)
Make sure you have checked all steps below.
Jira

[ X ] My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-XXX
In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

[ X ] Here are some details about my PR, including screenshots of any UI changes:

Adds a spark_binary param to the spark submit operator to allow folks to more easily configure the operator to use a different binary, as is needed for some distros of the Hadoop ecosystem which ship multiple version of Spark.
Tests

[ X ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Updates the existing test_spark_submit_operator to check for override
Commits

[ X ] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[ X ] In case of new functionality, my PR adds documentation that describes how to use it.

When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
All the public functions and the classes in the PR contain docstrings that explain what it does
docstring update

Code Quality

[ ] Passes flake8

Codecov Report

Merging #4360 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #4360   +/-   ##
=======================================
  Coverage   78.59%   78.59%
=======================================
  Files         204      204
  Lines       16453    16453
=======================================
  Hits        12932    12932
  Misses       3521     3521

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 51d0aca...3217c8c. Read the comment docs.

authors: codecov-io holdenk


text:
[BEAM-615] Add support for processing time timers in FlinkRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[x] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

R: @aljoscha or @mxm
I coded this very quickly without learning probably what I need to about the FlinkRunner or Flink. It seems to work for the naive examples I am experimenting with. Perhaps you can find errors and suggest corrections, or pull it and do it right?

Thanks for the PR @kennknowles. Processing time support definitely needs to be fixed. The changes look sensible to me but I would wait for @aljoscha opinion on this.

This would work but not in the presence of failures/recovery because timers need to be checkpointed, as we do with the event-time timers in WindowDoFnOperator.
I'm currently working on an addition to Flink that would provide user code with an interface that allows setting processing-time/event-time timers that would also be kept in a fault-tolerant way by the system. Maybe we should wait for that and then update WindowDoFnOperator to use that for both event-time timers and processing-time timers.

I see. I first considered duplicating the logic for event-time timers, but I had hoped the shortcut of using the underlying Flink timers would work. I will close this PR, leaving the open question of whether to emulate the event-time timer logic vs wait for the fault-tolerant Flink APIs.

I think this PR could be extended with not much work to checkpoint processing time timers in the snapshotOperatorState(..) method. It might be worth doing that because the next Flink release will only be available in a couple of months.

authors: aljoscha kennknowles mxm

repo: apache/incubator-beam


text:
Lazily create exception only when it needs to be thrown

Exceptions are expensive to create, so don't create a RuntimeException() unless we are actually going to throw something.

How about using Util.FoundOne.NULL, which is static?

Sounds good to me. I'll update the patch.

I've merged it to my https://github.com/julianhyde/incubator-calcite/tree/new-master branch. The pull request will get closed when I merge to apache master, after the release.

Thanks for the pull request!

authors: julianhyde mprudhom

repo: apache/incubator-calcite


text:
CALCITE-861 Refactor code location so compile/package work w/o instal…

…lation.

I don't like test as a top-level name much.
I expect more "test" modules might appear in the future, thus it would be nice if you could cut a more specific name.

I don't like test as a top-level name much.

Do you have a suggestion for something you might like more? core-test?

I expect more "test" modules might appear in the future

Hopefully not, but certainly possible.

Closing due to lack of interest in the change.

authors: joshelser vlsi


text:
[CELEBORN-1142] clear shuffleIdCache in shutdown method of ShuffleClientImpl

What changes were proposed in this pull request?
As Title
Why are the changes needed?
As Title
Does this PR introduce any user-facing change?
NO
How was this patch tested?
PASS GA

@ErikFang  @waitinfuture  please cc

thanks, merging to main(v0.4.0).

authors: cfmcgrady jiaoqingbo

repo: apache/incubator-celeborn


text:
Deep copy VariableMap when DataProcessor is copied

This copy needs to happen now that VariableMap is mutable, so sharing
the VariableMap with other processors that use/set the same variables
can cause issues. Deep copying the VariableMap resolves this issue.
DAFFODIL-2342

Test has been added and another similar potential issue was discovered by the test and fixed

repo: apache/incubator-daffodil


text:
[Bug][Jenkins] MavenModuleSet should be treated as a scope

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
The logic for categorizing Devlake Remote Scopes has changed in version v1.0.0-beta5. Previously, the default category was "scope," but now it is "group." This change has led to issues, such as hudson.maven.MavenModuleSet Jenkins project type being treated as a group instead of a scope. As a result, you cannot add a new Devlake Remote Scope because it's incorrectly categorized as a group.
What do you expect to happen
I expect that hudson.maven.MavenModuleSet Jenkins project is not categorized as group by Devlake
How to reproduce
Create a new Maven Project in Jenkins and try to add it through Devlake:

Anything else
No response
Version
v1.0.0-beta11
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Lovely

I like the report-and-fix vibe here. 👍

repo: apache/incubator-devlake


text:
[fix-5535]: raw_data_params structural and query change for safer deletion

Summary
What does this PR do?

Address the requirments of #5535
Updates the e2e test CSV files to include the new "plugin" entry in _raw_data_params; hence the large number of file changes.

Does this close any open issues?
Closes #5535
Closes #5520
Screenshots
Include any relevant screenshots here.

Other Information
Any other information that is important to this PR.

Closing in favor of #5551


text:
[BUG] resource download error

when resource file is under sub dir ,download goes error.

which version or branch?

I'll fix it, thanks

please describle detail . which branch ? detail problem ? Thx

authors: JinyLeeChina frankYu001 qiaozhanwei xingchun-chen

repo: apache/incubator-dolphinscheduler


text:
Dolphinscheduler Complete the installation Unable to initialize login

初始用户：admin
初始密码：dolphinscheduler123

please update the title in English

please check the api log

问题已解决


text:
[Improvement][UI] Improve parameter name

Tips

Thanks very much for contributing to Apache DolphinScheduler.
Please review https://dolphinscheduler.apache.org/en-us/community/index.html before opening a pull request.

What is the purpose of the pull request
Improve parameter name
Brief change log

Rename Command-line parameters to Main arguments(主程序参数)
Rename Other parameters to Option parameters(选项参数)

Verify this pull request
This change added tests and can be verified as follows:
(example:)

Manually verified the change by testing locally.

@break60 PTAL.Thx

authors: CalvinKirs chengshiwen


text:
[1.3.6-prepare][Improvement][Docker] Add workdir for dockerfile and remove redundant container_name and dolphinscheduler-postgresql-initdb

Brief change log

[1.3.6-prepare][Improvement][Docker] Add workdir for dockerfile
[1.3.6-prepare][Improvement][Docker] Remove redundant container_name and dolphinscheduler-postgresql-initdb

Removing container_name is convenient to scale master or worker
Like docker-compose up -d --scale dolphinscheduler-worker=3

@dailidong @CalvinKirs @xingchun-chen Please review🙏. This pr solves the scalability of master and worker, and I have tested it.
Removing container_name is convenient to scale master or worker like docker-compose up -d --scale dolphinscheduler-worker=3

+1

authors: chengshiwen xingchun-chen


text:
[fix] fix compile error

Proposed changes
Issue Number: close #xxx
Problem Summary:
fix compile error
Checklist(Required)

Does it affect the original behavior: (Yes/No/I Don't know)
Has unit tests been added: (Yes/No/No Need)
Has document been added or modified: (Yes/No/No Need)
Does it need to update dependencies: (Yes/No)
Are there any changes that cannot be rolled back: (Yes/No)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

merge it to fix master compile error

repo: apache/incubator-doris


text:
折线图 tooltip 溢出边界 可不可以scrollable （How to make the echarts tooltip scrollable? ）

What problem does this feature solve?
触发方式为axis时，如果tooltip中的数据太多，导致溢出边界，scrollable可以滚动查看；
What does the proposed API look like?
可以给tooltip设置一个高度，超过这个高度，让其中的内容scroll.

BROWSER_VERSION：google chrome 72.0.3626.121
浏览器版本：谷歌浏览器 72.0.3626.121

Tooltip is an HTML essentially, you can do anything just like you do with HTML.

可以实现
但是要将tooltips的触发triggerOn改为click
然后tooltips的formatter回调的html中设置height和overflow：scroll
即可实现

tooltips的触发triggerOn不影响overflow效果
但是position位置要调一下
即在鼠标移到tooltips框前
tooptip框不能消失

authors: Aslantoe HowToMeetYou deqingli

repo: apache/incubator-echarts


text:
Y轴为对数轴，柱状图数据为堆叠的，Y轴的刻度跟数据对不起来

This issue is not created by echarts-issue-helper and will be soon closed.


text:
在 linux下 使用 echarts 生成的 图片中文字体乱码 请问各位遇到过这种情况吗

在 linux下 使用 echarts 生成的 图片中文字体乱码 请问各位遇到过这种情况吗?

In the images downloaded from toolbox, or with chart.getDataURL()?

authors: Hysteriachen Ovilia


text:
[GEARPUMP-276] Remove ununsed 'shade' folder

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the commit message is formatted like:
[GEARPUMP-<Jira issue #>] Meaningful description of pull request
[ ] Make sure tests pass via sbt clean test.
[x] Make sure old documentation affected by the pull request has been updated and new documentation added for new functionality.

Codecov Report

Merging #153 into master will decrease coverage by -0.19%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #153      +/-   ##
==========================================
- Coverage   71.75%   71.57%   -0.19%
==========================================
  Files         191      191
  Lines        6068     6068
  Branches      522      522
==========================================
- Hits         4354     4343      -11
- Misses       1714     1725      +11

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a996d39...4a9a365. Read the comment docs.

+1, merging

authors: codecov-io huafengw manuzhang

repo: apache/incubator-gearpump


text:
[DNM] Test window optimization in Velox

What changes were proposed in this pull request?
https://github.com/facebookincubator/velox/pull/9025
How was this patch tested?
Existing tests

@zhztheplayer The CI is passed. Please help to pick this commit into oap velox in next round rebase. Thanks.

Committed Velox change in https://github.com/oap-project/velox/tree/2024_04_09

authors: JkSelf zhztheplayer

repo: apache/incubator-gluten


text:
HAWQ-1359. Fix bugs in policy tests for HAWQ with Ranger enabled and …

…add test for HAWQ-1367

+1

+1

repo: apache/incubator-hawq


text:
[INLONG-2137] version 0.12.0 cannot pass UT in devcloud environment, it is ok in github check flow. Environment adapter problem.

Title Name: [INLONG-2137][Inlong-DataProxy] version 0.12.0 cannot pass UT in devcloud environment, it is ok in github check flow. Environment adapter problem.
Fixes #2137
Motivation
Explain here the context, and why you're making that change. What is the problem you're trying to solve.
Modifications
Describe the modifications you've done.
Verifying this change

[ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

Codecov Report

Merging #2153 (1b6c573) into master (8a87b5d) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #2153   +/-   ##
=========================================
  Coverage     12.27%   12.27%
  Complexity     1157     1157
=========================================
  Files           413      413
  Lines         35243    35243
  Branches       5544     5544
=========================================
  Hits           4325     4325
  Misses        30147    30147
  Partials        771      771

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8a87b5d...1b6c573. Read the comment docs.

authors: codecov-commenter luchunliang

repo: apache/incubator-inlong


text:
DMN Header cell properties loses focus after one character is typed

There is an issue in the DMN Editor. Once user tries to set properties of the header cell, he/she is not able to write down more than one character. Once the one character is typed, the focus is lost.
Steps to reproduce

Open a DMN Diagram
Add a decision node
Open the decision node
Set its type as 'context'
open the properties panel
select the context header cell
try to fill in 'Allowed answers' using keyboard

Could you please add a video of the current behavior?

@ljmotta sure, will add it into description

authors: jomarko ljmotta

repo: apache/incubator-kie-issues


text:
[KYUUBI #1449][BACKPORT][FOLLOWUP] Fix import

Why are the changes needed?
Fix import to pass CI
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

Codecov Report

Merging #1551 (b2f9310) into branch-1.4 (3ff80cf) will increase coverage by 0.00%.
The diff coverage is 69.23%.

@@              Coverage Diff              @@
##             branch-1.4    #1551   +/-   ##
=============================================
  Coverage         59.02%   59.03%
  Complexity          172      172
=============================================
  Files               233      233
  Lines             11999    12014   +15
  Branches           1474     1476    +2
=============================================
+ Hits               7083     7093   +10
- Misses             4314     4320    +6
+ Partials            602      601    -1

Impacted Files
Coverage Δ

.../scala/org/apache/kyuubi/server/KyuubiServer.scala
58.66% <25.00%> (-2.45%)
:arrow_down:

...he/kyuubi/engine/spark/shim/CatalogShim_v2_4.scala
76.19% <50.00%> (-2.15%)
:arrow_down:

...ommon/src/main/scala/org/apache/kyuubi/Utils.scala
66.33% <75.00%> (+0.74%)
:arrow_up:

...apache/kyuubi/ctl/ServiceControlCliArguments.scala
77.91% <80.00%> (+1.69%)
:arrow_up:

.../engine/spark/session/SparkSQLSessionManager.scala
80.95% <100.00%> (+0.46%)
:arrow_up:

...rg/apache/kyuubi/metrics/JsonReporterService.scala
86.66% <100.00%> (ø)

...g/apache/kyuubi/session/KyuubiSessionManager.scala
97.29% <100.00%> (+0.07%)
:arrow_up:

...ache/kyuubi/engine/spark/SparkProcessBuilder.scala
84.00% <0.00%> (-3.00%)
:arrow_down:

...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala
91.00% <0.00%> (+1.00%)
:arrow_up:

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d22d616...b2f9310. Read the comment docs.

repo: apache/incubator-kyuubi


text:
Problems of bucking iterator in speech example

The bucking iterator cannot read all the training data, which will cause the evaluation being significantly different.

@AprilSongRits Can you give more details of this problem. Like a reproducible simple example will be great. Thanks!

@sandeep-krishnamurthy @nswamy @anirudh2290 Could you please close the issue due to lack of activity.
@AprilSongRits Please feel free to re-open the issue in case closed in error.

@mxnet-label-bot add [suggest-closed]

closing until requester reopens

authors: AprilSongRits ankkhedia apeforest cjolivier01 kalyc

repo: apache/incubator-mxnet


text:
Example link of the image classification show 404

I keep getting error 404 when I try to read the image classification examples--https://mxnet.apache.org/tutorials/python/predict_image.html

True link should be -- https://github.com/dmlc/mxnet-notebooks/blob/master/python/tutorials/predict_imagenet.ipynb

@zachgk assign @TEChopra1000

@TEChopra1000 Please comment if you want to take this issue.

@zachgk  I will take the issue.

@stereomatchingkiss would you please point me to the page you were on when you found the broken link?

In https://github.com/apache/incubator-mxnet/pull/16764, this image classification link was removed. @stereomatchingkiss @TEChopra1000


text:
get the names of different outputs of the same layer

When a layer has more than one output, how to get the names of different outputs?
For example, a custom layer outputs two symbols, when using get_internal(), they usually have the same name.

@gaosanyuan please ask this question in the discussion forum here - https://discuss.mxnet.io/
@sandeep-krishnamurthy please add the label - Question and close this issue.

authors: anirudhacharya gaosanyuan


text:
[MXNET-618] Clojure package - move NDArray and Symbol Code generation to Automatic

Description
From feedback from the Clojure package PR https://github.com/apache/incubator-mxnet/pull/11205, this changes the automatic generation of the ndarray and symbol code from a manual step to an automatic compile-time (AOT) step. The generated code is no longer checked into git as well.
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[x] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[x] Changes are complete (i.e. I finished coding on this PR)
[x] All changes have test coverage:
[x] Code is well-documented:
[x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Move ndarray and symbol code generation to compile time
Remove generated code files from git
Remove the alias to generate-code from the project file since it is now automatic

Testing
To test this change:
with lein test

Make sure there are no files in the src/org/apache/clojure_mxnet/gen directory
Run lein test
Tests should run fine and the generated files should be present in src/org/apache/clojure_mxnet/gen but ignored by git

with jars

Make sure there are no files in the src/org/apache/clojure_mxnet/gen directory
Run lein install to create target jar
cd examples/imclassification
run lein run to run the example with the installed jar

Hey Carin, I can't review your PR on my phone due to the number of deleted files. Please excuse if this is already answered.
The two processes you described, are they already part of CI or how are we testing it? If something takes a long time, I'm happy to add it to nightly.

@marcoabreu - I put manual testings instructions there. The first part with lein test is what the CI does. The second part with jars is not covered by CI right now. It is creating the jars and then using them in another project. Eventually this is what I hope integration tests to cover.
For this PR, if CI tests are green, we are good to go.
If someone else wants to test out the with jars step manually, that would be great too.

authors: gigasquid marcoabreu


text:
[v1.4.x] Disable flaky test test_random:test_randint_generator

…6 (#13480)
Description
Disable flaky test
Skip flaky test https://github.com/apache/incubator-mxnet/issues/1344…
[Temporary fix]

@mxnet-label-bot add [Test, pr-awaiting-review]

authors: ChaiBapchya vandanavk


text:
[DON'T MERGE]PoC for Conversion from FP32 to mixed precision model

Description
This is a PoC for https://github.com/apache/incubator-mxnet/issues/14584
prereq is the AMP PR: https://github.com/apache/incubator-mxnet/pull/14173
I have pulled some of the operator code for amp_cast and amp_multicast etc. into my branch for now.
Approach

Currently, the pass uses the fp16_op_names, fp32_op_names and widest_op_names to make decisions on operators to insert in the computation graph.
The logic for casting currently follows the PR: #14173.
For fp16_ops an amp_cast with target_dtype of fp16 is inserted before the op, fp32 is inserted before op for fp32 ops,  for widest_type_casts, amp_multicast is used.
The code is currently specific to fp16 and fp32 ops but i plan to change it to use target_dtype, low_precision_ops, full_precision_ops.

Testing
Test script for the PoC:
import mxnet as mx

data = mx.sym.var("data")
data2 = mx.sym.var("data2")
data3 = mx.sym.var("data3")
x = mx.sym.exp(data)
x2 = mx.sym.sin(data)
x3 = mx.sym.cos(data)
sym = x + x2 + x3
result = mx.sym.add_n(sym, data2, data3)
x = mx.viz.plot_network(result)
casted_result = mx.contrib.amp._convert_symbol(result, fp32_op_names=["elemwise_add"], fp16_op_names=["sin", "cos", "exp"], widest_type_op_names=["add_n"])
y = mx.viz.plot_network(casted_result)
x.render('test-output/round-table.gv', view=False)
y.render('test-output/round-table.gv', view=False)

Model before:

Model after:

@ptrendx @DickJC123 @Caenorst @ZhennanQin @pengzhao-intel

Closing this Pull Request. Final PR: https://github.com/apache/incubator-mxnet/pull/15118


text:
Update submodule dmlc-core

Description
This PR advances the 3rdparty dmlc-core submodule by the following 2 commits:
ca9f932  2019-11-05  Dick Carter            Fix initial-release nvcc v10.1 issue with thread_local.h (#579)
05fb5ad  2019-10-24  Philip Hyunsu Cho      Make yarn.py and launcher.py compatible with Python 3 (#564)

Importantly, this pulls in dmlc-core PR https://github.com/dmlc/dmlc-core/pull/579, which is the work-around for MXNet issue https://github.com/apache/incubator-mxnet/issues/16612.  Once this is merged, an MXNet build against the initial release of cuDNN v10.1 will not fail.  The version of dmlc-core included as a submodule of 3rdparty/tvm will not change of course (it depends on the tvm submodule hash), but this is not causing a compile problem for MXNet.
@larroy
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[X ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

preferrably this should also go in 1.6 if we are still taking bug fixes.