text:
[FLINK-8933] Avoid calling Class#newInstance

What is the purpose of the change
This pull request refactored the usage of Class#newInstance
Brief change log

Replaced Class#newInstance to Class#newInstance#getDeclaredConstructor#newInstance

Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
This change is already covered by existing tests*.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

cc @StephanEwen & @zentol just one commit~

The discussion on the JIRA clearly came tot he conclusion to do this change in 2 commits. Please separate the PR.

OK now closing this and create two new PRs...

authors: yanghua zentol


text:
[FLINK-1933] Add distance measure interface and basic implementation to machine learning library

This PR contains following changes:

Add dot method and magnitude method.
Add DistanceMeasure trait.
Add 7 basic implementation of DistanceMeasure.
Add tests for above changes.

Hi @chiwanpark. Really good work :-) I had only some minor comments.
One think to think about is where we place the distance measures. Currently it is in the flink.ml.common.distance package. Maybe we could put them in a metrics.distances package instead. We could either put it directly under flink.ml or under flink.ml.math. What do you think?

@tillrohrmann Oh. flink.ml.math.metrics.distances is much better. I will move all classes into the package. I will also fix some problem in dot product and test cases for that.

Well done @chiwanpark.
The only thing which is missing is an overview page with all supported distance measures for the Flink website. Could you add such a site? You find in flink/docs/libs/ml/ examples for other ML documentation. If you write 2-3 sentences and list all the different distance measures, then it would be perfect :-) I will then merge the PR.

@tillrohrmann Okay. I'll add them and update PR. :)

I added a overview documentation. Please review and comment if there are some errors.

@chiwanpark, looks really good. I had only one comment concerning the cosine distance.
Furthermore, I was wondering whether we should rename DistanceMeasure into Distance or DistanceMetric because in all the ScalaDocs we say "The XYZ distance metric ....". What do you think?
After clarifying these questions, I think it is good to be merged :-)

Renaming into DistanceMetric is great idea. :) I'll rename DistanceMeasure into DistanceMetric.

Ping me once you've renamed the classes, then I'll merge the PR.

@tillrohrmann I just renamed the classes. :)

Oh, the commit logs contains distance measure. I will fix them.

I updated them.

Great. I'll merge it then.

source: gharchive/pull-request

authors: chiwanpark tillrohrmann

repo: apache/flink

license_type: permissive

license_source: github-api


text:
[DO NOT MERGE][FLINK-11710][tests] Refactor SimpleSlotProvider to TestingLogicalSlo…

This change originally for FLINK-10569 and now I can see @leesf took over the successors like FLINK-12181. DO NOT MERGE since this issue possibly become invalid soon.
What is the purpose of the change
Refactor SimpleSlotProvider to TestingLogicalSlotProvider.
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector:(no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

cc @tillrohrmann @zentol @GJL

@tillrohrmann I think this refactor is ready to be reviewed. Changes are minor and it would be good if you allocate a bit to review it :-)

Hi @TisonKun, what is the status here? ExecutionGraphRestartTest looks like still using SimpleSlotProvider in #8226. Do you think we do not need this refactoring? If so, could you close this PR/issue? Otherwise, what do you think about class naming comments?

@azagrebin
SimpleSlotProvider based on SimpleSlot which is a legacy resource manage concept. We still need to remove SimpleSlot as part of FLINK-10392.
I'd like to close this pr but remain the issue because of my inactive in following days. About the naming comments, I agree with TestingSlotContext and TestingSlotProvider in test package.

authors: TisonKun azagrebin


text:
[FLINK-2410] [java api] PojoTypeInfo is not completely serializable

The type java.reflect.Field is not Serializable and has to be serialized manually, which was not implemented until now in PojoTypeInfo.

Thank you for the PR.
Can you add a test case validating the fix?

The lines are basically copied from PojoComparator but yes I can add a test case for it ;)

I would just use the Table API example you've posted in the JIRA and add another IT case for the Table API

I have created a separate test. But the idea is good. I will create a Table API example from my current code.

Why do we need to serialize the TypeInformation in the first place?

After merging this, we should update CsvInputFormat. Currently, we create Field objects in open method of CsvInputFormat. With this PR, we don't need to create Field manually.

The non-serializable problem of the PojoTypeInfo has been fixed in #1134. Thus, the PR can be closed.

I think more tests for the Table API can not harm. I will adapt the PR after #1134 is merged.

The test is included in the PR #1134.

So this PR can be closed?

authors: StephanEwen chiwanpark fhueske rmetzger tillrohrmann twalthr


text:
GEODE-5407: fix JMXMBeanReconnectDUnitTest timing

Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[x] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[x] Is your initial contribution a single, squashed commit?

[x] Does gradlew build run cleanly?

[x] Have you written or updated unit tests to verify your changes?

[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

Looks good and precheckin is green. Huzzah!

repo: apache/geode


text:
gfsh help now uses log4j log levels: update user guide

GEODE-2778: Where gfsh documentation listed log level params, replaced the "old" levels (fine, finer, finest, etc) with log4j levels (ALL, TRACE, DEBUG, etc).
Did not replace log level params in non-gfsh contexts (e.g. Java command lines or xml config files).
Also did not standardize "loglevel" vs "log-level", which are inconsistently used in the gfsh interface (see GEODE-2850).

PR associated itself with a secondary ticket. Will re-post with an improved comment.


text:
GEODE:8042- Create Integrated Distributed/HA test for Redis Module/Sp…

…ring Session
Authored-by: Jens Deppe jdeppe@vmware.com
Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[ ] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[ ] Is your initial contribution a single, squashed commit?

[ ] Does gradlew build run cleanly?

[ ] Have you written or updated unit tests to verify your changes?

[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

Note:  currently intended only for internal team review-- PLEASE DO NOT MERGE UNTIL ADDITIONAL NOTE IS ENTERED CONFIRMING OK TO MERGE--  thanks :) -J

Note: currently intended only for internal team review

@jhutchison - GitHub has draft PRs, which can be enabled by clicking the Convert to draft link under the reviewers on the right.  It will still run the PR checks, just won't allow merging til it is no longer in draft state

authors: jhutchison moleske


text:
GEODE-8763: run benchmark tests up to 5 times

Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[x] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[x] Is your initial contribution a single, squashed commit?

[x] Does gradlew build run cleanly?

[x] Have you written or updated unit tests to verify your changes?

[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.

this looks better. although I think lines 85-97 could be simplified to just:
set -e
./run_against_baseline.sh ...
set +e

Not really. -e causes errors to bubble up and exit the script, which in this case we want to avoid to loop 5 times, but only on the execution of this line. The logic for checking the iteration found could be simplified if we captured the error code and exited with that code after the 5th iteration.
exit_code=0
for ... {
  ...
  set +e
  ./run_against...
  set -e
  exit_code = $?

  if [[ $exit_code -eq 0 ]]; then
    break;
  fi
}
...
exit $exit_code

authors: nonbinaryprogrammer pivotal-jbarrett


text:
Persist the order of methods

to ensure the generated trait classes are the same for the same source code.
This is a one of the possible solutions to solve GROOVY-10704.
see #1753 for an alternative version which only sorts the methods when generating the trait

This is the approach I would take to address the issue. I'll check into the errors - one looks explainable at first glance.

Thank you @paulk-asert!

Merged, thanks!

authors: musketyr paulk-asert

repo: apache/groovy


text:
GUACAMOLE-926: Batch Import: Attempt to correct incorrectly-cased parameter values if possible

Yet another batch import bugfix / improvement!? Nobody could have seen this coming!
This change does two things:

Fixes an issue where file-level CSV errors didn't get surfaced to the user
Attempts to correct any invalidly-cased user-provided connection parameter values, when the parameter has a limited number of possible options.

The latter is intended to help out with the case where a user may, while editing a CSV in many popular spreadsheets, have their "true" values for boolean parameters converted to "TRUE", which the backend will not understand.

Looks good - only issues I see are a couple typos.

Spelling is hard

repo: apache/guacamole-client


text:
HADOOP-17840: Backport HADOOP-17837 to branch-3.2

https://issues.apache.org/jira/browse/HADOOP-17840

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
8m 18s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ branch-3.2 Compile Tests _

+1 :green_heart:
mvninstall
25m 57s

branch-3.2 passed

+1 :green_heart:
compile
15m  8s

branch-3.2 passed

+1 :green_heart:
checkstyle
0m 50s

branch-3.2 passed

+1 :green_heart:
mvnsite
1m 21s

branch-3.2 passed

+1 :green_heart:
javadoc
1m 12s

branch-3.2 passed

+1 :green_heart:
spotbugs
2m  7s

branch-3.2 passed

+1 :green_heart:
shadedclient
14m 18s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 49s

the patch passed

+1 :green_heart:
compile
14m 17s

the patch passed

+1 :green_heart:
javac
14m 17s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 48s

the patch passed

+1 :green_heart:
mvnsite
1m 20s

the patch passed

+1 :green_heart:
javadoc
1m 10s

the patch passed

+1 :green_heart:
spotbugs
2m 15s

the patch passed

+1 :green_heart:
shadedclient
14m 40s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
15m 45s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 51s

The patch does not generate ASF License warnings.

120m 41s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3275/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3275

JIRA Issue
HADOOP-17840

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 94246bdcc202 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
branch-3.2 / 71b07f76a9b2d33fefe01706e3fe47eb47bbb18b

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3275/1/testReport/

Max. process+thread count
1391 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3275/1/console

versions
git=2.17.1 maven=3.6.0 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

LGTM.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 43s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ branch-3.2 Compile Tests _

+1 :green_heart:
mvninstall
26m 39s

branch-3.2 passed

+1 :green_heart:
compile
15m  3s

branch-3.2 passed

+1 :green_heart:
checkstyle
0m 50s

branch-3.2 passed

+1 :green_heart:
mvnsite
1m 22s

branch-3.2 passed

+1 :green_heart:
javadoc
1m 14s

branch-3.2 passed

+1 :green_heart:
spotbugs
2m  7s

branch-3.2 passed

+1 :green_heart:
shadedclient
14m 36s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 48s

the patch passed

+1 :green_heart:
compile
14m 13s

the patch passed

+1 :green_heart:
javac
14m 13s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 48s

the patch passed

+1 :green_heart:
mvnsite
1m 20s

the patch passed

+1 :green_heart:
javadoc
1m  8s

the patch passed

+1 :green_heart:
spotbugs
2m 15s

the patch passed

+1 :green_heart:
shadedclient
14m 36s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
15m 52s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 50s

The patch does not generate ASF License warnings.

114m  3s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3275/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/3275

JIRA Issue
HADOOP-17840

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell

uname
Linux 461dc989ff37 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
branch-3.2 / bfd7b946bffddad1bff4894f7c6e37a377f5ca24

Default Java
Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3275/2/testReport/

Max. process+thread count
1388 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3275/2/console

versions
git=2.17.1 maven=3.6.0 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

repo: apache/hadoop


text:
HADOOP-19038. Improve create-release RUN script.

Description of PR
JIRA: HADOOP-19038. Improve create-release RUN script.
Using create-release will create a docker image locally, but three of the RUN scripts may fail to execute.

RUN groupadd --non-unique -g 0 root

=> ERROR [16/20] RUN groupadd --non-unique -g 0 root                                                            0.2s
------
 > [16/20] RUN groupadd --non-unique -g 0 root:
0.154 groupadd: group 'root' already exists
------
Dockerfile:100
--------------------
  98 |
  99 |     LABEL org.apache.hadoop.create-release="cr-19697"
 100 | >>> RUN groupadd --non-unique -g 0 root
 101 |     RUN useradd -g 0 -u 0 -m root
 102 |     RUN chown -R root /home/root

RUN useradd -g 0 -u 0 -m root

 > [17/20] RUN useradd -g 0 -u 0 -m root:
0.165 useradd: user 'root' already exists
------
Dockerfile:101
--------------------
  99 |     LABEL org.apache.hadoop.create-release="cr-12068"
 100 |     RUN groupadd --non-unique -g 0 root; exit 0;
 101 | >>> RUN useradd -g 0 -u 0 -m root
 102 |     RUN chown -R root /home/root
 103 |     ENV HOME /home/root

RUN chown -R root /home/root

 > [18/20] RUN chown -R root /home/root:
0.168 chown: cannot access '/home/root': No such file or directory
------
Dockerfile:102
--------------------
 100 |     RUN groupadd --non-unique -g 0 root; exit 0;
 101 |     RUN useradd -g 0 -u 0 -m root; exit 0;
 102 | >>> RUN chown -R root /home/root
 103 |     ENV HOME /home/root
 104 |     RUN mkdir -p /maven
--------------------

Even if these three scripts fail, subsequent steps can continue to be executed, so I added exit 0 after the script.

After the modification is completed, the script can be executed normally:

> [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 2B                                                                                0.0s
 => [internal] load build definition from Dockerfile                                                                                0.0s
 => => transferring dockerfile: 3.26kB                                                                                0.0s
 => [internal] load metadata for docker.io/library/ubuntu:focal                                                                                1.5s
 => [ 1/20] FROM docker.io/library/ubuntu:focal@sha256:f2034e7195f61334e6caff6ecf2e965f92d11e888309065da85ff50c617732b8                                                                                0.0s
 => [internal] load build context                                                                                0.0s
 => => transferring context: 997B                                                                                0.0s
 => CACHED [ 2/20] WORKDIR /root                                                                                0.0s
 => CACHED [ 3/20] RUN echo APT::Install-Recommends "0"; > /etc/apt/apt.conf.d/10disableextras                                                                                0.0s
 => CACHED [ 4/20] RUN echo APT::Install-Suggests "0"; >>  /etc/apt/apt.conf.d/10disableextras                                                                                0.0s
 => CACHED [ 5/20] COPY pkg-resolver pkg-resolver                                                                                0.0s
 => CACHED [ 6/20] RUN chmod a+x pkg-resolver/*.sh pkg-resolver/*.py     && chmod a+r pkg-resolver/*.json                                                                                0.0s
 => CACHED [ 7/20] RUN apt-get -q update     && apt-get -q install -y --no-install-recommends python3     && apt-get -q install -y --no-install-recommends $(pkg-resolver/resolve.py ubuntu:focal)     && apt-get clean     && rm -rf /var/lib/apt/lists/*                 0.0s
 => CACHED [ 8/20] RUN locale-gen en_US.UTF-8                                                                                0.0s
 => CACHED [ 9/20] RUN pkg-resolver/install-common-pkgs.sh                                                                                0.0s
 => CACHED [10/20] RUN pkg-resolver/install-maven.sh ubuntu:focal                                                                                0.0s
 => CACHED [11/20] RUN pkg-resolver/install-spotbugs.sh ubuntu:focal                                                                                0.0s
 => CACHED [12/20] RUN pkg-resolver/install-boost.sh ubuntu:focal                                                                                0.0s
 => CACHED [13/20] RUN pkg-resolver/install-protobuf.sh ubuntu:focal                                                                                0.0s
 => CACHED [14/20] RUN pkg-resolver/install-hadolint.sh ubuntu:focal                                                                                0.0s
 => CACHED [15/20] RUN pkg-resolver/install-intel-isa-l.sh ubuntu:focal                                                                                0.0s
 => CACHED [16/20] RUN groupadd --non-unique -g 0 root; exit 0;                                                                                0.0s
 => CACHED [17/20] RUN useradd -g 0 -u 0 -m root; exit 0;                                                                                0.0s
 => [18/20] RUN chown -R root /home/root; exit 0;                                                                                0.2s
 => [19/20] RUN mkdir -p /maven                                                                                0.3s
 => [20/20] RUN chown -R root /maven

How was this patch tested?
For code changes:

[ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
18m  0s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+0 :ok:
shelldocs
0m  0s

Shelldocs was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

_ trunk Compile Tests _

+0 :ok:
mvndep
14m  3s

Maven dependency ordering for branch

+1 :green_heart:
shadedclient
38m 22s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 29s

Maven dependency ordering for patch

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
shellcheck
0m  1s

No new issues.

+1 :green_heart:
shadedclient
36m 40s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
asflicense
0m 33s

The patch does not generate ASF License warnings.

112m 31s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6448/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6448

Optional Tests
dupname asflicense codespell detsecrets shellcheck shelldocs

uname
Linux f57beaab6d95 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / e8632c480e9d487b88dc93828eef28cd44977dfc

Max. process+thread count
540 (vs. ulimit of 5500)

modules
C:  U:

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6448/1/console

versions
git=2.25.1 maven=3.6.3 shellcheck=0.7.0

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@Hexiaoqiao Can you help review this PR again? Thank you very much!


text:
HIVE-21075 : Metastore: Drop partition performance downgrade with Postgres

This is rebase of PR #2323

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

recheck

recheck

repo: apache/hive


text:
HIVE-26415 : Add epoch time in the information_schema.scheduled_executions view

This commit doesn't contain any secrets.

What changes were proposed in this pull request?

It just adds an extra column in the scheduled_executions view of information_schema

Why are the changes needed?

Makes correlation between replication_metrics and scheduled_execution easier.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Manually tested by running the script through beeline interface.

+1. Looks good to me.
By the way, I need a write access as a committer. It's been 4 years. I'm new to Apache Hive on GitHub.

I tested the failing TestJdbcWithMiniHS2#testHttpRetryOnServerIdleTimeout on my laptop. It passed. As it's a trivial change, I'll merge it.

LGTM +1. I'll merge it.

authors: pudidic shreenidhiSaigaonkar


text:
SDK supports writing Date type

Describe This Problem
The document supports the Date type, but the SDK does not support writing the Date type.
Ref: https://horaedb.apache.org/sql/model/data_types.html
Proposal
SDK supports writing Date type.

[ ] Java
[ ] Pthon
[ ] Golang
[ ] Rust

Additional Context
No response

Hey @chunshao90 can I pick this up?

@adi-kmt Just go ahead, which sdk are you interested at?

Can start with Go and once I'm done with that can pick up rust and so on.

Sounds good, let me know if you have any problems.

@adi-kmt Any progress?

Sorry, got a bit busy with work
Just confirming @jiacai2050 , is the sdk the same inside the integration_test/sdk? Or is there some other path. Asking since i could only find one *.java file in the code, but no *.py file in that integration test

They are in different repository, please see https://horaedb.apache.org/docs/user-guide/sdk/

source: gharchive/issue

repo: apache/horaedb


text:
[HTTPCLIENT-2043] Rename classes documented as factories from *Provider

to *Factory.

Looking into...

Added:
This should be IgnoreCookieSpecFactory. The sole name IgnoreSpecFactory is too broad.

Thanks @michael-o

authors: garydgregory michael-o

repo: apache/httpcomponents-client


text:
[SUPPORT] Unable to Set Database Name for Hive Metadata Sync using Flink SQL

Describe the problem you faced
Using Flink SQL it's not possible to change Database Name for Hive Sync.
To Reproduce
Steps to reproduce the behavior:

Try to use Flink SQL to sink data to Hudi and metadata to Hive

CREATE TABLE IF NOT EXISTS test_tbl
(
    `eventDate`       TIMESTAMP(3),
    `operationType`   STRING
) WITH (
    'connector' = 'hudi',
    'path' = 'file:///opt/flink/hudi',
    'table.type' = 'COPY_ON_WRITE',
    'write.operation'='insert',
    'write.parquet.block.size'='1',
    'write.parquet.max.file.size'='5',
    'hoodie.database.name' = 'testdb',
    'hoodie.table.name' = 'test_tbl',
    'hive_sync.enable' = 'true',
    'hive_sync.mode' = 'hms',
    'hive_sync.metastore.uris' = 'thrift://172.17.0.1:9083'
    'hive_sync.database' = 'testdb',
);

Despite that both hoodie.database.name and hive_sync.database are set to 'testdb' - it still attempts to create default_database in Hive. Looking at the generated .hoodie/hoodie.properties I still see hoodie.database.name=default_database
Expected behavior
Hudi metadata is synced to Hive Metastore testdb database.
A clear and concise description of what you expected to happen.
Environment Description

Hudi version : 0.14.1

Spark version :

Hive version :

Hadoop version :

Storage (HDFS/S3/GCS..) :

Running on Docker? (yes/no) : yes

Using hudi-flink1.17-bundle-0.14.1.jar
Additional context
Whenever Hive Sync is configured in Flink SQL - it tries to create default_database and there is no way to change it,
Stacktrace
2024-01-29 15:36:25,611 ERROR org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [sync hive metadata for instant 20240129153624064] error
org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing test_tbl
	at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:171) ~[hudi-flink1.17-bundle-0.14.1.jar:0.14.1]
	at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:342) ~[hudi-flink1.17-bundle-0.14.1.jar:0.14.1]
	at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130) ~[hudi-flink1.17-bundle-0.14.1.jar:0.14.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]

Caused by: org.apache.hudi.hive.HoodieHiveSyncException: failed to create table test_tbl
	at org.apache.hudi.hive.ddl.HMSDDLExecutor.createTable(HMSDDLExecutor.java:140) ~[hudi-flink1.17-bundle-0.14.1.jar:0.14.1]
	at org.apache.hudi.hive.HoodieHiveSyncClient.createTable(HoodieHiveSyncClient.java:235) ~[hudi-flink1.17-bundle-0.14.1.jar:0.14.1]
	at org.apache.hudi.hive.HiveSyncTool.syncFirstTime(HiveSyncTool.java:332) ~[hudi-flink1.17-bundle-0.14.1.jar:0.14.1]
	at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:254) ~[hudi-flink1.17-bundle-0.14.1.jar:0.14.1]
	at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:180) ~[hudi-flink1.17-bundle-0.14.1.jar:0.14.1]
	at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:168) ~[hudi-flink1.17-bundle-0.14.1.jar:0.14.1]
	... 5 more
Caused by: org.apache.hadoop.hive.metastore.api.InvalidObjectException: default_database
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:54899) ~[hive-exec-3.1.3.jar:3.1.3]
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result$create_table_with_environment_context_resultStandardScheme.read(ThriftHiveMetastore.java:54876) ~[hive-exec-3.1.3.jar:3.1.3]
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_with_environment_context_result.read(ThriftHiveMetastore.java:54802) ~[hive-exec-3.1.3.jar:3.1.3]
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) ~[hive-exec-3.1.3.jar:3.1.3]
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_with_environment_context(ThriftHiveMetastore.java:1556) ~[hive-exec-3.1.3.jar:3.1.3]
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_with_environment_context(ThriftHiveMetastore.java:1542) ~[hive-exec-3.1.3.jar:3.1.3]
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:2867) ~[hive-exec-3.1.3.jar:3.1.3]
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:121) ~[hive-exec-3.1.3.jar:3.1.3]
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:837) ~[hive-exec-3.1.3.jar:3.1.3]
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:822) ~[hive-exec-3.1.3.jar:3.1.3]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) ~[hive-exec-3.1.3.jar:3.1.3]
	at com.sun.proxy.$Proxy50.createTable(Unknown Source) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2773) ~[hive-exec-3.1.3.jar:3.1.3]
	at com.sun.proxy.$Proxy50.createTable(Unknown Source) ~[?:?]
	at org.apache.hudi.hive.ddl.HMSDDLExecutor.createTable(HMSDDLExecutor.java:137) ~[hudi-flink1.17-bundle-0.14.1.jar:0.14.1]
	at org.apache.hudi.hive.HoodieHiveSyncClient.createTable(HoodieHiveSyncClient.java:235) ~[hudi-flink1.17-bundle-0.14.1.jar:0.14.1]
	at org.apache.hudi.hive.HiveSyncTool.syncFirstTime(HiveSyncTool.java:332) ~[hudi-flink1.17-bundle-0.14.1.jar:0.14.1]
	at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:254) ~[hudi-flink1.17-bundle-0.14.1.jar:0.14.1]
	at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:180) ~[hudi-flink1.17-bundle-0.14.1.jar:0.14.1]
	at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:168) ~[hudi-flink1.17-bundle-0.14.1.jar:0.14.1]
	... 5 more```

@vkhoroshko This is a table property and we can't change any of the table properties.

@vkhoroshko This is a table property and we can't change any of the table properties.

But docs state that this should be possible - https://hudi.apache.org/docs/configurations/#FLINK_SQL

The correct option for Flink SQL is "hive-sync.db" and not "hive-sync.database".
Closing the ticket.

repo: apache/hudi


text:
[SUPPORT] Transform from kafka complains about table not found when using transformer.sql

Dear experts,
we are trying to read data from a kafka topic and write it with hudi applying a transformation. Note our hudi command has worked before without transformation. Also we are working on a cluster that is configured very well and overall working fine.
**Command used"
the relevant part of the spark-submit command is:
spark-submit
--jars abfs://somelocation/hudi-utilities-bundle_2.11-0.6.0.jar \
--props /somepath/mypropfile.properties \
--target-base-path abfs://somepath/somedb/unmanaged/sometable \
--table-type COPY_ON_WRITE \
--transformer-class org.apache.hudyi.utilities.transform.SqlQueryBasedTransformer \
--hoodie-conf hoodie.deltastreamer.transformer.sql='select `fielda,fieldb from <SRC>';

Stacktrace
We keep on getting following error complaining about the table provided in target-base-path:
ERROR yarn.Client: Application diagnostics message: User class threw exception: org.apache.hudi.exception.TableNotFoundException: Hoodie table not found in path abfs://somepath/somedb/unmanaged/sometable/.hoodie'

We have defined an external table "sometable" for the table to land in the location above (abfs://somepath/somedb/unmanaged/sometable/), and it is empty so that hudi can write the data there.
Questions I have
1)in the "hoodie.deltastreamer.transformer.sql" statement, the SELECT ... FROM < SRC > -> what table do we choose here? do we really write < SRC > in the code? Does it know then from the props file that you have provided that he needs to fetch it from a kafka topic?
2)is our approach correct: we have a working kafka topic, and an empty external table, with the location that we provide in target-base-path hudi should read the data from the kafka topic and then write to the location of the external table?
Environment Description

Hudi version : 0.6.0

Spark version :2.4.5

Hive version :Hive 3.1 on Tez

Hadoop version :3.1.1

Storage (HDFS/S3/GCS..) : abfs (azure blob storage)

Running on Docker? (yes/no) :no

I have seen by coincidence @nsivabalan respond to similar questions.. maybe he is so kind to take a look?
Thanks!

This is similar to https://github.com/apache/hudi/issues/3906
@JB-data can you try not manually creating the path before job starts? deltastreamer is supposed to init a hoodie table for you. Also please consider upgrading to 0.8 or later since 0.6 is a bit too old.

Indeed, thanks for feedback.
This makes the error go away.
Maybe you also have input for this  question:
--> in the "hoodie.deltastreamer.transformer.sql" statement, the SELECT ... FROM < SRC > -> what table do we choose here? do we really write < SRC > in the code? Does it know then from the props file that you have provided that he needs to fetch it from a kafka topic?

@JB-data I have not used it myself. but from the code and example here, you write <SRC> which will be interpreted. See

org.apache.hudi.utilities.transform.TestSqlQueryBasedTransformer
org.apache.hudi.utilities.transform.SqlFileBasedTransformer#SRC_PATTERN

Closing this as the original issue was resolved.

Indeed, original issue clear.
Thanks for feedback.
Thanks for links: so indeed you write .
My confusion was: where does this SRC come from, what table is it?
But after looking at the java code you suggested (https://github.com/a0x8o/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/SqlQueryBasedTransformer.java), this is I believe what happens:
1)WITH SQL TRANSFORMATION
When you are using a transformation he will register  a temporary table in hive in order to perform the SQL transformation. So the data from the kafka topic is pulled into hive for a bit (in the default db), the SQL transformation is performed, and then the data is written according to the target schema.
This means we need to make sure that the user that is running the job has all permissions to access the hive metastore's default db as this is where he will temporarily register this  table, which in reality wil be called "HOODIE_SRC_TMP_TABLE_".
2)WITHOUT SQL TRANSFORMATION
While if you dont do any sql transformation, the data is directly landed in the target table and target db specified in the props.

@JB-data I don't think it goes to hive; it simply creates a spark sql table using that current spark session.

@xushiyan
Many thanks for all your input!
I think sparksql (at least in our setup-hadoop cluster with spark/hive/hdfs) registers this sql table in the default db in hive.  We noticed before at some point we were getting errors since the user running the job had no priviliges on the dfefault db in hive. After adding priviliges, this was solved.
Maybe if you are only running spark (which then uses its own embedded db for running queries and not hive), it is different.

authors: JB-data xushiyan


text:
[HUDI-5704] De-coupling column drop flag and schema validation flag (0.13.0)

Change Logs
De-coupling column drop flag and schema validation flag. Looks like we had tight coupling before.
for eg,
if table schema is col1, col2, col3
and new incoming schema is col1, col2
col drop config is set to false (which means col drop should not be supported), and schema validation is set to false, commit will succeed. Expectation is, commit should fail for this new batch.
Fixing the use-case in this patch, where we are de-coupling them. col drop flag will be honored irrespective of whether schema validation is enabled or not.
Impact
Column drop flag will be honored in all cases.
Risk level (write none, low medium or high below)
low.
Documentation Update
N/A
Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

@hudi-bot run azure

CI is green


text:
cpp network module LengthPrefixCodec has a bug

when using ignite cpp thin client, we find a bug of LengthPrefixCodec::Decode function, in file modules/platforms/cpp/network/src/network/length_prefix_codec.cpp

a call of LengthPrefixCodec::Decode with data length of 3, the packetSize reset to -1, and the packet length will be 3
following call, will enter first if, and the packet length of 3 reset to 0, the packet last 3 byte data will be lost

DataBuffer LengthPrefixCodec::Decode(DataBuffer& data)
        {
            if (packet.IsValid() && packet.Get()->Length() == (PACKET_HEADER_SIZE + packetSize))
            {
                packetSize = -1;
                packet.Get()->Length(0);
            }

            if (packetSize < 0)
            {
                Consume(data, PACKET_HEADER_SIZE);

                if (packet.Get()->Length() < PACKET_HEADER_SIZE)
                    return DataBuffer();

                packetSize = impl::binary::BinaryUtils::ReadInt32(*packet.Get(), 0);
            }

            Consume(data, PACKET_HEADER_SIZE + packetSize);

            if (packet.Get()->Length() == (PACKET_HEADER_SIZE + packetSize))
                return DataBuffer(packet, 0, PACKET_HEADER_SIZE + packetSize);

            return DataBuffer();
        }

can be fix as :
        DataBuffer LengthPrefixCodec::Decode(DataBuffer& data)
        {
            if (packet.IsValid() && packetSize != -1 && packet.Get()->Length() == (PACKET_HEADER_SIZE + packetSize))
            {
                packetSize = -1;
                packet.Get()->Length(0);
            }

Thanks for the bug report. I've created related Jira: https://issues.apache.org/jira/browse/IGNITE-23598. Sorry for the late replay.

authors: isapego morphad

repo: apache/ignite


text:
[AIRFLOW-782] Update Beam PipelineRunner name to 'DataflowRunner'

Dear Airflow maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
@fenglu-g please review and refer to a relevant commitor.
JIRA

[x] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-XXX

Description

[x] Here are some details about my PR, including screenshots of any UI changes:

DataflowPipelineRunner is no longer a valid pipeline runner name. DataflowRunner is.
Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

too straightforward change?
Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

This would brake people running it with the DataFlow v1 SDK. Note that v2 is still not officially out.

Codecov Report

Merging #2326 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2326   +/-   ##
=======================================
  Coverage   69.48%   69.48%
=======================================
  Files         144      144
  Lines       11028    11028
=======================================
  Hits         7663     7663
  Misses       3365     3365

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d165377...b163121. Read the comment docs.

Please note that even google-cloud-dataflow-0.6.0 which is up-to-date in pypi already removed DataflowPipelineRunner from its supported names.
The new name got introduced in 0.5.1 and the old name is officially removed in 0.6.0 (release note).
I know this is a breaking change, but anybody starting airflow today by pip install apache-airflow google-cloud-dataflow will get the error "DataflowPipelineRunner is not a valid PipelineRunner name."

Allright, I'm wondering if we should note this in UPDATING.md. So people are warned, as this is master this would be a new Airflow 1.9 section.

Added a note. Is it the correct place?

Why was this closed? Will you open a new one?

Ah, that was my mistake. I accidentally removed the branch that included this PR while cleaning up my repo.
Actually the PR was ready at the lastest update (May 25), but hasn't gotten much attention. (both the reviewer and I forgot to proceed)
I will revive again and see if there's any conflict to resolve.

FYI, I revived the branch and reopened the PR.
Not sure it's good to sync the branch myself, so the files are as they were
at the last commit. (I remember some projects prohibits syncing the branch
during review.)
On Wed, Jun 21, 2017 at 2:29 PM bolkedebruin notifications@github.com
wrote:

Why was this closed? Will you open a new one?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-airflow/pull/2326#issuecomment-310210606,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AXihKnMzHqd9uHcPAU_Xqt6t8Sc0r3Wuks5sGYsjgaJpZM4NmpuJ
.

Hi, I merged to another committed PR that did the same thing, but here still UPDATING.md needs updates.
@criccomini Please review. Thanks!

authors: alexvanboxel bolkedebruin codecov-io yk5

repo: apache/incubator-airflow

license_source: bigquery


text:
[AIRFLOW-1300] Enable table creation with TBLPROPERTIES in HiveCLIHook

Dear Airflow Maintainers,
Please accept this PR that addresses the following issues:
AIRFLOW-1300

Tblproperties? Any reason not to use the more pythonic table_properties?

You should also change mssql_to_hive and mysql_to_hive

@bolkedebruin the field is actually called TBLPROPERTIES in Hive (https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL)

Also, please change the formatting to match the template for the PR body

@saguziel @aoen @bolkedebruin made requested changes. please take another look.

Looks like tests are broken in master too, query renders correctly in stack trace
root: INFO: DROP TABLE IF EXISTS test_mysql_to_hive;
   CREATE TABLE IF NOT EXISTS test_mysql_to_hive (
   org_year INT,
       baby_name STRING,
       rate DOUBLE,
       sex STRING)
   ROW FORMAT DELIMITED
   FIELDS TERMINATED BY ','
   STORED AS textfile
   TBLPROPERTIES('test_property'='test_value')
   ;

Codecov Report

Merging #2364 into master will increase coverage by 0.01%.
The diff coverage is 87.5%.

@@            Coverage Diff             @@
##           master    #2364      +/-   ##
==========================================
+ Coverage   69.22%   69.23%   +0.01%
==========================================
  Files         145      145
  Lines       11134    11141       +7
==========================================
+ Hits         7707     7714       +7
  Misses       3427     3427

Impacted Files
Coverage Δ

airflow/operators/mssql_to_hive.py
0% <0%> (ø)
:arrow_up:

airflow/operators/s3_to_hive_operator.py
94.23% <100%> (+0.05%)
:arrow_up:

airflow/operators/mysql_to_hive.py
100% <100%> (ø)
:arrow_up:

airflow/hooks/hive_hooks.py
39.88% <100%> (+0.71%)
:arrow_up:

airflow/models.py
87.13% <0%> (+0.04%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fb21bcb...7500018. Read the comment docs.

@saguziel The tests are now successful

authors: bolkedebruin codecov-io krishnabhupatiraju saguziel


text:
[BEAM-894] Using @Teardown to remove temp files from failed bundles in Write.WriteBundles.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

R: @dhalperi @tgroh

PTAL

The leftover temp files issue is mitigated by https://github.com/apache/incubator-beam/pull/1278 now.
I am closing this PR since the benefits doesn't justify the cost for the additional method that Sink writers have to implement.

repo: apache/incubator-beam


text:
serevr端服务存在，client一直处于[E112]Not connected to

Describe the bug (描述bug)
client端链接一直处于[E112]Not connected to ，重试请求依然是[E112], 但是底层server服务是正常的，其他client可以正常链接发送
To Reproduce (复现方法)
Expected behavior (期望行为)
Versions (各种版本)
OS:
Compiler:
brpc:
protobuf:
Additional context/screenshots (更多上下文/截图)

tcpdump或者是wireshark抓包看一眼吧

authors: BillyJoy GTHubT

repo: apache/incubator-brpc


text:
[DLAB-1510] All resourses are stopped with project

According to related issue, the project's resourse should be stopped with initial project

Did you run tests?

The unitest has been adjusted, plus a new test has been added for the start resource

repo: apache/incubator-dlab


text:
[QUESTION] Reslove InterProcessMutex.acquire() hang

Worker's Network is not good , but connecting has no problem .
When worker using InterProcessMutex to acquire a lock, all worker's acquire() hang in a long time.
public InterProcessMutex acquireZkLock(CuratorFramework zkClient,String zNodeLockPath)throws Exception{
		InterProcessMutex mutex = new InterProcessMutex(zkClient, zNodeLockPath);
		mutex.acquire();
		return mutex;
	}

Has anyone else encountered the same problem ？

I may find the reason: https://github.com/apache/curator/pull/297

Fixes race in many Curator recipes whereby a pattern was used that called "notifyAll" in a synchronized block in response to a ZooKeeper watcher callback. This created a race and possible deadlock if the recipe instance was already in a synchronized block. This would result in the ZK event thread getting blocked which would prevent ZK connections from getting repaired. This change adds a new executor (available from CuratorFramework) that can be used to do the sync/notify so that ZK's event thread is not blocked.

repo: apache/incubator-dolphinscheduler


text:
fix: The toolbar buttons of Dag disable bug

Please see #2123 of details.

@break60

+1

authors: Rubik-W break60 dailidong


text:
[Fix-3077][ui] Fix the edit duplicated name verify

What is the purpose of the pull request
colse(#3077 )
Fix the edit duplicated name verify
Brief change log

Change the verify for edit and create process definition

Verify this pull request

Manually verified the change by testing locally.

Your pr caused the workflow to be uneditable, please refer to the screenshot below to fix


text:
build-thirdparty.sh fail on ubuntu

Describe the bug
In ubuntu (16.04 14.04）and many other Linux distribution, "sh" links to /bin/sh not /bin/bash. build-thirdparty.sh use source command which only exists in /bin/bash not /bin/sh.
Run build-thirdparty.sh by sh fails and downlaod-thirdparty.sh:65 use sh to run download-thirdparty.sh.
Because both first line of download-thirdpaty.sh and  build-thirdparty.sh is "#!/bin/bash", the bash scripts can run directly without “sh".
To Reproduce
Steps to reproduce the behavior:

sh ./thirdparty/build-thirdparty.sh
See error:
thirdparty/build-thirdparty.sh: 41: thirdparty/build-thirdparty.sh: source: not found

Expected behavior
build third-party source
Screenshots

Desktop (please complete the following information):

OS: Ubuntu
Version 16.04

Additional context
Add any other context about the problem here.

commit： ae19fbfc8b741a9cd70c780e6f70b9e9a08bb493 has fixed all the issues.

repo: apache/incubator-doris


text:
Duplicate rows in Aggregate Key Table

Describe the bug
A clear and concise description of what the bug is.
Duplicate rows in Aggregate Key Table.
To Reproduce
Steps to reproduce the behavior:

create dest tbl

create table test1(
id int,
dt date,
name varchar(10) replace_if_not_null
) AGGREGATE KEY(id, dt)
partition by range (dt)(
    PARTITION pmin VALUES LESS THAN ("2021-01-01"),
    PARTITION p20210101 VALUES LESS THAN ("2021-01-02"),
    PARTITION p20210102 VALUES LESS THAN ("2021-01-03"),
    PARTITION p20210103 VALUES LESS THAN ("2021-01-04"),
    PARTITION p20210104 VALUES LESS THAN ("2021-01-05"),
    PARTITION p20210105 VALUES LESS THAN ("2021-01-06"),
    PARTITION p20210106 VALUES LESS THAN ("2021-01-07"),
    PARTITION p20210107 VALUES LESS THAN ("2021-01-08"),
    PARTITION p20210108 VALUES LESS THAN ("2021-01-09"),
    PARTITION p20210109 VALUES LESS THAN ("2021-01-10"),
    ...
    PARTITION p20210130 VALUES LESS THAN ("2021-01-31")
)DISTRIBUTED BY HASH(`id`) BUCKETS 8
PROPERTIES (
    "replication_num" = "3",
    "storage_type"="column",
    "dynamic_partition.enable" = "true",
    "dynamic_partition.time_unit" = "DAY",
    "dynamic_partition.time_zone" = "Asia/Shanghai",
    "dynamic_partition.end" = "2",
    "dynamic_partition.prefix" = "p",
    "dynamic_partition.replication_num" = "3",
    "dynamic_partition.buckets" = "8"
);

insert into data

insert into test1
select id, dt, name
from source_tbl;

execute the Sql multi times in step 2
query the data in test1 tbl

select * from test1 where id = 123 and dt = '2021-01-01' ;

**There are duplicate lines in the result with the same primary key. **
Expected behavior
A clear and concise description of what you expected to happen.'
There is only one line if the data has the same primary key .
Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

hi，which version are you using now.

hi，which version are you using now.

hi，which version are you using now.

The code in master branch about 2020.12.20.

hi，which version are you using now.

The code in master branch about 2020.12.20.


text:
[DUBBO-2418]: merge pull request #2353 to 2.6.x to fix #2353

What is the purpose of the change
merge pull request #2418  to 2.6.x to fix #2353
Brief changelog
dubbo-config/dubbo-config-spring/src/main/java/com/alibaba/dubbo/config/spring/beans/factory/annotation/ServiceAnnotationBeanPostProcessor.java
dubbo-config/dubbo-config-spring/src/test/java/com/alibaba/dubbo/config/spring/ConfigTest.java
dubbo-config/dubbo-config-spring/src/test/java/com/alibaba/dubbo/config/spring/context/annotation/provider/HelloServiceImpl.java
Verifying this change
XXXXX
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[ ] Run mvn clean install -DskipTests & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
[ ] If this contribution is large, please follow the Software Donation Guide.

Codecov Report

Merging #2761 into 2.6.x will increase coverage by 0.04%.
The diff coverage is 100%.

@@             Coverage Diff              @@
##              2.6.x    #2761      +/-   ##
============================================
+ Coverage     47.38%   47.42%   +0.04%
- Complexity     4289     4293       +4
============================================
  Files           552      552
  Lines         24375    24375
  Branches       4313     4313
============================================
+ Hits          11551    11561      +10
+ Misses        11038    11031       -7
+ Partials       1786     1783       -3

Impacted Files
Coverage Δ
Complexity Δ

...annotation/ServiceAnnotationBeanPostProcessor.java
85.29% <100%> (+2.2%)
33 <0> (+2)
:arrow_up:

...baba/dubbo/remoting/transport/mina/MinaClient.java
57.81% <0%> (-1.57%)
8% <0%> (-1%)

.../com/alibaba/dubbo/monitor/dubbo/DubboMonitor.java
89.71% <0%> (+1.86%)
15% <0%> (ø)
:arrow_down:

...n/java/com/alibaba/dubbo/common/utils/JVMUtil.java
79.62% <0%> (+7.4%)
15% <0%> (+2%)
:arrow_up:

...baba/dubbo/config/spring/util/AnnotationUtils.java
52.63% <0%> (+10.52%)
5% <0%> (+1%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a1b27ab...67019e6. Read the comment docs.

repo: apache/incubator-dubbo


text:
Latest ECharts 4.4/3/2 is not working with script tag, while Echarts 2.2.7 is working

Version
4.4.0
Steps to reproduce
I am using the latest echarts.min.js file and trying to use it by script tag in my jsp page. The application is able to get the js file, but on call to echarts, it is throwing error "echarts is undefined". This is happening with echarts 4.4 or 4.3 or 4.2. But the same is working with 2.2.7 version.
Usage for 4.4:

Usage for 2.2.7

What is expected?
latest echarts 4.4 should work with script tag in jsp page
What is actually happening?
echarts.min.js is throwing error on calling echarts.init()
echarts is undefined

I have raised a previous ticket #11573, but since I got the echarts2.2.7 working I closed it. But with the echarts 2.2.7 version, I got it from cdn and there is no license information, and i cannot find a link for echarts 2.2.7 on echarts.apache.org site. Kindly let me know following:

how the echarts 2.2.7 different from echarts 4.4 which is not working with script tag ?
If I am using echarts 2.2.7, are there any licence information and a link from echarts website to use ?

Thanks

It probably be some basic "HTML/JS" developing issue, but not related to echarts.

How the echarts 2.2.7 different from 4.4, because when integrated with echarts2.2.7, I am able to get the object, but with 4.4, the echarts object is not present and throwing error "echarts is undefined". Is there any change needs to be done apart from adding the echarts in the script tag ?
Apart from that, do you still support echarts 2.2.7 and can I get a link for echarts 2.2.7 js files on echarts website?
NOTE: the browser is able to the js file, I can confirm this as I am seeing it in Network tab of my browser.
Thanks

You can get the global echarts variable. Here's an example: https://jsfiddle.net/ovilia/n6xc4df3/
Please check your HTML to make sure it is including the JavaScript correctly.

Hi Olivia,
I was not able to access the echarts variable because the echarts variable is not exported in the echarts.js file. I made the following highlighted change in the echarts.js file, after which I could access the echarts in my jsp page.

I am closing the ticket now as I can create charts using echarts, but with a limitation that I have to change the echarts.js file to access echarts variable.
Thanks for the help.

authors: 100pah Ovilia ankitagrahari

repo: apache/incubator-echarts


text:
关于点击图会闪动

One-line summary [问题简述]
点击下图的定义的一个单击事件, 图会有抖动，什么参数可以让其不抖动

Version & Environment [版本及环境]  最新

ECharts version [ECharts 版本]: 最新
Browser version [浏览器类型和版本]: chrome 最新
OS Version [操作系统类型和版本]: 最新

Expected behaviour [期望结果]
ECharts option [ECharts配置项]

option = {
    backgroundColor: new echarts.graphic.RadialGradient(0.3, 0.3, 0.8, [{
        offset: 0,
        color: '#f7f8fa'
    }, {
        offset: 1,
        color: '#cdd0d5'
    }]),
    // title: {
    //     text: "dsdsdS",
    //     subtext: "DsdsDs",
    //     top: "top",
    //     left: "center"
    // },
    tooltip: {
        show:true,
        trigger:'item',
        formatter:(params , ticket , callback)=> {
            //console.log ( "data" , params );
            return "
tooltip
";
        }

    },
    legend: [{
        // formatter: function(name) {
        //     return echarts.format.truncateText(name, 40, '14px Microsoft Yahei', '…');
        // },
        tooltip: {
            show: true
        },
        selectedMode: 'false',
        bottom: 20,
        data: ['有色金属', '物联网', '互联网', '通信设备']
    }],
    // toolbox: {
    //     show: true,
    //     feature: {
    //         dataView: {
    //             show: true,
    //             readOnly: true
    //         },
    //         restore: {
    //             show: true
    //         },
    //         saveAsImage: {
    //             show: true
    //         }
    //     }
    // },
    //animationDuration: 13000,
    //animationEasingUpdate: 'quinticInOut',
    series: [{
        name: '详情',
        type: 'graph',
        layout: 'force',
        force: {
            repulsion: 300
        },
        data: [{
            "name": "计算机应用",
            // "x": 0,
            // y: 0,
            "symbolSize": 70,
            "draggable": "true",
            "value": 27

        },{
            "name": "有色金属",
            "value": 15,
            "symbolSize": 50,
            "category": "有色金属",
            tip:"xxxxxxxxxxx",
            "draggable": "true",
            itemStyle: {
                color: 'green'
            }
        },{
            "name": "物联网",
            "symbolSize": 50,
            "category": "物联网",
            "draggable": "true",
            "value": 1
        },{
            "name": "互联网",
            "symbolSize": 50,
            "category": "互联网",
            "draggable": "true",
            "value": 1
        },{
            "name": "通信设备",
            "symbolSize": 50,
            "category": "通信设备",
            "draggable": "true",
            "value": 1
        }],
        links: [{
            "source": "计算机应用",
            "target": "有色金属",
            lineStyle:{
                color:'green',
                width:2,
                curveness:0.6
            }
        },{
            "source": "计算机应用",
            "target": "物联网",
            lineStyle:{
                color:'green',
                width:2,
                curveness:0.6
            }
        },{
            "source": "计算机应用",
            "target": "互联网",
            lineStyle:{
                color:'green',
                width:2,
                curveness:0.6
            }
        },{
            "source": "计算机应用",
            "target": "通信设备",
            lineStyle:{
                color:'green',
                width:2,
                curveness:0.6
            }
        }],
        categories: [{
            'name': '有色金属'
        },{
            'name':'物联网'
        },{
            'name':'互联网'
        },{
            'name':'通信设备'
        }],
        focusNodeAdjacency: true,
        roam: true,
        label: {
            normal: {
                show: true,
                position: 'top',

            }
        },
        lineStyle: {
            normal: {
                color: 'source',
                curveness: 0,
                type: "solid",
            },
        }
    }]
}

Other comments [其他信息]

没有复现出“抖动”，抖动指什么？高亮吗？给整个 option 配置个 option.silent: true 可以禁止交互

这个参数配置后所有的事件都失效了

那抖动是什么意思呢？

节点会重新渲染 位置会移动

无法复现，如果还有问题的话，建议 reopen 一下这个 issue，并且提供额外的复现条件。

authors: Ovilia cartWalk wwwsoftwares


text:
[Enhancement] Unused field

Search before asking

[X] I had searched in the issues and found no similar issues.

Enhancement Request

located at:
org/apache/eventmesh/runtime/boot/EventMeshTCPServer.java 133
analysis and explanation:
Unused field: org.apache.eventmesh.runtime.boot.EventMeshTCPServer.tcpRegisterTask
Describe the solution you'd like
This field is never used,consider removing it from the class.
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

I would like to work on this issue. @Alonexc @xwm1992 Can you please assign this task to me?

authors: Alonexc DeepikaMarthandanTW

repo: apache/incubator-eventmesh


text:
[Enhancement] Method check a map with containsKey(), before using get() [LocalUnSubscribeEventProcessor]

Search before asking

[X] I had searched in the issues and found no similar issues.

Enhancement Request

located at:
eventmesh-runtime/src/main/java/org/apache/eventmesh/runtime/core/protocol/http/processor/LocalUnSubscribeEventProcessor.java line 173
analysis and explanation:
This method checks for the presence of a key in a map using containsKey(), before attempting to fetch the value of the key using get(). This equates to doing two map lookups in a row.
Describe the solution you'd like
It is much simpler to just fetch the value with get, and checking for non null instead.
Are you willing to submit PR?

[x] Yes I am willing to submit a PR!

I want to fix it.

I want to fix it.

authors: Alonexc jonyangx


text:
Add Hudi changes for Presto MOR query support

Adds the necessary changes to hudi for support of presto querying hudi
merge-on-read table's realtime view.
To see the associated follow-up changes in Presto, please see this branch.
https://github.com/bschell/presto/commit/a3fb658c1cd70fd72f0a3021b3d994fe383303aa
Brief change log

Add @UseRecordReaderFromInputFormat annotation for HoodieParquetRealtimeInputFormat

Verify this pull request
Tested with associated Presto patch to successfully query hudi MOR realtime tables.
hudi-hadoop-mr tests pass

Codecov Report

Merging #1578 into master will increase coverage by 0.03%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #1578      +/-   ##
============================================
+ Coverage     71.81%   71.85%   +0.03%
  Complexity      294      294
============================================
  Files           385      385
  Lines         16542    16542
  Branches       1661     1661
============================================
+ Hits          11880    11886       +6
+ Misses         3932     3925       -7
- Partials        730      731       +1

Impacted Files
Coverage Δ
Complexity Δ

...oop/realtime/HoodieParquetRealtimeInputFormat.java
72.34% <ø> (ø)
0.00 <0.00> (ø)

...src/main/java/org/apache/hudi/metrics/Metrics.java
67.56% <0.00%> (+10.81%)
0.00% <0.00%> (ø%)

...g/apache/hudi/metrics/InMemoryMetricsReporter.java
80.00% <0.00%> (+40.00%)
0.00% <0.00%> (ø%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4b7162...c8b4297. Read the comment docs.

@bschell Thanks for your contribution. It would be better  to file a jira issue to track this work. More details, please refer to Hudi's contribution guide: http://hudi.apache.org/community.html#contributing

@bschell Thanks for your contribution. It would be better to file a jira issue to track this work. More details, please refer to Hudi's contribution guide: http://hudi.apache.org/community.html#contributing

I suppose this is tracked under - https://issues.apache.org/jira/browse/HUDI-305.
@bschell  can you reassign that Jira to yourself and also change the title of this PR to start with [HUDI-305] ?

Thanks for taking a look and sorry about the title. I completely spaced on adding [HUDI-305], just added.

LGTM. merging this.

authors: bhasudha bschell codecov-io yanghua

repo: apache/incubator-hudi


text:
java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

While creating Iceberg table using PartitionSpec.builderFor, I faced the java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V error.
I use the Iceberg-master and Spark 2.4.0.
import org.apache.hadoop.fs.Path
import org.apache.iceberg.hadoop.HadoopTables
import org.apache.iceberg.spark.SparkSchemaUtil
import org.apache.iceberg.PartitionSpec

val path = "hdfs://${TABLE_PATH}"
val df = spark.read.parquet("/user/hive/warehouse/tpcds.db/web_sales")

{
    val conf = spark.sparkContext.hadoopConfiguration
    val fs = new Path(path).getFileSystem(conf)
    fs.delete(new Path(path), true)
    val tables = new HadoopTables(conf)
    val icebergSchema = SparkSchemaUtil.convert(df.schema)
    val spec = PartitionSpec.builderFor(icebergSchema)
    .identity("ws_sold_date_sk")
    .identity("ws_sold_time_sk")
    .identity("ws_ship_date_sk")
    .identity("ws_item_sk")
    .identity("ws_bill_customer_sk")
    .identity("ws_bill_cdemo_sk")
    .identity("ws_bill_hdemo_sk")
    .identity("ws_bill_addr_sk")
    .identity("ws_ship_customer_sk")
    .identity("ws_ship_cdemo_sk")
    .identity("ws_ship_hdemo_sk")
    .identity("ws_ship_addr_sk")
    .identity("ws_web_page_sk")
    .identity("ws_web_site_sk")
    .identity("ws_ship_mode_sk")
    .identity("ws_warehouse_sk")
    .identity("ws_promo_sk")
    .identity("ws_order_number")
    .identity("ws_quantity")
    .identity("ws_wholesale_cost")
    .identity("ws_list_price")
    .identity("ws_sales_price")
    .identity("ws_ext_discount_amt")
    .identity("ws_ext_sales_price")
    .identity("ws_ext_wholesale_cost")
    .identity("ws_ext_list_price")
    .identity("ws_ext_tax")
    .identity("ws_coupon_amt")
    .identity("ws_ext_ship_cost")
    .identity("ws_net_paid")
    .identity("ws_net_paid_inc_tax")
    .identity("ws_net_paid_inc_ship")
    .identity("ws_net_paid_inc_ship_tax")
    .identity("ws_net_profit")
    .build()
    tables.create(icebergSchema, spec, path)
    tables.load(path).schema
}

java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
  at org.apache.iceberg.PartitionSpec$Builder.checkAndAddPartitionName(PartitionSpec.java:330)
  at org.apache.iceberg.PartitionSpec$Builder.identity(PartitionSpec.java:356)
  ... 93 elided

I appreciate if anyone can help me with this problem.

Solved by adding relocate 'com.google.common', 'com.google.common.version28' in build.gradle.

@HelloHorizon Where did you add this and were there any other steps? I'm facing the same issue. Thanks

@chris-alphonso For me, it is because of Guava version conflict. Since Spark v2.4.0 use 14.0.1 while iceberg use 28.0, you only need to add relocation for shardowjar in build.gradle. Add relocate 'com.google.common', 'com.google.common.version28' at line 415.

We ran into this issue as well, should the workaround land on master?

@rdblue I know this issue appears to be closed, but I noticed that we run into this problem when trying to use the iceberg artifact published on maven with spark 2.4.4. Is there a downside to "relocating" google commons in iceberg?

@Voltir, that sounds fine to me. I don't think I saw this issue when it was open so I didn't realize there was a conflict. Would you like to open a PR to relocate Guava?

@rdblue Sure, I will open a PR shortly.

authors: HelloHorizon Voltir chris-alphonso rdblue

repo: apache/incubator-iceberg


text:
[IOTDB-577] Cluster module build failure due to unapproved licenses

The cluster module cannot pass rat check.
The cluster_new branch build failure due to some files with unapproved licenses.

Hi @vesense , PR #958 has fixed the apache rat.
To reduce code conflict, I'd like to close this PR.
Before PR 958 is merged, we can disable apache-rat checker for passing the compiling.
How do you think?

@jixuan1989 I'm OK, and will close this PR.

repo: apache/incubator-iotdb


text:
This PR is a first attempt to minimize ngram array copies by utilizing DirectBuffers.

This is a work in progress but posted so that @mjpost  and @kpu can review (and do some speed comparisons).

@kpu there's still a lot of array copying going on here.  We've got some further work coming in this commit https://github.com/KellenSunderland/incubator-joshua/commit/6a88bfbdd130301a73248ed1633397dc7779b210 but we can't be bothered to fix the failing tests tonight.

This PR should now implement all of the JNI improvements we wanted to make at the MT Marathon.  Before merging would you be able to run a quick test and see what the performance is like?  If the second commit doesn't speed things up we could consider removing it to reduce code complexity.
I also noticed that language model ffs all were estimating with the following signature:
public float estimateCost(Rule rule, Sentence sentence)
Does it actually make sense to pass the sentence object here?  Each rule should be able to be estimated with only its local feature score + weights right?  It shouldn't need any additional context?

I thought the role of Sentence here is to scope the lifecycle of ChartState objects and disambiguate what the vector the indices refer to.

I want to dig in to why this test is returning differing results between linux and osx.  Temporarily closing this PR.

@kpu I partially agree about sentence.  The sentence object is meant to be scoped to a single translation of a sentence.  For calls to probRule this helps disambiguate.
For estimate I'm not so sure.  A rule can have estimateRuleCost called on it even when Joshua is not translation anything. For example as a result of calling sort on a grammar during startup before any requests are processed.  In fact the only usage of estimate being called perviously was passing null as the argument for sentence.  For the time being I'm creating the ChartState object once per call, and then immediately releasing it afterwards.  We'll have to test the performance impact of doing this, but this shouldn't actually be called when decoding so hopefully it won't affect decoding perf.

I did some timing tests on commit 3ba83f84e8258a784fcef509fc9b158d44f15c66, and didn't see any change.

Well in that case I can open a PR without the DirectBuffer changes included.  They're pretty complicated and if they don't improve performance it's probably better to leave them out.

For estimateRuleCost, are we allocating a ChartState on a vector then throwing it away to score grammar rules in isolation?  I thought that was a separate JNI path that didn't want a sentence.

authors: KellenSunderland kpu mjpost

repo: apache/incubator-joshua


text:
Any benchmark compare with ledis, ssdb or redix? simple test seems write performance much slower....

Describe the bug
A clear and concise description of what the bug is.

OS: [e.g. Centos 7]
Version [e.g. 1.1.5]

To Reproduce
Steps to reproduce the behavior:
1.
2.
3.
Expected behavior
A clear and concise description of what you expected to happen.

compare with Redis or others? can you paste the Redis benchmark log?

Closed as stale. I suppose you run redis-benchmark against these systems.
Looking forward to any feedback.

authors: binwangcn git-hulk tisonkun

repo: apache/incubator-kvrocks


text:
Require redis-cli for testing

Without redis-cli the command ./x.py test go is running but the test cases eventually fail with an error exec: "redis-cli": executable file not found in $PATH

@tisonkun Good observation.
@PragmaTwice Thank you for your hints.
Done.

Also reformatted x.py in a separate commit.

@torwig may I ask what tools you use to format the python code?

@tisonkun It was just Pycharm :)

Thanks all. Merging...

authors: PragmaTwice tisonkun torwig


text:
[KYUUBI #1174][FOLLOWUP] change configuration(ha.zookeeper.publish.configs) default value to false

Why are the changes needed?
Compatible with lower version of hive driver.
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

https://github.com/apache/incubator-kyuubi/pull/1175#issuecomment-993263295
cc @pan3793

Codecov Report

Merging #1553 (25300d5) into master (3594e08) will not change coverage.
The diff coverage is 100.00%.

:exclamation: Current head 25300d5 differs from pull request most recent head 908c475. Consider uploading reports for the commit 908c475 to get more accurate results

@@            Coverage Diff            @@
##             master    #1553   +/-   ##
=========================================
  Coverage     59.28%   59.28%
  Complexity      172      172
=========================================
  Files           241      241
  Lines         12224    12224
  Branches       1501     1501
=========================================
  Hits           7247     7247
  Misses         4364     4364
  Partials        613      613

Impacted Files
Coverage Δ

...la/org/apache/kyuubi/ha/HighAvailabilityConf.scala
98.85% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3594e08...908c475. Read the comment docs.

repo: apache/incubator-kyuubi


text:
[KYUUBI #2743] colorfully kyuubi logo support

We can make the logo printed by Kyuubi on the terminal colorful and vibrant.
Why are the changes needed?

When TTY is available, print the colorfully logo, else print default

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

Codecov Report

Merging #2744 (2205e79) into master (49352b5) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #2744      +/-   ##
============================================
- Coverage     61.79%   61.78%   -0.01%
  Complexity      385      385
============================================
  Files           433      433
  Lines         20345    20345
  Branches       2760     2760
============================================
- Hits          12572    12570       -2
- Misses         6541     6542       +1
- Partials       1232     1233       +1

Impacted Files
Coverage Δ

.../org/apache/kyuubi/operation/KyuubiOperation.scala
68.00% <0.00%> (-1.34%)
:arrow_down:

...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala
84.27% <0.00%> (-0.63%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 49352b5...2205e79. Read the comment docs.

@turboFei Have we use this logo in other places?

Thanks, merging to master

authors: codecov-commenter pan3793 wolfboys


text:
[KYUUBI #3720][FOLLOWUP] Using correct config version for CTL_BATCH_LOG_ON_FAILURE_TIMEOUT

Why are the changes needed?

Set the config version to 1.6.1.
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

thanks, merged to master and 1.6.1


text:
Multi-Training-Task on the same GPU card

For bugs or installation issues, please provide the following information.
The more information you provide, the more likely people will be able to help you.
Environment info
Operating System:
centos-7.2
Compiler:
gcc-4.8.5
Package used (Python/R/Scala/Julia):
Python
MXNet version:
0.9.3
Or if installed from source:
installed from source
MXNet commit hash (git rev-parse HEAD):
If you are using python package, please provide
python-2.7
Python version and distribution:
python-2.7
If you are using R package, please provide
R sessionInfo():
Error Message:
Please paste the full error message, including stack trace.
When I run a single process on a single gpu card, I got about 56 samples/sec;
When I run two single process on a single gpu card, I got about 27 samples/sec
I think it is mainly the band-width in IO for GPU which cause the low trainning speed
How can I run multi-trainning-task on the sample single GPU card with a high training performence?
And My GPU's version is k40(12G)
Thank you very much!
Minimum reproducible example
if you are using your own code, please provide a short script that reproduces the error.
Steps to reproduce
or if you are running standard examples, please provide the commands you have run that lead to the error.

What have you tried to solve it?

when trainning, the following problem occurs:
（1）One trainning task on one gpu card(k40c, 12G)，batchsize=64的，59 samples/ s ec;
(2) Two sample trainning task on one gpu card(k40c, 12G)，batchsize=32， 27samples / sec
why? Thank you very much!

You have two training at 27 samples/sec each one ?
GPU have power processing so if one training use 100% the second need to wait even if they share the memory.
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070    Off  | 0000:01:00.0     Off |                  N/A |
| 29%   44C    P8    14W / 151W |      0MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                              ^
                                                              |
                                                              |

This issue is closed due to lack of activity in the last 90 days. Feel free to ping me to reopen if this is still an active issue. Thanks!
Also, do please check out our forum (and Chinese version) for general "how-to" questions.

authors: edmBernard idealboy szha

repo: apache/incubator-mxnet


text:
Google index points to old version of install instructions

Description

Leads to

@marcoabreu - this issue was resolved by the redirect to the install page. Please close.

authors: aaronmarkham lebeg


text:
[MXNET-978] Higher Order Gradient Support clip, dropout.

Description
PR intends to add support for higher order gradient for clip, dropout.
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[x] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA-978 issue created (except PRs with tiny changes)
[x] Changes are complete (i.e. I finished coding on this PR)
[x] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[x] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[x] higher order gradient for a clip, dropout.
[x] unit test for the same.

@kshitij12345 could you trigger CI again?

@roywei @apeforest Have retriggered and the tests have ran successfully.


text:
[Large Tensor] Fix multi_lars op

Description
The multi_lars op was previously breaking on large tensor (dimension >= 2^32) data. With the following input:
run_performance_test(nd.multi_lars, inputs=[{"lrs": nd.random_normal(shape=(2**32 + 1, 1)), "weights_sum_sq": nd.random_normal(shape=(2**32 + 1, 1)), "grads_sum_sq": nd.random_normal(shape=(2**32 + 1, 1)), "wds": nd.random_normal(shape=(2**32 + 1, 1)), "eta": .1, "eps": .9}], run_backward=False, warmup=1, runs=1)

the following error was thrown:
Segmentation fault: 11

To root cause this issue, I ran the previous command in a Python script with GDB, and found that the underlying problem was in the data type of the MultiLARS kernel in multi_lars-inl.h.h. The index variable i used the int dtype when it should have been using index_t to properly handle long int indices. I switched this variable to index_t in the struct header and, after rebuilding, the previous input command displayed the correct output:
INFO:root:Begin Benchmark - multi_lars
INFO:root:Complete Benchmark - multi_lars
[{'multi_lars': [{'inputs': {'lrs': '<NDArray 4294967297x1 @cpu(0)>', 'weights_sum_sq': '<NDArray 4294967297x1 @cpu(0)>', 'grads_sum_sq': '<NDArray 4294967297x1 @cpu(0)>', 'wds': '<NDArray 4294967297x1 @cpu(0)>', 'eta': 0.1, 'eps': 0.9}, 'max_storage_mem_alloc_cpu/0': 16106127.0, 'avg_time_forward_multi_lars': 4904.6128}]}]

To ensure completeness and to prevent future breaking changes, I also added a nightly test for the multi_lars op with large tensor data in tests/nightly/test_large_array.py.
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[x] Changes are complete (i.e. I finished coding on this PR)
[x] All changes have test coverage
[x] Code is well-documented
[x] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

M src/operator/contrib/multi_lars-inl.h
M tests/nightly/test_large_array.py

Comments
Tested on r5dn.24xl-ubuntu 16.04 and p2.16xl-ubuntu 16.04 with

Individual op run
Full OpPerf run

Results
The key difference between CPU and GPU tests was the instance type (r5dn.24xl for CPU, p2.16xl for GPU). All relevant build flags remain the same, and both were tested using CPU context.
Single operator test - multi_lars op (GPU) - pending
Single operator test - multi_lars op (CPU)
Full OpPerf test (GPU) - pending
Full OpPerf test (CPU) - pending
@apeforest @access2rohit @ChaiBapchya

@mxnet-label-bot add [pr-awaiting-review]


text:
AMP support for Numpy ops

Description

Add all registered ops (mostly Numpy) to AMP lists
Removed "skipped" attribute from AMP tests
Some change in AMP monkey patching, since NumPy ops require a bit different handling.

@mxnet-bot run ci [centos-cpu, unix-cpu]

@mxnet-bot run ci [all]

[2020-09-24T01:42:24.553Z] ************* Module mxnet.contrib.amp.loss_scaler

[2020-09-24T01:42:24.553Z] python/mxnet/contrib/amp/loss_scaler.py:45:4: C0111: Missing method docstring (missing-docstring)

[2020-09-24T01:42:24.553Z] python/mxnet/contrib/amp/loss_scaler.py:52:8: W0105: String statement has no effect (pointless-string-statement)

@mxnet-bot run ci [unix-cpu]

@mxnet-bot run ci [centos-cpu, unix-cpu, unix-gpu]

@mxnet-bot run ci [centos-cpu, unix-cpu]

@mxnet-bot run ci [windows-gpu, macosx-x86_64]


text:
Improve building of platform on JDK11

Make sure one can build and test with:
incubator-netbeans$ ant build-platform -Dnbjdk.home=$HOME/bin/jdk-11/
incubator-netbeans$ ant test-platform -Dnbjdk.home=$HOME/bin/jdk-11/

I've created new jenkins job to test platform on JDK11. Once this PR is integrated, the job should compile and run the tests (with some failures).

FWIW, there is also "test.nbjdk.home" which is a JDK on which the tests are run (while the compilation still uses nbjdk.home).

Why isn't test.nbjdk.home derived from value of nbjdk.home?

Feel free to make test.nbjdk.home fallback to to nbjdk.home, although I am not quite sure if there's a point. The launcher used to run the tests will either be derived from test.nbjdk.home (if present and usable) or from nbjdk.home (otherwise). I.e. using test.nbjdk.home is optional, but allows to compile and run tests on different JDKs.

The problem is that without specifying test.nbjdk.home the tests fallback to JAVA_HOME and not to nbjdk.home. I found that confusing.

The job run #10 finished with 110 failed tests. Anyone who cares about JDK11 and NetBeans Platform is welcomed to fix some!

Not sure why you are seeing that fallback (of specifically test.nbjdk.home, nbjdk.home sometimes falls back to ${java.home}), but please feel free to fix.
The only use of test.nbjdk.home I see in the sources is in nbbuild/jdk.xml:
        <condition property="test.nbjdk.java" value="${test.nbjdk.home}/bin/java${.exe}" else="${nbjdk.home}/bin/java${.exe}">
            <and>
                <isset property="test.nbjdk.home" />
                <available file="${test.nbjdk.home}/bin/java${.exe}" type="file"/>
            </and>
        </condition>

I.e. the test.nbjdk.java property is used to infer test.nbjdk.java property, and that falls back to nbjdk.home. The uses of test.nbjdk.java are in common.xml, and specify the Java launcher for testing.

I am not sure either, but I had to make few corrective commits to the PR to make it really buildable on JDK-11. For some reason the tests were compiled with JDK8 classpath that contains methods and classes removed in JDK-11. E.g. the compilation of tests picked JAVA_HOME up somehow.

authors: jlahoda jtulach

repo: apache/incubator-netbeans


text:
arch: Call board_reset before up_irq_save and spin_trylock

Summary
since board_reset may call some kernel functions which try to acquire the lock again
Impact
Testing

@xiaoxiang781216 - if board reset call sys reset how does this work?

@xiaoxiang781216 - if board reset call sys reset how does this work?

It should work as before if the implementation is just simply write some registers. The change is only affected the more complex implementation such as need IPC to other core to request the reset.

1.3.428 CONFIG_BOARD_RESET_ON_ASSERT: Enable Board Reset after assert
Type: Integer
Default: 0
Dependencies: [CONFIG_LIB_BOARDCTL](https://cwiki.apache.org/confluence/display/NUTTX/Configuration+Variables#CONFIG_LIB_BOARDCTL), [CONFIG_BOARDCTL_RESET](https://cwiki.apache.org/confluence/display/NUTTX/Configuration+Variables#CONFIG_BOARDCTL_RESET)
Kconfig file: ./boards/Kconfig
== 0 up_assert never reset the machine >= 1 up_assert from interrupt handler or IDLE thread will reset the

machine
>= 2 up_assert from user or kernel thread will reset the machine.
The default behavior is just to kill the asserting thread.

What happens here  line for CONFIG_BOARD_RESET_ON_ASSERT
Wouldn't the CPU reset before it tries the lock?

1.3.428 CONFIG_BOARD_RESET_ON_ASSERT: Enable Board Reset after assert
Type: Integer
Default: 0
Dependencies: [CONFIG_LIB_BOARDCTL](https://cwiki.apache.org/confluence/display/NUTTX/Configuration+Variables#CONFIG_LIB_BOARDCTL), [CONFIG_BOARDCTL_RESET](https://cwiki.apache.org/confluence/display/NUTTX/Configuration+Variables#CONFIG_BOARDCTL_RESET)
Kconfig file: ./boards/Kconfig
== 0 up_assert never reset the machine >= 1 up_assert from interrupt handler or IDLE thread will reset the

machine
>= 2 up_assert from user or kernel thread will reset the machine.
The default behavior is just to kill the asserting thread.

What happens here line for CONFIG_BOARD_RESET_ON_ASSERT
Wouldn't the CPU reset before it tries the lock?

Yes, this is what this patch want, I already high light in the commit message.

OK I was confused by the commit message vs what the Title and Summary says.

repo: apache/incubator-nuttx


text:
Updating Node.js and Docker docs.

Adding common use-case instructions based on community feedback.
Description
Re-wrote Node.js and Docer action documentation to incorporate common issues users having been running into based on my interactions in the external community.
This includes...

Bundling external modules with native deps in Node.js
Using module bundlers to build single source file
Extending custom runtimes with Docker support
Better examples for --native flag.

My changes affect the following components

[ ] Documentation

Types of change

[ ] Documentation changes

Checklist:

[x] I signed an Apache CLA.
[x] I reviewed the style guides and followed the recommendations (Travis CI will check :).
[ ] I added tests to cover my changes.
[ ] My changes require further changes to the documentation.
[x] I updated the documentation where necessary.

Excellent rewrite.

Codecov Report

Merging #4451 into master will decrease coverage by 4.48%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4451      +/-   ##
==========================================
- Coverage   85.33%   80.84%   -4.49%
==========================================
  Files         170      170
  Lines        7929     7929
  Branches      552      552
==========================================
- Hits         6766     6410     -356
- Misses       1163     1519     +356

Impacted Files
Coverage Δ

...core/database/cosmosdb/RxObservableImplicits.scala
0% <0%> (-100%)
:arrow_down:

...core/database/cosmosdb/CosmosDBArtifactStore.scala
0% <0%> (-95.46%)
:arrow_down:

...sk/core/database/cosmosdb/CosmosDBViewMapper.scala
0% <0%> (-92.67%)
:arrow_down:

...whisk/core/database/cosmosdb/CosmosDBSupport.scala
0% <0%> (-84.62%)
:arrow_down:

...abase/cosmosdb/CosmosDBArtifactStoreProvider.scala
4% <0%> (-52%)
:arrow_down:

...in/scala/org/apache/openwhisk/common/Counter.scala
40% <0%> (-20%)
:arrow_down:

...penwhisk/core/database/cosmosdb/CosmosDBUtil.scala
81.81% <0%> (-15.16%)
:arrow_down:

...nwhisk/core/database/cosmosdb/CosmosDBConfig.scala
94.11% <0%> (-5.89%)
:arrow_down:

...rg/apache/openwhisk/common/ForcibleSemaphore.scala
92.3% <0%> (-3.85%)
:arrow_down:

...in/scala/org/apache/openwhisk/common/Logging.scala
69.75% <0%> (-0.62%)
:arrow_down:

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7ecae17...069e3d4. Read the comment docs.

authors: codecov-io jthomas rabbah

repo: apache/incubator-openwhisk