text:
YARN-11709. NodeManager should be shut down or blacklisted when it ca…

…nnot run program /var/lib/yarn-ce/bin/container-executor

Description of PR
How was this patch tested?
For code changes:

[x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
[ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

Thanks @ferdelyi for the change, seems good to me.
Can you please add some sort of unit test to this commit?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 33s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
46m 16s

trunk passed

+1 :green_heart:
compile
1m 26s

trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
compile
1m 31s

trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
checkstyle
0m 40s

trunk passed

+1 :green_heart:
mvnsite
0m 45s

trunk passed

+1 :green_heart:
javadoc
0m 45s

trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javadoc
0m 40s

trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
spotbugs
1m 30s

trunk passed

+1 :green_heart:
shadedclient
38m 32s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 36s

the patch passed

+1 :green_heart:
compile
1m 31s

the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javac
1m 31s

the patch passed

+1 :green_heart:
compile
1m 29s

the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
javac
1m 29s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 29s

the patch passed

+1 :green_heart:
mvnsite
0m 36s

the patch passed

+1 :green_heart:
javadoc
0m 36s

the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2

+1 :green_heart:
javadoc
0m 34s

the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

+1 :green_heart:
spotbugs
1m 35s

the patch passed

+1 :green_heart:
shadedclient
39m 40s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

-1 :x:
unit
25m 16s
/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
hadoop-yarn-server-nodemanager in the patch passed.

+1 :green_heart:
asflicense
0m 34s

The patch does not generate ASF License warnings.

166m 12s

Reason
Tests

Failed junit tests
hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks

Subsystem
Report/Notes

Docker
ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6960/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6960

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 65ca6bf4f6a6 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 061ff5d62e8ef9b1c32022328ad941164531d597

Default Java
Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6960/1/testReport/

Max. process+thread count
762 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6960/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 38s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
45m 27s

trunk passed

+1 :green_heart:
compile
1m 29s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
compile
1m 24s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
checkstyle
0m 40s

trunk passed

+1 :green_heart:
mvnsite
0m 47s

trunk passed

+1 :green_heart:
javadoc
0m 48s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
0m 41s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
1m 29s

trunk passed

+1 :green_heart:
shadedclient
34m 31s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 36s

the patch passed

+1 :green_heart:
compile
1m 19s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javac
1m 19s

the patch passed

+1 :green_heart:
compile
1m 18s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
javac
1m 18s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

-0 :warning:
checkstyle
0m 28s
/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 4 new + 5 unchanged - 0 fixed = 9 total (was 5)

+1 :green_heart:
mvnsite
0m 36s

the patch passed

+1 :green_heart:
javadoc
0m 34s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
0m 32s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
1m 26s

the patch passed

+1 :green_heart:
shadedclient
33m 56s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
24m 46s

hadoop-yarn-server-nodemanager in the patch passed.

+1 :green_heart:
asflicense
0m 38s

The patch does not generate ASF License warnings.

154m 56s

Subsystem
Report/Notes

Docker
ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6960/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6960

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 21643c411af2 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / e6a1038902833bf71d04a3a1de5b5319b7c45ff0

Default Java
Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6960/4/testReport/

Max. process+thread count
553 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6960/4/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 31s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  1s

codespell was not available.

+0 :ok:
detsecrets
0m  1s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:
test4tests
0m  0s

The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
48m 20s

trunk passed

+1 :green_heart:
compile
1m 38s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
compile
1m 33s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
checkstyle
0m 39s

trunk passed

+1 :green_heart:
mvnsite
0m 45s

trunk passed

+1 :green_heart:
javadoc
0m 49s

trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
0m 38s

trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
1m 29s

trunk passed

+1 :green_heart:
shadedclient
38m 48s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 36s

the patch passed

+1 :green_heart:
compile
1m 27s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javac
1m 27s

the patch passed

+1 :green_heart:
compile
1m 22s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
javac
1m 22s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 29s

the patch passed

+1 :green_heart:
mvnsite
0m 37s

the patch passed

+1 :green_heart:
javadoc
0m 35s

the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04

+1 :green_heart:
javadoc
0m 33s

the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05

+1 :green_heart:
spotbugs
1m 29s

the patch passed

+1 :green_heart:
shadedclient
39m 36s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
25m  4s

hadoop-yarn-server-nodemanager in the patch passed.

+1 :green_heart:
asflicense
0m 38s

The patch does not generate ASF License warnings.

168m 13s

Subsystem
Report/Notes

Docker
ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6960/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/6960

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 7306f0e9dabe 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / d5f5c4814201b3127341b031614d54507fe9e702

Default Java
Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6960/5/testReport/

Max. process+thread count
558 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6960/5/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Thanks @ferdelyi LGTM

Thanks for the patch @ferdelyi and thanks for checking it @zeekling. LGTM, merging to trunk.


text:
modified:add a small change.

What's changed?

add a small change.
Checklist

[ ]  I have read the Contributing Guide
[ ]  I have written the necessary doc or comment.
[ ]  I have added the necessary unit tests and all cases have passed.

Add or update API

[ ] I have added the necessary e2e tests and all cases have passed.

Codecov Report
Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 32.17%. Comparing base (948fcb7) to head (8b3b3ad).

Files with missing lines
Patch %
Lines

.../collector/collect/database/JdbcCommonCollect.java
0.00%
0 Missing and 2 partials :warning:

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #2878      +/-   ##
============================================
- Coverage     32.17%   32.17%   -0.01%
+ Complexity     1914     1913       -1
============================================
  Files           507      507
  Lines         18668    18669       +1
  Branches       2276     2277       +1
============================================
  Hits           6006     6006
- Misses        11953    11954       +1
  Partials        709      709

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

source: gharchive/pull-request

authors: TJxiaobao codecov-commenter

repo: apache/hertzbeat

license_type: permissive

license_source: github-api


text:
HIVE-25441: Incorrect deltas split for sub-compactions when using

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

recheck

repo: apache/hive


text:
HIVE-27451: Upgrade Kafka dependencies to latest 3.5.0

What changes were proposed in this pull request?

HIVE-27451
Why are the changes needed?

Keeping Kafka dependencies upto date
Does this PR introduce any user-facing change?

No
Is the change a dependency upgrade?

Yes
How was this patch tested?

Existing UT

Dependency Tree post upgrade: kafkaUpgrade.txt

@akshat0395 more context on the previous upgrade PR-4082:
It was needed to shade zstd-jni in hive-ql after upgrading parquet version dowstream, otherwise Tez couldn't work with parquet and zstd encryption. Parquet and kafka-client both use zstd-jni. In PR-4082 I upgraded kafka-client to latest version which used zstd-jni version close to the version in parquet.

Moving forward with a downgrade in Kafka version as of this point, closing this PR.
PR for downgrade: #4460


text:
Refactor APIs like failed(Exception) to failed(String, Exception)

Better error reporting with this patch gives us:
Exception in thread "main" java.util.concurrent.ExecutionException:
Socket bind failure for socket ServerSocket[unbound],
address=0.0.0.0/0.0.0.0:8080, BacklogSize=0: java.net.BindException:
Address already in use: bind
	at org.apache.hc.core5.concurrent.BasicFuture.getResult(BasicFuture.java:74)
	at org.apache.hc.core5.concurrent.BasicFuture.get(BasicFuture.java:87)
	at org.apache.hc.core5.http.examples.AsyncFileServerExample.main(AsyncFileServerExample.java:165)
Caused by: java.net.BindException: Address already in use: bind
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Unknown Source)
	at sun.nio.ch.Net.bind(Unknown Source)
	at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source)
	at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source)
	at org.apache.hc.core5.reactor.SingleCoreListeningIOReactor.processSessionRequests(SingleCoreListeningIOReactor.java:164)
	at org.apache.hc.core5.reactor.SingleCoreListeningIOReactor.processEvents(SingleCoreListeningIOReactor.java:102)
	at org.apache.hc.core5.reactor.SingleCoreListeningIOReactor.doExecute(SingleCoreListeningIOReactor.java:96)
	at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:81)
	at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
	at java.lang.Thread.run(Unknown Source)
Thu, 04 Apr 2019 20:17:06 GMT | HTTP server shutting down

Instead of just the BindException.

@garydgregory 117 changed files and a massive API change in order to have a slightly different error message? Is this really wise? Why not considering my suggestion of simply wrapping the original exception?

Hi @ok2c,
Why not considering my suggestion of simply wrapping the original exception?
That's how I coded it initially but it feels like a hack and when you look at result it looks like a hack to me; YMMV.
The idea here IMO is that since we are still in beta, is to consider what is the best API moving forward. So for me the breadth of the change is not important as long as the change is for the better.
The way I look at it is:

When I call a failed() API and I have more to say than the exception, having to nest an exception feels weird when all I want is to give some information.
When I implement a failed() API and I get a message and an exception, I can format things maybe in a nicer way rather than just dumping the stack trace and assuming that message at the top is meaningful. If I do dump the stack trace, then I have yet another level of stack frames to look at, which is not so nice in this case.

I'm open to doing it either way, but my guess is that this is not the first time this will come up. In this specific case, it is something that's been bugging me forever, I just never took the time to do anything about it ;-)
Gary

Hi @ok2c,
Please consider changing #processSessionRequests only without breaking the existing APIs.
Do you mean use a nested exception or only change the failed() method directly referenced from processSessionRequests .
Gary

@garydgregory I am afraid I will have to disagree on this being the best API. I think this change is redundant and its breadth shows how intrusive it is.
There is no need to chain exceptions if you think it makes the stack trace harder to read. One can re-throw an exception with more context in it. Please see 335f59b97cbb098a45f3451d8c24c4504b377dc2

@ok2c,
Like this then: https://github.com/apache/httpcomponents-core/pull/118
?
Gary

@garydgregory Yes, that would be my choice.

repo: apache/httpcomponents-core

license_source: bigquery


text:
[SUPPORT] HoodieSparkSqlWriter use sparkContext.hadoopConfiguration when initTable

Describe the problem you faced
In HoodieSparkSqlWriter, the init table() uses sparkContext.adoopConfiguration.
At this time, some configurations set in spark session may not take effect.
Maybe it is better to use sqlContext.sparkSession.sessionState.newHadoopConf here? Just as used by HoodieCatalogTable.initHoodieTable()
HoodieSparkSqlWriter
val tableMetaClient = HoodieTableMetaClient.withPropertyBuilder()
  .setTableType(tableType)
   ...
  .initTable(sparkContext.hadoopConfiguration, path)

HoodieCatalogTable
private val hadoopConf = spark.sessionState.newHadoopConf

HoodieTableMetaClient.withPropertyBuilder()
  .fromProperties(properties)
  ...
  .initTable(hadoopConf, tableLocation)

Environment Description

Hudi version : master

It will be better if you can provide an example that fails or doesn't work because of this.

source: gharchive/issue

authors: YannByron Zouxxyy

repo: apache/hudi


text:
[HUDI-1281] Add deltacommit to ActionType

What is the purpose of the pull request
To add deltacommit to ActionType
Brief change log

Updated ActionType enum to add deltacommit

Verify this pull request
This change added tests and can be verified as follows:

Added unit test testCompletedDeltaCommit in TestMetadataConversionUtils

Committer checklist

[X] Has a corresponding JIRA in PR title & commit

[X] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Codecov Report

Merging #3018 (12372a5) into master (e6a71ea) will increase coverage by 15.68%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master    #3018       +/-   ##
=============================================
+ Coverage     55.14%   70.83%   +15.68%
+ Complexity     3850      385     -3465
=============================================
  Files           485       54      -431
  Lines         23542     2016    -21526
  Branches       2522      241     -2281
=============================================
- Hits          12982     1428    -11554
+ Misses         9406      454     -8952
+ Partials       1154      134     -1020

Flag
Coverage Δ

hudicli
?

hudiclient
?

hudicommon
?

hudiflink
?

hudihadoopmr
?

hudisparkdatasource
?

hudisync
?

huditimelineservice
?

hudiutilities
70.83% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

.../main/java/org/apache/hudi/common/util/Option.java

.../hudi/common/bloom/InternalDynamicBloomFilter.java

...in/java/org/apache/hudi/schema/SchemaProvider.java

...til/jvm/HotSpotMemoryLayoutSpecification64bit.java

...udi/common/util/queue/BoundedInMemoryExecutor.java

...c/main/java/org/apache/hudi/hive/HiveSyncTool.java

.../hudi/common/config/SerializableConfiguration.java

...java/org/apache/hudi/common/util/ParquetUtils.java

.../org/apache/hudi/util/RowDataToAvroConverters.java

...org/apache/hudi/util/StringToRowDataConverter.java

... and 420 more

@nsivabalan  can you pls take a look at this PR

sure.

@vinothchandar  can you please review the PR

LGTM. Re-kicked the CI.

cc @prashantwason as FYI .

authors: codecov-commenter nsivabalan veenaypatil vinothchandar


text:
[minor]The Kryo of Hudi conflicts with the Kryo of Flink

Change Logs
Describe context and summary for this change. Highlight if any code was copied.
Impact
Describe any public API or user-facing feature change or any performance impact.
Risk level: none | low | medium | high
Choose one. If medium or high, explain what verification was done to mitigate the risks.
Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

@cuibo01 @danny0405 do you still plan to land this?

@cuibo01 @danny0405 do you still plan to land this?

Let's not land this until we clarify what exactly caused the conflict. https://github.com/apache/hudi/pull/6522#discussion_r1014118759

@danny0405 do you know if this is still an issue on master and in the recent Hudi releases?

I didn't see any reports about it in recent releases.

I didn't see any reports about it in recent releases.

Got it.  Closing this stale PR now.


text:
Supply a hint arrow schema for casting Parquet field types during scans

This is so as to avoid a potential schema mismatch resulting from upcasting arrow 8 and 16 bit integers to Iceberg 32 bit integer type.
This is one way to resolve #813. Note that this is dependent on https://github.com/apache/arrow-rs/pull/6892 getting merged (and picked up) first.
I still need to think of a proper test case for this too.
Closes #813.

Making this a draft as the upstream dependency is also a draft atm.

repo: apache/iceberg-rust


text:
IGNITE-12111 Cluster ID and tag to identify cluster

Thank you for submitting the pull request to the Apache Ignite.
In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:
The Contribution Checklist

[x] There is a single JIRA ticket related to the pull request.
[x] The web-link to the pull request is attached to the JIRA ticket.
[x] The JIRA ticket has the Patch Available state.
[ ] The pull request body describes changes that have been made.
The description explains WHAT and WHY was made instead of HOW.
[x] The pull request title is treated as the final commit message.
The following pattern must be used: IGNITE-XXXX Change summary where XXXX - number of JIRA issue.
[ ] A reviewer has been mentioned through the JIRA comments
(see the Maintainers list)
[ ] The pull request has been checked by the Teamcity Bot and
the green visa attached to the JIRA ticket (see TC.Bot: Check PR)

Notes

How to Contribute
Coding abbreviation rules
Coding Guidelines
Apache Ignite Teamcity Bot

If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com #ignite channel.

Can you please rebase this branch properly, so that it contains only your changes?

repo: apache/ignite


text:
[AIRFLOW-1213] Add hcatalog parameters to sqoop

Dear Airflow maintainers,
This commit will add additional parameters to specify a hive table instead of a hdfs directory. This will also source the schema from the hive metastore.
JIRA

[x] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-1213] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-1213

Description

[x] Here are some details about my PR, including screenshots of any UI changes:
This commit will add additional parameters to specify a hive table instead of a hdfs directory. This will also source the schema from the hive metastore.

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Codecov Report

Merging #2305 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2305      +/-   ##
==========================================
+ Coverage   69.11%   69.12%   +<.01%
==========================================
  Files         144      144
  Lines       11023    11023
==========================================
+ Hits         7619     7620       +1
+ Misses       3404     3403       -1

Impacted Files
Coverage Δ

airflow/models.py
87.1% <0%> (+0.04%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d2d3e49...039822b. Read the comment docs.

Adding these parameters before properties breaks functionality for people using positional arguments. Could you move to the end? Or does that not make sense w.r.t. usage.
Also, adding in every option seems a bit brittle ... As none of these fields are template fields, wouldn't it be better to let users specify a dict and build commands from that? That will work better sqoop API changes.
I can add in an example later.

Hi @hgrif, good catch. I think most people write it like:
sqoop_job = SqoopOperator(
    task_id='sqoop_job',
    conn_id='production_postgres',
    cmd_type='export',
    input_null_string="\\\\N",
    input_null_non_string="\\\\N",
    delimiter="|",
    table="sales",
    export_dir='/apps/hive/warehouse/bi/sales',
    num_mappers='1',
    dag=dag
)

Then it doesn't break, but just to be sure.
Regarding supplying a dict, I think it is more elegant. On the other side, the operator might be a bit brittle in terms of writing them, but this is done only once. Every time you use an operator, and you have to figure out the command, this might also open up room for errors.

OK!
Rewriting it is maybe something for another PR. 👍

authors: Fokko codecov-io hgrif

repo: apache/incubator-airflow


text:
Add http logger Chinese docs

NOTE: Please read the Contributing.md guidelines before submitting your patch:
https://github.com/apache/incubator-apisix/blob/master/Contributing.md#how-to-add-a-new-feature-or-change-an-existing-one
Summary
SUMMARY_HERE
Full changelog

[Implement ...]
[Add related tests]
...

Issues resolved
Fix #1562

@scaat many thx ^_^

authors: membphis scaat

repo: apache/incubator-apisix


text:
[CELEBORN-627][FLINK] Support read split partitions.

What changes were proposed in this pull request?
Support read split partitions.
Why are the changes needed?
Ditto.
Does this PR introduce any user-facing change?
NO.
How was this patch tested?
UT and cluster run.

This patch will be tested on the cluster after #1519 is merged.

@zhongqiangczq I have tested it with split disabled. We need to cooperate to support split reading partitions for Flink.

LGTM

repo: apache/incubator-celeborn


text:
[Feature][zentao] Collect&Extract&Convert stories

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
Collect&Extract&Convert stories
Use case
Collect&Extract&Convert stories
Related issues
#2961
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I am willing to submit a PR

resolved by #3181

repo: apache/incubator-devlake


text:
[Bug][starrocks] failed when run starrocks tasks: cached plan must not change result type

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
It report an error of cached plan must not change result type (SQLSTATE 0A000) when I using starrocks plugin to sync data from pg to starrocks.
What you expected to happen
Should not cause an error.
How to reproduce

Setup devlake with a new postgres database and a starrocks service.
Create pipeline with following plans:

{
    "name": "sync to starrocks",
    "plan": [
        [
            {
                "plugin": "starrocks",
                "options": {
                    "host": "starrocks-fe",
                    "port": 9030,
                    "user": "root",
                    "be_port": 8040,
                    "be_host": "starrocks-be",
                    "database": "lake",
                    "password": "password",
                    "batch_size": 10000,
                    "tables": [
                        "^[a-zA-Z].+$"
                    ]
                }
            }
        ]
    ]
}

Anything else
log
Version

Build based on release-v0.14 branch

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

starrocks the way of alter table need to update at func loadData().
such as the way of jira plugin:
https://github.com/apache/incubator-devlake/blob/main/plugins/jira/models/migrationscripts/20220716_add_init_tables.go
starrocks the way of alter table need to update at func loadData().
such as the way of jira plugin:
https://github.com/apache/incubator-devlake/blob/main/plugins/jira/models/migrationscripts/20220716_add_init_tables.go

authors: abeizn narrowizard


text:
Update load-json-format.md

stripe_outer_array to strip_outer_array
Proposed changes
Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.
Types of changes
What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation Update (if none of the other choices apply)
[ ] Code refactor (Modify the code structure, format the code, etc...)
[ ] Optimization. Including functional usability improvements and performance improvements.
[ ] Dependency. Such as changes related to third-party components.
[ ] Other.

Checklist
Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[ ] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail
[ ] Compiling and unit tests pass locally with my changes
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] If these changes need document changes, I have updated the document
[ ] Any dependent changes have been merged

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

You also need to check the English version of the documentation.

authors: qidaye wangliansong

repo: apache/incubator-doris


text:
longMin/longMax has different result, should unify?

two sql statement: (col < 0) is always false

select max(col) filter(where col < 0) from datasource;
select max(col) from datasource where col <0;

we convert these two into two http message while both use longMax aggregator, but 1. sql will return Long.MIN_VALUE, 2. sql will return empty. should these two results combined one?

What would SQL standard behavior be?
I think we're not standard for query (2), it should probably return null. I'm not sure what query (1) should return.

authors: dijkspicy gianm

repo: apache/incubator-druid


text:
user input check

frontend and backend validation check

use http status code from backend to handle response

repo: apache/incubator-dubbo-ops


text:
3.x dev cache

What is the purpose of the change
improve the implementation of cache，especially ThreadLocalCache and ExpiringCache, to prevents memory leak.
Brief changelog
XXXXX
Verifying this change
XXXXX
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[ ] Run mvn clean install -DskipTests & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
[ ] If this contribution is large, please follow the Software Donation Guide.

Codecov Report

Merging #2594 into 3.x-dev will decrease coverage by 0.06%.
The diff coverage is 0%.

@@              Coverage Diff              @@
##             3.x-dev    #2594      +/-   ##
=============================================
- Coverage      54.93%   54.87%   -0.07%
+ Complexity      5276     5272       -4
=============================================
  Files            573      573
  Lines          25517    25534      +17
  Branches        4531     4531
=============================================
- Hits           14018    14011       -7
- Misses          9395     9416      +21
- Partials        2104     2107       +3

Impacted Files
Coverage Δ
Complexity Δ

...bo/cache/support/threadlocal/ThreadLocalCache.java
58.33% <0%> (-41.67%)
3 <0> (ø)

.../org/apache/dubbo/cache/support/jcache/JCache.java
74.07% <0%> (-12.89%)
4 <0> (ø)

...a/org/apache/dubbo/cache/support/lru/LruCache.java
63.63% <0%> (-36.37%)
3 <0> (ø)

...he/dubbo/cache/support/expiring/ExpiringCache.java
71.42% <0%> (-28.58%)
3 <0> (ø)

...che/dubbo/remoting/transport/mina/MinaChannel.java
42.25% <0%> (-11.27%)
15% <0%> (-1%)

...org/apache/dubbo/rpc/filter/ActiveLimitFilter.java
77.77% <0%> (-11.12%)
5% <0%> (-2%)

...ubbo/rpc/protocol/dubbo/ChannelWrappedInvoker.java
37.5% <0%> (-4.17%)
3% <0%> (ø)

...rpc/protocol/dubbo/telnet/InvokeTelnetHandler.java
54.21% <0%> (-3.62%)
13% <0%> (-2%)

...he/dubbo/registry/multicast/MulticastRegistry.java
55.6% <0%> (-1.73%)
35% <0%> (-3%)

.../java/org/apache/dubbo/config/ReferenceConfig.java
50.71% <0%> (-0.36%)
42% <0%> (ø)

... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2faf5e7...40ad5ca. Read the comment docs.

Do not agree with this pr. Unless a complete set of expiration policies can be provided.

Hi, @dugenkui03
I will close this pr. Expect you to be able to provide a complete cleanup caching strategy.
:)

authors: carryxyh codecov-io dugenkui03

repo: apache/incubator-dubbo


text:
[Dubbo-2838] Fix UT failed on windows with FileNetworkerTest

What is the purpose of the change
Fix Dubbo-2838
Brief changelog
Fix FileNetworkerTest failure
Verifying this change
Run UT on windows platform
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn clean install -DskipTests & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
[x] If this contribution is large, please follow the Software Donation Guide.

Codecov Report

Merging #2839 into master will decrease coverage by 0.16%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2839      +/-   ##
==========================================
- Coverage    63.8%   63.63%   -0.17%
==========================================
  Files         577      577
  Lines       25930    25930
  Branches     4541     4541
==========================================
- Hits        16544    16501      -43
- Misses       7239     7269      +30
- Partials     2147     2160      +13

Impacted Files
Coverage Δ

...onfig/spring/extension/SpringExtensionFactory.java
68.42% <0%> (-15.79%)
:arrow_down:

.../remoting/transport/netty4/NettyClientHandler.java
75% <0%> (-13.89%)
:arrow_down:

.../apache/dubbo/qos/protocol/QosProtocolWrapper.java
69.23% <0%> (-12.83%)
:arrow_down:

...ache/dubbo/remoting/transport/AbstractChannel.java
75% <0%> (-12.5%)
:arrow_down:

...org/apache/dubbo/rpc/filter/ActiveLimitFilter.java
77.77% <0%> (-11.12%)
:arrow_down:

...ubbo/rpc/protocol/dubbo/ChannelWrappedInvoker.java
41.66% <0%> (-8.34%)
:arrow_down:

.../apache/dubbo/remoting/transport/AbstractPeer.java
58.69% <0%> (-6.53%)
:arrow_down:

.../dubbo/remoting/transport/netty4/NettyChannel.java
61.17% <0%> (-4.71%)
:arrow_down:

...ing/exchange/support/header/AbstractTimerTask.java
66.66% <0%> (-4.17%)
:arrow_down:

...rpc/protocol/dubbo/telnet/InvokeTelnetHandler.java
54.21% <0%> (-3.62%)
:arrow_down:

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 46b3458...311acf5. Read the comment docs.

Fix #2838

I have seen another pr to fix this issue:
https://github.com/apache/incubator-dubbo/pull/2848
Personally think that his modification is simpler and effective.
What do u think about it?

Please refer to https://en.wikipedia.org/wiki/File_URI_scheme
Actually an additional slash is only for Windows local file which hostname is omitted.
And for Unix like system remains the same. (file:///tmp/junit2484075406115019206/junit2016855624112146186.tmp ,three slashes in total including the root path one)
In fact the UT passed, so I think both pr are fine.  😄

Hi,
We don't need to judge our system, we just need to add a '/'. The content after the third '/' will be considered the path of the URL (the current way of parsing dubbo). I know that this problem is caused by the separator of WINDOWS. But in the way of fixing, I think we only need to add a '/' to move closer to dubbo's parsing.
Another thing is that that pr is simpler. So I personally prefer that one.  Thx for your pr, too. Very nice solution. 👍

authors: biyuhao carryxyh codecov-io


text:
Code review

What is the purpose of the change
XXXXX
Brief changelog
XXXXX
Verifying this change
XXXXX
Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a GITHUB_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GITHUB issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Format the pull request title like [Dubbo-XXX] Fix UnknownException when host config not exist #XXX. Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[ ] Run mvn clean install -DskipTests=false & mvn clean test-compile failsafe:integration-test to make sure unit-test and integration-test pass.
[ ] If this contribution is large, please follow the Software Donation Guide.

@zonghaishang It seems that the commits are not correctly linked to the author, which may be bad practice in GitHub repo and hard to track the author.


text:
官方实例中的中国地图是否与自然资源部地图技术审查中心的一致呢

What problem does this feature solve?
官方的ECharts 地图数据在线生成工具何时会完成整改？
What does the proposed API look like?

ECharts 官方不提供地图下载

repo: apache/incubator-echarts


text:
Is it possible to create an EEG chart?

First of all, thank you for providing these libraries.
Is it possible to create an EEG chart?
It's on one x-axis (TIME SERIRES)
Multiple y-axes with different ranges are indicated.
Thank you

One-line summary [问题简述]
Version & Environment [版本及环境]

ECharts version [ECharts 版本]:
Browser version [浏览器类型和版本]:
OS Version [操作系统类型和版本]:

Expected behaviour [期望结果]
ECharts option [ECharts配置项]

option = {

}

Other comments [其他信息]

See this example: http://gallery.echartsjs.com/editor.html?c=xrkeZYLl6G


text:
Look into making FluoConfiguration Serializable

Was looking into writing a Spark code using Fluo.  I was expecting FluoConfiguration to be serializable and it was not.

In the next release of Fluo, FluoConfiguration will be serializable.  There are many ways to work around this issue before the next release.  Below is an example of one way.
void doFluoStuff(FluoConfiguration fconf, JavaRDD<?> someRdd){
  //this map is serializable
  Map<String, String> confMap = fconf.toMap();

  someRdd.foreachPartition(iter->{
    FluoConfiguration fconf = new FluoConfiguration(confMap);
    try(FluoClient client = FluoFactory.newClient(fconf)){
      //do fluo stuff
    }
  }
}

repo: apache/incubator-fluo


text:
HAWQ-338. refer build instruction to wiki page

So we can only keep one copy, and the latest copy.

+1

The fix looks good. +1

Open for 29 days, +2 on votes, can we get this merged?

authors: cwelton huor radarwave yaoj2

repo: apache/incubator-hawq


text:
HAWQ-931. ORC optimized profile for PPD/CP

First version of the ORC optimized profile with support for Filter pushdown and Column projection.
We are still using Hive 1.2.x ORC apis.
Known issues: Operators on String data type doesn't work

@shivzone Can you explain why we need a separate Resolver for ORC?

LGTM, nice job

I will be sending a new PR with HAWQ-971 branch. All the above CR fixes will go into that.

authors: kavinderd shivzone


text:
Uploader implementations may leak storage when resubmitting the same analytic multiple times

Describe the bug
The Uploader logic in Heron adds a Random Long to the filename that causes memory leak in Bookkeeper.
The Uploader filename generation logic is here.
And the Dlog config that disables time and size based retention is here.
Eventually the Bookkeeper log runs out of space and no more topologies can be submitted.
The proposed fix is to remove the Random.long that is being added to the filename.

Would using RandomStringUtils#random with a character count of 11 fix the issue? Or is this a string length related issue?
If my math is correct it should generate 26 uppercase + 26 lowercase + 10 digits permute 11 count = 62 p 11 = 390,164,706,723,052,800 potential strings. That is more options than Random#long provides.

So my suggestion is to remove the Random.long all together. I'm not sure why this was added, but we don't have any ability to "roll-back" to the previous uploads. So having an endless series of entries in Bookkeeper doesn't make sense. If I resubmit the same topology with name, role, version coordinates, I would expect it to replace the previous entry in Bookkeeper (or whatever upload mechanism is used). For example when developing and testing an analytic job.
If the desire is to run different binaries, one should distinguish the name, role and/or version to create a unique coordinate in the Upload/Download mechanism.
We will have to test the Upload functionality to ensure that removing the Random.long doesn't break anything.

Is that possible to, either defined by user, or set it with timestamp instead?

authors: nicknezis surahman windhamwong

repo: apache/incubator-heron


text:
[INLONG-4040][Manager] Fix hive sink table data path

Fixes #4040
Tested in local env.

It turns out that the user should specify full path rather than just warehouse prefix.

repo: apache/incubator-inlong


text:
Review DRL6Expressions.g4 to replace IDENTIFIER with drlIdentifier

Parent issue

#5678

Failing tests
Not identified at the moment.
Issue
https://github.com/apache/incubator-kie-drools/issues/5776 covers only primary rule. This issue is to

Review whole DRL6Expressions.g4 to replace IDENTIFIER with drlIdentifier if required.
Add unit tests for the cases. They may be minor edge cases.

/take

Done by https://github.com/apache/incubator-kie-drools/pull/5875

repo: apache/incubator-kie-drools


text:
METRON-514: Snort kafka topic fails to get any data on a setup running for several days

This PR attempts to solve the issue of snort data not flowing after an extended amount of time by removing Flume as a dependency and simplifying the process.  Flume is replaced by a combination of the "tail" command and kafka-console-producer, making it much easier to troubleshoot.
The implementation was inspired by the yaf sensor.  The one significant change is in the "metron-deployment/roles/snort/templates/snort-producer" init script.  The yaf init script stops the child processes by name with "killall" which doesn't work in this case because "snort-producer" is not in the first "tail" command.  Instead the process group id is retrieved in line 64 and passed to the "kill" command in line 67.
This has been tested in quick-dev, including managing the sensor through Monit.  I searched for other Flume dependencies but could not find any, therefore Flume was removed completely.

Is there a jira for this?

Thanks @merrimanr, based on your testing I'm +1 by inspection (travis pending)

https://issues.apache.org/jira/browse/METRON-514
On Tue, Oct 25, 2016 at 9:57 PM, ottobackwards notifications@github.com
wrote:

Is there a jira for this?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-metron/pull/323#issuecomment-256236889,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AATCePIs_4uJ2Fj9SRMOqqyeYNqqX2tKks5q3sGSgaJpZM4KgKn4
.

@merrimanr - I was hoping to merge this in today, but it's conflicted. Think you'd have time to take a peek?

@dlyle65535 conflict is resolved

authors: dlyle65535 merrimanr ottobackwards

repo: apache/incubator-metron


text:
METRON-548 Improve Profiler documentation

Reorganized the README of the Profiler to be more sensible, adding links to other reference material like the Profiler Client, and added an additional example showing how the Profiler pairs nicely with Stellar's STATS package.

Thanks @mattf-horton .  I made the changes that you suggested.

+1 by inspection.  I like the stats example

authors: cestella nickwallen


text:
Inconsistent naming of pow/power between mx.symbol and mx.nd

mx.nd.pow and mx.symbol.power are named inconsistently. As a result, in a HybridBlock, one cannot do F.pow or F.powerl directly.

It's indeed a problem... To ensure backward compatibility I'll add both nd.power and sym.pow. How do you like such a fix?

@haojin2 That'll be perfect! Thanks!

@YutingZhang WIP, will let you know when the PR is out, should be within today.

@YutingZhang  Fix in #14606

@mxnet-label-bot update [Bug, operator]

@YutingZhang Fix merged. Please let me know if the latest master solves your issue.

Thanks! @haojin2

repo: apache/incubator-mxnet


text:
Why is LOG(INFO)  not working as supposed in mxnet C++ code ?

I ran example/image-classification/train_mnist.py with batch_size = 10000, so there would be 6 batches for the forword training ( val is set to None)，the Forword function of the first Fully-connect layer should be called 6 times. I added some debug codes as follows:

But the log printed  is not what I supposed:

why is "logger.info(load123)" in python executed 6 times as supposed, while  "LOG(INFO) << "forward123" in fully_connected-inl.h just once ?
@piiswrong

add execmanager.update_metric at the end

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!

authors: aaronwater piiswrong yajiedesign


text:
[CI branch] script for building openblas from source

Description
script for building openblas from source

@lanking520


text:
add pos_weight for SigmoidBinaryCrossEntropyLoss

Description
Add pos_weight for SigmoidBinaryCrossEntropyLoss.
A value pos_weights > 1 decreases the false negative count, hence increasing the recall.
Conversely setting pos_weights < 1 decreases the false positive count and increases the precision.
This can be seen from the fact that pos_weight is introduced as a multiplicative coefficient for the positive targets term in the loss expression:
label * -log(sigmoid(pred)) * pos_weight + (1 - label) * -log(1 - sigmoid(pred))
It's adopted from tensorflow's implementation
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[x] Changes are complete (i.e. I finished coding on this PR)
[x] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
[x] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes
Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

@eureka7mt Thanks for the contribution, could you add a unit test for this case?

@mxnet-label-bot add[Gluon, pr-awaiting-review]

@eureka7mt could you re-trigger the CI?

@eureka7mt Could you please look into the CI failures?

@mxnet-label-bot update [pr-awaiting-merge]

Add the if-else statement make an error.Though the default value of pos_weight is set to be 1,the pos_weight is an (1,N) NDArray.

@eureka7mt
I see. Since pos_weight is a tensor, it is better to  default pos_weight=None as you wrote before. I will update it.

@wkcn It could be a scalar for classifying a single class.But for multi-class and multi-label classifying,it should be a tensor.Because in this situation,the number of positive examples and negative examples isn't same for each class.
And in pytorch,it's also defined as a tensor.See the pytorch docs.

I change the order of SigmoidBinaryCrossEntropyLoss inputs from (self, F, pred, label, pos_weight=None, sample_weight=None) to (self, F, pred, label, sample_weight=None, pos_weight=None), since we need the compatibilty for other projects

Sorry that I trigger the sanity problem.
Could someone please help me solve it? Thanks!

Maybe the broadcast_mul isn't necessary.I think that a NDArray * a NDArray will do broadcast_mul automatically.

@eureka7mt I think we may pass Symbol into SigmoidBinaryCrossEntropyLoss.

The PR has been merged.
Thanks for your contribution: )

authors: anirudhacharya ankkhedia eureka7mt roywei vandanavk wkcn


text:
include/nuttx:  Fix improper use of inline

I finally figured out why the ez80 code has gotten so big.  It is because people have been put putting big inline functions in header files.  That is a violation of the coding standard, since only c89 compatibility is required in all common code. But we have been tolerating inline function it because include/nuttx/compiler.h defines 'inline' to be nothing for C89 compilers.
As a result, static inline functions declared within a C file not so bad; the inline qualifier is ignored, if not supported, but otherwise all is well.
But it is catastrophic in header files.  Those static inline functions are included as static functions and implemented in EVERY file that includes those header files, even if the functions are never called.  That makes the code base huge!So there is another PR coming to fix some of the worst offenders.
This commit fixes two of the worst offenders I have encountered so far:  include/nuttx/sempahore.h and cache.h.  But there may be a few other changes needed.  Under include/nuttx there are still inline functions thread.h, inclue/nuttx/list.h, mutex.h, tree.h, and include/nuttx/crypto/blake2s.h with no protection for compilers that do not handler the inline qualifier.  Otherwise we are clean.
With the changes to these two header files, the size of the z20x build is reduced by about 40%.  And incredible size savings.

On Mon, Mar 2, 2020 at 3:53 PM patacongo notifications@github.com wrote:

I finally figured out why the ez80 code has gotten so big. It is because
people have been put putting big inline functions in header files. That is
a violation of the coding standard, since only c89 compatibility is
required in all common code. But we have been tolerating inline function it
because include/nuttx/compiler.h defines 'inline' to be nothing for C89
compilers.
As a result, static inline functions declared within a C file not so bad;
the inline qualifier is ignored, if not supported, but otherwise all is
well.
But it is catastrophic in header files. Those static inline functions are
included as static functions and implemented in EVERY file that includes
those header files, even if the functions are never called. That makes the
code base huge!So there is another PR coming to fix some of the worst
offenders.
This commit fixes two of the worst offenders I have encountered so far:
include/nuttx/sempahore.h and cache.h. But there may be a few other changes
needed. Under include/nuttx there are still inline functions thread.h,
inclue/nuttx/list.h, mutex.h, tree.h, and include/nuttx/crypto/blake2s.h
with no protection for compilers that do not handler the inline qualifier.
Otherwise we are clean.
With the changes to these two header files, the size of the z20x build is
reduced by about 40%. And incredible size savings.

Wow that is significant!!
Good catch!
Nathan

Wow that is significant!!

Yes... the inline functions are not really large.  However, almost every C file in system includes nuttx/semaphore.h, either directly or indirectly.  With several hundreds of C files and several hundreds of copies of these functions, it adds up.  I expect to see a size improvement, but I as also rather astounded by the magnitude of the savings.

authors: hartmannathan patacongo

repo: apache/incubator-nuttx


text:
fix failing test and other test cleanup

fix for the CloudantMultiWorkerTests to use whisk properties instead of pureconfig
test cleanup

@dubeejw


text:
gradle build failed with Error response from daemon: no such id: scala

Environment details:
Distributed ubuntu14.04 env
docker version
Version:      1.12.0
API version:  1.24
Go version:   go1.6.3
Git commit:   8eab29e
Built:        Thu Jul 28 22:00:36 2016
OS/Arch:      linux/amd64
Steps to reproduce the issue:

gradle build fails while building for scala as  below:

./gradlew distDocker -PdockerHost=100.100.100.91:4243 -PdockerRegistry=100.100.100.91:5000
.
.
Sending build context to Docker daemon 479.2 kB
Step 1 : FROM buildpack-deps:trusty-curl
---> dd89f3a8227b
Step 2 : ENV DEBIAN_FRONTEND noninteractive
---> Running in 90a85dd4ff25
---> b0bad491b7e7
Removing intermediate container 90a85dd4ff25
Step 3 : RUN locale-gen en_US.UTF-8
---> Running in 858c587074eb
Generating locales...
en_US.UTF-8... done
Generation complete.
---> 10925676ce68
Removing intermediate container 858c587074eb
Step 4 : ENV LANG en_US.UTF-8
---> Running in ce567f0892e6
---> dc74d7cba2a9
Removing intermediate container ce567f0892e6
Step 5 : ENV LANGUAGE en_US:en
---> Running in 4049f8aa8d16
---> 9b56e6feadfa
Removing intermediate container 4049f8aa8d16
Step 6 : ENV LC_ALL en_US.UTF-8
---> Running in 21520a916c33

Task :common:scala:distDocker
Wed Dec 06 07:40:38 UTC 2017: Executing 'docker --host 100.100.100.91:4243 build -t scala /root/openwhisk/common/scala'

---> a0810f08e455
Removing intermediate container 21520a916c33
Step 7 : ENV VERSION 8
---> Running in 42b183ee95ab
---> 1ec97a8b361c
Removing intermediate container 42b183ee95ab
Step 8 : ENV UPDATE 141
---> Running in 5f2c82c4b2a3
---> 7c5b37147de6
Removing intermediate container 5f2c82c4b2a3
Step 9 : ENV BUILD 15
---> Running in 91af5eacc13c
---> bf681366e625
Removing intermediate container 91af5eacc13c
Step 10 : ENV SIG 336fa29ff2bb4ef291e347e091f7f4a7
---> Running in ac76a06ba91f
---> 7d37a12523aa
Removing intermediate container ac76a06ba91f
Step 11 : ENV JAVA_HOME /usr/lib/jvm/java-${VERSION}-oracle
---> Running in e823c44a0595
---> 6829e7b06f88
Removing intermediate container e823c44a0595
Step 12 : ENV JRE_HOME ${JAVA_HOME}/jre
---> Running in de3a92dca11e
---> 2987a7c66322
Removing intermediate container de3a92dca11e
Step 13 : ENV PATH $JAVA_HOME/bin:$PATH
---> Running in 0fca14db6955
---> 17562397b05d
Removing intermediate container 0fca14db6955
Step 14 : RUN curl --silent --location --retry 3 --cacert /etc/ssl/certs/GeoTrust_Global_CA.pem   --header "Cookie: oraclelicense=accept-securebackup-cookie;"   http://download.oracle.com/otn-pub/java/jdk/"${VERSION}"u"${UPDATE}"-b"${BUILD}"/"${SIG}"/jdk-"${VERSION}"u"${UPDATE}"-linux-x64.tar.gz   | tar xz -C /tmp &&   mkdir -p /usr/lib/jvm && mv /tmp/jdk1.${VERSION}.0_${UPDATE} "${JAVA_HOME}" &&   apt-get autoclean && apt-get --purge -y autoremove &&   rm -rf /tmp/* /var/tmp/*
---> Running in 0b7cfcc2e940
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/sh -c curl --silent --location --retry 3 --cacert /etc/ssl/certs/GeoTrust_Global_CA.pem   --header "Cookie: oraclelicense=accept-securebackup-cookie;"   http://download.oracle.com/otn-pub/java/jdk/"${VERSION}"u"${UPDATE}"-b"${BUILD}"/"${SIG}"/jdk-"${VERSION}"u"${UPDATE}"-linux-x64.tar.gz   | tar xz -C /tmp &&   mkdir -p /usr/lib/jvm && mv /tmp/jdk1.${VERSION}.0_${UPDATE} "${JAVA_HOME}" &&   apt-get autoclean && apt-get --purge -y autoremove &&   rm -rf /tmp/* /var/tmp/*' returned a non-zero code: 2
Sending build context to Docker daemon 479.2 kB

Task :common:scala:distDocker
Wed Dec 06 07:42:11 UTC 2017: Command 'docker --host 100.100.100.91:4243 build -t scala /root/openwhisk/common/scala' failed with exitCode 2, 2 retries left, retrying...
Wed Dec 06 07:42:11 UTC 2017: Executing 'docker --host 100.100.100.91:4243 build -t scala /root/openwhisk/common/scala'

Step 1 : FROM buildpack-deps:trusty-curl
---> dd89f3a8227b
Step 2 : ENV DEBIAN_FRONTEND noninteractive
---> Using cache
---> b0bad491b7e7
Step 3 : RUN locale-gen en_US.UTF-8
---> Using cache
---> 10925676ce68
Step 4 : ENV LANG en_US.UTF-8
---> Using cache
---> dc74d7cba2a9
Step 5 : ENV LANGUAGE en_US:en
---> Using cache
---> 9b56e6feadfa
Step 6 : ENV LC_ALL en_US.UTF-8
---> Using cache
---> a0810f08e455
Step 7 : ENV VERSION 8
---> Using cache
---> 1ec97a8b361c
Step 8 : ENV UPDATE 141
---> Using cache
---> 7c5b37147de6
Step 9 : ENV BUILD 15
---> Using cache
---> bf681366e625
Step 10 : ENV SIG 336fa29ff2bb4ef291e347e091f7f4a7
---> Using cache
---> 7d37a12523aa
Step 11 : ENV JAVA_HOME /usr/lib/jvm/java-${VERSION}-oracle
---> Using cache
---> 6829e7b06f88
Step 12 : ENV JRE_HOME ${JAVA_HOME}/jre
---> Using cache
---> 2987a7c66322
Step 13 : ENV PATH $JAVA_HOME/bin:$PATH
---> Using cache
---> 17562397b05d
Step 14 : RUN curl --silent --location --retry 3 --cacert /etc/ssl/certs/GeoTrust_Global_CA.pem   --header "Cookie: oraclelicense=accept-securebackup-cookie;"   http://download.oracle.com/otn-pub/java/jdk/"${VERSION}"u"${UPDATE}"-b"${BUILD}"/"${SIG}"/jdk-"${VERSION}"u"${UPDATE}"-linux-x64.tar.gz   | tar xz -C /tmp &&   mkdir -p /usr/lib/jvm && mv /tmp/jdk1.${VERSION}.0_${UPDATE} "${JAVA_HOME}" &&   apt-get autoclean && apt-get --purge -y autoremove &&   rm -rf /tmp/* /var/tmp/*
---> Running in 07a905979251
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/sh -c curl --silent --location --retry 3 --cacert /etc/ssl/certs/GeoTrust_Global_CA.pem   --header "Cookie: oraclelicense=accept-securebackup-cookie;"   http://download.oracle.com/otn-pub/java/jdk/"${VERSION}"u"${UPDATE}"-b"${BUILD}"/"${SIG}"/jdk-"${VERSION}"u"${UPDATE}"-linux-x64.tar.gz   | tar xz -C /tmp &&   mkdir -p /usr/lib/jvm && mv /tmp/jdk1.${VERSION}.0_${UPDATE} "${JAVA_HOME}" &&   apt-get autoclean && apt-get --purge -y autoremove &&   rm -rf /tmp/* /var/tmp/*' returned a non-zero code: 2
Sending build context to Docker daemon 479.2 kB

Task :common:scala:distDocker
Wed Dec 06 07:43:33 UTC 2017: Command 'docker --host 100.100.100.91:4243 build -t scala /root/openwhisk/common/scala' failed with exitCode 2, 1 retries left, retrying...
Wed Dec 06 07:43:33 UTC 2017: Executing 'docker --host 100.100.100.91:4243 build -t scala /root/openwhisk/common/scala'

Step 1 : FROM buildpack-deps:trusty-curl
---> dd89f3a8227b
Step 2 : ENV DEBIAN_FRONTEND noninteractive
---> Using cache
---> b0bad491b7e7
Step 3 : RUN locale-gen en_US.UTF-8
---> Using cache
---> 10925676ce68
Step 4 : ENV LANG en_US.UTF-8
---> Using cache
---> dc74d7cba2a9
Step 5 : ENV LANGUAGE en_US:en
---> Using cache
---> 9b56e6feadfa
Step 6 : ENV LC_ALL en_US.UTF-8
---> Using cache
---> a0810f08e455
Step 7 : ENV VERSION 8
---> Using cache
---> 1ec97a8b361c
Step 8 : ENV UPDATE 141
---> Using cache
---> 7c5b37147de6
Step 9 : ENV BUILD 15
---> Using cache
---> bf681366e625
Step 10 : ENV SIG 336fa29ff2bb4ef291e347e091f7f4a7
---> Using cache
---> 7d37a12523aa
Step 11 : ENV JAVA_HOME /usr/lib/jvm/java-${VERSION}-oracle
---> Using cache
---> 6829e7b06f88
Step 12 : ENV JRE_HOME ${JAVA_HOME}/jre
---> Using cache
---> 2987a7c66322
Step 13 : ENV PATH $JAVA_HOME/bin:$PATH
---> Using cache
---> 17562397b05d
Step 14 : RUN curl --silent --location --retry 3 --cacert /etc/ssl/certs/GeoTrust_Global_CA.pem   --header "Cookie: oraclelicense=accept-securebackup-cookie;"   http://download.oracle.com/otn-pub/java/jdk/"${VERSION}"u"${UPDATE}"-b"${BUILD}"/"${SIG}"/jdk-"${VERSION}"u"${UPDATE}"-linux-x64.tar.gz   | tar xz -C /tmp &&   mkdir -p /usr/lib/jvm && mv /tmp/jdk1.${VERSION}.0_${UPDATE} "${JAVA_HOME}" &&   apt-get autoclean && apt-get --purge -y autoremove &&   rm -rf /tmp/* /var/tmp/*
---> Running in 35f8c2fe3cbf
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/sh -c curl --silent --location --retry 3 --cacert /etc/ssl/certs/GeoTrust_Global_CA.pem   --header "Cookie: oraclelicense=accept-securebackup-cookie;"   http://download.oracle.com/otn-pub/java/jdk/"${VERSION}"u"${UPDATE}"-b"${BUILD}"/"${SIG}"/jdk-"${VERSION}"u"${UPDATE}"-linux-x64.tar.gz   | tar xz -C /tmp &&   mkdir -p /usr/lib/jvm && mv /tmp/jdk1.${VERSION}.0_${UPDATE} "${JAVA_HOME}" &&   apt-get autoclean && apt-get --purge -y autoremove &&   rm -rf /tmp/* /var/tmp/*' returned a non-zero code: 2

Task :common:scala:distDocker
Wed Dec 06 07:44:54 UTC 2017: Command 'docker --host 100.100.100.91:4243 build -t scala /root/openwhisk/common/scala' failed with exitCode 2, no more retries left, aborting...

Error response from daemon: no such id: scala
Error response from daemon: no such id: scala
Error response from daemon: no such id: scala

Task :common:scala:tagImage
Wed Dec 06 07:44:54 UTC 2017: Executing 'docker --host 100.100.100.91:4243 tag scala 100.100.100.91:5000/whisk/scala:latest'
Wed Dec 06 07:44:54 UTC 2017: Command 'docker --host 100.100.100.91:4243 tag scala 100.100.100.91:5000/whisk/scala:latest' failed with exitCode 1, 2 retries left, retrying...
Wed Dec 06 07:44:54 UTC 2017: Executing 'docker --host 100.100.100.91:4243 tag scala 100.100.100.91:5000/whisk/scala:latest'
Wed Dec 06 07:44:54 UTC 2017: Command 'docker --host 100.100.100.91:4243 tag scala 100.100.100.91:5000/whisk/scala:latest' failed with exitCode 1, 1 retries left, retrying...
Wed Dec 06 07:44:54 UTC 2017: Executing 'docker --host 100.100.100.91:4243 tag scala 100.100.100.91:5000/whisk/scala:latest'
Wed Dec 06 07:44:54 UTC 2017: Command 'docker --host 100.100.100.91:4243 tag scala 100.100.100.91:5000/whisk/scala:latest' failed with exitCode 1, no more retries left, aborting...

FAILURE: Build completed with 2 failures.
1: Task failed with an exception.

Where:
Script '/root/openwhisk/gradle/docker.gradle' line: 96
What went wrong:
Execution failed for task ':common:scala:distDocker'.

Wed Dec 06 07:44:54 UTC 2017: Command 'docker --host 100.100.100.91:4243 build -t scala /root/openwhisk/common/scala' failed with exitCode 2

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
===================================================================
2: Task failed with an exception.

Where:
Script '/root/openwhisk/gradle/docker.gradle' line: 96
What went wrong:
Execution failed for task ':common:scala:tagImage'.

Wed Dec 06 07:44:54 UTC 2017: Command 'docker --host 100.100.100.91:4243 tag scala 100.100.100.91:5000/whisk/scala:latest' failed with exitCode 1

Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
===================================================================

Additional information you deem important:

I am on the head.. cloned recently.
common/scala/Dockerfile looks like ==
FROM buildpack-deps:trusty-curl

ENV DEBIAN_FRONTEND noninteractive
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
ENV VERSION 8
ENV UPDATE 141
ENV BUILD 15
ENV SIG 336fa29ff2bb4ef291e347e091f7f4a7
ENV JAVA_HOME /usr/lib/jvm/java-${VERSION}-oracle
ENV JRE_HOME ${JAVA_HOME}/jre
ENV PATH $JAVA_HOME/bin:$PATH
RUN curl --silent --location --retry 3 --cacert /etc/ssl/certs/GeoTrust_Global_CA.pem
--header "Cookie: oraclelicense=accept-securebackup-cookie;"
http://download.oracle.com/otn-pub/java/jdk/"${VERSION}"u"${UPDATE}"-b"${BUILD}"/"${SIG}"/jdk-"${VERSION}"u"${UPDATE}"-linux-x64.tar.gz
| tar xz -C /tmp &&
mkdir -p /usr/lib/jvm && mv /tmp/jdk1.${VERSION}.0_${UPDATE} "${JAVA_HOME}" &&
apt-get autoclean && apt-get --purge -y autoremove &&
rm -rf /tmp/* /var/tmp/*
RUN update-alternatives --install "/usr/bin/java" "java" "${JRE_HOME}/bin/java" 1 &&
update-alternatives --install "/usr/bin/javac" "javac" "${JAVA_HOME}/bin/javac" 1 &&
update-alternatives --set java "${JRE_HOME}/bin/java" &&
update-alternatives --set javac "${JAVA_HOME}/bin/javac" &&
mkdir /logs
COPY transformEnvironment.sh /
RUN chmod +x transformEnvironment.sh

Tried running same curl command on VM, as
VERSION=8
BUILD=15
UPDATE=141
-> curl --silent --location --retry 3 --cacert /etc/ssl/certs/GeoTrust_Global_CA.pem   --header "Cookie: oraclelicense=accept-securebackup-cookie;" http://download.oracle.com/otn-pub/java/jdk/"${VERSION}"u"${UPDATE}"-b"${BUILD}"/"${SIG}"/jdk-"${VERSION}"u"${UPDATE}"-linux-x64.tar.gz -O

It took time. but able to download.
Thinking same..tried to increase "dockerTimeout" in docker.gradle file if in case curl take time to download. But changing "dockerTimeout" does not coming into effect.

I had successfully deployed it 4 months back and still that setup is working fine. However, This is a new setup which I am trying to deploy.
Please suggest.

It turn out to be an issue with curl.. it error out with "curl: (35) Unsupported SSL protocol version".
By modifying common/scala/Dockerfile, able to successfully build gradle and thus wsk.
Closing the Ticket.

repo: apache/incubator-openwhisk


text:
[Feature] Schema Change can be generated by data in CDC ingestion

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation
Especially in Kafka, there may be disorder, and the Schema Change may arrive later than the data. If there is a checkpoint difference between them, it may cause data blockage.
For schema changes, they can actually be generated from data and do not necessarily rely on cdc.
Solution
No response
Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

If the data is out of order, when receive data with the new schema and then trigger the Schema Change; Next , the data with old schema arrive , may also trigger the Schema Change , so it is also problematic to change the schema through the data.
data send order is :
{"id":"1","name":"2","tm":1}
{"id":"2","name":"2","tm":2}
{"id":"3","name":"3","age":3,"tm":3}

but data receive order is :

{"id":"1","name":"2","tm":1}
{"id":"3","name":"3","age":3,"tm":3}
{"id":"2","name":"2","tm":2}

id with 2,3 is out of order , this may cause trigger the Schema Change many time

@q977734161 No, schema can only be widened, not narrowed

Has this issue been completed? If not, please assign it to me. Thanks.

authors: JingsongLi TaoZex q977734161

repo: apache/incubator-paimon