text:
[CELEBORN-560][BUG] Rerun task in spark later then RSS stageEnd cause NPE then job failed

What changes were proposed in this pull request?
Spark rerun a success task (may be caused by TaskSuccess rpc delay)

But Celeborn side have handledStageEnd and remove shuffle key from shuffleAllocatedWorker
then the rerun task throw NPE
23/04/25 11:08:44 ERROR ShuffleClientImpl: Exception raised while registering shuffle 19 with 201 mapper and 200 partitions.
com.aliyun.emr.rss.common.exception.RssException: Exception thrown in awaitResult:
   at com.aliyun.emr.rss.common.util.ThreadUtils$.awaitResult(ThreadUtils.scala:224)
   at com.aliyun.emr.rss.common.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:74)
   at com.aliyun.emr.rss.common.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:90)
   at com.aliyun.emr.rss.client.ShuffleClientImpl.registerShuffle(ShuffleClientImpl.java:329)
   at com.aliyun.emr.rss.client.ShuffleClientImpl.lambda$getPartitionLocation$2(ShuffleClientImpl.java:379)
   at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
   at com.aliyun.emr.rss.common.network.util.JavaUtils$ConcurrentHashMapForJDK8.computeIfAbsent(JavaUtils.java:427)
   at com.aliyun.emr.rss.client.ShuffleClientImpl.getPartitionLocation(ShuffleClientImpl.java:378)
   at com.aliyun.emr.rss.client.write.DataPushQueue.takePushTasks(DataPushQueue.java:97)
   at com.aliyun.emr.rss.client.write.DataPusher$1.run(DataPusher.java:125)
Caused by: java.lang.NullPointerException
   at com.aliyun.emr.rss.client.write.LifecycleManager.handleRegisterShuffle(LifecycleManager.scala:475)
   at com.aliyun.emr.rss.client.write.LifecycleManager$$anonfun$receiveAndReply$1.applyOrElse(LifecycleManager.scala:429)
   at com.aliyun.emr.rss.common.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:110)
   at com.aliyun.emr.rss.common.rpc.netty.Inbox.safelyCall(Inbox.scala:214)
   at com.aliyun.emr.rss.common.rpc.netty.Inbox.process(Inbox.scala:107)
   at com.aliyun.emr.rss.common.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:223)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)
23/04/25 11:08:44 ERROR ShuffleClientImpl: Exception raised while registering shuffle 19 with 201 mapper and 200 partitions.
com.aliyun.emr.rss.common.exception.RssException: Exception thrown in awaitResult:
   at com.aliyun.emr.rss.common.util.ThreadUtils$.awaitResult(ThreadUtils.scala:224)
   at com.aliyun.emr.rss.common.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:74)
   at com.aliyun.emr.rss.common.rpc.RpcEndpointRef.askSync(RpcEndpointRef.scala:90)
   at com.aliyun.emr.rss.client.ShuffleClientImpl.registerShuffle(ShuffleClientImpl.java:329)
   at com.aliyun.emr.rss.client.ShuffleClientImpl.lambda$getPartitionLocation$2(ShuffleClientImpl.java:379)
   at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
   at com.aliyun.emr.rss.common.network.util.JavaUtils$ConcurrentHashMapForJDK8.computeIfAbsent(JavaUtils.java:427)
   at com.aliyun.emr.rss.client.ShuffleClientImpl.getPartitionLocation(ShuffleClientImpl.java:378)
   at com.aliyun.emr.rss.client.ShuffleClientImpl.pushOrMergeData(ShuffleClientImpl.java:514)
   at com.aliyun.emr.rss.client.ShuffleClientImpl.pushData(ShuffleClientImpl.java:783)
   at com.aliyun.emr.rss.client.write.DataPusher.pushData(DataPusher.java:191)
   at com.aliyun.emr.rss.client.write.DataPusher.access$500(DataPusher.java:38)
   at com.aliyun.emr.rss.client.write.DataPusher$1.run(DataPusher.java:128)
Caused by: java.lang.NullPointerException
   at com.aliyun.emr.rss.client.write.LifecycleManager.handleRegisterShuffle(LifecycleManager.scala:475)
   at com.aliyun.emr.rss.client.write.LifecycleManager$$anonfun$receiveAndReply$1.applyOrElse(LifecycleManager.scala:429)
   at com.aliyun.emr.rss.common.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:110)
   at com.aliyun.emr.rss.common.rpc.netty.Inbox.safelyCall(Inbox.scala:214)
   at com.aliyun.emr.rss.common.rpc.netty.Inbox.process(Inbox.scala:107)
   at com.aliyun.emr.rss.common.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:223)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   at java.lang.Thread.run(Thread.java:748)

In this pr we remove the behavior of remove shuffle key from shuffleAllocatedWorkers before unregister shuffle to avoid this.
Finally this will be remove during handle removeExpiredShuffle()
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?

ping @pan3793 @RexXiong @waitinfuture

repo: apache/incubator-celeborn


text:
[CELEBORN-1327] Support Spark 3.5 with JDK21

What changes were proposed in this pull request?
Updates Celeborn to account for JDK-8303083, which affects JDK21.  (similar to the Apache Spark change here:
https://github.com/apache/spark/pull/39909)
Why are the changes needed?
Without this change, Spark users of Celeborn will encounter a runtime error similar to the following:
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.IllegalStateException: java.lang.NoSuchMethodException: java.nio.DirectByteBuffer.<init>(long, int) [in thread "Executor task launch worker for task 0.0 in stage 0.0 (TID 0)"]         at org.apache.celeborn.common.unsafe.Platform.<clinit>(Platform.java:135)         ... 16 more
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Tested using standalone Spark 3.5.1 (Hadoop 3.2) and the Celeborn main branch with JDK21 build changes in https://github.com/apache/incubator-celeborn/pull/2385.  Reproduced the runtime error above and confirmed the patch resolves it.

Thanks, merged to main for 0.5.0.

source: gharchive/pull-request

license_type: permissive

license_source: github-api


text:
[Feature][server]  SqlTask should send notifications by alert server api

sub task of https://github.com/apache/incubator-dolphinscheduler/issues/3049

+1

+1

what do you mean ? No people doing at the feature now, do you want to work for it?

OK，I can complete this task.

OK，I can complete this task.

This feature involves two issues. Can you complete both issues?
https://github.com/apache/incubator-dolphinscheduler/issues/3754 Alert module support service for other service invoke
https://github.com/apache/incubator-dolphinscheduler/issues/3749 SqlTask should send notifications by alert server api

OK，I can complete this task.

This feature involves two issues. Can you complete both issues?
#3754 Alert module support service for other service invoke
#3749 SqlTask should send notifications by alert server api

ok,I can

OK，I can complete this task.

This feature involves two issues. Can you complete both issues?

#3754 Alert module support service for other service invoke

#3749 SqlTask should send notifications by alert server api

ok,I can

You can link your design in this issue first.

Sorry, guys. I'm busy recently. I don't have time to pay attention to the community. The contribution I wanted to make has not been completed. In order to promote the plug-in module, I hope other partners will continue

source: gharchive/issue

repo: apache/incubator-dolphinscheduler


text:
[Bug][LoggerServer] task log can not refresh in time

for example ，if 1s produce log size can not teet the default conditions --Constants.DEFAULT_LOG_ROWS_NUM（64）， and need to wait a long time。

例如，如果 1s 内产生的日志量不能满足代码中默认的64条，并且后续又需要等很久才有新的日志产生。 就会导致 日志很久都无法被刷新。

例如，如果 1s 内产生的日志量不能满足代码中默认的64条，并且后续又需要等很久才有新的日志产生。 就会导致 日志很久都无法被刷新。

for example ，if 1s produce log size can not teet the default conditions --Constants.DEFAULT_LOG_ROWS_NUM（64）， and need to wait a long time。

This is where the content that has not been written to the log is uniformly disposed of after the task has completed.

for example ，if 1s produce log size can not teet the default conditions --Constants.DEFAULT_LOG_ROWS_NUM（64）， and need to wait a long time。

This is where the content that has not been written to the log is uniformly disposed of after the task has completed.

https://github.com/apache/incubator-dolphinscheduler/pull/4618

for example ，if 1s produce log size can not teet the default conditions --Constants.DEFAULT_LOG_ROWS_NUM（64）， and need to wait a long time。

This is where the content that has not been written to the log is uniformly disposed of after the task has completed.
Probably because of efficiency

so，  if this task exec need long time， log can not write in time， inconvenient problem positioning。

for example
#!/bin/bash
echo "hello world"
sleep 10m
echo "hello world2"


text:
[Typo] Optimize some code comments style for 'be/src/common/*'

Proposed changes
Issue Number: close #xxx
Problem Summary:
The current code comment style is not uniform. I want to unify the code comment style to improve code readability. Thanks.
Checklist(Required)

Does it affect the original behavior: No
Has unit tests been added: No Need
Has document been added or modified: No Need
Does it need to update dependencies: No
Are there any changes that cannot be rolled back: No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Was the code formatted using the method here?
http://doris.incubator.apache.org/zh-CN/developer-guide/cpp-format-code.html#代码风格定��% B6

Was the code formatted using the method here? http://doris.incubator.apache.org/zh-CN/developer-guide/cpp-format-code.html#代码风格定��% B6

Yes, I am using clang format, it seems that there is a lot of code that does not conform to this format specification. Thanks!

authors: morningman xlwh

repo: apache/incubator-doris


text:
Declarative configs, parse Druid options docs from code

To understand the meaning of some option in code, I sometimes need to find this option in docs and read it's description there. Because there are no docs in code.
Suggestions:

Set config prefixes declaratively, e. g.

   @ConfigPrefix("druid.segmentCache") class SegmentLoaderConfig {}

Move descriptions of options from doc pages to Javadocs of corresponding fields in config classes. We may also add custom tags like @defaultValue, @required.
In docs markdown, we write something like [[optionDescriptions io.druid.segment.loading.SegmentLoaderConfig]], it finds corresponding class in the code using DocLet or something, parses fields with @JsonProperty annotations, their javadocs, etc. and builds an html table from all of that. We may also parse default values from the code itself.

This approach could also allow to improve the doc coverage of plugins, disabled by default metric emitters, etc.

repo: apache/incubator-druid


text:
Message rejection absolute date

Description
Add an option to druid kafka indexing to select an absolute date time for message rejection.
Use case: Migrating from a kafka pull (tranquillity) indexing to a kafka push indexing.
So is needed to select at an absolute time for starting new indexing.

This PR has:

[x] been self-reviewed.
[ ] added documentation for new or modified features or behaviors.
[ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[ ] added or updated version, license, or notice information in licenses.yaml
[x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[x] added unit tests or modified existing tests to cover new code paths.
[ ] added integration tests.
[x] been tested in a test Druid cluster.

can you please describe your use case for this feature ?
also fill appropriate items in PR checklist that cover things like whether this change is tested on a cluster etc ?

can you please describe your use case for this feature ?
also fill appropriate items in PR checklist that cover things like whether this change is tested on a cluster etc ?

Done!

thanks, can you also update the necessary documentation and mention the precedence of lateMessageRejectionStartDateTime vs lateMessageRejectionPeriod or maybe there should be some check ensuring user doesn't specify both as that is most likely an user error.

thanks, can you also update the necessary documentation and mention the precedence of lateMessageRejectionStartDateTime vs lateMessageRejectionPeriod or maybe there should be some check ensuring user doesn't specify both as that is most likely an user error.

I've update documentation and mentioned the precedence

please don't rebase/force-push after the PR is under review which makes it really difficult for me, as reviewer, to see what changed since I last reviewed. thanks. ( https://github.com/apache/incubator-druid/blob/master/CONTRIBUTING.md#if-your-pull-request-shows-conflicts-with-master )

authors: denever himanshug


text:
Markpoint's information on stack bar is incorrect

Version
4.5.0-rc.2
Steps to reproduce
Open this gallery https://gallery.echartsjs.com/editor.html?c=xUmiTvvyv&v=1
Max and min for 'cheese cocoa ' are wrong
What is expected?
'Cheese cocoa' 's max is 124.1, min is 67.2
What is actually happening?
max = 67.2, min = 86.4

Also average is wrong

repo: apache/incubator-echarts

license_source: bigquery


text:
地图中如何获取各个省份的渲染颜色

One-line summary [问题简述]
地图中如何获取各个省份的渲染颜色？
Version & Environment [版本及环境]

ECharts version [ECharts 版本]:
使用的 vue-echarts 组件@3.0.7, 基于ECharts@4.0.1

Browser version [浏览器类型和版本]:

OS Version [操作系统类型和版本]:

Expected behaviour [期望结果]
实现一下效果，显示红框处图例：

ECharts option [ECharts配置项]

options = {
      visualMap: {
          show: false,
          min: 10,
          max: 100,
          inRange: {
            color: ['#D8FAFE', '#B4E3F8', '#7FDAF8', '#4EA1E9']
          },
        },
       series: [{
          type: 'map',
          mapType: 'CHINA',
          itemStyle: { areaColor: '#f5f5f5' },
          data: [
            { name: '广东省', value: 12 },
            { name: '江苏省', value: 56 }
          ],
}

Other comments [其他信息]

解决了，在 series[i].label.formatter 中通过 params.color 获取渲染颜色。
之前搞错了在外层的label中和获取了。。。
series: [{
    label: {
            formatter: (params) => {
              const topFive = this.chartData.slice(0, 5).map(item => item.name);
              if (topFive.includes(params.name)) {
                const { name, value, color } = params;
                this.topFiveLegend.push({
                  name,
                  value,
                  color
                })
              }
              return `${params.name}: ${params.value}%`;
            }
          }
}]


text:
Feature: tree 增加直角线条功能

树形图会经常用于组织机构的展现,对于组织机构的展现一般用直角的线条会比较合适,但是目前tree 的节点线条都是平滑的,所以希望能够增加直角的线条展现配置功能

@elliot-cj 没有...

👍

同求

+1

+1

同求

同求

同求

同求

是的 我也遇到这个问题，期望线条样式可以有直线转角，而不只是曲线设置曲度。翻了很多资料 貌似echarts2.0可以实现，高版本的都不行。另外就是不同层级的节点之间的间距设置也没有相关介绍。

只有2.x版本是直角

I have completed the polyline edge style which will be released in version 4.7.0. So I will close this issue if you still have any problem feel free to reopen this issue.

我已经完成了折线边缘样式，该样式将在4.7.0版中发布。因此，如果您仍然有任何问题可以随时重新打开此问题，我将关闭此问题。

你好，我在4.7版本里面的文档并没有看到这个配置项啊？


text:
[Enhancement] 'catch' branch identical to 'SocketException' branch

Search before asking

[X] I had searched in the issues and found no similar issues.

Enhancement Request

located at:
eventmesh-common/src/main/java/org/apache/eventmesh/common/utils/IPUtils.java line 115
analysis and explanation:
'catch' branch identical to 'SocketException' branch
Describe the solution you'd like
reference:

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

i want to fix it

authors: Alonexc keranbingaa

repo: apache/incubator-eventmesh


text:
[VL] Results are mismatch with vanilla Spark when use regexp_replace('a{bc', '\{', '\[')

Backend
VL (Velox)
Bug description
spark-sql> set spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConstantFolding;
spark-sql> select regexp_replace('a{bc', '\\{', '\\[');

vanilla get
a[bc

gluten get
abc

Spark version
Spark-3.2.x
Spark configurations
No response
System information
Velox System Info v0.0.2
Commit: 3a459abad5af1a8cc268c349bc3813249511768c
CMake Version: 3.22.0
System: Linux-3.10.0-862.mt20190308.130.el7.x86_64
Arch: x86_64
CPU Name: Model name:            Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz
C++ Compiler: /opt/rh/devtoolset-9/root/usr/bin/c++
C++ Compiler Version: 9.3.1
C Compiler: /opt/rh/devtoolset-9/root/usr/bin/cc
C Compiler Version: 9.3.1
CMake Prefix Path: /usr/local;/usr;/;/usr/local/cmake;/usr/local;/usr/X11R6;/usr/pkg;/opt
Relevant logs
No response

thanks @kecookier I will take a look when have time

thanks @kecookier do you will help fix the issue ?

Yes, I will work on this issue.

@kecookier I also encountered this problem. Although it's not resolved, I can provide some information.
present situation
--- Query
with tb as (
select
    name
FROM Values
    ('12234') people (name)
)
select regexp_replace(name, '22', '\\|') c1, 'gluten-spark' c2 from (select /*+ repartition(2) */ * from tb)
union
select regexp_replace(name, '22', '\\|') c1, 'vanilla-spark' c2 from tb

-- Result
c1            c2
1|34      vanilla-spark
134       gluten-spark

Gluten Velox RE2
1. code
#include <iostream>
#include <re2/re2.h>

int main() {
    std::string input = "12234";
    RE2 pattern("22");
    std::string replacement = "\\|";
    int replacements = RE2::GlobalReplace(&input, pattern, replacement);
    std::cout << "Modified string: " << input << std::endl;
    return 0;
}

2. output：
re2/re2.cc:1059: invalid rewrite pattern: \|
Modified string: 134

3. compile：
g++ -o example example.cpp -L/usr/local/lib -lre2 -Wl,-rpath,/usr/local/lib

Spark java.util.regex
1. code
    import java.util.regex.Matcher
    import java.util.regex.Pattern

    val input = "12234"
    val pattern: Pattern = Pattern.compile("22")
    val replacement = "\\|"
    val matcher: Matcher = pattern.matcher(input)
    val result = matcher.replaceAll(replacement)
    println(s"Modified string: $result")

2. output：
Modified string: 1|34

@wang-zhun Sorry for the late reply, and thank you for the information. I have a work-in-progress PR that will fix this issue.

Posting code where re2 error happens for replacement string with "\\": https://github.com/google/re2/blob/6dcd83d60f7944926bfd308cc13979fc53dd69ca/re2/re2.cc#L1053.
While java.util.regex.Matcher views characters after "\\" as literal, i.e., using what it is after "\\".
Can we just change the replacement string to let Re2 do the correct replacement?

@PHILO-HE  Apologies for the delayed response. This PR attempts to fix the mismatch problems in regexp_replace: https://github.com/facebookincubator/velox/pull/10981

https://github.com/facebookincubator/velox/pull/10981  Already merged

authors: PHILO-HE gaoyangxiaozhu kecookier wang-zhun

repo: apache/incubator-gluten


text:
feat(ci): avoid running CI if pull request is still working in progress

Reason for this PR

as #602 describe
What changes are included in this PR?

Add check that avoid running CI if PR is still working in progress
Are these changes tested?

yes
Are there any user-facing changes?

no

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 66.65%. Comparing base (aff205c) to head (4e26416).

Additional details and impacted files
@@             Coverage Diff              @@
##               main     #615      +/-   ##
============================================
+ Coverage     62.45%   66.65%   +4.19%
- Complexity        0      226     +226
============================================
  Files            60       21      -39
  Lines          8319     2012    -6307
  Branches        877      179     -698
============================================
- Hits           5196     1341    -3855
+ Misses         3123      515    -2608
- Partials          0      156     +156

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: acezen codecov-commenter

repo: apache/incubator-graphar


text:
[HUDI-810] Migrate ClientTestHarness to JUnit 5

Migrate all HoodieClientTestHarness subclasses, residing in hudi-cli, hudi-client, and hudi-utilities
Follows #1530
Migration status (after merging)

Package
JUnit 5 lib
API migration
Restructure packages

hudi-cli
✅
✅
-

hudi-client
✅
🚧
-

hudi-common
✅
🚧
🚧

hudi-hadoop-mr
✅
✅
-

hudi-hive-sync
✅
✅
-

hudi-integ-test
✅
✅
N.A.

hudi-spark
✅
✅
-

hudi-timeline-service
✅
✅
-

hudi-utilities
✅
🚧
-

Committer checklist

[x] Has a corresponding JIRA in PR title & commit

[x] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Codecov Report

Merging #1553 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #1553      +/-   ##
============================================
+ Coverage     71.66%   71.68%   +0.01%
  Complexity      294      294
============================================
  Files           378      378
  Lines         16551    16551
  Branches       1670     1670
============================================
+ Hits          11861    11864       +3
+ Misses         3959     3956       -3
  Partials        731      731

Impacted Files
Coverage Δ
Complexity Δ

...ache/hudi/common/fs/inline/InMemoryFileSystem.java
89.65% <0.00%> (+10.34%)
0.00% <0.00%> (ø%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 19cc15c...79e937d. Read the comment docs.

@xushiyan If this PR is ready for review, please let me know.

@yanghua The PR is ready for review. Thanks.

@xushiyan There are some conflicting files.

@xushiyan There are some conflicting files.

@yanghua ok fixed.

@yanghua Fixed 2 new conflicts
hudi-client/src/test/java/org/apache/hudi/execution/TestBoundedInMemoryExecutor.java
hudi-client/src/test/java/org/apache/hudi/execution/TestBoundedInMemoryQueue.java

authors: codecov-io xushiyan yanghua

repo: apache/incubator-hudi


text:
chore: keep hadoop-hdfs-client and hadoop-common version consistent

refer to #456

hi @chenlixuan , in order to check the LICENSE conveniently, when we update the dependency, we need also update this file  at the same time (https://github.com/chenlixuan/incubator-hugegraph-toolchain/blob/master/hugegraph-dist/scripts/dependency/known-dependencies.txt #L92) and make the version of dependencies in the file consistent with pom.xml

repo: apache/incubator-hugegraph-toolchain


text:
Unify code path to support both HiveCatalog and HadoopCatalog

Current code path supports db.table, and HiveCatalog.
Should be improved to use a unified path to support HadoopCatalog (mainly namespace with multiple levels) as well

Addressed by this commit. Ready to close.

repo: apache/incubator-iceberg


text:
publish all 0.10.0 images to docker hub

Is your feature request related to a problem? Please describe.
publish all 0.10.0 images to docker hub.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

All docker images are published to inlong repo with tag 0.10.0-incubating by
mvn clean package -DskipTests -Pdocker
cd docker
sh publish.sh

repo: apache/incubator-inlong


text:
kie-kogito-serverless-operator-361: Add data-index and job service st…

…artupProbes to the workflow Deployment

Description of the change:
Motivation for the change:
Checklist

[X] Add or Modify a unit test for your change
[X] Have you verified that tall the tests are passing?

How to backport a pull request to a different branch?

In order to automatically create a backporting pull request please add one or more labels having the following format backport-<branch-name>, where <branch-name> is the name of the branch where the pull request must be backported to (e.g., backport-7.67.x to backport the original PR to the 7.67.x branch).

NOTE: backporting is an action aiming to move a change (usually a commit) from a branch (usually the main one) to another one, which is generally referring to a still maintained release branch. Keeping it simple: it is about to move a specific change or a set of them from one branch to another.

Once the original pull request is successfully merged, the automated action will create one backporting pull request per each label (with the previous format) that has been added.
If something goes wrong, the author will be notified and at this point a manual backporting is needed.

NOTE: this automated backporting is triggered whenever a pull request on main branch is labeled or closed, but both conditions must be satisfied to get the new PR created.

Thank you! But I'm missing the startupProbe in the workflow container. 😅

this startupProbe is already being set here: https://github.com/apache/incubator-kie-kogito-serverless-operator/blob/7ba75e5c8573829199f4d85f2a6f5e86a1fc597e/controllers/profiles/common/object_creators.go#L142

@wmedvede can u rebase?

authors: ricardozanini wmedvede


text:
Auto comment PR for linters

Why are the changes needed?

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

Codecov Report

Merging #2038 (3bb8183) into master (ce84627) will decrease coverage by 0.08%.
The diff coverage is n/a.

:exclamation: Current head 3bb8183 differs from pull request most recent head 262eea0. Consider uploading reports for the commit 262eea0 to get more accurate results

@@             Coverage Diff              @@
##             master    #2038      +/-   ##
============================================
- Coverage     60.59%   60.50%   -0.09%
  Complexity       69       69
============================================
  Files           305      305
  Lines         14908    14908
  Branches       1928     1928
============================================
- Hits           9033     9020      -13
- Misses         5104     5115      +11
- Partials        771      773       +2

Impacted Files
Coverage Δ

...org/apache/kyuubi/operation/ExecuteStatement.scala
80.89% <0.00%> (-6.75%)
:arrow_down:

...rg/apache/kyuubi/engine/trino/TrinoStatement.scala
65.06% <0.00%> (-4.82%)
:arrow_down:

.../org/apache/kyuubi/operation/KyuubiOperation.scala
62.66% <0.00%> (-2.67%)
:arrow_down:

...n/scala/org/apache/kyuubi/KyuubiSQLException.scala
92.77% <0.00%> (-1.21%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce84627...262eea0. Read the comment docs.

It will be great to have this automations but seems a long standing bug or resteiction of github actions not allow us to do this.

authors: codecov-commenter yaooqinn

repo: apache/incubator-kyuubi


text:
[KYUUBI #2628][FOLLOWUP] Refine kyuubi-ctl batch commands

Why are the changes needed?

Refine kyuubi-ctl batch commands
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

Codecov Report

Merging #2889 (7c5fd65) into master (b987a68) will decrease coverage by 0.03%.
The diff coverage is 48.38%.

:exclamation: Current head 7c5fd65 differs from pull request most recent head 3c94fb2. Consider uploading reports for the commit 3c94fb2 to get more accurate results

@@             Coverage Diff              @@
##             master    #2889      +/-   ##
============================================
- Coverage     63.78%   63.75%   -0.04%
  Complexity      205      205
============================================
  Files           507      508       +1
  Lines         22201    22232      +31
  Branches       2931     2941      +10
============================================
+ Hits          14161    14173      +12
- Misses         6711     6725      +14
- Partials       1329     1334       +5

Impacted Files
Coverage Δ

...n/java/org/apache/kyuubi/client/util/JsonUtil.java
30.00% <0.00%> (-7.50%)
:arrow_down:

...che/kyuubi/ctl/cmd/delete/DeleteBatchCommand.scala
53.84% <25.00%> (-46.16%)
:arrow_down:

...che/kyuubi/ctl/cmd/submit/SubmitBatchCommand.scala
90.90% <25.00%> (-2.64%)
:arrow_down:

...n/scala/org/apache/kyuubi/ctl/util/BatchUtil.scala
75.00% <75.00%> (ø)

...cala/org/apache/kyuubi/ctl/RestClientFactory.scala
88.63% <100.00%> (ø)

...rg/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala
92.85% <100.00%> (+6.65%)
:arrow_up:

...ala/org/apache/kyuubi/operation/LaunchEngine.scala
86.36% <0.00%> (-3.64%)
:arrow_down:

.../org/apache/kyuubi/session/KyuubiSessionImpl.scala
73.49% <0.00%> (-2.19%)
:arrow_down:

.../kyuubi/credentials/HadoopCredentialsManager.scala
92.80% <0.00%> (-1.32%)
:arrow_down:

.../engine/spark/session/SparkSQLSessionManager.scala
79.01% <0.00%> (-1.24%)
:arrow_down:

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b987a68...3c94fb2. Read the comment docs.

cc @lightning-L  @ulysses-you

thanks, merging to master

authors: codecov-commenter turboFei ulysses-you


text:
[K8S][HELM] Use recommended labels in helm chart

Why are the changes needed?
The changes are needed to be compliant with official Kubernetes and Helm recommendations:

https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/
https://helm.sh/docs/chart_best_practices/labels/#standard-labels

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

Codecov Report

Merging #3980 (937da0e) into master (0295c0f) will decrease coverage by 0.05%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #3980      +/-   ##
============================================
- Coverage     51.95%   51.90%   -0.06%
  Complexity       13       13
============================================
  Files           522      522
  Lines         29042    29042
  Branches       3887     3887
============================================
- Hits          15089    15074      -15
- Misses        12577    12587      +10
- Partials       1376     1381       +5

Impacted Files
Coverage Δ

.../kyuubi/server/mysql/constant/MySQLErrorCode.scala
13.84% <0.00%> (-6.16%)
:arrow_down:

...ache/kyuubi/server/mysql/MySQLCommandHandler.scala
77.77% <0.00%> (-4.05%)
:arrow_down:

...ache/kyuubi/operation/KyuubiOperationManager.scala
79.45% <0.00%> (-2.74%)
:arrow_down:

...ache/kyuubi/server/mysql/MySQLGenericPackets.scala
76.59% <0.00%> (-2.13%)
:arrow_down:

...g/apache/kyuubi/operation/BatchJobSubmission.scala
75.47% <0.00%> (-1.26%)
:arrow_down:

...g/apache/kyuubi/session/KyuubiSessionManager.scala
94.02% <0.00%> (-0.75%)
:arrow_down:

...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala
80.74% <0.00%> (-0.63%)
:arrow_down:

...apache/kyuubi/engine/JpsApplicationOperation.scala
77.41% <0.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Thanks, merging to master

authors: codecov-commenter dnskr pan3793


text:
NDarray should support setitem with empty key, value

Description
Consider
import numpy as np
a = np.arange(3)
b = np.array([])
a[:len(b)] = b

import mxnet as mx
a = mx.nd.arange(3)
b = mx.nd.array([])
a[:len(b)] = b

The first half works fine, but mxnet fails on the second half:
Traceback (most recent call last):
  File "mxnet_test.py", line 10, in <module>
    a[:len(b)] = b
  File "/Users/lllausen/anaconda3/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 444, in __setitem__
    self._set_nd_basic_indexing(key, value)
  File "/Users/lllausen/anaconda3/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 735, in _set_nd_basic_indexing
    dim_size = _get_dim_size(start, stop, step)
  File "/Users/lllausen/anaconda3/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 2215, in _get_dim_size
    assert start < stop
AssertionError

Environment info (Required)
At least mxnet 1.2 and master

@reminisce

MXNet does not support empty NDArrays in nature. What is the use case of creating an empty array in MXNet?

At least mx.nd.array([]) doesn't raise an error. Also indexing an array to obtain an empty one works. My impression was that NDarray aims to model np.ndarray as close as feasible?
Regarding use-case, consider having a list of 1D ndarrays of varying length, with some being empty. Padding them to the same length (and creating a 2D array) following the code above will raise the issue. If mxnet matches the numpy behavior code can be simplified. Otherwise an extra if, else statement needs to be added to handle the special case of empty arrays.

Operations on empty arrays will probably result in undefined behavior in MXNet. So not throwing exceptions is not a good indicator showing that it works.
MXNet NDArray is not designed to cover empty array use case as in numpy, because complete shape information (both inputs and outputs of operators) must be obtained in shape inference pass before moving to computation in either imperative or symbolic mode.

In that case, would it be better to be explicit about the lack of support for empty arrays and make sure they cannot be instantiated (i.e. raise an Exception)?

I don't know whether forbidding mx.nd.array([]) would lead to any backward compatibility concern, but it should fail when one of either inputs or outputs of an operator is shape-incomplete. I will check to see why slice does not fail when the output is an empty NDArray. Thanks.

authors: leezu reminisce szha

repo: apache/incubator-mxnet


text:
Many examples of operators in NDArray API are not executable

While there are many examples of NDArray API that are executable (meaning they can be copied to the clipboard and run in shell directly
There are quite a few others that aren't.
In order to:
a. Maintain consistency in Documentation on the website
b. Facilitate usability, ease of understanding and reproducibility
we should ensure all examples of the operators are executable (copy-pasted into the shell (Python shell in case of Python-specific API bindings) and executed)

@mxnet-label-bot add [Doc]


text:
[Flaky] test_numpy_op.test_np_einsum

Description
======================================================================

FAIL: test_numpy_op.test_np_einsum

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/work/mxnet/tests/python/unittest/common.py", line 177, in test_new

    orig_test(*args, **kwargs)

  File "/work/mxnet/python/mxnet/util.py", line 315, in _with_np_shape

    return func(*args, **kwargs)

  File "/work/mxnet/python/mxnet/util.py", line 499, in _with_np_array

    return func(*args, **kwargs)

  File "/work/mxnet/tests/python/unittest/test_numpy_op.py", line 3524, in test_np_einsum

    assert_almost_equal(out_mx.asnumpy(), expected_np, rtol=rtol, atol=atol)

  File "/work/mxnet/python/mxnet/test_utils.py", line 627, in assert_almost_equal

    raise AssertionError(msg)

AssertionError:

Items are not equal:

Error 1.669922 exceeds tolerance rtol=1.000000e+00, atol=1.000000e-01 (mismatch 25.000000%).

Location of maximum error: (0, 1), a=0.24609375, b=0.02963257

 ACTUAL: array([[156.    ,   0.2461],

       [433.2   , -30.8   ]], dtype=float16)

 DESIRED: array([[156.     ,   0.02963],

       [433.2    , -30.72   ]], dtype=float16)

-------------------- >> begin captured stdout << ---------------------

*** Maximum errors for vector of size 4:  rtol=1.0, atol=0.1

  1: Error 1.669922  Location of error: (0, 1), a=0.24609375, b=0.02963257

--------------------- >> end captured stdout << ----------------------

-------------------- >> begin captured logging << --------------------

root: INFO: NumPy-shape semantics has been activated in your code. This is required for creating and manipulating scalar and zero-size tensors, which were not supported in MXNet before, as in the official NumPy library. Please DO NOT manually deactivate this semantics while using `mxnet.numpy` and `mxnet.numpy_extension` modules.

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1120100586 to reproduce.

--------------------- >> end captured logging << ---------------------

Occurrences
Unrelated PR - #16599
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-16599/5/pipeline
What have you tried to solve it?
Nothing

Introduced in #15911

Also happened here: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-gpu/detail/PR-16639/1/pipeline
and here: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-16639/1/pipeline

Temporarily disable the flaky tests in #16643 .

@mxnet-label-bot update [Bug, Numpy]

It's still flaky in #16555: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-16555/18/pipeline

FAIL: test_numpy_op.test_np_einsum

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/work/mxnet/tests/python/unittest/common.py", line 177, in test_new

    orig_test(*args, **kwargs)

  File "/work/mxnet/python/mxnet/util.py", line 315, in _with_np_shape

    return func(*args, **kwargs)

  File "/work/mxnet/python/mxnet/util.py", line 499, in _with_np_array

    return func(*args, **kwargs)

  File "/work/mxnet/tests/python/unittest/test_numpy_op.py", line 3585, in test_np_einsum

    assert_almost_equal(grad[0][iop], grad[1][iop], rtol=rtol, atol=atol)

  File "/work/mxnet/python/mxnet/test_utils.py", line 627, in assert_almost_equal

    raise AssertionError(msg)

AssertionError:

Items are not equal:

Error 1.018927 exceeds tolerance rtol=1.000000e-03, atol=1.000000e-05 (mismatch 20.000000%).

Location of maximum error: (0, 1), a=-0.03388485, b=-0.03384018

 ACTUAL: array([[  97.04612   ,   -0.03388485, -148.55832   ,  -86.593506  ,

         -22.488245  ],

       [  97.04612   ,   -0.03388485, -148.55832   ,  -86.593506  ,...

 DESIRED: array([[  97.04615   ,   -0.03384018, -148.5583    ,  -86.5935    ,

         -22.48825   ],

       [  97.04615   ,   -0.03384018, -148.5583    ,  -86.5935    ,...

-------------------- >> begin captured stdout << ---------------------

*** Maximum errors for vector of size 25:  rtol=0.001, atol=1e-05

  1: Error 1.018927  Location of error: (0, 1), a=-0.03388485, b=-0.03384018

  2: Error 1.018927  Location of error: (1, 1), a=-0.03388485, b=-0.03384018

  3: Error 1.018927  Location of error: (2, 1), a=-0.03388485, b=-0.03384018

  4: Error 1.018927  Location of error: (3, 1), a=-0.03388485, b=-0.03384018

  5: Error 1.018927  Location of error: (4, 1), a=-0.03388485, b=-0.03384018

authors: ChaiBapchya TaoLv hzfan nickguletskii


text:
oneDNN 2 missing headerfiles

https://github.com/apache/incubator-mxnet/pull/19667 breaks Horovod (cf https://github.com/horovod/horovod/pull/2530) as some header files are missing in the pip wheel:/usr/local/lib/python3.6/dist-packages/mxnet/include/mkldnn/oneapi/dnnl/dnnl.hpp:23:10: fatal error: oneapi/dnnl/dnnl_config.h: No such file or directory
cc @bartekkuncer

Related code on MXNet side is at https://github.com/apache/incubator-mxnet/blob/91503f71e17fca9151779503fc9f5edefe26f2ef/tools/pip/setup.py#L148-L150

I've met this error too. @bartekkuncer  Is it possible to add the header to pip wheel? Also ping @szha

In terms of the wheel, I think the last wheel that works is 20201214.

@leezu I believe that changing L149 to: shutil.copytree(os.path.join(CURRENT_DIR, 'mxnet-build/3rdparty/mkldnn/include/oneapi/dnnl'), fixes the issue but as I am not familiar with horovod I do not know how to check if the fix works. Can you provide me with a way to reproduce the issue?

@bartekkuncer For me, I met this error when trying to install horovod (you may change the cuda version):

python3 -m pip install -U --pre "mxnet-cu102==2.0.0b20201217" -f https://dist.mxnet.io/python
HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_WITHOUT_GLOO=1 HOROVOD_WITH_MPI=1 HOROVOD_WITH_MXNET=1 HOROVOD_WITHOUT_PYTORCH=1 HOROVOD_WITHOUT_TENSORFLOW=1 python3 -m pip install --no-cache-dir horovod

@bartekkuncer I recommend the following for verification:

Download a wheel with the correct headers (e.g. the 20201214 one that @sxjscience mentioend) and one without (the ones after 1214)
Unzip them and examine the header content and see where the headers should be.
Correct the toolts/pip/setup.py and use the wheel build scripts to build correct wheel.
Verify if the headers are in the right location in the resulting wheel.

Thanks @bartekkuncer! I opened https://github.com/apache/incubator-mxnet/pull/19694 as weekend has started in your timezone

Horovod now fails with
    In file included from /tmp/pip-req-build-bhade3mm/horovod/mxnet/mpi_ops.h:24:0,
                     from /tmp/pip-req-build-bhade3mm/horovod/mxnet/mpi_ops.cc:21:
    /usr/local/lib/python3.6/dist-packages/mxnet/include/mxnet/ndarray.h:41:10: fatal error: mkldnn.hpp: No such file or directory
     #include <mkldnn.hpp>
              ^~~~~~~~~~~~
    compilation terminated.
    horovod/mxnet/CMakeFiles/mxnet.dir/build.make:758: recipe for target 'horovod/mxnet/CMakeFiles/mxnet.dir/mpi_ops.cc.o' failed

Horovod now fails with
�[0m�[91m    In file included from /tmp/pip-req-build-bhade3mm/horovod/mxnet/mpi_ops.h:24:0,
                     from /tmp/pip-req-build-bhade3mm/horovod/mxnet/mpi_ops.cc:21:
    /usr/local/lib/python3.6/dist-packages/mxnet/include/mxnet/ndarray.h:41:10: fatal error: mkldnn.hpp: No such file or directory
�[0m�[91m     #include <mkldnn.hpp>
�[0m�[91m              ^~~~~~~~~~~~
�[0m�[91m    compilation terminated.
�[0m�[91m    horovod/mxnet/CMakeFiles/mxnet.dir/build.make:758: recipe for target 'horovod/mxnet/CMakeFiles/mxnet.dir/mpi_ops.cc.o' failed

Yes, I saw that, working on the fix.

Thanks @bartekkuncer!

@leezu I think this PR https://github.com/apache/incubator-mxnet/pull/19706 should fix the problem.

@bartekkuncer looks like the dnnl_config.h is included in the wrong directory. It should be in oneapi/dnnl/dnnl_config.h at least horovod build fails with usr/local/lib/python3.6/dist-packages/mxnet/include/mkldnn/oneapi/dnnl/dnnl.hpp:23:10: fatal error: oneapi/dnnl/dnnl_config.h: No such file or directory  #include "oneapi/dnnl/dnnl_config.h")

The files included in are
mxnet/include/mkldnn
mxnet/include/mkldnn/mkldnn_version.h
mxnet/include/mkldnn/dnnl_debug.h
mxnet/include/mkldnn/mkldnn_debug.h
mxnet/include/mkldnn/dnnl_ocl.h
mxnet/include/mkldnn/dnnl_sycl.h
mxnet/include/mkldnn/dnnl_ocl.hpp
mxnet/include/mkldnn/mkldnn_types.h
mxnet/include/mkldnn/dnnl_version.h
mxnet/include/mkldnn/oneapi
mxnet/include/mkldnn/oneapi/dnnl
mxnet/include/mkldnn/oneapi/dnnl/dnnl_debug.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_ocl.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_sycl.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_ocl.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_types.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_sycl_types.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_threadpool_iface.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_threadpool.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_sycl.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_threadpool.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl.h
mxnet/include/mkldnn/mkldnn_config.h
mxnet/include/mkldnn/dnnl_types.h
mxnet/include/mkldnn/dnnl.hpp
mxnet/include/mkldnn/dnnl_config.h
mxnet/include/mkldnn/mkldnn.hpp
mxnet/include/mkldnn/mkldnn_dnnl_mangling.h
mxnet/include/mkldnn/dnnl_sycl_types.h
mxnet/include/mkldnn/dnnl_threadpool_iface.hpp
mxnet/include/mkldnn/dnnl_threadpool.hpp
mxnet/include/mkldnn/dnnl_sycl.hpp
mxnet/include/mkldnn/dnnl_threadpool.h
mxnet/include/mkldnn/mkldnn.h
mxnet/include/mkldnn/dnnl.h

I think we may need to update https://github.com/apache/incubator-mxnet/blob/3c5beb3596b6bc01f77bc7ddd14ed90221c31950/cd/mxnet_lib/static/Jenkins_pipeline.groovy#L36 to ensure that the config files are stashed correctly on the CD

we might also consider making it robust in setup.py by asserting the existence of these header files instead of only include when available.

The files included in are
mxnet/include/mkldnn
mxnet/include/mkldnn/mkldnn_version.h
mxnet/include/mkldnn/dnnl_debug.h
mxnet/include/mkldnn/mkldnn_debug.h
mxnet/include/mkldnn/dnnl_ocl.h
mxnet/include/mkldnn/dnnl_sycl.h
mxnet/include/mkldnn/dnnl_ocl.hpp
mxnet/include/mkldnn/mkldnn_types.h
mxnet/include/mkldnn/dnnl_version.h
mxnet/include/mkldnn/oneapi
mxnet/include/mkldnn/oneapi/dnnl
mxnet/include/mkldnn/oneapi/dnnl/dnnl_debug.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_ocl.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_sycl.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_ocl.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_types.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_sycl_types.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_threadpool_iface.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_threadpool.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_sycl.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_threadpool.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl.h
mxnet/include/mkldnn/mkldnn_config.h
mxnet/include/mkldnn/dnnl_types.h
mxnet/include/mkldnn/dnnl.hpp
mxnet/include/mkldnn/dnnl_config.h
mxnet/include/mkldnn/mkldnn.hpp
mxnet/include/mkldnn/mkldnn_dnnl_mangling.h
mxnet/include/mkldnn/dnnl_sycl_types.h
mxnet/include/mkldnn/dnnl_threadpool_iface.hpp
mxnet/include/mkldnn/dnnl_threadpool.hpp
mxnet/include/mkldnn/dnnl_sycl.hpp
mxnet/include/mkldnn/dnnl_threadpool.h
mxnet/include/mkldnn/mkldnn.h
mxnet/include/mkldnn/dnnl.h

I think we may need to update
https://github.com/apache/incubator-mxnet/blob/3c5beb3596b6bc01f77bc7ddd14ed90221c31950/cd/mxnet_lib/static/Jenkins_pipeline.groovy#L36
to ensure that the config files are stashed correctly on the CD

Thanks @leezu . I fixed it in CI but must have overlooked it in CD.

The files included in are
mxnet/include/mkldnn
mxnet/include/mkldnn/mkldnn_version.h
mxnet/include/mkldnn/dnnl_debug.h
mxnet/include/mkldnn/mkldnn_debug.h
mxnet/include/mkldnn/dnnl_ocl.h
mxnet/include/mkldnn/dnnl_sycl.h
mxnet/include/mkldnn/dnnl_ocl.hpp
mxnet/include/mkldnn/mkldnn_types.h
mxnet/include/mkldnn/dnnl_version.h
mxnet/include/mkldnn/oneapi
mxnet/include/mkldnn/oneapi/dnnl
mxnet/include/mkldnn/oneapi/dnnl/dnnl_debug.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_ocl.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_sycl.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_ocl.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_types.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_sycl_types.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_threadpool_iface.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_threadpool.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_sycl.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_threadpool.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl.h
mxnet/include/mkldnn/mkldnn_config.h
mxnet/include/mkldnn/dnnl_types.h
mxnet/include/mkldnn/dnnl.hpp
mxnet/include/mkldnn/dnnl_config.h
mxnet/include/mkldnn/mkldnn.hpp
mxnet/include/mkldnn/mkldnn_dnnl_mangling.h
mxnet/include/mkldnn/dnnl_sycl_types.h
mxnet/include/mkldnn/dnnl_threadpool_iface.hpp
mxnet/include/mkldnn/dnnl_threadpool.hpp
mxnet/include/mkldnn/dnnl_sycl.hpp
mxnet/include/mkldnn/dnnl_threadpool.h
mxnet/include/mkldnn/mkldnn.h
mxnet/include/mkldnn/dnnl.h

I think we may need to update
https://github.com/apache/incubator-mxnet/blob/3c5beb3596b6bc01f77bc7ddd14ed90221c31950/cd/mxnet_lib/static/Jenkins_pipeline.groovy#L36
to ensure that the config files are stashed correctly on the CD

Thanks @leezu . I fixed it in CI but must have overlooked it in CD.

It looks like there are still more issues with the CD. Horovod still fails with
    /usr/local/lib/python3.6/dist-packages/mxnet/include/mkldnn/oneapi/dnnl/dnnl.hpp:23:10: fatal error: oneapi/dnnl/dnnl_config.h: No such file or directory
     #include "oneapi/dnnl/dnnl_config.h"
              ^~~~~~~~~~~~~~~~~~~~~~~~~~~
    compilation terminated.
    horovod/mxnet/CMakeFiles/mxnet.dir/build.make:758: recipe for target 'horovod/mxnet/CMakeFiles/mxnet.dir/mpi_ops.cc.o' failed

It looks like there are still more issues with the CD. Horovod still fails with
    /usr/local/lib/python3.6/dist-packages/mxnet/include/mkldnn/oneapi/dnnl/dnnl.hpp:23:10: fatal error: oneapi/dnnl/dnnl_config.h: No such file or directory
     #include "oneapi/dnnl/dnnl_config.h"
              ^~~~~~~~~~~~~~~~~~~~~~~~~~~
    compilation terminated.
    horovod/mxnet/CMakeFiles/mxnet.dir/build.make:758: recipe for target 'horovod/mxnet/CMakeFiles/mxnet.dir/mpi_ops.cc.o' failed

It looks like there are still more issues with the CD. Horovod still fails with
�[0m�[91m    /usr/local/lib/python3.6/dist-packages/mxnet/include/mkldnn/oneapi/dnnl/dnnl.hpp:23:10: fatal error: oneapi/dnnl/dnnl_config.h: No such file or directory
�[0m�[91m     #include "oneapi/dnnl/dnnl_config.h"
�[0m�[91m              ^~~~~~~~~~~~~~~~~~~~~~~~~~~
�[0m�[91m    compilation terminated.
�[0m�[91m    horovod/mxnet/CMakeFiles/mxnet.dir/build.make:758: recipe for target 'horovod/mxnet/CMakeFiles/mxnet.dir/mpi_ops.cc.o' failed

Yes, I saw that. https://github.com/apache/incubator-mxnet/pull/19726 should fix the issue.

It looks like there are still more issues with the CD. Horovod still fails with
�[0m�[91m    /usr/local/lib/python3.6/dist-packages/mxnet/include/mkldnn/oneapi/dnnl/dnnl.hpp:23:10: fatal error: oneapi/dnnl/dnnl_config.h: No such file or directory
�[0m�[91m     #include "oneapi/dnnl/dnnl_config.h"
�[0m�[91m              ^~~~~~~~~~~~~~~~~~~~~~~~~~~
�[0m�[91m    compilation terminated.
�[0m�[91m    horovod/mxnet/CMakeFiles/mxnet.dir/build.make:758: recipe for target 'horovod/mxnet/CMakeFiles/mxnet.dir/mpi_ops.cc.o' failed

Yes, I saw that. https://github.com/apache/incubator-mxnet/pull/19726 should fix the issue.

Thank you @bartekkuncer!

Thank you @bartekkuncer!

authors: bartekkuncer leezu sxjscience szha


text:
Restore quantized RNN to master

Description
Restoring qunatized RNN: https://github.com/apache/incubator-mxnet/pull/20759

@mxnet-bot run ci [centos-cpu, unix-cpu, website, windows-gpu]

@mxnet-bot run ci [unix-cpu, windows-gpu, clang]

@mxnet-bot run ci [unix-cpu]

@mxnet-bot run ci [unix-cpu]

@mxnet-bot run ci [windows-gpu]

@mxnet-bot run ci [unix-cpu]

@mxnet-bot run ci [windows-gpu]

@mxnet-bot run ci [centos-cpu, unix-cpu]

@mxnet-bot run ci [centos-cpu, unix-cpu]

@mxnet-bot run ci [unix-cpu]

@mxnet-bot run ci [unix-cpu]

@mxnet-bot run ci [unix-cpu]


text:
NETBEANS-1663: Better support for JUnit 5 ParameterizedTest

See https://issues.apache.org/jira/browse/NETBEANS-1663.
I bumped the Java version for junit.ui to 8, let me know if this isn't okay.
If anyone has suggestions for how to test this automatically, please let me know. I tested this by hand on a project with @ParameterizedTest tests, as well as a mix of JUnit 5 and 4 regular @Test tests.

Looks okay to me, seems I missed looking at parameterized tests when implementing NETBEANS-6 thanks @srdo for spotting and providing a fix

authors: mcdonnell-john srdo

repo: apache/incubator-netbeans


text:
Mpfs switch to exception common

Summary
Switch mpfs target to use riscv_exception_common to reduce code copies
Also fix riscv_exception_common to support more than 2 CPUs in SMP mode, and allow defining multiple IRQ stacs even when not using SMP.
This is needed by mpfs bootloader build, where nuttx is being booted only on one CPU, but the startup code still handles SW IRQs from all harts.

Did the modifications suggested by @pkarashchenko

repo: apache/incubator-nuttx


text:
driver/sensor: rename /dev/sensor -> /dev/uorb

Summary
driver/sensor: rename /dev/sensor -> /dev/uorb
Because this directory contains all topics, not just sensors.
This PR is related to https://github.com/apache/incubator-nuttx-apps/pull/1257
Signed-off-by: Jiuzhu Dong dongjiuzhu1@xiaomi.com
Impact
rename  directory for sensor device
Testing
Local test

@xiaoxiang781216 is this a breaking change? I mean from out of tree users.

Yes, if they don't use uorb wrapper here:
https://github.com/apache/incubator-nuttx-apps/tree/master/system/uorb


text:
Revert "tools/ci: patch for gcc-arm-none-eabi to 11.3.Rel1 August 08, 2022"

Summary
Revert "tools/ci: patch for gcc-arm-none-eabi to 11.3.Rel1 August 08, 2022"
revert this change since docker image upload always fails
This reverts commit 0f3f5d091ad510f7b5f4556c8d1805b073279e51.
Signed-off-by: chao an anchao@xiaomi.com
Impact
N/A
Testing
ci check

let's merge the change directly to unlock ci.


text:
boards: Remove the duplicated prototype of CONFIG_INIT_ENTRYPOINT

Summary
Remove the duplication
Impact
code refactor only
Testing
Pass CI

Error: nor_main.c:78:5: error: conflicting types for 'nor_main'
   78 | int nor_main(int argc, char *argv)
      |     ^~~~~~~~
In file included from nor_main.c:25:
/github/workspace/sources/nuttx/include/sys/types.h:318:5: note: previous declaration of 'nor_main' was here
  318 | int CONFIG_INIT_ENTRYPOINT(int argc, FAR char *argv[]);

Error: nor_main.c:78:5: error: conflicting types for 'nor_main'
   78 | int nor_main(int argc, char *argv)
      |     ^~~~~~~~
In file included from nor_main.c:25:
/github/workspace/sources/nuttx/include/sys/types.h:318:5: note: previous declaration of 'nor_main' was here
  318 | int CONFIG_INIT_ENTRYPOINT(int argc, FAR char *argv[]);

Fixed.

Same with dram_main.c:76:5: error: conflicting types for 'dram_main'

Fix all.


text:
arch: cxd56xx: Fix stall bulk xfer when sending 512 byte data

Summary
Remove hardware zero length packet enhancement because of driver logic already processed the ZLP correctly. It is unnecessary and cause of IN interrupt lost.
Impact
only spresense
Testing

Merging it directly since CI is still not fully recovered

authors: SPRESENSE pkarashchenko


text:
feat(service/azfile): add azure file service support

part of: https://github.com/apache/incubator-opendal/issues/3047

I have a Azure File serivce account and will add integration for next PR.

I have a Azure File serivce account and will add integration test for next PR.

You can add the account to 1Password and share it with me. I will then copy it to our services vault.

repo: apache/incubator-opendal


text:
Add unit tests to support "in-line" code on "function" keyword

Need to assure we test for support of in-line code, for example:
function: |
          function main(params) {
            console.log(params);
            return params || {};
          }

we need to have unit tests in place for multiple langs. (e.g., python, java, swift, and php at the very least)

need to assure code "errors out: if typo and exists gracefully with an error message to user

repo: apache/incubator-openwhisk-wskdeploy


text:
Feed should be able to store state info

From reporter:
The github/webhook action creates a webhook if params.lifecycleEvent == CREATE. The response from Github is the ID of the webhook created there as well as the url pointing to it. Currently, there is no established way to store such information in order to be able to delete it later. A solution could be that we store the url in annotations but I'm not sure that is the right way to go forward.
From @rabbah :
I think it may be relevant here. a feed creation could return values to store in the trigger so that on the next life cycle event, those values are sent as parameters with the request. This might solve the problem here if I understood it. If the values contain sensitive information, then they can be encrypted by the feed so they are treated as opaque values. A further refinement on this is to allow the feed to specify information it wants to receive back on the next life cycle event but also parameters to add to the trigger firing.

Rethinking this: the feed has the trigger name and the API key. It can decorate the trigger any way it pleases. @csantanapr if you agree is this an issue at all then?

authors: csantanapr rabbah

repo: apache/incubator-openwhisk


text:
mqtt-streaming: fix flaky 'unsubscribe a client session' test

Previously, in this test, the server sent a 'publish' concurrent to the client sending the 'ack' for that publish. This fix makes sure the 'ack' is sent only after observing the 'publish'.
Hopefully fixes #458
This PR still fails due to #454 but #460 suggests it should be OK when combined with #456

@raboof Feel free to merge the PR when you want

authors: mdedetrich raboof

repo: apache/incubator-pekko-connectors


text:
acknowledge licensing issues in AbstractBoundedNodeQueue

https://github.com/apache/incubator-pekko/blob/532b7173c175c0c379074cb022e98f5002860c22/actor/src/main/java/org/apache/pekko/dispatch/AbstractBoundedNodeQueue.java#L18-L27

This seems reasonable to me

authors: mdedetrich pjfanning

repo: apache/incubator-pekko


text:
support creating ForkJoinPools with maximumPoolSize

ForkJoinPool has extra contructors in JDK9+ that allow maximumPoolSize to be set.
Relates to https://github.com/apache/incubator-pekko/discussions/482
Pekko still supports JDK8 so implementing this is not straightforward.
Options include:

adding a new dispatcher executor type of fork-join-executor-jdk9 which is similar to fork-join-executor but supports the extra maximumPoolSize config (the class for this could be added to jdk9+ build (we support having a scala-jdk9 source directory)
have fork-join-executor support the maximumPoolSize but ignore it or fail if Java 8 runtime is used. We might need to use Java reflection to do this.

I think it makes sense to solve this when we get multi-jar-release working so we can target jdk 9+ in a principled fashion. For this reason I think removing it from the 1.1.0-M2 milestone makes sense as I seriously doubt that we will get multi-jar-release working by then, @He-Pin @pjfanning do you agree?

IIRC, the multi-jar-release has bad IDE support, and @Glavo once told me about that too, for me the current implementation seems good, so does netty work this way, another library like reactor-core is using the multi-jar-release mechanism.

I don't believe in dropping useful features because one implementation that can't be done is preferred over one that is ready to go.

@mdedetrich Netty's PlatformDependent0 is doing the same, I don't think multi-jar-release should be a blocker for this, What about android? multi-jar-release can not ensure the code will work on Android too.

multi-jar-release can not ensure the code will work on Android too.

Android supports JDK 1.8+ which means it will ignore java 9 compiled classes in the multi-release-jar (its the java-9 compiled classes which will be a non standard location).
In other words it will work fine

I don't think multi-jar-release should be a blocker for this

I am not saying that its a blocker, just that multi-jar-release may already be done before the feature is complete

Some methods are missing in Android, AFAIK, we will see.

@mdedetrich It will be contagious when user packaging shadow jars, @glavo just told me。

@mdedetrich It will be contagious when user packaging shadow jars, @Glavo just told me。

I don't think this is a good reason not to use MR jars. Good shading tools support MR jars.

We already depend on jackson-core which is an MR jar.

@mdedetrich It will be contagious when user packaging shadow jars, @Glavo just told me。

I don't think this is a good reason not to use MR jars. Good shading tools support MR jars.

Agreed, I don't know what the aversion to multi release jar's comes from but its well support and a lot of critical Java OS libraries use it.
The issues currently with multi release jar's are more sbt specific and this is largely a historical artifact (which is that any new functionality in later JDK's typically had their own equivalent idiomatic Scala specific solution so there wasn't much incentive to support multi-release-jars in sbt as there was very little demand for it).

@He-Pin I'm open to pressing to get this into 1.1. My view is that user need outweighs implementation niceties. Do you need this change?

@pjfanning Yes， I'm using Pekko in several of our mission-critical systems, so I would like to keep my systems safe. If we like, we can re-implement it later with multi-jar-release.

authors: He-Pin mdedetrich pjfanning


text:
[PINOT-7328] Reduce lock contention in physical planning phase by reducing the total number of tasks

Performance tests show that for requests that require little server work (docs/entries scanned <100k), the latency increases quickly with QPS while CPU is hardly used. In one example, with a query that took ~2ms to execute, beyond 600QPS the latency shot up to 200+ms while CPU utilization was <20%.
Profiling/thread dumps showed that majority of the time was spent in taking tasks off the task queue in the physical planning phase - queries were essentially serialized on the LinkedBlockingQueue's lock.
This change bounds the total number of tasks that can be created/put on the queue, thus reducing contention and doing more work per task.
With this change in place, we were able to push the example query to 2000+ qps.
Testing done:

Added unit tests to cover functionality.
Perf tests as noted above.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@b4496b6). Click here to learn what that means.
The diff coverage is 90.9%.

@@            Coverage Diff            @@
##             master    #3470   +/-   ##
=========================================
  Coverage          ?   70.48%
  Complexity        ?        4
=========================================
  Files             ?      999
  Lines             ?    45681
  Branches          ?     6111
=========================================
  Hits              ?    32198
  Misses            ?    11370
  Partials          ?     2113

Impacted Files
Coverage Δ
Complexity Δ

.../com/linkedin/pinot/core/plan/CombinePlanNode.java
68.75% <90.9%> (ø)
0 <0> (?)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b4496b6...c921470. Read the comment docs.

authors: codecov-io sunithabeeram

repo: apache/incubator-pinot


text:
Adding a version option to pinot admin to show all the component versions

Description
Adding a new option: -v into pinot-admin script to show all the component versions.
Sample Output:
➜ bin/pinot-admin.sh -v
List All Pinot Component Versions:
Package: pinot-protobuf, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-kafka-2.0, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-avro, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-distribution, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-csv, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-s3, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-batch-ingestion-standalone, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-confluent-avro, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-thrift, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-orc, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-batch-ingestion-spark, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-gcs, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-hdfs, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-batch-ingestion-hadoop, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-adls, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-json, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01
Package: pinot-parquet, Version: 0.7.0-SNAPSHOT-23f7770ffc795b917c83efd3e36fcb369177bf01

Codecov Report

Merging #6380 (23f7770) into master (1beaab5) will decrease coverage by 21.39%.
The diff coverage is 38.60%.

@@             Coverage Diff             @@
##           master    #6380       +/-   ##
===========================================
- Coverage   66.44%   45.05%   -21.40%
===========================================
  Files        1075     1298      +223
  Lines       54773    62767     +7994
  Branches     8168     9112      +944
===========================================
- Hits        36396    28281     -8115
- Misses      15700    32148    +16448
+ Partials     2677     2338      -339

Flag
Coverage Δ

integration
45.05% <38.60%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...ot/broker/broker/AllowAllAccessControlFactory.java
100.00% <ø> (ø)

.../helix/BrokerUserDefinedMessageHandlerFactory.java
52.83% <0.00%> (-13.84%)
:arrow_down:

...org/apache/pinot/broker/queryquota/HitCounter.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...che/pinot/broker/queryquota/MaxHitRateTracker.java
0.00% <0.00%> (ø)

...ache/pinot/broker/queryquota/QueryQuotaEntity.java
0.00% <0.00%> (-50.00%)
:arrow_down:

...ker/routing/instanceselector/InstanceSelector.java
100.00% <ø> (ø)

...ceselector/StrictReplicaGroupInstanceSelector.java
0.00% <0.00%> (ø)

...roker/routing/segmentpruner/TimeSegmentPruner.java
0.00% <0.00%> (ø)

...roker/routing/segmentpruner/interval/Interval.java
0.00% <0.00%> (ø)

...r/routing/segmentpruner/interval/IntervalTree.java
0.00% <0.00%> (ø)

... and 1295 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 33de6dc...23f7770. Read the comment docs.


text:
Compatibility test for segment operations upload and delete

Description
Implemented compatibility test for Segment UPLOAD and DELETE operations.

UPLOAD: Operation requires the below input and performs segment generation, compress to tar.gz file and then upload to controller. Also, does the validation check for segment uploaded to the controller and whether it is in ONLINE state.

inputDataFileName
schemaFileName
tableConfigFileName
recordReaderConfigFileName
segmentName

DELETE: Operation requires the below input and deletes segment. Also, does the validation check for the segment deletion.

tableConfigFileName
segmentName

Note:

Segment UPLOAD and DELETE operation required TableOp to complete creating Schema and Table before calling SegmentOp.

Test Case:

UPLOAD - Tested Segment Generation, Compression to tar.gz file, Upload segment, validate upload segment successful, and cleanup of temp directories.
DELETE - Tested Segment deletion and validation to check segment deleted is successful.

Issue #4854
Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

[ ] Yes (Please label as backward-incompat, and complete the section below on Release Notes)

Does this PR fix a zero-downtime upgrade introduced earlier?

[ ] Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

Does this PR otherwise need attention when creating release notes? Things to consider:

New configuration options
Deprecation of configurations
Signature changes to public methods/interfaces
New plugins added or old plugins removed

[ ] Yes (Please label this PR as release-notes and complete the section on Release Notes)

Release Notes
If you have tagged this as either backward-incompat or release-notes,
you MUST add text here that you would like to see appear in release notes of the
next release.
If you have a series of commits adding or enabling a feature, then
add this section only in final commit that marks the feature completed.
Refer to earlier release notes to see examples of text
Documentation
If you have introduced a new feature or configuration, please add it to the documentation as well.
See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document

Codecov Report

Merging #6382 (5eca9cc) into master (1beaab5) will decrease coverage by 1.23%.
The diff coverage is 56.80%.

@@            Coverage Diff             @@
##           master    #6382      +/-   ##
==========================================
- Coverage   66.44%   65.21%   -1.24%
==========================================
  Files        1075     1298     +223
  Lines       54773    62769    +7996
  Branches     8168     9112     +944
==========================================
+ Hits        36396    40935    +4539
- Misses      15700    18912    +3212
- Partials     2677     2922     +245

Flag
Coverage Δ

unittests
65.21% <56.80%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...e/pinot/broker/api/resources/PinotBrokerDebug.java
0.00% <0.00%> (-79.32%)
:arrow_down:

...ot/broker/broker/AllowAllAccessControlFactory.java
71.42% <ø> (-28.58%)
:arrow_down:

.../helix/BrokerUserDefinedMessageHandlerFactory.java
33.96% <0.00%> (-32.71%)
:arrow_down:

...ker/routing/instanceselector/InstanceSelector.java
100.00% <ø> (ø)

...ava/org/apache/pinot/client/AbstractResultSet.java
66.66% <0.00%> (+9.52%)
:arrow_up:

.../main/java/org/apache/pinot/client/Connection.java
35.55% <0.00%> (-13.29%)
:arrow_down:

...inot/client/JsonAsyncHttpPinotClientTransport.java
10.90% <0.00%> (-51.10%)
:arrow_down:

...not/common/assignment/InstancePartitionsUtils.java
73.80% <ø> (+0.63%)
:arrow_up:

...common/config/tuner/NoOpTableTableConfigTuner.java
100.00% <ø> (ø)

...ot/common/config/tuner/RealTimeAutoIndexTuner.java
100.00% <ø> (ø)

... and 1151 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b8bc74f...5eca9cc. Read the comment docs.

Thanks, @amarnathkarthik for your contribution. Also, big thanks for fixing existing bugs in this area.

Thanks, @amarnathkarthik for your contribution. Also, big thanks for fixing existing bugs in this area.

authors: amarnathkarthik codecov-io mcvsubbu


text:
Add function-package-url support in function cli add url support to function cli

Motivation
Pulsar-Function supports package-url from which worker can download and load function package. But right now, cli tool is not having a way to pass url arguments while calling function api.
Modifications
add cli support for function url.
Result
User can submit function with url.

@jerrypeng @srkukarni can you review the latest change?

@jerrypeng @srkukarni made the change with server-side validation.

authors: rdhabalia sijie

repo: apache/incubator-pulsar


text:
Remove --advertised-address from docker getting-started section

Motivation
Fixes #1929
Changes
Per comment, removed --advertised-address after 1.22.0

retest this please

authors: merlimat sijie


text:
Temporarely disable SequenceIdWithErrorTest

Motivation
There is a race condition between the consumer and producer reconnecting after a ledger fenced error that causes the test to get stuck and timeout.
Disabling this test to unblock CI on master while I'm working on proper fix for the race condition.

@sijie Yes, created  #2141 to track the issue that causes the test to fail.


text:
[REEF-244]  Adding Group Communication tests cases for Yarn env

This PR is to add test cases for group communication running on YARN
JIRA: REEF-244. (https://issues.apache.org/jira/browse/REEF-244)
This closes #
Author: Julia Wang  Email: jwang98052@yahoo.com

To verify it, please run all tests in VS to make sure no regression caused by the test refactor.
The new Yarn test cases are marked as [Ignore] for now. They will be run in Yarn env.

I'll test this.

repo: apache/incubator-reef


text:
[REEF-390]  Allow serialized configuration pass to Evaluator as a string

Currently, some of the Nodes were dropped by proto buffer libary when Java deserializes a class hierarchy created in C#, resulting in Java driver not able to deserialize service configurations passed from C# which use those classes. A typical example is nested generic types for ICodec which client would like to pass to the evaluator as part of the service config.
This PR:
*Add a new API GetServiceConfiguration to allow the client to pass additional service configuration
IConfiguration GetServiceConfiguration(params IConfiguration[] clientConfigurations);
*Inside this method, serialize the clientConfigurations as a string and bind it with service config
*Updated ServiceConfiguration to inject the named parameter to get the string config and deserialze it, and then merge it with the service config
*Test cases are added for the hcange
JIRA: REEF-390(https://issues.apache.org/jira/browse/REEF-390)
This closes #
Author: Julia Wang  Email: jwang98052@yahoo.com

I'll take a look.

Done with my pass.

review comments addressed.

I'll do another pass.

This has me confused :-)
I thought the issue was that we needed to avoid Java trying to deserialize the Service Configuration, as that fails. In the fullness of time, this will work and I don't want to contaminate the public API with workarounds, which this change seems to do: It changes the group communications API and introduces a new concept of first creating a Service Configuration with its API and then passing that over to SubmitContextAndService.
I'd much prefer a solution that is purely on the side of the REEF implementation, maintaining current APIs. Couldn't we just change the implementation of SubmitContextAndService to do the following?

Take the IConfiguration passed and serialize it.
Bind the serialized form to a string in a new IConfiguration.
Pass that new IConfiguration on to the Java side.

Then, on the Evaluator, we'd have to add code to unwrap the service configuration for use. Both of these code paths should be marked as hotfixes such that we can find them again in the future, e.g. while working on the new Evaluator.

This is the API that Dhruv needed. He added this API and passed ICodec configuration to it and it triggers the bug. Without this client passed configuration, we were fine.
Yes, we can put entire service configuration as a string. Not sure it might need Java side changes. If we would like to change the protoco between Java and Evaluator, we might want to do it separately.

wrapped the entire service config into a string in the service config when passing over the bridge. The change is pushed in and rebased.

I'll have a look

I've done my pass.

Either leave WrapServiceConfigAsString inside ServiceConfiguration class, put relevant code together, or move it to the class where it is used as a private method, so that it is hidden to the public, both should work. I now changed it to latter.

I'll merge this.


text:
The documentation for Seata 2.1.0 should point to the Docker Image of apache/seata-server

Why you need it?
Is your feature request related to a problem? Please describe in details

The documentation for Seata 2.1.0 should point to the Docker Image of apache/seata-server. See https://hub.docker.com/r/apache/seata-server and https://github.com/apache/incubator-seata/pull/6349 .
Both https://seata.apache.org/docs/ops/deploy-by-docker and https://seata.apache.org/docs/ops/deploy-by-docker-compose point to the outdated seataio/seata-server and https://hub.docker.com/r/seataio/seata-server/tags .

How it could be?
A clear and concise description of what you want to happen. You can explain more about input of the feature, and output of it.

I'm not sure if the current issue should be moved to https://github.com/apache/incubator-seata-website ?

Other related information
Add any other context or screenshots about the feature request here.

An early investigation is at https://github.com/apache/shardingsphere/pull/33030 and https://github.com/apache/shardingsphere/pull/31934 .

@linghengqian thanks for your feedback.

authors: linghengqian slievrly

repo: apache/incubator-seata


text:
Add 2.1.2 docs

Add 2.1.2 docs

@zhongjiajie @CalvinKirs @ruanwenjun Hi, please take a look, thanks!

repo: apache/incubator-seatunnel-website


text:
[Suggest][seatunnel-connector-spark-file]   sink S3  by used  plugin -file  easily (spark)

Search before asking

[X] I had searched in the feature and found no similar feature requirement.

Description

Missing documents   S3 . The file(file/hdfs/s3) plug-in is used to synchronize data to S3, but S3 cannot be written  according to the start-seatunnel-spark.sh   startup. After debug, found that  spark-submit  command lack parameters   com.amazonaws.services.s3.enableV4. There can use the --variable/-i   java opts . Obviously, it doesn't match the notes very well (--variable, -i PROP=VALUE     Variable substitution, such as -i city=beijing, or -i date=20190318).  Can be optimized accordingly for easy use ？
If you need to write to S3 by spark ,  please refer to the following configuration:
bin/start-seatunnel-spark.sh  -c  ${config-path}  -m master  -e  client  -i  com.amazonaws.services.s3.enableV4
It's a good project

Usage Scenario
Using the file plug-in, it is easier to write s3 without specifying additional parameters.  Hotspot service scenario requirements.
Related issues
no
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[ ] I agree to follow this project's Code of Conduct

Would you like contribute this? If so, look forward to your contribution.

repo: apache/incubator-seatunnel


text:
feature common : support external cros filter config.

using shenyu.cros-filter-config.* config the cors filter:
shenyu:
  cros-filter-config:
    allowed-headers: qicz, custom
    allowed-expose: *
    allowed-methods: *
    allowed-origin: *
    max-age: 18000

the allowed-headers default is x-requested-with, authorization, Content-Type, Authorization, credential, X-XSRF-TOKEN, token, username, client.
the external config will append to the allowed-headers.  the example of allowed-headers = x-requested-with, authorization, Content-Type, Authorization, credential, X-XSRF-TOKEN, token, username, client,  qicz, custom.

Hi,maybe you can provider a pr, make cross filter use configuration instead of hard coding

Hi,maybe you can provider a pr, make cross filter use configuration instead of hard coding

see #2000

authors: SaberSola qicz

repo: apache/incubator-shenyu


text:
[ISSUE #2450] register shenyu instance by etcd

About #2450 .

Make sure that:

[ ] You have read the contribution guidelines.
[ ] You submit test cases (unit or integration tests) that back your changes.
[ ] Your local test passed mvn clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #2474 (98579f2) into master (2384656) will decrease coverage by 0.39%.
The diff coverage is 0.00%.

:exclamation: Current head 98579f2 differs from pull request most recent head 076939a. Consider uploading reports for the commit 076939a to get more accurate results

@@             Coverage Diff              @@
##             master    #2474      +/-   ##
============================================
- Coverage     59.28%   58.89%   -0.40%
+ Complexity     4051     4019      -32
============================================
  Files           639      639
  Lines         18127    18127
  Branches       1689     1688       -1
============================================
- Hits          10746    10675      -71
- Misses         6535     6602      +67
- Partials        846      850       +4

Impacted Files
Coverage Δ

...apache/shenyu/register/client/etcd/EtcdClient.java
7.14% <0.00%> (ø)

...rg/apache/shenyu/admin/utils/ThreadLocalUtils.java
0.00% <0.00%> (-66.67%)
:arrow_down:

...che/shenyu/sync/data/http/HttpSyncDataService.java
61.38% <0.00%> (-19.81%)
:arrow_down:

...henyu/plugin/grpc/resolver/ShenyuNameResolver.java
51.57% <0.00%> (-11.58%)
:arrow_down:

...a/org/apache/shenyu/common/utils/VersionUtils.java
67.85% <0.00%> (-10.72%)
:arrow_down:

.../apache/shenyu/admin/model/dto/BatchCommonDTO.java
50.00% <0.00%> (-7.15%)
:arrow_down:

...apache/shenyu/admin/model/dto/AuthPathWarpDTO.java
57.14% <0.00%> (-7.15%)
:arrow_down:

...org/apache/shenyu/admin/model/dto/AuthPathDTO.java
58.82% <0.00%> (-5.89%)
:arrow_down:

...in/response/strategy/NettyClientMessageWriter.java
69.56% <0.00%> (-4.35%)
:arrow_down:

...che/shenyu/admin/model/vo/DashboardUserEditVO.java
62.50% <0.00%> (-4.17%)
:arrow_down:

... and 24 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2384656...076939a. Read the comment docs.

authors: JooKS-me codecov-commenter


text:
[bug-1784][console]MySQL datasource cannot be used normally

close #1784

Sorry, this issue is not a bug, when the user chooses mysql as the database needs the user to actively place the mysql driver under lib instead of adding maven denpendcy under pom.xml.

Hi @shangeyao, thanks for you feedback.
The license of mysql dependency is an category-x[1], which is not expected by the asf-project.
If you want to use mysql as the database, you can place the mysql driver under the lib.
[1] https://apache.org/legal/resolved.html#category-x

closed. cc @wolfboys

authors: lvshaokang shangeyao youyangkou

repo: apache/incubator-streampark


text:
Superset Real Time Visualization

I have a mariadb database connected to apache superset for real time visualization, however the dashboards are a day behind so they are only showing data up to 23:59 the previous day. I have checked the timezone and it is correct, I have also tried auto-refreshing after 5 minutes and 'force refresh' but the dashboards remain the same. I have tried opening the slices' queries in mysql lab but it is impossible to edit them to fetch real time data. Does anyone know how to get real time visualization using superset?

Try setting your date time range on your chart to be one day ahead.
Example: 14-June-2019 to 15-June-2019 will only display data within June 14th. It will not include any data in June 15th.
If you set your date range to be 14-June-2019 to 1 day from now you will see 15-June-2019 data in realtime assuming you are auto-refreshing every 5 minutes.

Inspect the generated SQL on the explore view and make sure it looks as expected

@mashciku if you want charts to default to showing all datapoints today instead of yesterday 23:59, set
DEFAULT_RELATIVE_END_TIME = 'now' in superset_config.py: https://github.com/apache/incubator-superset/blob/0d122431b287b1a34f006d91717d1b4e09136869/superset/config.py#L615

@villebro I have added DEFAULT_RELATIVE_END_TIME = 'now'  to the config.py and made it executable but nothing has changed. It's still showing 23:59.

@villebro Actually, it has worked. Thank you.

authors: davidhassan mashciku mistercrunch villebro

repo: apache/incubator-superset


text:
Add pxy to tuweni client

Allow users to deploy pxy sites as part of the tuweni client.

Fixed with #237

repo: apache/incubator-tuweni


text:
[AutoTVM][RPCRunner] timeout is not passed correctly

In the RPCRunner class the timeout is not correctly passed to the LocalExecutor and therefore the default timeout is always used.
see previous problem description at TVM discuss

I created a matching pull request 6924 for this.
We can not see it in the logs, I checked it in the logs is the correct highl-level value but the LocalExectuor works with the inner default. Probably there the read from the remote args works?

Is not readable in the logs, because for the logs self.timeout is used, which is correct.
So with the pull request it should be ok.

repo: apache/incubator-tvm


text:
[INLONG-11494][Sort] Add Kafka connector on Flink 1.18

Fixes #11494 [Feature][Sort] Add Kafka connector on Flink 1.18
Motivation
This PR adds a Kafka connector to inlong-sort for Flink 1.18 support.
Modifications

Migrated the Kafka connector from flink-kafka-connector 3.2.0 to support Flink 1.18.
Updated the relevant Maven pom.xml to reflect the changes.

Verifying this Change

[ ] This change is a trivial rework/code cleanup without any test coverage.

[ ] This change is already covered by existing tests, such as:
(please describe tests)

[x] This change added tests and can be verified as follows:
Further formal testing will be introduced in #11495.
Currently, a KafkaProducer is used to send messages to Kafka, and these messages are consumed by sort-connectors-v1.18, which is introduced in this PR.
Since there is no verified functional sink yet, the built-in file-system sink is used in this test.
In the example below, we can see that the messages in Kafka are successfully extracted and written to /tmp/output_file.

Example Flink SQL:
CREATE TABLE kafka_source (
    `message` STRING
) WITH (
    'connector' = 'kafka-inlong',
    'topic' = 'test-topic',
    'properties.bootstrap.servers' = 'kafka:9092',
    'properties.group.id' = 'flink-group',
    'scan.startup.mode' = 'earliest-offset',
    'format' = 'json'
);

CREATE TABLE file_sink (
    `message` STRING
) WITH (
    'connector' = 'filesystem',
    'path' = 'file:///tmp/output_file',
    'format' = 'json'
);

INSERT INTO file_sink
SELECT * FROM kafka_source;

Output:
The messages in Kafka have been successfully extracted and transported to /tmp/output_file. See the screenshot below for the result:

Documentation

Does this pull request introduce a new feature? Yes
If yes, how is the feature documented? N/A
If a feature is not applicable for documentation, explain why: The version change of the sort connectors is not reflected on the official website.

Sink function of this connector not tested yet.

please add license if there are some copied codes from other project.

please add license if there are some copied codes from other project.

Thanks for the reminder!
All the copied codes are from flink-connector-kafka. I believe they call follow Apache 2.0 license, so I don't have to add the license once more.

Please ref to inlong/licenses/inlong-sort-connectors/LICENSE, you should add the copied files to this file.

Please ref to inlong/licenses/inlong-sort-connectors/LICENSE, you should add the copied files to this file.

oh I see, sorry for the misunderstanding

repo: apache/inlong


text:
使用0.12.0版本session客户端，插入数据报类找不到错误，jar包里确实没这个类

Caused by: java.lang.ClassNotFoundException: org.apache.iotdb.rpc.RedirectException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

normally it is caused that you do not import all dependencies..
We suggest you use maven to manage the dependencies:
        <dependency>
            <groupId>org.apache.iotdb</groupId>
            <artifactId>iotdb-session</artifactId>
            <version>0.12.0</version>
        </dependency>

这一般都是没有使用maven或者其他依赖管理工具，而是自己手动仅导入了session一个jar包导致的。 建议使用maven管理依赖，如上所示。
否则，可以通过编译源码生成 with-dependencies的依赖包，这样的话导入一个jar包即可。
由于with-dependencies 的依赖包无法从网上的maven仓库下载， 目前只能手动编译：
mvn package -pl session -am -P get-jar-with-dependencies，
之后在session的target目录下就有以with-dependencies.jar结尾的jar包了。
另，如果手动逐一导入jar包，则列表为：
[INFO] --- maven-dependency-plugin:3.1.1:list (default-cli) @ iotdb-session ---
[INFO]
[INFO] The following files have been resolved:
[INFO]    org.apache.iotdb:service-rpc:jar:<YOUR VERSION>:compile
[INFO]    org.apache.iotdb:tsfile:jar:<YOUR VERSION>:compile
[INFO]    org.apache.iotdb:iotdb-thrift:jar:<YOUR VERSION>:compile
[INFO]    org.apache.thrift:libthrift:jar:0.13.0:compile
[INFO]    javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO]    org.xerial.snappy:snappy-java:jar:1.1.8.4:compile

normally it is caused that you do not import all dependencies..
We suggest you use maven to manage the dependencies:
        <dependency>
            <groupId>org.apache.iotdb</groupId>
            <artifactId>iotdb-session</artifactId>
            <version>0.12.0</version>
        </dependency>

这一般都是没有使用maven或者其他依赖管理工具，而是自己手动仅导入了session一个jar包导致的。 建议使用maven管理依赖，如上所示。
否则，可以通过编译源码生成 with-dependencies的依赖包，这样的话导入一个jar包即可。
由于with-dependencies 的依赖包无法从网上的maven仓库下载， 目前只能手动编译：
mvn package -pl session -am -P get-jar-with-dependencies，
之后在session的target目录下就有以with-dependencies.jar结尾的jar包了。
另，如果手动逐一导入jar包，则列表为：
[INFO] --- maven-dependency-plugin:3.1.1:list (default-cli) @ iotdb-session ---
[INFO]
[INFO] The following files have been resolved:
[INFO]    org.apache.iotdb:service-rpc:jar:<YOUR VERSION>:compile
[INFO]    org.apache.iotdb:tsfile:jar:<YOUR VERSION>:compile
[INFO]    org.apache.iotdb:iotdb-thrift:jar:<YOUR VERSION>:compile
[INFO]    org.apache.thrift:libthrift:jar:0.13.0:compile
[INFO]    javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO]    org.xerial.snappy:snappy-java:jar:1.1.8.4:compile

谢谢，我是用maven管理依赖的，问题解决了，原因是之前我用的0.11.2的版本，后来改成了0.12.0版本，没有update,依赖中的service-rpc版本还是0.11.2版本的，导致没找到org.apache.iotdb.rpc.RedirectException这个类。

repo: apache/iotdb


text:
JAMES-3775 Increase RSpamD scoring stability

RSpamD scoring delegates to many async modules and then aggregate the result. The implicit default task_timeout is 8 seconds which is sometime not long enough to return the full result.
Increasing timeout to a bigger value (say 20s) could let RSpamD enough time to fully score a mail.

On my PC

Can we please run rspamd tests as unstable tests while you investigate this issue / relax assertions to not include scoring?

I investigated the issue and likely it is a bug regarding to RSpamD (reported) and no more we can do at least ATM.

Can we please run rspamd tests as unstable tests while you investigate this issue / relax assertions to not include scoring?

The only place we assert the score (on master) is on a checkHam test which is not observed an unstable score issue.
But sure, relaxing the score there also is better, just in case.

Please rebase @quantranhong1999

authors: chibenwa quantranhong1999 vttranlina

repo: apache/james-project