text:
[HUDI-5341] IncrementalCleaning consider clustering completed later

Change Logs
refer to https://github.com/apache/hudi/pull/7401 (close accidentally)
In some scenes, such as offline clustering or online sync clustering(Parallel),  later plans could completed before previous plans. If later plan belong to the next partation, and cleaning speed catch up with it，Incremental Cleaning mode for getPartitionPathsForIncrementalCleaning would ignore previous plans ，eventually lead to duplicate data.
partation : day=2022-12-06/hour=10/minute_per_10=0   ,  day=2022-12-06/hour=10/minute_per_10=1

cleaning catch up with clustering belong to partation day=2022-12-06/hour=10/minute_per_10=1
log:

instant:

the file in previous clustering plan belong to partation 2022-12-06/hour=10/minute_per_10=0  won't clean forever. and it will cause duplicate data when the instants archived.

Impact
Describe any public API or user-facing feature change or any performance impact.
Risk level (write none, low medium or high below)
If medium or high, explain what verification was done to mitigate the risks.
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

@danny0405 How aboud add a check when attempt to archive ,   varify that every instant  completed time should be earlier than clean time one by one.

5341.patch.zip
Hi, @zhuanshenbsj1 , i have tried to review and applied a patch, add a weak guarantee on the archiver pre condition check, do you think that is okey ? It would be costy of we resolve all the clean metadata to see whether the clustering replaced files are all been cleaned.
We can go ahead if you think it is okey, and it would be great if you can test it also.

@danny0405, I have reviewed the above patch and tested whether the data of the replaced files is duplicated. The test result is that the data of the replaced files isn't duplicated for the query engine. But I confused that the replaced files also couldn't be cleaned after applying the patch. Is the behavior as you expect?

5341.patch.zip Hi, @zhuanshenbsj1 , i have tried to review and applied a patch, add a weak guarantee on the archiver pre condition check, do you think that is okey ? It would be costy of we resolve all the clean metadata to see whether the clustering replaced files are all been cleaned.
We can go ahead if you think it is okey, and it would be great if you can test it also.

But this patch still not consider clustering completed time ?  If previous clustering instant in timeline compelted later than following clean instant cleaned cutoff time， there are still risks.

@danny0405,  I agree with @zhuanshenbsj1. The oldest instant to retain for clustering should depend on the completed timestamp of the clustering instant, not the clustering instant time itself.

@danny0405, @zhuanshenbsj1, @stream2000, IMO, the bugfix couldn't be complex as the pull request changed. Just modify the getEarliestCommitToRetain to keep the linear cleaning with the following:
if (config.getCleanerPolicy() == HoodieCleaningPolicy.KEEP_LATEST_COMMITS
        && commitTimeline.countInstants() > commitsRetained) {
      Option<HoodieInstant> earliestPendingCommits = hoodieTable.getMetaClient()
          .getActiveTimeline()
          .getCommitsTimeline()
          .filter(s -> !s.isCompleted()).firstInstant();
      if (earliestPendingCommits.isPresent()) {
        // Earliest commit to retain must not be later than the earliest pending commit
        earliestCommitToRetain =
            commitTimeline.nthInstant(commitTimeline.countInstants() - commitsRetained).map(nthInstant -> {
              if (nthInstant.compareTo(earliestPendingCommits.get()) <= 0) {
                return Option.of(nthInstant);
              } else {
                return commitTimeline.findInstantsBefore(earliestPendingCommits.get().getTimestamp()).lastInstant();
              }
            }).orElse(Option.empty());
      } else {
        earliestCommitToRetain = commitTimeline.nthInstant(commitTimeline.countInstants() -
            commitsRetained); //15 instants total, 10 commits to retain, this gives 6th instant in the list
      }
 }

WDYT?

@danny0405, @zhuanshenbsj1, @stream2000, IMO, the clean operation should modify the getEarliestCommitToRetain to keep the linear cleaning with the following:
if (config.getCleanerPolicy() == HoodieCleaningPolicy.KEEP_LATEST_COMMITS
        && commitTimeline.countInstants() > commitsRetained) {
      Option<HoodieInstant> earliestPendingCommits = hoodieTable.getMetaClient()
          .getActiveTimeline()
          .getCommitsTimeline()
          .filter(s -> !s.isCompleted()).firstInstant();
      if (earliestPendingCommits.isPresent()) {
        earliestCommitToRetain =
            commitTimeline.nthInstant(commitTimeline.countInstants() - commitsRetained).map(nthInstant -> {
              if (nthInstant.compareTo(earliestPendingCommits.get()) <= 0) {
                return Option.of(nthInstant);
              } else {
                return commitTimeline.findInstantsBefore(earliestPendingCommits.get().getTimestamp()).lastInstant();
              }
            }).orElse(Option.empty());
      } else {
        earliestCommitToRetain = commitTimeline.nthInstant(commitTimeline.countInstants() - commitsRetained); //15 instants total, 10 commits to retain, this gives 6th instant in the list
      }
 }

WDYT?

It still need to confirm whether clustering instant has been cleaned when archiving.

@zhuanshenbsj1, +1. I have modified the getOldestInstantToRetainForClustering of the above patch with the following to take the case you mentioned:
/**
   * Checks whether the latest clustering instant has a subsequent cleaning action. Returns
   * the clustering instant if there is such cleaning action or empty.
   *
   * @param activeTimeline The active timeline
   * @return the oldest instant to retain for clustering
   */
  public static Option<HoodieInstant> getOldestInstantToRetainForClustering(HoodieActiveTimeline activeTimeline)
      throws IOException {
    Option<HoodieInstant> cleanInstant =
        activeTimeline.getCleanerTimeline().filter(instant -> !instant.isCompleted()).firstInstant();
    if (cleanInstant.isPresent()) {
      // The first clustering instant of which timestamp is greater than or equal to the earliest commit to retain of
      // the clean metadata.
      HoodieCleanMetadata cleanMetadata =
          TimelineMetadataUtils.deserializeHoodieCleanMetadata(
              activeTimeline.getInstantDetails(cleanInstant.get()).get());
      return activeTimeline.getCompletedReplaceTimeline()
          .filter(instant ->
              HoodieTimeline.compareTimestamps(
                  instant.getTimestamp(),
                  HoodieTimeline.GREATER_THAN_OR_EQUALS,
                  cleanMetadata.getEarliestCommitToRetain()))
          .firstInstant();
    }
    return Option.empty();
  }

PTAL.

@danny0405, IMO, this pull request could be close because the 7568 could replace the pull request.

closing the PR as we have the PR https://github.com/apache/hudi/pull/7568

repo: apache/hudi


text:
rest: refresh the token when expired

When received Authorization Expired, we should re-fetch the token and retry.

request for self-assign

source: gharchive/issue

repo: apache/iceberg-rust

license_type: permissive

license_source: github-api


text:
update README and other docs to call the project Apache Answer instead of Answer

Maybe not needed everywhere but the first reference on a page or section should probably refer to Apache Answer instead of Answer.
You have the Incubator footer on your web site pages but you might want to consider mentioning Incubation status of the project in the main body of your main page.

Thanks for the feedback, the website has been updated.

authors: fenbox pjfanning

repo: apache/incubator-answer


text:
Docker compose not working

Fix for #37

@moonming isn't it good to have a script to always sync with the repository config.yaml file this will help to resolve the problems we are getting in the future.

source: gharchive/pull-request

repo: apache/incubator-apisix-docker


text:
[BEAM-25] Add support for stateful ParDo / DoFn to direct runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[x] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

R: @tgroh
Here's a quick summary of the approach:

Expand stateful ParDo.BoundMulti into GbkIntoKeyedWorkItems followed by new pseudo-primitive StatefulParDo.
Add StatefulParDoEvaluatorFactory:

Here is where the cleanup scheduling is owned
Delegates everything else to ParDoEvaluatorFactory and ParDoEvaluator, involved some refactor to support.

I'll need to add more tests for StatefuParDoEvaluatorFactory to make sure that the normal processing aspects of it are not botched. It should be able to also pass all normal ParDo tests. It would be great to be able to re-use them by re-running ParDoTest with an over-zealous override that always used the stateful expansion.

High level commentary most appreciated. A couple automated refractors were used, so I'm still self-deceiving detailed bits.

I've moved changes to other runners to other PRs and gotten Jenkins green. The unusual class cast errors that we discussed were due to compiler-inserted casts failing. I found and eliminated the rawtypes that let the type errors through.

Jenkins failure was unrelated breakage at head fixed by #1418. Please review as green! Rebasing past the fix so it will green up.

repo: apache/incubator-beam


text:
[BEAM-78] Move annotations in org.apache.beam.sdk package

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[X] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[X] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[X] Replace "<Jira issue #>" in the title with the actual Jira issue
number, if there is one.
[X] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

R: @davorbonaci
Nice!

Rebase and update on checkstyle as requested by @davorbonaci

LGTM. Merging.

Merged. Thanks JB.

authors: davorbonaci jbonofre


text:
Optimize Map and Flatmap when there are no side inputs.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

varargs and kwargs are expensive, even when they're empty.
This is especially true for otherwise one-argument Python calls
which are special cased in CPython.

R: @aaltay

LGTM. Thank you.

authors: aaltay robertwb


text:
[QUESTION] Should we need Beeline for hive connection?

Describe the question
DS current use hive jdbc to execute hive sql, but it can not good fit these:

Put multiple sql within a sql task
Dynamic table name. Each data task,  we use some temporary tables, those table name has a date argument,  like tmp.tmp_table_xxxx_20191208, DS can't pass a argument to as tableName
Argument as a part of string.

Which version of DolphinScheduler:

[1.1.0]

Additional context
We plan to migrate our data task from Oozie to DS, and we use beeline before, but we meet those problem with hive task. We know jdbc is much common for different datasource, so Beeline may not necessary, any good solution or workaround is accepted

can you use Beeline in shell task?

you can use shell task to use beeline like beeline -u jdbc:hive2:xxx -n ${tenant_id}, but if you not set-n ${tenant_id} param the hive table will belong to user=anonymous not the workflow ${tenant_id}

I believe invoke beeline in shell is a solution, but I think it’s not friendly for developers, since we need to put sql file on hdfs or DS resources manager, also we need to set a lot of hivevars within shell command if need to pass arguments. Current UI of sql task I think is friendly enough.

This maybe not necessary, I open this issue just want to know is anyone else meet this problem and how they solve this.

authors: KEN-LJQ Yeleights dailidong

repo: apache/incubator-dolphinscheduler


text:
Using Jackson instead of Fastjson

I replaced all fastjson except fastjson configuration related to SerializerFeature.

Codecov Report

Merging #2858 into dev will decrease coverage by 0.44%.
The diff coverage is 8.96%.

@@             Coverage Diff              @@
##                dev    #2858      +/-   ##
============================================
- Coverage     37.21%   36.77%   -0.45%
+ Complexity     2560     2538      -22
============================================
  Files           435      434       -1
  Lines         20020    20029       +9
  Branches       2426     2433       +7
============================================
- Hits           7451     7366      -85
- Misses        11904    12007     +103
+ Partials        665      656       -9

Impacted Files
Coverage Δ
Complexity Δ

...inscheduler/alert/utils/EnterpriseWeChatUtils.java
61.70% <0.00%> (+1.28%)
13.00 <0.00> (ø)

...pache/dolphinscheduler/alert/utils/ExcelUtils.java
78.94% <ø> (ø)
8.00 <0.00> (ø)

...olphinscheduler/api/service/DataSourceService.java
2.22% <0.00%> (ø)
3.00 <0.00> (ø)

...cheduler/api/service/ProcessDefinitionService.java
60.86% <0.00%> (ø)
75.00 <0.00> (ø)

...dolphinscheduler/api/service/ResourcesService.java
57.06% <ø> (ø)
67.00 <0.00> (ø)

...apache/dolphinscheduler/common/model/TaskNode.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

...che/dolphinscheduler/common/utils/HadoopUtils.java
34.17% <0.00%> (-0.51%)
21.00 <0.00> (ø)

.../dolphinscheduler/common/utils/ParameterUtils.java
40.00% <ø> (ø)
13.00 <0.00> (ø)

...cheduler/remote/command/TaskExecuteAckCommand.java
0.00% <ø> (ø)
0.00 <0.00> (ø)

...uler/remote/command/TaskExecuteRequestCommand.java
0.00% <ø> (ø)
0.00 <0.00> (ø)

... and 48 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8f8ddae...d9e6b31. Read the comment docs.


text:
fix create table sql exception

fix create table sql exception

small fix, no need reviewer


text:
[Improvement][Server] Improve dependent task unit tests

What is the purpose of the pull request
Improve dependent task unit tests
This pr is just about DependentTaskTest.java and closes the outdated pr #3442
The getTaskLogPath changes of MasterBaseTaskExecThread.java and MasterTaskExecThreadTest.java have been removed by the same author hsupu with the merged pr #3629.
Brief change log

Improve dependent task unit tests

Verify this pull request
This pull request is already covered by existing tests, such as (please describe tests).

Codecov Report

Merging #4911 (ca17246) into dev (54fc458) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                dev    #4911      +/-   ##
============================================
- Coverage     46.89%   46.88%   -0.01%
+ Complexity     3624     3622       -2
============================================
  Files           567      567
  Lines         23810    23810
  Branches       2742     2742
============================================
- Hits          11166    11164       -2
- Misses        11634    11635       +1
- Partials       1010     1011       +1

Impacted Files
Coverage Δ
Complexity Δ

...er/master/dispatch/host/assign/RandomSelector.java
77.77% <0.00%> (-5.56%)
3.00% <0.00%> (-1.00%)

...olphinscheduler/server/utils/DependentExecute.java
60.52% <0.00%> (-1.32%)
16.00% <0.00%> (-1.00%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 54fc458...ca17246. Read the comment docs.

authors: chengshiwen codecov-io


text:
[bug]一个简单查询，会时不时unknown error

Describe the bug
一个简单查询：
select id from 表  limit 1
在代码中：
while（true）{
JDBC 执行 SQL
}
会不定期出现:
Exception in thread "main" com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown error
在fe.warn.log中看到的日志是：
2020-09-22 17:21:13,368 WARN 24043 [StmtExecutor.execute():306] execute Exception
java.util.ConcurrentModificationException: null
at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211) ~[?:1.8.0_252]
at java.util.TreeMap$KeyIterator.next(TreeMap.java:1265) ~[?:1.8.0_252]
at org.apache.doris.common.util.RuntimeProfile.printChildCounters(RuntimeProfile.java:235) ~[palo-fe.jar:?]
at org.apache.doris.common.util.RuntimeProfile.prettyPrint(RuntimeProfile.java:212) ~[palo-fe.jar:?]
at org.apache.doris.common.util.RuntimeProfile.prettyPrint(RuntimeProfile.java:219) ~[palo-fe.jar:?]
at org.apache.doris.common.util.RuntimeProfile.prettyPrint(RuntimeProfile.java:219) ~[palo-fe.jar:?]
at org.apache.doris.common.util.RuntimeProfile.prettyPrint(RuntimeProfile.java:219) ~[palo-fe.jar:?]
at org.apache.doris.common.util.RuntimeProfile.prettyPrint(RuntimeProfile.java:219) ~[palo-fe.jar:?]
at org.apache.doris.qe.StmtExecutor.writeProfile(StmtExecutor.java:342) ~[palo-fe.jar:?]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:244) ~[palo-fe.jar:?]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:184) ~[palo-fe.jar:?]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:316) ~[palo-fe.jar:?]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:501) ~[palo-fe.jar:?]
at org.apache.doris.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:50) ~[palo-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
2020-09-22 17:21:13,385 WARN 24043 [ConnectProcessor.processOnce():491] Null packet received from network. remote: XXX.XXX.XXX.74:38188
2020-09-22 17:21:13,385 WARN 24043 [ReadListener.lambda$handleEvent$0():58] Exception happened in one session(org.apache.doris.mysql.nio.NConnectContext@4f53ffef).
java.io.IOException: Error happened when receiving packet.
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:492) ~[palo-fe.jar:?]
at org.apache.doris.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:50) ~[palo-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
补充：
fe配置：
max_routine_load_job_num = 270
max_layout_length_per_row = 100000000
max_routine_load_task_num_per_be = 27
label_keep_max_second = 3600
history_job_keep_max_second = 3600
label_clean_interval_second = 3600
max_running_txn_num_per_db = 300
audit_log_delete_age = 7d
sys_log_delete_age = 7d
query_colocate_join_memory_limit_penalty_factor = 1
thrift_server_max_worker_threads = 6144
be配置：
routine_load_thread_pool_size = 27
sys_log_roll_num = 1
base_compaction_num_threads_per_disk = 3
base_compaction_check_interval_seconds = 10
cumulative_compaction_check_interval_seconds = 1
cumulative_compaction_skip_window_seconds = 10
min_compaction_failure_interval_sec = 10
cumulative_compaction_num_threads_per_disk = 3
10个be节点，20分桶，无分区

不是很理解一个查询的SQL，为什么会出现一致性修改错误，一般不都是对iterator遍历的时候做删除，会出这种错误嘛

再次补充：
通过be.log，在查询失败附近，会发现：
INFO 393 [QeProcessorImpl.reportExecStatus():118] ReportExecStatus() runtime error, query TUniqueId(hi:-5892633407530515572, lo:-7549245074990846607) does not exist

再次补充，使用group by 或者 join 后的SQL，不会出现该错误

@li-ang-666 Hi,  @HangyuanLiu  has fixed this bug.
@HangyuanLiu Hi, please submit a PR when you have time, thanks!

@li-ang-666 Hi, @HangyuanLiu has fixed this bug.
@HangyuanLiu Hi, please submit a PR when you have time, thanks!

@kangkaisen 0.13.9 , 这个bug修复好了嘛

repo: apache/incubator-doris


text:
Kafka: Add warning to doc for earlyMessageRejectionPeriod.

See #6643.

👍

authors: dclim gianm

repo: apache/incubator-druid


text:
remove useless module-dubbo-test-integration

dubbo project looks like some modules are useless,such as dubbo-test-integration,would we remove it?

Could you pull a request for it? :D

we should remove dubbo-test completely from the main trunk.

authors: beiwei30 mercyblitz xeostream

repo: apache/incubator-dubbo


text:
想获取北京市石景山区的geoJson

One-line summary [问题简述]
Version & Environment [版本及环境]

ECharts version [ECharts 版本]:
Browser version [浏览器类型和版本]:
OS Version [操作系统类型和版本]:

Expected behaviour [期望结果]
ECharts option [ECharts配置项]

option = {

}

Other comments [其他信息]

https://github.com/echarts-maps/echarts-china-counties-js#直辖市

@chfw 能提供街道信息吗

我可以尝试一下。请发具体要求到我的联系信箱 wangc_2011@hotmail.com 。时间周期会长一点啊，我得找空做。

@chfw  你好，做这样类型的地图数据，采用什么方法或工具呢，有出教程吗？

authors: chfw hl-stone lishaoh

repo: apache/incubator-echarts

license_source: bigquery


text:
如何保存整张图片，而不是当前屏幕显示的部分

画了一个很大的关系图，在浏览器中可以拖动查看，
但保存图片时，只能保存当前屏幕显示的部分
如何能保存整张图片？

f12打开浏览器后台，再按ctrl+shift+p会出现一个搜索框，在搜索框中输入full,选中"Capture full size screenshot",浏览器会自动帮你下载完整页面的图片


text:
关于3D地球(about 3d earth)

我在地球上标记了一些国家，或者是一些点。地球在转动过程中，我能得知当前转动到哪个点或者国家吗?
I have marked some countries or some points on the earth. In the course of rotation, can I know which point or country to turn to?
应该可以通过计算转动角度来计算获取，但是这样我需要隔离用户的鼠标操作。我想知道是否有一种更加好用的方式来获取，类似事件？
It should be able to calculate the rotation angle, but I need to isolate the user's mouse operation. I want to know if there is a more convenient way to obtain similar events.
非常感谢！
Thank you very much!

请问有解决办法了吗，我也碰到了这个需求。

请问有解决办法了吗，我也碰到了这个需求。

如果不需要用户操作的话，你可以试试上面那个思路。。

请问如何获取转动角度？自己写个计时器去计算吗？


text:
[ISSUE #3085] Method uses integer based for loops to iterate over a List

Fixes #3085 .
Modifications
Changed the for loop to foreach loop at line 1538

Does this pull request introduce a new feature? (no)

@xwm1992 can you please take a look at the error and explain? I am unable to get this

@xwm1992 can you please take a look at the error and explain? I am unable to get this

A space may need to be added.

Hi @PreetAngadSingh, this issue has been fixed, so I will close this pull request.

authors: Alonexc PreetAngadSingh xwm1992

repo: apache/incubator-eventmesh


text:
NPM deployment automatically starts regardless of usage

The flagon-userale NPM package has a handful of issues. The most egregious (in my eyes) is that, even if I don't explicitly start it, it will begin collecting/sending logs as soon as it is imported. Maybe there is some argument that this is intended behavior, but having it do this is probably a surprise for developers. For example, this React project results in log collection/sending:
// in index.js
import React from 'react';
import ReactDOM from 'react-dom';
import App from './App';

ReactDOM.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>,
  document.getElementById('root')
);

// in App.jsx
import { packageCustomLog } from "flagon-userale";
import { useCallback, useEffect, useState } from "react";

function App() {
  const [secondsElapsed, setElapsed] = useState(0);
  const resetTimer = useCallback(() => {
    setElapsed(0);
    packageCustomLog({ elapsed: secondsElapsed, logType: "timerReset" }, undefined, false);
  }, [secondsElapsed, setElapsed]);

  useEffect(() => {
    const timerId = setTimeout(() => {
      setElapsed(secondsElapsed + 1);
      packageCustomLog({ elapsed: secondsElapsed, logType: "timerUpdate" }, undefined, false);
    }, 1000);

    return () => {
      clearTimeout(timerId);
    };
  }, [secondsElapsed, setElapsed]);

  return (

App has been running for {secondsElapsed} seconds.

      <button onClick={resetTimer}>Reset</button>

  );
}

export default App;

This surprises me, since it means that importing any part of userale will automatically start the logging process. I'd imagine we ought to figure out a way to have it not start by default when imported via NPM. Of course, this is a breaking change.

That's a good ticket. So far haven't heard much concern from the users I'm connected with, but they were expecting this behavior. Maybe should add something to './src/getInitialSettings' and some examples for triggering UserALE's start state as well as clearly defined default start behavior and more "customized" options... What do you think "most developers" will expect from a package like this? This behavior is identical to the script tag option. I imagine there would be some level of refactoring needed to configure the exports such that if you just wanted to use packageCustomLogs, you're only getting that export and related exports/imports.
Likely big reconfiguring of './src/main.js'. Currently, all the libraries in ./src are effectively daisy-chained because we originally concieved of this as a script tag. Not a module with optional imports.

Regarding developer expectations, I would not expect it to start logging everthing immediately. I'd also like to see this behavior turned off by default

I imagine there would be some level of refactoring needed to configure the exports such that if you just wanted to use packageCustomLogs, you're only getting that export and related exports/imports.

that is correct, we don't actually import from src at all anyways. we have the main in package.json set to the built script, meaning it is an all-or-nothing deal. i think it may just be surprising, as there is a start function exported, but userale will actually start whenever some part of it gets imported somewhere. it is then the user's responsibility to call stop and then call options (to actually configure it), and finally call start to actually begin. all of this, mind you, also means they can't actually use the resolution option in the config, since it is applied during attachHandlers, which only gets called once (during initial startup).

Cool. I can play around with this. Good ticket.

Did you make any progress on this? It came up in a discussion with some potential users.

This has become an issue for us. Has anyone started work on this? If not I'll have a crack at it.

@kevbrowndev I've started work on a proof-of-concept for this. In the meantime, a workaround for users who don't want userale to start automatically could be to call the stop function as mentioned above

Cool thanks I appreciate you looking at this.
I've started a proof of concept as well and hope to push it tomorrow for
comment and review.
Calling stop immediately does not work unfortunately. When the setup
function runs it will push right through and set on to true no matter what.
I was thinking of doing something with autostart. Do you think that makes
sense? I mean what is autostart = false for if not to say, hey, I don't
want userale sending logs?
Alternatively, maybe we could have another variable on config called
hardstop. That would be set in the stop function to true and it would be
checked in the setup function. Start wood set hardstop to false. Just a
thought.
On Fri, Mar 18, 2022, 5:14 PM UncleGedd @.***> wrote:

@kevbrowndev https://github.com/kevbrowndev I've started work on a
proof-of-concept for this. In the meantime, a workaround for users who
don't want userale to start automatically could be to call the stop
function as mentioned above
—
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-flagon-useralejs/issues/72#issuecomment-1072799786,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOZLZ6KMR2WZGFUDAQLXALVATWZXANCNFSM4ZXVGRIQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID:
@.***>

Ok, I may have stumbled into a super simple solution here. In getInitialSettings.js we set:
    settings.autostart = get('data-autostart') === 'false' ? false : true;

This causes userale to start up automatically unless the data-autostart property is explicity set to false. Which in the case of an NPM package, will not be set at all. This can be averted by simply inverting the logic to:
    settings.autostart = get('data-autostart') === 'true' ? true : false;

With this change, in an NPM deployment, the start function must be called before userale will start working, and in the case of a script deployment, users must set data-autostart to true. This is of course a breaking change for script users

Yes, thank you, I noticed this too! I was wondering if there was some
reason for it.
How to properly test this fix? Do you think it would be a good idea to add
a second, separate journey test, that tested autostart = false?
On Sat, Mar 19, 2022, 6:34 PM UncleGedd @.***> wrote:

Ok, I may have stumbled into a super simple solution here. In
getInitialSettings.js we set:
settings.autostart = get('data-autostart') === 'false' ? false : true;

This causes userale to start up automatically unless the data-autostart
property is explicity set to false. Which in the case of an NPM package,
will not be set at all. This can be averted by simply inverting the logic
to:
settings.autostart = get('data-autostart') === 'true' ? true : false;

With this change, in an NPM deployment, the start function must be called
before userale will start working, and in the case of a script deployment,
users must set data-autostart to true. This is of course a breaking
change for script users
—
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-flagon-useralejs/issues/72#issuecomment-1073122703,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOZLZ3UIGFYSI4O44H6L6DVAZI7ZANCNFSM4ZXVGRIQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID:
@.***>

With this change, in an NPM deployment, the start function must be called before userale will start working, and in the case of a script deployment, users must set data-autostart to true. This is of course a breaking change for script users

Instead of making a breaking change for script users, is there perhaps a way to detect whether we are in a script vs NPM deployment? Maybe we can make use of available environment variables? (though that implies a build process)

In order to prevent breaking changes, what about this?
In Userale, the configure() function could be altered at line 32 so that if
autostart is set via the options() function, as could happened if userale
is being used as an npm package, that value trumps the getInitialSettings()
version of autostart.
This way, the code should work the same as it does now, unless autostart
value is set to true in the Userale.options() function. In that case, no
Userale logs would be sent.
https://github.com/apache/incubator-flagon-useralejs/blob/master/src/configure.js
On Sun, Mar 20, 2022, 11:53 AM Rob @.***> wrote:

With this change, in an NPM deployment, the start function must be called
before userale will start working, and in the case of a script deployment,
users must set data-autostart to true. This is of course a breaking
change for script users
Instead of making a breaking change for script users, is there perhaps a
way to detect whether we are in a script vs NPM deployment? Maybe we can
make use of available environment variables? (though that implies a build
process)
—
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-flagon-useralejs/issues/72#issuecomment-1073279453,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOZLZ2SBLTIYHL6MNBRB7TVA5CXFANCNFSM4ZXVGRIQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID:
@.***>

Yes, thank you, I noticed this too! I was wondering if there was some reason for it. How to properly test this fix? Do you think it would be a good idea to add a second, separate journey test, that tested autostart = false?

Adding a journey test and asserting if a network request did not occur would likely add signficant time to our test suite (waiting for Cypress to time out). I would try a unit test approach first, or just make the change and edit the unit/journey tests that would undoubtedy break

In order to prevent breaking changes, what about this? In Userale, the configure() function could be altered at line 32 so that if autostart is set via the options() function, as could happened if userale is being used as an npm package, that value trumps the getInitialSettings() version of autostart. This way, the code should work the same as it does now, unless autostart value is set to true in the Userale.options() function. In that case, no Userale logs would be sent. https://github.com/apache/incubator-flagon-useralejs/blob/master/src/configure.js

I can't quite picture it, but I'm more than happy to take a look at a PR (a WIP is totally fine).
My bigger concern is around creating a userale behavior dichotomy between script vs NPM users. My opinion is that we should have Userale behave the same regardless of source. I'm defintely open to other opinions though

What about something like this?
Script tag Userale and NPM package Userale behave the same. Might be a tad kludgey...

authors: UncleGedd confusingstraw kevbrow kevbrowndev poorejc

repo: apache/incubator-flagon-useralejs


text:
Deleted AUTHORS file, added NOTICE file, and removed 'incubator' from…

… links to project website

Don't forget about the DISCLAIMERS file with the necessary notification that fluo isn't a TLP (but an on-going effort in the ASF incubator)
http://incubator.apache.org/guides/branding.html

e.g. https://github.com/apache/incubator-slider/blob/master/DISCLAIMER

Looks great! Thanks @mikewalch

Cool.  Added DISCLAIMERS file in 78447d2953434

+1

authors: joshelser keith-turner mikewalch

repo: apache/incubator-fluo


text:
[GRIFFIN-194] single call to fetch all tables of all dbs

Significantly improves opening speed for measure dialogs.
Previously taking 10+ seconds on huge schemas, with this call
it takes fraction of second to load.

@guoyuepeng can you please clarify your comments, please? Not very clear what changes are required, if any.

@chemikadze  LGTM.
Just rushed out for my meeting. sorry for confusing.

authors: chemikadze guoyuepeng

repo: apache/incubator-griffin


text:
HAWQ-148. Bump up optimizer and gpos version.

Bumping optimizer version number to enable stats derivation for all groups

Looks good to me. +1

+1

authors: entong hsyuan vraghavan78

repo: apache/incubator-hawq


text:
[Bug] pd make the RaftEngine constructor private

Bug Type (问题类型)
logic (逻辑设计问题)
Before submit

[X] 我已经确认现有的 Issues 与 FAQ 中没有相同 / 重复问题 (I have confirmed and searched that there are no similar problems in the historical issue and documents)

Environment (环境信息)
Server Version: 1.5
Expected & Actual behavior (期望与实际表现)
The RaftEngine is intended to be a global singleton, but the constructor is not private.
public RaftEngine() {
        this.stateMachine = new RaftStateMachine();
    }

Vertex/Edge example (问题点 / 边数据举例)
No response
Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)
No response

@dosu
The RaftEngine constructor is actually public, so I think this breaks the design of the singleton pattern

repo: apache/incubator-hugegraph


text:
Opt-in deploy SonataFlow as Knative Serving service instead of k8s Deployments

Description
Currently, the operator deploys workflows as regular Kubernetes Deployments. In some use cases, deploying a workflow as a Knative Serving resource makes sense.

Assess the possibility of deploying workflows as Knative Serving services by default if Knative is installed in the cluster
Add the possibility for users to opt-in to deploying as a ksvc or k8s deployment by labeling the CR.
Once labeled and deployed, this attribute must be immutable. We may assess the possibility of changing this field later in the field
Initially, revisions handling is out of scope, the operator won't reconcile this field so that users can change it based on their requirements. An assessment of revisions in the platform can be done later

Implementation ideas
No response

What about providing an option for deployment method (knative service or k8s deployment) at the SonataflowPlatform level? so for each production profile workflow, the same will method will be applied.

We might add that in the future, sure.

authors: masayag ricardozanini


text:
Move the nightly docker image organization to the apache

This closes #1019

I don't know whether we should update the secret for uploading permissions of apache org.
Request changes to prevent accidental merge. If CI passed then we can merge.

It should be fine since the old docker user/token is created by my personal account which also have applied the permission to push to the apache organization.

repo: apache/incubator-kvrocks


text:
[KYUUBI #3545][KYUUBI #3563] Support restrict spark configurations

Why are the changes needed?

ban end-user from security settings
How was this patch tested?

[x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

This PR aims to restrict configurations from a set statement(Via SET Statements).
We need to also handle:
1). modify configurations in the scala execution mode.
spark.sessionState.conf.setConfString(OPTIMIZER_EXCLUDED_RULES.key, somevalues)

2). set a configuration from the connection URL. Via JDBC Connection URL

@yaooqinn @Yikf @bowenliang123 @lordk911

@Yikf Spark does not provide any public API to change the default behavior, preventing the optimizer phase rule be excluded by the conf spark.sql.optimizer.excludedRules, even though I agree that intercepting the set config at optimizer runtime is a better approach.

also update the doc? cc @bowenliang123

explain for what, sorry.

I don't quite get the point after checking these comments. I can explain a bit about my idea, which may not the best solution anyway.
So,

What is the problem?

Configurations set by end-users may harm the whole system while the admin wants to ban them ahead.

Which way to set and what is the influence?

Configs though JDBC string

will change engine defaults if no live engine yet. End-users can do anything they want if we do not have a restricted list.
will change session defaults if the engine exists. End-users can change non-static SQL configurations only if we do not have a restricted list.

Configs though SET cmd. End-users can change non-static SQL configurations only if we do not have a restricted list.

So introduce ban lists?

For 2.1 and 2.2, we have kyuubi.session.conf.ignore.list and kyuubi.session.conf.restrict.list at serverside for banning session+ level changes.
For 2.3, like what I did in the submarine project at engineside for banning operation level changes.

For scala or any programming lang support in the feature?

If you let users run such code or program, it means that you give up sys security. otherwise, you shall ban it.
may we need another more complete ACL management framework for configs.

thanks.
Kent

Codecov Report

Merging #3564 (a8d35ef) into master (ef68aa0) will decrease coverage by 0.03%.
The diff coverage is 71.42%.

:exclamation: Current head a8d35ef differs from pull request most recent head e4e554f. Consider uploading reports for the commit e4e554f to get more accurate results

@@             Coverage Diff              @@
##             master    #3564      +/-   ##
============================================
- Coverage     51.71%   51.68%   -0.04%
  Complexity       13       13
============================================
  Files           482      483       +1
  Lines         26940    26964      +24
  Branches       3761     3765       +4
============================================
+ Hits          13933    13937       +4
- Misses        11653    11664      +11
- Partials       1354     1363       +9

Impacted Files
Coverage Δ

...uthz/ranger/AuthzConfigurationCheckExtension.scala
69.23% <69.23%> (ø)

...ugin/spark/authz/ranger/RangerSparkExtension.scala
100.00% <100.00%> (ø)

.../kyuubi/server/mysql/constant/MySQLErrorCode.scala
13.84% <0.00%> (-6.16%)
:arrow_down:

...ache/kyuubi/server/mysql/MySQLCommandHandler.scala
75.00% <0.00%> (-4.55%)
:arrow_down:

...apache/kyuubi/engine/JpsApplicationOperation.scala
77.41% <0.00%> (-3.23%)
:arrow_down:

...ache/kyuubi/server/mysql/MySQLGenericPackets.scala
76.59% <0.00%> (-2.13%)
:arrow_down:

...rg/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala
78.00% <0.00%> (-2.00%)
:arrow_down:

...mon/src/main/scala/org/apache/kyuubi/Logging.scala
52.63% <0.00%> (-1.32%)
:arrow_down:

...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala
80.74% <0.00%> (-0.63%)
:arrow_down:

...a/org/apache/kyuubi/service/TFrontendService.scala
91.17% <0.00%> (-0.30%)
:arrow_down:

... and 2 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

also update the doc? cc @bowenliang123

thanks all.

authors: cfmcgrady codecov-commenter yaooqinn

repo: apache/incubator-kyuubi


text:
[Bug] [ECP]The task error message is unclear when the ec plugin does not exist

Search before asking

[X] I searched the issues and found no similar issues.

Linkis Component
linkis-cg-engineConnplugin
What happened + What you expected to happen
Now the error thrown by ecp is correct, it will print ecp does not exist exception, as follows:
现在ecp抛出的错误时正确的，会打印ecp不存在异常，如下：
Caused by: org.apache.linkis.manager.engineplugin.common.exception.EngineConnPluginNotFoundException: errCode: 70063 ,desc: No plugin found shell-1, please check your configuration ,ip: bdpujes110003 ,port: 9103 ,serviceKind: linkis-cg-engineplugin
        at org.apache.linkis.manager.engineplugin.manager.loaders.DefaultEngineConnPluginLoader.loadEngineConnPluginInternal(DefaultEngineConnPluginLoader.java:173) ~[linkis-engineconn-plugin-loader-1.1.1-webank.jar:1.1.1-webank]
        at org.apache.linkis.manager.engineplugin.cache.GuavaEngineConnPluginCache.lambda$get$2(GuavaEngineConnPluginCache.java:122) ~[linkis-engineconn-plugin-cache-1.1.1-webank.jar:1.1.1-webank]
        at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4853) ~[guava-30.0-jre.jar:?]
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529) ~[guava-30.0-jre.jar:?]
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278) ~[guava-30.0-jre.jar:?]
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155) ~[guava-30.0-jre.jar:?]
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) ~[guava-30.0-jre.jar:?]
        at com.google.common.cache.LocalCache.get(LocalCache.java:3951) ~[guava-30.0-jre.jar:?]
        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4848) ~[guava-30.0-jre.jar:?]
        at org.apache.linkis.manager.engineplugin.cache.GuavaEngineConnPluginCache.get(GuavaEngineConnPluginCache.java:119) ~[linkis-engineconn-plugin-cache-1.1.1-webank.jar:1.1.1-webank]
        at org.apache.linkis.manager.engineplugin.manager.loaders.CacheablesEngineConnPluginLoader.getEngineConnPlugin(CacheablesEngineConnPluginLoader.java:82) ~[linkis-engineconn-plugin-loader-1.1.1-webank.jar:1.1.1-webank]
        at org.apache.linkis.engineplugin.server.service.DefaultEngineConnResourceFactoryService.getResourceFactoryBy(DefaultEngineConnResourceFactoryService.scala:36) ~[linkis-engineconn-plugin-server-1.1.1-webank.jar:1.1.1-webank]
        at org.apache.linkis.engineplugin.server.service.DefaultEngineConnResourceFactoryService.createEngineResource(DefaultEngineConnResourceFactoryService.scala:47) ~[linkis-engineconn-plugin-server-1.1.1-webank.jar:1.1.1-webank]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]

However, there is no clear information about the exception after RPC encapsulation obtained in the linkismanager.
2022-03-30 21:48:14.447 [INFO ] [AskEngineService-Thread-98              ] o.a.l.m.a.s.e.DefaultEngineAskEngineService (121) [apply] - Failed  to async(bdpujes110003:9101_48) createEngine, can Retry false org.apache.linkis.manager.am.exception.AMErrorException: errCode: 30001 ,desc: Failed to create engineResource ,ip: bdpujes110003 ,port: 9101 ,serviceKind: linkis-cg-linkismanager
        at org.apache.linkis.manager.am.pointer.DefaultEngineConnPluginPointer.createEngineResource(DefaultEngineConnPluginPointer.scala:39) ~[linkis-application-manager-1.1.1-webank.jar:1.1.1-webank]
        at org.apache.linkis.manager.am.service.engine.DefaultEngineCreateService.requestResource(DefaultEngineCreateService.scala:253) ~[linkis-application-manager-1.1.1-webank.jar:1.1.1-webank]
        at org.apache.linkis.manager.am.service.engine.DefaultEngineCreateService.createEngine(DefaultEngineCreateService.scala:181) ~[linkis-application-manager-1.1.1-webank.jar:1.1.1-webank]
        at org.apache.linkis.manager.am.service.engine.DefaultEngineAskEngineService$$anonfun$3.apply(DefaultEngineAskEngineService.scala:92) ~[linkis-application-manager-1.1.1-webank.jar:1.1.1-webank]
        at org.apache.linkis.manager.am.service.engine.DefaultEngineAskEngineService$$anonfun$3.apply(DefaultEngineAskEngineService.scala:82) ~[linkis-application-manager-1.1.1-webank.jar:1.1.1-webank]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[scala-library-2.11.12.jar:?]
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[scala-library-2.11.12.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

Relevent platform
all
Reproduction script

delete shell ecp
execute task

Anything else
It is recommended to add an error response message protocol for ecp resource requests, and then throw an error message
建议新增ecp资源请求的错误响应信息协议，然后抛出错误信息

Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

I will fix this

authors: legendtkl peacewong

repo: apache/incubator-linkis


text:
Floating point exception in mxnet.ndarray.op.SequenceMask

Description
(A clear and concise description of what the bug is.)
mxnet.ndarray.op.SequenceMask has floating point exception when given data has 0 in its shape. Please see the provided code for example.
Error Message
(Paste the complete error message. Please also include stack trace by setting environment variable DMLC_LOG_STACK_TRACE_DEPTH=10 before running your script.)

Floating point exception (core dumped)

To Reproduce
(If you developed your own code, please provide a short script that reproduces the error. For existing examples, please provide link.)
import mxnet
import numpy as np
data = mxnet.nd.array(np.random.rand(0,1,1))
mxnet.ndarray.op.SequenceMask(data)

Steps to reproduce
(Paste the commands you ran that produced the error.)

run the provided code in python interpreter or as a script

What have you tried to solve it?

Environment
We recommend using our script for collecting the diagnositc information. Run the following command and paste the outputs below:
curl --retry 10 -s https://raw.githubusercontent.com/dmlc/gluon-nlp/master/tools/diagnose.py | python

# paste outputs here

Got 404 when trying to get the script.
Some environment information:

OS: ubuntu 18.04
Python: 3.7.6
pip: 20.0.2
numpy: 1.18.5
mxnet: 1.6.0

So here's the problem:
% DMLC_LOG_STACK_TRACE_DEPTH=150 MXNET_ENGINE_TYPE=NaiveEngine lldb python3.7 -- test_18939.py
(lldb) target create "python3.7"
Current executable set to 'python3.7' (x86_64).
(lldb) settings set -- target.run-args  "test_18939.py"
(lldb) run
Process 72921 launched: '/usr/local/bin/python3.7' (x86_64)
Process 72921 stopped
* thread #2, stop reason = exec
    frame #0: 0x0000000100006000 dyld`_dyld_start
dyld`_dyld_start:
->  0x100006000 <+0>: popq   %rdi
    0x100006001 <+1>: pushq  $0x0
    0x100006003 <+3>: movq   %rsp, %rbp
    0x100006006 <+6>: andq   $-0x10, %rsp
(lldb) cont
Process 72921 resuming
[23:48:10] ../src/engine/engine.cc:55: MXNet start using engine: NaiveEngine
Process 72921 stopped
* thread #2, queue = 'com.apple.main-thread', stop reason = EXC_ARITHMETIC (code=EXC_I386_DIV, subcode=0x0)
    frame #0: 0x0000000117211d6d libmxnet.dylib`mxnet::op::SequenceMaskOp<mshadow::cpu, float, float>::Forward(this=0x00000001280dd950, ctx=0x00007ffeefbfc2b0, in_data=0x00000001280f5ae8, req=0x00007ffeefbfc310, out_data=0x00000001280f5b30, aux_args=0x00000001280f5b18) at sequence_mask-inl.h:100
   97  	    auto d0 = in_data[seq_mask::kData].size(0);
   98  	    auto d1 = in_data[seq_mask::kData].size(1);
   99  	    auto dsize = in_data[seq_mask::kData].Size();
-> 100 	    auto rest_size = dsize / (d0 * d1);
   101
   102 	    Shape<3> s3 = Shape3(d0, d1, rest_size);
   103 	    Tensor<xpu, 3, DType> data =

https://github.com/apache/incubator-mxnet/blob/9bdd4d6347c284770ee5bfe5ae98f1dabc283829/src/operator/sequence_mask-inl.h#L100
The code needs to guard against zero-size array for right operand of /, and we should add a smoke test to guard against such problem in this op, similar to https://github.com/apache/incubator-mxnet/pull/18972/files

authors: leeyeetonn szha

repo: apache/incubator-mxnet


text:
[1.x] website CI fails with 'module 'jinja2' has no attribute 'contextfilter''

Description
website CI fails with 'module 'jinja2' has no attribute 'contextfilter'':

Fixed.


text:
Split an existing symbol graph into 2 symbols

Say we have a symbol like AlexNet model, which outputs are
['data',
 'conv1_weight',
 'conv1_bias',
 'conv1_output',
 'activation0_output',
 'lrn0_output',
 'pooling0_output',
 'conv2_weight',
 'conv2_bias',
 'conv2_output',
 'activation1_output',
 'lrn1_output',
 'pooling1_output',
 'conv3_weight',
 'conv3_bias',
 'conv3_output',
 'activation2_output',
 'conv4_weight',
 'conv4_bias',
 'conv4_output',
 'activation3_output',
 'conv5_weight',
 'conv5_bias',
 'conv5_output',
 'activation4_output',
 'pooling2_output',
 'flatten0_output',
 'fc1_weight',
 'fc1_bias',
 'fc1_output',
 'activation5_output',
 'dropout0_output',
 'fc2_weight',
 'fc2_bias',
 'fc2_output',
 'activation6_output',
 'dropout1_output',
 'fc3_weight',
 'fc3_bias',
 'fc3_output',
 'softmax_label',
 'softmax_output']

Now I just want to split this symbol into 2 symbols, for example, the first symbol "bottom" includes nodes from data to flatten0_output, and the second symbol "top" ONLY includes nodes from flatten0_output to softmax_output. The former can be achieved by flatten = alexnet.get_internals()['flatten0_output'], but after getting node flatten0_output , I cannot cut the connection between flatten0_output and previous nodes, i.e the second symbol "top" is exactly the original symbol.
Splitting a symbol into 2 symbols is for model-parallelism usage, in this way we can put the first symbol "bottom" into a mxnet.mod.Module using GPU 1 and the second symbol "top"  into another mxnet.mod.Module using GPU2. This could be a method to implement model-parallelism for an existing symbol, but if you have another option, please let me know.
Thanks!

to_json and load_json

This issue is closed due to lack of activity in the last 90 days. Feel free to ping me to reopen if this is still an active issue. Thanks!

authors: bearsroom piiswrong szha


text:
[WIP] Add NEWS and README

Description
Added NEWS and README for 1.2.0 release.
@mbaijal Let me know on the decision with legal-discuss@ so we can update the release notes with known issues for the license.
Please see previews here: https://github.com/anirudh2290/mxnet/blob/v1.2.0/NEWS.md and https://github.com/anirudh2290/mxnet/blob/v1.2.0/README.md

@nswamy @spidyDev @anirudhacharya @reminisce @zheng-da @rahul003 @cjolivier01 @eric-haibin-lin @aaronmarkham @piiswrong Can you please help review ?

@KellenSunderland @larroy shall we mention that there's now a new way to build ARM and Jetson and that we output wheels?

@eric-haibin-lin @piiswrong is this good to merge ?

@anirudh2290 Would it be possible to format the PR a bit before merging.  I think it would make sense to either have 1 PR with two commits (merged without squashing), or two PRs (one for news, one with a fix).

I'd be in favour of the second approach. The news get squashed and merged and the backports get rebased and merged.

@KellenSunderland You raised a good point! Thanks. I have squashed all the README related changes into a single commit. I think this PR should now be merged without squashing.

Moved this back to WIP since need to add some more PRs for the release: #10477 #10415 #10565

All required commits have been cherry-picked. This PR is ready to be merged (without squashing).

authors: KellenSunderland anirudh2290 marcoabreu


text:
CSRNDArray from/to scipy csr_matrix; fix rand_shape_nd

Added preliminary/inefficient support so that user can

construct CSRNDArray from scipy csr_matrix.
convert CSRNDArray to scipy csr_matrix
throw exception when setters for indices/data/indptr are called.

Also fixed rand_shape_nd so that it returns tuple (consistent w/ rand_shape_2d)
TODO in next PR:

move copy logic to cpp backend to reduce copy/blocking.

@reminisce @anirudh2290

Done

Added better err msg for mx.nd.sparse.array.


text:
debug: nuttx-gdbinit not producing backtraces on armv7e-m

When I try to dump the threads with nxthread_bt_all nothing is shown and then it crashes the target. I am using this with STM32H7 and the ARM Embedded 2020-q2-update toolchain:
GNU gdb (GNU Arm Embedded Toolchain 9-2020-q2-update) 8.3.1.20191211-git

nx_start () at init/nx_start.c:832
832	      up_idle();
(gdb) bt
#0  nx_start () at init/nx_start.c:832
#1  0x080003ce in __start () at chip/stm32_start.c:426
(gdb) nxthread
target examined
_target_arch.name=armv7e-m
$_target_has_fpu : 0
$_target_has_smp : 0
saved current_tcb (pid=0)
Missing argument 0 in user function.
(gdb) nxthread_all_bt
* 0 Thread 0x24000504  (Name: Idle Task, State: Running, Priority: 0) 0x80025c6 in nx_start()
#0  nx_start () at init/nx_start.c:832
#1  0x080003ce in __start () at chip/stm32_start.c:426
  1 Thread 0x38000490  (Name: hpwork, State: Waiting,Signal, Priority: 224) 0x0 in No()
#0  0x00000000 in ?? ()
  2 Thread 0x38001130  (Name: lpwork, State: Waiting,Signal, Priority: 100) 0x0 in No()
#0  0x00000000 in ?? ()
  3 Thread 0x38001dd0  (Name: init, State: Waiting,Semaphore, Priority: 100) 0x0 in No()
#0  0x00000000 in ?? ()
  4 Thread 0x38002b60  (Name: usbhost, State: Waiting,Semaphore, Priority: 100) 0x0 in No()
#0  0x00000000 in ?? ()
  5 Thread 0x38003420  (Name: Telnet daemon, State: Waiting,Semaphore, Priority: 100) 0x0 in No()
#0  0x00000000 in ?? ()

@masayuki2009 any idea what is going on here? The Missing argument 0 in user function. does not look good.

@masayuki2009 any idea what is going on here? The Missing argument 0 in user function. does not look good.

What happens if you call info_nxthread before calling nxthread ?

Still looks like it is not grabbing the stackframes correctly:
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
stm32_phyread (phydevaddr=0, phyregaddr=1, value=0x38002866) at chip/stm32_ethernet.c:3228
3228	      if ((stm32_getreg(STM32_ETH_MACMDIOAR) & ETH_MACMDIOAR_MB) == 0)
(gdb) info_nxthreads
  0 Thread 0x240004a4  (Name: Idle Task, State: Ready, Priority: 0) 0x0 in No()
  1 Thread 0x38000490  (Name: hpwork, State: Waiting,Signal, Priority: 224) 0x0 in No()
  2 Thread 0x38001130  (Name: lpwork, State: Ready, Priority: 100) 0x0 in No()
  3 Thread 0x38001dd0  (Name: init, State: Running, Priority: 100) 0x0 in No()
  4 Thread 0x38002b60  (Name: usbhost, State: Ready, Priority: 100) 0x0 in No()
  5 Thread 0x38003420  (Name: Telnet daemon, State: Ready, Priority: 100) 0x0 in No()
(gdb) bt
#0  stm32_phyread (phydevaddr=0, phyregaddr=1, value=0x38002866) at chip/stm32_ethernet.c:3228
#1  0x0800a264 in stm32_phyinit (priv=0x240068e0 <g_stm32ethmac>) at chip/stm32_ethernet.c:3517
#2  0x0800a80c in stm32_ethconfig (priv=0x240068e0 <g_stm32ethmac>) at chip/stm32_ethernet.c:4312
#3  0x08009c90 in stm32_ifup (dev=0x2400693c <g_stm32ethmac+92>) at chip/stm32_ethernet.c:2580
#4  0x08032034 in netdev_ifup (dev=0x2400693c <g_stm32ethmac+92>) at netdev/netdev_ioctl.c:1833
#5  0x08031bce in netdev_ifr_ioctl (psock=0x38002164, cmd=1818, req=0x3800297c) at netdev/netdev_ioctl.c:907
#6  0x08031f6e in psock_vioctl (psock=0x38002164, cmd=1818, ap=...) at netdev/netdev_ioctl.c:1681
#7  0x08032006 in netdev_vioctl (sockfd=8, cmd=1818, ap=...) at netdev/netdev_ioctl.c:1810
#8  0x0801edfa in nx_vioctl (fd=8, req=1818, ap=...) at vfs/fs_ioctl.c:151
#9  0x0801ef38 in ioctl (fd=8, req=1818) at vfs/fs_ioctl.c:259
#10 0x0800add4 in netlib_ifup (ifname=0x8038ee4 "eth0") at netlib_setifstatus.c:97
#11 0x0800acb8 in netinit_net_bringup () at netinit.c:603
#12 0x0800ad60 in netinit_configure () at netinit.c:701
#13 0x0800ad6c in netinit_bringup () at netinit.c:1045
#14 0x0800aabe in nsh_consolemain (argc=1, argv=0x38002a58) at nsh_consolemain.c:104
#15 0x0800aa6c in nsh_main (argc=1, argv=0x38002a58) at nsh_main.c:168
#16 0x08006a00 in nxtask_startup (entrypt=0x800aa0d <nsh_main>, argc=1, argv=0x38002a58) at sched/task_startup.c:165
#17 0x08002b36 in nxtask_start () at task/task_start.c:144
#18 0x00000000 in ?? ()
(gdb)

Still looks like it is not grabbing the stackframes correctly:

Hmm, it's strange...
$ arm-none-eabi-gdb --version
GNU gdb (GNU Arm Embedded Toolchain 9-2020-q2-update) 8.3.1.20191211-git
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

I've just tried stm32f4discovery:rndis
(gdb) target extended-remote:3333
Remote debugging using :3333
up_idle () at chip/stm32_idle.c:208
208       END_IDLE();
(gdb) info_nxthreads
target examined
_target_arch.name=armv7e-m
$_target_has_fpu : 0
$_target_has_smp : 0
saved current_tcb (pid=0)
* 0 Thread 0x20000308  (Name: Idle Task, State: Running, Priority: 0) 0x80051c6 in up_idle()
  1 Thread 0x20004a80  (Name: lpwork, State: Waiting,Signal, Priority: 100) 0x8005778 in arm_switchcontext()
  2 Thread 0x200056c0  (Name: init, State: Waiting,Semaphore, Priority: 100) 0x8005778 in arm_switchcontext()
  3 Thread 0x20007820  (Name: Telnet daemon, State: Waiting,Semaphore, Priority: 100) 0x8005778 in arm_switchcontext()

Still looks like it is not grabbing the stackframes correctly:

@btashton
Which board and configuration are you using now?

I am using nucleo-h743zi:otg_fs_host which was just merged in, the only change I made is setting the thread name limit to 31 instead of 0.  When I used stm32f429i-disco:nsh everything was fine.

@btashton
Perhaps, I found the root cause.
Actually, nucleo-.h743zi:otg_fs_host uses FPU but it was not detected by the nuttx-gdbinit script.
Currently, it just checks the symbol named fpuconfig but stm32_fpuconfig was used instead.
For the interim solution, could you try it by replacing fpuconfig with stm32_fpuconfig?

@masayuki2009 I had to make an additional change to use lookup_symbol instead of lookup_global_symbol which returns a tuple with the first value as the symbol (or None).
python if (type(gdb.lookup_symbol("stm32_fpuconfig")[0]) is gdb.Symbol) : \
There are 22 different definitions of this {}_fpuconfig symbol so I am not quite sure what the change should be here.

There are 22 different definitions of this {}_fpuconfig symbol so I am not quite sure what the change should be here.

@btashton
I came up with a new idea to check if the target uses FPU.
It just checks the size of tcb.xcp.regs[].
I will try this idea later.

repo: apache/incubator-nuttx


text:
Documentation: Fix typos

Summary
Documentation/reference/os/wqueue.rst:

Fix spelling, grammar, and typos.
Improve wording in a few areas.
Re-wrap the paragraphs that were modified.

Impact
Improves documentation.
Testing
None.

I have additional fixes for comments in the source files in [1], but I'm waiting on PR #4035 since they could introduce unnecessary conflicts.
[1] https://github.com/hartmannathan/incubator-nuttx/tree/work-queue-typos


text:
API GW should adhere to "Accept" header dictates as much as possible

Here is one specific example, although other scenarios likely have the same issue.

Create an API that is configured with "Require applications to authenticate via API key" enabled.
Invoke the API using curl, but specifying the "Accept" media type as something besides application/json
A 401 is returns with a JSON body and "Content-Type" value of application/json

>curl https://service.us.apiconnect.ibmcloud.com/gws/apigateway/api/API-UNIQUE-ID/API-PATH -H "Accept: application/xml"

{"status":401,"message":"Error: Unauthorized"}

So, in your scenario, let's say you pass an Accept header of text/html. If an API key is required, the gateway responds with 401, but what Content-Type is the response body?

I guess the server might be able to support a few "standard" media types and respond with the one that matches the request Accept header.  If a server is unable to adhere to the requests Accept set of media types, should it return a 406?

I think 406 isn't terribly useful, since we don't know exactly what types are allowable and it would be preferable to tell the caller why the call failed even if it's in a format they didn't ask for. Since we'd set the Content-Type to whatever is being returned (which would be application/json by default), then the caller should be prepared to handle that case.

authors: mdeuser mhamann

repo: apache/incubator-openwhisk-apigateway


text:
Unit test of TestParseManifestForSingleLineParams failing

=== RUN   TestParseManifestForSingleLineParams
2017/08/28 23:03:51 error happened during unmarshal :yaml: unmarshal errors:
line 25: cannot unmarshal !!str into parsers.Parameter   line 26: cannot unmarshal !!str into parsers.Parameter
2017/08/28 23:03:51 yaml: unmarshal errors:
line 25: cannot unmarshal !!str into parsers.Parameter   line 26: cannot unmarshal !!str into parsers.Parameter
2017/08/28 23:03:51 Error happened during execution, please type 'wskdeploy -h' for help messages.
FAIL	github.com/apache/incubator-openwhisk-wskdeploy/parsers	0.015s

@daisy-ycguo @lzbj Did you see this issue?

@houshengbo it didn't have any issues in my laptop.
$ make test | grep 'TestParseManifestForSingleLineParams'
=== RUN   TestParseManifestForSingleLineParams
--- PASS: TestParseManifestForSingleLineParams (0.00s)

I got the same issue.
Installing dependencies
go get -d -t ./...
Testing
go test ./... -tags=unit
?   	github.com/apache/incubator-openwhisk-wskdeploy	[no test files]
ok  	github.com/apache/incubator-openwhisk-wskdeploy/cmd	1.254s
ok  	github.com/apache/incubator-openwhisk-wskdeploy/deployers	0.013s
Error:
Filepath: "  "
Package:  {   [] map[]     map[] map[] map[] map[] map[] map[] map[]}
PackageName: "  "
Number of Actions:  0
Actions:  map[]
2017/08/29 16:04:25 error happened during unmarshal :yaml: unmarshal errors:
line 25: cannot unmarshal !!str into parsers.Parameter   line 26: cannot unmarshal !!str into parsers.Parameter
2017/08/29 16:04:25 yaml: unmarshal errors:
line 25: cannot unmarshal !!str into parsers.Parameter   line 26: cannot unmarshal !!str into parsers.Parameter
2017/08/29 16:04:25 Error happened during execution, please type 'wskdeploy -h' for help messages.
FAIL	github.com/apache/incubator-openwhisk-wskdeploy/parsers	0.014s
?   	github.com/apache/incubator-openwhisk-wskdeploy/tests/src/integration/common	[no test files]
ok  	github.com/apache/incubator-openwhisk-wskdeploy/utils	0.054s
?   	github.com/apache/incubator-openwhisk-wskdeploy/wski18n	[no test files]
make: *** [test] Error 1

Raise the level to get some attention, since it will slow down the development.

@lzbj Strangely travis does not report this error, but it really exists when I run make test locally. Is there something wrong with manifest_validate_singleline_params.yaml?

@houshengbo , correct, so I doubt we hit a yaml lib bug or even a golang bug perhaps, need to dig deeper in this.

@lzbj Line 25 and Line 26 put '' and "" as the values. It seems the parser does not read them correctly.

@lzbj @houshengbo interesting issue. I run under ubuntu and it works fine.

@daisy-ycguo It is valid in mac. I tried ubuntu as well, there is no issue for linux.

@pritidesai  is our UT expert, please could you help take a look @pritidesai

@lzbj thanks for recognition :) you will be surprised to know I have no QA background 👎
didn't find any issues while running that test on mac:
go test ./parsers/ -tags=unit -run=TestParseManifestForSingleLineParams -v
=== RUN   TestParseManifestForSingleLineParams
--- PASS: TestParseManifestForSingleLineParams (0.00s)
PASS
ok      github.com/apache/incubator-openwhisk-wskdeploy/parsers 0.021s

thank you @pritidesai , it seems my env problems.
I run the same command, got the output.
go test ./parsers/ -tags=unit -run=TestParseManifestForSingleLineParams -v
=== RUN   TestParseManifestForSingleLineParams
2017/08/30 17:14:34 error happened during unmarshal :yaml: unmarshal errors:
line 25: cannot unmarshal !!str into parsers.Parameter   line 26: cannot unmarshal !!str into parsers.Parameter
2017/08/30 17:14:34 yaml: unmarshal errors:
line 25: cannot unmarshal !!str into parsers.Parameter   line 26: cannot unmarshal !!str into parsers.Parameter
2017/08/30 17:14:34 Error happened during execution, please type 'wskdeploy -h' for help messages.
exit status 1
FAIL    github.com/apache/incubator-openwhisk-wskdeploy/parsers 0.011s

I used go1.9

@lzbj can you please check if this issue is resolved for you?

@pritidesai , thank you, yes, it's fixed after I upgrade the yaml lib version, so close it.

authors: daisy-ycguo houshengbo lzbj pritidesai

repo: apache/incubator-openwhisk-wskdeploy


text:
[flink] Support remove-orphan-files action

Purpose

Linked issue: close #1571

Tests

RemoveOrphanFilesActionITCase
API and Format

Documentation
Understand Files - Remove Orphan Files

Some codes have referenced #2223

repo: apache/incubator-paimon


text:
Clean up unused and remove redundant metrics

This PR cleans up all the unused metrics and removes redundant metrics.
e.g. PINOT_CONTROLLER_PARTITION_LEADER can be removed because it can be replaced by getting the value of CONTROLLER_LEADER_PARTITION_COUNT plus calling the controller leader API.
Other metrics like CONTROLLER_INTERNAL_ERROR even get used in the code at all.

Codecov Report

Merging #4710 into master will increase coverage by 6.95%.
The diff coverage is 0%.

@@             Coverage Diff              @@
##             master    #4710      +/-   ##
============================================
+ Coverage     57.86%   64.82%   +6.95%
  Complexity       16       16
============================================
  Files          1213     1061     -152
  Lines         65116    56607    -8509
  Branches       9483     8414    -1069
============================================
- Hits          37682    36697     -985
+ Misses        24581    17188    -7393
+ Partials       2853     2722     -131

Impacted Files
Coverage Δ
Complexity Δ

...a/org/apache/pinot/common/metrics/ServerGauge.java
95.45% <ø> (-0.55%)
0 <0> (ø)

...apache/pinot/controller/LeadControllerManager.java
74.48% <ø> (-0.52%)
0 <0> (ø)

...g/apache/pinot/common/metrics/ControllerGauge.java
96.15% <ø> (-0.4%)
0 <0> (ø)

...a/org/apache/pinot/common/metrics/BrokerMeter.java
100% <ø> (ø)
0 <0> (ø)
:arrow_down:

...g/apache/pinot/common/metrics/ControllerMeter.java
100% <ø> (ø)
0 <0> (ø)
:arrow_down:

...pinot/broker/api/resources/PinotClientRequest.java
23.8% <0%> (-19.05%)
0 <0> (ø)

...er/validation/BrokerResourceValidationManager.java
25% <0%> (-56.25%)
0% <0%> (ø)

.../org/apache/pinot/client/PinotClientException.java
33.33% <0%> (-33.34%)
0% <0%> (ø)

...he/pinot/core/query/pruner/ValidSegmentPruner.java
57.14% <0%> (-28.58%)
0% <0%> (ø)

...egation/function/customobject/MinMaxRangePair.java
75.86% <0%> (-24.14%)
0% <0%> (ø)

... and 184 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4f997af...1b4f8c2. Read the comment docs.

authors: codecov-io jackjlli

repo: apache/incubator-pinot


text:
JSON column datatype support.

Description
This PR adds support for JSON column type. JSON column type can contain only valid JSONObject. JSON column type supports conversion to and from STRING.
A new property (storageFormat) was added to column metadata to record the current storage format of any given column type. This was done to allow for backward compatibility and segment data migration if the storage format of a column type is changed in future. By default, the storage format of all column types is set to original in segment metadata.properties file (column.jsoncolumn.storageFormat = original).
Columns of type JSON can be indexed using JsonIndex. Queries using JSON_EXTRACT_SCALAR and JSON_MATCH functions can run against a JSON column type. Test cases in JsonDatatypeTest are used to validate functionality.
Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)

[ ] Yes (Please label as backward-incompat, and complete the section below on Release Notes)

Does this PR fix a zero-downtime upgrade introduced earlier?

[ ] Yes (Please label this as backward-incompat, and complete the section below on Release Notes)

Does this PR otherwise need attention when creating release notes? Things to consider:

New configuration options
Deprecation of configurations
Signature changes to public methods/interfaces
New plugins added or old plugins removed

[ ] Yes (Please label this PR as release-notes and complete the section on Release Notes)

Release Notes

Documentation

Codecov Report

Merging #6878 (4bfae70) into master (1c09b78) will decrease coverage by 0.05%.
The diff coverage is 28.40%.

@@             Coverage Diff              @@
##             master    #6878      +/-   ##
============================================
- Coverage     65.48%   65.43%   -0.06%
  Complexity       12       12
============================================
  Files          1421     1421
  Lines         69980    70042      +62
  Branches      10112    10115       +3
============================================
+ Hits          45825    45829       +4
- Misses        20874    20926      +52
- Partials       3281     3287       +6

Flag
Coverage Δ
Complexity Δ

unittests
65.43% <28.40%> (-0.06%)
12.00 <0.00> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ
Complexity Δ

...rg/apache/pinot/common/function/FunctionUtils.java
97.72% <ø> (ø)
0.00 <0.00> (ø)

...java/org/apache/pinot/core/common/DataFetcher.java
89.95% <0.00%> (ø)
0.00 <0.00> (ø)

...ter/predicate/EqualsPredicateEvaluatorFactory.java
82.53% <ø> (ø)
0.00 <0.00> (ø)

.../filter/predicate/InPredicateEvaluatorFactory.java
71.28% <ø> (ø)
0.00 <0.00> (ø)

.../predicate/NotEqualsPredicateEvaluatorFactory.java
59.74% <ø> (ø)
0.00 <0.00> (ø)

...lter/predicate/NotInPredicateEvaluatorFactory.java
66.36% <ø> (ø)
0.00 <0.00> (ø)

...lter/predicate/RangePredicateEvaluatorFactory.java
68.07% <ø> (ø)
0.00 <0.00> (ø)

...ator/transform/function/CastTransformFunction.java
54.90% <0.00%> (-3.44%)
0.00 <0.00> (ø)

...e/pinot/core/query/reduce/HavingFilterHandler.java
89.13% <ø> (ø)
0.00 <0.00> (ø)

...egment/local/segment/creator/impl/V1Constants.java
12.50% <ø> (ø)
0.00 <0.00> (ø)

... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1c09b78...4bfae70. Read the comment docs.

authors: amrishlal codecov-commenter


text:
[DOCS]Update README.md

Purpose of this pull request

Check list

[ ] Code changed are covered with tests, or it does not need tests for reason:
[ ] Change does not need document change, or I will submit document change to https://github.com/apache/incubator-seatunnel-website later

Hi @CalvinKirs @nielifeng , let's remove these branches once the PR is merged, so that we don't get a bunch of unused branches in the future. I just removed all these branches that are merged.

repo: apache/incubator-seatunnel


text:
SCB-949   Configurable aggregate mode

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Format the pull request title like [SCB-XXX] Fixes bug in ApproximateQuantiles, where you replace SCB-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Run go build go test go fmt go vet to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Codecov Report

Merging #457 into master will decrease coverage by 0.04%.
The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master     #457      +/-   ##
==========================================
- Coverage   60.19%   60.15%   -0.05%
==========================================
  Files         156      156
  Lines       13407    13411       +4
==========================================
- Hits         8070     8067       -3
- Misses       4759     4763       +4
- Partials      578      581       +3

Impacted Files
Coverage Δ

server/plugin/pkg/registry/etcd/etcd.go
86.3% <66.66%> (-0.47%)
:arrow_down:

pkg/etcdsync/mutex.go
61.29% <0%> (-5.38%)
:arrow_down:

server/govern/service.go
73.33% <0%> (-0.34%)
:arrow_down:

server/broker/util.go
53.47% <0%> (+0.26%)
:arrow_up:

server/plugin/pkg/registry/etcd/tracing.go
81.81% <0%> (+9.09%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5fac325...3bb8742. Read the comment docs.

Coverage decreased (-0.02%) to 62.521% when pulling 3bb874272cd4123d6bd6b27a7ed11fed57fa053f on little-cui:master into 5fac325c02bc2c1c5e7a021dfb6ca94497a2475a on apache:master.

authors: codecov-io coveralls little-cui


text:
Standard List usage

According to standard, use LinkedList in priority, not use Lists.newArrayList() of Guava.
Changes proposed in this pull request:

Modify Lists.newArrayList() to new LinkedList<>

Pull Request Test Coverage Report for Build 9394

1 of 3 (33.33%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 64.577%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

shardingsphere-sql-parser/shardingsphere-sql-parser-mysql/src/main/java/org/apache/shardingsphere/sql/parser/visitor/MySQLDDLVisitor.java
0
2
0.0%

Totals

Change from base Build 9392:
0.0%

Covered Lines:
10960

Relevant Lines:
16972

💛  - Coveralls

authors: SteNicholas coveralls

repo: apache/incubator-shardingsphere


text:
[Question] performance problem in MotanProxyService

Question
In org.apache.shenyu.plugin.motan.proxy.MotanProxyService#genericInvoker, we have a performance problem :
    public Mono<Object> genericInvoker(final String body, final MetaData metaData, final ServerWebExchange exchange) throws ShenyuException {
        Map<String, Map<String, String>> rpcContext = exchange.getAttribute(Constants.GENERAL_CONTEXT);
        Optional.ofNullable(rpcContext).map(context -> context.get(PluginEnum.MOTAN.getName())).ifPresent(context -> {
            context.forEach((k, v) -> RpcContext.getContext().setRpcAttachment(k, v));
        });
        RefererConfig<CommonHandler> reference = ApplicationConfigCache.getInstance().get(metaData.getPath());
        if (Objects.isNull(reference) || StringUtils.isEmpty(reference.getServiceInterface())) {
            ApplicationConfigCache.getInstance().invalidate(metaData.getPath());
            reference = ApplicationConfigCache.getInstance().initRef(metaData);
        }
        CommonHandler commonHandler = reference.getRef();
        ApplicationConfigCache.MotanParamInfo motanParamInfo = ApplicationConfigCache.PARAM_MAP.get(metaData.getMethodName());
        Object[] params;
        if (Objects.isNull(motanParamInfo)) {
            params = new Object[0];
        } else {
            int num = motanParamInfo.getParamTypes().length;
            params = new Object[num];
            Map<String, Object> bodyMap = GsonUtils.getInstance().convertToMap(body);
            for (int i = 0; i < num; i++) {
                params[i] = bodyMap.get(motanParamInfo.getParamNames()[i]).toString();
            }
        }
        ResponseFuture responseFuture;
        //CHECKSTYLE:OFF IllegalCatch
        try {
            responseFuture = (ResponseFuture) commonHandler.asyncCall(metaData.getMethodName(), params, Object.class);
        } catch (Throwable e) {
            LOG.error("Exception caught in MotanProxyService#genericInvoker.", e);
            return null;
        }
        //CHECKSTYLE:ON IllegalCatch
        ResponseFuture finalResponseFuture = responseFuture;
//fixme If there are many time-consuming tasks, the ForkJoinPool with only a few threads will become a performance bottleneck
        CompletableFuture<Object> future = CompletableFuture.supplyAsync(finalResponseFuture::getValue);
        return Mono.fromFuture(future.thenApply(ret -> {
            if (Objects.isNull(ret)) {
                ret = Constants.MOTAN_RPC_RESULT_EMPTY;
            }
            exchange.getAttributes().put(Constants.RPC_RESULT, ret);
            exchange.getAttributes().put(Constants.CLIENT_RESPONSE_RESULT_TYPE, ResultEnum.SUCCESS.getName());
            return ret;
        })).onErrorMap(ShenyuException::new);
    }

so I'm going to adjust the thread pool to cached:
    private final ExecutorService threadPool = Executors.newCachedThreadPool(ShenyuThreadFactory.create("shenyu-motan", true));

    public Mono<Object> genericInvoker(final String body, final MetaData metaData, final ServerWebExchange exchange) throws ShenyuException {
        //......
        CompletableFuture<Object> future = CompletableFuture.supplyAsync(finalResponseFuture::getValue, threadPool);
        return Mono.fromFuture(future.thenApply(ret -> {
            if (Objects.isNull(ret)) {
                ret = Constants.MOTAN_RPC_RESULT_EMPTY;
            }
            exchange.getAttributes().put(Constants.RPC_RESULT, ret);
            exchange.getAttributes().put(Constants.CLIENT_RESPONSE_RESULT_TYPE, ResultEnum.SUCCESS.getName());
            return ret;
        })).onErrorMap(ShenyuException::new);
    }

see https://github.com/apache/incubator-shenyu/pull/3172

repo: apache/incubator-shenyu


text:
[Bug]  The K8S Application Mode deployment job has been in the INITIALIZING state since it was started

Search before asking

[X] I had searched in the issues and found no similar issues.

Java Version
No response
Scala Version
2.12.x
StreamPark Version
2.0.0
Flink Version
1.16.1
deploy mode
None
What happened
Deploying FlinkSQL jobs in the K8S Application Mode mode sometimes continues in the INITIALIZING state, but in fact it is already in the RUNNING state on the K8S. I checked the network and K8S cluster resources and they all look fine.
Later, I debugged the code and found that a cacheis maintained in the JobStatusCacheclass for job status caching, but it does not set a cache timeout.
Next, FlinkJobStatusWatcher will continuously obtain job status and push status update events through eventbus.
As shown in the following code:
  if (latest == null || latest.jobState != jobState.jobState || latest.jobId != jobState.jobId) {
    // put job status to cache
    watchController.jobStatuses.put(trackId, jobState)
    // set jobId to trackIds
    watchController.trackIds.update(trackId)
    eventBus.postSync(FlinkJobStatusChangeEvent(trackId, jobState))
  }

When the job starts normally, the cachewill be updated to RUNNING, and the monitoring will continue to obtain the RUNNING status. In the case that the cache will not be invalidated, if other methods directly update the database state to the INITIALIZING state without clearing the cache, then the job will continue to be in the INITIALIZING state and will not automatically change to the RUNNING state.
My modification suggestion: Modify the cache timeout in JobStatusCache to 30 seconds.
 private[this] lazy val cache: Cache[CacheKey, JobStatusCV] = Caffeine.newBuilder.expireAfterWrite(30,TimeUnit.SECONDS).build()

Error Exception
No response
Screenshots

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!(您是否要贡献这个PR?)

Code of Conduct

[X] I agree to follow this project's Code of Conduct

@senlizishi Hi Bin, Due to historical reasons, there are some non-idempotent issues with the read/write maintenance of the StreamPark's cache for Flink job statuses. From the perspective of quick fixes, making the corresponding local cache quickly invalid is a good hot fix solution. I agree with your proposal.

authors: Al-assad senlizishi

repo: apache/incubator-streampark


text:
Set autorefresh timer for a specific chart, not the whole Dashboard

Hey,
I was wondering if this is possible, and if yes, how ? I got a lot of Dashboards with a lot of charts ( nearly 30+ per dashboard ), I just need a few of them to autorefresh every xx minutes. If you have an idea on how to do that, I'll take it !
NB : I can't move the charts to a special Dashboard in order to set the whole Dashboard to autorefresh itself.

I've submitted a solution for this - see the latest docs for refresh_immune_slices

authors: DylanCa RichRadics

repo: apache/incubator-superset


text:
AUTH_USER_REGISTRATION_ROLE for Multiple Roles

I see that multiple roles are supported when I look at Users when logged in an Admin account.
Given this, I am wondering if it's possible to configure the AUTH_USER_REGISTRATION_ROLE to assign multiple roles to the account upon registration (e.g. "Gamma","sql_lab"), as I haven't been able to figure it out.
I just want to confirm whether this is possible, otherwise I know a workaround would be to create my own single role with the appropriate permissions from the multiple roles.

It's not too hard to customize the authentication in FAB. Here's an example of what we do at Airbnb. You can create and assign roles when user first login.
https://gist.github.com/mistercrunch/6d31af4a11c47edcedc1ba6ceb5f5fab

Hi @behanjc have you figure it out if it is possible to configure multiple role ?

authors: behanjc enzo-dechaene mistercrunch


text:
Remove TOP 50000 Big Number chart

Expected results
what you expected to happen.
Superset query all records in table when creating the Big Number chart
Actual results
Number of records to return is 50000
Screenshots

How to reproduce the bug

Go to 'Add new chart'
Choose 'Big Number'
View query

Environment

superset version: 0.28.1
python version: 3.6
node.js version: 10.16.0
npm version: 6.9.0
database: SQL Server

Checklist
Make sure these boxes are checked before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[ ] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Oh does the top actually limits the number of rows that are scanned in SQL Server, making your COUNT == 50000?
The equivalent LIMIT in most other databases limits what is returned, not what is scanned.

Thanks for the fix

Out of curiosity I tested this on an old MSSQL 2008R2 instance, and it did not limit the number of scanned rows, only the result. @syazwan0913 what version of MSSQL are you on? And are you sure your table doesn't happen to have 50,000 rows?

Sounds like this shouldn't affect the query plan's or perf in any important way.

@villebro @mistercrunch Actually it's don't affect the query at all. It is just showing TOP 50000 eventhough the query is correct. So, everything is okay :)

authors: mistercrunch syazwan0913 villebro


text:
Adding a new viz line-bar chart

This pull request is a very simple and straightforward implementation of a line-bar chart. I recently added this in my project and thought it would be an good addition. Nothing fancy but leveraging the nvd3 charting functionality to accomplish it.

Codecov Report

Merging #5144 into master will decrease coverage by 0.32%.
The diff coverage is 17.02%.

@@            Coverage Diff             @@
##           master    #5144      +/-   ##
==========================================
- Coverage   77.48%   77.16%   -0.33%
==========================================
  Files          44       44
  Lines        8737     8784      +47
==========================================
+ Hits         6770     6778       +8
- Misses       1967     2006      +39

Impacted Files
Coverage Δ

superset/viz.py
79.43% <17.02%> (-1.95%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f102eab...590f35f. Read the comment docs.

@VinodLouis can you fix the conflicts? We would really like to use your visualization!

@SpyderRivera Thanks for your response I have resolved the conflicts and pushed the changes.

Would be great to see this integrated!

I'll ping the dev list about this.

@VinodLouis  Thanks for adding this new visualization type to superset!
As the community of superset is getting larger, we are aware of the increasing number of new visualization types added to Superset. It is great, however, it makes the visualization code hard to maintain and make the js bundle size larger. Due to this reason, the Superset community has proposed https://github.com/apache/incubator-superset/issues/5692, a js plugin system, to separate the visualization code from the superset core infra.  You can also check one example we are working on https://github.com/apache-superset/superset-ui/pull/50. It will be great if you can follow the same structure and help us to embrace the visualization plug-in system and any suggestion are welcome!

authors: SpyderRivera VinodLouis codecov-io conglei jeremyhiggs


text:
[superset-client][datasource editor] replace ajax with SupersetClient

This PR is one of several that implement the final step 4) discussed in #5772, to refactor just JS datasource editor, modal, and control ajax calls for easier review.

dashboard #5854
explore #5869
charts #5875
sql lab #5896
logging #6133
datasource editor (this PR)

Tested with unit tests + functionally (including forcing errors to see proper handling)
@kristw @mistercrunch @graceguo-supercat @michellethomas @conglei

Codecov Report

Merging #6134 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #6134   +/-   ##
=======================================
  Coverage   76.91%   76.91%
=======================================
  Files          47       47
  Lines        9362     9362
=======================================
  Hits         7201     7201
  Misses       2161     2161

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4e2341a...ca88cc6. Read the comment docs.

authors: codecov-io williaster


text:
Traffic Ops ORT can update zero size file

Fix the issue #TC-120.

Tested and works as designed.

repo: apache/incubator-trafficcontrol


text:
[Bug][Sort][Manager] please declare primary key for sink table when query contains update/delete record.

What happened
When creating a binlog type data source, an exception occurs when initializing sort: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: please declare primary key for sink table when query contains update/delete record.

What you expected to happen
no error
How to reproduce
When creating a binlog type data source, an exception occurs when initializing sort.
Environment
centos7
InLong version
1.2.0
InLong Component
InLong Manager, InLong Sort
Are you willing to submit PR?

[ ] Yes, I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hi @Greedyu, could you please resolve this issue?

@healchow  ok, i'll try it later

authors: Greedyu healchow

repo: apache/inlong


text:
KAFKA-12388: Share broker channel between alterIsrManager and lifecycleManager

More detailed description of your change
There are some BorkerToControllerChannerManager in BrokerServer and KafkaServer,
We are planning to consolidate into two channels eventually:

broker to controller channel
client to controller channel

Auto topic creation and forwarding fall into the 2nd category, while AlterIsr, lifecycleManager, and logDirEventManager(see KAFKA-9837) would be the 1st category.
#10135 is consolidating the 2nd category, this pr is trying to consolidate the 1st category.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@abbccdda @mumrah @hachikuji to have a look.😉

Thanks for looking at this. The issue that I see with sharing channels (really sockets) in this way is that we don't want the broker heartbeat to get delayed by something else. If the heartbeat gets delayed too long, the consequences could be really bad.
I think what we should do here is get rid of head-of-line blocking for Kafka RPCs, so that they can be done in parallel. Really, the main reason why we still have head-of-line blocking is because of the difficulty of moving the producer and consumer off of that model. So we could get rid of HOL blocking for all requests except produce and consume without too many changes to the code, I think.
I think we need to fix the HOL blocking problem before we start consolidating channels, to avoid regressions in functionality. cc @ijuma @hachikuji @abbccdda

authors: cmccabe dengziming

repo: apache/kafka


text:
KAFKA-14805 KRaft controller supports pre-migration mode

Based on work by @cmccabe #13003
Pre-Migration mode in QuorumController
This patch adds the concept of pre-migration mode to the KRaft controller. While in this mode, the controller will only allow certain write operations. The purpose of this is to disallow metadata changes when the controller is waiting for the ZK migration records to be committed.
The following ControllerWriteEvent operations are permitted in pre-migration mode

completeActivation
maybeFenceReplicas
writeNoOpRecord
processBrokerHeartbeat
registerBroker
unregisterBroker

Raft events and other controller events do not follow the same code path as ControllerWriteEvent, so they are not affected by this new behavior.
New metric from KIP-866
The kafka.controller:type=KafkaController,name=ZkMigrationState metric is now supported by the KRaft controller.
Upgrade after 3.4 ZK migration
In 3.4.0, we wrote a single ZkMigrationStateRecord(PRE_MIGRATION) at the beginning of the migration, but nothing after it completed. This means in later versions, we need to tolerate the stray record. This patch adds a tagged field PreMigrationSupported to ZkMigrationStateRecord which allows us to handle this scenario.
System tests
Two new system tests are included in this patch:
zookeeper_migration_test.py::TestMigration.test_pre_migration_mode_3_4 Starts a KRaft cluster in 3.4 with migrations disabled. It then performs a rolling upgrade to 3.5 with migrations enabled. The test ensures that a migration does not happen and checks for a log message

ZkMigrationState is NONE which means this cluster should not be migrated from ZooKeeper

zookeeper_migration_test.py::TestMigration.test_upgrade_after_3_4_migration Starts a ZK cluster and KRaft controller in 3.4 and does a migration. The KRaft controller is upgraded to 3.5 still with migrations enabled. The test ensures that the migration does not happen again and checks for the log message

Read a ZkMigrationState of PRE_MIGRATION from a ZK migration on Apache Kafka 3.4.

The release artifacts from 3.4.0 have been uploaded to Confluent's public S3 bucket in order to support these new tests.
TODO:

[x] Add system test for upgrading from older KRaft version to 3.4, ensure we write the correct ZkMigrationStateRecord
[ ] Additional unit tests for completion event

@mumrah thanks for this patch. pardon me that I don't dig in this patch before. Do you think the bug related to both #13405 and ZkMigrationIntegrationTest#testDualWrite is fixed by this patch?

@chia7712 it's quite possible that #13405 is another source of failures for ZkMigrationIntegrationTest#testDualWrite. This patch will fix the main source of failures that I've seen which is that some metadata (ProducerId block) is written before
the ZK migration completes. I was hoping to take a look at your PR this week, but I've not found the time yet. Hopefully I can look tomorrow.

@mumrah thanks for the response. I raised the question because 3.5.0 will get released on 4/26. The bugs happen when testing Zk migration so it would be nice to get fix into 3.5.0
Feel free to close #13405 if this PR can fix #13405 :)