text:
[SUPPORT] A flow task batch task cannot write to the same table at the same time(flink sql)

Tips before filing an issue

Have you gone through our FAQs?
yes

Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced
Now we use Flink sql+hudi+presto as the solution of streaming and batching. Hudi is used for storage, Flink sql is used for processing, and presto is used as the query engine. Considering the data volume, stream processing only deals with the data of the current day, but the history will change. At this time, we need to repair the historical data by batch processing in the early morning every day. When we tested and verified according to this expectation, we found that when stream processing is running, batch processing can no longer be run (at this time, stream processing will report an error). We cannot directly load data only through stream processing. The status of historical data will change (order status). Therefore, batch processing is required for regular data repair. The final expectation of this solution is the same processing engine, the same code and the same target table. It is not so much a problem as a solution. At present, there are few online introductions in this regard. No suitable method was found. So ask for help here.
To Reproduce
Steps to reproduce the behavior:
1.Start a stream processing task, write data to partition table A, and only write today's data.
2.Use the same finksql code to start in batch mode and write data to partition table A. Reload data for the past 30 days.
3.After the batch task is submitted successfully, when it starts to write data to the hudi table, the stream processing task starts to report an error and restarts continuously. The error message is as follows:
Exception：Ignoring the request to fail job because the job is already failing. The ignored failure cause is
org.apache.hudi.exception.HoodieException: Executor executes action [handle write metadata event for instant 20221125180423588] error
Expected behavior
A clear and concise description of what you expected to happen.
Environment Description

Hudi version :
0.13

Spark version :

Hive version :
2.1.1

Hadoop version :
3.0.0

Storage (HDFS/S3/GCS..) :
HDFS

Running on Docker? (yes/no) :
NO

Additional context
Add any other context about the problem here.
Stacktrace
Add the stacktrace of the error.
Exception：Ignoring the request to fail job because the job is already failing. The ignored failure cause is
org.apache.hudi.exception.HoodieException: Executor executes action [handle write metadata event for instant 20221125180423588] error

@zhaizhenhua Thanks for raising this issue.  Essentially you're asking for the multi-writer support.  Hudi on Flink hasn't supported multi-writer yet.  It's on Hudi's roadmap.  Stay tuned for our updates.

authors: yihua zhaizhenhua


text:
[HUDI-7438] Fix Azure CI report check with new issue comments

Change Logs
This PR fixes Azure CI report check with new issue comments.
Impact
Fixes bug and improves Azure CI check.
Risk level
none
Documentation Update
N/A
Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

New comment

test comment

source: gharchive/pull-request

repo: apache/hudi

license_type: permissive

license_source: github-api


text:
[HUDI-1103]Fix Delete data demo in Quick-Start Guide

Fix Delete data demo in Quick-Start Guide
Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request
(For example: This pull request adds quick-start document.)
Brief change log
(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@nsivabalan can u take a look when free?


text:
[HUDI-1936] Introduce a optional property for conditional upsert

Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request
If anyone wants to use custom upsert logic then they have to override the Latest avro payload class which is only possible in java or scala .
Python developers have no such option .
Will be introducing a new payload class and a new key which can work in java , scala and python
This class will be responsible for custom upsert logic and a new key hoodie.update.key which will accept the columns which only need to be updated
"hoodie.update.keys": "admission_date,name",  #comma seperated key
"hoodie.datasource.write.payload.class": "com.hudiUpsert.hudiCustomUpsert" #custom upsert key
so this will only update the column admission_date and name in the target table
Brief change log
(for example:)

added hudi-common/src/main/java/org/apache/hudi/common/model/OverwriteWithCustomAvroPayload.java

Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Manually verified the change by running a job locally.

Committer checklist

[x] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

Codecov Report

Merging #3035 (26dadb6) into master (974b476) will decrease coverage by 45.74%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master   #3035       +/-   ##
============================================
- Coverage     55.01%   9.27%   -45.75%
+ Complexity     3850      48     -3802
============================================
  Files           485      54      -431
  Lines         23467    2016    -21451
  Branches       2497     241     -2256
============================================
- Hits          12911     187    -12724
+ Misses         9405    1816     -7589
+ Partials       1151      13     -1138

Flag
Coverage Δ

hudicli
?

hudiclient
?

hudicommon
?

hudiflink
?

hudihadoopmr
?

hudisparkdatasource
?

hudisync
?

huditimelineservice
?

hudiutilities
9.27% <ø> (-61.56%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...va/org/apache/hudi/utilities/IdentitySplitter.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...va/org/apache/hudi/utilities/schema/SchemaSet.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...a/org/apache/hudi/utilities/sources/RowSource.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../org/apache/hudi/utilities/sources/AvroSource.java
0.00% <0.00%> (-100.00%)
:arrow_down:

.../org/apache/hudi/utilities/sources/JsonSource.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...rg/apache/hudi/utilities/sources/CsvDFSSource.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...g/apache/hudi/utilities/sources/JsonDFSSource.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...apache/hudi/utilities/sources/JsonKafkaSource.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...pache/hudi/utilities/sources/ParquetDFSSource.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...lities/schema/SchemaProviderWithPostProcessor.java
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 462 more

cc @vingov do you mind taking a review at this, given its a python benefiting change

In some sense, with the Spark SQL support now, python users can do custom merges? does that satisfy your requirements?

@fanaticjo : We landed a partial payload support via https://github.com/apache/hudi/pull/4676.
Let us know if we can close this patch or if its possible to enhance the 4676 or if this patch is addressing something different.

authors: codecov-commenter fanaticjo nsivabalan vinothchandar


text:
[MINOR] fix typo in IntervalTreeBasedGlobalIndexFileFilter

What is the purpose of the pull request
this pull request fix typo in IntervalTreeBasedGlobalIndexFileFilter
Brief change log
fix typo

fix typo in IntervalTreeBasedGlobalIndexFileFilter

Verify this pull request
This pull request is a trivial rework / code cleanup without any test coverage.
Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@nsivabalan can you help to run workflows?


text:
[HUDI-2957] Shade kryo jar for flink bundle jar

Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.

What is the purpose of the pull request
(For example: This pull request adds quick-start document.)
Brief change log
(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@hudi-bot run azure

@hudi-bot run azure


text:
[MINOR] Fixing build failure

What is the purpose of the pull request
Locally mvn package fails w/ latest master.
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  31.148 s
[INFO] Finished at: 2022-03-28T20:06:19-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project hudi-aws: Compilation failure: Compilation failure:
[ERROR] /Users/nsb/Documents/personal/projects/nov26/hudi/hudi-aws/src/main/java/org/apache/hudi/aws/sync/AwsGlueCatalogSyncTool.java:[27,15] package jdk.jfr does not exist
[ERROR] /Users/nsb/Documents/personal/projects/nov26/hudi/hudi-aws/src/main/java/org/apache/hudi/aws/sync/AwsGlueCatalogSyncTool.java:[41,2] cannot find symbol
[ERROR]   symbol: class Experimental
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hudi-aws

Brief change log
(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

already fixed in master


text:
[Typo][Hoodie Metadata]Fix a typo for parameter in HoodieMetadata.

Change Logs
Describe context and summary for this change. Highlight if any code was copied.
[Typo][Hoodie Metadata]Fix a typo for parameter in HoodieMetadata.
Impact
Describe any public API or user-facing feature change or any performance impact.
Risk level (write none, low medium or high below)
If medium or high, explain what verification was done to mitigate the risks.
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[x] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[x] CI passed

Closing this as the spelling is correct.

authors: FlechazoW yihua


text:
add an option to automatically use the table name in the path

Feature Request / Improvement
currently by default, pyiceberg write all table inside the same path, which is arguable a very bad idea as later, there is no way to provide a fine level security, the current solution today is to explicitly write the path when creating a new table, which I don't think it is a great idea, as the user (IMHO) should not deal with storage path.
it would be nice if pyiceberg has an option to automatically generate a path based on a table name, if I write schema.table,  the path will be base_location/schema/table.
to be clear, I am not asking to change the iceberg catalog spec, but a new friendly functionality when using pyiceberg.

+1 for sensible defaults.
in some scenarios, the warehouse location is set to base_warehouse_location/database_name/table_name
https://github.com/apache/iceberg-python/blob/583a7e97db28afa4259cdf504611845222338893/pyiceberg/catalog/__init__.py#L954
I wonder if we just missed using this in some situations.
@djouallah how is your project setup currently?

In case it's relevant, there is an issue re: adopting a pluggable LocationProvider interface akin to what exists in iceberg java sdk.  If it does turn out that we're currently handling the location inconsistently in pyiceberg, this could be a good opportunity to begin the process of putting this behind a single interface (pluggability and alternate implementations could come later).

@kevinjqliu https://github.com/djouallah/Fabric_Notebooks_Demo/blob/main/iceberg/Iceberg_azure.ipynb
if i remove the location, I get an error

it seems to be a bug with the sql catalog, as it works fine with Polaris, fwiw, this how I am using the sqlcatalog
catalog = SqlCatalog(
      "default",
      **{
          "uri"                : userdata.get("postgresql_db"),
          "adlfs.account-name" : userdata.get("account_name") ,
          "adlfs.account-key"  : userdata.get ("AZURE_STORAGE_ACCOUNT_KEY"),
          "adlfs.tenant-id"    : userdata.get("azure_storage_tenant_id"),
          "py-io-impl"         : "pyiceberg.io.fsspec.FsspecFileIO",
          "legacy-current-snapshot-id": True
      },
                        )

source: gharchive/issue

authors: corleyma djouallah kevinjqliu

repo: apache/iceberg-python


text:
IGNITE-21648 Fix NPEs in networking-related tests

https://issues.apache.org/jira/browse/IGNITE-21648

Closing as not relevant anymore (fixed in IGNITE-20296)


text:
[AIRFLOW-2403] Fix License Headers

Fix License Headers (vertica_to_mysql operator).
Make sure you have checked all steps below.
JIRA

[x] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-2403

Description

[x] Here are some details about my PR, including screenshots of any UI changes:
Fixes wrong Apache license header

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:
Documentation only

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[x] In case of new functionality, my PR adds documentation that describes how to use it.

When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

[x] Passes git diff upstream/master -u -- "*.py" | flake8 --diff

Codecov Report

Merging #3285 into master will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3285      +/-   ##
==========================================
+ Coverage   75.87%   75.89%   +0.02%
==========================================
  Files         197      197
  Lines       14702    14702
==========================================
+ Hits        11155    11158       +3
+ Misses       3547     3544       -3

Impacted Files
Coverage Δ

airflow/jobs.py
82.83% <0%> (+0.26%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a7e8f48...0ce6419. Read the comment docs.

repo: apache/incubator-airflow

license_source: bigquery


text:
Add initial scaffolding for default bucket (take 2)

@tgroh, could you please take a look?

Provide a hook for determining the project number associated with a bucket
Provide an API for looking up the project number from a project ID

Re-opened because the other PR wasn't clearing the extra commit history, despite the branch being correct.

Method renamed, all ready to go.

Merging

Thanks for fixing the git history.
Compared this with a copy I made in lukecwik/incubator-beam:finish-pr-1014 and it looked the same.
Merging

authors: lukecwik sammcveety

repo: apache/incubator-beam


text:
Add ElementByteSizeObserver default constructor

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[*] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[*] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[*] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[*] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

This enables constructing an observer without the Counter to simplify
testing and use with other ways of reporting byte size.
Once compatibility issues are resolved, the class will become abstract
with the Counter-specific functionality removed.

R: @tgroh (oneof @dhalperi @kennknowles)

LGTM

Minor comments, but LGTM. Please self-merge.

authors: bjchambers dhalperi tgroh


text:
[Feature][GitHub plugin] Support transformations from GitHub Environments

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Use case
I want to classify Deployments and Production Deployments based on the GitHub Environment name that the Job is referring to, instead of only being able to use the Job name
Description
No response
Related issues
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

What does  GitHub Environment name refer to ?

What does  GitHub Environment name refer to ?

Refer to the name that are highlighed in these printscreens below:

Below is the documentation about the environment field in the Actions definition (ref)

Interesting, never notice this before.
@Startrekzky is following up

Interesting, never notice this before. @Startrekzky is following up

It's basically the same as the environment setting in a BitBucket pipeline step. We can reuse this logic if the environment keys and values are collected.

This is a valid feature, we need to do it.

Hi @Feggah , we are working on collecting GitHub deployments in #5949 . This will also collect the environment of GitHub deployments.
Although I've read GitHub's doc of how deployments are configured, since we're not using GitHub environments, I'm curious about how you configure them in real life. Do you mind sharing an example of your GitHub workflow or job to conduct deployments? Thank you in advance.

Can't help much any longer by giving real-life examples @Startrekzky, I'm not in the same company that I was before.
But our use case was simple: we used environments to apply specific policies to it. For example, for production environments, we had a policy that two people had to approve before the deployment job was executed.
For DevLake specifically, using GitHub Environments would be more reliable than the job name, because we had control over the GitHub Environments and it had a pattern across the company, but for pipeline/job names, each repository had its own different pattern.

Hi @Feggah , DevLake is going to support this in the upcoming v0.20. Please see the pic below. Feel free to reopen the issue if it can not meet your needs.

authors: Feggah Startrekzky klesh

repo: apache/incubator-devlake


text:
Datax controll the num of split task have no effects

the channel setting of core and job both get the dataXParameters. i think this lead to the number of channel is usually 1.it does't have any effect of the split task.
`
/**
* build datax setting config
*
* @return datax setting config JSONObject
*/
private JSONObject buildDataxJobSettingJson() {
JSONObject speed = new JSONObject();
speed.put("channel", DATAX_CHANNEL_COUNT);
    if (dataXParameters.getJobSpeedByte() > 0) {
        speed.put("byte", dataXParameters.getJobSpeedByte());
    }

    if (dataXParameters.getJobSpeedRecord() > 0) {
        speed.put("record", dataXParameters.getJobSpeedRecord());
    }

    JSONObject errorLimit = new JSONObject();
    errorLimit.put("record", 0);
    errorLimit.put("percentage", 0);

    JSONObject setting = new JSONObject();
    setting.put("speed", speed);
    setting.put("errorLimit", errorLimit);

    return setting;
}

private JSONObject buildDataxCoreJson() {
    JSONObject speed = new JSONObject();
    speed.put("channel", DATAX_CHANNEL_COUNT);

    if (dataXParameters.getJobSpeedByte() > 0) {
        speed.put("byte", dataXParameters.getJobSpeedByte());
    }

    if (dataXParameters.getJobSpeedRecord() > 0) {
        speed.put("record", dataXParameters.getJobSpeedRecord());
    }

    JSONObject channel = new JSONObject();
    channel.put("speed", speed);

    JSONObject transport = new JSONObject();
    transport.put("channel", channel);

    JSONObject core = new JSONObject();
    core.put("transport", transport);

    return core;
}

`

DATAX_CHANNEL_COUNT is 1 by default.
UI configuration does not support split table.
You can use custom config datax json.

DATAX_CHANNEL_COUNT is 1 by default.
UI configuration does not support split table.
You can use custom config datax json.

i have done some feature modify so that we can pass the addtional datax json param through the local param . mabye this way is not graceful.
`

if (otherDataxParamMap.size() > 0){
Configuration conf = Configuration.from(json);
for (Map.Entry<String,Object> el: otherDataxParamMap.entrySet()){
conf.set(el.getKey(),el.getValue());
}
json = conf.toJSON();
}
`

Good job.
Welcome to submit a PR.

repo: apache/incubator-dolphinscheduler


text:
BDB meta version log check bug

https://github.com/apache/incubator-doris/blob/master/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java#L515
Version check using MetaContext seems meaningless. Code should be
         if (version > FeConstants.meta_version) {
                LOG.error("meta data version is out of date, image: {}. meta: {}."
                     + "please update FeConstants.meta_version and restart.",
                     MetaContext.get().getMetaVersion(), FeConstants.meta_version);
                System.exit(-1);
         }

Yes, you are right. Could you please make a PR to fix this?

OK

authors: gengjun-git morningman

repo: apache/incubator-doris


text:
Can Druid use existed datasource and segments

Hello Everyone.
I have a question when I use druid.
If I have the datasource and segment created by the previous druid, now I have created a new druid and used the original datasource and segment.
Is this done in the current druid?
Thanks.
Max

For the most part yes. Going up one version (say from 0.12 to 0.13) is guaranteed to be safe always.
As it happens it is safe to go up quite a few versions recently. More details can be found in the release notes. What is the version of your old Druid?

authors: hueiyuanSu vogievetsky

repo: apache/incubator-druid


text:
【需求】图例(legend)需要能控制不触发brushSelected事件

What problem does this feature solve?
图表绑定了brushSelected事件，这个事件会通过图例点击触发。
现在要求brushSelected事件只有框选才会触发，
而图例点击时仅触发自身的互动效果，而不触发brushSelected事件。
What does the proposed API look like?
legend增加silent属性，默认false。
当silent为true时，不触发brushSelected事件

legend click interaction will affect the number of series selected by the brush. so it will trigger the brushSelected event. Why do you want to disable it, what problem you have met?

我有图表通过brushSelected事件和别的图表产生下钻联动，现在要求下钻仅通过框选触发，legend点击不要触发，legend仅影响图表本体。
如果这个实现比较麻烦，那就采取折中方案，给事件的处理参数加上来源标识，标识由什么元素触发的
api示范
chart.on('brushSelected', function (params) {
    if(params.eventTarget=="legend"){
          return;
    }else{
         //other something
    }
});

authors: comvir deqingli

repo: apache/incubator-echarts


text:
echart地图散点设置，symbolSize: 1,时会出现散点后逐渐消失，symbolSize: 0.5就正常了

### echart地图散点设置，symbolSize: 1,时会出现散点后逐渐消失，symbolSize: 0.5就正常了
series: [
        {
            type: 'scatter',
            postEffect: {
                enable: true
            },
            progressive: 0,
            coordinateSystem: 'geo',
            blendMode: 'lighter',
            large: true,
            silent: true,
            dimensions: ['lng', 'lat'],
            data: data
            symbolSize: 1,
            /*symbolSize: function (val,params) {
                return val[2] * 1;
            },*/
            label: {
                normal: {
                    show: false
                },
                emphasis: {
                    show: false
                }
            },
            large: true,
            useWorker: true,
            steps: 2,
            itemStyle: {
                normal: {
                    shadowBlur: 10,
                    color: '#bad8ef',
                    shadowColor: '#258CF9'
                },
                emphasis: {
                    borderColor: '#fff',
                    borderWidth: 1
                }
            }
        }
    ]

你好，能否提供一个可以复现的例子

progressive: 1e6,这个配置去掉就正常了

series: [
{
name: 'zbjsaas',
type: 'scatter',
coordinateSystem: 'geo',
large: true,
useWorker: true,
steps: 2,
silent: true,
// symbolSize: 0.4,
symbolSize: function(value,params) {
return 0.4;
},
data: new Float32Array(),
label: {
normal: {
show: false
},
emphasis: {
show: false
}
},
itemStyle: {
normal: {
shadowBlur:10,
color: '#bad8ef',
shadowColor: '#258CF9'
}
}
}
]
symbolSize: 0.4,// symbol显示正常
symbolSize: function(value,params) {
return 0.4;
},// symbol不显示

authors: pissang wangmiaoduan


text:
Sql

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the commit message is formatted like:
[GEARPUMP-<Jira issue #>] Meaningful description of pull request
[ ] Make sure tests pass via sbt clean test.
[ ] Make sure old documentation affected by the pull request has been updated and new documentation added for new functionality.

@buddhiayesha2015 could you squash your commits and get rid of the merge commit Merge remote-tracking branch 'origin/sql' into sql ?

repo: apache/incubator-gearpump


text:
[VL] Result mismatch in CollectList when partial sort is involved

Backend
VL (Velox)
Bug description
Describe the issue
Reproducing SQL:
CREATE OR REPLACE TEMP VIEW temp_table AS
SELECT * FROM VALUES
  (1, 'a'), (1, 'b'), (1, 'c'),
  (2, 'd'), (2, 'e'), (2, 'f'),
  (3, 'g'), (3, 'h'), (3, 'i')
AS t(id, value);

SELECT id, collect_list(value) AS values_list
FROM (
  SELECT * FROM
  (SELECT id, value
   FROM temp_table
   DISTRIBUTE BY rand())  -- Forces a shuffle
  DISTRIBUTE BY id SORT BY id, value
) t
GROUP BY id;

Results:

The vanilla result is deterministic and values_list is sorted by value column:
id   values_list
1    ["a", "b", "c"]
2    ["d", "e", "f"]
3    ["g", "h", "i"]

The gluten result is non-deterministic and values_list is not sorted, e.g. :
id   values_list
1    ["a", "c", "b"]
3    ["g", "i", "h"]
2    ["f", "e", "d"]

The gluten physical plan:
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
   VeloxColumnarToRowExec
   +- ^(9) HashAggregateTransformer(keys=[id#0], functions=[velox_collect_list(value#1)], isStreamingAgg=false, output=[id#0, values_list#22])
      +- ^(9) HashAggregateTransformer(keys=[id#0], functions=[partial_velox_collect_list(value#1)], isStreamingAgg=false, output=[id#0, buffer#29])
         +- ^(9) InputIteratorTransformer[id#0, value#1]
            +- ShuffleQueryStage 1
               +- ColumnarExchange hashpartitioning(id#0, 20), REPARTITION_WITH_NUM, [id#0, value#1], [id=#1293], [id=#1293], [OUTPUT] List(id:IntegerType, value:StringType), [OUTPUT] List(id:IntegerType, value:StringType)
                  +- VeloxAppendBatches 3276
                     +- ^(8) ProjectExecTransformer [hash(id#0, 42) AS hash_partition_key#31, id#0, value#1]
                        +- ^(8) InputIteratorTransformer[id#0, value#1, _nondeterministic#24]
                           +- ShuffleQueryStage 0
                              +- ColumnarExchange hashpartitioning(_nondeterministic#24, 20), REPARTITION_WITH_NUM, [id#0, value#1, _nondeterministic#24], [id=#1239], [id=#1239], [OUTPUT] List(id:IntegerType, value:StringType, _nondeterministic:DoubleType), [OUTPUT] List(id:IntegerType, value:StringType, _nondeterministic:DoubleType)
                                 +- VeloxAppendBatches 3276
                                    +- ^(7) ProjectExecTransformer [hash(_nondeterministic#24, 42) AS hash_partition_key#30, id#0, value#1, _nondeterministic#24]
                                       +- ^(7) InputIteratorTransformer[id#0, value#1, _nondeterministic#24]
                                          +- RowToVeloxColumnar
                                             +- LocalTableScan [id#0, value#1, _nondeterministic#24]
+- == Initial Plan ==
   SortAggregate(key=[id#0], functions=[velox_collect_list(value#1)], output=[id#0, values_list#22])
   +- SortAggregate(key=[id#0], functions=[partial_velox_collect_list(value#1)], output=[id#0, buffer#29])
      +- Sort [id#0 ASC NULLS FIRST, value#1 ASC NULLS FIRST], false, 0
         +- Exchange hashpartitioning(id#0, 20), REPARTITION_WITH_NUM, [id=#1211]
            +- Project [id#0, value#1]
               +- Exchange hashpartitioning(_nondeterministic#24, 20), REPARTITION_WITH_NUM, [id=#1209]
                  +- LocalTableScan [id#0, value#1, _nondeterministic#24]

Even though the collect_list function is non-deterministic, as stated in the documentation, some ETL tasks in our production environment depend on this behavior in vanilla Spark.
Root cause for this issue
We can see the Sort operator is removed through the gluten plan. This change appears to be due to this code snippet: code link.
class ReplaceSingleNode() extends LogLevelUtil with Logging {

    def doReplace(p: SparkPlan): SparkPlan = {
// ....
    case plan: SortAggregateExec =>
      logDebug(s"Columnar Processing for ${plan.getClass} is currently supported.")
      HashAggregateExecBaseTransformer.from(plan)(SortUtils.dropPartialSort)
// ...
}
object SortUtils {
  def dropPartialSort(plan: SparkPlan): SparkPlan = plan match {
    case RewrittenNodeWall(p) => RewrittenNodeWall(dropPartialSort(p))
    case PartialSortLike(child) => child
    // from pre/post project-pulling
    case ProjectLike(PartialSortLike(ProjectLike(child))) if plan.outputSet == child.outputSet =>
      child
    case ProjectLike(PartialSortLike(child)) => plan.withNewChildren(Seq(child))
    case _ => plan
  }
}

I'm  wondering why the partial sort added by SQL 'sort by' needs to be removed for SortAggregateExec. Would it be possible to retain the partial sort operator for resolving this issue?
Spark version
None
Spark configurations
No response
System information
No response
Relevant logs
No response

@NEUpanning, in Gluten sort agg is replaced with hash agg, which makes the prior sort operator unnecessary possibly. So Gluten will remove the sort operator unless it is needed for ensuring ordering requirement.

I believe the relevant sort-removal code is here. I think we could preserve the sort as long as vanilla Spark plan has it with a hash agg. This requires for a fix.

I think we could preserve the sort as long as vanilla Spark plan has it with a hash agg.

@zhztheplayer I see this feature is implemented in gluten 1.2 branch, but main branch doesn't include it for some reason. For this issue, CollectList function is replaced by VeloxCollectList function in logical optimization phase. Here is the spark plan:
== Parsed Logical Plan ==
Aggregate [id#0], [id#0, collect_list(value#1, 0, 0) AS values_list#13]
+- SubqueryAlias t
   +- Project [id#0, value#1]
      +- RepartitionByExpression [_nondeterministic#15], 20
         +- Project [id#0, value#1, rand(5386921442550703776) AS _nondeterministic#15]
            +- Project [id#0, value#1]
               +- SubqueryAlias temp_table
                  +- Project [id#0, value#1]
                     +- SubqueryAlias t
                        +- LocalRelation [id#0, value#1]

== Analyzed Logical Plan ==
id: int, values_list: array<string>
Aggregate [id#0], [id#0, collect_list(value#1, 0, 0) AS values_list#13]
+- SubqueryAlias t
   +- Project [id#0, value#1]
      +- RepartitionByExpression [_nondeterministic#15], 20
         +- Project [id#0, value#1, rand(5386921442550703776) AS _nondeterministic#15]
            +- Project [id#0, value#1]
               +- SubqueryAlias temp_table
                  +- Project [id#0, value#1]
                     +- SubqueryAlias t
                        +- LocalRelation [id#0, value#1]

== Optimized Logical Plan ==
Aggregate [id#0], [id#0, velox_collect_list(value#1) AS values_list#13]
+- Project [id#0, value#1]
   +- RepartitionByExpression [_nondeterministic#15], 20
      +- LocalRelation [id#0, value#1, _nondeterministic#15]

== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
   VeloxColumnarToRowExec
   +- ^(3) HashAggregateTransformer(keys=[id#0], functions=[velox_collect_list(value#1)], isStreamingAgg=false, output=[id#0, values_list#13])
      +- ^(3) InputIteratorTransformer[id#0, buffer#20]
         +- CustomShuffleReader coalesced
            +- ShuffleQueryStage 1
               +- ColumnarExchange hashpartitioning(id#0, 20), ENSURE_REQUIREMENTS, [id#0, buffer#20], [id=#1048], [id=#1048], [OUTPUT] List(id:IntegerType, buffer:ArrayType(StringType,false)), [OUTPUT] List(id:IntegerType, buffer:ArrayType(StringType,false))
                  +- VeloxAppendBatches 3276
                     +- ^(2) ProjectExecTransformer [hash(id#0, 42) AS hash_partition_key#22, id#0, buffer#20]
                        +- ^(2) FlushableHashAggregateTransformer(keys=[id#0], functions=[partial_velox_collect_list(value#1)], isStreamingAgg=false, output=[id#0, buffer#20])
                           +- ^(2) ProjectExecTransformer [id#0, value#1]
                              +- ^(2) InputIteratorTransformer[id#0, value#1, _nondeterministic#15]
                                 +- ShuffleQueryStage 0
                                    +- ColumnarExchange hashpartitioning(_nondeterministic#15, 20), REPARTITION_WITH_NUM, [id#0, value#1, _nondeterministic#15], [id=#966], [id=#966], [OUTPUT] List(id:IntegerType, value:StringType, _nondeterministic:DoubleType), [OUTPUT] List(id:IntegerType, value:StringType, _nondeterministic:DoubleType)
                                       +- VeloxAppendBatches 3276
                                          +- ^(1) ProjectExecTransformer [hash(_nondeterministic#15, 42) AS hash_partition_key#21, id#0, value#1, _nondeterministic#15]
                                             +- ^(1) InputIteratorTransformer[id#0, value#1, _nondeterministic#15]
                                                +- RowToVeloxColumnar
                                                   +- LocalTableScan [id#0, value#1, _nondeterministic#15]
+- == Initial Plan ==
   SortAggregate(key=[id#0], functions=[velox_collect_list(value#1)], output=[id#0, values_list#13])
   +- Sort [id#0 ASC NULLS FIRST], false, 0
      +- Exchange hashpartitioning(id#0, 20), ENSURE_REQUIREMENTS, [id=#936]
         +- SortAggregate(key=[id#0], functions=[partial_velox_collect_list(value#1)], output=[id#0, buffer#20])
            +- Sort [id#0 ASC NULLS FIRST], false, 0
               +- Project [id#0, value#1]
                  +- Exchange hashpartitioning(_nondeterministic#15, 20), REPARTITION_WITH_NUM, [id=#928]
                     +- LocalTableScan [id#0, value#1, _nondeterministic#15]

This leads to Spark using SortAggregateExec
instead of ObjectHashAggregateExec as aggregate operator. So I think the sort also should be preserved if aggregate expressions include VeloxCollectList or VeloxCollectSet.

authors: NEUpanning PHILO-HE zhztheplayer

repo: apache/incubator-gluten


text:
[GLUTEN-5307][VL] Fix Potential Overflow Issue in VeloxShuffleWriter Due to Mismatched Data Types of RowNumber

What changes were proposed in this pull request?
Fixes: #5307

Thanks. Does this cause overflow in real cases? The uint16_t is used on purpose to benefit L2 cache, and is not supposed to cause the overflow with the 32k limitation of the batch size. Please check https://github.com/apache/incubator-gluten/pull/4401#issuecomment-1895360554

Thanks. Does this cause overflow in real cases? The uint16_t is used on purpose to benefit L2 cache, and is not supposed to cause the overflow with the 32k limitation of the batch size. Please check #4401 (comment)
cc: @FelixYBW

Indeed, using uint16_t in this scenario can lead to overflow.   The specific statements and conditions for reproducing this issue have already been outlined in issue #5307, and I will be providing a dataset shortly to further illustrate this point.

@yangzhg Thanks for your detailed explanation. There are configurations in Gluten to explicitly limit the batch size within 32k https://github.com/marin-ma/gluten/blob/cf391fae10da22d73c451aced5cd05e474e3bccd/shims/common/src/main/scala/io/glutenproject/GlutenConfig.scala#L929-L946, (the configurations are passed to velox QueryContext here https://github.com/apache/incubator-gluten/blob/c60b5d90e0be23430672d732f9d574267b67c06e/cpp/velox/compute/WholeStageResultIterator.cc#L477-L480), and the prerequisites of doing that is we expect the batch size produced by Velox pipeline always <= 32k. If the input batch size doesn't exceed this limitation, there shouldn't be a problem to use uint16_t.
However, it seems like there are cases that the output from Velox pipeline can exceed the configured batch size. We are not sure if it's an expected case in Velox. Could you help to raise an issue in Velox to illustrate the case you've, that the Hashagg output can exceed the configured batch size? Thanks!

@yangzhg Thanks for your detailed explanation. There are configurations in Gluten to explicitly limit the batch size within 32k
https://github.com/apache/incubator-gluten/blob/c60b5d90e0be23430672d732f9d574267b67c06e/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala#L478-L487
, the configurations are passed to velox QueryContext here
https://github.com/apache/incubator-gluten/blob/c60b5d90e0be23430672d732f9d574267b67c06e/cpp/velox/compute/WholeStageResultIterator.cc#L477-L480
, and the prerequisites of doing that is we expect the batch size produced by Velox pipeline always <= 32k. If the input batch size doesn't exceed this limitation, there shouldn't be a problem to use uint16_t.
However, it seems like there are cases that the output from Velox pipeline can exceed the configured batch size. We are not sure if it's an expected case in Velox. Could you help to raise an issue in Velox to illustrate the case you've met, that the Hashagg output can exceed the configured batch size? Thanks!

OK! Just out of curiosity, how much performance improvement can using uint16_t bring compared to uint32_t?

OK! Just out of curiosity, how much performance improvement can using uint16_t bring compared to uint32_t?

It's up to queries and CPU. You may test in your case. Memory latency can be >50x of L2 cache latency. Split operation is top 5 hotspot in our TPCH test.
If batch size is unlimited, it may more hurt Velox pipeline's performance. Velox is vectorized engine, basic policy is that one operator's output is used by following operator's input. So if batch size is too large it will leads to big cache miss and hurt performance. It's why we use 4K rows as default now.

/Benchmark Velox

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query
log/native_5326_time.csv
log/native_master_04_17_2024_9b3f59a1c_time.csv
difference
percentage

q1
35.81
36.61
0.796
102.22%

q2
23.56
24.12
0.555
102.35%

q3
37.76
37.01
-0.749
98.02%

q4
39.72
38.03
-1.688
95.75%

q5
70.34
70.92
0.581
100.83%

q6
9.58
5.81
-3.767
60.67%

q7
84.01
86.12
2.110
102.51%

q8
83.77
82.86
-0.909
98.92%

q9
119.76
123.45
3.697
103.09%

q10
43.75
45.70
1.950
104.46%

q11
20.40
20.35
-0.050
99.75%

q12
25.26
28.15
2.882
111.41%

q13
55.48
54.46
-1.022
98.16%

q14
21.81
17.88
-3.933
81.97%

q15
29.29
30.69
1.403
104.79%

q16
14.27
14.00
-0.269
98.12%

q17
102.13
101.27
-0.856
99.16%

q18
142.38
143.83
1.449
101.02%

q19
13.68
13.61
-0.063
99.54%

q20
29.30
27.86
-1.439
95.09%

q21
293.40
285.86
-7.540
97.43%

q22
16.66
14.42
-2.242
86.55%

total
1312.12
1303.01
-9.105
99.31%

@yangzhg Sorry that I didn't notice this PR wasn't got merged. There's no performance impact. Thanks for your fix!

@yangzhg Did you create an issue in Velox community?

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query
log/native_5326_time.csv
log/native_master_04_17_2024_9b3f59a1c_time.csv
difference
percentage

q1
37.26
36.61
-0.650
98.25%

q2
23.85
24.12
0.269
101.13%

q3
37.99
37.01
-0.972
97.44%

q4
39.90
38.03
-1.875
95.30%

q5
71.86
70.92
-0.943
98.69%

q6
5.69
5.81
0.122
102.14%

q7
81.87
86.12
4.245
105.18%

q8
86.18
82.86
-3.320
96.15%

q9
126.41
123.45
-2.957
97.66%

q10
44.86
45.70
0.843
101.88%

q11
19.72
20.35
0.629
103.19%

q12
26.70
28.15
1.443
105.40%

q13
54.46
54.46
0.002
100.00%

q14
18.33
17.88
-0.447
97.56%

q15
28.73
30.69
1.963
106.83%

q16
14.25
14.00
-0.253
98.23%

q17
100.80
101.27
0.474
100.47%

q18
144.23
143.83
-0.408
99.72%

q19
16.59
13.61
-2.975
82.07%

q20
28.91
27.86
-1.044
96.39%

q21
284.71
285.86
1.157
100.41%

q22
15.15
14.42
-0.730
95.18%

total
1308.44
1303.01
-5.428
99.59%

authors: FelixYBW GlutenPerfBot marin-ma yangzhg


text:
HAWQ-839. Fix libyarn coredump when failover to standby RM

Please review, Thanks!

+1

authors: huor linwen

repo: apache/incubator-hawq


text:
Error when building MXNet with "USE_DIST_KVSTORE=1" inside docker

Note: Providing complete information in the most concise form is the best way to get help. This issue template serves as the checklist for essential information to most of the technical issues and bug reports. For non-technical issues and feature requests, feel free to present the information in what you believe is the best form.
For Q & A and discussion, please start a discussion thread at https://discuss.mxnet.io
Description
I am unable to build mxnet on docker image based on ubuntu 16.04.
Build info (Required if built from source)
Compiler (gcc/clang/mingw/visual studio):
MXNet commit hash:
dd1f21b4369371f4d20fc8a88c1d10834f8cf53b
Build config:
RUN make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas USE_DIST_KVSTORE=1
Error Message:
OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/engine/naive_engine.cc -o build/src/engine/naive_engine.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/engine/openmp.cc -o build/src/engine/openmp.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/engine/threaded_engine_pooled.cc -o build/src/engine/threaded_engine_pooled.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/engine/engine.cc -o build/src/engine/engine.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/engine/threaded_engine.cc -o build/src/engine/threaded_engine.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/engine/threaded_engine_perdevice.cc -o build/src/engine/threaded_engine_perdevice.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/storage/storage.cc -o build/src/storage/storage.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/c_api/c_api_symbolic.cc -o build/src/c_api/c_api_symbolic.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/c_api/c_api_profile.cc -o build/src/c_api/c_api_profile.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/c_api/c_api_ndarray.cc -o build/src/c_api/c_api_ndarray.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/c_api/c_api_executor.cc -o build/src/c_api/c_api_executor.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/c_api/c_predict_api.cc -o build/src/c_api/c_predict_api.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/c_api/c_api_function.cc -o build/src/c_api/c_api_function.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/c_api/c_api.cc -o build/src/c_api/c_api.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/c_api/c_api_error.cc -o build/src/c_api/c_api_error.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/executor/inplace_addto_detect_pass.cc -o build/src/executor/inplace_addto_detect_pass.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/executor/infer_graph_attr_pass.cc -o build/src/executor/infer_graph_attr_pass.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/executor/graph_executor.cc -o build/src/executor/graph_executor.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/executor/attach_op_execs_pass.cc -o build/src/executor/attach_op_execs_pass.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/executor/attach_op_resource_pass.cc -o build/src/executor/attach_op_resource_pass.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/kvstore/gradient_compression.cc -o build/src/kvstore/gradient_compression.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/kvstore/kvstore_utils.cc -o build/src/kvstore/kvstore_utils.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/kvstore/kvstore.cc -o build/src/kvstore/kvstore.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/resource.cc -o build/src/resource.o
g++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/incubator-mxnet/mshadow/ -I/incubator-mxnet/dmlc-core/include -fPIC -I/incubator-mxnet/nnvm/include -I/incubator-mxnet/dlpack/include -I/incubator-mxnet/nnvm/tvm/include -Iinclude -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -DMSHADOW_USE_CUDA=0 -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK  -DMXNET_USE_DIST_KVSTORE -I/incubator-mxnet/ps-lite/include -I/incubator-mxnet/deps/include -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/initialize.cc -o build/src/initialize.o
make CXX=g++ DEPS_PATH=/incubator-mxnet/deps -C /incubator-mxnet/ps-lite ps
make[1]: Entering directory '/incubator-mxnet/ps-lite'
rm -rf zeromq-4.1.4.tar.gz zeromq-4.1.4
wget https://raw.githubusercontent.com/mli/deps/master/build/zeromq-4.1.4.tar.gz && tar --no-same-owner -zxf zeromq-4.1.4.tar.gz
make/deps.mk:22: recipe for target '/incubator-mxnet/deps/include/zmq.h' failed
make[1]: Leaving directory '/incubator-mxnet/ps-lite'
/bin/sh: 1: wget: not found
make[1]: *** [/incubator-mxnet/deps/include/zmq.h] Error 127
Makefile:436: recipe for target 'PSLITE' failed
make: *** [PSLITE] Error 2
make: *** Waiting for unfinished jobs....
The command '/bin/sh -c make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas USE_DIST_KVSTORE=1' returned a non-zero code: 2

I solved the issue

Hello, I met the same problem. How did you solve this problem?

@wanderingpj I am very sorry, I am not sure what I did to solve that issue.

it's ok.

authors: mouradmourafiq wanderingpj

repo: apache/incubator-mxnet


text:
Proposal: Generalize broadcast_like

Now that broadcast_like has been merged (#11820), I would like to discuss the extension discussed here by @taliesinb. Quoting it:

There is a generalization that would be extremely useful for this operator to have. The generalization is very similar to one that was discussed at https://discuss.mxnet.io/t/reshaping-broadcasting-without-hardcoding-target-dimensions/851/6 (you can skip to the last 4 comments, the thread contains an irrelevant proposal although the motivation is relevant).
In short, the generalization would allow only specific dimensions to be copied from the 'other' tensor. For example:
input.shape = (1, 2, 1, 3)
other.shape = (5, 6, 7, 8)
output = broadcast_like(input, other, input_axes:(0,2), other_axes:(1,3))
output.shape = (6, 2, 8, 3)
In other words, what's happening here is that the you can pick exactly which axes of the other tensor you want to use to "fill in" axes of the input tensor. This is how broadcast_axes works, except instead of providing the values via a size parameter, you are providing them from specific axes in the other tensor.
The reason this is so valuable is that it is common to have another tensor that contains the dimension you want to broadcast amongst a set of irrelevant dimensions. There is simply no other way of "extracting" the relevant dimension from elsewhere in the net, so currently you have to hardcode that dimension into a parameter list, which forces expensive workarounds like bucketing where otherwise cheap reshaping would work to make a net that is compatible with multiple sequence lengths, for example.
The current behavior of broadcast_like in the PR would be consistent with this generalization if the default value of input_axis is the empty tuple, which means "all axes".

I would like to open this design up for discussion.
@ifeherva

Sounds reasonable to me.

@sbodenstein I was working on broadcast_like briefly a few weeks ago (#11443 ). This specs looks very reasonable. I am just wondering if you would be comfortable that I work on top of @ifeherva new broadcast_like, and generalize it as discussed above?

@rondogency: @ifeherva had also expressed interest to implement it, but whoever has more time this week should probably do it (we need it for next Monday, so I would add it neither of you has time this week).

I see. I won't be available for it until Saturday. If you need it by Monday, I may not be able to make it on time. Sorry for the inconvenience.

authors: ifeherva rondogency sbodenstein


text:
Python, Bug: Speech_recognition crashes in threaded_engine.h:379: Error: compute_ctc_loss, stat = execution failed

Any insights in how to fix this would be greatly appreciated
Description
Speech_recognition training on LibriSpeech crashes in threaded_engine.h:379: Error: compute_ctc_loss, stat = execution failed.
Environment info (Required)
ubuntu 16.04, cuda 9.2 Cudnn7.1.4, nccl 2.1.2 4 Nvidia V100s, CUDA_VISIBLE_DEVICES=0
deepspeech.cfg set to only use 1 GPU
What to do:
1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
2. Run the script using `python diagnose.py` and paste its output here.
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                28
On-line CPU(s) list:   0-27
Thread(s) per core:    1
Core(s) per socket:    14
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
Stepping:              1
CPU MHz:               2189.890
CPU max MHz:           3500.0000
CPU min MHz:           1200.0000
BogoMIPS:              5189.99
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              35840K
NUMA node0 CPU(s):     0-13
NUMA node1 CPU(s):     14-27
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt ibrs ibpb stibp kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts
----------Python Info----------
('Version      :', '2.7.12')
('Compiler     :', 'GCC 5.4.0 20160609')
('Build        :', ('default', 'Dec  4 2017 14:50:18'))
('Arch         :', ('64bit', 'ELF'))
------------Pip Info-----------
('Version      :', '9.0.1')
('Directory    :', '/usr/local/lib/python2.7/dist-packages/pip')
----------MXNet Info-----------
('Version      :', '1.3.0')
('Directory    :', '/home/levinth/mxnet/python/mxnet')
Hashtag not found. Not installed from pre-built package.
----------System Info----------
('Platform     :', 'Linux-4.4.0-130-generic-x86_64-with-Ubuntu-16.04-xenial')
('system       :', 'Linux')
('node         :', 'zt-gpu-lin')
('release      :', '4.4.0-130-generic')
('version      :', '#156-Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018')
----------Hardware Info----------
('machine      :', 'x86_64')
('processor    :', 'x86_64')
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0267 sec, LOAD: 0.4810 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0077 sec, LOAD: 0.4934 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.2532 sec, LOAD: 0.2584 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0342 sec, LOAD: 0.3129 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0159 sec, LOAD: 0.1362 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0608 sec, LOAD: 1.6993 sec.

Package used (Python/R/Scala/Julia):
python
Build info (Required if built from source)
Compiler (gcc/clang/mingw/visual studio):
gcc/nvcc
gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
MXNet commit hash:
git rev-parse HEAD
1fa04f2c9a7ba0c3273d080afa7fc993b927f114
Build config:
make -j USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 USE_NCCL=1 USE_NCCL_PATH=/usr/local/nccl
config.mk is default except for uncommenting the 2 lines for including warp-ctc
Error Message:
[    INFO][2018/08/03 10:57:31.331] optimizer_params_dictionary = {"momentum":0.9}
[    INFO][2018/08/03 10:57:31.332] clip_gradient = 100
[    INFO][2018/08/03 10:57:31.332] weight_decay = 0.
[    INFO][2018/08/03 10:57:31.332]
[    INFO][2018/08/03 10:57:59.214] ---------train---------
[10:58:00] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:109: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
terminate called after throwing an instance of 'dmlc::Error'
what():  [10:58:03] src/engine/./threaded_engine.h:379: Error: compute_ctc_loss, stat = execution failed
A fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.
Stack trace returned 9 entries:
[bt] (0) /home/levinth/mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::StackTraceabi:cxx11+0x5b) [0x7f9dfe87440b]
[bt] (1) /home/levinth/mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x28) [0x7f9dfe874f78]
[bt] (2) /home/levinth/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock*)+0xfa9) [0x7f9e015b3a59]
[bt] (3) /home/levinth/mxnet/python/mxnet/../../lib/libmxnet.so(void mxnet::engine::ThreadedEnginePerDevice::GPUWorker<(dmlc::ConcurrentQueueType)0>(mxnet::Context, bool, mxnet::engine::ThreadedEnginePerDevice::ThreadWorkerBlock<(dmlc::ConcurrentQueueType)0>, std::shared_ptrdmlc::ManualEvent const&)+0xeb) [0x7f9e015c9d0b]
[bt] (4) /home/levinth/mxnet/python/mxnet/../../lib/libmxnet.so(std::_Function_handler<void (std::shared_ptrdmlc::ManualEvent), mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock, bool)::{lambda()#4}::operator()() const::{lambda(std::shared_ptrdmlc::ManualEvent)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptrdmlc::ManualEvent&&)+0x4e) [0x7f9e015c9f7e]
[bt] (5) /home/levinth/mxnet/python/mxnet/../../lib/libmxnet.so(std::thread::_Impl<std::_Bind_simple<std::function<void (std::shared_ptrdmlc::ManualEvent)> (std::shared_ptrdmlc::ManualEvent)> >::_M_run()+0x4a) [0x7f9e015b299a]
[bt] (6) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f9e4e10dc80]
[bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f9e581196ba]
[bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f9e57e4f41d]
Minimum reproducible example
default deepspeech with LibriSpeech data set prepared per instructions
train.py edited to use

#summary_writer = SummaryWriter(tblog_dir)
summary_writer = tf.summary.FileWriter(tblog_dir)

Steps to reproduce
see attachment for installation and invocation notes
mxnet_deepspeech_build_for_bug.txt

@anirudh2290 Can you please add labels: Python, Bug

done
On Mon, Aug 6, 2018 at 11:52 AM, Roshani Nagmote notifications@github.com
wrote:

@anirudh2290 https://github.com/anirudh2290 Can you please add labels:
Python, Bug
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-mxnet/issues/12024#issuecomment-410813935,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIUuT0CnCD3ui7TKQVycJ2eVl5x3Krztks5uOJBXgaJpZM4VuhrR
.

I am in the process of replicating the issue - haven't yet got to the failure that you are seeing.
In the meantime, just had a couple of questions to try and understand the issue/replication steps better:
Why use tf.summary.FileWriter instead of SummaryWriter?
Also, have you tried mxboard's FileWriter/SummaryWriter in this example?
Any inputs on this would be helpful.

@David-Levinthal have you had a chance to try out the solutions listed in https://github.com/apache/incubator-mxnet/issues/6121, https://github.com/apache/incubator-mxnet/issues/7002 or https://github.com/apache/incubator-mxnet/issues/6603?

SummaryWriter is deprecated. Use an R1.9 TF installation for the source
I used TF as I have source builds
if mxboard is required perhaps a minor change to the readme for
installation would be a solution.
d
On Thu, Aug 16, 2018 at 11:42 PM, Vandana Kannan notifications@github.com
wrote:

I am in the process of replicating the issue - haven't yet got to the
failure that you are seeing.
In the meantime, just had a couple of questions to try and understand the
issue/replication steps better:
Why use tf.summary.FileWriter instead of SummaryWriter?
Also, have you tried mxboard's FileWriter/SummaryWriter in this example?
Any inputs on this would be helpful.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-mxnet/issues/12024#issuecomment-413773081,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIUuTzzq1aBDwgfWPed-r1JG_3JTKeMRks5uRmXRgaJpZM4VuhrR
.

Update: I don't see the issue with mxboard.
Changes required (https://github.com/vandanavk/incubator-mxnet/commit/d1bc989afc6366404147a6d30dbcfd5b912a8350)
It's been executing for more than 1 hour now.
[    INFO][2018/08/18 00:37:36.313] clip_gradient = 100
[    INFO][2018/08/18 00:37:36.313] weight_decay = 0.
[    INFO][2018/08/18 00:37:36.313]
[    INFO][2018/08/18 00:38:00.737] ---------train---------
[00:38:02] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:109: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[    INFO][2018/08/18 00:52:28.763] label: and be like the self sacrificing heroines she loved to act
[    INFO][2018/08/18 00:52:28.763] pred :       , cer: 0.864865 (distance: 32/ label length: 37)
[    INFO][2018/08/18 00:52:28.764] label: ain't that something done after you've done all that
[    INFO][2018/08/18 00:52:28.765] pred :      , cer: 0.875000 (distance: 28/ label length: 32)
[    INFO][2018/08/18 00:52:28.766] label: so said the captain in a voice so stern it made joe wince
[    INFO][2018/08/18 00:52:28.766] pred :     , cer: 0.921053 (distance: 35/ label length: 38)
[    INFO][2018/08/18 00:52:28.767] label: that my words seem to you utterly unnecessary and out of place
[    INFO][2018/08/18 00:52:28.767] pred :       , cer: 0.875000 (distance: 35/ label length: 40)
[    INFO][2018/08/18 00:52:28.768] label: who dethroned his father you are welcome brave jason
[    INFO][2018/08/18 00:52:28.768] pred :       , cer: 0.852941 (distance: 29/ label length: 34)
[    INFO][2018/08/18 00:52:28.770] label: several times that day as he perceived coulson's jealous sullenness
[    INFO][2018/08/18 00:52:28.770] pred :         , cer: 0.829268 (distance: 34/ label length: 41)
[    INFO][2018/08/18 00:52:28.771] label: but she turned it off with assumed lightness oh yes
[    INFO][2018/08/18 00:52:28.771] pred :       , cer: 0.848485 (distance: 28/ label length: 33)
[    INFO][2018/08/18 00:52:28.772] label: missus ludlow's mental motions were sufficiently various
[    INFO][2018/08/18 00:52:28.772] pred :       , cer: 0.843750 (distance: 27/ label length: 32)
[    INFO][2018/08/18 00:52:28.774] label: the children said she did not shed one tear but prayed all the while
[    INFO][2018/08/18 00:52:28.774] pred :       , cer: 0.888889 (distance: 40/ label length: 45)
[    INFO][2018/08/18 00:52:28.775] label: many that did ill under physicians hands have happily escaped
[    INFO][2018/08/18 00:52:28.775] pred :      , cer: 0.894737 (distance: 34/ label length: 38)
[    INFO][2018/08/18 00:52:28.776] label: while his face assumed a hard determined expression
[    INFO][2018/08/18 00:52:28.776] pred :       , cer: 0.838710 (distance: 26/ label length: 31)
[    INFO][2018/08/18 00:52:28.778] label: and white and furnished with light and heat hot or cold water if desired

as has been stated:
the issue is with using tensorflow/tensorboard..
.if mxboard is required.then the instructions need to say that
On Fri, Aug 17, 2018 at 6:48 PM, Vandana Kannan notifications@github.com
wrote:

Update: I don't see the issue with mxboard.
Changes required (vandanavk@d1bc989
https://github.com/vandanavk/incubator-mxnet/commit/d1bc989afc6366404147a6d30dbcfd5b912a8350
)
It's been executing for more than 1 hour now.
[    INFO][2018/08/18 00:37:36.313] clip_gradient = 100
[    INFO][2018/08/18 00:37:36.313] weight_decay = 0.
[    INFO][2018/08/18 00:37:36.313]
[    INFO][2018/08/18 00:38:00.737] ---------train---------
[00:38:02] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:109: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[    INFO][2018/08/18 00:52:28.763] label: and be like the self sacrificing heroines she loved to act
[    INFO][2018/08/18 00:52:28.763] pred :       , cer: 0.864865 (distance: 32/ label length: 37)
[    INFO][2018/08/18 00:52:28.764] label: ain't that something done after you've done all that
[    INFO][2018/08/18 00:52:28.765] pred :      , cer: 0.875000 (distance: 28/ label length: 32)
[    INFO][2018/08/18 00:52:28.766] label: so said the captain in a voice so stern it made joe wince
[    INFO][2018/08/18 00:52:28.766] pred :     , cer: 0.921053 (distance: 35/ label length: 38)
[    INFO][2018/08/18 00:52:28.767] label: that my words seem to you utterly unnecessary and out of place
[    INFO][2018/08/18 00:52:28.767] pred :       , cer: 0.875000 (distance: 35/ label length: 40)
[    INFO][2018/08/18 00:52:28.768] label: who dethroned his father you are welcome brave jason
[    INFO][2018/08/18 00:52:28.768] pred :       , cer: 0.852941 (distance: 29/ label length: 34)
[    INFO][2018/08/18 00:52:28.770] label: several times that day as he perceived coulson's jealous sullenness
[    INFO][2018/08/18 00:52:28.770] pred :         , cer: 0.829268 (distance: 34/ label length: 41)
[    INFO][2018/08/18 00:52:28.771] label: but she turned it off with assumed lightness oh yes
[    INFO][2018/08/18 00:52:28.771] pred :       , cer: 0.848485 (distance: 28/ label length: 33)
[    INFO][2018/08/18 00:52:28.772] label: missus ludlow's mental motions were sufficiently various
[    INFO][2018/08/18 00:52:28.772] pred :       , cer: 0.843750 (distance: 27/ label length: 32)
[    INFO][2018/08/18 00:52:28.774] label: the children said she did not shed one tear but prayed all the while
[    INFO][2018/08/18 00:52:28.774] pred :       , cer: 0.888889 (distance: 40/ label length: 45)
[    INFO][2018/08/18 00:52:28.775] label: many that did ill under physicians hands have happily escaped
[    INFO][2018/08/18 00:52:28.775] pred :      , cer: 0.894737 (distance: 34/ label length: 38)
[    INFO][2018/08/18 00:52:28.776] label: while his face assumed a hard determined expression
[    INFO][2018/08/18 00:52:28.776] pred :       , cer: 0.838710 (distance: 26/ label length: 31)
[    INFO][2018/08/18 00:52:28.778] label: and white and furnished with light and heat hot or cold water if desired
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-mxnet/issues/12024#issuecomment-414024311,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIUuT3ZrTX3kKZtHIuNm8uk9kbDfxKhzks5uR3KGgaJpZM4VuhrR
.

PR https://github.com/apache/incubator-mxnet/pull/12291 submitted

@David-Levinthal The changes in the PR have been approved. Please verify if the changes work at your end too.

may take a few days...my lab was moved to a new building...
:-)
what fun!
:-(
On Tue, Aug 28, 2018 at 7:46 AM Vandana Kannan notifications@github.com
wrote:

@David-Levinthal https://github.com/David-Levinthal The changes in the
PR have been approved. Please verify if the changes work at your end too.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-mxnet/issues/12024#issuecomment-416613073,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIUuT7v_cxzkOyLhVzXWpAb0EGG3XRNbks5uVVfSgaJpZM4VuhrR
.

@David-Levinthal just wanted to followup to check if you still see this issue

sorry..seriously overwhelmed..know a good cloning lab?
d
On Mon, Sep 24, 2018 at 10:03 PM Vandana Kannan notifications@github.com
wrote:

@David-Levinthal https://github.com/David-Levinthal just wanted to
followup to check if you still see this issue
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-mxnet/issues/12024#issuecomment-424208040,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIUuT6CQ9OgTxolOVAFJ5_97aFFGwwWBks5uebk1gaJpZM4VuhrR
.

@David-Levinthal
requesting an update from your side, if the issue is fixed , we can close this issue

@David-Levinthal  since the issue, has been resolved in a PR, I am requesting committers to close this issue, Feel free to reopen it if the error persists on your side
@sandeep-krishnamurthy Requesting to close this issue since it has been resolved in a PR

authors: David-Levinthal Roshrini vandanavk vrakesh


text:
Floating Point Exception mxnet.ndarray.contrib.bipartite_matching

Description
(A clear and concise description of what the bug is.)

mxnet.ndarray.contrib.bipartite_matching gives floating point exception when input data's shape contains 0.
Curiously, it only gives floating point exception when 0 is at certain location in shape, and gives dmlc::Error for other locations. See the provided code snippets for example.

Error Message
(Paste the complete error message. Please also include stack trace by setting environment variable DMLC_LOG_STACK_TRACE_DEPTH=10 before running your script.)

Floating point exception(core dumped)

To Reproduce
(If you developed your own code, please provide a short script that reproduces the error. For existing examples, please provide link.)
The code below produces floating point exception:
import mxnet as mx
import numpy as np

input = mx.nd.array(np.random.rand(0,2))
mx.ndarray.contrib.bipartite_matching(input, threshold=0.1)

gives:

Floating point exception (core dumped)

If swap the position of the 0 in shape, it gives dmlc::Error instead:
import mxnet as mx
import numpy as np

input = mx.nd.array(np.random.rand(2,0))
mx.ndarray.contrib.bipartite_matching(input, threshold=0.1)

gives:

terminate called after throwing an instance of 'dmlc::Error'
what():  [22:15:23] src/imperative/./imperative_utils.h:146: Operator _contrib_bipartite_matching inferring shapes failed.
input shapes:
[2,-1]
output shapes:
[2]
[-1]
operator attributes:
threshold : 0.1
Stack trace:
[bt] (0) /root/miniconda3/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x307d3b) [0x7f5828e96d3b]
[bt] (1) /root/miniconda3/lib/python3.7/site-packages/mxnet/libmxnet.so(mxnet::imperative::SetShapeType(mxnet::Context const&, nnvm::NodeAttrs const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, std::vector<mxnet::NDArray*, std::allocatormxnet::NDArray* > const&, mxnet::DispatchMode*)+0x363b) [0x7f582c0323bb]
Aborted (core dumped)

Steps to reproduce
(Paste the commands you ran that produced the error.)

run the above code snippets in python Interpreter or run it as a python script

What have you tried to solve it?

Environment
We recommend using our script for collecting the diagnositc information. Run the following command and paste the outputs below:
curl --retry 10 -s https://raw.githubusercontent.com/dmlc/gluon-nlp/master/tools/diagnose.py | python

# paste outputs here

I got 404 when trying to get the script.
Some environment information:

OS: ubuntu 18.04
Python: 3.7.6
pip: 20.0.2
numpy: 1.18.5
mxnet: 1.6.0

See the above PR for the fix.

authors: leeyeetonn szha


text:
gradients using loss=F.make_loss(myloss) are not same as gradients not using loss=F.make_loss(myloss) in Hybrid programming

This operator accepts a customized loss function symbol as a terminal loss and the symbol should be an operator with no backward dependency. The output of this function is the gradient of loss with respect to the input data.

The description of ndarray.make_loss is same as the description of symbol.make_loss. And it only explains symbol, not ndarray.
I want to know what F.make_ loss() will do when I use net.hybridize() and loss.backward().
https://mxnet.apache.org/versions/1.7.0/api/python/docs/api/ndarray/ndarray.html?highlight=make_loss#mxnet.ndarray.make_loss
https://mxnet.apache.org/versions/1.7.0/api/python/docs/api/symbol/symbol.html#mxnet.symbol.make_loss
class Myloss(mx.gluon.nn.HybridBlock):
    def __init__(self):
        super(Myloss, self).__init__()
    def hybrid_forward(self, F, pred, gt):
        loss_l2 = F.sum(F.square(pred - gt), axis=1) / 2
        return F.make_loss(loss_l2)
net=resnet()
net.hybridize()
x = net(input)
myloss = Myloss()
loss = myloss(x, y)
loss.backward()

@Sundrops there's no need to use make_loss in Gluon as all values can be used as head gradients now.

authors: Sundrops szha


text:
refine Nano setup directions

Description
After another run-through on a fresh Nano, plus trying out these new pip wheels, we found some refinements that should make the setup go a little more cleanly.

Nudging CI again...


text:
[v1.x] Onnx export support for batch_dot

Description
Onnx export support for batch_dot
Checklist
Essentials

[ ] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage
[ ] Code is well-documented

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

@mxnet-bot run ci [unix-gpu]


text:
0.11.1

@piiswrong

@szha what is this?

authors: domdivakaruni szha


text:
extend gdbinit script to support xtensa

Summary
This updates the gdbinit script to support xtensa arch.
Impact
None
Testing
Debugging

If someone more familiar with the arch can verify the TCB handling I would appreciate it

We tried this before both @masayuki2009 and myself.  This won't work the same way as other archs due to the differences between the logical registers shown in a window and the physical registers.  I haven't had any motivation to try and solve this issue for this script.  However, Espressif's OpenOCD fork (https://github.com/espressif/openocd-esp32/) does support Xtensa for NuttX.
Note that a bunch of refactoring was done to the NuttX's code in Espressif's OpenOCD, and one of the changes is the name of the RTOS: from "nuttx" to "NuttX".

@Ouss4
Please see https://github.com/apache/incubator-nuttx/pull/2015

My interest is usually on the info_nxthreads, since I can get good information without having to go into NSH console, which is sometimes inaccessible. What I always wondered is: do we really need to iterate through tcbs and switch/save to do this? I understand switch/save is needed when focusing on a thread or changing execution into another one, but couldn't the info_nxthreads command simply iterate through the TCBs? I suspect it could also be much faster.
Regarding openocd: the problem is that it requires building with a specific configuration. Maybe that could somehow improved (I tried once but went the wrong path). Still, I think we couldn't get info such as used stack like with gdb.

repo: apache/incubator-nuttx


text:
Fix examples of config.toml for oli

In bin/oli/README.md, some examples of config.toml are introduced like as follows.
[<profile_name>]
configuration_key1 = "value1"
configuration_key2 = "value2"

[<another_profile_name>]
configuration_key3 = "value3"
configuration_key4 = "value4"

But profile names should be profiles.<profile_name>.

Thanks!

authors: Xuanwo sarutak

repo: apache/incubator-opendal


text:
change file name

Standardising licence file name to markdown

Thanks for the suggestion.
I expect that as this repository is moved to the Apache organization (which is an ongoing process), this file will become simply LICENSE (see for example https://github.com/apache/kafka/blob/trunk/LICENSE).

@csantanapr @mrutkows is there a standard practice for how to name the license file? i see other apache repos use LICENSE.

I think both LICENSE and LICENSE.txt are fine. I see some projects uses LICESE.txt and some project uses LICENSE.

@KrishMunot  do you have an Apache CLI on file?

shall we rename CREDITS.txt as well?

@KrishMunot thanks for this suggestion. If I look at the two files:
https://github.com/KrishMunot/openwhisk/blob/c1b836d685466b799583e671f3bf5a7c03edec88/LICENSE.md
and
https://github.com/apache/incubator-openwhisk/blob/master/LICENSE.txt
Notice in the second:

https://help.github.com/articles/licensing-a-repository/
either is accepted - i think we'll keep it as text though since the license isn't really using markdown.

authors: KrishMunot daisy-ycguo rabbah

repo: apache/incubator-openwhisk


text:
[core] Remove the duplicated ContinuousCompactorFollowUpScanner

Purpose
The ContinuousCompactorFollowUpScanner is same with DeltaFollowUpScanner, So I think we can reuse the DeltaFollowUpScanner

Tests

API and Format

Documentation

CC @tsreaper

repo: apache/incubator-paimon


text:
Issue 286: migrate cluster k8s java

https://github.com/apache/incubator-pekko/issues/286
This MR copies the files from https://github.com/akka/akka-sample-cluster-kubernetes-java and converts them to pekko.

Issue fixed:
$  kubectl logs appka-54c88575f4-2hmp8 | grep MemberUp
[2023-04-25 16:06:12,996] [INFO] [org.apache.pekko.cluster.bootstrap.demo.DemoApp$MemberEventLogger] [] [Appka-pekko.actor.default-dispatcher-11] - MemberEvent: MemberUp(Member(pekko://Appka@10.244.0.121:17355, Up)) MDC: {sourceActorSystem=Appka, pekkoAddress=pekko://Appka@10.244.0.123:17355, pekkoSource=pekko://Appka/user/listener}
[2023-04-25 16:06:14,079] [INFO] [org.apache.pekko.cluster.bootstrap.demo.DemoApp$MemberEventLogger] [] [Appka-pekko.actor.default-dispatcher-15] - MemberEvent: MemberUp(Member(pekko://Appka@10.244.0.122:17355, Up)) MDC: {sourceActorSystem=Appka, pekkoAddress=pekko://Appka@10.244.0.123:17355, pekkoSource=pekko://Appka/user/listener}
[2023-04-25 16:06:14,088] [INFO] [org.apache.pekko.cluster.bootstrap.demo.DemoApp$MemberEventLogger] [] [Appka-pekko.actor.default-dispatcher-12] - MemberEvent: MemberUp(Member(pekko://Appka@10.244.0.123:17355, Up)) MDC: {sourceActorSystem=Appka, pekkoAddress=pekko://Appka@10.244.0.123:17355, pekkoSource=pekko://Appka/user/listener}

waiting for https://github.com/apache/incubator-pekko-samples/pull/37 to go in

repo: apache/incubator-pekko-samples


text:
Add OptionConverters.toScala methods for java Optional primitives

As pointed out in https://github.com/apache/incubator-pekko-http/pull/142#pullrequestreview-1417421049 we should do our best to avoid usage of the Rich* implicit classes within Java code especially since Java doesn't do inlining (unlike Scala).
I checked the pekko-http code and within Java sources we only use the .toScala methods (the Scala to Java conversion cases are done within Scala source which uses the standard extension methods).

@pjfanning @He-Pin @nvollmar Would you be able to review this? Its the last PR needed for https://github.com/apache/incubator-pekko-http/pull/142

repo: apache/incubator-pekko


text:
Fix NPE while unparsing single key DSL output

Fix NPE that can happen if unparsing of a single key DSL fails.

This has been tested using 4,310 table config files used in production at Linkedin, which have been tested for equality between the old and new configuration formats (both before and after merging) to ensure that the code behaves correctly across a large number of varied use cases. For obvious reasons, I'm not checking those in.
The overall code is also exercised as part of #2937.

This will get done once the config format is fully stable.

repo: apache/incubator-pinot


text:
Fixed the union-all elimiation case where some project expression is not an AttributeReference.

This PR fixed for the case where the only one child of the union-all query is non-empty, and the child has a non-AttributeReference project expression:
SELECT 1 AS a FROM not_empty
UNION ALL
SELECT 0 AS a FROM empty;

LGTM! Merging.

authors: jianqiao zuyu

repo: apache/incubator-quickstep