text:
[SUPPORT] Hudi Concurrent write (OCC) with upsert tables random errors

Hello,
We are running jobs on AWS EMR to compact tables stored in S3 and maintaining Athena tables through Hudi Hive sync.
Recently we started exploring Hudi multi writer and we were experiencing some issues when running concurrent Hudi upsert jobs with Hudi OCC where each job save data into a distinct partition.
The errors seem happen pretty randomly and get more frequently as the number of concurrent jobs increases.
Environment Description

Hudi version : 0.8.0

Spark version : 2.4.7

Hive version : 2.3.7

Hadoop version : 2.10.1

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) : no

AWS EMR: 5.33.0

Errors
Type 1 error:
FileAlreadyExistsException
User class threw exception: org.apache.hudi.exception.HoodieIOException: Failed to create file s3://hudi_debug/test_db/hudi_occ_debug_zoo_1_batch_v2_8_concurrent_jobs/.hoodie/20211011064521.commit.requested
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.createImmutableFileInPath(HoodieActiveTimeline.java:526)
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.createFileInMetaPath(HoodieActiveTimeline.java:487)
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.createNewInstant(HoodieActiveTimeline.java:147)
at org.apache.hudi.client.AbstractHoodieWriteClient.startCommit(AbstractHoodieWriteClient.java:714)
at org.apache.hudi.client.AbstractHoodieWriteClient.startCommitWithTime(AbstractHoodieWriteClient.java:700)
at org.apache.hudi.client.AbstractHoodieWriteClient.startCommitWithTime(AbstractHoodieWriteClient.java:691)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:185)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:145)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
...
Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: File already exists:s3://hudi_debug/test_db/hudi_occ_debug_zoo_1_batch_v2_8_concurrent_jobs/.hoodie/20211011064521.commit.requested
at com.amazon.ws.emr.hadoop.fs.s3.upload.plan.RegularUploadPlanner.checkExistenceIfNotOverwriting(RegularUploadPlanner.java:36)

Type 2 error:
IllegalArgumentException
java.lang.IllegalArgumentException
	at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:31)
	at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:396)
	at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:377)
	at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.saveAsComplete(HoodieActiveTimeline.java:154)
	at org.apache.hudi.client.AbstractHoodieWriteClient.commit(AbstractHoodieWriteClient.java:212)
	at org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:185)
	at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:121)
	at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:476)
	at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:222)
	at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:145)
	at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)

Type 3 error:
HoodieRollbackException
 User class threw exception: org.apache.hudi.exception.HoodieRollbackException: Failed to rollback s3://hudi_debug/test_db/hudi_occ_debug_zoo_1_batch_v2_8_concurrent_jobs commits 20211011064933
at org.apache.hudi.client.AbstractHoodieWriteClient.rollback(AbstractHoodieWriteClient.java:593)
at org.apache.hudi.client.AbstractHoodieWriteClient.rollbackFailedWrites(AbstractHoodieWriteClient.java:808)
at org.apache.hudi.client.AbstractHoodieWriteClient.rollbackFailedWrites(AbstractHoodieWriteClient.java:797)
at org.apache.hudi.client.AbstractHoodieWriteClient.lambda$clean$1cda88ee$1(AbstractHoodieWriteClient.java:648)
at org.apache.hudi.common.util.CleanerUtils.rollbackFailedWrites(CleanerUtils.java:135)
at org.apache.hudi.client.AbstractHoodieWriteClient.clean(AbstractHoodieWriteClient.java:647)
at org.apache.hudi.client.AbstractHoodieWriteClient.clean(AbstractHoodieWriteClient.java:630)
at org.apache.hudi.client.AbstractHoodieWriteClient.clean(AbstractHoodieWriteClient.java:661)
at org.apache.hudi.client.AbstractHoodieWriteClient.autoCleanOnCommit(AbstractHoodieWriteClient.java:494)
at org.apache.hudi.client.AbstractHoodieWriteClient.postCommit(AbstractHoodieWriteClient.java:431)
at org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:186)
at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:121)
at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:476)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:222)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:145)
...
 Caused by: org.apache.hudi.exception.HoodieIOException: Could not delete instant [==>20211011064933__commit__REQUESTED]
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.deleteInstantFile(HoodieActiveTimeline.java:196)
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.deletePending(HoodieActiveTimeline.java:173)
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.deleteInflightAndRequestedInstant(BaseRollbackActionExecutor.java:225)
at

Type 4 error:
Athena table wasn't updated by Hudi to include new partition properly.
Our understanding is that with hoodie.datasource.hive_sync.enable set to true, Hudi is suppose to keep the corresponding Athena table up-to-date for any schema change and and new partitions.
However, we found that sometime the partition is missing in Athena table after a job completed successfully.
We were able to query the data from the missing partition by in a Hudi snapshot table, which suggests that the data is there it's just that for some reason Athena table didn't pick up that partition.

Error Reproduction
To reproduce the errors, I built a Scala jar that contains the logic to generate a testing data and save with optimistic_concurrency_control enabled, and ran 8 spark steps concurrently.
package com.hudioccdebug

import org.apache.spark.sql.{DataFrame, SaveMode, SparkSession}
import org.apache.hadoop.hive.conf.HiveConf
import org.apache.spark.sql.functions.{current_timestamp, lit, udf}

object HudiOccDebug {
  val spark: SparkSession = SparkSession.builder.enableHiveSupport().getOrCreate()
  val sqlContext = new org.apache.spark.sql.SQLContext(spark.sparkContext)
  import sqlContext.implicits._

  def main(args: Array[String]): Unit = {
    val dbName = "test_db"
    val tableName = args(0)
    val rowCount = args(1).toInt
    val batchNum = args(2).toInt
    val lockProvider = args(3)

    val s3Location = "s3://hudi_debug"
    val r = scala.util.Random
    val n = 60

    for (i <- 1 to batchNum) {
      println(s"Starting batch $i")

      val m = (r.nextInt % n + n) % n
      println(s"sleep for $m seconds before upsert")
      Thread.sleep(m * 1000)

      val df = generateTestingDF(spark, rowCount, spark.sparkContext.applicationId)
      saveDf(s3Location, dbName, tableName, df, getHudiOptions(dbName, tableName, lockProvider))
    }
  }

  def generateTestingDF(spark: SparkSession, rowCount: Int = 1000000, partitionKey: String): DataFrame = {
    val uuid = udf(() => java.util.UUID.randomUUID().toString)
    def randomStringGen(length: Int) = scala.util.Random.alphanumeric.take(length).mkString

    val df = spark.sparkContext.parallelize(
      Seq.fill(rowCount){(randomStringGen(4), randomStringGen(4), randomStringGen(6))}, 10
    ).toDF("col_1", "col_2", "col_3").withColumn("partition", lit(partitionKey)).withColumn("uuid", uuid()).withColumn("ts", current_timestamp())
    df
  }

  def getHudiOptions(dbName:String, tableName: String, lockProvider: String) : Map[String, String] = {
    def getHiveServerURI: String = {
        val hiveMetastoreURIs = new HiveConf().get("hive.metastore.uris")
        val parsedMetastoreURI = if (hiveMetastoreURIs != null) hiveMetastoreURIs.replaceAll("thrift://", "") else ":"
        val hiveServer2URI = parsedMetastoreURI.substring(0, parsedMetastoreURI.lastIndexOf(":"))
        hiveServer2URI
    }
    var m = Map[String, String](
        "hoodie.table.name" -> tableName,
        "hoodie.datasource.write.table.name" -> tableName,
        "hoodie.consistency.check.enabled" -> "true",
        "hoodie.datasource.write.table.type" -> "COPY_ON_WRITE",
        "hoodie.datasource.write.recordkey.field" -> "uuid",
        "hoodie.datasource.write.keygenerator.class" -> "org.apache.hudi.keygen.ComplexKeyGenerator",
        "hoodie.datasource.write.partitionpath.field" -> "partition",
        "hoodie.datasource.write.precombine.field" -> "ts",
        "hoodie.parquet.max.file.size" -> String.valueOf(500 * 1024 * 1024),
        "hoodie.datasource.hive_sync.enable" -> "true",
        "hoodie.datasource.hive_sync.jdbcurl" -> s"jdbc:hive2://${getHiveServerURI}:10000",
        "hoodie.datasource.hive_sync.database" -> dbName,
        "hoodie.datasource.hive_sync.table" -> tableName,
        "hoodie.datasource.hive_sync.partition_fields" -> "partition",
        "hoodie.datasource.hive_sync.partition_extractor_class" -> "org.apache.hudi.hive.MultiPartKeysValueExtractor",
        "hoodie.datasource.write.operation" -> "upsert",
        "hoodie.fail.on.timeline.archiving" -> "false"
    )

    if (lockProvider.equals("ZookeeperBasedLockProvider")) {
      print("Using ZookeeperBasedLockProvider")
      m += (
        "hoodie.write.lock.zookeeper.url" -> "ip-10-0-227-209.ec2.internal",
        "hoodie.write.lock.zookeeper.port" -> "2181",
        "hoodie.write.lock.zookeeper.lock_key" -> "test_table",
        "hoodie.write.lock.zookeeper.base_path" -> "/test",
        "hoodie.write.concurrency.mode" -> "optimistic_concurrency_control",
        "hoodie.cleaner.policy.failed.writes" -> "LAZY",
        "hoodie.write.lock.provider" -> "org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider",
        "hoodie.write.lock.client.num_retries" -> "20",
        "hoodie.write.lock.wait_time_ms" -> "20000",
        "hoodie.write.lock.wait_time_ms_between_retry" -> "5000"
      )
    }

    m
  }

  def saveDf(
        s3Location:String,
        dbName:String,
        tableName:String,
        inputDf: DataFrame,
        options: Map[String, String]
    ): Unit = {
    inputDf.write
      .format("org.apache.hudi")
      .options(options)
      .mode(SaveMode.Append)
      .save("%s/%s/%s".format(s3Location, dbName, tableName))
  }

}

spark-submit \
--deploy-mode cluster \
--executor-memory 43g \
--driver-memory 43g \
--executor-cores 6 \
--class com.hudioccdebug.HudiOccDebug \
--conf "spark.serializer=org.apache.spark.serializer.KryoSerializer"\
--conf "spark.sql.hive.convertMetastoreParquet=false" \
--jars /usr/lib/hudi/hudi-spark-bundle.jar,/usr/lib/spark/external/lib/spark-avro.jar \
s3://hudi_debug/jars/hudi_occ_debug.jar "hudi_occ_debug_zoo_1_batch_v2_8_concurrent_jobs" "500000" "1" "ZookeeperBasedLockProvider"

@umehrot2 Do you have a patch that can be tested?

Hey folks. We have made few bug fixes in 0.10.0. Can you give it a try and let us know if the issues are resolved.

@ChiehFu : Did you get a chance to test out 0.10.0 for multi-writers. Let us know how it goes.

Closing this out since we fixed bunch of multi writer bugs with 0.10.0. Feel free to re-open this issue or open a new one if you encounter issues. would be happy to help.

authors: ChiehFu codope nsivabalan


text:
[SUPPORT] [BUG] BUCKET Index with Flink 1.14 + Hudi 0.11 (master）

Describe the problem you faced
USE BUCKET index， when flink chkp that throw Exception java.util.NoSuchElementException: No value present in Option
A clear and concise description of the problem.
To Reproduce
BUCKET index, COW table , insert op
0. chkp

hudi config

'connector' = 'hudi',
  'path' = 'hdfs://xxxx',
  'index.bootstrap.enabled' = 'false',
  'index.state.ttl' = '0.0',
  'index.type' = 'BUCKET',
  'hoodie.parquet.compression.codec'= 'snappy',
  'table.type' = 'COPY_ON_WRITE',
  'write.operation' = 'insert',
  'write.precombine' = 'true',
  'write.precombine.field' = 'ts',
  'write.tasks' = '1',
  'write.bucket_assign.tasks' = '1',
  'hive_sync.enable'='false',
  'hive_sync.table'='xxxx',
  'hive_sync.db'='xxx',
  'hive_sync.mode' = 'hms',
  'hive_sync.metastore.uris' = 'thrift://xxxx.com:9083',
  'hive_sync.skip_ro_suffix' = 'true',
  'write.insert.cluster' = 'true',
  'write.ignore.failed' = 'true',

Expected behavior
A clear and concise description of what you expected to happen.
Environment Description

Hudi version : master 0.11

Flink version : 1.14.3

Hadoop version : 3.0.0-cdh6

Storage (HDFS/S3/GCS..) : HDFS

Running on Docker? (yes/no) : no

Additional context
Add any other context about the problem here.
Stacktrace
2022-03-17 20:38:31
java.io.IOException: Could not perform checkpoint 3 for operator bucket_write -> Sink: clean_commits (1/1)#2.
	at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1274)
	at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:147)
	at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.triggerCheckpoint(SingleCheckpointBarrierHandler.java:287)
	at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.access$100(SingleCheckpointBarrierHandler.java:64)
	at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler$ControllerImpl.triggerGlobalCheckpoint(SingleCheckpointBarrierHandler.java:493)
	at org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.triggerGlobalCheckpoint(AbstractAlignedBarrierHandlerState.java:74)
	at org.apache.flink.streaming.runtime.io.checkpointing.AbstractAlignedBarrierHandlerState.barrierReceived(AbstractAlignedBarrierHandlerState.java:66)
	at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.lambda$processBarrier$2(SingleCheckpointBarrierHandler.java:234)
	at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.markCheckpointAlignedAndTransformState(SingleCheckpointBarrierHandler.java:262)
	at org.apache.flink.streaming.runtime.io.checkpointing.SingleCheckpointBarrierHandler.processBarrier(SingleCheckpointBarrierHandler.java:231)
	at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.handleEvent(CheckpointedInputGate.java:181)
	at org.apache.flink.streaming.runtime.io.checkpointing.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:159)
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110)
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:496)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 3 for operator bucket_write -> Sink: clean_commits (1/1)#2. Failure reason: Checkpoint was declined.
	at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:265)
	at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:170)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:348)
	at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:233)
	at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:206)
	at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:186)
	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:605)
	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:315)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$14(StreamTask.java:1329)
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1315)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:1258)
	... 22 more
Caused by: java.util.NoSuchElementException: No value present in Option
	at org.apache.hudi.common.util.Option.get(Option.java:88)
	at org.apache.hudi.io.HoodieMergeHandle.<init>(HoodieMergeHandle.java:118)
	at org.apache.hudi.io.FlinkMergeHandle.<init>(FlinkMergeHandle.java:70)
	at org.apache.hudi.io.FlinkConcatHandle.<init>(FlinkConcatHandle.java:53)
	at org.apache.hudi.client.HoodieFlinkWriteClient.getOrCreateWriteHandle(HoodieFlinkWriteClient.java:480)
	at org.apache.hudi.client.HoodieFlinkWriteClient.insert(HoodieFlinkWriteClient.java:170)
	at org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$0(StreamWriteFunction.java:181)
	at org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:461)
	at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
	at org.apache.hudi.sink.StreamWriteFunction.flushRemaining(StreamWriteFunction.java:454)
	at org.apache.hudi.sink.StreamWriteFunction.snapshotState(StreamWriteFunction.java:131)
	at org.apache.hudi.sink.common.AbstractStreamWriteFunction.snapshotState(AbstractStreamWriteFunction.java:157)
	at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)
	at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:87)
	at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:219)
	... 33 more

see https://github.com/apache/hudi/pull/5018

see #5018

thx, It looks good, and run without Exception.

source: gharchive/issue

authors: Guanpx wxplovecc

repo: apache/hudi

license_type: permissive

license_source: github-api


text:
[HUDI-5400] Fix read issues when Hudi-FULL schema evolution is not enabled

Change Logs
Prior to Hudi's FULL Schema evolution (HFSE) support, Hudi relies on Avro's schema-resolution to perform schema evolution.
The exhaustive list of permitted schema-changes that Avro's schema-resolution allows for can be found here:
https://avro.apache.org/docs/1.10.2/spec.html#Schema+Resolution
A summary of the type changes  is listed down below:
Supported cast conversions:
 - Integer => Long, Float, Double, Decimal*, String*
 - Long => Float, Double, Decimal*, String*
 - Float => Double, Decimal*, String*
 - Double => Decimal*, String*
 - Decimal => Decimal*, String*
 - String => Byte, Decimal*, Date*
 - Byte=> String
 - Date => String*

*type conversions that are supported by HFSE, but not in Native Avro's schema-resolution

The current write execution flow is as such:

deduceWriterSchema to check if the incoming schema is compatible with table's schema (via org.apache.hudi.avro.AvroSchemaCompatibility.ReaderWriterCompatibilityChecker#calculateCompatibility)
deduceWriterSchema's validation is an adaptation of Avro's schema compatibility check (org.apache.avro.SchemaCompatibility.ReaderWriterCompatiblityChecker#calculateCompatibility); if Avro permits such operation, allow execution to proceed
As such if there are implicit schema changes which are compatible with Avro's schema resolution feature, HFSE does not need to be enabled
If writer is writing to a different filegroup, this filegroup will be written with a new schema; while existing filegroups that are not written to will contain the old schema

When reading:

When reading, the same schema will be used for all the filegroup since nothing is written to .schema.
Parquet reader will throw errors due to type mismatches when reading

This PR fixes the Spark-read issues when implicit schema changes are made without enabling HFSE.
The scope of this fix is limited to Spark-Read + Spark-Write.
TODO: Check if issue exists in Flink reader/writer.
Impact
None; no public APIs changed.
Risk level (write none, low medium or high below)
low
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[x] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

@voonhous
Maybe we need a parameter to control this feature, not all tables need to follow this logic

@voonhous Maybe we need a parameter to control this feature, not all tables need to follow this logic

Hmmm, CMIIW, Hudi has been relying on ASR for schema resolution since Hudi-0.7. As such, I was under the impression that this should be a default behaviour.
Nonetheless, a configuration key can be introduced where in this behaviour is enabled by default.
However, validation will need to be performed such that the choice between ASR/HFSE is mutually exclusive. i.e. if ASR is enabled, HFSE should be disabled and vice-versa.

Looks good.
maybe we also need a pr for flink

@voonhous Maybe we need a parameter to control this feature, not all tables need to follow this logic

Hmmm, CMIIW, Hudi has been relying on ASR for schema resolution since hudi-0.7. As such, I was under the impression that this should be a default behaviour.
Nonetheless, a configuration key can be introduced where in this behaviour is enabled by default.
However, validation will need to be performed such that the choice between ASR/HFSE is mutually exclusive. i.e. if ASR is enabled, HFSE should be disabled and vice-versa. WDYT?

@xiarixiaoyao I looked at the code and realised that there is no way validate configuration values based on other configuration values.
I wanted to add a AVRO_SCHEMA_RESOLUTION_ENABLE configuration key with the description:
Enable support for schema evolution using Avro's Schema Resolution (ASR). This configuration is mutually exclusive to Hudi's Full/Comprehensive Schema Evolution (HFSE) feature via the configuration key (hoodie.schema.on.read.enable).

The choice between ASR/HFSE is mutually exclusive. i.e. if ASR is enabled, HFSE should be disabled and vice-versa.

HFSE will take precedence over ASR. i.e. Enabling both HFSE and ASR will cause Hudi to default to HFSE for schema evolution.

Given that this is the intended behaviour and lack of configuration validation, I see no benefit for introducing AVRO_SCHEMA_RESOLUTION_ENABLE.
Since SCHEMA_EVOLUTION_ENABLE will take precedence over AVRO_SCHEMA_RESOLUTION_ENABLE, I think we can rely on the former (SCHEMA_EVOLUTION_ENABLE) to determine if ASR should be used.
If  SCHEMA_EVOLUTION_ENABLE is enabled, use HFSE, else, fallback to ASR.
WDYT?

@voonhous Maybe we need a parameter to control this feature, not all tables need to follow this logic

Hmmm, CMIIW, Hudi has been relying on ASR for schema resolution since hudi-0.7. As such, I was under the impression that this should be a default behaviour.
Nonetheless, a configuration key can be introduced where in this behaviour is enabled by default.
However, validation will need to be performed such that the choice between ASR/HFSE is mutually exclusive. i.e. if ASR is enabled, HFSE should be disabled and vice-versa. WDYT?

@xiarixiaoyao I looked at the code and realised that there is no way validate configuration values based on other configuration values.
I wanted to add a AVRO_SCHEMA_RESOLUTION_ENABLE configuration key with the description:
Enable support for schema evolution using Avro's Schema Resolution (ASR). This configuration is mutually exclusive to Hudi's Full/Comprehensive Schema Evolution (HFSE) feature via the configuration key (hoodie.schema.on.read.enable).

The choice between ASR/HFSE is mutually exclusive. i.e. if ASR is enabled, HFSE should be disabled and vice-versa.

HFSE will take precedence over ASR. i.e. Enabling both HFSE and ASR will cause Hudi to default to HFSE for schema evolution.

Given that this is the intended behaviour and lack of configuration validation, I see no benefit for introducing AVRO_SCHEMA_RESOLUTION_ENABLE.
Since SCHEMA_EVOLUTION_ENABLE will take precedence over AVRO_SCHEMA_RESOLUTION_ENABLE, I think we can rely on the former (SCHEMA_EVOLUTION_ENABLE) to determine if ASR should be used.
If SCHEMA_EVOLUTION_ENABLE is enabled, use HFSE, else, fallback to ASR.
WDYT?
agree

@voonhous
pls rebase code，
once ci pass，we can merge it .

@voonhous pls rebase code， once ci pass，we can merge it .

Done!

@xiarixiaoyao I have added support for Hudi tables that are schema-evolved via ASR for Spark2.4.
Can you please help to review the PR again?
Thank you!

@voonhous
Thank you for your support for spark2.4, although I personally think we don't need to support 2.4.
let‘s  extract  buildImplicitSchemaChangeInfo and isDataTypeEqual to a helper class to reuse.

@voonhous Thank you for your support for spark2.4, although I personally think we don't need to support 2.4. let‘s extract buildImplicitSchemaChangeInfo and isDataTypeEqual to a helper class to reuse.

Done!

@hudi-bot run azure

@voonhous
Thanks for your contribution

source: gharchive/pull-request

authors: voonhous xiarixiaoyao


text:
Result malformed.

diagrams are not drawn following this error:
after this query:
SELECT * from cypher('text_test', $$
          MATCH (V)-[R:connects]-(V2)
          RETURN V,R,V2
$$) as (V agtype, R agtype, V2 agtype);

$ SELECT * from cypher('text_test', $$ MATCH (V)-[R:connects]-(V2) RETURN V,R,V2 $$) as (V agtype, R agtype, V2 agtype);
Converting circular structure to JSON
--> starting at object with constructor 'Object'
--- property 'properties' closes the circle
info: ::ffff:172.17.0.1 - - [22/Nov/2021:13:44:17 +0000] "POST /api/v1/cypher HTTP/1.1" 500 47
 {"timestamp":"2021-11-22 13:44:17"}
info: ::ffff:172.17.0.1 - - [22/Nov/2021:14:06:22 +0000] "GET /api/v1/db HTTP/1.1" 200 134
 {"timestamp":"2021-11-22 14:06:22"}
info: ::ffff:172.17.0.1 - - [22/Nov/2021:14:06:22 +0000] "GET /api/v1/db/metaChart HTTP/1.1" 500 2
 {"timestamp":"2021-11-22 14:06:22"}
info: ::ffff:172.17.0.1 - - [22/Nov/2021:14:06:23 +0000] "GET /api/v1/db/meta HTTP/1.1" 304 -
 {"timestamp":"2021-11-22 14:06:23"}
error: syntax error at or near "MATCH"
    at Parser.parseErrorMessage (/app/backend/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (/app/backend/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/app/backend/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/app/backend/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (events.js:376:20)
    at addChunk (internal/streams/readable.js:309:12)
    at readableAddChunk (internal/streams/readable.js:284:9)
    at Socket.Readable.push (internal/streams/readable.js:223:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:188:23) {
  length: 93,
  severity: 'ERROR',
  code: '42601',
  detail: undefined,
  hint: undefined,
  position: '1',
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: undefined,
  table: undefined,
  column: undefined,
  dataType: undefined,
  constraint: undefined,
  file: 'scan.l',
  line: '1129',
  routine: 'scanner_yyerror'
}
info: ::ffff:172.17.0.1 - - [22/Nov/2021:14:14:52 +0000] "POST /api/v1/cypher HTTP/1.1" 500 47
 {"timestamp":"2021-11-22 14:14:52"}
info: ::ffff:172.17.0.1 - - [22/Nov/2021:14:15:41 +0000] "POST /api/v1/cypher HTTP/1.1" 200 137
 {"timestamp":"2021-11-22 14:15:41"}
TypeError: Converting circular structure to JSON
    --> starting at object with constructor 'Object'
    --- property 'properties' closes the circle
    at JSON.stringify (<anonymous>)
    at stringify (/app/backend/node_modules/express/lib/response.js:1123:12)
    at ServerResponse.json (/app/backend/node_modules/express/lib/response.js:260:14)
    at _callee$ (/app/backend/build/controllers/cypherController.js:59:33)
    at tryCatch (/app/backend/node_modules/regenerator-runtime/runtime.js:63:40)
    at Generator.invoke [as _invoke] (/app/backend/node_modules/regenerator-runtime/runtime.js:293:22)
    at Generator.next (/app/backend/node_modules/regenerator-runtime/runtime.js:118:21)
    at asyncGeneratorStep (/app/backend/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
    at _next (/app/backend/node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
info: ::ffff:172.17.0.1 - - [22/Nov/2021:14:16:03 +0000] "POST /api/v1/cypher HTTP/1.1" 500 154
 {"timestamp":"2021-11-22 14:16:03"}
info: ::ffff:172.17.0.1 - - [22/Nov/2021:14:16:13 +0000] "GET /api/v1/db/meta HTTP/1.1" 200 217
 {"timestamp":"2021-11-22 14:16:13"}

Funny, I ran into the same error today as well.
What I found is that it works fine as soon as the relations have at least one property set.
I.e. I get the graph displayed when creating the relations like this:
MATCH (a:Person),(b:Book) WHERE a.name='Jack' AND b.name='Hitchhiker' CREATE (a)-[r:Owns {name: a.name+'->'+b.name} ]->(b)
while the following results in the described error:
MATCH (a:Person),(b:Book) WHERE a.name='Jack' AND b.name='Hitchhiker' CREATE (a)-[r:Owns]->(b)

Just noticed: This is most likely the same issue as #9

so, each edge must be added a property?

@david-m-m I have a looping issue, can you please have a look at it?

so, each edge must be added a property? Thanks for the insight, I would have to amend my code.

Looks like it. Though I'm not familiar with the code, found out more by trial-and-error.
Hopefully the developers of this project can gain & give more insight into this issue.

@david-m-m I have a looping issue, can you please have a look at it?

First: I'm not a developer and am not (yet?) deep into the matter.
But I had issues addressing objects by their ID (when fiddling around on the console). So it may not be possible or work different that what we think.

Thanks for waiting response.
I can't reproduce this issue. Are you using Docker?

yes, i'm using docker here,

@afidegnum
I just updated it on latest tag. I'll close this issue :)

authors: afidegnum david-m-m emotionbug

repo: apache/incubator-age-viewer


text:
[AIRFLOW-2525] Fix a bug introduced by commit dabf1b9

Make sure you have checked all steps below.
JIRA

[x] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-2525
In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a JIRA issue.

Description

[x] Here are some details about my PR, including screenshots of any UI changes:

The previous commit on this issue (#3421)
introduced a new bug on COPY FROM operation.
This PR fixes it by adding a new parameter
to PostgresHook.copy_expert for specifying
file opening mode, instead of using
read and write file opening mode.
Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Fixed tests.hooks.test_postgres_hook:TestPostgresHook.test_copy_expert in accordance with PostgresHook.copy_expert's change.

Tested with PostgreSQL 9.5.12 as follows:

Initial state:
airflow=# SELECT * FROM t;
  c
-----
 foo
 bar
 baz
(3 rows)

run COPY TO:
In [1]: from airflow.hooks.postgres_hook import PostgresHook
/home/sekikn/a3/lib/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)

In [2]: h = PostgresHook()

In [3]: h.copy_expert("COPY t TO STDOUT", "/tmp/t")
[2018-05-26 10:14:54,078] {base_hook.py:83} INFO - Using connection to: localhost

dumped correctly:
$ cat /tmp/t
foo
bar
baz

the opposite operation:
In [4]: h.copy_expert("COPY t FROM STDIN", "/tmp/t", mode="r")
[2018-05-26 10:15:23,329] {base_hook.py:83} INFO - Using connection to: localhost

loaded successfully:
airflow=# SELECT * FROM t;
  c
-----
 foo
 bar
 baz
 foo
 bar
 baz
(6 rows)

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[x] In case of new functionality, my PR adds documentation that describes how to use it.

When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

[x] Passes git diff upstream/master -u -- "*.py" | flake8 --diff

@jgao54 Would you take a look? Sorry for taking your time.

Codecov Report

Merging #3423 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master   #3423      +/-   ##
=========================================
+ Coverage   76.59%   76.6%   +<.01%
=========================================
  Files         203     203
  Lines       15027   15027
=========================================
+ Hits        11510   11511       +1
+ Misses       3517    3516       -1

Impacted Files
Coverage Δ

airflow/hooks/postgres_hook.py
92.59% <100%> (ø)
:arrow_up:

airflow/models.py
87.35% <0%> (+0.04%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 432ac71...0177978. Read the comment docs.

Wouldn't it be friendlier by just opening the file in rw mode by default?

Thanks for the comment @Fokko @jgao54! Updated the PR.
Tested with PostgreSQL 9.5.12 and confirmed that it works as follows:

Initial state:

airflow=# SELECT * FROM t;
  c
-----
 foo
 bar
 baz
(3 rows)

$ test -f /tmp/t
$ echo $?
1

Test COPY TO (new file):

In [1]: from airflow.hooks.postgres_hook import PostgresHook

In [2]: h = PostgresHook()

In [3]: h.copy_expert("COPY t TO STDOUT", "/tmp/t")

$ cat /tmp/t
foo
bar
baz

Test COPY FROM:

In [4]: h.copy_expert("COPY t FROM STDIN", "/tmp/t")

airflow=# SELECT * FROM t;
  c
-----
 foo
 bar
 baz
 foo
 bar
 baz
(6 rows)

Test COPY TO (overwrite, # of existing lines in the file < newly output lines):

In [5]: h.copy_expert("COPY t TO STDOUT", "/tmp/t")

$ cat /tmp/t
foo
bar
baz
foo
bar
baz

Test COPY TO (overwrite, # of existing lines in the file > newly output lines):

airflow=# DELETE FROM t WHERE c <> 'foo';
DELETE 4
airflow=# SELECT * FROM t;
  c
-----
 foo
 foo
(2 rows)

In [6]: h.copy_expert("COPY t TO STDOUT", "/tmp/t")

$ cat /tmp/t
foo
foo

one nit-pick otherwise lgtm

Thanks for the review @jgao54, fixed and rebased!

authors: Fokko codecov-io jgao54 sekikn

repo: apache/incubator-airflow

license_source: bigquery


text:
[AIRFLOW-3011][CLI] Add cmd function printing config

…efined
Make sure you have checked all steps below.
Jira

[x] My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-3011] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-3011
In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

[x] Here are some details about my PR, including screenshots of any UI changes:
Example of usage:
Bash command: airflow config`
Example of printed config is described on a screenshoot below.

Output of the same command could be redirected into file and then used for further operations.

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[ ] In case of new functionality, my PR adds documentation that describes how to use it.

When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

[ ] Passes git diff upstream/master -u -- "*.py" | flake8 --diff

Codecov Report

Merging #3867 into master will decrease coverage by 0.01%.
The diff coverage is 20%.

@@            Coverage Diff             @@
##           master    #3867      +/-   ##
==========================================
- Coverage   77.51%   77.49%   -0.02%
==========================================
  Files         200      200
  Lines       15815    15820       +5
==========================================
+ Hits        12259    12260       +1
- Misses       3556     3560       +4

Impacted Files
Coverage Δ

airflow/bin/cli.py
65.2% <20%> (-0.28%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7cd3a85...f494dd0. Read the comment docs.

@dispower  the output (to the file) should not contain the logs of the cli, it defeats the purpose (to get a working config file). Not sure how to handle that though.

Can you give an example when you want to use this? The config is likely to contain secrets, so we should be careful exposing this.

@Fokko my initial use was to:

create a container that was setup with environment variables (including some in the entrypoint.sh, as in https://github.com/puckel/docker-airflow) and no custom configuration.
generate the config file to persist it so that when executing commands in the container, airflow CLI would work without having to setup all the environment variables needed

The problem being that the CLI (and I suppose other daemons) needs some of the same environment variables than the running webserver.
My current workaround is to run commands through entrypoint.sh, which sets up all the needed environment variables. This works as long as the entrypoint script is not making mutations to the filesystem.
Maybe the original issue wasn't the correct way to tackle this problem though... I will leave that decision to the airflow team :)

Thanks @victornoel
This won't fix the problem, since the original configuration file will be read, and the environment variables are ignored. Then you should generate a config using the configuration class: https://github.com/apache/incubator-airflow/blob/master/airflow/configuration.py
This isn't trivial though since you will need to iterate over all the environment variables, and populate the keys in the conf file, while keeping the order into account (environment variables have priority over the config file).

@Fokko thanks for clarifying this: basically what you just described is the original requirement of the jira issue: https://issues.apache.org/jira/browse/AIRFLOW-3011
So the PR does not solve the opened issue at all @dispower :)

authors: Fokko codecov-io dispower victornoel


text:
[AIRFLOW-XXX] Removed spot in Airflow logo

Make sure you have checked all steps below.
Jira

[x] My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-XXX
In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

[x] Here are some details about my PR, including screenshots of any UI changes:

The pin-large.png logo contains a small spot which looks like a blemish, and not a purposeful spot. I see people all over the internet copy this picture and see the spot everywhere. E.g. the GCP Composer documentation:

Seems like a mistake, so I removed the spot from the pin-large.png.

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

I don't test pictures.
Commits

[x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[x] In case of new functionality, my PR adds documentation that describes how to use it.

When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

N/A
Code Quality

[x] Passes flake8

Nice spot

authors: BasPH Fokko


text:
[AIRFLOW-3233] Fix deletion of DAGs in the UI

Make sure you have checked all steps below.
Jira

[X] My PR addresses the following Airflow Jira issues and references them in the PR title.

Description

[X] Here are some details about my PR, including screenshots of any UI changes:
Currently deleting DAGs that exist on the scheduler but not the webserver is not possible because the DAG id in the deletion URL comes from the webserver DAG object.
e.g. http://localhost:8080/delete?dag_id= instead of http://localhost:8080/delete?dag_id=example_kubernetes_executor

Tests

[X] My PR adds the following unit tests OR does not need testing for this extremely good reason:
UI Fix, tested both RBAC and regular.

Commits

[X] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[X] In case of new functionality, my PR adds documentation that describes how to use it.

When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

[X] Passes flake8

Codecov Report

Merging #4069 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #4069   +/-   ##
=======================================
  Coverage   75.91%   75.91%
=======================================
  Files         199      199
  Lines       15961    15961
=======================================
  Hits        12117    12117
  Misses       3844     3844

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f5e3b03...71bb5c2. Read the comment docs.

Should be possible to test this by creating a Dag object in the DB and ensuring the correct delete link appears in the output

authors: aoen ashb codecov-io


text:
APEXMALHAR-1950 #resolve #comment added stable annotation to semantic…

… version inclusion list

@tweise please review and merge

repo: apache/incubator-apex-malhar


text:
Fixes and tests twoPass Mac LE bug

Fixes bug where data containing CRs aren't replaced with LFs during
twoPass testing.
DAFFODIL-2209

Codecov Report

Merging #271 into master will decrease coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #271      +/-   ##
==========================================
- Coverage   80.38%   80.36%   -0.03%
==========================================
  Files         395      395
  Lines       23271    23272       +1
  Branches     2699     2699
==========================================
- Hits        18706    18702       -4
- Misses       4565     4570       +5

Impacted Files
Coverage Δ

...org/apache/daffodil/tdml/TDMLInfosetInputter.scala
82.35% <100%> (+0.35%)
:arrow_up:

...scala/org/apache/daffodil/tdml/TDMLException.scala
70% <0%> (-30%)
:arrow_down:

...ache/daffodil/xml/DaffodilConstructingLoader.scala
84.21% <0%> (-5.27%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9c5a6b1...6a1eff8. Read the comment docs.

+1

authors: codecov-io olabusayoT stevedlawrence

repo: apache/incubator-daffodil


text:
All source code files must begin with the Apache License Header

Description
Subtask of #1909
Describe the solution you'd like
Add apache header to every single source code files
header can be found from all apache project, like https://github.com/apache/servicecomb-java-chassis/blob/master/core/src/main/java/org/apache/servicecomb/core/bootstrap/SCBBootstrap.java
Has the Feature been Requested Before?
No

ok, i'll add it

ok, i'll add it

finished

authors: klesh likyh

repo: apache/incubator-devlake


text:
[Question][Github] Automatically add new github repositories to data scopes

Question
We have new repositories being added or removed to our Github organization.
What would be the best way to automatically synch the Github repos to the data-scopes in DevLake?
Is there an existing script that could provide such feature?

There isn't a script for this purpose currently.  I suggest using the API to accomplish this.

Thanks klesh.
I'm analysing the API calls performed by the frontend, so I can reproduce the necessary steps.
What I see is a PATCH operation against the api/blueprints/<x> endpoint.
Do I understand correctly I need to push the entire JSON payload, containing the entire list of repos, every time I perform a change?

The PATCH operation only updates specific fields of a resource. To remove elements from a list, you'll need to provide a complete, updated list in the request body. You can omit unchanged fields like name if they don't require modification.

That was indeed the case!
I'll proceed with my experiments. Thank you for your help. I'll close this.

authors: klesh nicolavolpini


text:
[Question] How to specify the checkpoint directory when setting up flink nodes and task restart

设置flink节点，处理实时数据的时候，如何设置checkpoint，当任务彻底挂掉需要人为干预启动的时候，通过指定的checkpoint目录重启。
当任务需要修改部分代码重启，在将任务下线的时候指定savepoint，flink on yarn 目前停止任务是直接通过yarn application -kill，并不能指定savepoint
Which version of DolphinScheduler:
-[1.3.3-preview]

English is global and necessary, please update the title in English, thanks

English is global and necessary, please update the title in English, thanks

@wuzhjian
Under yarn cluster deploy mode, you can

set -yD state.backend=filesystem -yD state.checkpoints.dir=hdfs://namenode:9000/flink/checkpoints -yD state.savepoints.dir=hdfs://namenode:9000/flink/savepoints in other parameters (其它参数) to configure state options
trigger a savepoint manually by flink savepoint <jobId> [savepointDirectory] -yid <yarnAppId>
set -s <savepointPath> -n in other parameters to restore from the specific savepoint path

In other deploy mode such as local and session, you can only set state.backend, state.checkpoints.dir and state.savepoints.dir in the configuration file flink-conf.yaml.

@wuzhjian
Under yarn cluster deploy mode, you can

set -yD state.backend=filesystem -yD state.checkpoints.dir=hdfs://namenode:9000/flink/checkpoints -yD state.savepoints.dir=hdfs://namenode:9000/flink/savepoints in other parameters (其它参数) to configure state options
trigger a savepoint manually by flink savepoint <jobId> [savepointDirectory] -yid <yarnAppId>
set -s <savepointPath> -n in other parameters to restore from the specific savepoint path

In other deploy mode such as local and session, you can only set state.backend, state.checkpoints.dir and state.savepoints.dir in the configuration file flink-conf.yaml.

Close this question since it has been replied

authors: chengshiwen wuzhjian xingchun-chen

repo: apache/incubator-dolphinscheduler


text:
[Improvement][worker] Optimize KillYarnJob

When KillYarnJob, determine whether the target host is the local machine, if it is, rpc communication is not required
org.apache.dolphinscheduler.server.worker.processor.TaskKillProcessor#killYarnJob

Assign to me

authors: CalvinKirs ruanwenjun


text:
Unifies load flow

What is problem of old load framework
There are different type of load use the different framework of load.

The most directly problem is that the same feature of load has been implemented many times in different type of load. The duplicated code is too complex to maintenance and enhance.

Also, the plan of broker and mini load cost more disk I/O and network transform. The performance of stream load is better then broker and mini load

Due to the non-streaming plan of broker load, it could not support to load the large size of file in limited memory.

What is the solution
Add a new framework of load which uses the unified load plan(streaming load plan) in be and quickly schedule logic in fe.
The design of solution

According to the new framework of load in fe:
The new stage of load consists of PENDING -> LOADING -> FINISHED which removes the ETL stage. Also, the new framework uses the TaskCallback and TxnStateChangeCallback instead of the LoadChecker which is used to trigger the next step of load.
In the new framework
Step1: New scheduler named LoadScheduler only support the broker load now. It will pick the PENDING jobs and call the execute function of jobs which is performed to begin txn and submit tasks. After this step, the state of job will be changed to LOADING.
Step2.0~2.N: The OnTaskFinished will be invoked after the task is finished. It is used to update the progress of job or trigger the next step of load. If all of tasks are finished, the txn will be committed.
Step3: The AfterVisible will be invoked after txn is visible which is used to change job state to FINISHED.

According to the broker load
Step1: The pending task will be submit by execute function of broker load job. There are preparations that need to be done on pending task.
Step2.1: OnPendingTaskFinished which is sub-function of OnTaskFinsihed will be invoked after pending task is finished. It is used to submit the loading task which is created based on attachment of pending task.
Step2.2: OnLoadingTaskFinished which is sub-function of OnTaskFinsihed too will be invoked after loding task is finished. It is used to record the commit info and commit txn when all of task has been finished.
Step3: The AfterVisible will be invoked after txn is visible which is used to change job state to FINISHED.

According to the new load plan:
The broker load, mini load, multi mini load will use the BrokerScanNode and OlapTableSink instead of CSVScanNode and DataSplitSink. The new load plan is streaming which support to load the huge size of file without Exceeding memory limit.

https://github.com/apache/incubator-doris/pull/1076

#https://github.com/apache/incubator-doris/pull/1076

https://github.com/apache/incubator-doris/pull/1134 Change the label of broker load and fix the LabelAlreadyUsedException

#1076

https://github.com/apache/incubator-doris/pull/1191

#1191 1. Add persistence of new load 2. Change non-streaming insert to streaming insert

#1323 The streaming mini load has been implemented in this commit

#1338 Fix the hung bug in streaming broker load

#1335 Add the streaming broker load in load jobs proc

I propose to test the performance of streaming broker load and mini load.
The environment:
Node 1:

Memory: about 3G
Disk: SATA about 2T
Network:

Node 2,3,4:

Memory: about 95G
Disk:  SSD about 369G
Network:

The prepare of test:
0. Install doris:

One frontend and three backends.
Three backends are located in different node.
One frontend is located Node1.

Create table: The table has only one tablet and 3 replicas.

You can use store_sales in TPC-DS. It is may be better.
The create table statement of store_sales can be elaborated in place.

repo: apache/incubator-doris


text:
loading empty segment group failed

Describe the bug
When segment group is loaded, it will fail because the empty flag is not set.
Reason
When DeltaWriter write a new AlphaRowset, the rowset can not be flush, But the empty flag is set in flush function of AlphaRowset, so the empty flag of segment group is invalid, leading to this problem.
It will eventually cause query failure.

Fixed by #865


text:
Coordinator console doesn't show the port of the historical node

Historical nodes can share the same host, that way having a single IP, while the different ports aren't visible in a tier nodes usage summary layout.

As well as it's not present in a segment description layout.

That've complicated operators work during production issues a bunch of times.

Usually, if the historical nodes share the same host, why not just start one historical node with higher memory and more threads? I just wonder that.

@dragonls cost reduction 😉

I suspect this has the greatest likelihood of showing up in a containerized environment where you may or may not have guarantees over isolation at the network level. In the world of "make one config you know works and stamp it out with only superficial care for what the underlying hardware looks like" (aka: the promise of containers), a scenario can easily arise where you have multiple JVMs on the same network interface, but different ports.

The new console as of Druid 0.16.0

Shows all the ports for all the nodes and even tells you is they are TLS or plaintext.
Closing as solved.

authors: dragonls drcrallen egor-ryashin vogievetsky

repo: apache/incubator-druid


text:
力导向布局图形加载时，绑定的加载完毕事件rendered 和 finished 两个事件效果一样，请问是我的使用有什么问题吗

Version
4.1.0-release
Steps to reproduce
力导向布局图形加载时，每一帧变化打印rendered，图形全部渲染结束时打印finished
使用到的事件
使用方法
实际效果
What is expected?
力导向布局图形加载时，每一帧变化打印rendered，图形全部渲染结束时打印finished
What is actually happening?
每一帧变化都打印 rendered和finished

看起来 和 zrender 里面的触发机制有关系，mark + 1

看起来 和 zrender 里面的触发机制有关系，mark + 1

看起来 和 zrender 里面的触发机制有关系，mark + 1

可能是吧，不知道官方有没有解决这个问题

现在因为上层没有去管力引导的布局，所以渲染完就算结束了，但是实际上布局会异步进行，这一块是需要优化一下

authors: WantK pissang wf123537200

repo: apache/incubator-echarts


text:
Aira Separator Update

Brief Information
This pull request is in the type of:

[ ] bug fixing
[ ] new feature
[x] others

What does this PR do?
Changed the separator of generating description of the chart.
Fixed issues

Details
Before: What was the problem?
There are no space after comma in the description of the chart (only in english version).
After: How is it fixed in this PR?
Change
separator: {
                middle: ',',
                end: '.'
            }

to
separator: {
                middle: ', ',
                end: '.'
            }

Are there any API changes?
No.
Merging options

[ ] Please squash the commits into a single one when merge.

Other information

Please modify the source file instead of distribution file


text:
[GLUTEN-1632][CH]Daily Update Clickhouse Version (20240320)

Auto commit by gluten daily build, please check the build status and merge it if it's green.

Run Gluten Clickhouse CI

repo: apache/incubator-gluten


text:
[GLUTEN-6863][VL] Pre-alloc and reuse compress buffer to avoid OOM in spill

During sort-shuffle spill, allocating compressed buffer can trigger another spill and lead to OOM. Because sortBuffer has fixed-size(1M), the maximum compressed buffer size can be computed at the very begining, and the compressed buffer can be pre-allocated and reused for spill.

Should we allocate the buffer using global allocator which is counted into overhead memory?

@jinchengchenghh can you take a look?

@jinchengchenghh Do you have further comments? Thanks!

@jinchengchenghh Do you have further comments? Thanks!

authors: FelixYBW marin-ma


text:
[GOBBLIN-553] Fix FileAwareInputStreamDataWriterTest access to public…

… key
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[X] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-553

Description

[X] Here are some details about my PR, including screenshots (if applicable):
Copy the public key from the jar into a file in the temporary directory.

Tests

[X] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[X] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

+1 Trivial test fixes.

repo: apache/incubator-gobblin


text:
[WIP] Separate optimizer implementations

This PR is based on a pending PR by @maropu that is sent before Hivemall entered Apache Incubator.
See JIRA for tracking the status of this issue.

@Lewuathe Thanks for your review! I'll check later

I'm considering to review and merge this PR after finishing #13 .

LossFunction should be selectable, not fixed.

@myui @maropu @Lewuathe JFYI: This PR has been moved to #79, and now I'm finalizing this huge feature.
@myui Could you close this PR?

Moved to #79

authors: maropu myui takuti

repo: apache/incubator-hivemall


text:
[INLONG-4301][Manager]Support oracle source

Title Name:[INLONG-4301][Manager]Support oracle source
where XYZ should be replaced by the actual issue number.
Fixes #4301
support oracle source
Explain here the context, and why you're making that change. What is the problem you're trying to solve?
Adaptative oracle extract node source
Describe the modifications you've done.
Add some utils  and objects to adaptative  oracle datasource in sort
(Please pick either of the following options)

[ ] This change is a trivial rework/code cleanup without any test coverage.

[ ] This change is already covered by existing tests, such as:
(please describe tests)

[ ] This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation

please some UT

authors: LvJiancheng gong

repo: apache/incubator-inlong


text:
[Umbrella][KPIP-1] Support RESTful API for Kyuubi

Describe the proposal

See the proposal document: #917
Task list

[x] #952 @yanghua
[x] #979 @yanghua
[x] #983 @yanghua
[x] #999 @yanghua
[x] #1011 @yanghua
[x] #1031 @yanghua
[x] #1052 @yanghua
[x] #1088 @yanghua
[x] #1092 @yanghua
[x] #1102 @yanghua
[x] #1106  @simon824
[x] #1135 @cfmcgrady
[x] #1258 @simon824
[x] #1275 @simon824
[x] #1294 @simon824
[x] #1351 @simon824
[ ] #1444 @simon824
[x] #1612
[x] #1611
[x] #1614
[x] #1620 @simon824
[x] #1575 @simon824
[x] #1631
[x] #1639
[ ] #1672 @yaooqinn

这个功能好

seems all of them are completed.
should we mark it as finished? @yanghua

seems all of them are completed.
should we mark it as finished? @yanghua

It's not completed yet, still having many things to do.

ping @yanghua @simon824, branch-1.5's been cut. Does anything more need to be fixed in this ticket? Otherwise, we shall mark this work as completed.
BTW, I notice that we have nothing in doc about this feature, right? It's not good for our users to use this feature.

ping @yanghua @simon824, branch-1.5's been cut. Does anything more need to be fixed in this ticket? Otherwise, we shall mark this work as completed.
BTW, I notice that we have nothing in doc about this feature, right? It's not good for our users to use this feature.

Yes, it can be marked as completed, if there is any bug in the future, can ping me to fix , and I will add the document later.

@simon824 thanks for the confirmation. actually, they're a lot of APIs through Rest, we can summarize and improve them in the future.

authors: mahengyang simon824 turboFei yanghua yaooqinn

repo: apache/incubator-kyuubi


text:
Failed to build from source when set USE_CPP_PACKAGE = 1, fatal error C1083: unabel to open file: “mxnet-cpp/op.h”: No such file or directory

Description
I 'm trying to compile mxnet from source code. However, when I set USE_CPP_PACKAGE=1, VS gives me 12 same errors showing below:
\mxnet\cpp-package\example..\include\mxnet-cpp/optimizer.hpp(37): fatal error C1083: unabel to open file: “mxnet-cpp/op.h”: No such file or directory.
In the build tutorial, op.h should be generate dynamicaly while compiling. If there has other option besides USE_CPP_PACKAGE?
If i set USE_CPP_PACKAGE=0, then the compile can be done succesfully. And it does generate libmxnet.dll. So I think the configuration of other dependent library is ok.
And also, there is a warning when I generate .sln by cmake:
CMake Warning at tests/CMakeLists.txt:61 (message):
Google Test not found
Belows are my setting in cmake:

Environment info (Required)
windows 10
VS2015
CMake 3.11.0 (gui)
OpenCV 3.2
OpenBlas
No CUDA No CuDNN
Build info (Required if built from source)
Compiler : visual studio 2015
What have you tried to solve it?

I have tried to paste an op.h file from other's project to my mxnet-cpp folder. But when I compile the code in VS2015, the error happens again and it delete the op.h which I put automatically.

@nswamy Can you please add labels-
CPP package, Installation

The unfortunate current situation is that mxnet-cpp/op.h should be generated from source by cpp-package/scripts/OpWrapperGenerator.py. But it requires that the actual mxnet library is there already. I suggest you make sure you have python installed and it's on the PATH and that you run 2 consecutive builds - 1 without cpp package and another with.

@MccreeZhao thank you for filing this!
This is not an installation issue, rather a Build issue.
@marcoabreu can you please update labels: remove "Installation", add "Build", add "Windows"

I have the same issues when build the mxnet from source on windows, some questions.

Would this error stop me from inference? I don't need to train the models by cpp api.
My cmake always show me "Could NOT find PythonInterp (missing: PYTHON_EXECUTABLE) ", even I add the PYTHON_EXECUTABLE as "C:\Python27", "C:\Python27\python.exe" or "C:\Python27\pythonw.exe"

If you have still the problem, what is the message in the output of the compiler for making project cpp_package_op.h ??
Typical problems can be that you do not have the Python interpreter executable in the PATH or any of the dependencies (dll's) of mxnet in the PATH. Like OPENCV, MKL or OPENBLAS, or CUDA / CUDNN

authors: Bumblebee1964 MccreeZhao Roshrini lebeg lupesko stereomatchingkiss

repo: apache/incubator-mxnet


text:
what is the accuray of imagenet-11k--place365-challenge mean?

I read the doc here:
https://github.com/apache/incubator-mxnet/tree/master/example/image-classification#imagenet-11k--place365-challenge
Is the model accuray is caculated with the imaget11k and place365 365 classes? Or only calculated with imaget11k classes?

Hi @universewill For questions, you can use MXNet discuss forum - https://discuss.mxnet.io/ where there are more people who can answer your query. We want to use Github issues mainly for bugs and feature request.
Closing the issue here. Please reopen if you are finding any issues.

authors: sandeep-krishnamurthy universewill


text:
CPU speed problem: MxNet 4 times slow than Caffe

I convert same model and parameter from caffe to MxNet. Everything works fine. But time spend more than about 4 times. Only using CPU
They both use atlas, same machine

caffe and mxnet all in the cpu model?

@qiaohaijun yes, CPU on same machine

@tqchen @piiswrong

Could you help to do some profile like in http://www.network-theory.co.uk/docs/gccintro/gccintro_80.html ?

I encountered the same issue, CPU version of mxnet is 3 times slower than caffe both using MKL. Did you solved the problem?  @wangzhangup

I encountered the same issue, CPU version of mxnet is 3 times slower than caffe both using MKL. Did you solved the problem?  @wangzhangup

I encountered the same issue, CPU version of mxnet is 3 times slower than caffe both using MKL. Did you solved the problem?  @wangzhangup

I encountered the same issue, CPU version of mxnet is 3 times slower than caffe both using MKL. Did you solved the problem?  @wangzhangup

I encountered the same issue, CPU version of mxnet is 3 times slower than caffe both using MKL. Did you solved the problem?  @wangzhangup

I encountered the same issue, CPU version of mxnet is 3 times slower than caffe both using MKL. Did you solved the problem?  @wangzhangup

I encountered the same issue, CPU version of mxnet is 3 times slower than caffe both using MKL. Did you solve the problem? @wangzhangup

@wangzhangup @uestcbingo Curious about this issue, mind listing the caffe model configuration?

Guys, is there any performance profiling tools in mxnet? I am also trying
to figure out the issue on CPU. Currently mxnet does not fully utilize my
CPUs on Windows.
On Tue, Feb 2, 2016 at 12:28 AM, Yizhi Liu notifications@github.com wrote:

@wangzhangup https://github.com/wangzhangup @uestcbingo
https://github.com/uestcbingo Curious about this issue, mind listing
the caffe model configuration?
—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/1222#issuecomment-178056479.

--
HONG Chuntao
System Research Group
Microsoft Research Asia

@hjk41 Not that I'm aware of. I have been thinking about adding profiling to the engine for a while. For example add a string tag to each Push and count time for each tag. But @tqchen Thinks this has negative performance implications.
BTW, there is a pretty easy fix for not fully using CPU: change mshadow/mshadow/tensor_cpu-inl.h:143 to a OpenMP parallel for.

ok. Back in Minerva v2, we had some kind of lightweight profiling which can
be turned on when needed. I think we can do the same for mxnet.
For the mshadow fix, why isn't this enabled by default?
On Tue, Feb 2, 2016 at 10:06 AM, Eric Junyuan Xie notifications@github.com
wrote:

@hjk41 https://github.com/hjk41 Not that I'm aware of. I have been
thinking about adding profiling to the engine for a while. For example add
a string tag to each Push and count time for each tag. But @tqchen
https://github.com/tqchen Thinks this has negative performance
implications.
BTW, there is a pretty easy fix for not fully using CPU: change
mshadow/mshadow/tensor_cpu-inl.h:143 to a OpenMP parallel for.
—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/1222#issuecomment-178311441.

--
HONG Chuntao
System Research Group
Microsoft Research Asia

@wangzhangup I am looking into this issue. Is it possible to give me the program that you are using? Both for mxnet and caffe?

Because... Its a Pr people have been too lazy to push?
On Feb 1, 2016 6:10 PM, "Chuntao Hong" notifications@github.com wrote:

ok. Back in Minerva v2, we had some kind of lightweight profiling which can
be turned on when needed. I think we can do the same for mxnet.
For the mshadow fix, why isn't this enabled by default?
On Tue, Feb 2, 2016 at 10:06 AM, Eric Junyuan Xie <
notifications@github.com>
wrote:

@hjk41 https://github.com/hjk41 Not that I'm aware of. I have been
thinking about adding profiling to the engine for a while. For example
add
a string tag to each Push and count time for each tag. But @tqchen
https://github.com/tqchen Thinks this has negative performance
implications.
BTW, there is a pretty easy fix for not fully using CPU: change
mshadow/mshadow/tensor_cpu-inl.h:143 to a OpenMP parallel for.
—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/1222#issuecomment-178311441.

--
HONG Chuntao
System Research Group
Microsoft Research Asia
—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/1222#issuecomment-178313054.

Just tried the mshadow trick. It does boosts the CPU utilization to 100%,
but lowers the overall speed. 😅 I guess enabling OpenMP in that loop
causes a lot of cache coherence overhead.
On Tue, Feb 2, 2016 at 10:15 AM, Eric Junyuan Xie notifications@github.com
wrote:

Because... Its a Pr people have been too lazy to push?
On Feb 1, 2016 6:10 PM, "Chuntao Hong" notifications@github.com wrote:

ok. Back in Minerva v2, we had some kind of lightweight profiling which
can
be turned on when needed. I think we can do the same for mxnet.
For the mshadow fix, why isn't this enabled by default?
On Tue, Feb 2, 2016 at 10:06 AM, Eric Junyuan Xie <
notifications@github.com>
wrote:

@hjk41 https://github.com/hjk41 Not that I'm aware of. I have been
thinking about adding profiling to the engine for a while. For example
add
a string tag to each Push and count time for each tag. But @tqchen
https://github.com/tqchen Thinks this has negative performance
implications.
BTW, there is a pretty easy fix for not fully using CPU: change
mshadow/mshadow/tensor_cpu-inl.h:143 to a OpenMP parallel for.
—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/1222#issuecomment-178311441.

--
HONG Chuntao
System Research Group
Microsoft Research Asia
—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/1222#issuecomment-178313054.

—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/1222#issuecomment-178314214.

--
HONG Chuntao
System Research Group
Microsoft Research Asia

try reducing the number of omp thread or limit mxnet's number of cpu
workers. They are probably competing.
On Feb 1, 2016 6:38 PM, "Chuntao Hong" notifications@github.com wrote:

Just tried the mshadow trick. It does boosts the CPU utilization to 100%,
but lowers the overall speed. 😅 I guess enabling OpenMP in that loop
causes a lot of cache coherence overhead.
On Tue, Feb 2, 2016 at 10:15 AM, Eric Junyuan Xie <
notifications@github.com>
wrote:

Because... Its a Pr people have been too lazy to push?
On Feb 1, 2016 6:10 PM, "Chuntao Hong" notifications@github.com wrote:

ok. Back in Minerva v2, we had some kind of lightweight profiling which
can
be turned on when needed. I think we can do the same for mxnet.
For the mshadow fix, why isn't this enabled by default?
On Tue, Feb 2, 2016 at 10:06 AM, Eric Junyuan Xie <
notifications@github.com>
wrote:

@hjk41 https://github.com/hjk41 Not that I'm aware of. I have been
thinking about adding profiling to the engine for a while. For
example
add
a string tag to each Push and count time for each tag. But @tqchen
https://github.com/tqchen Thinks this has negative performance
implications.
BTW, there is a pretty easy fix for not fully using CPU: change
mshadow/mshadow/tensor_cpu-inl.h:143 to a OpenMP parallel for.
—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/1222#issuecomment-178311441.

--
HONG Chuntao
System Research Group
Microsoft Research Asia
—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/1222#issuecomment-178313054.

—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/1222#issuecomment-178314214.

--
HONG Chuntao
System Research Group
Microsoft Research Asia
—
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/1222#issuecomment-178323122.

did a benchmark on mnist with lenet (batch size = 256) and openblas

lenet (img/s)
cpubenchmark.net

2 x e5-2630 v3
676
12877

2 x e5-2670
604
12514

e5-1620 v3
540
9658

i7-4790
546
10019

i5-3470
476
6567

it looks like mxnet is failed to use all cpu powers, especially for dual socket. i also looked into the performance,
   9.65%  python   [kernel.kallsyms]            [k] update_curr                                                                                ▒
   8.45%  python   libopenblas.so.0             [.] sgemm_kernel_SANDYBRIDGE                                                                                ▒
   7.62%  python   [kernel.kallsyms]            [k] __schedule                                                                                ▒
   6.37%  python   [kernel.kallsyms]            [k] pick_next_entity                                                                                ▒
   6.20%  python   [kernel.kallsyms]            [k] pick_next_task_fair                                                                                ▒
   6.07%  python   libmxnet.so                  [.] _ZN7mshadow6MapExpINS_2sv6savetoENS_6TensorINS_3cpuELi2EfEELi2EfNS_4expr13MakeTensorExpINS6_20UnpackPatchToColXExpINS3_IS4_Li4EfEEfLi4EEES9_Li2EfEELi3EEEvPNS_7TRValueIT0_S4_XT1_ET2_EERKNS6_3ExpIT3_SE_XT4_EEE                ▒
   5.66%  python   [kernel.kallsyms]            [k] __calc_delta                                                                                ▒
   3.66%  python   [kernel.kallsyms]            [k] cpuacct_charge                                                                                ▒
   3.56%  python   libmxnet.so                  [.] _ZN7mshadow6MapExpINS_2sv6savetoENS_6TensorINS_3cpuELi4EfEELi4EfNS_4expr13MakeTensorExpINS6_11CroppingExpINS7_INS6_12UnPoolingExpINS_3red7maximumENS7_INS6_10PaddingExpIS5_fLi4EEES5_Li4EfEEfLi4EEESE_Li4EfEEfLi4EEESG_Li4EfEEL▒
   3.34%  python   libc-2.19.so                 [.] __sched_yield                                                                                ▒
   3.17%  python   [kernel.kallsyms]            [k] update_min_vruntime                                                                                ▒
   3.05%  python   [kernel.kallsyms]            [k] system_call                                                                                ▒
   2.64%  python   libm-2.19.so                 [.] __expm1f                                                                                ▒
   2.20%  python   [kernel.kallsyms]            [k] system_call_after_swapgs                                                                                ▒
   2.11%  python   [kernel.kallsyms]            [k] _raw_spin_lock_irq                                                                                ▒
   2.09%  python   [kernel.kallsyms]            [k] clear_buddies                                                                                ▒
   2.05%  python   [kernel.kallsyms]            [k] _raw_spin_lock                                                                                ▒
   1.93%  python   [kernel.kallsyms]            [k] native_read_tsc                                                                                ▒
   1.61%  python   libmxnet.so                  [.] _ZN7mshadow6MapExpINS_2sv6savetoENS_6TensorINS_3cpuELi3EfEELi3EfNS_4expr13MakeTensorExpINS6_10ReshapeExpINS7_INS6_11SwapAxisExpINS3_IS4_Li4EfEEfLi4ELi3ELi0EEESA_Li4EfEEfLi3ELi4EEESC_Li3EfEELi3EEEvPNS_7TRValueIT0_S4_XT1_ET2_▒
   1.56%  python   libopenblas.so.0             [.] 0x00000000003dc027                                                                                ▒
   1.43%  python   [kernel.kallsyms]            [k] sys_sched_yield                                                                                ▒
   1.34%  python   [kernel.kallsyms]            [k] check_cfs_rq_runtime                                                                                ▒
   1.15%  python   libopenblas.so.0             [.] sgemm_otcopy_SANDYBRIDGE                                                                                ▒
   1.07%  python   libopenblas.so.0             [.] sgemm_beta_SANDYBRIDGE                                                                                ▒
   1.04%  python   [kernel.kallsyms]            [k] native_sched_clock                                                                                ▒
   1.02%  python   libmxnet.so                  [.] _ZN7mshadow6MapExpINS_2sv6savetoENS_6TensorINS_3cpuELi4EfEELi4EfNS_4expr13MakeTensorExpINS6_10PoolingExpINS_3red7maximumENS7_INS6_10PaddingExpIS5_fLi4EEES5_Li4EfEEfLi4EEESD_Li4EfEELi3EEEvPNS_7TRValueIT0_S4_XT1_ET2_EERKNS6_3▒
   1.02%  python   libmxnet.so                  [.] _ZN7mshadow6MapExpINS_2sv6savetoENS_6TensorINS_3cpuELi4EfEELi4EfNS_4expr13MakeTensorExpINS6_18PackColToPatchXExpINS3_IS4_Li2EfEEfLi4EEES9_Li4EfEELi3EEEvPNS_7TRValueIT0_S4_XT1_ET2_EERKNS6_3ExpIT3_SE_XT4_EEE                  ▒
   0.99%  python   [kernel.kallsyms]            [k] update_rq_clock.part.81                                                                                ▒
   0.91%  python   libm-2.19.so                 [.] __tanhf                                                                                ▒
   0.87%  python   [kernel.kallsyms]            [k] yield_task_fair                                                                                ▒
   0.77%  python   [kernel.kallsyms]            [k] sysret_check                                                                                ▒
   0.71%  python   libmxnet.so                  [.] _ZN7mshadow6MapExpINS_2sv6savetoENS_6TensorINS_3cpuELi4EfEELi4EfNS_4expr13MakeTensorExpINS6_11SwapAxisExpINS7_INS6_10ReshapeExpINS3_IS4_Li3EfEEfLi4ELi3EEESA_Li4EfEEfLi4ELi3ELi0EEESC_Li4EfEELi3EEEvPNS_7TRValueIT0_S4_XT1_ET2_▒
   0.65%  python   libopenblas.so.0             [.] sgemm_oncopy_SANDYBRIDGE                                                                                ▒
   0.58%  python   [kernel.kallsyms]            [k] rcu_note_context_switch

there is a large system overhead, probably due to most threads are idle. in the most two expensive mshadow functions, there are a large mount of cmp, mov, and imul. hope it helps

I have the same problems, it seems due to the python wrapper, because prediction from c++ works 4-5 times faster.

@u1234x1234 Thanks for reporting this. @hjk41 What do you think?

We are tuning CPU performance these days. We are doing it with MxNet.cpp,
though. Will report back if we have some findings.
On Sun, Mar 20, 2016 at 3:26 AM, Eric Junyuan Xie notifications@github.com
wrote:

@u1234x1234 https://github.com/u1234x1234 Thanks for reporting this.
@hjk41 https://github.com/hjk41 What do you think?
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/1222#issuecomment-198774168

--
HONG Chuntao
System Research Group
Microsoft Research Asia

Great!

i am trying get predictions for images using imagenet-21k-inception model but the prediction on cpu is very slow.
The prediction takes about 1.06 seconds for a single image and i tried to give images in batch but still there is no speed gain. I thought that giving images in batch will speed things up.
Is there something that i am doing wrong?
Is there a way to make predictions fast on cpu?

I have the same problem, my testing code is in c++ with MXNet.cpp , and do the prediction on cpu, it's about 2 times slower than caffe.

me too

Some points for CPU inference in Caffe:

For tiny models, there are a few unnecessary allocations in the forward pass in a few layers during reshaping. Only reshaping when shapes change helps a bit.

Inlining execution (when you can guarantee that all dependencies of a child operator have concluded whe executing a parent, running the child directly instead of throwing back into a job queue) helps a lot for some networks.

Specifically for convolutional networks on Haswell or above Intel CPUs at reasonable batch sizes (>32 or so), using NNPACK helps a lot (see https://github.com/BVLC/caffe/issues/4264 for an example integration). I'd suggest wrapping that in MXNet as a plugin, it's a really useful library in some scenarios.

Thnks @ajtulloch  the inclining execution is added in #2496
The NNPack could be an interesting thing to add

Tuning OMP_NUM_THREADS and OPENBLAS_NUM_THREADS can make much difference.
According to our experiments, setting OMP_NUM_THREADS to the number of
cores of a single CPU performs best in a multi-socket machine. The
difference can be more than 2X.
NNPACK seems interesting. Intel has been working on similar things in MKL.
It is said that a DNN-optimized MKL version will be released at Q4 this
year.
On Wed, Jun 22, 2016 at 12:32 PM, Tianqi Chen notifications@github.com
wrote:

Thnks @ajtulloch https://github.com/ajtulloch the inclining execution
is added in #2496 https://github.com/dmlc/mxnet/pull/2496
The NNPack could be an interesting thing to add
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dmlc/mxnet/issues/1222#issuecomment-227640521, or mute
the thread
https://github.com/notifications/unsubscribe/ABFI4RjY5MXBxG70TDVku79VJRyrO0fsks5qOLrygaJpZM4HBqqR
.

--
HONG Chuntao
System Research Group
Microsoft Research Asia

@hjk41 so MxNet is still slower than caffe on CPU for now?

@hjk41 @piiswrong

community has some plan to integrate NNPACK or MKLDNN library?
mxnet cpu patch2cols is one element-by-element copy from src images to dst cols. It will slow down performance even though openmp on. Community has plan to improve it?

@Darwin2011  NNPACK is there already if you use the CaffeOp and pull https://github.com/ajtulloch/caffe/tree/nnpack-pr into your Caffe branch btw :)

MKLDNN will not be available until late this year. They do have a tech
preview, though. I think we should have someone look into it. I have just
moved to a new company so I won't have much time for this. I will post a
request for contribution in the issue and see if someone is interested.
On Tue, Aug 9, 2016 at 5:24 AM, Andrew Tulloch notifications@github.com
wrote:

@Darwin2011 https://github.com/Darwin2011 NNPACK is there already if
you use the CaffeOp and pull https://github.com/ajtulloch/
caffe/tree/nnpack-pr into your Caffe branch btw :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dmlc/mxnet/issues/1222#issuecomment-238382345, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABFI4dtIqjKLOt5FHUwbQ3aZbF8FOi2Mks5qd56KgaJpZM4HBqqR
.

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!

authors: Darwin2011 ajtulloch careyjiang hjk41 javelinjs mli passerbythesun phunterlau piiswrong qiaohaijun realzheng rhythm9229 starimpact tqchen u1234x1234 uestcbingo wangzhangup winstywang


text:
[CI] Flaky test in CI windows-gpu

Description
windows-gpu CI has some flaky tests.
03:18:11  FAILED tests/python/gpu/test_operator_gpu.py::test_sparse_nd_pickle - mxnet.b...
03:18:11  FAILED tests/python/gpu/test_operator_gpu.py::test_sparse_nd_save_load[save]
03:18:11  FAILED tests/python/gpu/test_operator_gpu.py::test_sparse_ndarray_load_csr_npz_scipy[save]

Occurrences
https://jenkins.mxnet-ci.amazon-ml.com/job/mxnet-validation/job/windows-gpu/job/master/2524/execution/node/155/log/

@leezu Could you find someone to help?

I ran into failures on these exact 3 tests.  I've tracked down the problem: the unittest test_zero_sized_dim does not clean up after itself, and leaves is_np_shape_thread_local_ = 1.  Subsequent tests run by the same pytest worker might fail as a result.  So I get a failure on:
pytest --verbose -s tests/python/gpu/test_operator_gpu.py::{test_zero_sized_dim,test_sparse_nd_pickle}

but not on:
pytest --verbose -s tests/python/gpu/test_operator_gpu.py::test_sparse_nd_pickle

Looking into the most appropriate fix...


text:
Download pretrained model

Sorry to ask the problem here, but I don't know how to get the pretrained model vgg 16 after read the caffe documents, because I don't know  what is the gist id crresponding to model. Can anyone help?

[rd@idc01-rank-gpu-01 scripts]$ ./download_model_from_gist.sh
usage: download_model_from_gist.sh <gist_id>

@George19890617 you can download the vgg16 of mxnet in the readme of https://github.com/dmlc/mxnet/tree/master/example/fcn-xs
the http is https://pan.baidu.com/s/1bgz4PC#path=%252Ffcn-xs, name is  VGG_FC_ILSVRC_16_layers-***

@tornadomeet thanks so much, I found that seems to be the networking problem.

@tornadomeet I found it write in
https://github.com/dmlc/mxnet/tree/master/example/fcn-xs

(when using the newest mxnet, you'd better using larger learning rate, such as 1e-4, 1e-5, 1e-6 instead, because the newest mxnet will do gradient normalization in SoftmaxOutput)

But isn't it the default is set to not do the normalization of gradients? I am not clear about the quote sentence means.

yes, the mxnet version right now will do it auto, but the model i provide at that time will not.

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!


text:
Freeze weights during training

Is there any solution to freeze weights? For example, I want to keep some weights unchanged when them satisfy some condition(like weight<0.5) during training.

One solution might be using optimizer interface. You can set individual learning rate multipler for parameters. http://mxnet.io/api/python/model.html?highlight=optimizer#optimizer-api-reference

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!

authors: BingzheWu kevinthesun yajiedesign


text:
update raspberry pi install instructions

Description
The current install instructions for Raspberry Pi did not work for me. I kept at it until I got it working and now I've updated the instructions based on my experience using the cross-compile approach. The wheel I provide is from using this route.
I think that the dependencies could be leaner, but I know that these worked.
I did not test the native approach. I made one edit there because I noticed it doesn't install python-dev and that's needed for the Pi wine detector example.

@larroy @IvyBazan - can you review?

@aaronmarkham  Thanks for the contribution
@mxnet-label-bot  add [pr-awaiting-review]
@marcoabreu @larroy @lebeg
Could you help review this PR?

LGTM

authors: IvyBazan aaronmarkham ankkhedia


text:
Fix SliceChannel Type inference

Description
Fix SliceChannel Type Inference. Do forward and backward inference for slice channel with ElemwiseAttr logic. Remove exception thrown from infer_type logic in slice channel.
Adds some gluoncv models to AMP examples.
@ptrendx
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

I don't know enough to comment on the changes to the example etc., but the main fix from this PR (InferType in SliceChannel) looks good. @anirudh2290 Searching for First input must have specified type gives many other hits, do you intend to look into those as well?

@ptrendx how did you come up with that regex ?

@ptrendx good point about the other places where error is raised. The scope of the PR increased :) . Can we start with this PR first and I will follow up with another PR/PRs for fixes in other ops. Sorry, little busy with other deliverables too and cannot do it immediately.
For the examples, I have tested running the gluon models for conversion and dummy inference in a loop.

Sure, having this PR fix just SliceChannel is totally fine.


text:
[mxnet2.0] gluon 2.0 data: deprecate dataset transform= argument

Description
Fix https://github.com/apache/incubator-mxnet/issues/11141
This is part of changes for gluon 2.0 which addresses the usability issue.
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

Is there any timeline when to drop the deprecated argument?

Ideally the first release after 2.0

@mxnet-bot run ci [all]


text:
Remove check for subgraph with cycles

Description
Partitioning a graph with potential subgraphs that have a cycle with the input or output fails instead of ignoring these subgraphs. For instance, Faster-RCNN (available in the GluonCV model zoo) optimization failed.
Change
This PR simply removes the check that raise an exception if the subgraph candidate has a cycle, so LabelSubgraph can return false.

Thanks @Kh4L! this is a really great addition to the partitioning logic to be able to skip subgraphs rather than error out when theres a loop. It took me a while to come up to speed with the complexity of the logic here, can you add some more comments so that next time it will be easier?

@mxnet-bot run ci [unix-cpu]

@mxnet-bot run ci [clang]

@mxnet-bot run ci [centos-cpu]


text:
[v1.x] Update unittests for object detection models

Description
To address the prediction mismatching due to different implementations of nms operator between mxnet and onnx.
Checklist
Essentials

[ ] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage
[ ] Code is well-documented

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

@mxnet-bot run ci [sanity]

run ci [centos-gpu, edge]

@mxnet-bot run ci [edge]


text:
[NEMO-379] Change javadoc goal to a proper one

JIRA: NEMO-379: Change javadoc goal to a proper one
Major changes:

Change javadoc test command from javadoc to test-javadoc. This prevents tests from creating unwanted javdoc files, caused by #139 .

Minor changes to note:

None

Tests for the changes:

None

Other comments:

None

Closes #212

LGTM. I will merge it

authors: taegeonum wonook

repo: apache/incubator-nemo


text:
Integrate existing patches

When I worked on these, I didn't push them because new feature phase was already past. So I just attached them to each issue. I hope that these are merged because I'm not sure whether I can apply patches correctly after license headers are changed.

If someone has no objection, I'll merge this later.

repo: apache/incubator-netbeans


text:
xtensa/esp32: Fix the memory regions with regards to the range used by the ROM.

Summary
Static allocation sections should end at the beginning of the ROM data.
The rest of memory (End of ROM data --> End of DRAM) is added to the
heap.
Impact
Fix the memories and avoid clobbering ROM data.
Testing
Several configs with ESP32.

Thank you! @Ouss4

repo: apache/incubator-nuttx


text:
Fix TX ID bits 18-20 of mcp2515 can driver

Summary
Fix TX ID bits 18-20 of mcp2515 can driver, register TXB0SIDL is being filled inappropriately, values being overwritten.
Impact
Driver.
Testing
With mcp2515 shield in stm32h7.

Hi @robsonsopran thanks for finding this issue! There is a long line in the file:
drivers/can/mcp2515.c:1562:93: error: Long line found
This line was already there, but could you please break this line to be <=78 characters?

@acassis the long line is hard to fix, because it's a URL:
https://github.com/apache/incubator-nuttx/blob/master/drivers/can/mcp2515.c#L1562
Anyway, let's merge this PR and come up a fix later.

authors: acassis robsonsopran xiaoxiang781216


text:
Remove the unused make file variables

Summary

boards/nrf52: Remove ARCHCCVERSION/ARCHCCMAJOR which isn't used
boards/nrf52: Remove toolchain macro which is already defined in Toolchain.defs
boards/bm3803: Remove toolchain macro which is already defined in Toolchain.defs

Impact
No, code refactor
Testing
Pass CI

@pkarashchenko how about this PR?

Will merge as soon as CI pass.