text:
[SUPPORT] deltacommit for client 172.16.116.102 already exists

Environment Description

Hudi version :
0.6.0
Spark version :
spark-2.4.4-bin-hadoop2.7
Hive version :
hive-2.3.4
Hadoop version :
hadoop2.7.3
Storage (HDFS/S3/GCS..) :
hdfs
Running on Docker? (yes/no) :
no

1.when i write data to hudi,the error

Logical Plan:
RepartitionByExpression [dbName#23, tblName#24], 6
+- Project [row#21.dbName AS dbName#23, row#21.tblName AS tblName#24, row#21.opr AS opr#25, row#21.datalakeLogicalDeletion AS datalakeLogicalDeletion#26, row#21.etlTime AS etlTime#27L, row#21.jsonData AS jsonData#28]
+- Project [jsontostructs(StructField(dbName,StringType,true), StructField(tblName,StringType,true), StructField(opr,StringType,true), StructField(datalakeLogicalDeletion,IntegerType,true), StructField(etlTime,LongType,true), StructField(jsonData,StringType,true), cast(value#8 as string), Some(PRC)) AS row#21]
+- StreamingExecutionRelation KafkaV2[Subscribe[datalake_advertise]], [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13]
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:297)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:297)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:193)
Caused by: org.apache.hudi.exception.HoodieIOException: Failed to create file /user/datalake/hudi/hbase/f_mid_business_card/.hoodie/20210114202219.deltacommit
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.createImmutableFileInPath(HoodieActiveTimeline.java:449)
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:333)
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.createImmutableFileInPath(HoodieActiveTimeline.java:449)
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:333)
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:308)
at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.saveAsComplete(HoodieActiveTimeline.java:143)
at org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:124)
at org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:124)
at org.apache.hudi.client.AbstractHoodieWriteClient.commit(AbstractHoodieWriteClient.java:99)
at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:397)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:205)	at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:205)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:125)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)

2.my hoodieWriteConfig :

hoodie.filesystem.view.incr.timeline.sync.enable -> false,
hoodie.bulkinsert.sort.mode -> GLOBAL_SORT,
hoodie.avro.schema.externalTransformation -> false,
hoodie.bootstrap.parallelism -> 1500,
hoodie.delete.shuffle.parallelism -> 1500,
hoodie.simple.index.use.caching -> true,
hoodie.bloom.index.filter.type -> DYNAMIC_V0,
hoodie.filesystem.view.remote.port -> 26754,
hoodie.datasource.write.operation -> upsert,
hoodie.cleaner.parallelism -> 200,
hoodie.global.simple.index.parallelism -> 100,
hoodie.bootstrap.mode.selector.regex -> .*,
hoodie.parquet.page.size -> 1048576,
hoodie.datasource.write.table.type -> MERGE_ON_READ,
hoodie.datasource.hive_sync.table -> f_mid_business_card,
hoodie.compaction.daybased.target.partitions -> 10,
hoodie.metrics.reporter.class -> ,
hoodie.parquet.block.size -> 125829120,
hoodie.cleaner.delete.bootstrap.base.file -> false,
hoodie.consistency.check.max_interval_ms -> 300000,
hoodie.insert.shuffle.parallelism -> 100,
hoodie.upsert.shuffle.parallelism -> 100,
hoodie.bulkinsert.shuffle.parallelism -> 1500,
hoodie.write.commit.callback.on -> false,
hoodie.cleaner.fileversions.retained -> 3,
hoodie.datasource.hive_sync.partition_extractor_class -> org.apache.hudi.hive.NonPartitionedExtractor,
hoodie.parquet.compression.codec -> gzip,
hoodie.datasource.write.hive_style_partitioning -> true,
hoodie.copyonwrite.insert.split.size -> 500000,
hoodie.optimistic.consistency.guard.sleep_time_ms -> 500,
hoodie.datasource.hive_sync.use_jdbc -> true,
hoodie.metrics.reporter.type -> GRAPHITE,
hoodie.bootstrap.index.class -> org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex,
hoodie.filesystem.remote.backup.view.enable -> true,
hoodie.logfile.to.parquet.compression.ratio -> 0.35,
hoodie.filesystem.view.spillable.mem -> 104857600,
hoodie.write.status.storage.level -> MEMORY_AND_DISK_SER,
hoodie.write.commit.callback.http.timeout.seconds -> 3,
hoodie.copyonwrite.insert.auto.split -> true,
hoodie.logfile.data.block.max.size -> 268435456,
hoodie.index.type -> BLOOM,
hoodie.keep.min.commits -> 6,
hoodie.memory.spillable.map.path -> /tmp/,
hoodie.filesystem.view.rocksdb.base.path -> /tmp/hoodie_timeline_rocksdb,
hoodie.compact.inline -> false,
hoodie.clean.async -> true,
hoodie.record.size.estimation.threshold -> 1.0,
hoodie.metrics.graphite.host -> localhost,
hoodie.simple.index.update.partition.path -> false,
hoodie.bloom.index.filter.dynamic.max.entries -> 100000,
hoodie.compaction.reverse.log.read -> false,
hoodie.metrics.jmx.port -> 9889,
hoodie.writestatus.class -> org.apache.hudi.client.WriteStatus,
hoodie.datasource.hive_sync.enable -> true,
hoodie.finalize.write.parallelism -> 1500,
hoodie.rollback.parallelism -> 100,
hoodie.index.bloom.num_entries -> 60000,
hoodie.memory.merge.max.size -> 131072,
hoodie.bootstrap.mode.selector.regex.mode -> METADATA_ONLY,
hoodie.rollback.using.markers -> false,
hoodie.copyonwrite.record.size.estimate -> 1024,
hoodie.bloom.index.input.storage.level -> MEMORY_AND_DISK_SER,
hoodie.simple.index.parallelism -> 50,
hoodie.consistency.check.enabled -> false,
hoodie.bloom.index.use.caching -> true,
hoodie.metrics.on -> false,
hoodie.memory.compaction.max.size -> 1048576,
hoodie.parquet.small.file.limit -> 104857600,
hoodie.combine.before.insert -> false,
hoodie.cleaner.commits.retained -> 2,
hoodie.embed.timeline.server -> true,
hoodie.bootstrap.mode.selector -> org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector,
hoodie.filesystem.view.secondary.type -> MEMORY,
_.hoodie.allow.multi.write.on.same.instant -> false,
hoodie.datasource.write.partitionpath.field -> ,
_hoodie.optimistic.consistency.guard.enable -> true,
hoodie.datasource.hive_sync.database -> hbase,
hoodie.bloom.index.update.partition.path -> true,
hoodie.fail.on.timeline.archiving -> true,
hoodie.markers.delete.parallelism -> 100,
hoodie.filesystem.view.type -> MEMORY,
hoodie.parquet.max.file.size -> 125829120,
hoodie.datasource.write.keygenerator.class -> org.apache.hudi.keygen.NonpartitionedKeyGenerator,
hoodie.bootstrap.partitionpath.translator.class -> org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPartitionPathTranslator,
hoodie.bloom.index.prune.by.ranges -> true,
hoodie.base.path -> /user/datalake/hudi/hbase/f_mid_business_card,
hoodie.index.class -> ,
hoodie.clean.automatic -> true,
hoodie.filesystem.view.remote.host -> localhost,
hoodie.compaction.lazy.block.read -> false,
hoodie.memory.writestatus.failure.fraction -> 0.1,
hoodie.metrics.graphite.port -> 4756,
hoodie.cleaner.policy -> KEEP_LATEST_COMMITS,
hoodie.logfile.max.size -> 1073741824,
hoodie.filesystem.view.spillable.compaction.mem.fraction -> 0.01,
hoodie.datasource.write.recordkey.field -> datalake_rowkey,
hoodie.avro.schema.validate -> false,
hoodie.simple.index.input.storage.level -> MEMORY_AND_DISK_SER,
hoodie.timeline.layout.version -> 1,
hoodie.consistency.check.max_checks -> 7,
hoodie.consistency.check.initial_interval_ms -> 2000,
hoodie.keep.max.commits -> 8,
hoodie.compact.inline.max.delta.commits -> 5,
hoodie.parquet.compression.ratio -> 0.1,
hoodie.memory.dfs.buffer.max.size -> 16777216,
hoodie.auto.commit -> true,
hoodie.write.commit.callback.http.api.key -> hudi_write_commit_http_callback,
hoodie.assume.date.partitioning -> false,
hoodie.filesystem.view.spillable.dir -> /tmp/view_map/,
hoodie.compaction.strategy -> org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy,
hoodie.combine.before.upsert -> true,
hoodie.bloom.index.keys.per.bucket -> 10000000,
hoodie.write.commit.callback.class -> org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback,
hoodie.bloom.index.parallelism -> 0,
hoodie.cleaner.incremental.mode -> true,
hoodie.commits.archival.batch -> 5,
hoodie.datasource.hive_sync.partition_fields -> ,
hoodie.compaction.target.io -> 512000,
hoodie.table.name -> f_mid_business_card,
hoodie.bloom.index.bucketized.checking -> true,
hoodie.compaction.payload.class -> org.apache.hudi.common.model.OverwriteWithLatestAvroPayload,
hoodie.combine.before.delete -> true,
hoodie.datasource.write.precombine.field -> ts,
hoodie.filesystem.view.spillable.bootstrap.base.file.mem.fraction -> 0.05,
hoodie.metrics.jmx.host -> localhost,
hoodie.index.bloom.fpp -> 0.000000001,
hoodie.datasource.hive_sync.jdbcurl -> jdbc:hive2://172.16.116.102:10000,
hoodie.bloom.index.use.treebased.filter -> true

To Reproduce
Key code is

val hudiDF = buildHudiDF(tableConfig, partitionDF, dbNameSource, tableNameSource)
val hoodieWriteConfigMap = buildHudiWriteConfig(tableConfig)
hudiDF.write.format("hudi")
.options(hoodieWriteConfigMap)
.mode(SaveMode.Append)
.save(hoodieWriteConfigMap.getOrElse(BASE_PATH_PROP, "/tmp"))

private def buildHudiDF(tableConfig: TableConfig, sourceDataFrame: DataFrame, dbNameKafka: String, tblNameKafka: String): DataFrame = {
val dataLakeColumns = tableConfig.getDataLakeColumns.asScala.toArray
val commonSchema = new StructType(dataLakeColumns.map(column => StructField(column.getColumnName, StringType)))
val castArray = dataLakeColumns.map(column => col(column.getColumnName).cast(column.getColumnType).toString())
var targetDataFrame = sourceDataFrame
.select(from_json(col("jsonData"), commonSchema).alias("row"), col("datalakeLogicalDeletion"))
.select(col("row."), col("datalakeLogicalDeletion"))
.selectExpr(castArray :+ (col("datalakeLogicalDeletion").alias("datalake_logical_deletion").toString()): _)
.filter(col("dbName").isNull.or(col("dbName").equalTo(dbNameKafka)).and(col("tblName").equalTo(tblNameKafka)))
val precombineField = tableConfig.getPrecombineFieldOptKey
if (StringUtils.isEmpty(precombineField) || !commonSchema.contains(StructField(precombineField, StringType))) {
targetDataFrame = targetDataFrame
.withColumn("ts", current_timestamp())
}
val complexRecordKey = tableConfig.getRecordkeyFieldOptKey
if (StringUtils.isNotEmpty(complexRecordKey)) {
val keyArray = complexRecordKey.split(StringUtils.COMMA_SYMBOL)
var conditionChain = col(keyArray(0)).isNotNull.and(col(keyArray(0)).notEqual(StringUtils.EMPTY))
keyArray.tail.foreach(key => conditionChain.or(col(key).isNotNull.and(col(key).notEqual(StringUtils.EMPTY))))
targetDataFrame = targetDataFrame.filter(conditionChain)
}
targetDataFrame
}

private def buildHudiWriteConfig(tableConfig: TableConfig): mutable.Map[String, String] = {
val options = new mutable.HashMap[String, String]
val dbName = tableConfig.getDbName
val tblName = tableConfig.getTableName
val partition_field = tableConfig.getPartitionpathFieldOptKey
options += (TABLE_TYPE_OPT_KEY -> tableConfig.getTableTypeOptKey)
options += (OPERATION_OPT_KEY -> tableConfig.getOperationOptKey)
options += (RECORDKEY_FIELD_OPT_KEY -> tableConfig.getRecordkeyFieldOptKey)
options += (PRECOMBINE_FIELD_OPT_KEY -> tableConfig.getPrecombineFieldOptKey)
options += (HIVE_SYNC_ENABLED_OPT_KEY -> "true")
options += (INDEX_TYPE_PROP -> tableConfig.getIndexTypeProp)
options += (BLOOM_INDEX_FILTER_TYPE -> BloomFilterTypeCode.DYNAMIC_V0.name())
options += (KEYGENERATOR_CLASS_OPT_KEY -> tableConfig.getKeygeneratorClassOptKey)
options += (HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY -> tableConfig.getHivePartitionExtractorClassOptKey)
options += (HIVE_STYLE_PARTITIONING_OPT_KEY -> "true")
options += (PARTITIONPATH_FIELD_OPT_KEY -> partition_field)
options += (HIVE_PARTITION_FIELDS_OPT_KEY -> partition_field.split(",").map(pair => pair.split(":")(0)).mkString(","))
options += (HIVE_DATABASE_OPT_KEY -> dbName)
options += (HIVE_TABLE_OPT_KEY -> tblName)
options += (HIVE_USE_JDBC_OPT_KEY -> tableConfig.getHiveUseJdbcOptKey)
options += (HIVE_URL_OPT_KEY -> tableConfig.getHiveUrlOptKey)
val hoodieCompactionConfig = HoodieCompactionConfig.newBuilder()
.retainCommits(2)
.withCommitsArchivalBatchSize(5)
.archiveCommitsWith(6, 8)
.withAsyncClean(true)
.build()
val hoodieIndexConfig = HoodieIndexConfig.newBuilder()
.withBloomIndexUpdatePartitionPath(true)
.build()
val hoodieMemoryConfig = HoodieMemoryConfig.newBuilder()
.withMaxMemoryMaxSize(128 * 1024, 1024 * 1024)
.build()
val hoodieWriteConfig = HoodieWriteConfig.newBuilder()
.withParallelism(tableConfig.getShuffleParallelism.toInt, tableConfig.getShuffleParallelism.toInt)
.withIndexConfig(hoodieIndexConfig)
.withCompactionConfig(hoodieCompactionConfig)
.withMemoryConfig(hoodieMemoryConfig)
.withProps(options.asJava)
.forTable(tblName)
.withPath(StringUtils.concat(tableConfig.getBasePath, File.separator, tableConfig.getDbName, File.separator, tableConfig.getTableName))
.build()
logger.info(s"hoodieWriteConfig -> {${hoodieWriteConfig.getProps.asScala.toString()}}")
hoodieWriteConfig.getProps.asScala
}

but ,when the hoodieWriteConfig like below,it is ok.but the log file is too big(2GB+),and small data file(1MB-) too much. and cause OOM error each hour.

hoodie.datasource.hive_sync.table->f_mid_order_details,
hoodie.bloom.index.update.partition.path->true,
hoodie.bloom.index.filter.type->DYNAMIC_V0,
hoodie.datasource.write.keygenerator.class->org.apache.hudi.keygen.SimpleKeyGenerator,
hoodie.datasource.hive_sync.database->hudi,
hoodie.datasource.write.table.type->MERGE_ON_READ,
hoodie.datasource.write.partitionpath.field->payment_date,
hoodie.datasource.hive_sync.partition_fields->payment_date,
hoodie.datasource.hive_sync.partition_extractor_class->org.apache.hudi.hive.MultiPartKeysValueExtractor,
hoodie.datasource.write.recordkey.field->key,
hoodie.datasource.hive_sync.enable->true,
hoodie.upsert.shuffle.parallelism->100,
hoodie.index.type->GLOBAL_BLOOM,
hoodie.datasource.hive_sync.jdbcurl->jdbc:hive2://172.16.117.73:10000,
hoodie.compact.inline->true,
hoodie.datasource.write.precombine.field->ts,
hoodie.table.name->f_mid_order_details,
hoodie.datasource.write.hive_style_partitioning->true,
hoodie.datasource.write.operation->upsert

Can you provide the full dump of the logs and .hoodie/ folder ?

Can you provide the full dump of the logs and .hoodie/ folder ?

Can you provide the full dump of the logs and .hoodie/ folder ?

the log has been clear, but still have some picture

when i set hoodie.auto.commit = false,the error is gone.
but how to limit the log file size,my log file is so big(3GB+),log file version always 1.

when i change hoodie.cleaner.policy = KEEP_LATEST_FILE_VERSIONS and hoodie.cleaner.fileversions.retained = 1,the old data file can be clean,but how to clean the old log file(or clean the old log file commit,set hoodie.cleaner.policy = KEEP_LATEST_COMMITS and hoodie.cleaner.commits.retained = 1 is useless)
now my config is

hoodie.filesystem.view.incr.timeline.sync.enable -> false,
hoodie.bulkinsert.sort.mode -> GLOBAL_SORT,
hoodie.avro.schema.externalTransformation -> false,
hoodie.bootstrap.parallelism -> 1500,
hoodie.delete.shuffle.parallelism -> 1500,
hoodie.simple.index.use.caching -> true,
hoodie.bloom.index.filter.type -> DYNAMIC_V0,
hoodie.filesystem.view.remote.port -> 26754,
hoodie.datasource.write.operation -> upsert,
hoodie.cleaner.parallelism -> 200,
hoodie.global.simple.index.parallelism -> 100,
hoodie.bootstrap.mode.selector.regex -> .*,
hoodie.parquet.page.size -> 1048576,
hoodie.datasource.write.table.type -> MERGE_ON_READ,
hoodie.datasource.hive_sync.table -> f_mid_business_card,
hoodie.compaction.daybased.target.partitions -> 10,
hoodie.metrics.reporter.class -> ,
hoodie.parquet.block.size -> 125829120,
hoodie.cleaner.delete.bootstrap.base.file -> false,
hoodie.consistency.check.max_interval_ms -> 300000,
hoodie.insert.shuffle.parallelism -> 100,
hoodie.upsert.shuffle.parallelism -> 100,
hoodie.bulkinsert.shuffle.parallelism -> 1500,
hoodie.write.commit.callback.on -> false,
hoodie.cleaner.fileversions.retained -> 1,
hoodie.datasource.hive_sync.partition_extractor_class -> org.apache.hudi.hive.NonPartitionedExtractor,
hoodie.parquet.compression.codec -> gzip,
hoodie.datasource.write.hive_style_partitioning -> true,
hoodie.copyonwrite.insert.split.size -> 500000,
hoodie.optimistic.consistency.guard.sleep_time_ms -> 500,
hoodie.datasource.hive_sync.use_jdbc -> true,
hoodie.metrics.reporter.type -> GRAPHITE,
hoodie.bootstrap.index.class -> org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex,
hoodie.filesystem.remote.backup.view.enable -> true,
hoodie.logfile.to.parquet.compression.ratio -> 0.35,
hoodie.filesystem.view.spillable.mem -> 104857600,
hoodie.write.status.storage.level -> MEMORY_AND_DISK_SER,
hoodie.write.commit.callback.http.timeout.seconds -> 3,
hoodie.copyonwrite.insert.auto.split -> true,
hoodie.logfile.data.block.max.size -> 268435456,
hoodie.index.type -> BLOOM,
hoodie.keep.min.commits -> 6,
hoodie.memory.spillable.map.path -> /tmp/,
hoodie.filesystem.view.rocksdb.base.path -> /tmp/hoodie_timeline_rocksdb,
hoodie.compact.inline -> false,
hoodie.clean.async -> true,
hoodie.record.size.estimation.threshold -> 1.0,
hoodie.metrics.graphite.host -> localhost,
hoodie.simple.index.update.partition.path -> false,
hoodie.bloom.index.filter.dynamic.max.entries -> 100000,
hoodie.compaction.reverse.log.read -> false,
hoodie.metrics.jmx.port -> 9889,
hoodie.writestatus.class -> org.apache.hudi.client.WriteStatus,
hoodie.datasource.hive_sync.enable -> true,
hoodie.finalize.write.parallelism -> 1500,
hoodie.rollback.parallelism -> 100,
hoodie.index.bloom.num_entries -> 60000,
hoodie.memory.merge.max.size -> 134217728,
hoodie.bootstrap.mode.selector.regex.mode -> METADATA_ONLY,
hoodie.rollback.using.markers -> false,
hoodie.copyonwrite.record.size.estimate -> 1024,
hoodie.bloom.index.input.storage.level -> MEMORY_AND_DISK_SER,
hoodie.simple.index.parallelism -> 50,
hoodie.consistency.check.enabled -> false,
hoodie.bloom.index.use.caching -> true,
hoodie.metrics.on -> false,
hoodie.memory.compaction.max.size -> 1073741824,
hoodie.parquet.small.file.limit -> 104857600,
hoodie.combine.before.insert -> false,
hoodie.cleaner.commits.retained -> 1,
hoodie.embed.timeline.server -> true,
hoodie.bootstrap.mode.selector -> org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector,
hoodie.filesystem.view.secondary.type -> MEMORY,
_.hoodie.allow.multi.write.on.same.instant -> false,
hoodie.datasource.write.partitionpath.field -> ,
_hoodie.optimistic.consistency.guard.enable -> true,
hoodie.datasource.hive_sync.database -> hbase,
hoodie.bloom.index.update.partition.path -> true,
hoodie.fail.on.timeline.archiving -> true,
hoodie.markers.delete.parallelism -> 100,
hoodie.filesystem.view.type -> MEMORY,
hoodie.parquet.max.file.size -> 125829120,
hoodie.datasource.write.keygenerator.class -> org.apache.hudi.keygen.NonpartitionedKeyGenerator,
hoodie.bootstrap.partitionpath.translator.class -> org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPartitionPathTranslator,
hoodie.bloom.index.prune.by.ranges -> true,
hoodie.base.path -> /user/datalake/hudi/hbase/f_mid_business_card,
hoodie.index.class -> ,
hoodie.clean.automatic -> true,
hoodie.filesystem.view.remote.host -> localhost,
hoodie.compaction.lazy.block.read -> false,
hoodie.memory.writestatus.failure.fraction -> 0.1,
hoodie.metrics.graphite.port -> 4756,
hoodie.cleaner.policy -> KEEP_LATEST_FILE_VERSIONS,
hoodie.logfile.max.size -> 1073741824,
hoodie.filesystem.view.spillable.compaction.mem.fraction -> 0.01,
hoodie.datasource.write.recordkey.field -> datalake_rowkey,
hoodie.avro.schema.validate -> false,
hoodie.simple.index.input.storage.level -> MEMORY_AND_DISK_SER,
hoodie.timeline.layout.version -> 1,
hoodie.consistency.check.max_checks -> 7,
hoodie.consistency.check.initial_interval_ms -> 2000,
hoodie.keep.max.commits -> 8,
hoodie.compact.inline.max.delta.commits -> 5,
hoodie.parquet.compression.ratio -> 0.1,
hoodie.memory.dfs.buffer.max.size -> 16777216,
hoodie.auto.commit -> false,
hoodie.write.commit.callback.http.api.key -> hudi_write_commit_http_callback,
hoodie.assume.date.partitioning -> false,
hoodie.filesystem.view.spillable.dir -> /tmp/view_map/,
hoodie.compaction.strategy -> org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy,
hoodie.combine.before.upsert -> true,
hoodie.bloom.index.keys.per.bucket -> 10000000,
hoodie.write.commit.callback.class -> org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback,
hoodie.bloom.index.parallelism -> 0,
hoodie.cleaner.incremental.mode -> true,
hoodie.commits.archival.batch -> 5,
hoodie.datasource.hive_sync.partition_fields -> ,
hoodie.compaction.target.io -> 512000,
hoodie.table.name -> f_mid_business_card,
hoodie.bloom.index.bucketized.checking -> true,
hoodie.compaction.payload.class -> org.apache.hudi.common.model.OverwriteWithLatestAvroPayload,
hoodie.combine.before.delete -> true,
hoodie.datasource.write.precombine.field -> ts,
hoodie.filesystem.view.spillable.bootstrap.base.file.mem.fraction -> 0.05,
hoodie.metrics.jmx.host -> localhost,
hoodie.index.bloom.fpp -> 0.000000001,
hoodie.datasource.hive_sync.jdbcurl -> jdbc:hive2://172.16.116.102:10000,
hoodie.bloom.index.use.treebased.filter -> true

Can you provide the full dump of the logs and .hoodie/ folder ?

the log has been clear, but still have some picture

when i set hoodie.auto.commit = false,the error is gone.
but how to limit the log file size,my log file is so big(3GB+),log file version always 1.

when i change hoodie.cleaner.policy = KEEP_LATEST_FILE_VERSIONS and hoodie.cleaner.fileversions.retained = 1,the old data file can be clean,but how to clean the old log file(or clean the old log file commit,set hoodie.cleaner.policy = KEEP_LATEST_COMMITS and hoodie.cleaner.commits.retained = 1 is useless)
now my config is

hoodie.filesystem.view.incr.timeline.sync.enable -> false,
hoodie.bulkinsert.sort.mode -> GLOBAL_SORT,
hoodie.avro.schema.externalTransformation -> false,
hoodie.bootstrap.parallelism -> 1500,
hoodie.delete.shuffle.parallelism -> 1500,
hoodie.simple.index.use.caching -> true,
hoodie.bloom.index.filter.type -> DYNAMIC_V0,
hoodie.filesystem.view.remote.port -> 26754,
hoodie.datasource.write.operation -> upsert,
hoodie.cleaner.parallelism -> 200,
hoodie.global.simple.index.parallelism -> 100,
hoodie.bootstrap.mode.selector.regex -> .*,
hoodie.parquet.page.size -> 1048576,
hoodie.datasource.write.table.type -> MERGE_ON_READ,
hoodie.datasource.hive_sync.table -> f_mid_business_card,
hoodie.compaction.daybased.target.partitions -> 10,
hoodie.metrics.reporter.class -> ,
hoodie.parquet.block.size -> 125829120,
hoodie.cleaner.delete.bootstrap.base.file -> false,
hoodie.consistency.check.max_interval_ms -> 300000,
hoodie.insert.shuffle.parallelism -> 100,
hoodie.upsert.shuffle.parallelism -> 100,
hoodie.bulkinsert.shuffle.parallelism -> 1500,
hoodie.write.commit.callback.on -> false,
hoodie.cleaner.fileversions.retained -> 1,
hoodie.datasource.hive_sync.partition_extractor_class -> org.apache.hudi.hive.NonPartitionedExtractor,
hoodie.parquet.compression.codec -> gzip,
hoodie.datasource.write.hive_style_partitioning -> true,
hoodie.copyonwrite.insert.split.size -> 500000,
hoodie.optimistic.consistency.guard.sleep_time_ms -> 500,
hoodie.datasource.hive_sync.use_jdbc -> true,
hoodie.metrics.reporter.type -> GRAPHITE,
hoodie.bootstrap.index.class -> org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex,
hoodie.filesystem.remote.backup.view.enable -> true,
hoodie.logfile.to.parquet.compression.ratio -> 0.35,
hoodie.filesystem.view.spillable.mem -> 104857600,
hoodie.write.status.storage.level -> MEMORY_AND_DISK_SER,
hoodie.write.commit.callback.http.timeout.seconds -> 3,
hoodie.copyonwrite.insert.auto.split -> true,
hoodie.logfile.data.block.max.size -> 268435456,
hoodie.index.type -> BLOOM,
hoodie.keep.min.commits -> 6,
hoodie.memory.spillable.map.path -> /tmp/,
hoodie.filesystem.view.rocksdb.base.path -> /tmp/hoodie_timeline_rocksdb,
hoodie.compact.inline -> false,
hoodie.clean.async -> true,
hoodie.record.size.estimation.threshold -> 1.0,
hoodie.metrics.graphite.host -> localhost,
hoodie.simple.index.update.partition.path -> false,
hoodie.bloom.index.filter.dynamic.max.entries -> 100000,
hoodie.compaction.reverse.log.read -> false,
hoodie.metrics.jmx.port -> 9889,
hoodie.writestatus.class -> org.apache.hudi.client.WriteStatus,
hoodie.datasource.hive_sync.enable -> true,
hoodie.finalize.write.parallelism -> 1500,
hoodie.rollback.parallelism -> 100,
hoodie.index.bloom.num_entries -> 60000,
hoodie.memory.merge.max.size -> 134217728,
hoodie.bootstrap.mode.selector.regex.mode -> METADATA_ONLY,
hoodie.rollback.using.markers -> false,
hoodie.copyonwrite.record.size.estimate -> 1024,
hoodie.bloom.index.input.storage.level -> MEMORY_AND_DISK_SER,
hoodie.simple.index.parallelism -> 50,
hoodie.consistency.check.enabled -> false,
hoodie.bloom.index.use.caching -> true,
hoodie.metrics.on -> false,
hoodie.memory.compaction.max.size -> 1073741824,
hoodie.parquet.small.file.limit -> 104857600,
hoodie.combine.before.insert -> false,
hoodie.cleaner.commits.retained -> 1,
hoodie.embed.timeline.server -> true,
hoodie.bootstrap.mode.selector -> org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector,
hoodie.filesystem.view.secondary.type -> MEMORY,
_.hoodie.allow.multi.write.on.same.instant -> false,
hoodie.datasource.write.partitionpath.field -> ,
_hoodie.optimistic.consistency.guard.enable -> true,
hoodie.datasource.hive_sync.database -> hbase,
hoodie.bloom.index.update.partition.path -> true,
hoodie.fail.on.timeline.archiving -> true,
hoodie.markers.delete.parallelism -> 100,
hoodie.filesystem.view.type -> MEMORY,
hoodie.parquet.max.file.size -> 125829120,
hoodie.datasource.write.keygenerator.class -> org.apache.hudi.keygen.NonpartitionedKeyGenerator,
hoodie.bootstrap.partitionpath.translator.class -> org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPartitionPathTranslator,
hoodie.bloom.index.prune.by.ranges -> true,
hoodie.base.path -> /user/datalake/hudi/hbase/f_mid_business_card,
hoodie.index.class -> ,
hoodie.clean.automatic -> true,
hoodie.filesystem.view.remote.host -> localhost,
hoodie.compaction.lazy.block.read -> false,
hoodie.memory.writestatus.failure.fraction -> 0.1,
hoodie.metrics.graphite.port -> 4756,
hoodie.cleaner.policy -> KEEP_LATEST_FILE_VERSIONS,
hoodie.logfile.max.size -> 1073741824,
hoodie.filesystem.view.spillable.compaction.mem.fraction -> 0.01,
hoodie.datasource.write.recordkey.field -> datalake_rowkey,
hoodie.avro.schema.validate -> false,
hoodie.simple.index.input.storage.level -> MEMORY_AND_DISK_SER,
hoodie.timeline.layout.version -> 1,
hoodie.consistency.check.max_checks -> 7,
hoodie.consistency.check.initial_interval_ms -> 2000,
hoodie.keep.max.commits -> 8,
hoodie.compact.inline.max.delta.commits -> 5,
hoodie.parquet.compression.ratio -> 0.1,
hoodie.memory.dfs.buffer.max.size -> 16777216,
hoodie.auto.commit -> false,
hoodie.write.commit.callback.http.api.key -> hudi_write_commit_http_callback,
hoodie.assume.date.partitioning -> false,
hoodie.filesystem.view.spillable.dir -> /tmp/view_map/,
hoodie.compaction.strategy -> org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy,
hoodie.combine.before.upsert -> true,
hoodie.bloom.index.keys.per.bucket -> 10000000,
hoodie.write.commit.callback.class -> org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback,
hoodie.bloom.index.parallelism -> 0,
hoodie.cleaner.incremental.mode -> true,
hoodie.commits.archival.batch -> 5,
hoodie.datasource.hive_sync.partition_fields -> ,
hoodie.compaction.target.io -> 512000,
hoodie.table.name -> f_mid_business_card,
hoodie.bloom.index.bucketized.checking -> true,
hoodie.compaction.payload.class -> org.apache.hudi.common.model.OverwriteWithLatestAvroPayload,
hoodie.combine.before.delete -> true,
hoodie.datasource.write.precombine.field -> ts,
hoodie.filesystem.view.spillable.bootstrap.base.file.mem.fraction -> 0.05,
hoodie.metrics.jmx.host -> localhost,
hoodie.index.bloom.fpp -> 0.000000001,
hoodie.datasource.hive_sync.jdbcurl -> jdbc:hive2://172.16.116.102:10000,
hoodie.bloom.index.use.treebased.filter -> true

@peng-xin : Can you attach the contents of hoodie.properties file here. This is most likely due to setting  timeline layout version wrongly.
Can you start from a clean slate (new base path) and not pass "hoodie.timeline.layout.version" in configs and try and let us know ?

@peng-xin : Can you attach the contents of hoodie.properties file here. This is most likely due to setting  timeline layout version wrongly.
Can you start from a clean slate (new base path) and not pass "hoodie.timeline.layout.version" in configs and try and let us know ?

@peng-xin : Can you attach the contents of hoodie.properties file here. This is most likely due to setting timeline layout version wrongly.
Can you start from a clean slate (new base path) and not pass "hoodie.timeline.layout.version" in configs and try and let us know ?

the old hoodie.properties is there:

i drop the old data and removed the config hoodie.timeline.layout.version,but that happened again.

now the config is there:

hoodie.filesystem.view.incr.timeline.sync.enable -> false,
hoodie.bulkinsert.sort.mode -> GLOBAL_SORT,
hoodie.bootstrap.parallelism -> 1500,
hoodie.avro.schema.externalTransformation -> false,
hoodie.delete.shuffle.parallelism -> 1500,
hoodie.simple.index.use.caching -> true,
hoodie.bloom.index.filter.type -> DYNAMIC_V0,
hoodie.filesystem.view.remote.port -> 26754,
hoodie.datasource.write.operation -> upsert,
hoodie.cleaner.parallelism -> 200,
hoodie.global.simple.index.parallelism -> 100,
hoodie.bootstrap.mode.selector.regex -> .*,
hoodie.parquet.page.size -> 1048576,
hoodie.datasource.write.table.type -> MERGE_ON_READ,
hoodie.datasource.hive_sync.table -> f_mid_business_card,
hoodie.compaction.daybased.target.partitions -> 10,
hoodie.metrics.reporter.class -> ,
hoodie.parquet.block.size -> 125829120,
hoodie.cleaner.delete.bootstrap.base.file -> false,
hoodie.consistency.check.max_interval_ms -> 300000,
hoodie.insert.shuffle.parallelism -> 100,
hoodie.upsert.shuffle.parallelism -> 100,
hoodie.bulkinsert.shuffle.parallelism -> 1500,
hoodie.write.commit.callback.on -> false,
hoodie.cleaner.fileversions.retained -> 1,
hoodie.datasource.hive_sync.partition_extractor_class -> org.apache.hudi.hive.NonPartitionedExtractor,
hoodie.parquet.compression.codec -> gzip,
hoodie.datasource.write.hive_style_partitioning -> true,
hoodie.copyonwrite.insert.split.size -> 500000,
hoodie.optimistic.consistency.guard.sleep_time_ms -> 500,
hoodie.datasource.hive_sync.use_jdbc -> true,
hoodie.metrics.reporter.type -> GRAPHITE,
hoodie.bootstrap.index.class -> org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex,
hoodie.logfile.to.parquet.compression.ratio -> 0.35,
hoodie.filesystem.remote.backup.view.enable -> true,
hoodie.filesystem.view.spillable.mem -> 104857600,
hoodie.write.status.storage.level -> MEMORY_AND_DISK_SER,
hoodie.write.commit.callback.http.timeout.seconds -> 3,
hoodie.copyonwrite.insert.auto.split -> true,
hoodie.logfile.data.block.max.size -> 268435456,
hoodie.index.type -> BLOOM,
hoodie.keep.min.commits -> 6,
hoodie.memory.spillable.map.path -> /tmp/,
hoodie.filesystem.view.rocksdb.base.path -> /tmp/hoodie_timeline_rocksdb,
hoodie.compact.inline -> false,
hoodie.clean.async -> true,
hoodie.record.size.estimation.threshold -> 1.0,
hoodie.simple.index.update.partition.path -> false,
hoodie.bloom.index.filter.dynamic.max.entries -> 100000,
hoodie.metrics.graphite.host -> localhost,
hoodie.compaction.reverse.log.read -> false,
hoodie.metrics.jmx.port -> 9889,
hoodie.datasource.hive_sync.enable -> true,
hoodie.writestatus.class -> org.apache.hudi.client.WriteStatus,
hoodie.finalize.write.parallelism -> 1500,
hoodie.rollback.parallelism -> 100,
hoodie.index.bloom.num_entries -> 60000,
hoodie.memory.merge.max.size -> 134217728,
hoodie.bootstrap.mode.selector.regex.mode -> METADATA_ONLY,
hoodie.rollback.using.markers -> false,
hoodie.copyonwrite.record.size.estimate -> 1024,
hoodie.bloom.index.input.storage.level -> MEMORY_AND_DISK_SER,
hoodie.simple.index.parallelism -> 50,
hoodie.consistency.check.enabled -> false,
hoodie.bloom.index.use.caching -> true,
hoodie.memory.compaction.max.size -> 1073741824,
hoodie.metrics.on -> false,
hoodie.parquet.small.file.limit -> 104857600,
hoodie.combine.before.insert -> false,
hoodie.cleaner.commits.retained -> 1,
hoodie.embed.timeline.server -> true,
hoodie.bootstrap.mode.selector -> org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector,
hoodie.datasource.write.partitionpath.field -> ,
_.hoodie.allow.multi.write.on.same.instant -> false,
hoodie.filesystem.view.secondary.type -> MEMORY,
_hoodie.optimistic.consistency.guard.enable -> true,
hoodie.datasource.hive_sync.database -> hbase,
hoodie.bloom.index.update.partition.path -> true,
hoodie.fail.on.timeline.archiving -> true,
hoodie.markers.delete.parallelism -> 100,
hoodie.datasource.write.keygenerator.class -> org.apache.hudi.keygen.NonpartitionedKeyGenerator,
hoodie.parquet.max.file.size -> 125829120,
hoodie.filesystem.view.type -> MEMORY,
hoodie.bootstrap.partitionpath.translator.class -> org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPartitionPathTranslator,
hoodie.bloom.index.prune.by.ranges -> true,
hoodie.base.path -> /user/datalake/hudi/hbase/f_mid_business_card,
hoodie.clean.automatic -> true,
hoodie.index.class -> ,
hoodie.compaction.lazy.block.read -> false,
hoodie.filesystem.view.remote.host -> localhost,
hoodie.memory.writestatus.failure.fraction -> 0.1,
hoodie.metrics.graphite.port -> 4756,
hoodie.cleaner.policy -> KEEP_LATEST_FILE_VERSIONS,
hoodie.logfile.max.size -> 1073741824,
hoodie.filesystem.view.spillable.compaction.mem.fraction -> 0.01,
hoodie.datasource.write.recordkey.field -> datalake_rowkey,
hoodie.simple.index.input.storage.level -> MEMORY_AND_DISK_SER,
hoodie.avro.schema.validate -> false,
hoodie.consistency.check.max_checks -> 7,
hoodie.keep.max.commits -> 8,
hoodie.consistency.check.initial_interval_ms -> 2000,
hoodie.compact.inline.max.delta.commits -> 5,
hoodie.parquet.compression.ratio -> 0.1,
hoodie.memory.dfs.buffer.max.size -> 16777216,
hoodie.auto.commit -> false,
hoodie.write.commit.callback.http.api.key -> hudi_write_commit_http_callback,
hoodie.assume.date.partitioning -> false,
hoodie.filesystem.view.spillable.dir -> /tmp/view_map/,
hoodie.compaction.strategy -> org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy,
hoodie.bloom.index.keys.per.bucket -> 10000000,
hoodie.combine.before.upsert -> true,
hoodie.cleaner.incremental.mode -> true,
hoodie.bloom.index.parallelism -> 0,
hoodie.write.commit.callback.class -> org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback,
hoodie.commits.archival.batch -> 5,
hoodie.compaction.target.io -> 512000,
hoodie.datasource.hive_sync.partition_fields -> ,
hoodie.table.name -> f_mid_business_card,
hoodie.bloom.index.bucketized.checking -> true,
hoodie.compaction.payload.class -> org.apache.hudi.common.model.OverwriteWithLatestAvroPayload,
hoodie.datasource.write.precombine.field -> ts,
hoodie.combine.before.delete -> true,
hoodie.filesystem.view.spillable.bootstrap.base.file.mem.fraction -> 0.05,
hoodie.metrics.jmx.host -> localhost,
hoodie.index.bloom.fpp -> 0.000000001,
hoodie.bloom.index.use.treebased.filter -> true,
hoodie.datasource.hive_sync.jdbcurl -> jdbc:hive2://172.16.116.102:10000

@peng-xin : Can you attach the contents of hoodie.properties file here. This is most likely due to setting timeline layout version wrongly.
Can you start from a clean slate (new base path) and not pass "hoodie.timeline.layout.version" in configs and try and let us know ?

the old hoodie.properties is there:

i drop the old data and removed the config hoodie.timeline.layout.version,but that happened again.

now the config is there:

hoodie.filesystem.view.incr.timeline.sync.enable -> false,
hoodie.bulkinsert.sort.mode -> GLOBAL_SORT,
hoodie.bootstrap.parallelism -> 1500,
hoodie.avro.schema.externalTransformation -> false,
hoodie.delete.shuffle.parallelism -> 1500,
hoodie.simple.index.use.caching -> true,
hoodie.bloom.index.filter.type -> DYNAMIC_V0,
hoodie.filesystem.view.remote.port -> 26754,
hoodie.datasource.write.operation -> upsert,
hoodie.cleaner.parallelism -> 200,
hoodie.global.simple.index.parallelism -> 100,
hoodie.bootstrap.mode.selector.regex -> .*,
hoodie.parquet.page.size -> 1048576,
hoodie.datasource.write.table.type -> MERGE_ON_READ,
hoodie.datasource.hive_sync.table -> f_mid_business_card,
hoodie.compaction.daybased.target.partitions -> 10,
hoodie.metrics.reporter.class -> ,
hoodie.parquet.block.size -> 125829120,
hoodie.cleaner.delete.bootstrap.base.file -> false,
hoodie.consistency.check.max_interval_ms -> 300000,
hoodie.insert.shuffle.parallelism -> 100,
hoodie.upsert.shuffle.parallelism -> 100,
hoodie.bulkinsert.shuffle.parallelism -> 1500,
hoodie.write.commit.callback.on -> false,
hoodie.cleaner.fileversions.retained -> 1,
hoodie.datasource.hive_sync.partition_extractor_class -> org.apache.hudi.hive.NonPartitionedExtractor,
hoodie.parquet.compression.codec -> gzip,
hoodie.datasource.write.hive_style_partitioning -> true,
hoodie.copyonwrite.insert.split.size -> 500000,
hoodie.optimistic.consistency.guard.sleep_time_ms -> 500,
hoodie.datasource.hive_sync.use_jdbc -> true,
hoodie.metrics.reporter.type -> GRAPHITE,
hoodie.bootstrap.index.class -> org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex,
hoodie.logfile.to.parquet.compression.ratio -> 0.35,
hoodie.filesystem.remote.backup.view.enable -> true,
hoodie.filesystem.view.spillable.mem -> 104857600,
hoodie.write.status.storage.level -> MEMORY_AND_DISK_SER,
hoodie.write.commit.callback.http.timeout.seconds -> 3,
hoodie.copyonwrite.insert.auto.split -> true,
hoodie.logfile.data.block.max.size -> 268435456,
hoodie.index.type -> BLOOM,
hoodie.keep.min.commits -> 6,
hoodie.memory.spillable.map.path -> /tmp/,
hoodie.filesystem.view.rocksdb.base.path -> /tmp/hoodie_timeline_rocksdb,
hoodie.compact.inline -> false,
hoodie.clean.async -> true,
hoodie.record.size.estimation.threshold -> 1.0,
hoodie.simple.index.update.partition.path -> false,
hoodie.bloom.index.filter.dynamic.max.entries -> 100000,
hoodie.metrics.graphite.host -> localhost,
hoodie.compaction.reverse.log.read -> false,
hoodie.metrics.jmx.port -> 9889,
hoodie.datasource.hive_sync.enable -> true,
hoodie.writestatus.class -> org.apache.hudi.client.WriteStatus,
hoodie.finalize.write.parallelism -> 1500,
hoodie.rollback.parallelism -> 100,
hoodie.index.bloom.num_entries -> 60000,
hoodie.memory.merge.max.size -> 134217728,
hoodie.bootstrap.mode.selector.regex.mode -> METADATA_ONLY,
hoodie.rollback.using.markers -> false,
hoodie.copyonwrite.record.size.estimate -> 1024,
hoodie.bloom.index.input.storage.level -> MEMORY_AND_DISK_SER,
hoodie.simple.index.parallelism -> 50,
hoodie.consistency.check.enabled -> false,
hoodie.bloom.index.use.caching -> true,
hoodie.memory.compaction.max.size -> 1073741824,
hoodie.metrics.on -> false,
hoodie.parquet.small.file.limit -> 104857600,
hoodie.combine.before.insert -> false,
hoodie.cleaner.commits.retained -> 1,
hoodie.embed.timeline.server -> true,
hoodie.bootstrap.mode.selector -> org.apache.hudi.client.bootstrap.selector.MetadataOnlyBootstrapModeSelector,
hoodie.datasource.write.partitionpath.field -> ,
_.hoodie.allow.multi.write.on.same.instant -> false,
hoodie.filesystem.view.secondary.type -> MEMORY,
_hoodie.optimistic.consistency.guard.enable -> true,
hoodie.datasource.hive_sync.database -> hbase,
hoodie.bloom.index.update.partition.path -> true,
hoodie.fail.on.timeline.archiving -> true,
hoodie.markers.delete.parallelism -> 100,
hoodie.datasource.write.keygenerator.class -> org.apache.hudi.keygen.NonpartitionedKeyGenerator,
hoodie.parquet.max.file.size -> 125829120,
hoodie.filesystem.view.type -> MEMORY,
hoodie.bootstrap.partitionpath.translator.class -> org.apache.hudi.client.bootstrap.translator.IdentityBootstrapPartitionPathTranslator,
hoodie.bloom.index.prune.by.ranges -> true,
hoodie.base.path -> /user/datalake/hudi/hbase/f_mid_business_card,
hoodie.clean.automatic -> true,
hoodie.index.class -> ,
hoodie.compaction.lazy.block.read -> false,
hoodie.filesystem.view.remote.host -> localhost,
hoodie.memory.writestatus.failure.fraction -> 0.1,
hoodie.metrics.graphite.port -> 4756,
hoodie.cleaner.policy -> KEEP_LATEST_FILE_VERSIONS,
hoodie.logfile.max.size -> 1073741824,
hoodie.filesystem.view.spillable.compaction.mem.fraction -> 0.01,
hoodie.datasource.write.recordkey.field -> datalake_rowkey,
hoodie.simple.index.input.storage.level -> MEMORY_AND_DISK_SER,
hoodie.avro.schema.validate -> false,
hoodie.consistency.check.max_checks -> 7,
hoodie.keep.max.commits -> 8,
hoodie.consistency.check.initial_interval_ms -> 2000,
hoodie.compact.inline.max.delta.commits -> 5,
hoodie.parquet.compression.ratio -> 0.1,
hoodie.memory.dfs.buffer.max.size -> 16777216,
hoodie.auto.commit -> false,
hoodie.write.commit.callback.http.api.key -> hudi_write_commit_http_callback,
hoodie.assume.date.partitioning -> false,
hoodie.filesystem.view.spillable.dir -> /tmp/view_map/,
hoodie.compaction.strategy -> org.apache.hudi.table.action.compact.strategy.LogFileSizeBasedCompactionStrategy,
hoodie.bloom.index.keys.per.bucket -> 10000000,
hoodie.combine.before.upsert -> true,
hoodie.cleaner.incremental.mode -> true,
hoodie.bloom.index.parallelism -> 0,
hoodie.write.commit.callback.class -> org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback,
hoodie.commits.archival.batch -> 5,
hoodie.compaction.target.io -> 512000,
hoodie.datasource.hive_sync.partition_fields -> ,
hoodie.table.name -> f_mid_business_card,
hoodie.bloom.index.bucketized.checking -> true,
hoodie.compaction.payload.class -> org.apache.hudi.common.model.OverwriteWithLatestAvroPayload,
hoodie.datasource.write.precombine.field -> ts,
hoodie.combine.before.delete -> true,
hoodie.filesystem.view.spillable.bootstrap.base.file.mem.fraction -> 0.05,
hoodie.metrics.jmx.host -> localhost,
hoodie.index.bloom.fpp -> 0.000000001,
hoodie.bloom.index.use.treebased.filter -> true,
hoodie.datasource.hive_sync.jdbcurl -> jdbc:hive2://172.16.116.102:10000

@peng-xin : Can you enable hoodie.compact.inline -> true and hoodie.auto.commit -> true. The log files are growing because they need to be compacted and if you set the first config, it will periodically run compactions. Cleaner will eventually remove old log files and parquet files after that.

@peng-xin : Can you enable hoodie.compact.inline -> true and hoodie.auto.commit -> true. The log files are growing because they need to be compacted and if you set the first config, it will periodically run compactions. Cleaner will eventually remove old log files and parquet files after that.

@peng-xin : Can you enable hoodie.compact.inline -> true and hoodie.auto.commit -> true. The log files are growing because they need to be compacted and if you set the first config, it will periodically run compactions. Cleaner will eventually remove old log files and parquet files after that.

thank you so much.
when i set hoodie.compact.inline -> true,the size of log be limited.
but  hoodie.auto.commit -> true will cause the same error

@peng-xin : Can you enable hoodie.compact.inline -> true and hoodie.auto.commit -> true. The log files are growing because they need to be compacted and if you set the first config, it will periodically run compactions. Cleaner will eventually remove old log files and parquet files after that.

thank you so much.
when i set hoodie.compact.inline -> true,the size of log be limited.
but  hoodie.auto.commit -> true will cause the same error

@peng-xin Are you able to proceed with hoodie.compact.inline -> true and hoodie.auto.commit -> false ?

Guess you might have to fix the max file size. I see currently you set it to very high value. Was that intentional ?
hoodie.parquet.max.file.size -> 125829120,
hoodie.logfile.max.size -> 1073741824
Can you try setting the values as follows:
parquet max file size: 120Mb
@vinothchandar : any recommendation for log file max size?

@peng-xin : few quick questions as we triage the issue.

Were you running older version of Hudi and encountered this after upgrade? in other words, older Hudi version you were able to run successfully and with 0.7.0 there is a bug.
Is this affecting your production? trying to gauge the severity.
Or you are trying out a POC ? and this is the first time trying out Hudi.

if you set "hoodie.compact.inline -> true",this means compaction  changes inline, but async?

nope. inline means sync. if not its async.

but I need async, if I set "hoodie.compact.inline -> true",that is not async.

@root18039532923 : Please look at https://hudi.apache.org/blog/async-compaction-deployment-model/ for running async compactions

INLINE_COMPACT_NUM_DELTA_COMMITS_PROP is the configuration of sync?why does async need to add this?
I know that The default value of ASYNC_COMPACT_ENABLE_OPT_KEY is true.

@peng-xin Since we haven't heard from you in a while and this issue has not been reported by anyone else, I'm assuming this to be a transient issue with some of your settings. Let me know if you need further help.
@root18039532923 Please feel free to re-open if you are still confused about how to use async compaction.

source: gharchive/issue

repo: apache/hudi


text:
NoClassDefFoundError: org/apache/parquet/schema/LogicalTypeAnnotation$LogicalTypeAnnotationVisitor Caused by: ClassNotFoundException: org.apache.parquet.schema.LogicalTypeAnnotation$LogicalTypeAnnotationVisitor

Hi Team ,
Getting the below issue in hudi job while executing  through databricks.
ERROR details:-
NoClassDefFoundError: org/apache/parquet/schema/LogicalTypeAnnotation$LogicalTypeAnnotationVisitor Caused by: ClassNotFoundException: org.apache.parquet.schema.LogicalTypeAnnotation$LogicalTypeAnnotationVisitor
We are running the hudi code through databricks .
Environment Description

Hudi version : 0.11.0

Spark version : 3.1.2

Scala Version :2.12

Storage (HDFS/S3/GCS..) : AZURE Blob Storage

AVRO SOURCE we are using to read the kafka data .

Used  9.1 LTS  Databricks RUNTIME version

With same configurations job succeeded in non prod environment  getting error in prod environment
Please find the attached details and log file ,Let me know the resolution steps
Full log -
log4j-active (43).txt

can you share the complete spark-submit command with args? need to see what jars are used.

With same configurations job succeeded in non prod environment getting error in prod environment

so this is related to your environment discrepancy. are you able to check from your side what jars not present in your prod env?

Hi @xushiyan ,
i verified  the both non prod and prod jars , it is same in both environments .
Please find the attached screenshot of Databrick job details.

Parameters that we used for the job :-
["--table-type","COPY_ON_WRITE","--source-ordering-field","CDC_TS","--source-limit","1000","--source-class","com.optum.df.hudi.sources.DFAvroKafkaSource","--target-base-path","/mnt/ulp/dataassets-lake/metrics/","--target-table","metrics","--schemaprovider-class","org.apache.hudi.utilities.schema.SchemaRegistryProvider","--props","/mnt/ulp/artifacts/properties/metrics.properties"]

spark-avro package you used is mismatch; it's for spark 3.2.1 but your main spark is 3.1. can you refer to 0.11 release note migration guide? spark-avro is not required. And also you don't need spark3-bundle, as you already used utilities-bundle

HI @xushiyan ,
I have made the changes suggested by you, removed spark3-bundle and spark_avro and spark3 bundle dependencies are removed and submitted the job it's giving the another error .
ERROR details:-
HoodieException: Commit 20220530133502327 failed and rolled-back !
at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:649)
Please find the attached screen shot  and log file .
log4j-active (44).txt

The new exception is application-level error, specific to your data and scenario, not related to dependency. Please examine your executor logs and data for details. Closing this as the original problem about the dependency was resolved.

license_type: permissive

license_source: github-api


text:
[SUPPORT] Spark multi writer failed ! ! !

Describe the problem you faced
when i use config 1 ,The exception A occurred in one of the two spark jobs writing the same table;
when i use config 2 ,The exception B occurred in one of the two spark jobs writing the same table.
there was not clustering config .
To Reproduce
config 1:
hoodie.write.concurrency.mode=optimistic_concurrency_control
hoodie.cleaner.policy.failed.writes=LAZY
hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
hoodie.write.lock.zookeeper.url=10.1.2.12
hoodie.write.lock.zookeeper.port=2181
hoodie.write.lock.zookeeper.lock_key=ss_bucket_lock
hoodie.write.lock.zookeeper.base_path=/ss_bucket_lock
config 2 :
hoodie.write.concurrency.mode=optimistic_concurrency_control
hoodie.cleaner.policy.failed.writes=LAZY
hoodie.write.lock.provider=org.apache.hudi.hive.HiveMetastoreBasedLockProvider
hoodie.write.lock.hivemetastore.database=default
hoodie.write.lock.hivemetastore.table=occ_lock_ss_bucket_dsj
hoodie.write.lock.hivemetastore.uris=thrift://host-10-4-6-15:9083
Expected behavior
Two spark jobs could write the same hudi table  normally at the same time !
Environment Description

Hudi version :
Hudi-0.11.1
Spark version :
spark-3.1.x
Hive version :
None
Hadoop version :
Hadoop-3.3.0
Storage (HDFS/S3/GCS..) :
Hdfs
Running on Docker? (yes/no) :
No

**Stacktrace  ，Exception A **
22/08/05 15:01:15 INFO CommitUtils: Creating  metadata for INSERT numWriteStats:1numReplaceFileIds:0
22/08/05 15:01:15 INFO TransactionManager: Transaction starting for Option{val=[==>20220805150110559__commit__INFLIGHT]} with latest completed transaction instant Option{val=[20220805150050505__commit__COMPLETED]}
22/08/05 15:01:15 INFO ZookeeperBasedLockProvider: ACQUIRING lock atZkBasePath = /ss_bucket_lock, lock key = ss_bucket_lock
22/08/05 15:01:18 INFO ZookeeperBasedLockProvider: ACQUIRED lock atZkBasePath = /ss_bucket_lock, lock key = ss_bucket_lock
22/08/05 15:01:18 INFO TransactionManager: Transaction started for Option{val=[==>20220805150110559__commit__INFLIGHT]} with latest completed transaction instant Option{val=[20220805150050505__commit__COMPLETED]}
22/08/05 15:01:18 INFO HoodieTableMetaClient: Loading HoodieTableMetaClient from /tmp/hudi/ss_bucket_dsj
22/08/05 15:01:18 INFO HoodieTableConfig: Loading table properties from /tmp/hudi/ss_bucket_dsj/.hoodie/hoodie.properties
22/08/05 15:01:18 INFO HoodieTableMetaClient: Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from /tmp/hudi/ss_bucket_dsj
22/08/05 15:01:18 INFO HoodieTableMetaClient: Loading Active commit timeline for /tmp/hudi/ss_bucket_dsj
22/08/05 15:01:18 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220805150110559__commit__INFLIGHT]}
22/08/05 15:01:18 INFO FileSystemViewManager: Creating View Manager with storage type :MEMORY
22/08/05 15:01:18 INFO FileSystemViewManager: Creating in-memory based Table View
22/08/05 15:01:18 INFO HoodieBucketIndex: use bucket index, numBuckets=1
22/08/05 15:01:18 INFO HoodieActiveTimeline: Loaded instants upto : Option{val=[==>20220805150110559__commit__INFLIGHT]}
22/08/05 15:01:18 INFO SimpleConcurrentFileWritesConflictResolutionStrategy: Found conflicting writes between first operation = {actionType=commit, instantTime=20220805150110559, actionState=INFLIGHT'}, second operation = {actionType=commit, instantTime=20220805150100333, actionState=COMPLETED'} , intersecting file ids [00000000-4447-4513-8068-326e01720c62-0]
22/08/05 15:01:18 INFO TransactionUtils: Conflict encountered between current instant = {actionType=commit, instantTime=20220805150110559, actionState=INFLIGHT'} and instant = {actionType=commit, instantTime=20220805150100333, actionState=COMPLETED'}, attempting to resolve it...
22/08/05 15:01:18 INFO TransactionManager: Transaction ending with transaction owner Option{val=[==>20220805150110559__commit__INFLIGHT]}
22/08/05 15:01:18 INFO ZookeeperBasedLockProvider: RELEASING lock atZkBasePath = /ss_bucket_lock, lock key = ss_bucket_lock
22/08/05 15:01:18 INFO ZookeeperBasedLockProvider: RELEASED lock atZkBasePath = /ss_bucket_lock, lock key = ss_bucket_lock
22/08/05 15:01:18 INFO TransactionManager: Transaction ended with transaction owner Option{val=[==>20220805150110559__commit__INFLIGHT]}
22/08/05 15:01:18 INFO MapPartitionsRDD: Removing RDD 167 from persistence list
22/08/05 15:01:18 INFO MapPartitionsRDD: Removing RDD 159 from persistence list
22/08/05 15:01:18 INFO BlockManager: Removing RDD 167
22/08/05 15:01:18 ERROR HoodieStreamingSink: Micro batch id=32 threw following exception:
org.apache.hudi.exception.HoodieWriteConflictException: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
        at org.apache.hudi.client.transaction.SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict(SimpleConcurrentFileWritesConflictResolutionStrategy.java:102)
        at org.apache.hudi.client.utils.TransactionUtils.lambda$resolveWriteConflictIfAny$0(TransactionUtils.java:85)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
        at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
        at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
        at org.apache.hudi.client.utils.TransactionUtils.resolveWriteConflictIfAny(TransactionUtils.java:79)
        at org.apache.hudi.client.SparkRDDWriteClient.preCommit(SparkRDDWriteClient.java:473)
        at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:233)
        at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:122)
        at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:651)
        at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:315)
        at org.apache.hudi.HoodieStreamingSink.$anonfun$addBatch$2(HoodieStreamingSink.scala:91)
        at scala.util.Try$.apply(Try.scala:213)
        at org.apache.hudi.HoodieStreamingSink.$anonfun$addBatch$1(HoodieStreamingSink.scala:90)
        at org.apache.hudi.HoodieStreamingSink.retry(HoodieStreamingSink.scala:166)
        at org.apache.hudi.HoodieStreamingSink.addBatch(HoodieStreamingSink.scala:89)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$16(MicroBatchExecution.scala:586)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$15(MicroBatchExecution.scala:584)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runBatch(MicroBatchExecution.scala:584)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:226)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:194)
        at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:57)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:188)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:333)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:244)
Caused by: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
        ... 38 more
22/08/05 15:01:18 INFO HoodieStreamingSink: Ignore the exception and move on streaming as per hoodie.datasource.write.streaming.ignore.failed.batch configuration
22/08/05 15:01:18 INFO HoodieStreamingSink: Micro batch id=32 succeeded
22/08/05 15:01:18 INFO BlockManager: Removing RDD 159
22/08/05 15:01:18 INFO CheckpointFileManager: Writing atomically to hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/32 using temp file hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/.32.a829b364-c1b9-4b4b-8fae-f7d866ed76e5.tmp
22/08/05 15:01:18 INFO CheckpointFileManager: Renamed temp file hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/.32.a829b364-c1b9-4b4b-8fae-f7d866ed76e5.tmp to hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/32

**Stacktrace  for Exception B **
22/08/05 14:35:08 INFO TransactionManager: Transaction starting for Option{val=[==>20220805143450492__commit__INFLIGHT]} with latest completed transaction instant Option{val=[20220805143430140__commit__COMPLETED]}
22/08/05 14:35:08 INFO HiveMetastoreBasedLockProvider: ACQUIRING lock at database default and table occ_lock_ss_bucket_dsj
22/08/05 14:35:08 INFO LockManager: Retrying to acquire lock...
22/08/05 14:35:18 INFO HiveMetastoreBasedLockProvider: ACQUIRING lock at database default and table occ_lock_ss_bucket_dsj
22/08/05 14:35:18 INFO MapPartitionsRDD: Removing RDD 6 from persistence list
22/08/05 14:35:18 INFO BlockManager: Removing RDD 6
22/08/05 14:35:18 INFO MapPartitionsRDD: Removing RDD 14 from persistence list
22/08/05 14:35:18 INFO BlockManager: Removing RDD 14
22/08/05 14:35:18 ERROR HoodieStreamingSink: Micro batch id=133 threw following exception:
java.lang.IllegalArgumentException: ALREADY_ACQUIRED
        at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:40)
        at org.apache.hudi.hive.HiveMetastoreBasedLockProvider.acquireLock(HiveMetastoreBasedLockProvider.java:136)
        at org.apache.hudi.hive.HiveMetastoreBasedLockProvider.tryLock(HiveMetastoreBasedLockProvider.java:112)
        at org.apache.hudi.client.transaction.lock.LockManager.lock(LockManager.java:67)
        at org.apache.hudi.client.transaction.TransactionManager.beginTransaction(TransactionManager.java:53)
        at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:230)
        at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:122)
        at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:651)
        at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:315)
        at org.apache.hudi.HoodieStreamingSink.$anonfun$addBatch$2(HoodieStreamingSink.scala:91)
        at scala.util.Try$.apply(Try.scala:213)
        at org.apache.hudi.HoodieStreamingSink.$anonfun$addBatch$1(HoodieStreamingSink.scala:90)
        at org.apache.hudi.HoodieStreamingSink.retry(HoodieStreamingSink.scala:166)
        at org.apache.hudi.HoodieStreamingSink.addBatch(HoodieStreamingSink.scala:89)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$16(MicroBatchExecution.scala:586)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$15(MicroBatchExecution.scala:584)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runBatch(MicroBatchExecution.scala:584)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:226)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:194)
        at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:57)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:188)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:333)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:244)
22/08/05 14:35:18 INFO HoodieStreamingSink: Ignore the exception and move on streaming as per hoodie.datasource.write.streaming.ignore.failed.batch configuration
22/08/05 14:35:18 INFO HoodieStreamingSink: Micro batch id=133 succeeded
22/08/05 14:35:18 ERROR MicroBatchExecution: Query [id = d0ebeccf-4600-4aba-b097-420cd8356448, runId = 233f22f7-66d9-47a8-9093-0381f157428a] terminated with error
java.lang.AssertionError: assertion failed: Concurrent update to the commit log. Multiple streaming jobs detected for 133
        at scala.Predef$.assert(Predef.scala:223)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$17(MicroBatchExecution.scala:602)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.withProgressLocked(MicroBatchExecution.scala:613)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runBatch(MicroBatchExecution.scala:598)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:226)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:194)
        at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:57)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:188)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:333)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:244)
Exception in thread "main" org.apache.spark.sql.streaming.StreamingQueryException: assertion failed: Concurrent update to the commit log. Multiple streaming jobs detected for 133
=== Streaming Query ===
Identifier: [id = d0ebeccf-4600-4aba-b097-420cd8356448, runId = 233f22f7-66d9-47a8-9093-0381f157428a]
Current Committed Offsets: {KafkaV2[Subscribe[user_hive4]]: {"user_hive4":{"2":49415,"1":47279,"0":49423}}}
Current Available Offsets: {KafkaV2[Subscribe[user_hive4]]: {"user_hive4":{"2":49752,"1":47604,"0":49760}}}

Current State: ACTIVE
Thread State: RUNNABLE

Logical Plan:
Project [id#23, cast(score#24 as double) AS score#31, sex#25, to_utc_timestamp(cast(ts#26 as timestamp), Asia/Shanghai) AS ts#32]
+- SubqueryAlias s
   +- Project [split(kafka_value#21, ,, -1)[0] AS id#23, split(kafka_value#21, ,, -1)[1] AS score#24, split(kafka_value#21, ,, -1)[2] AS sex#25, split(kafka_value#21, ,, -1)[3] AS ts#26]
      +- SubqueryAlias t
         +- Project [cast(value#8 as string) AS kafka_value#21]
            +- StreamingDataSourceV2Relation [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13], org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan@6a155037, KafkaV2[Subscribe[user_hive4]]

        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:354)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:244)
Caused by: java.lang.AssertionError: assertion failed: Concurrent update to the commit log. Multiple streaming jobs detected for 133
        at scala.Predef$.assert(Predef.scala:223)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$17(MicroBatchExecution.scala:602)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.withProgressLocked(MicroBatchExecution.scala:613)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runBatch(MicroBatchExecution.scala:598)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:226)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:357)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:355)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:194)
        at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:57)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:188)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:333)
        ... 1 more

I think the second issue was fixed in the 0.12 branch, can you try it?

I also met the second issue, will create a pr to fix it

I also met the second issue, this should already be fixed in master branch , you can try it out

@fengjian428 Thank you for your reply，I'll try the main branch..

I also met the second issue, this should already be fixed in master branch , you can try it out

@fengjian428 Thank you for your reply，I'll try the master branch..

The second issue was indeed fixed , thank you !

@eric9204 For the first failure, can you share the full write configs? Was there any pending/inflight compaction and clustering? Can you share the .hoodie directory under the base path of the table? I think the lock provider is running fine but conflict resolution failed. The timeline will help us build the sequence of events and see why the resolution failed.

@eric9204 For the first failure, can you share the full write configs? Was there any pending/inflight compaction and clustering? Can you share the .hoodie directory under the base path of the table? I think the lock provider is running fine but conflict resolution failed. The timeline will help us build the sequence of events and see why the resolution failed.

2022-08-08 13:33 /tmp/hudi/ss_bucket_dsj/.hoodie/.aux
2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/.heartbeat
2022-08-08 13:33 /tmp/hudi/ss_bucket_dsj/.hoodie/.schema
2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/.temp
1.5 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134200577.rollback
    0 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134200577.rollback.inflight
1.3 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134200577.rollback.requested
1.5 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134211686.rollback
    0 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134211686.rollback.inflight
1.3 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134211686.rollback.requested
1.5 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134216453.rollback
    0 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134216453.rollback.inflight
1.3 K 2022-08-08 13:42 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134216453.rollback.requested
1.5 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134916968.rollback
    0 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134916968.rollback.inflight
1.3 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134916968.rollback.requested
1.5 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134917661.rollback
    0 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134917661.rollback.inflight
1.3 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134917661.rollback.requested
1.5 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134918933.rollback
    0 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134918933.rollback.inflight
1.3 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134918933.rollback.requested
1.5 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134919571.rollback
    0 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134919571.rollback.inflight
1.3 K 2022-08-08 13:49 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134919571.rollback.requested
1.5 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134920158.rollback
    0 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134920158.rollback.inflight
1.3 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134920158.rollback.requested
1.5 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134921135.rollback
    0 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134921135.rollback.inflight
1.3 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808134921135.rollback.requested
1.5 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135002565.rollback
    0 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135002565.rollback.inflight
1.3 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135002565.rollback.requested
1.5 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135003386.rollback
    0 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135003386.rollback.inflight
1.3 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135003386.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135013265.rollback
    0 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135013265.rollback.inflight
1.3 K 2022-08-08 13:50 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135013265.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135022179.rollback
    0 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135022179.rollback.inflight
1.3 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135022179.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135033233.rollback
    0 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135033233.rollback.inflight
1.3 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135033233.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135036166.rollback
    0 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135036166.rollback.inflight
1.3 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135036166.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135037732.rollback
    0 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135037732.rollback.inflight
1.3 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135037732.rollback.requested
1.5 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135043606.rollback
    0 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135043606.rollback.inflight
1.3 K 2022-08-08 13:51 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135043606.rollback.requested
1.5 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135212947.rollback
    0 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135212947.rollback.inflight
1.3 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135212947.rollback.requested
1.5 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135213720.rollback
    0 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135213720.rollback.inflight
1.3 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135213720.rollback.requested
1.5 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135215256.rollback
    0 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135215256.rollback.inflight
1.3 K 2022-08-08 13:52 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135215256.rollback.requested
1.5 K 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135330116.rollback
    0 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135330116.rollback.inflight
1.3 K 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135330116.rollback.requested
1.5 K 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135406717.rollback
    0 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135406717.rollback.inflight
1.3 K 2022-08-08 13:54 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135406717.rollback.requested
2.5 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135436042.commit
    0 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135436042.commit.requested
3.0 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135436042.inflight
1.5 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135447952.clean
1.6 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135447952.clean.inflight
1.6 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135447952.clean.requested
2.5 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135459430.commit
    0 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135459430.commit.requested
3.0 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135459430.inflight
1.5 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135509582.clean
1.6 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135509582.clean.inflight
1.6 K 2022-08-08 13:55 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135509582.clean.requested
2.5 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135540714.commit
    0 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135540714.commit.requested
3.0 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135540714.inflight
1.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552939.clean
1.6 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552939.clean.inflight
1.6 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552939.clean.requested
1.5 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552965.rollback
    0 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552965.rollback.inflight
1.3 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135552965.rollback.requested
1.5 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135624549.rollback
    0 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135624549.rollback.inflight
1.3 K 2022-08-08 13:56 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135624549.rollback.requested
2.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135634238.commit
    0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135634238.commit.requested
3.0 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135634238.inflight
1.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645734.clean
1.6 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645734.clean.inflight
1.6 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645734.clean.requested
1.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645758.rollback
    0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645758.rollback.inflight
1.3 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645758.rollback.requested
    0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645810.commit.requested
3.0 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135645810.inflight
1.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135646329.rollback
    0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135646329.rollback.inflight
1.3 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135646329.rollback.requested
2.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135659559.commit
    0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135659559.commit.requested
3.0 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135659559.inflight
1.5 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135711854.clean
1.6 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135711854.clean.inflight
1.6 K 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135711854.clean.requested
2.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135719322.commit
    0 2022-08-08 13:57 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135719322.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135719322.inflight
1.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135730874.clean
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135730874.clean.inflight
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135730874.clean.requested
2.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135731862.commit
    0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135731862.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135731862.inflight
    0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135734515.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135734515.inflight
1.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135736528.clean
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135736528.clean.inflight
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135736528.clean.requested
2.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135746974.commit
    0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135746974.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135746974.inflight
1.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135751508.clean
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135751508.clean.inflight
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135751508.clean.requested
    0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135753793.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135753793.inflight
2.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135754854.commit
    0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135754854.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135754854.inflight
1.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135802038.clean
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135802038.clean.inflight
1.6 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135802038.clean.requested
2.5 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135815814.commit
    0 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135815814.commit.requested
3.0 K 2022-08-08 13:58 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135815814.inflight
1.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135819555.clean
1.6 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135819555.clean.inflight
1.6 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135819555.clean.requested
2.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135849144.commit
    0 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135849144.commit.requested
3.0 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135849144.inflight
1.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906862.clean
1.6 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906862.clean.inflight
1.6 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906862.clean.requested
1.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906879.rollback
    0 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906879.rollback.inflight
1.3 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135906879.rollback.requested
1.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135907610.rollback
    0 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135907610.rollback.inflight
1.3 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135907610.rollback.requested
2.5 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135911957.commit
    0 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135911957.commit.requested
3.0 K 2022-08-08 13:59 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135911957.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135920430.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135920430.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135920430.clean.requested
2.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135925137.commit
    0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135925137.commit.requested
3.0 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135925137.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135930305.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135930305.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135930305.clean.requested
2.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135935514.commit
    0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135935514.commit.requested
3.0 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135935514.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135945906.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135945906.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808135945906.clean.requested
2.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140000922.commit
    0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140000922.commit.requested
3.0 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140000922.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140004948.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140004948.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140004948.clean.requested
2.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140006350.commit
    0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140006350.commit.requested
3.0 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140006350.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140010972.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140010972.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140010972.clean.requested
2.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140013016.commit
    0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140013016.commit.requested
3.0 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140013016.inflight
1.5 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140018094.clean
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140018094.clean.inflight
1.6 K 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140018094.clean.requested
2.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140020180.commit
    0 2022-08-08 14:00 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140020180.commit.requested
3.0 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140020180.inflight
1.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140028764.clean
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140028764.clean.inflight
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140028764.clean.requested
2.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140031809.commit
    0 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140031809.commit.requested
3.0 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140031809.inflight
1.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140035523.clean
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140035523.clean.inflight
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140035523.clean.requested
2.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140042109.commit
    0 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140042109.commit.requested
3.0 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140042109.inflight
1.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140046546.clean
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140046546.clean.inflight
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140046546.clean.requested
2.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140050105.commit
    0 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140050105.commit.requested
3.0 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140050105.inflight
1.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140055595.clean
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140055595.clean.inflight
1.6 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140055595.clean.requested
2.5 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140109135.commit
    0 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140109135.commit.requested
3.0 K 2022-08-08 14:01 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140109135.inflight
1.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140118730.clean
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140118730.clean.inflight
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140118730.clean.requested
2.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140132300.commit
    0 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140132300.commit.requested
3.0 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140132300.inflight
1.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140144323.clean
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140144323.clean.inflight
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140144323.clean.requested
2.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140204915.commit
    0 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140204915.commit.requested
3.0 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140204915.inflight
1.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140210226.clean
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140210226.clean.inflight
1.6 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140210226.clean.requested
    0 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140215991.commit.requested
3.0 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140215991.inflight
2.5 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140223090.commit
    0 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140223090.commit.requested
3.0 K 2022-08-08 14:02 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140223090.inflight
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140227701.clean
1.6 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140227701.clean.inflight
1.6 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140227701.clean.requested
2.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140232295.commit
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140232295.commit.requested
3.0 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140232295.inflight
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140238784.commit.requested
3.0 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140238784.inflight
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140246252.rollback
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140246252.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140246252.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140247998.rollback
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140247998.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140247998.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140253101.rollback
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140253101.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140253101.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140255842.rollback
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140255842.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140255842.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140303187.rollback
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140303187.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140303187.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140305407.rollback
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140305407.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140305407.rollback.requested
1.5 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140306513.rollback
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140306513.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140306513.rollback.requested
    0 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140308986.rollback.inflight
1.3 K 2022-08-08 14:03 /tmp/hudi/ss_bucket_dsj/.hoodie/20220808140308986.rollback.requested
    0 2022-08-08 13:41 /tmp/hudi/ss_bucket_dsj/.hoodie/archived
  781 2022-08-08 13:33 /tmp/hudi/ss_bucket_dsj/.hoodie/hoodie.properties

@codope  The hudi table type is cow,and there was not any config about compaction or clustering.

@eric9204 : just so you are aware of how conflict resolution happens w/in hudi.
If two writers are writing to same file, hudi will fail one of them and will succeed the other. But if two writers are writing to two different files, both should succeed (assuming both are concurrent writers).
For eg, lets say writer1 ingests 100 records in commit C1.
and at time t10, wirter1 updates the same 100 records and ingests to hudi. wirter2 updates the same 100 records and ingests to hudi. Hudi may not be in a position to choose the winner. and so will succeed one and will fail the other. In this situations, you will see ConflictResolutionFailed error messages.
But lets say writer1 ingests 100 records in commit C1 and writer2 ingests diff set of 100 records to a diff partition in commit C2.
at time t10, wirter1 updates records as part of C1 and ingests to hudi. writer2 updates the records as part of C2 and ingests to hudi. Both write will succeed if it happens concurrently. bcoz, there is no overlap.
Hope that clarifies why one of the writer failed in your case. you can get some additions details here.

@nsivabalan I know what you mean, there is another problem.
If one of writers fail, this hoodie commit transaction was failed at hudi side, but this batch was successful at spark side. So the next micro batch will consume record from last successful commit of offset. the data of last successful micro batch which is failed at hoodie side actually may be lost 。
22/08/05 15:01:18 INFO HoodieStreamingSink: Ignore the exception and move on streaming as per hoodie.datasource.write.streaming.ignore.failed.batch configuration
22/08/05 15:01:18 INFO HoodieStreamingSink: Micro batch id=32 succeeded
22/08/05 15:01:18 INFO BlockManager: Removing RDD 159
22/08/05 15:01:18 INFO CheckpointFileManager: Writing atomically to hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/32 using temp file hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/.32.a829b364-c1b9-4b4b-8fae-f7d866ed76e5.tmp
22/08/05 15:01:18 INFO CheckpointFileManager: Renamed temp file hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/.32.a829b364-c1b9-4b4b-8fae-f7d866ed76e5.tmp to hdfs://host-10-4-6-18:8020/tmp/hudi/ckp1/commits/32

Could the community add a retry strategy to make it successful instead of just discarding it.

yes. we have a config which you can flip to ensure spark streaming fails if hudi write fails.
hoodie.datasource.write.streaming.ignore.failed.batch
default value for this config is set to true until 2 weeks back. we have flipped the default to false. if you can set this config value to false, atleast you won' see any data loss. if there are any valid errors, your pipeline should fail.

@nsivabalan thanks! I've retested with hoodie.datasource.write.streaming.ignore.failed.batch=false ,the spark micro-batch indeed fail when hudi commit fail.
So, should I close this issue?

yes, thanks!