text:
[SUPPORT] - Hudi Read on a MOR table is failing with ArrayIndexOutOfBound exception

Tips before filing an issue

Have you gone through our FAQs?

Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced
A clear and concise description of the problem.
To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.
Environment Description

Hudi version :

Spark version :

Hive version :

Hadoop version :

Storage (HDFS/S3/GCS..) :

Running on Docker? (yes/no) :

Additional context
Add any other context about the problem here.
Stacktrace
Add the stacktrace of the error.
Detailed Notes -
We have Incoming Delta transactions from an Oracle based application that are being pushed into S3 endpoint using AWS DMS services. These CDC records are applied as upserts on to already existing Hudi table in a different S3 bucket (Initial Load data). The UPSERTS are happening by running below Spark Submits -
spark-submit
--deploy-mode client
--class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer
--conf spark.shuffle.service.enabled=true
--conf spark.default.parallelism=500
--conf spark.dynamicAllocation.enabled=true
--conf spark.dynamicAllocation.initialExecutors=3
--conf spark.dynamicAllocation.cachedExecutorIdleTimeout=90s
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer
--conf spark.app.name=<table_1>
--jars /usr/lib/spark/external/lib/spark-avro.jar,/usr/lib/hive/lib/hbase-client.jar /usr/lib/hudi/hudi-utilities-bundle.jar
--table-type MERGE_ON_READ
--op UPSERT
--hoodie-conf hoodie.datasource.hive_sync.jdbcurl=jdbc:hive2://localhost:10000
--source-ordering-field dms_seq_no
--props s3://bucket/cdc.properties
--hoodie-conf hoodie.datasource.hive_sync.database=glue_db
--target-base-path s3://bucket/table_1
--target-table table_1
--transformer-class org.apache.hudi.utilities.transform.SqlQueryBasedTransformer
--hoodie-conf hoodie.deltastreamer.source.dfs.root=s3://bucket/
--source-class org.apache.hudi.utilities.sources.ParquetDFSSource --enable-sync
This table  will be subsequently read with hudi options and joined with other hudi tables to populate the Final Enriched layer. While reading a Hudi table we are facing the ArrayIndexOutOfbound exception.
Below are the Hudi props and Spark Submits we execute to read and populate the downstream.
hoodie.datasource.write.partitionpath.field=
hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.NonpartitionedKeyGenerator
hoodie.datasource.hive_sync.enable=true
hoodie.datasource.hive_sync.assume_date_partitioning=false
hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.NonPartitionedExtractor
hoodie.parquet.small.file.limit=134217728
hoodie.parquet.max.file.size=1048576000
hoodie.cleaner.policy=KEEP_LATEST_COMMITS
hoodie.cleaner.commits.retained=1
hoodie.deltastreamer.transformer.sql=select CASE WHEN Op='D' THEN TRUE ELSE FALSE END AS _hoodie_is_deleted,* from
hoodie.datasource.hive_sync.support_timestamp=true
hoodie.datasource.compaction.async.enable=true
hoodie.index.type=BLOOM
hoodie.compact.inline=true
hoodiecompactionconfig.INLINE_COMPACT_NUM_DELTA_COMMITS_PROP=5
hoodie.metadata.compact.max.delta.commits=5
hoodie.clean.automatic=true
hoodie.clean.async=true
hoodie.datasource.hive_sync.table=table_1
hoodie.datasource.write.recordkey.field=table_1_ID
spark-submit --deploy-mode client --conf spark.yarn.appMasterEnv.SPARK_HOME=/prod/null --conf spark.executorEnv.SPARK_HOME=/prod/null --conf spark.shuffle.service.enabled=true --jars /usr/lib/hudi/hudi-spark-bundle.jar,/usr/lib/spark/external/lib/spark-avro.jar s3://pythonscripts/hudi_read.py
TaskSetManager: Lost task 32.2 in stage 6.0 (TID 253) on ip-172-31-16-236.ec2.internal, executor 1: java.lang.ArrayIndexOutOfBoundsException (null) [duplicate 1]
22/07/21 15:50:26 INFO TaskSetManager: Starting task 32.3 in stage 6.0 (TID 296, ip-172-31-16-236.ec2.internal, executor 1, partition 32, PROCESS_LOCAL, 8887 bytes)
22/07/21 15:50:26 INFO TaskSetManager: Lost task 33.2 in stage 6.0 (TID 256) on ip-172-31-16-236.ec2.internal, executor 1: java.lang.ArrayIndexOutOfBoundsException (null) [duplicate 2]

@soma1712 could you share how you read the Hudi table in s3://pythonscripts/hudi_read.py and the full stacktrace as well?  Which Hudi release do you use?

hudi_read.txt is actually a .py file. As the system was not supporting to update a .py, I had to change it to .txt
hudi_read.txt
results.txt

can you share contents of .hoodie if its feasible.

@soma1712 if you can please fill out the issue following the format, that would greatly help us in investigation (for ex, what version of Hudi are you using?)
Can you please paste full stack-trace of the exception?

@soma1712 are you sure you're running 0.11.1?
It seems like you're running an earlier version: looking at your stacktrace, i see method being used (createRowWithRequiredSchema) which was removed in 0.11.1:
22/07/30 04:09:14 WARN TaskSetManager: Lost task 2.0 in stage 2.0 (TID 7, ip-172-31-18-101.ec2.internal, executor 2): java.lang.ArrayIndexOutOfBoundsException: 9157
        at org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainLongDictionary.decodeToLong(PlainValuesDictionary.java:165)
        at org.apache.spark.sql.execution.datasources.parquet.ParquetDictionary.decodeToLong(ParquetDictionary.java:36)
        at org.apache.spark.sql.execution.vectorized.OnHeapColumnVector.getLong(OnHeapColumnVector.java:364)
        at org.apache.spark.sql.execution.vectorized.MutableColumnarRow.getLong(MutableColumnarRow.java:120)
        at org.apache.spark.sql.execution.vectorized.MutableColumnarRow.get(MutableColumnarRow.java:189)
        at org.apache.hudi.HoodieMergeOnReadRDD$$anon$3$$anonfun$createRowWithRequiredSchema$1.apply(HoodieMergeOnReadRDD.scala:278)
        at org.apache.hudi.HoodieMergeOnReadRDD$$anon$3$$anonfun$createRowWithRequiredSchema$1.apply(HoodieMergeOnReadRDD.scala:276)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
        at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at org.apache.spark.sql.types.StructType.foreach(StructType.scala:99)
        at org.apache.hudi.HoodieMergeOnReadRDD$$anon$3.createRowWithRequiredSchema(HoodieMergeOnReadRDD.scala:275)
        at org.apache.hudi.HoodieMergeOnReadRDD$$anon$3.hasNext(HoodieMergeOnReadRDD.scala:236)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:585)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:297)
        at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:289)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:123)
        at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

@soma1712 I have been running a deltastreamer MOR table job for over 30+ commits with sql transformer (though my setup is a bit different, I'm not using DMS). So far it's going well. No issues. We have had long-running deltastreamer tests in our test infra. Could you try with the latest Hudi?

authors: alexeykudinkin codope nsivabalan soma1712 yihua

repo: apache/hudi


text:
[SUPPORT] HUDI exclude columns from update

Is there any way in HUDI to exclude certain columns from update?

@ranjani1993 The update logic is controlled by the payload class, hoodie.datasource.write.payload.class.  Default is OverwriteWithLatestAvroPayload.  You can implement your own update logic to exclude certain columns.  You can also take a look at PartialUpdateAvroPayload in the latest master to see if that satisfies your requirement.

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
[SUPPORT] Archiving the commits failing for Hudi job

Tips before filing an issue

Have you gone through our FAQs? Yes

Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced
Hi Team, We have data pipeline that writes data into Hudi tables after reading from sources. After few days of successful runs, job started failing while archiving the commit timelines.
To Reproduce:
We couldn't produce the error, as it happened only once and then all the subsequent runs failed after that.
Expected behavior
A clear and concise description of what you expected to happen.
Environment Description : Apache Hudi on EMR cluster

Hudi version : 2.12-0.7.0-amzn-0.jar

Spark version : 3.1.1-amzn-0.1

Hive version :

Hadoop version :

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) : No

Additional context
Add any other context about the problem here.
Stacktrace
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
22/10/29 11:05:34 INFO HoodieLogFormat$WriterBuilder: Computed the next log version for commits in s3://bucket/dqip_data/.hoodie/archived as 35 with write-token 1-0-1
22/10/29 11:05:34 INFO HoodieLogFormat$WriterBuilder: HoodieLogFile on path s3://bucket/dqip_data/.hoodie/archived/.commits_.archive.35_1-0-1
22/10/29 11:05:34 INFO HoodieTimelineArchiveLog: Archiving instants [[==>20221028122716__replacecommit__REQUESTED], [==>20221028122716__replacecommit__INFLIGHT], [20221028122716__replacecommit__COMPLETED], [==>20221028141111__replacecommit__REQUESTED], [==>20221028141111__replacecommit__INFLIGHT], [20221028141111__replacecommit__COMPLETED], [==>20221028141409__replacecommit__REQUESTED], [==>20221028141409__replacecommit__INFLIGHT], [20221028141409__replacecommit__COMPLETED], [==>20221028142716__replacecommit__REQUESTED], [==>20221028142716__replacecommit__INFLIGHT], [20221028142716__replacecommit__COMPLETED], [==>20221028161416__replacecommit__REQUESTED], [==>20221028161416__replacecommit__INFLIGHT], [20221028161416__replacecommit__COMPLETED], [==>20221028162846__replacecommit__REQUESTED], [==>20221028162846__replacecommit__INFLIGHT], [20221028162846__replacecommit__COMPLETED], [==>20221028165845__replacecommit__REQUESTED], [==>20221028165845__replacecommit__INFLIGHT], [20221028165845__replacecommit__COMPLETED], [==>20221028181416__replacecommit__REQUESTED], [==>20221028181416__replacecommit__INFLIGHT], [20221028181416__replacecommit__COMPLETED], [==>20221028182709__replacecommit__REQUESTED], [==>20221028182709__replacecommit__INFLIGHT], [20221028182709__replacecommit__COMPLETED], [==>20221028201423__replacecommit__REQUESTED], [==>20221028201423__replacecommit__INFLIGHT], [20221028201423__replacecommit__COMPLETED], [==>20221028203806__replacecommit__REQUESTED], [==>20221028203806__replacecommit__INFLIGHT], [20221028203806__replacecommit__COMPLETED]]
22/10/29 11:05:34 INFO HoodieTimelineArchiveLog: Wrapper schema {"type":"record","name":"HoodieArchivedMetaEntry","namespace":"org.apache.hudi.avro.model","fields":[{"name":"hoodieCommitMetadata","type":["null",{"type":"record","name":"HoodieCommitMetadata","fields":[{"name":"partitionToWriteStats","type":["null",{"type":"map","values":{"type":"array","items":{"type":"record","name":"HoodieWriteStat","fields":[{"name":"fileId","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"path","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"prevCommit","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"numWrites","type":["null","long"],"default":null},{"name":"numDeletes","type":["null","long"],"default":null},{"name":"numUpdateWrites","type":["null","long"],"default":null},{"name":"totalWriteBytes","type":["null","long"],"default":null},{"name":"totalWriteErrors","type":["null","long"],"default":null},{"name":"partitionPath","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"totalLogRecords","type":["null","long"],"default":null},{"name":"totalLogFiles","type":["null","long"],"default":null},{"name":"totalUpdatedRecordsCompacted","type":["null","long"],"default":null},{"name":"numInserts","type":["null","long"],"default":null},{"name":"totalLogBlocks","type":["null","long"],"default":null},{"name":"totalCorruptLogBlock","type":["null","long"],"default":null},{"name":"totalRollbackBlocks","type":["null","long"],"default":null},{"name":"fileSizeInBytes","type":["null","long"],"default":null}]}},"avro.java.string":"String"}],"default":null},{"name":"extraMetadata","type":["null",{"type":"map","values":{"type":"string","avro.java.string":"String"},"avro.java.string":"String"}],"default":null},{"name":"version","type":["int","null"],"default":1},{"name":"operationType","type":["null",{"type":"string","avro.java.string":"String"}],"default":null}]}],"default":null},{"name":"hoodieCleanMetadata","type":["null",{"type":"record","name":"HoodieCleanMetadata","fields":[{"name":"startCleanTime","type":{"type":"string","avro.java.string":"String"}},{"name":"timeTakenInMillis","type":"long"},{"name":"totalFilesDeleted","type":"int"},{"name":"earliestCommitToRetain","type":{"type":"string","avro.java.string":"String"}},{"name":"partitionMetadata","type":{"type":"map","values":{"type":"record","name":"HoodieCleanPartitionMetadata","fields":[{"name":"partitionPath","type":{"type":"string","avro.java.string":"String"}},{"name":"policy","type":{"type":"string","avro.java.string":"String"}},{"name":"deletePathPatterns","type":{"type":"array","items":{"type":"string","avro.java.string":"String"}}},{"name":"successDeleteFiles","type":{"type":"array","items":{"type":"string","avro.java.string":"String"}}},{"name":"failedDeleteFiles","type":{"type":"array","items":{"type":"string","avro.java.string":"String"}}}]},"avro.java.string":"String"}},{"name":"version","type":["int","null"],"default":1},{"name":"bootstrapPartitionMetadata","type":["null",{"type":"map","values":"HoodieCleanPartitionMetadata","avro.java.string":"String","default":null}],"default":null}]}],"default":null},{"name":"hoodieCompactionMetadata","type":["null",{"type":"record","name":"HoodieCompactionMetadata","fields":[{"name":"partitionToCompactionWriteStats","type":["null",{"type":"map","values":{"type":"array","items":{"type":"record","name":"HoodieCompactionWriteStat","fields":[{"name":"partitionPath","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"totalLogRecords","type":["null","long"],"default":null},{"name":"totalLogFiles","type":["null","long"],"default":null},{"name":"totalUpdatedRecordsCompacted","type":["null","long"],"default":null},{"name":"hoodieWriteStat","type":["null","HoodieWriteStat"],"default":null}]}},"avro.java.string":"String"}]}]}],"default":null},{"name":"hoodieRollbackMetadata","type":["null",{"type":"record","name":"HoodieRollbackMetadata","fields":[{"name":"startRollbackTime","type":{"type":"string","avro.java.string":"String"}},{"name":"timeTakenInMillis","type":"long"},{"name":"totalFilesDeleted","type":"int"},{"name":"commitsRollback","type":{"type":"array","items":{"type":"string","avro.java.string":"String"}}},{"name":"partitionMetadata","type":{"type":"map","values":{"type":"record","name":"HoodieRollbackPartitionMetadata","fields":[{"name":"partitionPath","type":{"type":"string","avro.java.string":"String"}},{"name":"successDeleteFiles","type":{"type":"array","items":{"type":"string","avro.java.string":"String"}}},{"name":"failedDeleteFiles","type":{"type":"array","items":{"type":"string","avro.java.string":"String"}}},{"name":"rollbackLogFiles","type":["null",{"type":"map","values":"long","avro.java.string":"String"}],"default":null},{"name":"writtenLogFiles","type":["null",{"type":"map","values":"long","avro.java.string":"String"}],"default":null}]},"avro.java.string":"String"}},{"name":"version","type":["int","null"],"default":1},{"name":"instantsRollback","type":{"type":"array","items":{"type":"record","name":"HoodieInstantInfo","fields":[{"name":"commitTime","type":{"type":"string","avro.java.string":"String"}},{"name":"action","type":{"type":"string","avro.java.string":"String"}}]},"default":null},"default":null}]}],"default":null},{"name":"hoodieSavePointMetadata","type":["null",{"type":"record","name":"HoodieSavepointMetadata","fields":[{"name":"savepointedBy","type":{"type":"string","avro.java.string":"String"}},{"name":"savepointedAt","type":"long"},{"name":"comments","type":{"type":"string","avro.java.string":"String"}},{"name":"partitionMetadata","type":{"type":"map","values":{"type":"record","name":"HoodieSavepointPartitionMetadata","fields":[{"name":"partitionPath","type":{"type":"string","avro.java.string":"String"}},{"name":"savepointDataFile","type":{"type":"array","items":{"type":"string","avro.java.string":"String"}}}]},"avro.java.string":"String"}},{"name":"version","type":["int","null"],"default":1}]}],"default":null},{"name":"commitTime","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"actionType","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"version","type":["int","null"],"default":1},{"name":"hoodieCompactionPlan","type":["null",{"type":"record","name":"HoodieCompactionPlan","fields":[{"name":"operations","type":["null",{"type":"array","items":{"type":"record","name":"HoodieCompactionOperation","fields":[{"name":"baseInstantTime","type":["null",{"type":"string","avro.java.string":"String"}]},{"name":"deltaFilePaths","type":["null",{"type":"array","items":{"type":"string","avro.java.string":"String"}}],"default":null},{"name":"dataFilePath","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"fileId","type":["null",{"type":"string","avro.java.string":"String"}]},{"name":"partitionPath","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"metrics","type":["null",{"type":"map","values":"double","avro.java.string":"String"}],"default":null},{"name":"bootstrapFilePath","type":["null",{"type":"string","avro.java.string":"String"}],"default":null}]}}],"default":null},{"name":"extraMetadata","type":["null",{"type":"map","values":{"type":"string","avro.java.string":"String"},"avro.java.string":"String"}],"default":null},{"name":"version","type":["int","null"],"default":1}]}],"default":null},{"name":"hoodieCleanerPlan","type":["null",{"type":"record","name":"HoodieCleanerPlan","fields":[{"name":"earliestInstantToRetain","type":["null",{"type":"record","name":"HoodieActionInstant","fields":[{"name":"timestamp","type":{"type":"string","avro.java.string":"String"}},{"name":"action","type":{"type":"string","avro.java.string":"String"}},{"name":"state","type":{"type":"string","avro.java.string":"String"}}]}],"default":null},{"name":"policy","type":{"type":"string","avro.java.string":"String"}},{"name":"filesToBeDeletedPerPartition","type":["null",{"type":"map","values":{"type":"array","items":{"type":"string","avro.java.string":"String"}},"avro.java.string":"String"}],"default":null},{"name":"version","type":["int","null"],"default":1},{"name":"filePathsToBeDeletedPerPartition","type":["null",{"type":"map","values":{"type":"array","items":{"type":"record","name":"HoodieCleanFileInfo","fields":[{"name":"filePath","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"isBootstrapBaseFile","type":["null","boolean"],"default":null}]}},"avro.java.string":"String"}],"doc":"This field replaces the field filesToBeDeletedPerPartition","default":null}]}],"default":null},{"name":"actionState","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"hoodieReplaceCommitMetadata","type":["null",{"type":"record","name":"HoodieReplaceCommitMetadata","fields":[{"name":"partitionToWriteStats","type":["null",{"type":"map","values":{"type":"array","items":"HoodieWriteStat"},"avro.java.string":"String"}],"default":null},{"name":"extraMetadata","type":["null",{"type":"map","values":{"type":"string","avro.java.string":"String"},"avro.java.string":"String"}],"default":null},{"name":"version","type":["int","null"],"default":1},{"name":"operationType","type":["null",{"type":"string","avro.java.string":"String"}],"default":null},{"name":"partitionToReplaceFileIds","type":["null",{"type":"map","values":{"type":"array","items":{"type":"string","avro.java.string":"String"}},"avro.java.string":"String"}],"default":null}]}],"default":null}]}
22/10/29 11:05:34 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:35 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:35 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:35 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:35 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:35 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:35 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:35 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:35 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:35 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:35 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:36 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:36 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:36 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:36 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:36 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:36 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:36 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:36 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:36 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:36 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:36 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:36 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:36 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:37 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:37 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:37 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:37 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:37 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:37 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:37 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:37 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:37 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:37 INFO AbstractTableFileSystemView: Took 2568 ms to read  31 instants, 36565 replaced file groups
22/10/29 11:05:37 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:37 WARN ClusteringUtils: No content found in requested file for instant [==>20221029110511__replacecommit__INFLIGHT]
22/10/29 11:05:37 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/10/29 11:05:37 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:38 WARN DAGScheduler: Broadcasting large task binary with size 2.2 MiB
22/10/29 11:05:40 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:40 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:40 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:40 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:40 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:40 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:40 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:40 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:40 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:40 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:41 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:41 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:41 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:41 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:41 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:41 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:41 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:41 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:41 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:41 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:41 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:42 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:42 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:42 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:42 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:42 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:42 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:42 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:43 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:43 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:43 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:43 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:43 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:43 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:43 INFO AbstractTableFileSystemView: Took 3338 ms to read  31 instants, 36565 replaced file groups
22/10/29 11:05:43 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:43 WARN ClusteringUtils: No content found in requested file for instant [==>20221029110511__replacecommit__INFLIGHT]
22/10/29 11:05:43 INFO ClusteringUtils: Found 0 files in pending clustering operations
22/10/29 11:05:43 WARN HadoopFileSystemOwner: found no group information for livy (auth:SIMPLE), using livy as primary group
22/10/29 11:05:44 ERROR default: ****************An error occurred while calling o177.save.
: org.apache.hudi.exception.HoodieCommitException: Failed to archive commits
at org.apache.hudi.table.HoodieTimelineArchiveLog.archive(HoodieTimelineArchiveLog.java:322)
at org.apache.hudi.table.HoodieTimelineArchiveLog.archiveIfRequired(HoodieTimelineArchiveLog.java:138)
at org.apache.hudi.client.AbstractHoodieWriteClient.postCommit(AbstractHoodieWriteClient.java:426)
at org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:188)
at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:110)
at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:442)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:218)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:134)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:185)
at org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:223)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:220)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:181)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:134)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:133)
at org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:989)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:110)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:135)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:232)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:135)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:253)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:134)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:772)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:989)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:438)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:415)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:293)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.IllegalArgumentException: Positive number of partitions required
at org.apache.spark.rdd.ParallelCollectionRDD$.slice(ParallelCollectionRDD.scala:118)
at org.apache.spark.rdd.ParallelCollectionRDD.getPartitions(ParallelCollectionRDD.scala:96)
at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:300)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:296)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:300)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:296)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2303)
at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1030)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
at org.apache.spark.rdd.RDD.collect(RDD.scala:1029)
at org.apache.spark.api.java.JavaRDDLike.collect(JavaRDDLike.scala:362)
at org.apache.spark.api.java.JavaRDDLike.collect$(JavaRDDLike.scala:361)
at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
at org.apache.hudi.client.common.HoodieSparkEngineContext.map(HoodieSparkEngineContext.java:73)
at org.apache.hudi.client.ReplaceArchivalHelper.deleteReplacedFileGroups(ReplaceArchivalHelper.java:72)
at org.apache.hudi.table.HoodieTimelineArchiveLog.deleteReplacedFileGroups(HoodieTimelineArchiveLog.java:341)
at org.apache.hudi.table.HoodieTimelineArchiveLog.archive(HoodieTimelineArchiveLog.java:303)
... 45 more

22/10/29 11:05:44 ERROR ApplicationMaster: User application exited with status 1

Looks like this is a older version of hudi. with 0.12.0, I am not seeing the class named HoodieTimelineArchiveLog.
Or are you using an internal hudi version that you maintain internally in your company?
If not, can you point me which exact hudi version in OSS are you using.
we can investigate further

@mahi4uin : any updates here please


text:
[hudi-flink]fix assign function npe error

Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request

from https://github.com/apache/hudi/pull/2581#discussion_r581156455
fix npe problem when BucketAssignFunction restore from the checkpoint.

flink keystate could be access after keycontext'current key be setted，so i move checkPartitionsLoaded into processElement to avoid checkKeyNamespacePreconditions throw exception.

	private void checkKeyNamespacePreconditions(K key, N namespace) {
		Preconditions.checkNotNull(key, "No key set. This method should not be called outside of a keyed context.");
		Preconditions.checkNotNull(namespace, "Provided namespace is null.");
	}

Brief change log
(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Codecov Report

Merging #2603 (90a2d1b) into master (97864a4) will increase coverage by 18.26%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master    #2603       +/-   ##
=============================================
+ Coverage     51.19%   69.46%   +18.26%
+ Complexity     3226      356     -2870
=============================================
  Files           438       53      -385
  Lines         20089     1929    -18160
  Branches       2068      230     -1838
=============================================
- Hits          10285     1340     -8945
+ Misses         8958      456     -8502
+ Partials        846      133      -713

Flag
Coverage Δ
Complexity Δ

hudicli
?
?

hudiclient
100.00% <ø> (ø)
0.00 <ø> (ø)

hudicommon
?
?

hudiflink
?
?

hudihadoopmr
?
?

hudisparkdatasource
?
?

hudisync
?
?

huditimelineservice
?
?

hudiutilities
69.46% <ø> (ø)
0.00 <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ
Complexity Δ

...rg/apache/hudi/common/bloom/SimpleBloomFilter.java

...ache/hudi/cli/commands/ArchivedCommitsCommand.java

...rg/apache/hudi/exception/HoodieIndexException.java

.../apache/hudi/hadoop/RecordReaderValueIterator.java

.../hudi/common/bloom/InternalDynamicBloomFilter.java

.../org/apache/hudi/cli/commands/MetadataCommand.java

...rg/apache/hudi/common/table/HoodieTableConfig.java

.../main/java/org/apache/hudi/util/AvroConvertor.java

...op/realtime/HoodieCombineRealtimeRecordReader.java

...e/hudi/common/table/log/HoodieFileSliceReader.java

... and 374 more

source: gharchive/pull-request

authors: codecov-io hk-lrzy


text:
[HUDI-4436] Invalidate cached table in Spark after write

Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.

What is the purpose of the pull request
JIRA https://issues.apache.org/jira/browse/HUDI-4436
(For example: This pull request adds quick-start document.)
Brief change log
(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

cc @umehrot2 @zhedoubushishi

Azure ci timing out https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=10124&view=results

Sounds good thanks @codope i think @umehrot2 will also take a second look as well.

IT failure is due to Flink which is unrelated.

authors: rahil-c yihua


text:
[HUDI-6190] Append description in the HoodieTableFactory.checkRecordKey exception.

Change Logs
When set 'uuid' is recordkey field and the table schema doesn't contain this column. user would be noticed that “did not modify the recordkey .”
Now just append the description to let hudi beginner check whether the default recordkey in his schema.
Impact
Enrich a exception description.
Risk level (write none, low medium or high below)
No risk, no matter using uuid or not.
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

@danny0405 Hi, after last time we chat, I prefer to add a little tip when user set primary is uuid but there is no uuid field in the schema

CI is green.


text:
[AIRFLOW-1590] Fix unused module and variable

Dear Airflow maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[x] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-1590

Description

[x] Here are some details about my PR, including screenshots of any UI changes:
Fixed unused module import and variable.

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:
Does not need testing, small fix.

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Reopened instead of https://github.com/apache/incubator-airflow/pull/2591

Tests crushed without a reason

Master is failing, this needs to be fixed first.

There is no feedback in jobs logs. Could the problem be with those tests themselves? In previous pull request I had same output in 1 of containers, then I re-pushed my code, tests re-run and all of them passed.

Can you rebase against master?

Done

Codecov Report

Merging #2652 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master   #2652      +/-   ##
=========================================
+ Coverage   71.49%   71.5%   +<.01%
=========================================
  Files         153     153
  Lines       11746   11745       -1
=========================================
  Hits         8398    8398
+ Misses       3348    3347       -1

Impacted Files
Coverage Δ

airflow/jobs.py
77.7% <ø> (ø)
:arrow_up:

airflow/operators/sensors.py
67.7% <ø> (+0.23%)
:arrow_up:

airflow/utils/dates.py
85.71% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c3a65a...a2d48a2. Read the comment docs.

LGTM +1

authors: Fokko ProstoMaxim codecov-io

repo: apache/incubator-airflow

license_source: bigquery


text:
Add syntax to use variables from Jinja template

Make sure you have checked all steps below.
JIRA

[x] My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-XXX

Description

[x] Here are some details about my PR, including screenshots of any UI changes:

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

[ ] Passes git diff upstream/master -u -- "*.py" | flake8 --diff

Simple change in the documentation for the Variable usage from jinja template

Codecov Report

Merging #2847 into master will increase coverage by 0.08%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2847      +/-   ##
==========================================
+ Coverage   73.93%   74.01%   +0.08%
==========================================
  Files         160      160
  Lines       12131    12131
==========================================
+ Hits         8969     8979      +10
+ Misses       3162     3152      -10

Impacted Files
Coverage Δ

airflow/executors/dask_executor.py
79.06% <0%> (-2.33%)
:arrow_down:

airflow/jobs.py
79.89% <0%> (+0.44%)
:arrow_up:

airflow/utils/helpers.py
56.32% <0%> (+2.87%)
:arrow_up:

airflow/task_runner/bash_task_runner.py
100% <0%> (+6.66%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 06b41fb...8ebef4a. Read the comment docs.

Please still add a JIRA so that we can generate clean-looking release notes. Follow the contributor guidelines.

@r39132 Done!

+1 thx

FYI.. you should not make the JIRA resolved yourself -- it's automatically marked Resolved when committers/maintainers merge your fix.

authors: codecov-io moe-nadal-ck r39132


text:
[AIRFLOW-XXX] Fix Typo in SFTPOperator docstring

Make sure you have checked all steps below.
Jira

[x] My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"

https://issues.apache.org/jira/browse/AIRFLOW-XXX

Description

[x] Here are some details about my PR, including screenshots of any UI changes:
Fix Typo in SFTPOperator docstring

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:
N/a - Docstring change

Commits

[x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[x] In case of new functionality, my PR adds documentation that describes how to use it.

When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

[x] Passes git diff upstream/master -u -- "*.py" | flake8 --diff

Codecov Report

Merging #4016 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #4016   +/-   ##
======================================
  Coverage    75.5%   75.5%
======================================
  Files         199     199
  Lines       15949   15949
======================================
  Hits        12043   12043
  Misses       3906    3906

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 92b54bb...643c0bd. Read the comment docs.

authors: codecov-io kaxil


text:
The review count in the Admin is not correct

To Reproduce
Steps to reproduce the behavior:

A new question in review queue
Click Reject button
The question is automatically deleted
The review count still show "1" in Admin Dashboard

Expected behavior
There is no review tasks left. The review count should be "0".
Screenshots

Platform

Browser and version: Chrome
Version: v1.3.1

@LinkinStars  Can you assign this to me?

authors: fenbox sy-records

repo: apache/incubator-answer


text:
Main icons disappear

https://github.com/apache/incubator-apisix-dashboard/issues/124

repo: apache/incubator-apisix-dashboard


text:
[BEAM-794] Differ combining in case of merging windows with sideInputs.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[x] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

R: @kennknowles / @dhalperi / @bjchambers who were involved in this so far (and committers).

Jenkins fails on DirectRunner test. Not relevant to this PR.

repo: apache/incubator-beam


text:
FAILED in "Linking protoc-gen-mcpack“

Describe the bug (描述bug)
按照指导：https://github.com/apache/incubator-brpc/blob/master/docs/cn/getting_started.md#compile-brpc-with-cmake编译brpc提示错误
To Reproduce (复现方法)
1.拉取官方docker image  gcc-5.3.0
2.docker run
3.apt-get install -y git g++ make libssl-dev libgflags-dev libprotobuf-dev libprotoc-dev protobuf-compiler libleveldb-dev
4.sh config_brpc.sh --headers=/usr/include --libs=/usr/lib
5.make clean
6.make
Expected behavior (期望行为)
编译成功
Versions (各种版本)
OS:  Linux 81c1f755ca74 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 GNU/Linux
Compiler:
gcc (GCC) 5.3.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
g++ (GCC) 5.3.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
brpc:
master分支
commit 878e32b6a893c6e93f0a21fc74160c9b64779fce
Merge: b3a948c 885eaec
Author: Jiashun Zhu zhujiashun2010@gmail.com
Date:   Tue May 18 22:23:33 2021 +0200
Merge pull request #1345 from zyearn/fix_flamegraph_dep

Remove FlameGraph dependency to solve category b license conflict.

protobuf:
2.6.1
this is  output of apt-get install:
Setting up libssl-dev:amd64 (1.0.1t-1+deb8u12) ...
Setting up libgflags2 (2.0-2.1) ...
Setting up libsnappy1 (1.1.2-3) ...
Setting up libleveldb1:amd64 (1.17-1) ...
Setting up libgflags-dev (2.0-2.1) ...
Setting up libleveldb-dev:amd64 (1.17-1) ...
Setting up libprotobuf9:amd64 (2.6.1-1) ...
Setting up libprotobuf-lite9:amd64 (2.6.1-1) ...
Setting up libprotobuf-dev:amd64 (2.6.1-1) ...
Setting up libprotoc9:amd64 (2.6.1-1) ...
Setting up libprotoc-dev:amd64 (2.6.1-1) ...
Setting up protobuf-compiler (2.6.1-1) ...
Processing triggers for libc-bin (2.19-18+deb8u4) ...
/sbin/ldconfig.real: /usr/local/lib64/libstdc++.so.6.0.21-gdb.py is not an ELF file - it has the wrong magic bytes at the start.
Additional context/screenshots (更多上下文/截图)
错误提示信息：
Linking protoc-gen-mcpack
src/mcpack2pb/generator.o: In function google::protobuf::internal::StringTypeTraits::Get(int, google::protobuf::internal::ExtensionSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)': /usr/include/google/protobuf/extension_set.h:789: undefined reference to google::protobuf::internal::ExtensionSet::GetString(int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const'
src/mcpack2pb/generator.o: In function mcpack2pb::generate_declarations(std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, google::protobuf::io::Printer&)': /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:187: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:200: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' src/mcpack2pb/generator.o: In function generate_parsing':
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:276: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:489: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:494: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:571: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:579: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:552: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:590: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:499: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:535: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:517: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:511: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:505: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
src/mcpack2pb/generator.o:/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:523: more undefined references to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' follow src/mcpack2pb/generator.o: In function generate_parsing':
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:284: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:387: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:398: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:289: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:295: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:301: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
src/mcpack2pb/generator.o:/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:307: more undefined references to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' follow src/mcpack2pb/generator.o: In function generate_parsing':
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:474: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' src/mcpack2pb/generator.o: In function generate_serializing':
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:864: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:872: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:897: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1244: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1265: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' src/mcpack2pb/generator.o: In function mcpack2pb::McpackToProtobuf::Generate(google::protobuf::FileDescriptor const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, google::protobuf::compiler::GeneratorContext*, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) const':
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1404: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' src/mcpack2pb/generator.o: In function generate_serializing':
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1210: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1227: undefined reference to google::protobuf::io::Printer::Print(char const, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1255: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1218: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1207: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1214: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1227: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:911: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
src/mcpack2pb/generator.o:/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:911: more undefined references to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' follow src/mcpack2pb/generator.o: In function generate_serializing':
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:933: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:906: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:906: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:953: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:933: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:933: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
src/mcpack2pb/generator.o:/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:915: more undefined references to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' follow src/mcpack2pb/generator.o: In function generate_serializing':
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1022: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1029: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1032: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1037: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1151: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1193: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1174: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1154: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1159: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1164: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1080: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' src/mcpack2pb/generator.o:/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1099: more undefined references to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)' follow
src/mcpack2pb/generator.o: In function generate_serializing': /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:993: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1054: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:930: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:930: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:919: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:919: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' src/mcpack2pb/generator.o:/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:924: more undefined references to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)' follow
src/mcpack2pb/generator.o: In function generate_registration': /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1303: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1314: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1322: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1327: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/mcpack2pb/generator.cpp:1340: undefined reference to google::protobuf::io::Printer::Print(char const*, char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
libbrpc.a(logging.o): In function __static_initialization_and_destruction_0': /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/butil/logging.cc:1389: undefined reference to google::RegisterFlagValidator(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const*, bool ()(char const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&))'
libbrpc.a(variable.o): In function dumping_thread': /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/bvar/variable.cpp:705: undefined reference to google::GetCommandLineOption(char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator >)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/bvar/variable.cpp:709: undefined reference to google::GetCommandLineOption(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/bvar/variable.cpp:714: undefined reference to google::GetCommandLineOption(char const, std::__cxx11::basic_string<char, std::char_traits, std::allocator >)'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/bvar/variable.cpp:719: undefined reference to google::GetCommandLineOption(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/bvar/variable.cpp:723: undefined reference to google::GetCommandLineOption(char const, std::__cxx11::basic_string<char, std::char_traits, std::allocator >)'
libbrpc.a(variable.o): In function __static_initialization_and_destruction_0': /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/bvar/variable.cpp:825: undefined reference to google::RegisterFlagValidator(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, bool ()(char const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&))'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/bvar/variable.cpp:827: undefined reference to google::RegisterFlagValidator(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, bool (*)(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/bvar/variable.cpp:829: undefined reference to google::RegisterFlagValidator(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const*, bool ()(char const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&))'
/home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/bvar/variable.cpp:831: undefined reference to google::RegisterFlagValidator(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, bool (*)(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&))' /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/bvar/variable.cpp:833: undefined reference to google::RegisterFlagValidator(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const*, bool ()(char const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&))'
libbrpc.a(gflag.o): In function bvar::GFlag::get_value[abi:cxx11]() const': /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/bvar/gflag.cpp:81: undefined reference to google::GetCommandLineOption(char const*, std::__cxx11::basic_string<char, std::char_traits, std::allocator >)'
libbrpc.a(gflag.o): In function bvar::GFlag::set_value(char const*)': /home/build/brpc/0.9.7/incubator-brpc-0.9.7/src/bvar/gflag.cpp:88: undefined reference to google::SetCommandLineOption[abi:cxx11](char const, char const*)'
/tmp/ccOUHznP.o: In function protobuf_AssignDesc_idl_5foptions_2eproto()': idl_options.pb.cc:(.text+0x4a): undefined reference to google::protobuf::DescriptorPool::FindFileByName(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const'
/tmp/ccOUHznP.o: In function protobuf_AddDesc_idl_5foptions_2eproto()': idl_options.pb.cc:(.text+0x1b9): undefined reference to google::protobuf::MessageFactory::InternalRegisterGeneratedFile(char const*, void (*)(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&))'
collect2: error: ld returned 1 exit status
Makefile:240: recipe for target 'protoc-gen-mcpack' failed
make: *** [protoc-gen-mcpack] Error 1

https://gcc.gnu.org/legacy-ml/gcc-help/2014-08/msg00053.html
ldconfig 会默认你的/usr/local/lib64/ 下的所有文件都是elf文件。将/usr/local/lib64/libstdc++.so.6.0.21-gdb.py 移动到别的目录再重新编译下试试呢。

我更新了链接的错误提示信息，感觉是gcc5.3的问题，我升级gcc8.5同样步骤是可以编译成功的。

怀疑 gcc5.3 的镜像中 protobuf 编译时未启用 CXX11_ABI, 而编译 brpc 时默认启用了导致链接失败. 在 gcc8.5 中可能所有依赖库都使用了 CXX11_ABI 从而可以编译成功.
可以试试添加 -D_GLIBCXX_USE_CXX11_ABI=0 到 CXXFLAGS.

authors: TousakaRin jiangsongHW wasphin

repo: apache/incubator-brpc


text:
[Bug][Azure DevOps] subtask extractAzuredevopsGitpullrequests ended unexpectedly

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
At first pipeline run I received an error 'Task failed'
subtask extractAzuredevopsGitpullrequests ended unexpectedly Wraps: (2) remote error response: | ERROR: DataError (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush bluck if this flush is occuring prematurely) | (MySQLdb.DataError) (1406, "Data too long for column 'description' at row 1) | [SQL: INSERT INTO _tool_azuredevops_gitpullrequests (...)]
What do you expect to happen
The pipeline runs without an error.
How to reproduce
After configure the datasource, define the first blue print and run the pipeline.
Anything else
Every run.
Version
v0.17.0-beta6
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

This is already fixed by #5191 which has been cherry-picked to v0.17.0-beta6.
However this fix works only together with #5081 which is missing v0.17.0-beta6.

@CamilleTeruel the idea is to cherry-pick 5081 to beta6? Or will it enter the next beta version?

@CamilleTeruel the idea is to cherry-pick 5081 to beta6? Or will it enter the next beta version?

Next beta version

@andremattosferraz v0.17.0-beta8 has been released with the required cherry-pick. Let us know if it still doesn't work.

@KlemenDanfoss @andremattosferraz v0.17.0-beta8 has been released with the required cherry-pick. Let us know if it still doesn't work.

subtask extractAzuredevopsGitpullrequest ... 1048 "Column 'merge_commit_sha' cannot be null"
:)

#5298 still occurring :(

Fixed in #5191, the other reported error is tracked by #5298

authors: CamilleTeruel KlemenDanfoss andremattosferraz hezyin klesh

repo: apache/incubator-devlake


text:
dolphinscheduler server memory problems

在初始化脚本里服务启动内存设置xms=1g，是指Master Api Worker Logger Alert 每个服务1g，那就是最少5g，再考虑如果都整成分布式高可用，每台服务器占用就是5g，资源占用较大。因此xms 能够调多小，会有哪些影响。

1.并不是所有的server都装到一个机器上，正常都会分发到多个节点部署，每个节点装2-3个server
2.如果量级小，也可以设置128m，256m去尝试启动，生产环境应该都不会设置这么小，毕竟它是一个java进程。

authors: liangpingliu zhuangchong

repo: apache/incubator-dolphinscheduler


text:
bump calcite version to 1.18.0 in druid-sql

This PR bumps calcite version to 1.18.0 in druid-sql to pick up new features, improvements, and bug-fixes in calcite-1.18.0.
The updated test cases are mainly for compatible of sql planning changes in calcite-1.18.0. The special two cases which have different results are testNullStringEquality and testCountStarWithDegenerateFilter  due to different plan...
For testNullStringEquality
1.17.0
...
rel#27:LogicalFilter.NONE.[[]](input=rel#20:Subset#1.NONE.[],condition=CASE(=($0, 'a'), CAST(true):BOOLEAN, =($0, null)))
  call#64 rule [ReduceExpressionsRule(Filter)]
    rel#21:LogicalFilter.NONE.[](input=rel#20:Subset#1.NONE.[],condition==(CASE(=($0, 'a'), null, $0), null))
...

1.18.0
...
rel#27:LogicalFilter.NONE.[](input=rel#20:Subset#1.NONE.[],condition=IS TRUE(=($0, 'a')))
  call#64 rule [ReduceExpressionsRule(Filter)]
    rel#21:LogicalFilter.NONE.[](input=rel#20:Subset#1.NONE.[],condition==(CASE(=($0, 'a'), null, $0), null))
...

For testCountStarWithDegenerateFilter
1.17.0
...
rel#31:LogicalFilter.NONE.[[]](input=rel#21:Subset#1.NONE.[],condition=AND(=($1, 'a'), OR(>($0, 'a'), IS NOT NULL($0))))
  call#64 rule [ReduceExpressionsRule(Filter)]
    rel#22:LogicalFilter.NONE.[](input=rel#21:Subset#1.NONE.[],condition=AND(=($1, 'a'), OR(>($0, 'a'), <($0, 'b'))))
...
rel#39:DruidQueryRel.NONE.[[]](query={"queryType":"scan","dataSource":{"type":"table","name":"foo"},"intervals":{"type":"intervals","intervals":["-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"]},"virtualColumns":[],"resultFormat":"compactedList","batchSize":20480,"limit":9223372036854775807,"filter":{"type":"and","fields":[{"type":"selector","dimension":"dim2","value":"a","extractionFn":null},{"type":"or","fields":[{"type":"bound","dimension":"dim1","lower":"a","upper":null,"lowerStrict":true,"upperStrict":false,"extractionFn":null,"ordering":{"type":"lexicographic"}},{"type":"not","field":{"type":"selector","dimension":"dim1","value":null,"extractionFn":null}}]}]},"columns":["__time","cnt","dim1","dim2","dim3","m1","m2","unique_dim1"],"legacy":false,"context":{"defaultTimeout":300000,"maxScatterGatherBytes":9223372036854775807,"sqlCurrentTimestamp":"2000-01-01T00:00:00Z","sqlQueryId":"dummy"},"descending":false,"granularity":{"type":"all"}},signature={__time:LONG, cnt:LONG, dim1:STRING, dim2:STRING, dim3:STRING, m1:FLOAT, m2:DOUBLE, unique_dim1:COMPLEX})
  call#362 rule [DruidQueryRule(WHERE_FILTER)]
    rel#32:LogicalFilter.NONE.[[]](input=rel#19:Subset#0.NONE.[],condition=AND(=($3, 'a'), OR(>($2, 'a'), IS NOT NULL($2))))
...

1.18.0
...
rel#34:LogicalProject.NONE.[](input=rel#33:Subset#5.NONE.[],dim1=$2,dim2=$3)
  call#84 rule [FilterProjectTransposeRule]
    rel#22:LogicalFilter.NONE.[](input=rel#21:Subset#1.NONE.[],condition=AND(=($1, 'a'), OR(>($0, 'a'), <($0, 'b'))))
...
rel#42:DruidQueryRel.NONE.[](query={"queryType":"scan","dataSource":{"type":"table","name":"foo"},"intervals":{"type":"intervals","intervals":["-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"]},"virtualColumns":[],"resultFormat":"compactedList","batchSize":20480,"limit":9223372036854775807,"filter":{"type":"selector","dimension":"dim2","value":"a","extractionFn":null},"columns":["__time","cnt","dim1","dim2","dim3","m1","m2","unique_dim1"],"legacy":false,"context":{"defaultTimeout":300000,"maxScatterGatherBytes":9223372036854775807,"sqlCurrentTimestamp":"2000-01-01T00:00:00Z","sqlQueryId":"dummy"},"descending":false,"granularity":{"type":"all"}},signature={__time:LONG, cnt:LONG, dim1:STRING, dim2:STRING, dim3:STRING, m1:FLOAT, m2:DOUBLE, unique_dim1:COMPLEX})
  call#360 rule [DruidQueryRule(WHERE_FILTER)]
    rel#31:LogicalFilter.NONE.[](input=rel#19:Subset#0.NONE.[],condition=AND(=($3, 'a'), OR(>($2, 'a'), <($2, 'b'))))
...

@nishantmonu51 any chance you can review this one?

@xueyumusic, thanks for looking into this. Please check out the comment on the test "testNullStringEquality" - what do you think?

Hi, @gianm , thanks a lot for review.
The plan [CASE(=($0, 'a'), =(null, null), =($0, null))]'s part =($0, null) is simplified as unknownAs value which is false at here. Since the UNKNOWN policy of equal is ANY so if at least one of its arguments is null then expr value is null(unknown).
For =(null, null), DruidRexExecutor reduces it as True
One way might be that modify DruidRexExecutor to keep the same policy as calcite as following (adding strong.isNull check)
        if (sqlTypeName == SqlTypeName.BOOLEAN) {
          if (strong.isNull(constExp) && constExp.getKind() == SqlKind.EQUALS) {
            literal = rexBuilder.makeLiteral(false); //unknownAs is false
          } else {
            literal = rexBuilder.makeLiteral(exprResult.asBoolean(), constExp.getType(), true);
          }
        }

But this way will reduce =(null, null) and =(null, '')(useDefaultValueForNull) as false, so the test testEmptyStringEquality's result will be affected too and =(null, null) as false is a little counter-intuitive
What do you think, thanks!

It looks Calcite released 1.19.0 recently.. @gianm Do you think which way is make more sense, updating this pr to upgrade Calcite to 1.19.0 or first to 1.18.0 and then 1.19.0. Both version met the same issue discussed above..thanks

It looks Calcite released 1.19.0 recently.. @gianm Do you think which way is make more sense, updating this pr to upgrade Calcite to 1.19.0 or first to 1.18.0 and then 1.19.0. Both version met the same issue discussed above..thanks

IMO, updating this PR to go straight to 1.19.0 is best.

One way might be that modify DruidRexExecutor to keep the same policy as calcite as following (adding strong.isNull check)
...
But this way will reduce =(null, null) and =(null, '')(useDefaultValueForNull) as false, so the test testEmptyStringEquality's result will be affected too and =(null, null) as false is a little counter-intuitive
What do you think, thanks!

Hmm. I think in Druid's default mode (NullHandling.replaceWithDefault() true) then treating VARCHAR nulls as empty strings is generally the right thing to do. In SQL-compatible mode it's best to be SQL compatible of course. If there is some way we can accomplish that, it seems ideal.

If it’s not possible (it is fighting against the SQL grain a bit) then I’d also be ok with something that is at least in the spirit of treating nulls and empty strings somewhat equivalently, to the degree that Calcite lets us.

@gianm How about writing a new rule such as DruidReduceEqualNullRule which makes a=null as a='' when nessesary.. thanks

@gianm How about writing a new rule such as DruidReduceEqualNullRule which makes a=null as a='' when nessesary.. thanks

That seems reasonable if it gets us the behavior we want. I'm not sure if we will be able to get 100% there, though, because consider operators like COALESCE(x, y). In the default mode, we want this to return y if x is an empty string. But in SQL compliant mode, we want this to return x if x is an empty string (and return y if x is null). Right now, I think this works -- see "testCoalesceColumns" in CalciteQueryTest -- but it might be "by accident".

Hi, @gianm , I updated the PR to upgrade to calcite-1.19.0. Adding DruidRewriteEqualNullRule to process a=null as a='' and adding cast(BIGINT NOT NULL) as BIGINT process logic in DruidQuery to make query SUM(CASE WHEN dim1 <> '1' THEN 1 ELSE 0 END)  which will be rewrote by CaseFilteredAggregatorRule pass test

authors: drcrallen gianm xueyumusic

repo: apache/incubator-druid


text:
annotation @Active is inconsistent with it's document

source code：

/dubbo-common/src/main/java/com/alibaba/dubbo/common/extension/ExtensionLoader.java

private boolean isMatchGroup(String group, String[] groups) {
        if (group == null || group.length() == 0) {
            return true;
        }
        if (groups != null && groups.length > 0) {
            for (String g : groups) {
                if (group.equals(g)) {
                    return true;
                }
            }
        }
        return false;
    }

by the code, @Activate without property group can not be matched, also can not be activated.
but document say yes.

https://dubbo.incubator.apache.org/books/dubbo-dev-book-en/SPI.html#fn_1

import com.alibaba.dubbo.common.extension.Activate;
import com.alibaba.dubbo.rpc.Filter;

@Activate // Active for any condition
public class XxxFilter implements Filter {
    // ...
}
}

@darknesstm As the code shown, it will be activated when group is not set.
        if (group == null || group.length() == 0) {
            return true;
        }

Feel free to reopen it if it also confuse u.

I have just been confused about this issue, and I understand that "Active for any condition" is available to both providers and consumers, but the following configuration should be what I want:
@Activate(group = {Constants.CONSUMER, Constants.PROVIDER})

I have just been confused about this issue, and I understand that "Active for any condition" is available to both providers and consumers, but the following configuration should be what I want:
@Activate(group = {Constants.CONSUMER, Constants.PROVIDER})
Your reply is inaccurate, the condition of filtering is the second groups，So I think the document annotation should explain the problem more clearly.
if (groups != null && groups.length > 0) {             for (String g : groups) {                 if (group.equals(g)) {                     return true;                 }             }         }

@darknesstm

@diecui1202

authors: darknesstm diecui1202 linweizhen

repo: apache/incubator-dubbo