text:
[SUPPORT] Incoming batch schema is not compatible with the table's one

Tips before filing an issue

As is proposed before by another guy, there is a bug( #https://issues.apache.org/jira/browse/HUDI-4588 ) when upsert records less of  some columns which already exist in hudi table. So I made a test in the latest version 0.13.0.
Firstly, start up spark-shell like

spark-shell --packages org.apache.hudi:hudi-spark3-bundle_2.12:0.13.0 \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \
--conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog'

I wrote records had three Column:
import org.apache.hudi.QuickstartUtils._
import scala.collection.JavaConversions._
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._
import org.apache.hudi.common.model.HoodieRecord

val tableName = "dwd_hudi_test"
val basePath = "s3://xxx/test"
case class userinfo3(username:String,age:Int,addr:String)
val df = spark.sparkContext.parallelize(Seq(userinfo3("wanglei",25,"123"),userinfo3("zhangmeii",31,"sdf"))).toDF
df.write.format("hudi")
  .options(getQuickstartWriteConfigs)
  .option(PRECOMBINE_FIELD_OPT_KEY, "username")
  .option(RECORDKEY_FIELD_OPT_KEY, "username")
  .option(TABLE_NAME, tableName)
  .option("hoodie.datasource.hive_sync.database","dw")
  .option("hoodie.datasource.hive_sync.table",tableName)
  .option("hoodie.datasource.hive_sync.enable","true")
  .option("hoodie.schema.on.read.enable","true")
  .option("hoodie.datasource.write.reconcile.schema","false")
  .mode(Overwrite)
  .save(basePath)

then I appended records having 2 columns, missing one of columns.

case class userinfo2(username:String,age:Int)
val df = spark.sparkContext.parallelize(Seq(userinfo2("hanlei",22),userinfo2("luxin",31))).toDF
df.write.format("hudi")
  .options(getQuickstartWriteConfigs)
  .option(PRECOMBINE_FIELD_OPT_KEY, "username")
  .option(RECORDKEY_FIELD_OPT_KEY, "username")
  .option(TABLE_NAME, tableName)
  .option("hoodie.datasource.hive_sync.database","dw")
  .option("hoodie.datasource.hive_sync.table",tableName)
  .option("hoodie.datasource.hive_sync.enable","true")
  .option("hoodie.schema.on.read.enable","true")
  .option("hoodie.datasource.write.reconcile.schema","false")
  .mode(Append)
  .save(basePath)

Expected behavior
I thought this bug has been fixed is version 0.13.0, so I expeted the last two records could be wrote into hudi table successfully . But I got exception instead. So I don't known why this would happen. May anyone help me?  : )
(And this is my first issue in github)
Environment Description

Hudi version : 0.13

Spark version : 3.3.1

Hive version : 3.1.3

Hadoop version : 3.2.1

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) : no

Additional context
Add any other context about the problem here.
Stacktrace
ERROR HoodieSparkSqlWriter$: Incoming batch schema is not compatible with the table's one.
Incoming schema {
  "type" : "record",
  "name" : "dwd_hudi_test",
  "namespace" : "hoodie.dwd_hudi_test",
  "fields" : [ {
    "name" : "username",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "age",
    "type" : "int"
  } ]
}
Incoming schema (canonicalized) {
  "type" : "record",
  "name" : "dwd_hudi_test",
  "namespace" : "hoodie.dwd_hudi_test",
  "fields" : [ {
    "name" : "username",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "age",
    "type" : "int"
  } ]
}
Table's schema {
  "type" : "record",
  "name" : "dwd_hudi_test",
  "namespace" : "hoodie.dwd_hudi_test",
  "fields" : [ {
    "name" : "username",
    "type" : [ "null", "string" ],
    "default" : null
  }, {
    "name" : "age",
    "type" : "int"
  }, {
    "name" : "addr",
    "type" : [ "null", "string" ],
    "default" : null
  } ]
}

org.apache.hudi.exception.SchemaCompatibilityException: Incoming batch schema is not compatible with the table's one
  at org.apache.hudi.HoodieSparkSqlWriter$.deduceWriterSchema(HoodieSparkSqlWriter.scala:487)
  at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:305)
  at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:150)
  at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
  at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:103)
  at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
  at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
  at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:114)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$7(SQLExecution.scala:139)
  at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
  at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:139)
  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:245)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:138)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
  at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:100)
  at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:96)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:615)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:177)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:615)
  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:591)
  at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:96)
  at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:83)
  at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:81)
  at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:124)
  at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
  at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
  at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
  at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
  ... 59 elided
.```

Yes indeed. That could resolve this problem. Thanks a lot!


text:
[HUDI-1005] fix NPE in HoodieWriteClient.clean

Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request

Fix NPE in HoodieWriteClient.clean

Brief change log
(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Codecov Report

Merging #1714 into master will increase coverage by 0.05%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master    #1714      +/-   ##
============================================
+ Coverage     18.19%   18.24%   +0.05%
- Complexity      857      863       +6
============================================
  Files           348      348
  Lines         15358    15391      +33
  Branches       1525     1525
============================================
+ Hits           2794     2808      +14
- Misses        12206    12225      +19
  Partials        358      358

Impacted Files
Coverage Δ
Complexity Δ

...java/org/apache/hudi/client/HoodieWriteClient.java
20.37% <0.00%> (ø)
12.00 <0.00> (ø)

...i/common/table/view/HoodieTableFileSystemView.java
39.56% <0.00%> (+1.62%)
15.00% <0.00%> (+6.00%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fb28393...5239c9e. Read the comment docs.

@vinothchandar can you take a look at this?

I was wondering if there was a way to just throw an exception or make it an Option.. merged.. let's punt on this for now

I was wondering if there was a way to just throw an exception or make it an Option.. merged.. let's punt on this for now

When I try to run HoodieDeltaStreamer with metrics on, it's no clean action in hoodie meta, then  HoodieTable.create(config, hadoopConf).clean(jsc, cleanInstantTime) will return null.
hongdeMacBook-Pro hongshen$ ll .hoodie/
total 56
drwxr-xr-x  13 hongshen  wheel   416 Jun 10 10:11 ./
drwxr-xr-x   4 hongshen  wheel   128 Jun 10 10:11 ../
-rw-r--r--   1 hongshen  wheel    32 Jun 10 10:11 .20200610101112.commit.crc
-rw-r--r--   1 hongshen  wheel     8 Jun 10 10:11 .20200610101112.commit.requested.crc
-rw-r--r--   1 hongshen  wheel    12 Jun 10 10:11 .20200610101112.inflight.crc
drwxr-xr-x   2 hongshen  wheel    64 Jun 10 10:11 .aux/
-rw-r--r--   1 hongshen  wheel    12 Jun 10 10:11 .hoodie.properties.crc
drwxr-xr-x   2 hongshen  wheel    64 Jun 10 10:11 .temp/
-rw-r--r--   1 hongshen  wheel  2676 Jun 10 10:11 20200610101112.commit
-rw-r--r--   1 hongshen  wheel     0 Jun 10 10:11 20200610101112.commit.requested
-rw-r--r--   1 hongshen  wheel   380 Jun 10 10:11 20200610101112.inflight
drwxr-xr-x   2 hongshen  wheel    64 Jun 10 10:11 archived/
-rw-r--r--   1 hongshen  wheel   205 Jun 10 10:11 hoodie.properties

source: gharchive/pull-request

repo: apache/hudi

license_type: permissive

license_source: github-api


text:
[WIP][HUDI-4345] Enable partition pruning and projection for incremental query on COW table

What is the purpose of the pull request
This PR refactors IncrementalRelation to enable partition pruning and projection for incremental query on COW table.
Brief change log
Verify this pull request
This pull request is already covered by existing tests.
Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@alexeykudinkin : can you review this PR.

authors: nsivabalan yihua


text:
Support merge manifests on writes (MergeAppend)

Add MergeAppendFiles. This PR will enable the following configurations:

commit.manifest-merge.enabled: Controls whether to automatically merge manifests on writes.
commit.manifest.min-count-to-merge: Minimum number of manifests to accumulate before merging.
commit.manifest.target-size-bytes: Target size when merging manifest files.

Since commit.manifest-merge.enabled is default to True, we need to make MergeAppend as the default way to append data to align with the property definition and java implementation

Sorry for the long wait. I've fixed the sequence number inheritance issue. Previously some manifest entry incorrectly persist the -1 sequence number inherited from a newly constructed ManifestFile. I added a wrapper in ManifestWriter to ensure the sequence number None when unassigned.
I will add tests and update the doc soon

I'm seeing some odd behavior:
from pyiceberg.catalog.sql import SqlCatalog
from datetime import datetime, timezone, date
import uuid
import pyarrow as pa

pa_schema = pa.schema([
    ("bool", pa.bool_()),
    ("string", pa.large_string()),
    ("string_long", pa.large_string()),
    ("int", pa.int32()),
    ("long", pa.int64()),
    ("float", pa.float32()),
    ("double", pa.float64()),
    # Not supported by Spark
    # ("time", pa.time64('us')),
    ("timestamp", pa.timestamp(unit="us")),
    ("timestamptz", pa.timestamp(unit="us", tz="UTC")),
    ("date", pa.date32()),
    # Not supported by Spark
    # ("time", pa.time64("us")),
    # Not natively supported by Arrow
    # ("uuid", pa.fixed(16)),
    ("binary", pa.large_binary()),
    ("fixed", pa.binary(16)),
])

TEST_DATA_WITH_NULL = {
    "bool": [False, None, True],
    "string": ["a", None, "z"],
    # Go over the 16 bytes to kick in truncation
    "string_long": ["a" * 22, None, "z" * 22],
    "int": [1, None, 9],
    "long": [1, None, 9],
    "float": [0.0, None, 0.9],
    "double": [0.0, None, 0.9],
    # 'time': [1_000_000, None, 3_000_000],  # Example times: 1s, none, and 3s past midnight #Spark does not support time fields
    "timestamp": [datetime(2023, 1, 1, 19, 25, 00), None, datetime(2023, 3, 1, 19, 25, 00)],
    "timestamptz": [
        datetime(2023, 1, 1, 19, 25, 00, tzinfo=timezone.utc),
        None,
        datetime(2023, 3, 1, 19, 25, 00, tzinfo=timezone.utc),
    ],
    "date": [date(2023, 1, 1), None, date(2023, 3, 1)],
    # Not supported by Spark
    # 'time': [time(1, 22, 0), None, time(19, 25, 0)],
    # Not natively supported by Arrow
    # 'uuid': [uuid.UUID('00000000-0000-0000-0000-000000000000').bytes, None, uuid.UUID('11111111-1111-1111-1111-111111111111').bytes],
    "binary": [b"\01", None, b"\22"],
    "fixed": [
        uuid.UUID("00000000-0000-0000-0000-000000000000").bytes,
        None,
        uuid.UUID("11111111-1111-1111-1111-111111111111").bytes,
    ],
}

catalog = SqlCatalog("test_sql_catalog", uri="sqlite:///:memory:", warehouse=f"/tmp/")

pa_table = pa.Table.from_pydict(TEST_DATA_WITH_NULL, schema=pa_schema)

catalog.create_namespace(('some',))

tbl = catalog.create_table(identifier="some.table", schema=pa_schema, properties={
    "commit.manifest.min-count-to-merge": "2"
})

for num in range(5):
    print(f"Appended: {num}")
    tbl.merge_append(pa_table)

It tries to read a corrupt file (or a bug in our reader):
---------------------------------------------------------------------------
EOFError                                  Traceback (most recent call last)
Cell In[2], line 71
     69 for num in range(5):
     70     print(f"Appended: {num}")
---> 71     tbl.merge_append(pa_table)

File ~/work/iceberg-python/pyiceberg/table/__init__.py:1424, in Table.merge_append(self, df, snapshot_properties)
   1411 """
   1412 Shorthand API for appending a PyArrow table to a table transaction and merging manifests on write.
   1413
   (...)
   1421     snapshot_properties: Custom properties to be added to the snapshot summary
   1422 """
   1423 with self.transaction() as tx:
-> 1424     tx.merge_append(df=df, snapshot_properties=snapshot_properties)

File ~/work/iceberg-python/pyiceberg/table/__init__.py:472, in Transaction.merge_append(self, df, snapshot_properties)
    468 data_files = _dataframe_to_data_files(
    469     table_metadata=self._table.metadata, write_uuid=update_snapshot.commit_uuid, df=df, io=self._table.io
    470 )
    471 for data_file in data_files:
--> 472     update_snapshot.append_data_file(data_file)

File ~/work/iceberg-python/pyiceberg/table/__init__.py:1899, in UpdateTableMetadata.__exit__(self, _, value, traceback)
   1897 def __exit__(self, _: Any, value: Any, traceback: Any) -> None:
   1898     """Close and commit the change."""
-> 1899     self.commit()

File ~/work/iceberg-python/pyiceberg/table/__init__.py:1895, in UpdateTableMetadata.commit(self)
   1894 def commit(self) -> None:
-> 1895     self._transaction._apply(*self._commit())

File ~/work/iceberg-python/pyiceberg/table/__init__.py:2966, in _SnapshotProducer._commit(self)
   2965 def _commit(self) -> UpdatesAndRequirements:
-> 2966     new_manifests = self._manifests()
   2967     next_sequence_number = self._transaction.table_metadata.next_sequence_number()
   2969     summary = self._summary(self.snapshot_properties)

File ~/work/iceberg-python/pyiceberg/table/__init__.py:2935, in _SnapshotProducer._manifests(self)
   2932 delete_manifests = executor.submit(_write_delete_manifest)
   2933 existing_manifests = executor.submit(self._existing_manifests)
-> 2935 return self._process_manifests(added_manifests.result() + delete_manifests.result() + existing_manifests.result())

File ~/work/iceberg-python/pyiceberg/table/__init__.py:3111, in MergeAppendFiles._process_manifests(self, manifests)
   3102 unmerged_deletes_manifests = [manifest for manifest in manifests if manifest.content == ManifestContent.DELETES]
   3104 data_manifest_merge_manager = _ManifestMergeManager(
   3105     target_size_bytes=self._target_size_bytes,
   3106     min_count_to_merge=self._min_count_to_merge,
   3107     merge_enabled=self._merge_enabled,
   3108     snapshot_producer=self,
   3109 )
-> 3111 return data_manifest_merge_manager.merge_manifests(unmerged_data_manifests) + unmerged_deletes_manifests

File ~/work/iceberg-python/pyiceberg/table/__init__.py:3987, in _ManifestMergeManager.merge_manifests(self, manifests)
   3985 merged_manifests = []
   3986 for spec_id in reversed(groups.keys()):
-> 3987     merged_manifests.extend(self._merge_group(first_manifest, spec_id, groups[spec_id]))
   3989 return merged_manifests

File ~/work/iceberg-python/pyiceberg/table/__init__.py:3974, in _ManifestMergeManager._merge_group(self, first_manifest, spec_id, manifests)
   3963     return output_manifests
   3965 # executor = ExecutorFactory.get_or_create()
   3966 # futures = [executor.submit(merge_bin, b) for b in bins]
   3967
   (...)
   3971 # for future in concurrent.futures.as_completed(futures):
   3972 #     completed_futures.add(future)
-> 3974 bin_results: List[List[ManifestFile]] = [merge_bin(b) for b in bins]
   3976 return [manifest for bin_result in bin_results for manifest in bin_result]

File ~/work/iceberg-python/pyiceberg/table/__init__.py:3974, in <listcomp>(.0)
   3963     return output_manifests
   3965 # executor = ExecutorFactory.get_or_create()
   3966 # futures = [executor.submit(merge_bin, b) for b in bins]
   3967
   (...)
   3971 # for future in concurrent.futures.as_completed(futures):
   3972 #     completed_futures.add(future)
-> 3974 bin_results: List[List[ManifestFile]] = [merge_bin(b) for b in bins]
   3976 return [manifest for bin_result in bin_results for manifest in bin_result]

File ~/work/iceberg-python/pyiceberg/table/__init__.py:3961, in _ManifestMergeManager._merge_group.<locals>.merge_bin(manifest_bin)
   3959     output_manifests.extend(manifest_bin)
   3960 else:
-> 3961     output_manifests.append(self._create_manifest(spec_id, manifest_bin))
   3963 return output_manifests

File ~/work/iceberg-python/pyiceberg/table/__init__.py:3934, in _ManifestMergeManager._create_manifest(self, spec_id, manifest_bin)
   3932 with self._snapshot_producer.new_manifest_writer(spec=self._snapshot_producer.spec(spec_id)) as writer:
   3933     for manifest in manifest_bin:
-> 3934         for entry in self._snapshot_producer.fetch_manifest_entry(manifest=manifest, discard_deleted=False):
   3935             if entry.status == ManifestEntryStatus.DELETED and entry.snapshot_id == self._snapshot_producer.snapshot_id:
   3936                 #  only files deleted by this snapshot should be added to the new manifest
   3937                 writer.delete(entry)

File ~/work/iceberg-python/pyiceberg/table/__init__.py:3034, in _SnapshotProducer.fetch_manifest_entry(self, manifest, discard_deleted)
   3033 def fetch_manifest_entry(self, manifest: ManifestFile, discard_deleted: bool = True) -> List[ManifestEntry]:
-> 3034     return manifest.fetch_manifest_entry(io=self._io, discard_deleted=discard_deleted)

File ~/work/iceberg-python/pyiceberg/manifest.py:611, in ManifestFile.fetch_manifest_entry(self, io, discard_deleted)
    609 print(f"MANIFEST: {self.manifest_path}")
    610 input_file = io.new_input(self.manifest_path)
--> 611 with AvroFile[ManifestEntry](
    612     input_file,
    613     MANIFEST_ENTRY_SCHEMAS[DEFAULT_READ_VERSION],
    614     read_types={-1: ManifestEntry, 2: DataFile},
    615     read_enums={0: ManifestEntryStatus, 101: FileFormat, 134: DataFileContent},
    616 ) as reader:
    617     return [
    618         _inherit_from_manifest(entry, self)
    619         for entry in reader
    620         if not discard_deleted or entry.status != ManifestEntryStatus.DELETED
    621     ]

File ~/work/iceberg-python/pyiceberg/avro/file.py:172, in AvroFile.__enter__(self)
    170 with self.input_file.open() as f:
    171     self.decoder = new_decoder(f.read())
--> 172 self.header = self._read_header()
    173 self.schema = self.header.get_schema()
    174 if not self.read_schema:

File ~/work/iceberg-python/pyiceberg/avro/file.py:220, in AvroFile._read_header(self)
    219 def _read_header(self) -> AvroFileHeader:
--> 220     return construct_reader(META_SCHEMA, {-1: AvroFileHeader}).read(self.decoder)

File ~/work/iceberg-python/pyiceberg/avro/reader.py:333, in StructReader.read(self, decoder)
    331 for pos, field_reader in self._field_reader_functions:
    332     if pos is not None:
--> 333         struct[pos] = field_reader(decoder)  # later: pass reuse in here
    334     else:
    335         field_reader(decoder)

File ~/work/iceberg-python/pyiceberg/avro/reader.py:469, in MapReader.read(self, decoder)
    467         block_count = decoder.read_int()
    468 else:
--> 469     block_count = decoder.read_int()
    470     while block_count != 0:
    471         if block_count < 0:

File ~/work/iceberg-python/pyiceberg/avro/decoder_fast.pyx:85, in pyiceberg.avro.decoder_fast.CythonBinaryDecoder.read_int()

File ~/work/iceberg-python/pyiceberg/avro/decoder_fast.pyx:92, in pyiceberg.avro.decoder_fast.CythonBinaryDecoder.read_int()

EOFError: EOF: read 1 bytes

It tries to read this file, which turns out to be empty?
avro-tools tojson /tmp/some.db/table/metadata/94206240-2ae8-47e7-bffe-fd4a1b35d91d-m0.avro
24/06/30 21:44:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

avro-tools getmeta /tmp/some.db/table/metadata/94206240-2ae8-47e7-bffe-fd4a1b35d91d-m0.avro
24/06/30 21:45:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
schema	{"type":"struct","fields":[{"id":1,"name":"bool","type":"boolean","required":false},{"id":2,"name":"string","type":"string","required":false},{"id":3,"name":"string_long","type":"string","required":false},{"id":4,"name":"int","type":"int","required":false},{"id":5,"name":"long","type":"long","required":false},{"id":6,"name":"float","type":"float","required":false},{"id":7,"name":"double","type":"double","required":false},{"id":8,"name":"timestamp","type":"timestamp","required":false},{"id":9,"name":"timestamptz","type":"timestamptz","required":false},{"id":10,"name":"date","type":"date","required":false},{"id":11,"name":"binary","type":"binary","required":false},{"id":12,"name":"fixed","type":"fixed[16]","required":false}],"schema-id":0,"identifier-field-ids":[]}
partition-spec	{"spec-id":0,"fields":[]}
partition-spec-id	0
format-version	2
content	data
avro.schema	{"type": "record", "fields": [{"name": "status", "field-id": 0, "type": "int"}, {"name": "snapshot_id", "field-id": 1, "type": ["null", "long"], "default": null}, {"name": "data_sequence_number", "field-id": 3, "type": ["null", "long"], "default": null}, {"name": "file_sequence_number", "field-id": 4, "type": ["null", "long"], "default": null}, {"name": "data_file", "field-id": 2, "type": {"type": "record", "fields": [{"name": "content", "field-id": 134, "type": "int", "doc": "File format name: avro, orc, or parquet"}, {"name": "file_path", "field-id": 100, "type": "string", "doc": "Location URI with FS scheme"}, {"name": "file_format", "field-id": 101, "type": "string", "doc": "File format name: avro, orc, or parquet"}, {"name": "partition", "field-id": 102, "type": {"type": "record", "fields": [], "name": "r102"}, "doc": "Partition data tuple, schema based on the partition spec"}, {"name": "record_count", "field-id": 103, "type": "long", "doc": "Number of records in the file"}, {"name": "file_size_in_bytes", "field-id": 104, "type": "long", "doc": "Total file size in bytes"}, {"name": "column_sizes", "field-id": 108, "type": ["null", {"type": "array", "items": {"type": "record", "name": "k117_v118", "fields": [{"name": "key", "type": "int", "field-id": 117}, {"name": "value", "type": "long", "field-id": 118}]}, "logicalType": "map"}], "default": null, "doc": "Map of column id to total size on disk"}, {"name": "value_counts", "field-id": 109, "type": ["null", {"type": "array", "items": {"type": "record", "name": "k119_v120", "fields": [{"name": "key", "type": "int", "field-id": 119}, {"name": "value", "type": "long", "field-id": 120}]}, "logicalType": "map"}], "default": null, "doc": "Map of column id to total count, including null and NaN"}, {"name": "null_value_counts", "field-id": 110, "type": ["null", {"type": "array", "items": {"type": "record", "name": "k121_v122", "fields": [{"name": "key", "type": "int", "field-id": 121}, {"name": "value", "type": "long", "field-id": 122}]}, "logicalType": "map"}], "default": null, "doc": "Map of column id to null value count"}, {"name": "nan_value_counts", "field-id": 137, "type": ["null", {"type": "array", "items": {"type": "record", "name": "k138_v139", "fields": [{"name": "key", "type": "int", "field-id": 138}, {"name": "value", "type": "long", "field-id": 139}]}, "logicalType": "map"}], "default": null, "doc": "Map of column id to number of NaN values in the column"}, {"name": "lower_bounds", "field-id": 125, "type": ["null", {"type": "array", "items": {"type": "record", "name": "k126_v127", "fields": [{"name": "key", "type": "int", "field-id": 126}, {"name": "value", "type": "bytes", "field-id": 127}]}, "logicalType": "map"}], "default": null, "doc": "Map of column id to lower bound"}, {"name": "upper_bounds", "field-id": 128, "type": ["null", {"type": "array", "items": {"type": "record", "name": "k129_v130", "fields": [{"name": "key", "type": "int", "field-id": 129}, {"name": "value", "type": "bytes", "field-id": 130}]}, "logicalType": "map"}], "default": null, "doc": "Map of column id to upper bound"}, {"name": "key_metadata", "field-id": 131, "type": ["null", "bytes"], "default": null, "doc": "Encryption key metadata blob"}, {"name": "split_offsets", "field-id": 132, "type": ["null", {"type": "array", "element-id": 133, "items": "long"}], "default": null, "doc": "Splittable offsets"}, {"name": "equality_ids", "field-id": 135, "type": ["null", {"type": "array", "element-id": 136, "items": "long"}], "default": null, "doc": "Field ids used to determine row equality in equality delete files."}, {"name": "sort_order_id", "field-id": 140, "type": ["null", "int"], "default": null, "doc": "ID representing sort order for this file"}], "name": "r2"}}], "name": "manifest_entry"}
avro.codec	null

Doing some testing with avro-tools, asserting the state after 5 append operations with "commit.manifest.min-count-to-merge": "2"
V1 Table
Manifest-list
5th manifest-list
{
    "manifest_path": "/tmp/some.db/table/metadata/80ba9f84-99af-4af1-b8f5-4caa254645c2-m1.avro",
    "manifest_length": 6878,
    "partition_spec_id": 0,
    "content": 0,
    "sequence_number": 5,
    "min_sequence_number": 1,
    "added_snapshot_id": 6508090689697406000,
    "added_files_count": 1,
    "existing_files_count": 4,
    "deleted_files_count": 0,
    "added_rows_count": 3,
    "existing_rows_count": 12,
    "deleted_rows_count": 0,
    "partitions": {
        "array": []
    },
    "key_metadata": null
}

4th manifest-list
{
    "manifest_path": "/tmp/some.db/table/metadata/88807344-0e23-413c-827e-2a9ec63c6233-m1.avro",
    "manifest_length": 6436,
    "partition_spec_id": 0,
    "content": 0,
    "sequence_number": 4,
    "min_sequence_number": 1,
    "added_snapshot_id": 3455109142449701000,
    "added_files_count": 1,
    "existing_files_count": 3,
    "deleted_files_count": 0,
    "added_rows_count": 3,
    "existing_rows_count": 9,
    "deleted_rows_count": 0,
    "partitions": {
        "array": []
    },
    "key_metadata": null
}

Manifests
We have 5 manifests as expected:
avro-tools tojson /tmp/some.db/table/metadata/80ba9f84-99af-4af1-b8f5-4caa254645c2-m1.avro | wc -l
       5

Last one:
{
    "status": 1,
    "snapshot_id": {
        "long": 6508090689697406000
    },
    "data_sequence_number": null,
    "file_sequence_number": null,
    "data_file": {
        "content": 0,
        "file_path": "/tmp/some.db/table/data/00000-0-80ba9f84-99af-4af1-b8f5-4caa254645c2.parquet",
        "file_format": "PARQUET",
        "partition": {},
        "record_count": 3,
        "file_size_in_bytes": 5459,
        "column_sizes": { ... },
        "value_counts": { ... },
        "null_value_counts": { ... },
        "nan_value_counts": { ... },
        "lower_bounds": { ... },
        "upper_bounds": { ... },
        "key_metadata": null,
        "split_offsets": {
            "array": [
                4
            ]
        },
        "equality_ids": null,
        "sort_order_id": null
    }
}

First one:
{
    "status": 0,
    "snapshot_id": {
        "long": 6508090689697406000
    },
    "data_sequence_number": {
        "long": 1
    },
    "file_sequence_number": {
        "long": 1
    },
    "data_file": {
        "content": 0,
        "file_path": "/tmp/some.db/table/data/00000-0-bbd4029c-510a-48e6-a905-ab5b69a832e8.parquet",
        "file_format": "PARQUET",
        "partition": {},
        "record_count": 3,
        "file_size_in_bytes": 5459,
        "column_sizes": { ... },
        "value_counts": { ... },
        "null_value_counts": { ... },
        "nan_value_counts": { ... },
        "lower_bounds": { ... },
        "upper_bounds": { ... },
        "key_metadata": null,
        "split_offsets": {
            "array": [
                4
            ]
        },
        "equality_ids": null,
        "sort_order_id": null
    }
}

This looks good, except for one thing: the snapshot_id is off, as from the spec:

Snapshot id where the file was added, or deleted if status is 2. Inherited when null.

This should be the ID of the first append operation.
V2 Table
Manifest list
5th manifest-list
{
    "manifest_path": "/tmp/some.db/tablev2/metadata/93717a88-1cea-4e3d-a69a-00ce3d087822-m1.avro",
    "manifest_length": 6883,
    "partition_spec_id": 0,
    "content": 0,
    "sequence_number": 5,
    "min_sequence_number": 1,
    "added_snapshot_id": 898025966831056900,
    "added_files_count": 1,
    "existing_files_count": 4,
    "deleted_files_count": 0,
    "added_rows_count": 3,
    "existing_rows_count": 12,
    "deleted_rows_count": 0,
    "partitions": {
        "array": []
    },
    "key_metadata": null
}

4th manifest-list
{
    "manifest_path": "/tmp/some.db/tablev2/metadata/5c64a07c-4b8a-4be1-a751-d4fd339560e2-m0.avro",
    "manifest_length": 5127,
    "partition_spec_id": 0,
    "content": 0,
    "sequence_number": 1,
    "min_sequence_number": 1,
    "added_snapshot_id": 1343032504684197000,
    "added_files_count": 1,
    "existing_files_count": 0,
    "deleted_files_count": 0,
    "added_rows_count": 3,
    "existing_rows_count": 0,
    "deleted_rows_count": 0,
    "partitions": {
        "array": []
    },
    "key_metadata": null
}

Manifests
last manifest file in manifest-list
{
    "status": 1,
    "snapshot_id": {
        "long": 898025966831056900
    },
    "data_sequence_number": null,
    "file_sequence_number": null,
    "data_file": {
        "content": 0,
        "file_path": "/tmp/some.db/tablev2/data/00000-0-93717a88-1cea-4e3d-a69a-00ce3d087822.parquet",
        "file_format": "PARQUET",
        "partition": {},
        "record_count": 3,
        "file_size_in_bytes": 5459,
        "column_sizes": { ... },
        "value_counts": { ... },
        "null_value_counts": { ... },
        "nan_value_counts": { ... },
        "lower_bounds": { ... },
        "upper_bounds": { ... },
        "key_metadata": null,
        "split_offsets": {
            "array": [
                4
            ]
        },
        "equality_ids": null,
        "sort_order_id": null
    }
}

First manifest in manifest-list
{
    "status": 0,
    "snapshot_id": {
        "long": 898025966831056900
    },
    "data_sequence_number": {
        "long": 1
    },
    "file_sequence_number": {
        "long": 1
    },
    "data_file": {
        "content": 0,
        "file_path": "/tmp/some.db/tablev2/data/00000-0-5c64a07c-4b8a-4be1-a751-d4fd339560e2.parquet",
        "file_format": "PARQUET",
        "partition": {},
        "record_count": 3,
        "file_size_in_bytes": 5459,
        "column_sizes": { ... },
        "value_counts": { ... },
        "null_value_counts": { ... },
        "nan_value_counts": { ... },
        "lower_bounds": { ... },
        "upper_bounds": { ... },
        "key_metadata": null,
        "split_offsets": {
            "array": [
                4
            ]
        },
        "equality_ids": null,
        "sort_order_id": null
    }
}

Except for the snapshot-id and https://github.com/apache/iceberg-python/issues/893 this looks great! 🥳

Another test with commit.manifest.min-count-to-merge set to 100, and doing 500 append operations:
avro-tools tojson /tmp/some.db/woooo/metadata/snap-3952911087333379496-0-27b9a632-7ee0-4246-aaf2-fc6d8cb1dce5.avro
{"manifest_path":"/tmp/some.db/woooo/metadata/27b9a632-7ee0-4246-aaf2-fc6d8cb1dce5-m0.avro","manifest_length":5125,"partition_spec_id":0,"content":0,"sequence_number":500,"min_sequence_number":500,"added_snapshot_id":3952911087333379496,"added_files_count":1,"existing_files_count":0,"deleted_files_count":0,"added_rows_count":3,"existing_rows_count":0,"deleted_rows_count":0,"partitions":{"array":[]},"key_metadata":null}
{"manifest_path":"/tmp/some.db/woooo/metadata/dac5af38-f01b-4a59-9e4c-b14a26706e75-m0.avro","manifest_length":5126,"partition_spec_id":0,"content":0,"sequence_number":499,"min_sequence_number":499,"added_snapshot_id":8943105647176444976,"added_files_count":1,"existing_files_count":0,"deleted_files_count":0,"added_rows_count":3,"existing_rows_count":0,"deleted_rows_count":0,"partitions":{"array":[]},"key_metadata":null}
{"manifest_path":"/tmp/some.db/woooo/metadata/ed164af5-dda7-4e3e-9b67-fcb2fd78771b-m0.avro","manifest_length":5125,"partition_spec_id":0,"content":0,"sequence_number":498,"min_sequence_number":498,"added_snapshot_id":723002263384967579,"added_files_count":1,"existing_files_count":0,"deleted_files_count":0,"added_rows_count":3,"existing_rows_count":0,"deleted_rows_count":0,"partitions":{"array":[]},"key_metadata":null}
{"manifest_path":"/tmp/some.db/woooo/metadata/e2d3e14e-8caf-4ca0-9515-c0a19c2a5658-m0.avro","manifest_length":5126,"partition_spec_id":0,"content":0,"sequence_number":497,"min_sequence_number":497,"added_snapshot_id":6977509396340474362,"added_files_count":1,"existing_files_count":0,"deleted_files_count":0,"added_rows_count":3,"existing_rows_count":0,"deleted_rows_count":0,"partitions":{"array":[]},"key_metadata":null}
{"manifest_path":"/tmp/some.db/woooo/metadata/3cc77cfe-b68c-4071-9f70-41cc3933f0af-m1.avro","manifest_length":222800,"partition_spec_id":0,"content":0,"sequence_number":496,"min_sequence_number":1,"added_snapshot_id":7132518699806947299,"added_files_count":1,"existing_files_count":495,"deleted_files_count":0,"added_rows_count":3,"existing_rows_count":1485,"deleted_rows_count":0,"partitions":{"array":[]},"key_metadata":null}

I don't think it merges the manifests as it should:
➜  iceberg-python git:(manifest_compaction) avro-tools tojson /tmp/some.db/woooo/metadata/3cc77cfe-b68c-4071-9f70-41cc3933f0af-m1.avro | wc -l
24/07/04 21:04:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
     496
➜  iceberg-python git:(manifest_compaction) avro-tools tojson /tmp/some.db/woooo/metadata/27b9a632-7ee0-4246-aaf2-fc6d8cb1dce5-m0.avro | wc -l
24/07/04 21:04:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
       1

I would expect the manifest-entries to be distributed more evenly over the manifests to ensure maximum parallelization.

Another test with commit.manifest.min-count-to-merge set to 100, and doing 500 append operations:

I think the observed behavior is correct, although far from optimal. Each time we do one append, we add one manifest. At 100th append, when the number of manifest reach 100, the merge manager merge all of them to a new manifest file because they are all in the same "bin". This happens whenever the number of manifest reach 100, thus leaving us with a large manifest and 4 small ones.
I use spark to do the similar thing and get a similar result
@pytest.mark.integration
def test_spark_ref_behavior(spark: SparkSession, session_catalog: Catalog, arrow_table_with_null: pa.Table) -> None:
    identifier = "default.test_spark_ref_behavior"
    tbl = _create_table(session_catalog, identifier,
                        {"commit.manifest-merge.enabled": "true", "commit.manifest.min-count-to-merge": "10", "format-version": 2}, [])
    spark_df = spark.createDataFrame(arrow_table_with_null.to_pandas())

    for i in range(50):
        spark_df.writeTo(f"integration.{identifier}").append()
    tbl = session_catalog.load_table(identifier)
    tbl_a_manifests = tbl.current_snapshot().manifests(tbl.io)
    for manifest in tbl_a_manifests:
        print(
            f"Manifest: added: {manifest.added_files_count}, existing: {manifest.existing_files_count}, deleted: {manifest.deleted_files_count}")
=====
Manifest: added: 3, existing: 0, deleted: 0
Manifest: added: 3, existing: 0, deleted: 0
Manifest: added: 3, existing: 0, deleted: 0
Manifest: added: 3, existing: 0, deleted: 0
Manifest: added: 3, existing: 135, deleted: 0

To distribute manifest entries among manifests, I think we need to adjust the commit.manifest.target-size-bytes accordingly since this property controls the size of the bin.

To distribute manifest entries more evenly, I think we need to adjust the commit.manifest.target-size-bytes accordingly since this property controls the size of the bin.

Thanks, that makes actually a lot of sense 👍

Whoo 🥳 Thanks @HonahX for working on this, and thanks @syun64 for the review 🙌

authors: Fokko HonahX

repo: apache/iceberg-python


text:
uses actions/setup-python@v5 with python 3.8

Using multiple python version syntax from: https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md
Download and set up multiple Python versions:
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-python@v5
      with:
        python-version: |
            3.8
            3.9
            3.10
    - run: python my_script.py

Thanks @syun64 for the quick fix. Adding some context here:
We observed an error when running the release action for macos-14:
+ delocate-wheel --require-archs arm64 -w /private/var/folders/6g/_ypptv3n2fs80_zc0xz8p1d40000gn/T/cibw-run-4dh7emlt/cp38-macosx_arm64/repaired_wheel -v /private/var/folders/6g/_ypptv3n2fs80_zc0xz8p1d40000gn/T/cibw-run-4dh7emlt/cp38-macosx_arm64/built_wheel/pyiceberg-0.7.0-cp38-cp38-macosx_14_0_x86_64.whl
  Fixing: /private/var/folders/6g/_ypptv3n2fs80_zc0xz8p1d40000gn/T/cibw-run-4dh7emlt/cp38-macosx_arm64/built_wheel/pyiceberg-0.7.0-cp38-cp38-macosx_14_0_x86_64.whl
  Traceback (most recent call last):
    File "/private/var/folders/6g/_ypptv3n2fs80_zc0xz8p1d40000gn/T/cibw-run-4dh7emlt/cp38-macosx_arm64/build/venv/lib/python3.8/site-packages/delocate/delocating.py", line 789, in _calculate_minimum_wheel_name
      arch_version[arch] = max(version, version_dkt[arch])
  KeyError: 'x86_64'

This is similar to https://github.com/pypa/cibuildwheel/issues/1828. The suggested way is to explicitly specify python3.8 using actions/setup-python@v5


text:
IGNITE-22996 Move Ignite sources to JDK11

Thank you for submitting the pull request to the Apache Ignite.
In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:
The Contribution Checklist

[ ] There is a single JIRA ticket related to the pull request.
[ ] The web-link to the pull request is attached to the JIRA ticket.
[ ] The JIRA ticket has the Patch Available state.
[ ] The pull request body describes changes that have been made.
The description explains WHAT and WHY was made instead of HOW.
[ ] The pull request title is treated as the final commit message.
The following pattern must be used: IGNITE-XXXX Change summary where XXXX - number of JIRA issue.
[ ] A reviewer has been mentioned through the JIRA comments
(see the Maintainers list)
[ ] The pull request has been checked by the Teamcity Bot and
the green visa attached to the JIRA ticket (see TC.Bot: Check PR)

Notes

How to Contribute
Coding abbreviation rules
Coding Guidelines
Apache Ignite Teamcity Bot

If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com #ignite channel.

@ptupitsyn Hi, Pavel! Could you please have a look? Especially part related to .NET platform. Thanks!

repo: apache/ignite


text:
[BEAM-764] Remove cloneAs from PipelineOptions

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

R: @lukecwik @davorbonaci

This got merged but not closed as evidenced by: https://github.com/apache/incubator-beam/commit/71c69b31b6894064bf8111007f947150ff725528

Please close.

authors: lukecwik peihe

repo: apache/incubator-beam


text:
[BEAM-115] Make in-process GroupByKey consistent with Beam model

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[x] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

The commits in this PR stand individual but have strong dependencies. They each build towards making the InProcessPipelineRunner correspond to the intended runner API / Beam model.

Actually, failed to revert a couple tests.


text:
[BEAM-604] Use Watermark to Finish Streaming Job in TestDataflowRunner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

Add checkMaxWatermark() function in TestDataflowRunner, so that when testing on streaming pipeline with bounded input, the job can be canceled as soon as all watermark reach to max value (by default is -2). Then, verification steps can be executed.
Add WindowedWordCountIT as a basic example of testing on streaming job.
Add non-terminated check before canceling steaming job.

TODO:

Create verifier for WindowedWordCountIT. (BigQuery verifier)

+R: @peihe

A couple of smaller things and some comments for what to watch out for in the future but overall this looks good. Thanks Mark!

Thanks!
PTAL: @jasonkuster @peihe

+R: @lukecwik

PTAL: @lukecwik

PTAL: @lukecwik

PTAL: @peihe

PTAL @lukecwik

LGTM, will merge after travis/jenkins

authors: jasonkuster lukecwik markflyhigh


text:
用mold链接brpc报duplicate symbol

Describe the bug (描述bug)
用ld/lld/gold链接都是可以的，但是mold链接就会报这个错，不知道是mold的问题还是brpc的问题？
To Reproduce (复现方法)
mold: error: duplicate symbol: /home/bitetheddddt/dbdev/incubator-doris/thirdparty/installed/lib64/libbrpc.a(collector.cpp.o): src/service/CMakeFiles/palo_be.dir/doris_main.cpp.o: guard variable for butil::detail::ClassNameHelper<long>::name[abi:cxx11]
mold: error: duplicate symbol: /home/bitetheddddt/dbdev/incubator-doris/thirdparty/installed/lib64/libbrpc.a(acceptor.cpp.o): src/exec/libExec.a(data_sink.cpp.o): guard variable for butil::detail::ClassNameHelper<int>::name[abi:cxx11]
mold: error: duplicate symbol: /home/bitetheddddt/dbdev/incubator-doris/thirdparty/installed/lib64/libbrpc.a(execution_queue.cpp.o): src/exec/libExec.a(data_sink.cpp.o): guard variable for butil::detail::ClassNameHelper<bvar::detail::AddTo<long> >::name[abi:cxx11]
mold: error: duplicate symbol: /home/bitetheddddt/dbdev/incubator-doris/thirdparty/installed/lib64/libbrpc.a(collector.cpp.o): src/service/CMakeFiles/palo_be.dir/doris_main.cpp.o: guard variable for butil::detail::ClassNameHelper<bvar::detail::MaxTo<long> >::name[abi:cxx11]
mold: error: duplicate symbol: /home/bitetheddddt/dbdev/incubator-doris/thirdparty/installed/lib64/libbrpc.a(acceptor.cpp.o): src/exec/libExec.a(data_sink.cpp.o): guard variable for butil::detail::ClassNameHelper<bvar::detail::AddTo<int> >::name[abi:cxx11]
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Expected behavior (期望行为)
Versions (各种版本)
OS:
Compiler:
brpc:
protobuf:
Additional context/screenshots (更多上下文/截图)

看起来是mold的行为和其它ld不兼容，参考：https://github.com/rui314/mold/issues/524
有一个解决方案是gcc编译源码时加上-fno-gnu-unique选项

从clang14升级到clang15，又遇到了类似的问题
ld.lld: error: duplicate symbol: guard variable for butil::detail::ClassNameHelper<long>::name[abi:cxx11]
>>> defined at doris_main.cpp
>>>            src/service/CMakeFiles/doris_be.dir/doris_main.cpp.o:(guard variable for butil::detail::ClassNameHelper<long>::name[abi:cxx11])
>>> defined at key.cpp.o:(.bss._ZGVN5butil6detail15ClassNameHelperIlE4nameB5cxx11E+0x0) in archive /mnt/disk1/pxl/doris/thirdparty/installed//lib64/libbrpc.a

ld.lld: error: duplicate symbol: guard variable for butil::detail::ClassNameHelper<bvar::detail::MaxTo<long>>::name[abi:cxx11]
>>> defined at doris_main.cpp
>>>            src/service/CMakeFiles/doris_be.dir/doris_main.cpp.o:(guard variable for butil::detail::ClassNameHelper<bvar::detail::MaxTo<long>>::name[abi:cxx11])
>>> defined at key.cpp.o:(.bss._ZGVN5butil6detail15ClassNameHelperIN4bvar6detail5MaxToIlEEE4nameB5cxx11E+0x0) in archive /mnt/disk1/pxl/doris/thirdparty/installed//lib64/libbrpc.a

ld.lld: error: duplicate symbol: guard variable for butil::detail::ClassNameHelper<bvar::detail::AddTo<long>>::name[abi:cxx11]
>>> defined at rpc_fn_call.cpp
>>>            rpc_fn_call.cpp.o:(guard variable for butil::detail::ClassNameHelper<bvar::detail::AddTo<long>>::name[abi:cxx11]) in archive src/exprs/libExprs.a
>>> defined at task_control.cpp.o:(.bss._ZGVN5butil6detail15ClassNameHelperIN4bvar6detail5AddToIlEEE4nameB5cxx11E+0x0) in archive /mnt/disk1/pxl/doris/thirdparty/installed//lib64/libbrpc.a

ld.lld: error: duplicate symbol: guard variable for butil::detail::ClassNameHelper<int>::name[abi:cxx11]
>>> defined at rpc_fn_call.cpp
>>>            rpc_fn_call.cpp.o:(guard variable for butil::detail::ClassNameHelper<int>::name[abi:cxx11]) in archive src/exprs/libExprs.a
>>> defined at index_service.cpp.o:(.bss._ZGVN5butil6detail15ClassNameHelperIiE4nameB5cxx11E+0x0) in archive /mnt/disk1/pxl/doris/thirdparty/installed//lib64/libbrpc.a

ld.lld: error: duplicate symbol: guard variable for butil::detail::ClassNameHelper<bvar::detail::AddTo<int>>::name[abi:cxx11]
>>> defined at rpc_fn_call.cpp
>>>            rpc_fn_call.cpp.o:(guard variable for butil::detail::ClassNameHelper<bvar::detail::AddTo<int>>::name[abi:cxx11]) in archive src/exprs/libExprs.a
>>> defined at index_service.cpp.o:(.bss._ZGVN5butil6detail15ClassNameHelperIN4bvar6detail5AddToIiEEE4nameB5cxx11E+0x0) in archive /mnt/disk1/pxl/doris/thirdparty/installed//lib64/libbrpc.a
clang-15: error: linker command failed with exit code 1

研究了一下发现需要加inline即可，我会提交一个pr

Fixed by https://github.com/apache/incubator-brpc/pull/1936

source: gharchive/issue

authors: BiteTheDDDDt wwbmmm

repo: apache/incubator-brpc


text:
bthread_cond_timedwait 偶尔出现超时不返回

Describe the bug (描述bug)
当 bthread_cond_timedwait 设置的超时时间较小时（比如1ms） 偶尔会出现超时后 bthread_cond_timedwait  不返回， 最终可能导致 bthread 一直处于等待状态。
To Reproduce (复现方法)
bthread_cond_timedwait  设置较小的超时时间（注意：出现概率较低）
Expected behavior (期望行为)
调用 bthread_cond_timedwait  时， 在超时后正确退出， 无论超时时间多小。
Versions (各种版本)
brpc <= 0.9.6， 此bug 与butex 自身实现有关， 与其他库的版本无关

见 #617


text:
Replaces arrayIndex with occursIndex

Now the debugger uses "info occursIndex" instead of "info arrayIndex"
DAFFODIL 1478

+1

authors: olabusayoT stevedlawrence

repo: apache/incubator-daffodil


text:
dolphinscheduler 部署 k8s 的问题

请问，dolphinscheduler 可以部署在k8s 上吗?
如果可以部署，worker 节点调度hadoop , flink ，hive 等平台的时候，需要在 worker 镜像里面一起制作这些平台的客户端在里面吗？

Of course, hadoop, flink, hive, and etc. can integrate into worker, if you need them, you must install packages into docker image.
Please use English.

repo: apache/incubator-dolphinscheduler


text:
[Improvement][Readme] update DolphinScheduler slack invitation url

the slack invitation url has been expired, please update the url address in README.md and README_zh_CN.md
Join the slack channel by [this invitation link](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-mzqu52gi-rCggPkSHQ0DZYkwbTxO1Gw ).

please use this new slack invitation url: https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw
this invitation url never expires

I will fix it.

authors: KonsonChow dailidong


text:
Add ExcelUtilsTest.java , the unit test for ExcelUtils

What is the purpose of the pull request
For #1465 .
Add ExcelUtilsTest.java , the unit test for ExcelUtils.
Brief change log

Add ExcelUtilsTest.java
Edit ExcelUtils.java
Add /alert/utils/ExcelUtilsTest.java to root pom.xml

Verify this pull request
This change added tests and can be verified as follows:

Added ExcelUtilsTest to verify the change.

Codecov Report

Merging #1500 into dev will increase coverage by 0.32%.
The diff coverage is 100%.

@@           Coverage Diff            @@
##             dev   #1500      +/-   ##
========================================
+ Coverage   2.77%    3.1%   +0.32%
========================================
  Files        272     272
  Lines      13602   13598       -4
  Branches    2160    2160
========================================
+ Hits         378     422      +44
+ Misses     13205   13153      -52
- Partials      19      23       +4

Impacted Files
Coverage Δ

...pache/dolphinscheduler/alert/utils/ExcelUtils.java
78% <100%> (+78%)
:arrow_up:

...apache/dolphinscheduler/alert/utils/JSONUtils.java
50% <0%> (+50%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d8d74a...17b2004. Read the comment docs.

authors: codecov-io nauu


text:
[Fix][api] Add queryProjectCreatedAndAuthorizedByUser function

What is the purpose of the pull request
This pull request adds queryProjectCreatedAndAuthorizedByUser fuction
When performing batch copy or move, you need to select the target projects. The purpose of this pr  is as follows:

When the user is admin, all projects can be the target project
When the user is an ordinary user, the projects he created and authorized can be  the target project

please solve conflicting files locally, thx

Codecov Report

Merging #3658 into dev will increase coverage by 0.02%.
The diff coverage is 84.61%.

@@             Coverage Diff              @@
##                dev    #3658      +/-   ##
============================================
+ Coverage     39.85%   39.87%   +0.02%
- Complexity     2923     2924       +1
============================================
  Files           460      460
  Lines         21822    21831       +9
  Branches       2651     2652       +1
============================================
+ Hits           8698     8706       +8
- Misses        12300    12301       +1
  Partials        824      824

Impacted Files
Coverage Δ
Complexity Δ

...hinscheduler/api/controller/ProjectController.java
5.55% <0.00%> (ø)
2.00 <0.00> (ø)

.../org/apache/dolphinscheduler/api/enums/Status.java
100.00% <100.00%> (ø)
5.00 <0.00> (ø)

...scheduler/api/service/impl/ProjectServiceImpl.java
87.86% <100.00%> (+0.58%)
42.00 <2.00> (+2.00)

...eduler/server/worker/runner/TaskExecuteThread.java
55.20% <0.00%> (-0.80%)
12.00% <0.00%> (-1.00%)

...inscheduler/service/zk/CuratorZookeeperClient.java
60.97% <0.00%> (ø)
7.00% <0.00%> (ø%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3fdc557...aa2d218. Read the comment docs.

one PR need one corresponding issue, so please create an issue for PR next time

authors: codecov-io dailidong zixi0825


text:
dubbo LruCache 错误

public LruCache(URL url) {
    final int max = url.getParameter("cache.size", 1000);
    this.store = new LinkedHashMap<Object, Object>() {
        private static final long serialVersionUID = -3834209229668463829L;
        @Override
        protected boolean removeEldestEntry(Entry<Object, Object> eldest) {
            return size() > max;
        }
    };
}

new LinkedHashMap()  应该使用下面的construct:
          public LinkedHashMap(int initialCapacity,
                                                float loadFactor,
                                                boolean accessOrder)
     accessOrder must is true

@denglt 你说的是写法，能测试发现到不预期的情况的说明吗？
或是给出你这个做法优势的说明吗？

public LinkedHashMap(int initialCapacity,
float loadFactor,
boolean accessOrder)
accessOrder - the ordering mode - true for access-order, false for insertion-order
LruCache  must be  access-order.

public class LRUCache<K, V> extends LinkedHashMap<K, V> {
public LRUCache(int maxCapacity) {
super(16, DEFAULT_LOAD_FACTOR, true);
this.maxCapacity = maxCapacity;
}
}
This issue has been fixed.
&NEED-TO-CLOSE&

authors: carryxyh denglt oldratlee

repo: apache/incubator-dubbo


text:
about custom Exception which extends RuntimeException

interface project.

interface class.

provider implements class.

consumer class

In consumer class, I caught the DemoException, so I can consider it running correctly.
But why it logged error  in the provider progress?
Should the log line logger.error(xxx) move bellow the code bellow in ExceptionFilter.java ?
// directly throw if it's dubbo exception
  if (exception instanceof RpcException) {
      return result;
  }

RuntimeException is not predicted, maybe it is caused by a programming error, so dubbo does not know if the exception is throw by developer explicitly.

And does a error log cause some problems ?


text:
Fix test building problem caused by kafka version and remove unused files

Fix test building problem caused by kafka version and remove unused files

LGTM

repo: apache/incubator-eagle

license_source: bigquery


text:
EAGLE-745 add site delete err message

add site delete err message

LGTM


text:
Y-Axis Min & Max are not calculated automatically

Version
4.2.1
Reproduction link
https://stackblitz.com/edit/angular-ngx-echarts-v6hdqm
Steps to reproduce
Try the provided example.
What is expected?
I expect that the min and max values of each y axis to be calculated automatically and y-axis label are displayed as it is mentioned in the document.
What is actually happening?
It does only show values for the first y-axis. It only shows values when I set min and max explicitly which I don't want.

I didn't see any data in the chart. Did the option be set correctly?

@Ovilia Actually there's data. There's only one data point though and that should be sufficient. I've updated the link with two more data points.

You didn't set yAxisIndex in your series, which makes both of the series to be at the first y axis (on the left side). So there is no data in the right y axis, and thus showing no min max value.

I figured that couple of days ago but forgot to update here. Thanks @Ovilia

authors: HMubaireek Ovilia

repo: apache/incubator-echarts


text:
打包一个支持 ES6 import 的 build

What problem does this feature solve?
可以在较新的浏览器里使用import ECharts from './echarts.min'
What does the proposed API look like?
可以在较新的浏览器里使用import ECharts from './echarts.min'

浏览器里直接使用
<script type="module">
import ECharts from './echarts.min'
</script>

引入？

嗯。
但是我没找到什么方式怎么判断浏览器是否支持了export/import……

This is not an issue directly related to ECharts. Please search for solutions elsewise.

如果这是一个暂时无法解决的问题，是一个共性问题，那如你所说。但是如果这个很简单，你们能搞定，怎么就和你们没关系了呢，就像你们发布的默认是commonjs版本的一样，发布一个echarts.module.js就可以了。现在浏览器进步了，大家用各种构建工具做出来的东西，反而跟不上浏览器的进步？

Please include a "echarts.module.js" or "echarts.mjs" in your dist folder, which can be imported as an ES6 module as stated above.
Highcharts is doing it since v6.1.0 April 2018!!!
https://github.com/highcharts/highcharts#load-highcharts-from-the-cdn-as-ecmascript-modules

Please @pissang or @Ovilia  reopen!

Would be nice if this could fit into new Echart 5.0 stable!

Added in https://github.com/apache/incubator-echarts/commit/cb5cfdc0682402380824ba9756bf116b3b945d14

authors: Ovilia maicWorkGithub pissang zevero


text:
map,visualMap存在时,如何单独给某个省份设置颜色?

One-line summary [问题简述]
给data单独设置的颜色会被visualMap颜色所覆盖,希望当value为0时,能单独设置颜色#eee,而不使用visualMap的颜色lightskyblue,代码如下:
visualMap颜色设置
visualMap: {// 颜色的设置  dataRange
left: 'left',
bottom: '10%',
min: 0,
max: 500,
calculable: true,// 颜色呈条状
text: ['高', '低'],// 文本，默认为数值文本
inRange: {
color: ['lightskyblue', 'yellow', 'orangered']
}
},
给data数据单独设置颜色
data: [
{
name: '北京', value: 0,
itemStyle: {
areaColor: '#eee'
}
},
{
name: '天津', value: 0,
itemStyle: {
areaColor: '#eee'
}
}
]

暂时不行


text:
[Enhancement] Method calls equals on an enum instance [EventMeshTcp2Client]

Search before asking

[X] I had searched in the issues and found no similar issues.

Enhancement Request

located at:
eventmesh-runtime/src/main/java/org/apache/eventmesh/runtime/core/protocol/tcp/client/EventMeshTcp2Client.java line 156
analysis and explanation:
This method calls the equals(Object) method on an enum instance. Since enums values are singletons, you can use == to safely compare two enum values. In fact, the implementation for Enum.equals does just that.
Describe the solution you'd like
Change to use "==".
Are you willing to submit PR?

[x] Yes I am willing to submit a PR!

I want to fix it.

authors: Alonexc jonyangx

repo: apache/incubator-eventmesh


text:
GUACAMOLE-229: fixed clearing intervals when polling

https://issues.apache.org/jira/browse/GUACAMOLE-229

Thanks, @teschnei.
The changes here look fine code-wise, but can you clarify the commit message a bit? Just saying "fixed [thing which was apparently broken]" is too vague. Reading the message, it's not clear what high level issue is being fixed.
If you could also format the message such that it doesn't stand out next to the rest of the history (ie: starts with a capital letter, ends with a period), that would be nice. Looking at the history for this file, for example, you see:
GUACAMOLE-229: fixed clearing intervals when polling
GUACAMOLE-44: Expose tunnel UUID to JavaScript. Document allowed internal use of the empty opcode.
GUACAMOLE-1: Relicense C and JavaScript files.
GUAC-791: Fix type expressions to comply with JSDoc 3.
GUAC-1430: Do not move on to the next chained tunnel on upstream timeout.
...

The style of the GUACAMOLE-229 message is completely different from the rest

Certainly! I will amend it when I am next able to.
On Mar 2, 2017 11:56 AM, "Mike Jumper" notifications@github.com wrote:

Thanks, @teschnei https://github.com/teschnei.
The changes here look fine code-wise, but can you clarify the commit
message a bit? Just saying "fixed [thing which was apparently broken]" is
too vague. Reading the message, it's not clear what high level issue is
being fixed.
If you could also format the message such that it doesn't stand out next
to the rest of the history (ie: starts with a capital letter, ends with a
period), that would be nice. Looking at the history for this file, for
example, you see:
GUACAMOLE-229: fixed clearing intervals when polling
GUACAMOLE-44: Expose tunnel UUID to JavaScript. Document allowed internal use of the empty opcode.
GUACAMOLE-1: Relicense C and JavaScript files.
GUAC-791: Fix type expressions to comply with JSDoc 3.
GUAC-1430: Do not move on to the next chained tunnel on upstream timeout.
...
The style of the GUACAMOLE-229 message is completely different from the
rest
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-guacamole-client/pull/126#issuecomment-283745032,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABGI_4FyAT-41uMeOaD7QHwr6CfpVrjrks5rhxDhgaJpZM4MRWF-
.

Amended

authors: mike-jumper teschnei

repo: apache/incubator-guacamole-client


text:
HAWQ-1369. Update for RAT status and external doc link in README.md

Updating embedded external CI status images to include the HAWQ-rat project. Additionally, update documentation link to point at open source doc set.
The visible changes to the README.md can be seen here: https://github.com/edespino/incubator-hawq/tree/HAWQ-1369
reviewers: @paul-guo- @radarwave

+1

+1

authors: edespino huor radarwave

repo: apache/incubator-hawq


text:
[HIVEMALL-76][SPARK] Fix worng ranks in top-K funcs

What changes were proposed in this pull request?
This pr fixed the Spark each_top_k/top_k_join behaviour along with Hive ones.
What type of PR is it?
Bug Fix
What is the Jira issue?
https://issues.apache.org/jira/browse/HIVEMALL-76
How was this patch tested?
Added tests in HivemallOpsSuite.

travis, go for it!!!

@maropu build failing due to scala style.
https://travis-ci.org/apache/incubator-hivemall/jobs/206229890

Changes Unknown when pulling 89355895a81024dc0a3fdbb5cc0322d5930d25bc on maropu:HIVEMALL-76 into ** on apache:master**.

@myui passed.

authors: coveralls maropu myui

repo: apache/incubator-hivemall


text:
Fix the size in GenericDataFile

Fixes the size after removing file_ordinal and sort_columns in #914

Good catch, @garawalid. Thanks for submitting this!

authors: garawalid rdblue

repo: apache/incubator-iceberg


text:
Fix duplicately join the task runner

This may close #1448
Currently, server->Join will join the task runner before the cron thread, so it may have the data race since the cron thread may access the task runner as well. We can avoid this by removing the task runner join which will do in the task runner destructor.

@torwig Thanks for your quick review.

Could you specify the location of another Join call of the task runner?

Could you specify the location of another Join call of the task runner?

Sure, the TaskRunner destructor will join as well: https://github.com/apache/incubator-kvrocks/blob/unstable/src/common/task_runner.h#L49

LGTM. Although a dtor with a long time waiting may cause some misunderstanding.

Yes, I think we can remove the Cancel and Join in dtor since the Server::Stop will cancel the task runner first. Can take a look again. @PragmaTwice @torwig

@git-hulk Yes, Server::Stop will cancel and Server::Join will join all threads.
Additionally, here https://github.com/apache/incubator-kvrocks/blob/unstable/src/common/task_runner.cc#L48 should we stop joining threads when the first call to util::ThreadJoin or just log an error and continue joining other threads?

@git-hulk Yes, Server::Stop will cancel and Server::Join will join all threads. Additionally, here https://github.com/apache/incubator-kvrocks/blob/unstable/src/common/task_runner.cc#L48 should we stop joining threads when the first call to util::ThreadJoin or just log an error and continue joining other threads?

oooh, my bad. I didn't see that.

@git-hulk My idea was to change the loop to the following:
for (auto &thread : threads_) {
  if (auto s = util::ThreadJoin(thread); !s) {
    LOG(WARNING) << "Failed to join thread: " << s.Msg();
    continue;
  }
}

What do you think about that? @PragmaTwice What is your opinion on that?

LOG(WARNING) << "Failed to join thread: " << s.Msg();

Yes, it's good to only log an error when failing to join since other places like the worker thread only prints an error.

@git-hulk My idea was to change the loop to the following:
for (auto &thread : threads_) {
  if (auto s = util::ThreadJoin(thread); !s) {
    LOG(WARNING) << "Failed to join thread: " << s.Msg();
    continue;
  }
}

What do you think about that? @PragmaTwice What is your opinion on that?

Yes, it's good to only log an error when failing to join since other places like the worker thread only prints an error.

@git-hulk My idea was to change the loop to the following:
for (auto &thread : threads_) {
  if (auto s = util::ThreadJoin(thread); !s) {
    LOG(WARNING) << "Failed to join thread: " << s.Msg();
    continue;
  }
}

What do you think about that? @PragmaTwice What is your opinion on that?

Seems good to me.

Thanks all, merging...

authors: PragmaTwice git-hulk torwig

repo: apache/incubator-kvrocks


text:
Set CMAKE_EXPORT_COMPILE_COMMANDS to ON as a default behavior

Both clangd, ccls, clang-tidy and other C++ language tools (like cppcheck) need compile_commands.json (a database for compile commands) to retrieve compile information like include dirs, standard, compile options, and more.
CMake provide an option CMAKE_EXPORT_COMPILE_COMMANDS to generate the file.
So setting it to ON as a default behavior can make these tools more easier to use.

Could you describe a concrete use case that could benefits from this change? It seems we don't even configure CMAKE_EXPORT_COMPILE_COMMANDS for current cppcheck.

Yeah, cppcheck allow user to not provide a compile_commands.json, but it will greatly affect the detection accuracy, and user need to provide all include dirs, standard, compile options manually to fix this.
In static analysis and language server of C++, compile_commands.json is just a standard.
Many C++ project like apache/doris, oneflow, clickhouse did something like this PR.


text:
fix: ip was used as port number.

In the previous test, 127.0.0.1 has been resolved to port number 127 for testing.

@git-hulk  @PragmaTwice  Hi. Do you have time to review?

@git-hulk @PragmaTwice Hi. Do you have time to review?

Yes, will take a look tonight.

Yes, will take a look tonight.

Thank you very much.

authors: git-hulk tanruixiang


text:
Fix compatibility for spark authz with spark v3.3

Why are the changes needed?

Some of the commands were migrated to v2 in spark v3.3, this PR is used to match them
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

cc @cfmcgrady @zhaomin1423 @packyan

Codecov Report

Merging #2675 (defd582) into master (b40bcbd) will decrease coverage by 0.27%.
The diff coverage is 9.63%.

@@             Coverage Diff              @@
##             master    #2675      +/-   ##
============================================
- Coverage     64.24%   63.96%   -0.28%
  Complexity       82       82
============================================
  Files           385      385
  Lines         18673    18737      +64
  Branches       2531     2549      +18
============================================
- Hits          11997    11986      -11
- Misses         5532     5584      +52
- Partials       1144     1167      +23

Impacted Files
Coverage Δ

...park/authz/ranger/FilterDataSourceV2Strategy.scala
57.14% <0.00%> (-22.86%)
:arrow_down:

...n/spark/authz/ranger/FilteredShowObjectsExec.scala
42.10% <0.00%> (-37.90%)
:arrow_down:

...k/authz/ranger/RuleReplaceShowObjectCommands.scala
60.00% <0.00%> (-2.07%)
:arrow_down:

...he/kyuubi/plugin/spark/authz/util/AuthZUtils.scala
46.80% <0.00%> (-4.36%)
:arrow_down:

.../kyuubi/plugin/spark/authz/PrivilegesBuilder.scala
68.84% <8.00%> (-11.09%)
:arrow_down:

...ache/kyuubi/plugin/spark/authz/OperationType.scala
57.64% <25.00%> (-12.36%)
:arrow_down:

.../org/apache/kyuubi/operation/KyuubiOperation.scala
68.00% <0.00%> (-1.34%)
:arrow_down:

...in/scala/org/apache/kyuubi/config/KyuubiConf.scala
96.62% <0.00%> (-0.12%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b40bcbd...defd582. Read the comment docs.

Since the spark-3.3 support is in progress, I have run a GA job on my forked repo. see https://github.com/cfmcgrady/incubator-kyuubi/pull/16

Since the spark-3.3 support is in progress, I have run a GA job on my forked repo. see cfmcgrady#16

CI passed!

thanks @cfmcgrady for checking

Thanks, merging to master

authors: cfmcgrady codecov-commenter pan3793 yaooqinn

repo: apache/incubator-kyuubi


text:
[KYUUBI #3441] Change default Spark version to 3.3.0

Why are the changes needed?

Fix https://github.com/apache/incubator-kyuubi/issues/3441, this pr aims to change default Spark version to 3.3.0
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

Codecov Report

Merging #3471 (6824092) into master (92942e9) will decrease coverage by 0.12%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #3471      +/-   ##
============================================
- Coverage     51.44%   51.31%   -0.13%
  Complexity       13       13
============================================
  Files           481      481
  Lines         26734    26734
  Branches       3750     3750
============================================
- Hits          13752    13719      -33
- Misses        11638    11667      +29
- Partials       1344     1348       +4

Impacted Files
Coverage Δ

...pache/kyuubi/operation/log/LogDivertAppender.scala
33.33% <0.00%> (-11.12%)
:arrow_down:

...ver/mysql/authentication/MySQLNativePassword.scala
73.91% <0.00%> (-4.35%)
:arrow_down:

...ache/kyuubi/engine/spark/events/SessionEvent.scala
81.48% <0.00%> (-3.71%)
:arrow_down:

...pache/kyuubi/engine/YarnApplicationOperation.scala
62.96% <0.00%> (-3.71%)
:arrow_down:

...apache/kyuubi/engine/JpsApplicationOperation.scala
77.41% <0.00%> (-3.23%)
:arrow_down:

...he/kyuubi/ctl/cmd/create/CreateServerCommand.scala
86.48% <0.00%> (-2.71%)
:arrow_down:

...he/kyuubi/plugin/spark/authz/util/AuthZUtils.scala
44.18% <0.00%> (-2.33%)
:arrow_down:

...ache/kyuubi/engine/trino/TrinoProcessBuilder.scala
91.66% <0.00%> (-2.09%)
:arrow_down:

...apache/kyuubi/engine/jdbc/JdbcProcessBuilder.scala
87.75% <0.00%> (-2.05%)
:arrow_down:

...rg/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala
78.00% <0.00%> (-2.00%)
:arrow_down:

... and 29 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

There are a bunch of things that need to update as well, please refer
https://github.com/apache/incubator-kyuubi/blob/e87baa75055e2c1d3bfdb3fcf81c4fd16f71d83c/pom.xml#L2176-L2191

Please update GitHub Action as well.
Currently, we have 2 cross version verification workflows, verify-on-spark-3.1-binary, verify-on-spark-3.3-binary, it should be updated to build w/ Spark 3.3(default), and verify on Spark 3.1 and 3.2.

Seems that several TPC-DS queries failed w/ Spark 3.3.0 ...

Please update build/release/create-package.sh as well, and I guess we can remove the -Pspark-3.2

Spark3.3.0 has issues with k8s pod name check which caused the failure. We can wait for Spark3.3.1, see https://github.com/apache/spark/pull/36999

Hi @Yikf, have you looked that why upgrading causes some TPC-DS queries failure, is there some regression on Spark 3.3 comparing to 3.2

@Yikf I think you are talking about the issue of https://github.com/apache/spark/pull/36953, 3.3.1 will fine

Spark 3.3.1 is on the way, let's keep it open and test against 3.3.1 during the RC phase

@Yikf  are you sure you are running using master ? The code line from error stack is not correct.
For example: SubqueryBroadcastExec line 81, but the code is:
https://github.com/apache/spark/blob/3d14b745773f66d50c5ee5b3d7835b5f11132ec8/sql/core/src/main/scala/org/apache/spark/sql/execution/SubqueryBroadcastExec.scala#L81-L82

@Yikf  thank you for your investigation!

Spark 3.3.1 RC1 is out, ping @Yikf
https://www.mail-archive.com/dev@spark.apache.org/msg29407.html

@Yikf I don't get your point, the voting for 3.3.1 rc1 is started, which means it is already available.

Both staging maven repo and tgz download links are available during RC phase, and it's designed for validation. If RC1 voting failed, then we will test RC2.

The RC artifacts are stored in the staging repo, please read the VOTE mail. They will be moved to the release repo if voting success.

@Yikf you can refer to the Hudi PR https://github.com/apache/hudi/pull/6707 and spark.archive.name should be updated as well for Kyuubi

authors: Yikf codecov-commenter pan3793 ulysses-you


text:
[Possibly A Feature Request] Better strides support?

Hi!
I just took a cursory look at the code and it seems that strides as in https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.strides.html is not very well supported in MXNet, at least in the high level. mshadow seems to have used the idea of strides when transposing and slicing tensors, but still provides no more flexible way of accessing it. The higher level does not care about strides at all, it seems, with things like transposing directly delegated to mshadow, and due to the fact that mshadow does not support dynamic numbers of dimensions (ndim is specified as a C++ template parameter), transpose in MXNet supports at most 6 dimensions (it could have supported more, of course, but never arbitrarily many in this fashion).
As far as I know, with better strides support it would be more convenient, efficient and flexible to access a contiguous block of data. There could be much more useful views for one single block of data. Take the diag operator as an example. If we want to take the diagonal w.r.t. a specified set of axes of a given tensor, we can simply create a new view with the strides of these axes merged together. There would be no need for creating new memory space and no need for explicitly forwarding or backwarding.
As I am new to MXNet, especially to its huge and complex implementation, there might be a lot of things I have misunderstood. Please comment and help. Thanks!

Thank you for the feature request @jasonyu1996
@mxnet-label-bot [Feature request]

repo: apache/incubator-mxnet


text:
test_inter_methods_in_augmenter timeout

http://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/mxnet-validation/pipelines/unix-cpu/branches/PR-18319/runs/36/nodes/350/steps/489/log/?start=0
[2020-07-15T19:49:55.751Z] ______________________ test_inter_methods_in_augmenter[9] ______________________
[2020-07-15T19:49:55.751Z] [gw2] linux -- Python 3.6.9 /usr/bin/python3
[2020-07-15T19:49:55.751Z]
[2020-07-15T19:49:55.751Z] inter_method = 9
[2020-07-15T19:49:55.751Z] tmpdir = local('/tmp/pytest-of-jenkins_slave/pytest-0/popen-gw2/test_inter_methods_in_augmente1')
[2020-07-15T19:49:55.751Z]
[2020-07-15T19:49:55.751Z]     @pytest.mark.parametrize('inter_method', [0,1,2,3,4,9,10])
[2020-07-15T19:49:55.751Z]     def test_inter_methods_in_augmenter(inter_method, tmpdir):
[2020-07-15T19:49:55.751Z]         path = str(tmpdir)
[2020-07-15T19:49:55.751Z] >       get_cifar10(path)
[2020-07-15T19:49:55.751Z]
[2020-07-15T19:49:55.751Z] tests/python/unittest/test_io.py:97:
[2020-07-15T19:49:55.751Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2020-07-15T19:49:55.751Z] python/mxnet/test_utils.py:1728: in get_cifar10
[2020-07-15T19:49:55.751Z]     verify_ssl=False)
[2020-07-15T19:49:55.751Z] python/mxnet/gluon/utils.py:330: in download
[2020-07-15T19:49:55.751Z]     for chunk in r.iter_content(chunk_size=1024):
[2020-07-15T19:49:55.751Z] /usr/local/lib/python3.6/dist-packages/requests/models.py:751: in generate
[2020-07-15T19:49:55.751Z]     for chunk in self.raw.stream(chunk_size, decode_content=True):
[2020-07-15T19:49:55.751Z] /usr/local/lib/python3.6/dist-packages/urllib3/response.py:576: in stream
[2020-07-15T19:49:55.751Z]     data = self.read(amt=amt, decode_content=decode_content)
[2020-07-15T19:49:55.751Z] /usr/local/lib/python3.6/dist-packages/urllib3/response.py:519: in read
[2020-07-15T19:49:55.751Z]     data = self._fp.read(amt) if not fp_closed else b""
[2020-07-15T19:49:55.751Z] /usr/lib/python3.6/http/client.py:459: in read
[2020-07-15T19:49:55.751Z]     n = self.readinto(b)
[2020-07-15T19:49:55.751Z] /usr/lib/python3.6/http/client.py:503: in readinto
[2020-07-15T19:49:55.751Z]     n = self.fp.readinto(b)
[2020-07-15T19:49:55.751Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2020-07-15T19:49:55.751Z]
[2020-07-15T19:49:55.751Z] self = <socket.SocketIO object at 0x7f843f715438>
[2020-07-15T19:49:55.751Z] b = <memory at 0x7f862e920288>
[2020-07-15T19:49:55.751Z]
[2020-07-15T19:49:55.751Z]     def readinto(self, b):
[2020-07-15T19:49:55.751Z]         """Read up to len(b) bytes into the writable buffer *b* and return
[2020-07-15T19:49:55.751Z]         the number of bytes read.  If the socket is non-blocking and no bytes
[2020-07-15T19:49:55.751Z]         are available, None is returned.
[2020-07-15T19:49:55.751Z]
[2020-07-15T19:49:55.751Z]         If *b* is non-empty, a 0 return value indicates that the connection
[2020-07-15T19:49:55.751Z]         was shutdown at the other end.
[2020-07-15T19:49:55.751Z]         """
[2020-07-15T19:49:55.751Z]         self._checkClosed()
[2020-07-15T19:49:55.751Z]         self._checkReadable()
[2020-07-15T19:49:55.751Z]         if self._timeout_occurred:
[2020-07-15T19:49:55.751Z]             raise OSError("cannot read from timed out object")
[2020-07-15T19:49:55.751Z]         while True:
[2020-07-15T19:49:55.751Z]             try:
[2020-07-15T19:49:55.751Z] >               return self._sock.recv_into(b)
[2020-07-15T19:49:55.751Z] E               Failed: Timeout >1200.0s
[2020-07-15T19:49:55.751Z]
[2020-07-15T19:49:55.751Z] /usr/lib/python3.6/socket.py:586: Failed
[2020-07-15T19:49:55.751Z] ---------------------------- Captured stderr setup -----------------------------
[2020-07-15T19:49:55.751Z] DEBUG:root:np/mx/python random seeds are set to 1197329715, use MXNET_TEST_SEED=1197329715 to reproduce.
[2020-07-15T19:49:55.751Z] ------------------------------ Captured log setup ------------------------------
[2020-07-15T19:49:55.751Z] DEBUG    root:conftest.py:193 np/mx/python random seeds are set to 1197329715, use MXNET_TEST_SEED=1197329715 to reproduce.
[2020-07-15T19:49:55.751Z] ----------------------------- Captured stdout call -----------------------------
[2020-07-15T19:49:55.751Z] Downloading /tmp/pytest-of-jenkins_slave/pytest-0/popen-gw2/test_inter_methods_in_augmente1/cifar10.zip from http://data.mxnet.io/mxnet/data/cifar10.zip...
[2020-07-15T19:49:55.751Z] ----------------------------- Captured stderr call -----------------------------
[2020-07-15T19:49:55.751Z] DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): data.mxnet.io:80
[2020-07-15T19:49:55.751Z] DEBUG:urllib3.connectionpool:http://data.mxnet.io:80 "GET /mxnet/data/cifar10.zip HTTP/1.1" 301 275
[2020-07-15T19:49:55.751Z] DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): data.mxnet.io.s3-website-us-west-1.amazonaws.com:80
[2020-07-15T19:49:55.751Z] DEBUG:urllib3.connectionpool:http://data.mxnet.io.s3-website-us-west-1.amazonaws.com:80 "GET /mxnet/data/cifar10.zip HTTP/1.1" 200 132571290
[2020-07-15T19:49:55.751Z]
[2020-07-15T19:49:55.751Z] ~~~~~~~~~~~~~~~~~~~~~ Stack of <unknown> (140214861137664) ~~~~~~~~~~~~~~~~~~~~~
[2020-07-15T19:49:55.751Z]   File "/usr/local/lib/python3.6/dist-packages/execnet/gateway_base.py", line 285, in _perform_spawn
[2020-07-15T19:49:55.751Z]     reply.run()
[2020-07-15T19:49:55.751Z]   File "/usr/local/lib/python3.6/dist-packages/execnet/gateway_base.py", line 220, in run
[2020-07-15T19:49:55.751Z]     self._result = func(*args, **kwargs)
[2020-07-15T19:49:55.751Z]   File "/usr/local/lib/python3.6/dist-packages/execnet/gateway_base.py", line 967, in _thread_receiver
[2020-07-15T19:49:55.751Z]     msg = Message.from_io(io)
[2020-07-15T19:49:55.751Z]   File "/usr/local/lib/python3.6/dist-packages/execnet/gateway_base.py", line 432, in from_io
[2020-07-15T19:49:55.751Z]     header = io.read(9)  # type 1, channel 4, payload 4
[2020-07-15T19:49:55.751Z]   File "/usr/local/lib/python3.6/dist-packages/execnet/gateway_base.py", line 400, in read
[2020-07-15T19:49:55.751Z]     data = self._read(numbytes - len(buf))
[2020-07-15T19:49:55.751Z] ------------------------------ Captured log call -------------------------------
[2020-07-15T19:49:55.751Z] DEBUG    urllib3.connectionpool:connectionpool.py:230 Starting new HTTP connection (1): data.mxnet.io:80
[2020-07-15T19:49:55.751Z] DEBUG    urllib3.connectionpool:connectionpool.py:442 http://data.mxnet.io:80 "GET /mxnet/data/cifar10.zip HTTP/1.1" 301 275
[2020-07-15T19:49:55.751Z] DEBUG    urllib3.connectionpool:connectionpool.py:230 Starting new HTTP connection (1): data.mxnet.io.s3-website-us-west-1.amazonaws.com:80
[2020-07-15T19:49:55.751Z] DEBUG    urllib3.connectionpool:connectionpool.py:442 http://data.mxnet.io.s3-website-us-west-1.amazonaws.com:80 "GET /mxnet/data/cifar10.zip HTTP/1.1" 200 132571290
[2020-07-15T19:49:55.751Z] --------------------------- Captured stderr teardown ---------------------------
[2020-07-15T19:49:55.751Z] INFO:root:np/mx/python random seeds are set to 1197329715, use MXNET_TEST_SEED=1197329715 to reproduce.
[2020-07-15T19:49:55.751Z] ---------------------------- Captured log teardown -----------------------------
[2020-07-15T19:49:55.751Z] INFO     root:conftest.py:210 np/mx/python random seeds are set to 1197329715, use MXNET_TEST_SEED=1197329715 to reproduce.
[2020-07-15T19:49:55.751Z] =============================== warnings summary ===============================

Network issue..


text:
compile error with clang

Note: Providing complete information in the most concise form is the best way to get help. This issue template serves as the checklist for essential information to most of the technical issues and bug reports. For non-technical issues and feature requests, feel free to present the information in what you believe is the best form.
For Q & A and discussion, please start a discussion thread at https://discuss.mxnet.io
Description
I can compile successfully with g++ but error appears with clang.
Environment info (Required)
----------Python Info----------
Version      : 3.6.3
Compiler     : GCC 7.2.0
Build        : ('default', 'Oct 24 2017 14:48:20')
Arch         : ('64bit', '')
------------Pip Info-----------
Version      : 9.0.1
Directory    : /usr/lib/python3.6/site-packages/pip
----------MXNet Info-----------
No MXNet installed.
----------System Info----------
Platform     : Linux-4.13.12-1-ARCH-x86_64-with-arch
system       : Linux
node         : Arch-R720
release      : 4.13.12-1-ARCH
version      : #1 SMP PREEMPT Wed Nov 8 11:54:06 CET 2017
----------Hardware Info----------
machine      : x86_64
processor    :
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               158
Model name:          Intel(R) Core(TM) i5-7300HQ CPU @ 2.50GHz
Stepping:            9
CPU MHz:             3283.002
CPU max MHz:         3500.0000
CPU min MHz:         800.0000
BogoMIPS:            4993.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
NUMA node0 CPU(s):   0-3
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0252 sec, LOAD: 1.2449 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.1978 sec, LOAD: 6.1024 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 1.0039 sec, LOAD: 9.4451 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.1305 sec, LOAD: 1.4144 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0125 sec, LOAD: 7.6978 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0104 sec, LOAD: 0.6309 sec.

Package used (Python/R/Scala/Julia):
(I'm using c++)
For Scala user, please provide:

Java version: (java -version)
Maven version: (mvn -version)
Scala runtime if applicable: (scala -version)

For R user, please provide R sessionInfo():
Build info (Required if built from source)
Compiler (gcc/clang/mingw/visual studio):
clang
MXNet commit hash:
17427d1dc949cb921a190151bc8de74ed50504ef
Build config:
export CC = gcc
export CXX = clang++
export NVCC = nvcc

# whether compile with options for MXNet developer
DEV = 0

# whether compile with debug
DEBUG = 0

# whether compile with profiler
USE_PROFILER =

# whether to turn on segfault signal handler to log the stack trace
USE_SIGNAL_HANDLER =

# the additional link flags you want to add
ADD_LDFLAGS =

# the additional compile flags you want to add
ADD_CFLAGS =

#---------------------------------------------
# matrix computation libraries for CPU/GPU
#---------------------------------------------

# whether use CUDA during compile
USE_CUDA = 1

# add the path to CUDA library to link and compile flag
# if you have already add them to environment variable, leave it as NONE
# USE_CUDA_PATH = /usr/local/cuda
USE_CUDA_PATH = /opt/cuda

# whether use CuDNN R3 library
USE_CUDNN = 1

#whether to use NCCL library
USE_NCCL = 0
#add the path to NCCL library
USE_NCCL_PATH = NONE

# whether use opencv during compilation
# you can disable it, however, you will not able to use
# imbin iterator
USE_OPENCV = 1

#whether use libjpeg-turbo for image decode without OpenCV wrapper
USE_LIBJPEG_TURBO = 0
#add the path to libjpeg-turbo library
USE_LIBJPEG_TURBO_PATH = NONE

# use openmp for parallelization
USE_OPENMP = 1

# MKL ML Library for Intel CPU/Xeon Phi
# Please refer to MKL_README.md for details

# MKL ML Library folder, need to be root for /usr/local
# Change to User Home directory for standard user
# For USE_BLAS!=mkl only
MKLML_ROOT=/usr/local

# whether use MKL2017 library
USE_MKL2017 = 0

# whether use MKL2017 experimental feature for high performance
# Prerequisite USE_MKL2017=1
USE_MKL2017_EXPERIMENTAL = 0

# whether use NNPACK library
USE_NNPACK = 0

# choose the version of blas you want to use
# can be: mkl, blas, atlas, openblas
# in default use atlas for linux while apple for osx
UNAME_S := $(shell uname -s)
ifeq ($(UNAME_S), Darwin)
USE_BLAS = apple
else
USE_BLAS = openblas
endif

# whether use lapack during compilation
# only effective when compiled with blas versions openblas/apple/atlas/mkl
USE_LAPACK = 1

# path to lapack library in case of a non-standard installation
USE_LAPACK_PATH =

# by default, disable lapack when using MKL
# switch on when there is a full installation of MKL available (not just MKL2017/MKL_ML)
ifeq ($(USE_BLAS), mkl)
USE_LAPACK = 1
endif

# add path to intel library, you may need it for MKL, if you did not add the path
# to environment variable
USE_INTEL_PATH = NONE

# If use MKL only for BLAS, choose static link automatically to allow python wrapper
ifeq ($(USE_MKL2017), 0)
ifeq ($(USE_BLAS), mkl)
USE_STATIC_MKL = 1
endif
else
USE_STATIC_MKL = NONE
endif

#----------------------------
# Settings for power and arm arch
#----------------------------
ARCH := $(shell uname -a)
ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64))
	USE_SSE=0
else
	USE_SSE=1
endif

#----------------------------
# distributed computing
#----------------------------

# whether or not to enable multi-machine supporting
USE_DIST_KVSTORE = 0

# whether or not allow to read and write HDFS directly. If yes, then hadoop is
# required
USE_HDFS = 0

# path to libjvm.so. required if USE_HDFS=1
LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server

# whether or not allow to read and write AWS S3 directly. If yes, then
# libcurl4-openssl-dev is required, it can be installed on Ubuntu by
# sudo apt-get install -y libcurl4-openssl-dev
USE_S3 = 0

#----------------------------
# performance settings
#----------------------------
# Use operator tuning
USE_OPERATOR_TUNING = 1

# Use gperftools if found
USE_GPERFTOOLS = 1

# Use JEMalloc if found, and not using gperftools
USE_JEMALLOC = 1

#----------------------------
# additional operators
#----------------------------

# path to folders containing projects specific operators that you don't want to put in src/operators
EXTRA_OPERATORS =

#----------------------------
# other features
#----------------------------

# Create C++ interface package
USE_CPP_PACKAGE = 1

#----------------------------
# plugins
#----------------------------

# whether to use caffe integration. This requires installing caffe.
# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH
# CAFFE_PATH = $(HOME)/caffe
# MXNET_PLUGINS += plugin/caffe/caffe.mk

# WARPCTC_PATH = $(HOME)/warp-ctc
# MXNET_PLUGINS += plugin/warpctc/warpctc.mk

# whether to use sframe integration. This requires build sframe
# git@github.com:dato-code/SFrame.git
# SFRAME_PATH = $(HOME)/SFrame
# MXNET_PLUGINS += plugin/sframe/plugin.mk

Error Message:
clang++ -std=c++11 -c -DMSHADOW_FORCE_STREAM -Wall -Wsign-compare -O3 -DNDEBUG=1 -I/home/coin/build/mxnet/mshadow/ -I/home/coin/build/mxnet/dmlc-core/include -fPIC -I/home/coin/build/mxnet/nnvm/include -I/home/coin/build/mxnet/dlpack/include -Iinclude -funroll-loops -Wno-unused-variable -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs -msse3 -I/opt/cuda/include -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_RABIT_PS=0 -DMSHADOW_DIST_PS=0 -DMSHADOW_USE_PASCAL=0 -DMXNET_USE_OPENCV=1 -I/usr/include/opencv -fopenmp -DMXNET_USE_OPERATOR_TUNING=1 -DMXNET_USE_LAPACK=1 -DMSHADOW_USE_CUDNN=1 -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free  -I/home/coin/build/mxnet/3rdparty/cub -DMXNET_USE_NCCL=0 -DMXNET_USE_LIBJPEG_TURBO=0 -MMD -c src/operator/random/multisample_op.cc -o build/src/operator/random/multisample_op.o
In file included from src/operator/random/multisample_op.cc:26:
In file included from src/operator/random/./multisample_op.h:34:
src/operator/random/./sampler.h:93:10: error: use of undeclared identifier
      'omp_get_num_threads'
  return omp_get_num_threads();
         ^
1 error generated.
make: *** [Makefile:372: build/src/operator/random/multisample_op.o] Error 1

Minimum reproducible example
I just compile the library by running
make

Steps to reproduce
(Paste the commands you ran that produced the error.)

clean all

make clean

recompile

make

What have you tried to solve it?

@CoinCheung
clang++ compiler does not support OpenMP. you have to disable it. set USE_OPENMP=0.
if you want to run with clang and openmp, try with clang-omp(https://clang-omp.github.io/).

Clang 3.7+ has OpenMP support. Though it is a separate library and not all installs ship it (e.g. Apple's copy for instance). However it is trivial to build yourself. Also you can get copies of the complete Clang toolchain from your package manager of choice or Anaconda if you use that.

Any updates? I also try to compile mxnet with clang to avoid ABI issue. But even I specifiy CC=clang CXX=clang++, the compiling will use gcc somewhere, and I got unrecognize -fopenmp=libopenmp error:
[ 70%] Building CXX object CMakeFiles/mxnet_static.dir/3rdparty/tvm/nnvm/src/pass/infer_shape_type.cc.o
[ 70%] Building CXX object CMakeFiles/mxnet_static.dir/3rdparty/tvm/nnvm/src/pass/order_mutation.cc.o
[ 70%] Building CXX object CMakeFiles/mxnet_static.dir/3rdparty/tvm/nnvm/src/pass/place_device.cc.o
[ 70%] Building CXX object CMakeFiles/mxnet_static.dir/3rdparty/tvm/nnvm/src/pass/plan_memory.cc.o
[ 70%] Building CXX object CMakeFiles/mxnet_static.dir/3rdparty/tvm/nnvm/src/pass/print_graph_ir.cc.o
[ 70%] Building CXX object CMakeFiles/mxnet_static.dir/3rdparty/tvm/nnvm/src/pass/saveload_json.cc.o
[ 70%] Building CUDA object CMakeFiles/mxnet_static.dir/src/common/random_generator.cu.o
gcc: error: unrecognized command line option ‘-fopenmp=libomp’; did you mean ‘-fopenmp-simd’?
make[2]: *** [CMakeFiles/mxnet_static.dir/build.make:3872: CMakeFiles/mxnet_static.dir/src/common/random_generator.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:88: CMakeFiles/mxnet_static.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

authors: CoinCheung hubutui jakirkham mwbyeon


text:
[MXNET-241] Module API for distributed training w/ row_sparse weight

Description
This PR implements the case for 'update_on_kvstore=True' in  https://github.com/apache/incubator-mxnet/issues/9921
@szha @anirudh2290 @zheng-da
verified on the mat-fact example by comparing dense & rowsparse weights. The results are the same.
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

Added more description to update and prepare to explain why we need this function and what the argument means. @piiswrong


text:
Update KL Divergence formula

Errors fix.
Description
(Brief description on what this PR is about)
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

@goldmermaid I think the 2nd change is valid, i.e., replacing pred with prob.

authors: goldmermaid sxjscience