text:
[SUPPORT]0.10 cow table insert mode cannot merge small files

Describe the problem you faced
In hudi 0.9, using Flink 1.12.2 SQL client sink logs to hudi cow table in insert mode. The small files would be merged into a few parquet files.
But hudi 0.10, same code would produce lots of small parquet files.  And turn  on write.insert.cluster option	 metioned in doc had no effect.
To Reproduce
Steps to reproduce the behavior:

hudi table option:

...
WITH (
  'connector' = 'hudi',
  'path' = 'hdfs://xxx:8020/hudi/xxx',
  'write.precombine.field' = 'time',
  'write.operation' = 'insert',
  'write.insert.cluster' = 'true',
  'write.insert.deduplicate' = 'true',
  'hoodie.datasource.write.recordkey.field' = 'distinct_id,time,event,_track_id',
  'read.streaming.enabled' = 'true',  -- this option enable the streaming read
  'read.streaming.start-commit' = '20211001000000', -- specifies the start commit instant time
  'read.streaming.check-interval' = '60', -- specifies the check interval for finding new source commits, default 60s.
  'table.type' = 'COPY_ON_WRITE', -- this creates a MERGE_ON_READ table, by default is COPY_ON_WRITE
  'hive_sync.enable' = 'true',     -- Required. To enable hive synchronization
  'hive_sync.mode' = 'hms',         -- Required. Setting hive sync mode to hms, default jdbc
  'hive_sync.table'='xxx',                          -- required, hive table name
  'hive_sync.db'='xxx',
  'hive_sync.metastore.uris' = 'thrift://xxx9083' -- Required. The port need set on hive-site.xml
);

Expected behavior
Small files get  merged.
Environment Description

Hudi version : 0.10-snapshot/0.11-snapshot

Spark version :NA

Hive version :2.1.1

Hadoop version :3.0

Storage (HDFS/S3/GCS..) : HDFS

Running on Docker? (yes/no) : no

Flink: 1.13.2

Additional context
NA
Stacktrace
NA

Thanks for your feedback, may i see your running DAG of the pipeline ?

Thanks for your feedback, may i see your running DAG of the pipeline ?

This is Flink 1.13.2 with master branch hudi (0.11-snapshot).

The pipeline is right, there is a bucket assigner there which would try to assign records to small buckets, do you insert continuously and see the file size ? The clean may clean the old small files then.

The pipeline is right, there is a bucket assigner there which would try to assign records to small buckets, do you insert continuously and see the file size ? The cleaner may clean the old small files then.

I started two task to consume same Kafka topic with different group id, sink to different  Hudi table:
1: Flink 1.12.2 with Hudi 0.9
2: Flink 1.13.2 with latest Hudi version(Started at 05:49 PM for test)
Two jobs pipeline are same, but the files under partition are different:
Job one keep 13 files, and new data are merged into those files, and be merged into two files(because no much data)  at second day.
Job two just add new files, and there were more than 2000 files under every partition.

Okey, let me have a test on local cluster.

Fixed in https://github.com/apache/hudi/commit/934fe54cc57b508875383cd807735b1323fef754


text:
[SUPPORT] Job became slow because reading archived commit files

Tips before filing an issue

Have you gone through our FAQs?

Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced
We are running Glue streaming with Hudi 0.12.1 to write data from Kafka. The table is MOR for quick ingestion and commit method is inline as we don't want to run another job to perform compaction.
Initially each commit took 1.5 minutes. But after one day, it became slow to 3-4 minutes. After a few days, it became more than 10 minutes.
We noticed the slowness is due to Hudi was reading archived commits files after each commit. Because there were tons of commits files under .hoodie/archive directory on S3, it really took time. After I deleted those files, the speed became quick. But it would come down later.
I did some research and I found currently Hudi does not support purge these archived files automatically. There is a feature to merge small archived files. I tried it and the situation became better. But it was not fast enough.
Further research shows that might be related to sync hive meta. But I do need to sync. Is there any way to speed up this procedure? Thank you.
A clear and concise description of the problem.
To Reproduce
Steps to reproduce the behavior:

Set up a Kafka

Write pyspark to read stream from Kafka and then sink to hudi on S3.

Hudi config:
 'className' : 'org.apache.hudi',
 'hoodie.datasource.hive_sync.use_jdbc':'false',
 'hoodie.datasource.write.partitionpath.field': partitionKey,
 'hoodie.datasource.write.recordkey.field': primaryKey,
 'hoodie.datasource.write.precombine.field': timestamp,
 'hoodie.datasource.write.operation': 'insert',
 'hoodie.table.name': targetTableName,
 'hoodie.consistency.check.enabled': 'true',
 'hoodie.datasource.write.hive_style_partitioning':'true',
 'hoodie.datasource.write.table.type': 'MERGE_ON_READ',
 'hoodie.datasource.hive_sync.database': targetDBName,
 'hoodie.datasource.hive_sync.table': targetTableName,
 'hoodie.datasource.hive_sync.enable': 'true',
 'hoodie.datasource.hive_sync.partition_fields': partitionKey,
 'hoodie.datasource.hive_sync.partition_extractor_class': 'org.apache.hudi.hive.MultiPartKeysValueExtractor',
 'hoodie.datasource.hive_sync.support_timestamp':'true',
 'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.SimpleKeyGenerator',
 'hoodie.archive.merge.enable':'true',
 'hoodie.index.type':'BLOOM',  #SIMPLE
 'hoodie.compact.inline': 'true',

Expected behavior
A clear and concise description of what you expected to happen.
Environment Description

Hudi version : 0.12.1

Spark version : 3.1 (AWS Glue 3.0)

Hive version :

Hadoop version :

Storage (HDFS/S3/GCS..) : S3

Running on Docker? (yes/no) :

Additional context
Add any other context about the problem here.
Stacktrace
Add the stacktrace of the error.

Just happened to find the PR https://github.com/apache/hudi/pull/7561. It seems be helpful. Is that true? Thank you.

yes, we already fixed it and will be part of 0.13.0
thanks for reporting.
will go ahead and close out the issue since its already fixed.

source: gharchive/issue

authors: nsivabalan xmubeta

repo: apache/hudi

license_type: permissive

license_source: github-api


text:
[MINOR] Add parallel listing of existing partitions

Change Logs
Currently, the Glue sync works next way(simplified for particular problem explanation):

Read partitions which were upserted(create/update is indistinguishable) and deleted from the commitline(since previous sync time).
Read existing partitions from Glue.
Iterate to understand which AWS call should be done for the changed partition(create/update/delete).

To facilitate this, there are 2 clients, 1 is AWSGlueCatalogSyncClient, second is HoodieHiveSyncClient. In the Spark-EMR case, the second relies on the AWS's Hive-Glue interface implementation. There is no public code of the actual version, but there is of some previous, the interesting part is this.
Saying short for AWS Glue exists a mechanism of segments which allows to query long lists in parallel. I added code similar to what exists in AWS Hive-Glue implementation.
The real use-case for us was that we have been using HiveSyncTool (table with ~200k partitions) - and it was behaving slow when a lot of partitions were changing/in case of resync - so we decided to try out AWSGlueCatalogSyncClient which gave about 2x boost in write speed, but after we noticed a significant degradation in speed of listing(roughly 40 sec vs 210 on the ~same amount of partitions - 200k).
This improvement should bring the same listing speed to the tool.
Impact
There is a new config added with the default value from AWS Glue-Hive(parallelism of 5) - while before it was sequential, so I don't expect it to be failing, but we may set parallelism to 1 to be backward-compatible for whatever reasons.
Risk level (write none, low medium or high below)
Low
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change
Need to adjust the website, but please let me know if you are ok with the changes, I'll proceed then.

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

@hudi-bot run azure

@yihua will you review this if I fix conflicts?

@yihua @nsivabalan is there any chance you'll be able to take a look on it? It's a significant improvement and makes sync much faster... We've been running it in production for a month already and there are no issues.

@yihua @nsivabalan

@VitoMakarevich Thanks for the contribution.  Could you rebase the PR on the latest master and resolve the conflicts?

cc @VitoMakarevich for updating and I think it is a valuable contribution.

Thanks for the review, will take a look this week.

@yihua I found that functionality I'm trying to bring heavily collides with https://github.com/apache/hudi/pull/10572.
So for the case of loading a list of partitions, there are now the following mechanisms:

Get all - was present always, simply get everything - does not well scale with a lot of partitions - in particular our case can spend 10+ minutes in each commit just getting partitions.
Try to generate the pushdown filter(quite recent addition with improvements recently) - if approximately it can fit 2048 characters - generate a list of all partitions, otherwise get min/max from changed partitions and read within range. This again has an error in 2048 limit - as it depends on partition depth/name length and so on, and min/max is better than nothing but suffers from entropy.
Mechanism I use - simply call batchGetPartition with all partitions changed - scales almost indefinitely.

And basically 1st and 2nd are what is in master branch - but with my approach, we don't need it at all, since get all will be needed only initially(when creating the Glue database), then we can use incremental only. But since it's in master, we may need to have backward compatibility, can you suggest a course forward? I can make a feature flag to use this behavior under certain feature flags - but would like to use it only for HiveSyncTool - where it makes more sense, and for AWS's implementation use my mechanism.
As for the rest - such as improvement of all partitions listing/parallelized update/create/delete - it does not collide with anything.
Waiting for your advice, I'm now stuck because of falling IT - but I need to know the direction we go.

@parisni can you take a look at this PR if it addresses your concerns? We are operating 2M+ partition syncs with this code within seconds.

@parisni I will add improvements in the following PR. The list I plan to add:

IT as you mentioned
.excludeColumnSchema(true) - although as I remember seeing AWS Glue -there was no schema in partitions, anyway it's one line so I will add it.
Add documentation to the web page so that new parameters are covered.
If there is something besides it, please comment, will work on it next week.

thanks @VitoMakarevich

although as I remember seeing AWS Glue -there was no schema in partitions, anyway it's one line so I will add it.

indeed in glue, any partition is associated with it's own schema. When one create a partition, glue picks up the current table schema and associates it with the partition. Then when you fetch partition, default is to also fetch the associated schema, which adds resource overhead AFAIK

@danny0405 small follow-up just to confirm - config changes will be picked up automatically right? I opened asf-site branch and it looks like if I just add a new parameter to an existing class - it should be visible once someone rebuilds a site

yeah, the web-site would rebuild in minutes.

source: gharchive/pull-request

authors: VitoMakarevich danny0405 parisni yihua


text:
[MINOR] Make ordering deterministic in small file selection

Change Logs
Makes the ordering deterministic to get consistent results and avoid any issues in tests
Impact
Small file selection is consistent (mostly helps tests be consistent)
Risk level (write none, low medium or high below)
None
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

@hudi-bot run azure


text:
[HUDI-974] fix fields out of order in MOR mode when using Hive

Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request
Fields out of order in MOR mode when using Hive,with setting hoodie.realtime.merge.skip = true
Brief change log
modify getWriterSchema in RealtimeUnmergedRecordReader to getHiveSchema
Verify this pull request
This pull request is already covered by existing tests TestHoodieRealtimeRecordReader.
Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Codecov Report

Merging #1711 into master will decrease coverage by 54.06%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master    #1711       +/-   ##
=============================================
- Coverage     72.25%   18.19%   -54.07%
- Complexity      294      857      +563
=============================================
  Files           374      348       -26
  Lines         16371    15358     -1013
  Branches       1654     1525      -129
=============================================
- Hits          11829     2794     -9035
- Misses         3806    12206     +8400
+ Partials        736      358      -378

Impacted Files
Coverage Δ
Complexity Δ

.../hadoop/realtime/RealtimeUnmergedRecordReader.java
0.00% <0.00%> (-96.97%)
0.00 <0.00> (ø)

.../java/org/apache/hudi/client/HoodieReadClient.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

.../java/org/apache/hudi/metrics/MetricsReporter.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

.../java/org/apache/hudi/common/model/ActionType.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

...java/org/apache/hudi/io/HoodieRangeInfoHandle.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

.../java/org/apache/hudi/hadoop/InputPathHandler.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

...a/org/apache/hudi/exception/HoodieIOException.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

...org/apache/hudi/table/action/commit/SmallFile.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

...rg/apache/hudi/index/bloom/KeyRangeLookupTree.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

...g/apache/hudi/exception/HoodieInsertException.java
0.00% <0.00%> (-100.00%)
0.00% <0.00%> (ø%)

... and 336 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update acb1ada...3abe325. Read the comment docs.


text:
The button miss text in the mobile view

Describe the bug
The button miss text in Admin->Content->Questions in the mobile view.
Expected behavior
text: More
Screenshots

Platform

Device: Mobile
OS: iOS
Browser and version: Safari
Version: v1.3.0

@fenbox
i'll do this

@fenbox
here is the PR https://github.com/apache/incubator-answer/pull/921

authors: fenbox zahash

repo: apache/incubator-answer


text:
抽象类如何转换成object pool形式来管理

google::protobuf::Message本身是一个abstract class，如何在一开始的时候，如何将这种形式的：
std::unique_ptrgoogle::protobuf::Message req;
ret.reset(svc->GetRequestPrototype(method).New());
转换成object pool形式来管理？

解决了

repo: apache/incubator-brpc


text:
Config UI container has neither liveness nor healthiness probe

Hello,
is there a special reason why the container has neiter liveness nor healthiness probe defined? The lake deployment has the property .Values.lake.livenessProbe but for config-ui there is none. Isn't it best practice to have every container define both probes?
Cheers

hi @rbraeunlich, thanks for the suggestion, we will add the probes

repo: apache/incubator-devlake-helm-chart


text:
[Bug][Module Name] Bug title devlake docker container always restarting, even after trying with option suggested does not help

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
While installing devlake through docker compose method, I see the 'devlake' container is always in 'restarting' status.
I did notice this as known issue and it was suggested to use version 2 of compose and check health, but that really did not help much.
Iam using latest available.
sha256:0c0d74f44c6171c829418bceae76b335f3bebff92a24dee1938ef4cb632ff128
any help would be appreciated to unblock me.
Note: I can only pull latest version of image through our internal docker hub.
What do you expect to happen
I expect the devlake container to be UP and running fine.
How to reproduce
Tried to reproduce in couple of other systems, and I could do that every time.
by simply running docker compose and pointing to our internal docker hub to pull the latest available image, i.e
sha256:0c0d74f44c6171c829418bceae76b335f3bebff92a24dee1938ef4cb632ff128
Anything else
No response
Version
sha256:0c0d74f44c6171c829418bceae76b335f3bebff92a24dee1938ef4cb632ff128
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Thanks @dosu . I checked for port 8080 if it is in use, and it is not in use.
and I can confirm that internal docker hub works fine, as I'm able to create other containers without any problems.
I request you to check if the image with sha id that I'm using has any known issue. If not, then I need help for sure from you/forum on how to fix this.

I did set the ENCRYPTION_SECRET as mentioned in the documentation, however docker logs say the error is with the variable.
I tried setting it in .env, and later also as local variable on bash shell.  Still it gives the same/below error.
Version:  v1.0.1@e061ef2
panic: ENCRYPTION_SECRET must be set in environment variable or .env file

Pls ignore this.
I found there is another variable defined as blank by default in the package, which was overwriting what I was setting up.

repo: apache/incubator-devlake


text:
heartbeat for zk failed

[ERROR] 2020-06-29 18:16:59.995 org.apache.dolphinscheduler.common.zk.AbstractZKClient:[172] - heartbeat for zk failed : KeeperErrorCode = ConnectionLoss for /dolphinscheduler/workers/127.0.0.1_0000000008
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /dolphinscheduler/workers/127.0.0.1_0000000008
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1336)
at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:282)
at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:278)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:275)
at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:261)
at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:34)
at org.apache.dolphinscheduler.common.zk.AbstractZKClient.heartBeatForZk(AbstractZKClient.java:169)
at org.apache.dolphinscheduler.server.worker.WorkerServer$2.run(WorkerServer.java:293)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[INFO] 2020-06-29 18:17:00.009 org.apache.dolphinscheduler.server.worker.WorkerServer:[223] - worker server is stopping ..., cause : heartbeat for zk exception, release resources and stop myself
[INFO] 2020-06-29 18:17:03.010 org.apache.dolphinscheduler.server.worker.WorkerServer:[240] - heartbeat service stopped
[ERROR] 2020-06-29 18:17:07.612 org.apache.dolphinscheduler.common.queue.TaskQueueZkImpl:[369] - get all tasks from tasks queue exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /dolphinscheduler/tasks_kill
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1659)
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:230)
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:219)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:216)
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:207)
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:40)
at org.apache.dolphinscheduler.common.queue.TaskQueueZkImpl.smembers(TaskQueueZkImpl.java:361)
at org.apache.dolphinscheduler.server.worker.WorkerServer$3.run(WorkerServer.java:324)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[INFO] 2020-06-29 18:17:38.707 org.apache.dolphinscheduler.server.worker.WorkerServer:[248] - threadpool service stopped
[INFO] 2020-06-29 18:17:38.707 org.apache.dolphinscheduler.server.worker.WorkerServer:[255] - worker kill executor service stopped
[INFO] 2020-06-29 18:17:38.707 org.apache.dolphinscheduler.server.worker.WorkerServer:[262] - worker fetch task service stopped
[ERROR] 2020-06-29 18:17:38.707 org.apache.dolphinscheduler.common.queue.TaskQueueZkImpl:[285] - delete task:2_6853_2_6766_-1 from zookeeper fail, exception:
java.lang.InterruptedException: null
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1407)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:880)
at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:250)
at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:244)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:241)
at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:225)
at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:35)
at org.apache.dolphinscheduler.common.queue.TaskQueueZkImpl.removeNode(TaskQueueZkImpl.java:282)
at org.apache.dolphinscheduler.server.worker.runner.FetchTaskThread.removeNodeFromTaskQueue(FetchTaskThread.java:251)
at org.apache.dolphinscheduler.server.worker.runner.FetchTaskThread.run(FetchTaskThread.java:234)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[INFO] 2020-06-29 18:17:38.726 org.apache.zookeeper.ZooKeeper:[693] - Session: 0x10058d42144008b closed
[ERROR] 2020-06-29 18:17:38.731 org.apache.curator.framework.imps.CuratorFrameworkImpl:[566] - Background exception was not retry-able or retry gave up
java.lang.IllegalStateException: Client is not started
at org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:176)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:507)
at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:221)
at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:35)
at org.apache.curator.framework.imps.FailedDeleteManager.addFailedDelete(FailedDeleteManager.java:55)
at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:274)
at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:225)
at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:35)
at org.apache.curator.framework.recipes.locks.LockInternals.deleteOurPath(LockInternals.java:339)
at org.apache.curator.framework.recipes.locks.LockInternals.releaseLock(LockInternals.java:123)
at org.apache.curator.framework.recipes.locks.InterProcessMutex.release(InterProcessMutex.java:154)
at org.apache.dolphinscheduler.common.zk.AbstractZKClient.releaseMutex(AbstractZKClient.java:497)
at org.apache.dolphinscheduler.server.worker.runner.FetchTaskThread.run(FetchTaskThread.java:240)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[INFO] 2020-06-29 18:17:38.733 org.apache.dolphinscheduler.common.zk.AbstractZKClient:[138] - zookeeper close ...
[INFO] 2020-06-29 18:17:38.733 org.apache.dolphinscheduler.server.worker.WorkerServer:[270] - zookeeper service stopped

first，please check if there is packet loss on the network,
if the network is fine, please show zookeeper.properties info under conf directory, the default value of two following items is too short
zookeeper.session.timeout=60000 zookeeper.connection.timeout=30000

@antlers-lv yes, @dailidong gave perfect suggestions, you can try the above parameters. I will close this ticket, feel free to reopen it necessarily

authors: antlers-lv dailidong gabrywu

repo: apache/incubator-dolphinscheduler


text:
Add tdigest compression param for pencentile_approx function

Someone use doris and test the pencentile_approx function. He found that this function in doris is much slower than that in clickhouse.
The algorithms of pencentile_approx function in doris and clickhouse are the same. One of the reasons is that digest compresion in clickhouse is 2048, but it is 10000 in doris. That means we can get a more precise result in doris with the cost of much time.
So, we enable uses to set the compression param to trade off the precision and time cost by themselves.

Patch is here Add tdigest compression param for pencentile_approx function

With the different value of compression, we can get results as follows.
mysql>  select percentile_approx(siteid, 0.8,100) from table1;
+-----------------------------------------+
| percentile_approx(`siteid`, 0.8, 100.0) |
+-----------------------------------------+
|                      67062264.459699914 |
+-----------------------------------------+
1 row in set (5.54 sec)

mysql>  select percentile_approx(siteid, 0.8,1000) from table1;
+------------------------------------------+
| percentile_approx(`siteid`, 0.8, 1000.0) |
+------------------------------------------+
|                       67026452.899213754 |
+------------------------------------------+
1 row in set (6.71 sec)

mysql>  select percentile_approx(siteid, 0.8,10000) from table1;
+-------------------------------------------+
| percentile_approx(`siteid`, 0.8, 10000.0) |
+-------------------------------------------+
|                        67022356.721001081 |
+-------------------------------------------+
1 row in set (8.15 sec)

mysql> select count(*) from table1;
+----------+
| count(*) |
+----------+
| 83777768 |
+----------+
1 row in set (2.09 sec)

repo: apache/incubator-doris


text:
Added IntelliJ Inspection

Issue #6856
Make IntelliJ inspection "Iteration over keySet() could be replaced with entrySet()" a error

@hpandeycodeit would you fix the violations of this rule? There are 18 of them, according to CI: https://teamcity.jetbrains.com/viewLog.html?buildId=1937670&tab=Inspection&buildTypeId=OpenSourceProjects_Druid_InspectionsPullRequests

@leventov yes, working on it.

@leventov I created a new PR for this issue as the current one is messed up.
https://github.com/apache/incubator-druid/pull/7067

authors: hpandeycodeit leventov

repo: apache/incubator-druid


text:
6088 - Time Ordering On Scans

This PR addresses points 2 and 3 of issue #6088, is based on proposal #7036, and stems from a refactor of the spaghetti in PR #7024.
This PR introduces two strategies for time-ordering results from scan queries:

Priority Queue: Each segment on a Historical is opened sequentially.  Every row is added to a bounded priority queue which is ordered by timestamp.  For every row above the result set limit, the row with the earliest (if descending) or latest (if ascending) timestamp will be dequeued.  After every row has been processed, the sorted contents of the priority queue are streamed back to the Broker(s) in batches.

K-Way/N-Way Merge: Each segment on a Historical is opened in parallel.  Since each segment's rows are already time-ordered, a k-way merge can be performed on the results from each segment.  This approach doesn't persist the entire result set in memory (like the Priority Queue) as it streams back batches as they are returned from the merge function.

Each strategy has a corresponding configurable limit to protect historicals from OOMEs:

The druid.query.scan.maxRowsQueuedForTimeOrdering property protects
from OOMEs by limiting the number of rows in the query result set when time ordering is used.

The druid.query.scan.maxSegmentsTimeOrderedInMemory limit protects from creating too many decompression/decoding buffers by capping the number of segments opened per historical when time ordering is used.

Seems that you have to scan your PR for checkstyle rules:
mvn checkstyle:check
in order to get it successfully built.

The way this PR implements time ordering, the entire query will fail if the limits are exceeded on 1 or more historical.  This behaviour makes sense, right?

@justinborromeo Yeah, that's fine

I don't think this needs to be something done in this PR, but it's worth mentioning that moving the "time chunking" behavior to the broker, and issuing several subqueries to the historical could be a possible optimization:
This could help in situations like the following:

I want an ascending time scan with limit X, across three days worth of data, with DAY granularity segments
The first day of data has enough data to hit limit X
I have three historicals: the first has data for the first day, and the second has data for the second 2 day, and the third has data only for third day
With the current approach in this PR, historical 2 and 3 would also have to do work to answer this query (it would go and return up to limit X rows, even though none of them would be included in the final result set), when only historical 1 really needs to be queried
If the broker split the query up into daily subqueries, this would cut down on the overall workload across historicals

This would require more change to the broker to be able to support this new type of query, so I think the current approach is fine for now.

For the unordered case, there is no performance regression as demonstrated by the results of the ScanBenchmark before and after the PR below.  The benchmark was run on an EC2 instance.
java -Xmx8g -server -jar target/benchmarks.jar ScanBenchmark -f 1 -i 50 -wi 25 -p schemaAndQuery=basic.A -p ordering=NONE -p limit=0,1000,99999

After PR:
# Run complete. Total time: 00:33:03

Benchmark                                  (limit)  (numProcessingThreads)  (numSegments)  (rowsPerSegment)  (schemaAndQuery)  Mode  Cnt        Score        Error  Units
ScanBenchmark.queryMultiQueryableIndex           0                       2              2            200000           basic.A  avgt   50  1047299.585 ± 100832.876  us/op
ScanBenchmark.queryMultiQueryableIndex           0                       2              4            200000           basic.A  avgt   50  3327877.446 ± 340896.932  us/op
ScanBenchmark.queryMultiQueryableIndex        1000                       2              2            200000           basic.A  avgt   50     2871.893 ±      1.312  us/op
ScanBenchmark.queryMultiQueryableIndex        1000                       2              4            200000           basic.A  avgt   50     2932.073 ±      3.538  us/op
ScanBenchmark.queryMultiQueryableIndex       99999                       2              2            200000           basic.A  avgt   50   230683.292 ±   1364.525  us/op
ScanBenchmark.queryMultiQueryableIndex       99999                       2              4            200000           basic.A  avgt   50   231715.616 ±   1572.446  us/op

ScanBenchmark.querySingleIncrementalIndex        0                       2              2            200000           basic.A  avgt   50   349069.471 ±   4376.445  us/op
ScanBenchmark.querySingleIncrementalIndex        0                       2              4            200000           basic.A  avgt   50   340061.203 ±   3059.043  us/op
ScanBenchmark.querySingleIncrementalIndex     1000                       2              2            200000           basic.A  avgt   50     1258.837 ±      0.716  us/op
ScanBenchmark.querySingleIncrementalIndex     1000                       2              4            200000           basic.A  avgt   50     1320.102 ±      1.458  us/op
ScanBenchmark.querySingleIncrementalIndex    99999                       2              2            200000           basic.A  avgt   50   155196.629 ±   1357.245  us/op
ScanBenchmark.querySingleIncrementalIndex    99999                       2              4            200000           basic.A  avgt   50   162660.413 ±   1560.482  us/op

ScanBenchmark.querySingleQueryableIndex          0                       2              2            200000           basic.A  avgt   50   492261.267 ±   5470.700  us/op
ScanBenchmark.querySingleQueryableIndex          0                       2              4            200000           basic.A  avgt   50   480649.890 ±   5540.968  us/op
ScanBenchmark.querySingleQueryableIndex       1000                       2              2            200000           basic.A  avgt   50     2795.713 ±      1.473  us/op
ScanBenchmark.querySingleQueryableIndex       1000                       2              4            200000           basic.A  avgt   50     2864.783 ±      2.959  us/op
ScanBenchmark.querySingleQueryableIndex      99999                       2              2            200000           basic.A  avgt   50   240433.943 ±   1845.204  us/op
ScanBenchmark.querySingleQueryableIndex      99999                       2              4            200000           basic.A  avgt   50   234122.746 ±   1507.052  us/op

Before PR:
# Run complete. Total time: 00:33:02

Benchmark                                  (limit)  (numProcessingThreads)  (numSegments)  (rowsPerSegment)  (schemaAndQuery)  Mode  Cnt        Score        Error  Units
ScanBenchmark.queryMultiQueryableIndex           0                       2              2            200000           basic.A  avgt   50  1172830.904 ± 201573.754  us/op
ScanBenchmark.queryMultiQueryableIndex           0                       2              4            200000           basic.A  avgt   50  3267726.979 ± 319137.068  us/op
ScanBenchmark.queryMultiQueryableIndex        1000                       2              2            200000           basic.A  avgt   50     2824.916 ±      3.079  us/op
ScanBenchmark.queryMultiQueryableIndex        1000                       2              4            200000           basic.A  avgt   50     2777.685 ±      3.450  us/op
ScanBenchmark.queryMultiQueryableIndex       99999                       2              2            200000           basic.A  avgt   50   231104.252 ±   1376.867  us/op
ScanBenchmark.queryMultiQueryableIndex       99999                       2              4            200000           basic.A  avgt   50   231335.680 ±   1355.400  us/op

ScanBenchmark.querySingleIncrementalIndex        0                       2              2            200000           basic.A  avgt   50   342828.419 ±   4551.446  us/op
ScanBenchmark.querySingleIncrementalIndex        0                       2              4            200000           basic.A  avgt   50   333952.670 ±   3096.050  us/op
ScanBenchmark.querySingleIncrementalIndex     1000                       2              2            200000           basic.A  avgt   50     1307.223 ±      0.686  us/op
ScanBenchmark.querySingleIncrementalIndex     1000                       2              4            200000           basic.A  avgt   50     1288.650 ±      1.255  us/op
ScanBenchmark.querySingleIncrementalIndex    99999                       2              2            200000           basic.A  avgt   50   158636.920 ±    781.898  us/op
ScanBenchmark.querySingleIncrementalIndex    99999                       2              4            200000           basic.A  avgt   50   162173.353 ±    961.087  us/op

ScanBenchmark.querySingleQueryableIndex          0                       2              2            200000           basic.A  avgt   50   488685.621 ±   5538.093  us/op
ScanBenchmark.querySingleQueryableIndex          0                       2              4            200000           basic.A  avgt   50   486952.922 ±   5868.651  us/op
ScanBenchmark.querySingleQueryableIndex       1000                       2              2            200000           basic.A  avgt   50     2879.226 ±      1.755  us/op
ScanBenchmark.querySingleQueryableIndex       1000                       2              4            200000           basic.A  avgt   50     2823.509 ±     21.211  us/op
ScanBenchmark.querySingleQueryableIndex      99999                       2              2            200000           basic.A  avgt   50   238340.781 ±   1514.616  us/op
ScanBenchmark.querySingleQueryableIndex      99999                       2              4            200000           basic.A  avgt   50   235502.141 ±   1402.612  us/op

authors: jon-wei justinborromeo kamaci


text:
Too many DubboClientHandler threads in consumer-消费者中建立了太多DubboClientHandler 线程

Dubbo Version: 2.5.3
I have a large provider cluster with more than 30 instances. And I found that in client side, the consumer uses a stand alone thread pools for each instances, so there are 30 thread pool to consume the service, which I can find out from the jstack that I have many diffrent thread groups named like DubboClientHandler-IP1-XXX.
Question1:
I guess the consumer uses stand alone cached thread pool for each provider instances but with the max pool size of "default.actives", am I right?
Question2:
But the problem is that, I hope to maximize the tps , so I provide a very large cluster, but now the consumer will suffer from high cpu consumption due to large thread counts.
Can I change the thread model in consumer side? Like I use a shared thread pool so that I can control the maximum thread size.

我有一个非常大的provider 集群（30个实例）。然后我们发现在客户端，为每一个实例，都准备了一个独立线程池，所以我们看到有30个线程池以去发起消费请求，这个从jstack堆栈上可以证明，因为里面很多线程组的名字是类似DubboClientHandler-IP1-XXX，其中ip都是不一样的。
问题1：
按照我的猜测，每个实例是应该会有一个cached的线程池（可回收），然后其中的最大线程数是default.actives，是否如此？
问题2：
之所以我们提供如此大的服务集群，是因为我们希望最大化TPS，但现在集群是大了，但是消费者又因此导致巨大的线程消耗（高CPU）。我们能否通过配置改变这个线程的模型，例如用一个共享的线程池去处理所有的服务实例，这样最大的线程数便可控了。

Duplicate of #2013

authors: Jaskey chickenlj

repo: apache/incubator-dubbo


text:
在mouseover的时候，使用setoption设置radar的indicator的时候，数据会无限叠加

在mouseover的时候，使用setoption设置radar的indicator的时候，数据会无限叠加

设置radar的indicator中经过的字体名称的颜色的时候，数据一直叠加
		options = {
				 radar: [{
					 indicator: setindicator
					 }],
		}
		radarEchartsBox.setOption(options)

Please use issue template to create an issue and provide an online demo from jsfiddle.

authors: IIIIIIIIF Ovilia

repo: apache/incubator-echarts

license_source: bigquery


text:
[ISSUE #790] Knative Connector: Initial Implementation of Consumer.

Fixing ISSUE #790.
Motivation
Implemented and tested Consumer module to subscribe an event from Knative (ISSUE #790).
The PRs for ISSUE #790 will be merged into knative-connector branch.
Modifications
Implementation

KnativeConsumer base on Pull communication model

subscribe method
unsubscribe method

Unit test for KnativeConsumer
Integration test for KnativeProducer and KnativeConsumer

Thanks @qqeasonchen and @xwm1992 for their super helpful suggestions in the design and implementation.
How to Test
Assume the cloudevents-player Knative service has already been set up on your local machine; otherwise, please follow the steps in https://knative.dev/docs/getting-started/first-source/#sending-an-event to set up a Knative service (cloudevents-player) as source and sink.
# Set eventMesh.connector.plugin.type=knative in eventmesh-runtime/conf/eventmesh.properties
# Do integration test and subscribed event message can be seen in the console:
$ cd incubator-eventmesh/eventmesh-connector-plugin/eventmesh-connector-knative
$ ../../gradlew clean test --tests KnativeConnectorTest.testConnect -i

Documentation

Does this pull request introduce a new feature? Yes.
If yes, how is the feature documented? Not documented.
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation. The documentation will be created in another PR.

@xwm1992 Please have a look when you have time. Thanks very much in advance!

@pchengma you may need to create a new issue link to this pr.

@pchengma you may need to create a new issue link to this pr.

Let me create a new issue and link it to this PR.

@pchengma you may need to create a new issue link to this pr.

Let me create a new issue and link it to this PR.

Done with the new issue creation and linked it to this PR.

authors: pchengma qqeasonchen

repo: apache/incubator-eventmesh


text:
[ISSUE #481] Use YAML file to store configuration

The main change:

Use eventmesh.yml to replace eventmesh.properties
Use static field to get the configuration property, the advantage of this change is that there is no need to pass the configuration objects.

Codecov Report

Merging #483 (a2d31b0) into develop (fefe391) will decrease coverage by 0.31%.
The diff coverage is 9.18%.

:exclamation: Current head a2d31b0 differs from pull request most recent head 70f4e32. Consider uploading reports for the commit 70f4e32 to get more accurate results

@@              Coverage Diff              @@
##             develop     #483      +/-   ##
=============================================
- Coverage      11.23%   10.92%   -0.32%
- Complexity       326      330       +4
=============================================
  Files            228      229       +1
  Lines          10816    10731      -85
  Branches         907      891      -16
=============================================
- Hits            1215     1172      -43
+ Misses          9504     9462      -42
  Partials          97       97

Impacted Files
Coverage Δ

...in/java/org/apache/eventmesh/common/Constants.java
0.00% <ø> (ø)

...he/eventmesh/common/EventMeshRuntimeException.java
0.00% <0.00%> (ø)

...e/eventmesh/common/config/CommonConfiguration.java
0.00% <0.00%> (-91.23%)
:arrow_down:

...ntime/admin/controller/ClientManageController.java
0.00% <0.00%> (ø)

...he/eventmesh/runtime/boot/EventMeshHTTPServer.java
0.00% <0.00%> (ø)

...apache/eventmesh/runtime/boot/EventMeshServer.java
0.00% <0.00%> (ø)

...pache/eventmesh/runtime/boot/EventMeshStartup.java
0.00% <0.00%> (ø)

...che/eventmesh/runtime/boot/EventMeshTCPServer.java
0.00% <0.00%> (ø)

...time/configuration/EventMeshHTTPConfiguration.java
0.00% <0.00%> (ø)

...ntime/configuration/EventMeshTCPConfiguration.java
0.00% <0.00%> (ø)

... and 28 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fefe391...70f4e32. Read the comment docs.

@ruanwenjun  hello，In TCP mode, there is a problem with Connection refused. Doesn't exist in the development version.

@Jackzeng1224 Thanks a lot, I have solved, please help me test again.

@Jackzeng1224 Thanks a lot, I have solved, please help me test again.
OK.


text:
[GOBBLIN-512]Add containers to hold unchecked exceptions from closing resources in broker

Add containers to hold unchecked exceptions from closing resources in broker, to make sure all resources are correctly handled, and log exceptions afterwards.
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[x] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-512] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-512

Description

[x] Here are some details about my PR, including screenshots (if applicable):

Tests

[ ] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[x] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

@zxcware @ibuenros
Can you help review ? Thanks.

repo: apache/incubator-gobblin


text:
fix building problem with source tarball

When building with the release candidate source tarball, we found it fails with the following issue:
ERROR: Process exited with status 1: Process exited with status 1
fatal: Not a git repository (or any of the parent directories): .git
Failed to run command to check head: git rev-parse --abbrev-ref HEAD
INFO: Elapsed time: 431.371s, Critical Path: 143.15s
FAILED: Build did NOT complete successfully

This is because the scripts/release/status.sh script relies on some git information while the source tarball excludes .git directory.
This PR fixes the building failure issue and will generate the following version information when the heron client is built with a source tar ball.
[tw-mbp-nlu incubator-heron-v-0.20.0-incubating-candidate-2]$ ~/.heron/bin/heron version
heron.build.git.revision : incubator-heron-v-0.20.0-incubating-candidate-2
heron.build.git.status : Clean
heron.build.host : tw-mbp-nlu
heron.build.time : Wed Aug 22 17:33:58 PDT 2018
heron.build.timestamp : 1534984438000
heron.build.user : nlu
heron.build.version : incubator-heron-v-0.20.0-incubating-candidate-2

@nwangtw Please take a look.

repo: apache/incubator-heron


text:
publish all 1.0.0 images to docker hub #1502

Description
No response
Use case
No response
Are you willing to submit PR?

[X] Yes, I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

docker images with 1.0.0-incubating tag are pushed to https://hub.docker.com/repository/docker/inlong/dashboard

repo: apache/incubator-inlong


text:
[KYUUBI #2569]Change the acquisition method of flinkHome to keep it consistent with other engines

Why are the changes needed?

fix #2569
How was this patch tested?

[x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

Codecov Report

Merging #2570 (1523d43) into master (1932ad7) will decrease coverage by 0.13%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master    #2570      +/-   ##
============================================
- Coverage     63.86%   63.73%   -0.14%
  Complexity       73       73
============================================
  Files           376      377       +1
  Lines         17956    18098     +142
  Branches       2405     2443      +38
============================================
+ Hits          11468    11534      +66
- Misses         5415     5466      +51
- Partials       1073     1098      +25

Impacted Files
Coverage Δ

...ache/kyuubi/engine/flink/FlinkProcessBuilder.scala
95.12% <100.00%> (+3.89%)
:arrow_up:

...apache/kyuubi/engine/hive/HiveProcessBuilder.scala
84.61% <100.00%> (ø)

...rg/apache/kyuubi/engine/trino/TrinoStatement.scala
66.66% <0.00%> (-2.30%)
:arrow_down:

.../engine/spark/session/SparkSQLSessionManager.scala
79.76% <0.00%> (-1.20%)
:arrow_down:

.../apache/kyuubi/server/api/v1/BatchesResource.scala
90.19% <0.00%> (ø)

...apache/kyuubi/server/api/v1/SessionsResource.scala
71.21% <0.00%> (ø)

...ache/kyuubi/server/api/v1/OperationsResource.scala
67.46% <0.00%> (ø)

...kyuubi/server/api/SwaggerScalaModelConverter.scala
51.26% <0.00%> (ø)

...ache/kyuubi/operation/KyuubiOperationManager.scala
85.24% <0.00%> (+1.63%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1932ad7...1523d43. Read the comment docs.

authors: codecov-commenter jiaoqingbo

repo: apache/incubator-kyuubi


text:
[KYUUBI #3842] [Improvement] Support maven pom.xml code style check with spotless plugin

Why are the changes needed?

Introduce code style check support for Maven's pom.xml with sortPom in spotless maven plugin.
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

The key change in the parent pom.xml, by adding <sortPom/> with in <pom/> element configs under spotless-maven-plugin plugin settings.

Codecov Report

Merging #3843 (3c65459) into master (a482c49) will decrease coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #3843      +/-   ##
============================================
- Coverage     52.96%   52.93%   -0.03%
  Complexity       13       13
============================================
  Files           497      497
  Lines         28023    28023
  Branches       3864     3864
============================================
- Hits          14841    14834       -7
- Misses        11780    11785       +5
- Partials       1402     1404       +2

Impacted Files
Coverage Δ

...ache/kyuubi/operation/KyuubiOperationManager.scala
79.45% <0.00%> (-2.74%)
:arrow_down:

...pache/kyuubi/engine/YarnApplicationOperation.scala
62.96% <0.00%> (-1.86%)
:arrow_down:

...rg/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala
63.93% <0.00%> (-1.64%)
:arrow_down:

...mon/src/main/scala/org/apache/kyuubi/Logging.scala
52.63% <0.00%> (-1.32%)
:arrow_down:

...yuubi/server/metadata/jdbc/JDBCMetadataStore.scala
89.27% <0.00%> (-0.70%)
:arrow_down:

...g/apache/kyuubi/operation/BatchJobSubmission.scala
74.64% <0.00%> (ø)

...apache/kyuubi/engine/JpsApplicationOperation.scala
77.41% <0.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Thanks, merging to master


text:
METRON-230: Bro parser should throw exception

Right now, if an invalid message comes to the bro parser, it returns null, which is interpreted as no messages from the parser.  INstead, we should throw an exception so the message can be routed to the error queue.

+1 works great in quick-dev-vagrant.

👍

authors: cestella dlyle65535 nickwallen

repo: apache/incubator-metron


text:
[MXNET-913] Java API -- NDArray Macros remove Option field

Description
The Plan 7 discussed in the mxnet Scala Group
@nswamy @andrewfayres @yzhliu @zachgk
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[x] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[x] To my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Thanks for your contribution @lanking520
@mxnet-label-bot[pr-awaiting-review]

Looked through this and everything looks good. In the morning I'll clone this and build so that I can take a peek at the generated code.

Stop the experiment and ship with this solution https://github.com/apache/incubator-mxnet/pull/12772

repo: apache/incubator-mxnet


text:
Backport CI fixes

Backport https://github.com/apache/incubator-mxnet/pull/17962

Maybe not required. Let's see if http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwindows-gpu/detail/v1.x/5/pipeline passes

It's not required to backport right now (old version still passes).
Let's reconsider backporting if we run into any Windows toolchain issues.


text:
Updating license for dmlc-core for v1.8 release

Description
update dmlc-core's LICENSE #19427
Checklist
Essentials

[x] Changes are complete (i.e. I finished coding on this PR)

Thank you.

Items that changed in dmlc-core:
dmlc-core/include/dmlc/any.h: Copyright (c) 2016 by Contributors
dmlc-core/include/dmlc/build_config_default.h: Copyright (c) 2018 by Contributors
dmlc-core/include/dmlc/endian.h: Copyright (c) 2017 by Contributors
dmlc-core/include/dmlc/strtonum.h: Copyright (c) 2015-2018 by Contributors

Close this PR as it was covered in https://github.com/apache/incubator-mxnet/pull/19440


text:
L2CAP LE COC signaling part plus some additional fixes

In this pull LE COC signaling part has been implemented together with extensions to Bletiny app needed to test it. It is tested against BlueZ and PTS and works fine.
Note: This pull will not enable LE COC,  BLE_L2CAP_COC_MAX_NUM is 0
As you will notice I change a way of creating and sending signal commands. It is now done similar to what BLE_SM does. I plan to do same for l2cap update command.
Please have a look on API as I change it a bit comparing to what we discussed on mailing list. I found it more useful when writing Bletiny extensions.
In addition in this pull you can find some fixes and cleaning.

ah...just run newt test @apache-mynewt-core/net/nimble/host/test - so yes, I need to fix some test cases

Tests fixed

repo: apache/incubator-mynewt-core


text:
apps: Fix the bug that setsockopt didn't check return value

Signed-off-by: weizihan weizihan@xiaomi.com
Summary
netutils/webclient/webclient.c
Fix the bug that setsockopt didn't check return value
Impact
webclient.c check return value one by one
Testing
passed vela CI

@Weizihan please fix the warning:
https://github.com/apache/incubator-nuttx-apps/runs/7855724603?check_suite_focus=true

repo: apache/incubator-nuttx-apps


text:
Implement the fine-grained spin lock

Right now, NuttX support one big spin lock:
irqstate_t spin_lock_irqsave(void);
void spin_unlock_irqrestore(irqstate_t flags);

Which isn't good for SMP machine. Compare with Linux(https://elixir.bootlin.com/linux/latest/source/include/linux/irqflags.h#L200), the above API should rename to:
irqstate_t local_irq_irqsave(void);
void local_irqrestore(irqstate_t flags);

and add the real spinlock(multiple instance) support:
https://www.kernel.org/doc/Documentation/locking/spinlocks.txt

Right now, NuttX support one big spin lock:

@xiaoxiang781216
NuttX has several spinlocks (e.g., g_cpu_irqlock, g_cpu_sched_lock, ... ) inside the kernel.
However, yes, the current  spin_lock_irqsave() and spin_unlock_irqrestore() uses a global spinlock named g_irq_spin. The reason why I introduced these APIs instead of multiple spinlock instance is to improve performance just by replacing some critical section APIs where we do not need to block. (e.g. just protecting register access). Also the APIs are compatible for non-SMP kernel.

the above API should rename to:
irqstate_t local_irq_irqsave(void);
void local_irqrestore(irqstate_t flags);

In my understanding, these APIs are the same as the current up_irq_save() and up_irq_restore() and also these APIs do not use a spinlock while spin_lock_irqsave() and spin_unlock_irqrestore() use a spinlock. So the behaviours are different.

and add the real spinlock(multiple instance) support:

I agree with you.

@masayuki2009 you are right. Let's introduce spinlock first and replace the usage step by step as note here: https://github.com/apache/incubator-nuttx/issues/1144.

After more thinking, we should support two type synchronization:

Multiple instance spinlock
Multiple instance critical section

The major difference is that the caller can't sleep afte hold the spinlock, but critical section can.

After more thinking, we should support two type synchronization:
Multiple instance spinlock

@xiaoxiang781216
I've just started this work by enhancing the existing APIs.
My idea is simple by just passing a spinlock
#if defined(CONFIG_SMP) && defined(CONFIG_SPINLOCK_IRQ)
-irqstate_t spin_lock_irqsave(void);
+irqstate_t spin_lock_irqsave(spinlock_t *lock);
 #else
-#  define spin_lock_irqsave() enter_critical_section()
+#  define spin_lock_irqsave(l) enter_critical_section()
 #endif

#if defined(CONFIG_SMP) && defined(CONFIG_SPINLOCK_IRQ)
-void spin_unlock_irqrestore(irqstate_t flags);
+void spin_unlock_irqrestore(spinlock_t *lock, irqstate_t flags);
 #else
-#  define spin_unlock_irqrestore(f) leave_critical_section(f)
+#  define spin_unlock_irqrestore(l, f) leave_critical_section(f)

Here, if the spinlock is NULL, it uses the global spinlock (i.e. g_irq_spin) for SMP.
And for a non-SMP case, it should be NULL as well.
For example, the imxrt does not support SMP, so imxrt_serial.c will have
flags  = spin_lock_irqsave(NULL);
...
spin_unlock_irqrestore(NULL, flags);

The behavior is the same as before. (i.e. It just disable/enable CPU interrupt for non-SMP case)
For SMP cases, you can also use the above call (i.e. spinlock is NULL). In this case, the behavior is also the same as before.  And this will be useful for migration. For the first step, I will do this by just replacing the existing API call with the NULL spinlock.
Next step, some drivers would have a local spinlock for SMP, for example, cxd56_serial.c  would be something like
--- a/arch/arm/src/cxd56xx/cxd56_serial.c
+++ b/arch/arm/src/cxd56xx/cxd56_serial.c
@@ -85,6 +85,9 @@ struct up_dev_s
   bool dtrdir;        /* DTR pin is the direction bit */
 #endif
   void *pmhandle;
+#ifdef CONFIG_SMP
+  spinlock_t lock;
+#endif
 };

...

   .oflow     = false, /* flow control is not supported */
 #endif
+#ifdef CONFIG_SMP
+  .lock      = SP_UNLOCKED,
+#endif
 };

static uart_dev_t g_uart2port =
@@ -288,7 +297,7 @@ static inline void up_disableuartint(FAR struct up_dev_s *priv,
 {
   irqstate_t flags;

-  flags = spin_lock_irqsave();
+  flags = spin_lock_irqsave(&priv->lock);
   if (ier)
     {
       *ier = priv->ier & UART_INTR_ALL;
@@ -296,7 +305,7 @@ static inline void up_disableuartint(FAR struct up_dev_s *priv,

   priv->ier &= ~UART_INTR_ALL;
   up_serialout(priv, CXD56_UART_IMSC, priv->ier);
-  spin_unlock_irqrestore(flags);
+  spin_unlock_irqrestore(&priv->lock, flags);
 }

In this case, each UART will have a spinlock but only for the SMP case. For the non-SMP case, the spinlock is not allocated but it's OK because the parameter (&priv->lock) will be ignored in the macro.
What do you think?

After more thinking, we should support two type synchronization:
Multiple instance spinlock

@xiaoxiang781216
I've just started this work by enhancing the existing APIs.
My idea is simple by just passing a spinlock
#if defined(CONFIG_SMP) && defined(CONFIG_SPINLOCK_IRQ)
-irqstate_t spin_lock_irqsave(void);
+irqstate_t spin_lock_irqsave(spinlock_t *lock);
 #else
-#  define spin_lock_irqsave() enter_critical_section()
+#  define spin_lock_irqsave(l) enter_critical_section()

It's better change to:
#  define spin_lock_irqsave(l) up_irq_save()

#endif
#if defined(CONFIG_SMP) && defined(CONFIG_SPINLOCK_IRQ)
-void spin_unlock_irqrestore(irqstate_t flags);
+void spin_unlock_irqrestore(spinlock_t *lock, irqstate_t flags);
#else
-#  define spin_unlock_irqrestore(f) leave_critical_section(f)
+#  define spin_unlock_irqrestore(l, f) leave_critical_section(f)

It's better change to:
#  define spin_unlock_irqrestore(l, f) up_irq_restore(f)

Here, if the spinlock is NULL, it uses the global spinlock (i.e. g_irq_spin) for SMP.
And for a non-SMP case, it should be NULL as well.
For example, the imxrt does not support SMP, so imxrt_serial.c will have
flags  = spin_lock_irqsave(NULL);
...
spin_unlock_irqrestore(NULL, flags);

The behavior is the same as before. (i.e. It just disable/enable CPU interrupt for non-SMP case)
For SMP cases, you can also use the above call (i.e. spinlock is NULL). In this case, the behavior is also the same as before. And this will be useful for migration. For the first step, I will do this by just replacing the existing API call with the NULL spinlock.

Yes, it is a good compromise.

Next step, some drivers would have a local spinlock for SMP, for example, cxd56_serial.c would be something like
--- a/arch/arm/src/cxd56xx/cxd56_serial.c
+++ b/arch/arm/src/cxd56xx/cxd56_serial.c
@@ -85,6 +85,9 @@ struct up_dev_s
   bool dtrdir;        /* DTR pin is the direction bit */
 #endif
   void *pmhandle;
+#ifdef CONFIG_SMP
+  spinlock_t lock;
+#endif
 };

...

   .oflow     = false, /* flow control is not supported */
 #endif
+#ifdef CONFIG_SMP
+  .lock      = SP_UNLOCKED,
+#endif
 };

Can we add some macros for definition and initialization? so we can avoid spread #ifded/#endif in the whole code base.

static uart_dev_t g_uart2port =
@@ -288,7 +297,7 @@ static inline void up_disableuartint(FAR struct up_dev_s *priv,
{
irqstate_t flags;

flags = spin_lock_irqsave();

flags = spin_lock_irqsave(&priv->lock);
if (ier)
{
*ier = priv->ier & UART_INTR_ALL;
@@ -296,7 +305,7 @@ static inline void up_disableuartint(FAR struct up_dev_s *priv,
priv->ier &= ~UART_INTR_ALL;
up_serialout(priv, CXD56_UART_IMSC, priv->ier);

spin_unlock_irqrestore(flags);

spin_unlock_irqrestore(&priv->lock, flags);
}

In this case, each UART will have a spinlock but only for the SMP case. For the non-SMP case, the spinlock is not allocated but it's OK because the parameter `(&priv->lock)` will be ignored in the macro.

What do you think?

The propose looks great!

@xiaoxiang781216
Thanks for your comments.

It's better change to:
#  define spin_lock_irqsave(l) up_irq_save()

If we apply the above change, the behavior will be different in the case of CONFIG_SMP=y and CONFIG_SPINLOCK_IRQ=n.
#if defined(CONFIG_SMP) && defined(CONFIG_SPINLOCK_IRQ)
-irqstate_t spin_lock_irqsave(void);
+irqstate_t spin_lock_irqsave(spinlock_t *lock);
 #else
-#  define spin_lock_irqsave() enter_critical_section()
+# define spin_lock_irqsave(l) up_irq_save()

However, I think it's better to remove CONFIG_SPINLOCK_IRQ from Kconfig to avoid complexity.
In this case, finally, we get
#if defined(CONFIG_SMP)
-irqstate_t spin_lock_irqsave(void);
+irqstate_t spin_lock_irqsave(spinlock_t *lock);
 #else
-#  define spin_lock_irqsave() enter_critical_section()
+# define spin_lock_irqsave(l) up_irq_save()

What do you think?

Oh, I forgot SPINLOCK_IRQ config, but this option doesn't make sense for me, because the user can freely select:

spin_lock/spin_unlock to hold only spinlock
Use spin_lock_irqsave/spin_unlock_irqrestore to hold both spinlock and irq
Two set API is already flexible enough for all use case.
On the other hand, it most likely generate synchronization error if user disable SPINLOCK_IRQ in defconfig in the current code base.

@xiaoxiang781216

Oh, I forgot SPINLOCK_IRQ config, but this option doesn't make sense for me, because the user can freely select:

I wrote the code but I agree with you.
OK, I will create a separate PR to remove CONFIG_SPINLOCK_IRQ from Kconfig.

The fine grant spinlock is implemented, let's close it.

repo: apache/incubator-nuttx


text:
exit() and pthread_exit() limitations

The main thread of a task should be able to call pthread_exit(), of course, calling pthread_self() first.  Certainly this behavior is not prohibited by the specifications at OpenGroup.org and there are several posts that specifically claim that this is possible.  See for example:  https://programmer.help/blogs/the-use-of-pthread-exit-in-main-thread.html
In this case, the main thread of the process/task group will terminate, but the process/task group will persist and other threads will continue running.  I believe that calling pthread_exit() from the main thread will currently lead to an assertion or crash due to the different forms of the main and pthread TCBs and the fact that global data is retained in the main thread's stack.
Similarly, a pthread should be able to call exit() and terminate the entire process/  See for example https://www.informit.com/articles/article.aspx?p=2085690&seqNum=5.  In fact, this is often the case when exit() is called from a signal handler in a multi-threaded task because we never know for sure which thread the signal handler will run on.
Again, a crash would be likely due to the same reasoning.

The main thread of a task should be able to call pthread_exit(), of course, calling pthread_self() first. Certainly this behavior is not prohibited by the specifications at OpenGroup.org and there are several posts that specifically claim that this is possible. See for example: https://programmer.help/blogs/the-use-of-pthread-exit-in-main-thread.html
In this case, the main thread of the process/task group will terminate, but the process/task group will persist and other threads will continue running. I believe that calling pthread_exit() from the main thread will currently lead to an assertion or crash due to the different forms of the main and pthread TCBs

Yes, this is major reason why we can't handle main thread and pthread equally. The best fix could be unify two tcb struct into one and:

Move main thread specific data to task_info
Move pthread specific data to tls_info

and the fact that global data is retained in the main thread's stack.

It's fixed by patch: https://github.com/apache/incubator-nuttx/pull/3992

Similarly, a pthread should be able to call exit() and terminate the entire process/ See for example https://www.informit.com/articles/article.aspx?p=2085690&seqNum=5. In fact, this is often the case when exit() is called from a signal handler in a multi-threaded task because we never know for sure which thread the signal handler will run on.
Again, a crash would be likely due to the same reasoning.

Wouldn't you also need some kind of reference count on the task data so that it persists until the last thread exits?

The reference count is already there: group_s(and then task_info_s) get destroyed only after the last thread in the process exit.

Should this issue be closed?  Is there no longer an issue?

Here is a test case:
static void *thread_func(FAR void *parameter)
{
  /* Wait a bit then call exit */

  printf("Child: Started\n");
  sleep(1);

  printf("Child: exit()\n");
  exit(0);
}

int main(int argc, FAR char *argv[])
{
  pthread_t thread;
  int status;

  /* Start the thread instances */

  printf("main: Starting thread\n");
  status = pthread_create(&thread, NULL, thread_func, (pthread_addr_t)NULL);
  if (status != 0)
    {
      printf("ERROR in thread creation: %d\n", status);
    }

  /* And exit the main thread */

  printf("main: pthread_exit()\n");
  pthread_exit(NULL);
}

When I run this test, I get:
nsh> exit
main: Starting thread
Child: Started
main: pthread_exit()
nsh> Child: exit()

nsh>

So I conclude that there is no problem.  Sorry for the false alarm.  I will close this issue now.

I could like to add a new issue to unify task and thread tcb into one.


text:
build: Makefile should reference CONFIG_HOST_xxx

Summary
to unify the host detection logic.
Impact
build on cygwin and msys2
Testing

@btashton could you help review this PR?

Sure, I can also do some windows builds. Need to look at msys I have not built with it in some time.

Sure, I can also do some windows builds. Need to look at msys I have not built with it in some time.

@btashton the build script doesn't do the special action for msys, so the change should be work with msys.


text:
docs: Add README for oli

We should let users know:

What's oli is?
How to use oli?
How to help develop oli?

Hi @Xuanwo , I am interested in this issue, could you assign it to me?

and where should I place the README file,  is bin/oli/README.md  OK for that?

Hi @Xuanwo , I am interested in this issue, could you assign it to me?

Thanks!

Hi @STRRL, do you need assistance with this issue?

No I think, I would open a PR later.

No I think, I would open a PR later.

Thanks a lot!

authors: STRRL Xuanwo

repo: apache/incubator-opendal


text:
binding/python: export enabled scheme

https://github.com/apache/incubator-opendal/pull/3331

/assign

Let's hold off on taking action until we have specific use cases. Do you have any intended uses in mind?

Can't see an usage, let's close.

authors: Xuanwo Zheaoli


text:
WIP: adding API Gateway functionality

This is still in progress and way far from completion.
Closes #686

no longer applicable, closing it.

repo: apache/incubator-openwhisk-wskdeploy


text:
Enforce Pinot code style

Use maven plugin to check the code style and fail the build if the check does not pass

+1
Shall we have one PR to format all the code and adding the code style plugin?

repo: apache/incubator-pinot


text:
[ROCKETMQ-200]-Cluster name is always missing when fetch ClusterInfo from name server

https://issues.apache.org/jira/browse/ROCKETMQ-200

Coverage increased (+0.01%) to 37.899% when pulling c52edde359f170530f77a36d2f0c6df50f2de1a9 on Jaskey:ROCKETMQ-200-Cluster-missing into 1630f277b9c9f8e85c9bdb09a323d39e13e797e4 on apache:develop.

LGTM @zhouxinyu @vongosling

@zhouxinyu @lizhanhui  it is time to merge this PR now, could you please have a double check?

+1, thanks @Jaskey

authors: Jaskey coveralls dongeforever zhouxinyu

repo: apache/incubator-rocketmq


text:
[Refactor][seatunnel-core-flink] refactor flink-sql module

Purpose of this pull request
refactor flink sql module, ref https://github.com/apache/incubator-seatunnel/issues/1753 and dev mail for some context
Check list

[yes] Code changed are covered with tests, or it does not need tests for reason:
[yes] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[yes] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

Hi, @ruanwenjun , would you help review this?
Just as we discussed in the dev mailing list, I send the PR for step 1, namely "refactor start-seatunnel-sql.sh. Now start-seatunnel-flink.sh and start-seatunnel-spark.sh have been refactored, and the main logic has
been rewritten by java code. I think we can first keep them consistent."
Thanks,
Kelu

repo: apache/incubator-seatunnel


text:
SCB-938 Should check self presevation max ttl

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Format the pull request title like [SCB-XXX] Fixes bug in ApproximateQuantiles, where you replace SCB-XXX with the appropriate JIRA issue.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Run go build go test go fmt go vet to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Codecov Report

Merging #450 into master will decrease coverage by <.01%.
The diff coverage is 88.88%.

@@            Coverage Diff             @@
##           master     #450      +/-   ##
==========================================
- Coverage   59.77%   59.76%   -0.01%
==========================================
  Files         149      149
  Lines       13174    13176       +2
==========================================
  Hits         7875     7875
  Misses       4727     4727
- Partials      572      574       +2

Impacted Files
Coverage Δ

server/core/backend/common.go
100% <ø> (ø)
:arrow_up:

server/core/backend/defer_instance.go
86.02% <88.88%> (+0.3%)
:arrow_up:

server/plugin/infra/registry/etcd/tracing.go
72.72% <0%> (-9.1%)
:arrow_down:

server/broker/util.go
53.2% <0%> (-0.27%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 95013d4...65d86e3. Read the comment docs.

Coverage decreased (-0.1%) to 62.219% when pulling 65d86e315d988c296cdbfe79de675a61fb682518 on little-cui:bug into 95013d48d6c3db50c10f575a29d0b773fb9c4128 on apache:master.

authors: codecov-io coveralls little-cui


text:
4.0.0-RC2 ResultSetMetaData  data was lost

version 4.0.0-RC2
question
codes
org.apache.shardingsphere.shardingjdbc.jdbc.core.resultset.EncryptResultSetMetaData#getLogicColumn
org.apache.shardingsphere.shardingjdbc.jdbc.core.resultset.ShardingResultSetMetaData#getLogicColumn
 private String getLogicColumn(final String actualColumn) throws SQLException {
        for (Entry<String, String> entry : logicAndActualColumns.entrySet()) {
            if (entry.getValue().contains(actualColumn)) {
                return entry.getKey();
            }
        }
        throw new SQLException(String.format("Can not get logic column by %s.", actualColumn));
    }

entry.getValue().contains(actualColumn)

desc
When there are two fields in a table, one field is the content of another field, such as
address
certificate_address
It is possible to match the columnName of certificate_address when looking up the columnName of the field address again, which results in the loss of the address field in the result set.

I will try to solve the problem

How to understand one field is the content of another field ?

Closed for no response any more.

authors: kalvan2019 terrymanu tswstarplanet

repo: apache/incubator-shardingsphere


text:
Cannot find table rule and default data source with logic tables

Please, when only databases need to shard and tables need not to shard, how can I omit the following configuration,
such as: spring.shardingsphere.sharding.tables.tb_order.actual-data-nodes=
If it has to be configured, if there are hundreds of tables, does it need to be configured hundreds of times?

Configuration is a one-time job, and if needn't shard, you can config nothing for them as default tables.

if i config nothing,  when i query ,  the system will report an error：Cannot find table rule and default data source with logic table: 'tb_order'

You may try for inline expression, for example: $tb_order{0...99}

You may try for inline expression, for example: tb_order_$->{0...99}

it's not what i want.  i just want to shard databases,  but if there are many tables,  do i have to config all of these tables for inline expression?

not at all, as default tables, no need to config.

Sorry, I can't understand it. The situation is like this. I need to shard databases at present, and I don't need to shard tables. Now there are many tables with the sharding-column, which can be divided to the database. I need to configure actual-data-nodes for all of these tables to query. If you don't configure them, an error will be reported.  Can you tell me how to simplify the configuration？

You can config nothing for tables.
Those non-sharding tables will be configured as default tables automically.

spring.shardingsphere.datasource.names=tenant-meituan,tenant-taobao,tenant-jd,tenant-common
spring.shardingsphere.datasource.tenant-meituan.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.tenant-meituan.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.tenant-meituan.url=jdbc:mysql://localhost:3306/tenant_meituan
spring.shardingsphere.datasource.tenant-meituan.username=root
spring.shardingsphere.datasource.tenant-meituan.password=password
spring.shardingsphere.datasource.tenant-taobao.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.tenant-taobao.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.tenant-taobao.url=jdbc:mysql://localhost:3306/tenant_taobao
spring.shardingsphere.datasource.tenant-taobao.username=root
spring.shardingsphere.datasource.tenant-taobao.password=password
spring.shardingsphere.datasource.tenant-jd.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.tenant-jd.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.tenant-jd.url=jdbc:mysql://localhost:3306/tenant_jd
spring.shardingsphere.datasource.tenant-jd.username=root
spring.shardingsphere.datasource.tenant-jd.password=password
spring.shardingsphere.datasource.tenant-common.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.tenant-common.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.tenant-common.url=jdbc:mysql://localhost:3306/tenant_common
spring.shardingsphere.datasource.tenant-common.username=root
spring.shardingsphere.datasource.tenant-common.password=password
spring.shardingsphere.sharding.default-data-source-name=tenant-common
hundreds of tables
spring.shardingsphere.sharding.tables.tb_order.actual-data-nodes=tenant-meituan.tb_order,tenant-taobao.tb_order,tenant-jd.tb_order
spring.shardingsphere.sharding.tables.tb_order_item.actual-data-nodes=tenant-meituan.tb_order_item,tenant-taobao.tb_order_item,tenant-jd.tb_order_item
spring.shardingsphere.sharding.tables.tb_user_info.actual-data-nodes=tenant-meituan.tb_user_info,tenant-taobao.tb_user_info,tenant-jd.tb_user_info
......
spring.shardingsphere.sharding.tables.tb_sys_dict.actual-data-nodes=tenant-meituan.tb_sys_dict,tenant-taobao.tb_sys_dict,tenant-jd.tb_sys_dict
spring.shardingsphere.sharding.default-database-strategy.standard.sharding-column=merchant
spring.shardingsphere.sharding.default-database-strategy.standard.precise-algorithm-class-name=com.bsi.shardingsphere.demo.algorithm.DbShardingAlgorithm
spring.shardingsphere.props.sql.show=true
Above are all my configurations， there's no problem, but the configuration is troublesome，because I need to config actual-data-nodes for hundreds of tables.   Is there any other solution?

authors: Zhumingj kimmking terrymanu


text:
[STREAMPIPES-125] Add Redis as an optional external service

Hi there,
This PR will add Redis as an optional external service to be used in Kubernetes deployment.
Fixes: https://issues.apache.org/jira/browse/STREAMPIPES-125
Thanks,
Grainier.

Hi @grainier - thanks for the update on the k8s integration for Redis.

authors: grainier wipatrick

repo: apache/incubator-streampipes-installer


text:
fix: Enable ESLint rule default-props-match-prop-types

SUMMARY
Fixes a bug introduced in https://github.com/apache/incubator-superset/pull/10868 (reverted by https://github.com/apache/incubator-superset/pull/11050)
Cypress test was added to make sure that the tabs switch correctly.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

@graceguo-supercat It fixes a bug reported by you, can you please take a look?
@adam-stasiak There are some changes in dashboard tests that you're refactoring

Codecov Report

Merging #11058 into master will decrease coverage by 4.80%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #11058      +/-   ##
==========================================
- Coverage   65.79%   60.99%   -4.81%
==========================================
  Files         816      816
  Lines       38422    38402      -20
  Branches     3621     3621
==========================================
- Hits        25280    23422    -1858
- Misses      13034    14794    +1760
- Partials      108      186      +78

Flag
Coverage Δ

#cypress
?

#javascript
61.76% <ø> (-0.01%)
:arrow_down:

#python
60.53% <ø> (-0.88%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

superset-frontend/src/CRUD/Field.jsx
22.22% <ø> (ø)

...src/SqlLab/components/ExploreCtasResultsButton.jsx
7.14% <ø> (-6.20%)
:arrow_down:

...rontend/src/SqlLab/components/SqlEditorLeftBar.jsx
40.00% <ø> (-16.00%)
:arrow_down:

...end/src/SqlLab/components/TemplateParamsEditor.jsx
88.57% <ø> (-2.86%)
:arrow_down:

superset-frontend/src/components/Hotkeys.jsx
100.00% <ø> (ø)

...t-frontend/src/components/Select/OnPasteSelect.jsx
95.23% <ø> (ø)

...dashboard/components/ColorSchemeControlWrapper.jsx
81.25% <ø> (ø)

...tend/src/dashboard/components/DashboardBuilder.jsx
77.27% <ø> (-7.58%)
:arrow_down:

...ntend/src/dashboard/components/PropertiesModal.jsx
46.42% <ø> (-14.29%)
:arrow_down:

...t-frontend/src/dashboard/components/SliceAdder.jsx
80.59% <ø> (-16.42%)
:arrow_down:

... and 208 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d056e3d...7a6155f. Read the comment docs.

authors: codecov-commenter kgabryje

repo: apache/incubator-superset


text:
Fix #3612 - reverse sign in difference calculation

fixes https://github.com/apache/incubator-superset/issues/3612

Coverage remained the same at 70.091% when pulling cdd5374640cc9320e9e5922a452a922394d4afa1 on mistercrunch:fix_diff into 4a3c09187a2aff72fe3715bf543f625d5966e5a8 on apache:master.

Coverage remained the same at 70.091% when pulling cdd5374640cc9320e9e5922a452a922394d4afa1 on mistercrunch:fix_diff into 4a3c09187a2aff72fe3715bf543f625d5966e5a8 on apache:master.

Coverage remained the same at 70.091% when pulling cdd5374640cc9320e9e5922a452a922394d4afa1 on mistercrunch:fix_diff into 4a3c09187a2aff72fe3715bf543f625d5966e5a8 on apache:master.

lgtm

authors: coveralls michellethomas mistercrunch


text:
Assertion failure on sgx_tstd/hashbrown

Hi,
For some reason, I keep getting a panic from an assertion in sgx_tstd/hashbrown/src/raw/sse2.rs.
This is on the current master (c2698dc2685f8dcd9550086c62077bceff15ded0).
The error is the following:
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `8`,
 right: `0`', /<redacted>/sgx-sdk/sgx_tstd/hashbrown/src/raw/mod.rs:1388:9
fatal runtime error: failed to initiate panic, error 5

This seems to be checking some alignment, and, by the looks of it, it enforces that pointers should be aligned to a 16B boundary. However, AFAIK rust only requires 8B aligned pointers and I don't know how to force them to be 16B aligned.
Is there something I'm doing wrong?

Update: manually overriding the cfg_if to use the implementation in generic.rs instead of the sse2.rs seems to work.
However, looking at my /proc/cpuinfo file it seems that my machine has sse2 support.

repo: apache/incubator-teaclave-sgx-sdk


text:
YUNIKORN-545: JSON based API error responses

What is this PR for?
Return JSON based error responses for all REST API's.
What type of PR is it?
Feature
Todos

[ ] - Task

What is the Jira issue?
https://issues.apache.org/jira/browse/YUNIKORN-545
How should this be tested?
Screenshots (if appropriate)
Questions:

[ ] - The licenses files need update.
[ ] - There is breaking changes for older versions.
[ ] - It needs documentation.

@wilfred-s @yangwwei can you please take a look?

repo: apache/incubator-yunikorn-core


text:
Clone a  note

Adding ability to clone a note.
New button on top of note to clone it

Dialog asking for a name for the new note(clone)

The cloned note

LGTM!
Thanks

Tested and working nicely.
Current icon "icon-grid"

is not very intuitive to me.
What do you think "fa fa-copy"

?
Cloning notebook does not copy interpreter bindings. How about copying interpreter bindings, too?

+1 on copy icon and interpreter bindings

@Leemoonsoo The copy icon looks better :-).
@Leemoonsoo @felixcheung I have also added code to copy the interpreter bindings.

This is a much needed feature! Love it.

LGTM!

this feature sounds very good and helpful!  thanks for your work! Love it very much!
can I ask that, what's the schedule of this feature or is this work already finished?
when will this feature merge to master?

@zenglinxi0615  the feature is complete. U can pull my branch to get hands on.
As to when it will be merged , tagging @Leemoonsoo

Can we resolve the following issue.?
https://issues.apache.org/jira/browse/ZEPPELIN-126

@Karuppayya could tell me your jira id?

@Leemoonsoo  my id is karuppayya1990

@Karuppayya can not find your jira apache id karuppayya1990. Could you confirm?

Oops.. id is karup1990 @Leemoonsoo

authors: Karuppayya Leemoonsoo felixcheung jitenderaswani r-kamath zenglinxi0615

repo: apache/incubator-zeppelin


text:
[IOTDB-1396] Restructure logical query operator

The design doc is in https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181306231

Coverage decreased (-0.003%) to 67.915% when pulling 8ec9cdf709a549bd4d34a664f061f3e557306bcc on RequeryV2 into 16167373d924b4f457ce380144a2564be76b65b2 on master.

authors: Alima777 SteveYurongSu coveralls

repo: apache/iotdb


text:
JAMES-3405 Add Prometheus config

The guide part for ES are restored.
Added metric configuration with Prometheus + example.

I disagree with changes to docs/modules/servers/pages/distributed/run-docker.adoc
A proper 3.6.0 docker image set will be uploaded for the release, we should then remember to update the docker-compose then.

I like that you create :
docs/modules/servers/pages/distributed/operate/metrics_es.adoc
docs/modules/servers/pages/distributed/operate/metrics_prometheus.adoc

Then we should move all content of docs/modules/servers/pages/distributed/operate/metrics.adoc  linked to Prometheus to metrics_prometheus.adoc  and all linked to ES to metrics_es.adoc

repo: apache/james-project


text:
KAFKA-12375: fix concurrency issue in application shutdown

enforce rebalance is now used in a thread safe way
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

One unrelated test failure: kafka.server.ListOffsetsRequestTest.testResponseIncludesLeaderEpoch
Going to merge this now

Merged to trunk and cherrypicked to 2.8 cc @vvcephei

repo: apache/kafka


text:
Re-using callbackHandler for refreshing Kerberos TGT when keytab is not used

When keytab file is not used, and the necessary configuration data are provided by the SASL callback handler, the Kerberos TGT renewal fails because the code is not re-using the configured CallbackHandler in the re-login sequence.
The error is:
javax.security.auth.login.LoginException: No CallbackHandler available to garner authentication information from the user

The change preserves the instance of the CallbackHandler that was used to login into Kerberos and passes it to the LoginContext when TGT needs to be renewed.
The change is tested in DIT with live Kafka and AD KRB instances in our current project.

Some tests are failed, but apparently it has nothing to do with Kerberos:
testSendNonCompressedMessageWithCreateTime(String) > kafka.api.PlaintextProducerSendTest.testSendNonCompressedMessageWithCreateTime(String)[1] FAILED
[2023-01-05T21:14:03.931Z]     java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 2 larger than available brokers: 1.

@vladimirdyuzhev Thanks for the PR. Since this is a security-related change, can we create a JIRA with the details from the PR description and include the ticket in the PR title? Can we also add a test to the PR?

Created JIRA KAFKA-14681

authors: rajinisivaram vladimirdyuzhev


text:
KAFKA-14781: Downgrade MM2 log message severity when no ACL authorizer is configured on source broker

Jira
Instead of logging an ERROR-level message when topic syncing is enabled (which it is by default) and no ACL authorizer is configured on the source Kafka cluster, we first issue an INFO-level message that suggests to the user that they disable topic ACL syncing, and then for each subsequent time, issue a DEBUG-level message.
Some thoughts while implementing this:

The number of testing-only constructors for MirrorSourceConnector is getting a bit out of hand. We might consider replacing all of them with a package-private start(MirrorSourceConfig) method, which can help unify testing and non-testing code paths, as well as reduce the testing-only code that we have to tack on to non-testing code.
We could possibly expand on the test case introduced here to simulate an ACL authorizer being added to the source Kafka cluster during the lifetime of the connector. I've omitted this as I don't believe it's a very common case and the changes here do not seem likely to introduce any risk for it.

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@mimaison would you mind taking a look?

Thanks @mimaison!


text:
KAFKA-15238: Move DLQ reporter setup from the DistributedHerder's tick thread to the sink task thread

From https://issues.apache.org/jira/browse/KAFKA-15238:

When Kafka Connect is run in distributed mode - if a sink connector's task is restarted (via a worker's REST API), the following sequence of steps will occur (on the DistributedHerder's thread):

The existing sink task will be stopped (ref)
A new sink task will be started (ref)
As a part of the above step, a new WorkerSinkTask will be instantiated (ref)
The DLQ reporter (see KIP-298) for the sink task is also instantiated and configured as a part of this (ref)
The DLQ reporter setup involves two synchronous admin client calls to list topics and create the DLQ topic if it isn't already created (ref)

All of these are occurring synchronously on the herder's tick thread - in this portion here where external requests are run. If the admin client call in the DLQ reporter setup step blocks for some time (due to auth failures and retries or network issues or whatever other reason), this can cause the Connect worker to become non-functional (REST API requests will timeout) and even fall out of the Connect cluster and become a zombie (since the tick thread also drives group membership functions - see here, here).

This patch fixes the issue by moving the DLQ reporter setup from the DistributedHerder's tick thread to the sink task's thread itself (in the initializeAndStart phase). A Supplier is used to avoid passing a bunch of otherwise irrelevant parameters to the WorkerSinkTask (such as admin configs, producer configs etc.) and / or refactoring some of the Worker methods.
This way, any blocking in the admin client calls during DLQ setup will only affect the sink task for which the DLQ is being setup (by delaying its startup) instead of affecting the entire Connect worker.
In case the DLQ setup fails, the sink task's status will still be updated with the failure just like it would prior to this patch.
A small integration test is also added to verify the DLQ topic functionality.
Another thing to note is that no explicit timeout is being used for the blocking admin client calls which isn't ideal. We should probably add a reasonable timeout value (the default admin client timeout values are fairly large) for the overall listTopics + (potential) createTopics operation in either this patch or a separate one.

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Hi Chris, would you be able to take a look at this whenever you get a chance?

Thanks for the swift reviews Chris! And yeah, even I was surprised by how brief the changes ended up being 😄


text:
KAFKA-6301 changing regular expression in ops.html from '' to '.'

The documentation for section "Mirroring data between clusters" states the following:
Or you could mirror all topics using --whitelist ''
The regular expression should be '.' instead.
This fix makes the change directly to the ops.html file.

MirrorMaker is now deprecated and as this PR is stalled I'll close it out for now.

authors: mimaison waleedfateem


text:
KAFKA-7255: Fix timing issue with create/update in SimpleAclAuthorizer

ACL updates currently  get (currentAcls, currentVersion) for the resource from ZK and do a conditional update using (currentAcls+newAcl, currentVersion). This supports concurrent atomic updates if the resource path already exists in ZK. If the path doesn't exist, we currently do a conditional createOrUpdate using (newAcl, -1). But -1 has a special meaning in ZooKeeper for update operations - it means match any version. So two brokers adding acls using (newAcl1, -1) and (newAcl2, -1) will result in one broker creating the path and setting newAcl1, while the other broker can potentially update the path with (newAcl2, -1), losing newAcl1. The timing window is very small, but we have seen intermittent failures in SimpleAclAuthorizerTest.testHighConcurrencyModificationOfResourceAcls as a result of this window.
This PR fixes the version used for conditional updates in ZooKeeper. It also replaces the confusing ZkVersion.NoVersion=-1 used for set(any-version) and get(return not-found) with ZkVersion.MatchAnyVersion for set(any-version) and ZkVersion.UnknownVersion for get(return not-found) to avoid the return value from get matching arbitrary values in set.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

LGTM.
Thanks for fixing this.  I tried to fix this in the past, but I was not able to get to the bottom of the issue.

@junrao @omkreddy @satishd Thanks for the reviews. Will merge to trunk and cherry-pick/backport to older branches.

authors: omkreddy rajinisivaram