text:
fix  #722

The branch of 0.2.0 is master-0.2.0. If your PR is to solve the problem of 0.2.0, please submit the PR to it.
Please do not create a Pull Request without creating an issue first.
What is the purpose of the change
#733
make service search table work.
Brief changelog
1.clean service route and add icon,show search service in menu.
2.add service page languages.
3.rebuild service page.
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[ ] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist.
[ ] Run `mvn clean compile --batch-mode -DskipTests=false -Dcheckstyle.skip=false -Drat.skip=false -Dmaven.javadoc.skip=true to make sure basic checks pass.

#722

repo: apache/dubbo-admin


text:
Provide ScopeModel Related Docs

FrameworkModel -> ApplicationModel -> ModuleModel，中间的映射都是 1:N 的
抽象这三个能力是为了实现 Dubbo 的多实例支持，FrameworkModel 是实现类似 JVM 租户级别的隔离，ApplicationModel 是为了实现一个机器上发布多个应用（如 demo-application1 和 demo-application2 一起发布），ModuleModel 是为了实现服务生命周期的独立管理（如一个 demo-application 可以由多个 Spring 容器共同提供）。
所以一个 Dubbo 服务是对应到 ModuleModel 上的。
举一个复杂一些的例子，比如:
目前需要发布 DemoInterface1、DemoInterface2、DemoInterface3、DemoInterface4 四个服务
订阅 DemoService1、DemoService2、DemoService3、DemoService4 四个服务
其中 DemoInterface1 和 DemoInterface2 由 demo-application-1 这个应用名发布（而且由 2 个 Spring Context 分别管理），DemoInterface3 和 DemoInterface4 demo-application-2 这个应用名发布 ，DemoService1、DemoService2、DemoService3、DemoService4 由 demo-application-3 这个应用名订阅。此外出于多租户的考虑，DemoInterface1、DemoInterface2、DemoInterface3、DemoInterface4 在 20880 端口和 20881 端口都独立发布，这 8 个服务（多租户 2 * 4 个服务）的实现都不一样。
那么实际上 Scope 的层级结构是：
FrameworkModel (1) -> ApplicationModel (1.1 demo-application-1) -> ModuleModel (1.1.1 Spring Context 1) -> DemoInterface1 (bind on 20880)
FrameworkModel (1) -> ApplicationModel (1.1 demo-application-1) -> ModuleModel (1.1.2 Spring Context 2) -> DemoInterface2 (bind on 20880)
FrameworkModel (1) -> ApplicationModel (1.1 demo-application-2) -> ModuleModel (1.2.1 Spring Context) -> DemoInterface3 (bind on 20880)
FrameworkModel (1) -> ApplicationModel (1.1 demo-application-2) -> ModuleModel (1.2.1 Spring Context) -> DemoInterface4 (bind on 20880)
FrameworkModel (1) -> ApplicationModel (1.1 demo-application-3) -> ModuleModel (1.3.1 Spring Context) -> DemoService1
FrameworkModel (1) -> ApplicationModel (1.1 demo-application-3) -> ModuleModel (1.3.1 Spring Context) -> DemoService2
FrameworkModel (1) -> ApplicationModel (1.1 demo-application-3) -> ModuleModel (1.3.1 Spring Context) -> DemoService3
FrameworkModel (1) -> ApplicationModel (1.1 demo-application-3) -> ModuleModel (1.3.1 Spring Context) -> DemoService4

FrameworkModel (2) -> ApplicationModel (2.1 demo-application-1) -> ModuleModel (2.1.1 Spring Context 1) -> DemoInterface1 (bind on 20881)
FrameworkModel (2) -> ApplicationModel (2.1 demo-application-1) -> ModuleModel (2.1.2 Spring Context 2) -> DemoInterface2 (bind on 20881)
FrameworkModel (2) -> ApplicationModel (2.1 demo-application-2) -> ModuleModel (2.2.1 Spring Context) -> DemoInterface3 (bind on 20881)
FrameworkModel (2) -> ApplicationModel (2.1 demo-application-2) -> ModuleModel (2.2.1 Spring Context) -> DemoInterface4 (bind on 20881)
FrameworkModel (2) -> ApplicationModel (2.1 demo-application-3) -> ModuleModel (2.3.1 Spring Context) -> DemoService1
FrameworkModel (2) -> ApplicationModel (2.1 demo-application-3) -> ModuleModel (2.3.1 Spring Context) -> DemoService2
FrameworkModel (2) -> ApplicationModel (2.1 demo-application-3) -> ModuleModel (2.3.1 Spring Context) -> DemoService3
FrameworkModel (2) -> ApplicationModel (2.1 demo-application-3) -> ModuleModel (2.3.1 Spring Context) -> DemoService4

上述所有服务独立工作，互相不影响
这一块需要出一个具体的文档来说明
Originally posted by @AlbumenJ in https://github.com/apache/dubbo/issues/10829#issuecomment-1291402547

Please assign me.

请问这块有文档了吗，感觉还是不好理解

请问这块有文档了吗，感觉还是不好理解
https://cn.dubbo.apache.org/zh-cn/blog/1/01/01/dubbo-3-中的三层配置隔离/
可以参考一下，后边会继续补充

source: gharchive/issue

authors: AlbumenJ Sakuragi27 namelessssssssssss

repo: apache/dubbo

license_type: permissive

license_source: github-api


text:
如何扩展元数据的自定义维度？

[x] I have searched the issues of this repository and believe that this is not a duplicate.
[x] I have checked the FAQ of this repository and believe that this is not a duplicate.

Environment

Dubbo version: 2.7.1
Operating System version: macos
Java version: 1.8

Steps to reproduce this issue
目前元数据上报至元数据中心的参数维度不能够满足我的业务，所以想扩展元数据来实现一些个性化参数的上报，以便于其他消费者或网关更好的使用一些特殊场景？

可以直接扩展MetadataReport扩展点，覆盖已有实现或提供新实现
public interface MetadataReport {

    void storeProviderMetadata(MetadataIdentifier providerMetadataIdentifier, FullServiceDefinition serviceDefinition);

    void storeConsumerMetadata(MetadataIdentifier consumerMetadataIdentifier, Map<String, String> serviceParameterMap);

}


text:
服务调用时,version="*"报错

spring-cloud-starter-dubbo 2.1.0.RELEASE

下面是错误代码，如果version="具体版本号"，则没有问题
@Reference(version = "*",check = false)
ProductService productService;
错误信息：
org.apache.dubbo.rpc.RpcException: Failed to invoke the method findById in the service com.oracle.tao.service.ProductService. No provider available for the service com.oracle.tao.service.ProductService:* from registry localhost:9090 on the consumer 10.191.9.71 using the dubbo version 2.7.3. Please check if the providers have been started and registered.
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.checkInvokers(AbstractClusterInvoker.java:271) ~[dubbo-2.7.3.jar:2.7.3]
at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:59) ~[dubbo-2.7.3.jar:2.7.3]
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:248) ~[dubbo-2.7.3.jar:2.7.3]
at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:78) ~[dubbo-2.7.3.jar:2.7.3]
at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:55) ~[dubbo-2.7.3.jar:2.7.3]
at org.apache.dubbo.common.bytecode.proxy0.findById(proxy0.java) ~[dubbo-2.7.3.jar:2.7.3]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_251]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_251]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_251]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_251]
at org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor$ReferenceBeanInvocationHandler.invoke(ReferenceAnnotationBeanPostProcessor.java:260) ~[dubbo-2.7.3.jar:2.7.3]
at com.sun.proxy.$Proxy121.findById(Unknown Source) ~[na:na]
at com.oracle.tao.controller.OrderController.getProductById(OrderController.java:18) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_251]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_251]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_251]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_251]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.29.jar:9.0.29]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.29.jar:9.0.29]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar:9.0.29]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar:9.0.29]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar:9.0.29]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:108) ~[spring-boot-actuator-2.2.2.RELEASE.jar:2.2.2.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar:9.0.29]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.2.RELEASE.jar:5.2.2.RELEASE]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) [tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367) [tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860) [tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1591) [tomcat-embed-core-9.0.29.jar:9.0.29]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.29.jar:9.0.29]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_251]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.29.jar:9.0.29]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_251]

version=''是不是的是不是认为版本号在比较了？

    public static boolean isMatch(URL consumerUrl, URL providerUrl) {
        String consumerInterface = consumerUrl.getServiceInterface();
        String providerInterface = providerUrl.getServiceInterface();
        //FIXME accept providerUrl with '*' as interface name, after carefully thought about all possible scenarios I think it's ok to add this condition.
        if (!(ANY_VALUE.equals(consumerInterface)
                || ANY_VALUE.equals(providerInterface)
                || StringUtils.isEquals(consumerInterface, providerInterface))) {
            return false;
        }

        if (!isMatchCategory(providerUrl.getParameter(CATEGORY_KEY, DEFAULT_CATEGORY),
                consumerUrl.getParameter(CATEGORY_KEY, DEFAULT_CATEGORY))) {
            return false;
        }
        if (!providerUrl.getParameter(ENABLED_KEY, true)
                && !ANY_VALUE.equals(consumerUrl.getParameter(ENABLED_KEY))) {
            return false;
        }

        String consumerGroup = consumerUrl.getParameter(GROUP_KEY);
        String consumerVersion = consumerUrl.getParameter(VERSION_KEY);
        String consumerClassifier = consumerUrl.getParameter(CLASSIFIER_KEY, ANY_VALUE);

        String providerGroup = providerUrl.getParameter(GROUP_KEY);
        String providerVersion = providerUrl.getParameter(VERSION_KEY);
        String providerClassifier = providerUrl.getParameter(CLASSIFIER_KEY, ANY_VALUE);
        return (ANY_VALUE.equals(consumerGroup) || StringUtils.isEquals(consumerGroup, providerGroup) || StringUtils.isContains(consumerGroup, providerGroup))
                && (ANY_VALUE.equals(consumerVersion) || StringUtils.isEquals(consumerVersion, providerVersion))
                && (consumerClassifier == null || ANY_VALUE.equals(consumerClassifier) || StringUtils.isEquals(consumerClassifier, providerClassifier));
    }

String ANY_VALUE = "*";
consumer he provider match rule like this, compare the element with (group、version、classifer)，if the problem occur，i think should think about the bootstarter


text:
[Dubbo-4556] fix dubbo startup is slow

What is the purpose of the change
fix #4556
Brief changelog
modify:dubbo/dubbo-config/dubbo-config-api/src/main/java/org/apache/dubbo/config/ServiceConfig.java

Codecov Report

Merging #4601 into master will increase coverage by <.01%.
The diff coverage is 58.84%.

@@             Coverage Diff              @@
##             master    #4601      +/-   ##
============================================
+ Coverage     64.11%   64.11%   +<.01%
+ Complexity      452      451       -1
============================================
  Files           768      768
  Lines         32874    32891      +17
  Branches       5180     5183       +3
============================================
+ Hits          21077    21089      +12
- Misses         9379     9381       +2
- Partials       2418     2421       +3

Impacted Files
Coverage Δ
Complexity Δ

...in/java/org/apache/dubbo/config/ServiceConfig.java
58.84% <58.84%> (+0.42%)
0 <0> (ø)
:arrow_down:

...che/dubbo/remoting/transport/mina/MinaChannel.java
43.42% <0%> (-10.53%)
16% <0%> (-1%)

.../dubbo/remoting/transport/netty4/NettyChannel.java
61.17% <0%> (-4.71%)
0% <0%> (ø)

...bo/rpc/cluster/support/FailbackClusterInvoker.java
67.21% <0%> (-3.28%)
0% <0%> (ø)

...onfig/spring/extension/SpringExtensionFactory.java
78.04% <0%> (-2.44%)
0% <0%> (ø)

...g/apache/dubbo/registry/consul/ConsulRegistry.java
61.87% <0%> (-0.63%)
29% <0%> (ø)

...pache/dubbo/registry/support/AbstractRegistry.java
80.07% <0%> (+1.53%)
0% <0%> (ø)
:arrow_down:

.../apache/dubbo/remoting/transport/AbstractPeer.java
67.39% <0%> (+4.34%)
0% <0%> (ø)
:arrow_down:

...ng/exchange/support/header/HeartbeatTimerTask.java
78.94% <0%> (+5.26%)
0% <0%> (ø)
:arrow_down:

.../remoting/transport/netty4/NettyClientHandler.java
68.42% <0%> (+7.01%)
0% <0%> (ø)
:arrow_down:

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8a50b2f...a2d3d92. Read the comment docs.

Wrong newline character, resubmit later.

source: gharchive/pull-request

authors: codecov-io crossoverJie


text:
draggable labelLayout option don't work with textBorderWidth

Version
5.0.2
Reproduction link
https://jsfiddle.net/x7qg3jpL/
Steps to reproduce

set label  textBorderWidth property to a gratter value then 0.
set labelLayout  draggable option to true.

What is expected?
That i can change label position with drag and drop.
What is actually happening?
I can't change label position with drag and drop..

解决了吗？

textBorderWidth should be a numeric (not character) value, like textBorderWidth:2

authors: WedersonCD cyyjs helgasoft

repo: apache/echarts


text:
[Feature] Add new positions to series-bar.markLine.data.0.label. position

What problem does this feature solve?
When markLines exist very close together on yAxis, it is hard to read.

I have tried offsetting vertically with distance and padding but both don't seem to move the label in the vertical axis.
What does the proposed API look like?
Along with  label positions: 'start', 'middle', 'end', 'insideStartTop', 'insideStartBottom', 'insideMiddleTop', 'insideMiddleBottom', 'insideEndTop', 'insideEndBottom', to add 'endTop' and 'endBottom'.
option = {
  xAxis: {
    type: 'category',
    data: ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']
  },
  yAxis: {
    type: 'value'
  },
  series: [
    {
      data: [150, 230, 224, 218, 135, 147, 260],
      type: 'line',
            markLine: {
        data: [
             {
            type: 'test',
            name: 'test',
            yAxis: 197,
            label: {
              backgroundColor: 'pink',
              distance: [20, 300],
              padding: [50,0,0,0]
            }
          },
          {
            type: 'average',
            name: 'Avg',
          }
        ]
      }
    }
  ]
};

Use offset instead of distance to manually position the label.


text:
Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.doris.flink.cfg.DorisOptions to field org.apache.flink.cdc.connectors.doris.sink.DorisMetadataApplier.dorisOptions of type org.apache.doris.flink.cfg.DorisOptions in instance of org.apache.flink.cdc.connectors.doris.sink.DorisMetadataApplier

Version:
mysql:8.0.31
doris-2.1.3-rc09
flink-1.19.0-bin-scala_2.12.tgz
flink-cdc-3.1.0-bin.tar.gz
flink-cdc-pipeline-connector-mysql-3.1.0.jar
flink-cdc-pipeline-connector-doris-3.1.0.jar
mysql-connector-java-8.0.27.jar

Doris 3FE and 3BE
mysql-to-doris.yaml
################################################################################
# Description: Sync MySQL all tables to Doris
################################################################################
source:
  type: mysql
  hostname: localhost
  port: 3305
  username: root
  password: 123456
  tables: doris-test.\.*
  server-id: 5400-5404
  server-time-zone: UTC

sink:
  type: doris
  fenodes: localhost:8031
  username: root
  password: ""
  table.create.properties.light_schema_change: true
  table.create.properties.replication_num: 1

pipeline:
  name: Sync MySQL Database to Doris
  parallelism: 2

./bin/flink-cdc.sh
./bin/flink-cdc.sh mysql-to-doris.yaml
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/brin/work/tools/doris_build/doris-1fe1be/tools/flink-1.19.0/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/brin/data/doris-runtime/flink/flink-1.19.0/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "main" org.apache.flink.util.FlinkException: Failed to execute job 'Sync MySQL Database to Doris'.
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2455)
	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2421)
	at org.apache.flink.cdc.composer.flink.FlinkPipelineExecution.execute(FlinkPipelineExecution.java:43)
	at org.apache.flink.cdc.cli.CliExecutor.run(CliExecutor.java:74)
	at org.apache.flink.cdc.cli.CliFrontend.main(CliFrontend.java:71)
Caused by: java.lang.RuntimeException: org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster.
	at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
	at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:75)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
	at java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:483)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1491)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:2073)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2035)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)
Caused by: org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster.
	at org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1570)
Caused by: java.util.concurrent.CompletionException: java.lang.RuntimeException: org.apache.flink.runtime.JobException: Cannot instantiate the coordinator for operator Source: Flink CDC Event Source: mysql -> SchemaOperator -> PrePartition
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770)
	... 3 more
Caused by: java.lang.RuntimeException: org.apache.flink.runtime.JobException: Cannot instantiate the coordinator for operator Source: Flink CDC Event Source: mysql -> SchemaOperator -> PrePartition
	at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
	at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
	... 3 more
Caused by: org.apache.flink.runtime.JobException: Cannot instantiate the coordinator for operator Source: Flink CDC Event Source: mysql -> SchemaOperator -> PrePartition
	at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:182)
	at org.apache.flink.runtime.executiongraph.ExecutionJobVertex$Factory.createExecutionJobVertex(ExecutionJobVertex.java:671)
	at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.attachJobVertices(DefaultExecutionGraph.java:860)
	at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.attachJobGraph(DefaultExecutionGraph.java:833)
	at org.apache.flink.runtime.executiongraph.DefaultExecutionGraphBuilder.buildGraph(DefaultExecutionGraphBuilder.java:221)
	at org.apache.flink.runtime.scheduler.DefaultExecutionGraphFactory.createAndRestoreExecutionGraph(DefaultExecutionGraphFactory.java:173)
	at org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:381)
	at org.apache.flink.runtime.scheduler.SchedulerBase.<init>(SchedulerBase.java:224)
	at org.apache.flink.runtime.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:140)
	at org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:162)
	at org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:121)
	at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:379)
	at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:356)
	at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
	at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
	at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
	... 4 more
Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.doris.flink.cfg.DorisOptions to field org.apache.flink.cdc.connectors.doris.sink.DorisMetadataApplier.dorisOptions of type org.apache.doris.flink.cfg.DorisOptions in instance of org.apache.flink.cdc.connectors.doris.sink.DorisMetadataApplier
	at java.base/java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2102)
	at java.base/java.io.ObjectStreamClass$FieldReflector.checkObjectFieldValueTypes(ObjectStreamClass.java:2066)
	at java.base/java.io.ObjectStreamClass.checkObjFieldValueTypes(ObjectStreamClass.java:1355)
	at java.base/java.io.ObjectInputStream$FieldValues.defaultCheckFieldValues(ObjectInputStream.java:2682)
	at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2483)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2269)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1747)
	at java.base/java.io.ObjectInputStream$FieldValues.<init>(ObjectInputStream.java:2603)
	at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2454)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2269)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1747)
	at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:525)
	at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:483)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:539)
	at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:527)
	at org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
	at org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.create(OperatorCoordinatorHolder.java:496)
	at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.createOperatorCoordinatorHolder(ExecutionJobVertex.java:294)
	at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:173)
	... 19 more

As required by Apache Flink, please report bugs or new features on Apache Jira
under the project Flink using component tag Flink CDC. You must have a JIRA account in order to log cases and issues.
If you don’t have an ASF JIRA account, you can request one at the ASF Self-serve portal,
account creation requires review by the PMC member of the application project, which normally takes one to two working days to be approved.

did this question figure out,I meet the same bug.but I use the version of flink is 1.18

在dinky使用的Cdc版本不一样 导致的
---Original---
From: @.>
Date: Thu, Nov 21, 2024 09:40 AM
To: @.>;
Cc: @.@.>;
Subject: Re: [apache/flink-cdc] Caused by: java.lang.ClassCastException:cannot assign instance of org.apache.doris.flink.cfg.DorisOptions to fieldorg.apache.flink.cdc.connectors.doris.sink.DorisMetadataApplier.dorisOptionsof type org.apache.doris.flink.cfg.DorisOptions in instance oforg.apache.flink.cdc.connectors.doris.sink.DorisMetadataApplier (Issue #3406)
遇到同一个问题了
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.***>

如何解决的 为什么issue closed

dinky 1.2.0
flink 1.18
flink cdc 3.2.1/3.2.0
碰到同样问题。这个issue连回答都懒得回答就直接关闭？这就是kpi吗，太秀了。。

Just a reminder for all the guys who encounter the same problem.
The pipeline-doris jar package of version 3.2.x is a fat jar, which has shaded the classes of flink-doris-connector. Please delete all the files under org/apache/doris in the pipeline-doris fat jar and repackage it, and then put it into the dependency directory of Dinky.

Please refer https://github.com/apache/flink-cdc/issues/3814 for more information

repo: apache/flink-cdc


text:
[FLINK-18844][json][maxwell] Support maxwell-json format to read Maxwell changelogs

What is the purpose of the change
support maxwell-json for Maxwell binlog collector
Brief change log

add DeserializationSchema for deserialize byte[]
add SerializationSchema for serialize data to byte[]
add formatfactory for DeserializationSchema&SerializationSchema

Verifying this change
This change added tests and can be verified as follows:

Added java tests for deserialize byte[] from table source & serialize data to byte[] into table sink

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): ( no)
The public API, i.e., is any changed class annotated with @Public(Evolving): ( no)
The serializers: ( no )
The runtime per-record code paths (performance sensitive): ( no )
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: ( no )
The S3 file system connector: ( no)

Documentation

Does this pull request introduce a new feature? (yes )
If yes, how is the feature documented? ( docs / JavaDocs )

@flinkbot run azure

@flinkbot run travis

Besides, it would be great if you can add the corresponding documentation in this PR.
But create a following up JIRA issue is also fine to me.

Besides, it would be great if you can add the corresponding documentation in this PR.
But create a following up JIRA issue is also fine to me.

i will finish these bugs soon.
Documentation will be add next PR .

@flinkbot run azure

CI report:

d2a7bde Azure: FAILURE

Bot commands

CI report:

d2a7bde Azure: FAILURE

Bot commands

@flinkbot  run azure

Build is passed.
Merging...

authors: lonelyGhostisdog wuchong

repo: apache/flink


text:
[FLINK-2972] [JavaAPI] Remove Chill dependency from flink-java.

This PR replaces flink-java's chill dependency by chill-java to decouple flink-java from Scala.
Chill is used within the KryoSerializer to obtain a "Scala-preconfigured" Kryo instance. Hence, Chill is still required for Scala DataSet or DataStream programs. Separate KryoSerializers for Scala and Java are hard to provide, because GenericTypeInformation can be generated through the TypeExtractor, TypeParser, and the Scala compiler marko.
In this PR, the KryoSerializer tries to dynamically load Chill's ScalaKryoInstantiator and generate a Kryo instance through reflection to circumvent the Chill dependency. As a fallback, a regular Kryo instance is created. Consequently, Chill's Kryo instance is used whenever Chill is present in the classpath. To ensure that client, JM, and TM have Chill in their classpath, we add Chill as a dependency to flink-client.
Note: Before merging, we need to check if this works for Java and Scala programs which are submitted to a cluster (including some collection data sources which are serialized at the client).

Sorry, I linked the PR to the wrong JIRA issue.
Will close this PR and open a new one.


text:
[FLINK-20966][table-planner-blink] Rename Stream(/Batch)ExecIntermediateTableScan to Stream(/Batch)PhysicalIntermediateTableScan

What is the purpose of the change
Rename Stream(/Batch)ExecIntermediateTableScan to Stream(/Batch)PhysicalIntermediateTableScan
Brief change log

Rename StreamExecIntermediateTableScan to StreamPhysicalIntermediateTableScan
Rename BatchExecIntermediateTableScan to BatchPhysicalIntermediateTableScan

Verifying this change
This change is a refactoring rework covered by existing tests.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@flinkbot run azure

@flinkbot run azure

@flinkbot run azure

@flinkbot run azure


text:
[FLINK-21822][table] Support the new factory stack for CatalogFactory

What is the purpose of the change
Support the newer factory stack based on Factory for CatalogFactory. This still supports the old TableFactory stack as well, but deprecates it, for renewal in a later version.

[x] Migrate the built-in catalog factories (GenericInMemory, Hive, JDBC)
[x] Migrate catalogs used in tests (SimpleCatalogFactory, 2 * TestCatalogFactory, TestHiveCatalogFactory, TestClassLoaderCatalogFactory)
[x] Docs

Brief change log

CatalogFactory extends both TableFactory and Factory with default implementations for compatibility.
CatalogFactory#createCatalog(String, Map) is deprecated.

Verifying this change
This change added tests and can be verified as follows:

CatalogITCase has an existing test for a custom user-defined catalog factory using the legacy stack
CatalogITCase has been extended by another test using the new stack

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? docs (tbd) + JavaDocs

Thanks @twalthr. I've updated the PR and will continue with the remaining things to do as well.

I don't think the one remaining failure has anything to do with this PR:

This task has failed 49 times across 534 pipeline runs in the last 14 days.


text:
Flink 16504 dynamodb connector backpressure

What is the purpose of the change
(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)
Brief change log
(for example:)

The TaskInfo is stored in the blob store on job creation time as a persistent artifact
Deployments RPC transmits only the blob storage reference
TaskManagers retrieve the TaskInfo from the blob cache

Verifying this change
(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (100MB)
Extended integration test for recovery after master (JobManager) failure
Added test that validates that TaskInfo is transferred only once across recoveries
Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Created by mistake, dropping it.


text:
[FLINK-26864][metrics] Fix performance regression from mailbox latency measuement

What is the purpose of the change

Fix performance regression from mailbox latency measuement
Note: The performance implications, i.e., recovery of performance values, are documented in https://issues.apache.org/jira/browse/FLINK-26864. (Please review the performance results in the linked issue.)

Brief change log

Trigger mailbox latency measurement from MailboxProcessor class instead from StreamTask class. This way, measurement can be initiated selectively only when mails are executed which is, in particular, on any non-poison mail.
To resolve cyclic dependencies between MailboxProcessor which starts mailbox measurement with StreamTask which holds references to MailboxExecutor and TimerServices that are both needed for scheduling mailbox latency measurements an additional class MailboxMetricsController is introduced that holds the dependencies and provides control functions.

Verifying this change
Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing
This change is already covered by existing tests that are updated respectively. The relevant tests are:

StreamTaskTest#testMailboxMetricsScheduling
StreamTaskTest#testMailboxMetricsMeasurement
TaskMailboxProcessorTest#testRunDefaultActionAndMails

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): yes
Anything that affects deployment or recovery: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

@flinkbot run azure

@pnowojski There was a merge conflict. Therefore, I pushed an update and reran the ci check which succeeded.


text:
[FLINK-4115] FsStateBackend filesystem verification can cause classpath exceptions

There are two changes to FsStateBackend to avoid classpath exceptions when submitting a job from a Flink client which does not have the necessary file system classes on its classpath:

The filesystem is no longer initialised in the FsStateBackend constructor
The verification checks for the checkpoint directory in the FsStateBackend constructor are skipped if Flink does not have built-in support for the URI scheme

Thanks for the PR! I'll have a look and try and figure out if it's the right way to go.

Thanks for your work again! I merged it. 😃

Cool, thanks!

authors: aljoscha joshfg


text:
[FLINK-4431] [core] Introduce a "VisibleForTesting" annotation.

This annotations declares that a function, field, constructor, or entire type, is only visible for testing purposes.
This annotation is typically attached when for example a method should be {@code private} (because it is not intended to be called externally), but cannot be declared private, because some tests need to have access to it.
Guava offers a similar annotation, but I would like to not rely on Guava for this.

Makes sense +1

+1. Very useful one. Also I saw some of the interfaces though they are supposed to be Public it is not marked with those types. Is it better to fix all of them ? Or add a test case that ensures that any new class/existing class if it misses annotation either Public or Private or what ever could be a new type, then we should fail the build. We have such a procedure in our project. Sorry if this is offtopic. Just thought of it so saying here.

+1, when adding this we should probably replace all uses of the Guava version and check that we don't use it using checkstyle rules. (If our goal is to one day remove our dependency on Guava.)

@ramkrish86 Only some project are API projects and need annotations. Other internal projects have no stability annotations.

I would merge this and file for a followup issue to replace the Guava annotations and add a checkstyle rule.

@StephanEwen
Sorry for continuing in this thread. I did not meant to mean the StabilityAnnotations. I was only talking about other interfaces without annotations - Public/ Internal/PublicEvolving that flink is already using.
For eg - TimeStampAssigner and the other interfaces extending it. They don't have types annotated. I meant such interfaces.

@ramkrish86 Within the public API projects, all classes should have such annotations. We currently have no automated way of checking that. Would not hurt to have that, I agree.

Thanks @StephanEwen . I can work on such Util. Will get back to it.

authors: StephanEwen aljoscha ramkrish86 uce


text:
Fixed comment about scala versions.

What is the purpose of the change
(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)
Brief change log
(for example:)

The TaskInfo is stored in the blob store on job creation time as a persistent artifact
Deployments RPC transmits only the blob storage reference
TaskManagers retrieve the TaskInfo from the blob cache

Verifying this change
(Please pick either of the following options)
This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (100MB)
Extended integration test for recovery after master (JobManager) failure
Added test that validates that TaskInfo is transferred only once across recoveries
Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

This is incorrect as Flink 1.7 also builds with scala 2.12. The entire section nee

This is incorrect as Flink 1.7 also builds with scala 2.12.

This is incorrect as Flink 1.7 also builds with scala 2.12.
👍

@zentol PTAL.


text:
[scripts] cleanup release script and make it more robust

fail on command error
make Mac OS X compatible
fix uberjar file limit detection
cleanup

changes look good modulo inline comments

Thanks for the review.

Is there anything blocking this? If not, feel free to go ahead and merge this. :)

Thanks, merging then.

authors: mxm uce


text:
Create FluoOracle command

For #842, create command that runs Oracle if given connection.properties and the application name.
java org.apache.fluo.core.command.FluoOracle /path/to/connection.properties myAppName

Closed by #883

repo: apache/fluo


text:
Reopen HDDS-2034 Async RATIS pipeline creation and destroy through heartbeat commands

https://issues.apache.org/jira/browse/HDDS-2034

BTW, thank you very much @ChenSammi the re-creation of the patch. The build results look nice, full green.

"I think we should consider AllocatedPipelines during pipelineCount."
The allocated one means it is not reported from DN yet, which we don't know if it is in a health state.

"I think we should consider AllocatedPipelines during pipelineCount."
The allocated one means it is not reported from DN yet, which we don't know if it is in a health state.

If we don't consider them, then it will be always like a fresh installed cluster, as none will be in open state. As this count is got, during startup.(Where all pipelines in DB are set to initial state)
I think when data node is decommissioned, then we should remove those pipelines from SCM.

And also one more thing I have seen is in DeadnodeHandler, we fire the event to send close command to DN, and immediately removing pipeline from DB and calling trigger pipeline creation.
Now Datanode becomes stale means it might not be running. So, in this case below flow looks incorrect.
SCM PipelineManager.java:
private void destroyPipeline(Pipeline pipeline) throws IOException {
pipelineFactory.close(pipeline.getType(), pipeline);
// remove the pipeline from the pipeline manager
removePipeline(pipeline.getId());
triggerPipelineCreation();
}
DeadNodeHandler.java
pipelineManager.finalizeAndDestroyPipeline -> destoryPipeline.

So, in this case below flow looks incorrect or is this is fine?, as we are removing from DB, without removal.
I think we should be fine. Here is an example: one stale datanode releases two other datanodes of the destroyed pipeline. A new pipeline can be created if there are one of two extra datanodes available, they can be used to create a new pipeline.

If we don't consider them, then it will be always like a fresh installed cluster, as none will be in open state. As this count is got, during startup.(Where all pipelines in DB are set to initial state)

Good point. I somehow remember the DB always has the matching state update with the in-memory statemanager, which is not the case here. As a result, we will need to consider both ALLOCATED and OPEN pipeline here. Given that the in memory pipeline map always been initialized to ALLOCATED state at the time of the safemode rule is initialized, we could just check ALLOCATED state as well.

So, in this case below flow looks incorrect or is this is fine?, as we are removing from DB, without removal.

I think we should be fine. Here is an example: one stale datanode releases two other datanodes of the destroyed pipeline. A new pipeline can be created if there are one of two extra datanodes available, they can be used to create a new pipeline.

Yes, I agree. But without ratis group remove call, we are considering them to form a new pipeline.

So, in this case below flow looks incorrect or is this is fine?, as we are removing from DB, without removal.

I think we should be fine. Here is an example: one stale datanode releases two other datanodes of the destroyed pipeline. A new pipeline can be created if there are one of two extra datanodes available, they can be used to create a new pipeline.

Yes, I agree. But without ratis group remove call, we are considering them to form a new pipeline. Will this not be a problem?

So, in this case below flow looks incorrect or is this is fine?, as we are removing from DB, without removal.

I think we should be fine. Here is an example: one stale datanode releases two other datanodes of the destroyed pipeline. A new pipeline can be created if there are one of two extra datanodes available, they can be used to create a new pipeline.

Yes, I agree. But without ratis group remove call, we are considering them to form a new pipeline. Will this not be a problem?

Do you mean form a new pipeline with the datanodes which might should close the previous pipeline and remove the ratis group first, will it to be a problem?  Correct me if you don't mean this.
The answer to above question is No. It will not become a problem since the underneath Ratis library supports multi-raft.

So, in this case below flow looks incorrect or is this is fine?, as we are removing from DB, without removal.

I think we should be fine. Here is an example: one stale datanode releases two other datanodes of the destroyed pipeline. A new pipeline can be created if there are one of two extra datanodes available, they can be used to create a new pipeline.

Yes, I agree. But without ratis group remove call, we are considering them to form a new pipeline. Will this not be a problem?

Do you mean form a new pipeline with the datanodes which might should close the previous pipeline and remove the ratis group first, will it to be a problem? Correct me if you don't mean this.
The answer to above question is No. It will not become a problem since the underneath Ratis library supports multi-raft.

Do you mean form a new pipeline with the datanodes which might should close the previous pipeline and remove the ratis group first, will it to be a problem? Correct me if you don't mean this.

Yes this was my question. Thanks for the reply, then I think it should be okay here.

Hi @elek , the acceptance is cancelled.  Should I re-trigger the CI and make sure it succeeds?

/retest

@ChenSammi /retest doesn't work for Github Actions.  Checks can be re-run from the Checks page (for committers).
https://help.github.com/assets/images/help/repository/rerun-checks-drop-down.png

I will implement a /retest later, until that please push a new commit to have new results:
git commit --allow-empty -m "retrigger build" && git push

"Cancelled" means time out after 6 hours. On the master the acceptance tests can be executed until 1 hours, the timeout caused by the patch.
And actually this was the root cause of the revert. Without the patch it can work well.
It may be a problem with the new safe mode check?

It may be a problem with the new safe mode check?

Safe mode check looks good to me based on the code, and the last secure cluster passed the check:
2019-11-22T04:43:04.8986872Z SCM is out of safe mode.
2019-11-22T04:43:04.8987697Z Safe mode is off

But something is wrong with the execution of the secure test:
https://pipelines.actions.githubusercontent.com/O2JnqO9VD7g40zn5cnS8ZQ3t6IkBNUXf9dtdL8xzpLtWhEsOm2/_apis/pipelines/1/runs/279/signedlogcontent/44?urlExpires=2019-11-25T13%3A35%3A19.9767125Z&urlSigningMethod=HMACV1&urlSignature=Kzda9PFputac%2BHePoESDsVBJz8zyio8TIHOBvcMscIQ%3D

I tried to define a test timeout. Hopefully it makes possible to download real test log to identify the problem.

@elek ozonesecure acceptance test being stuck at Ozonefs test (and timing out after 6 hours) is happening intermittently in other PRs and post-commit builds, too.
https://github.com/apache/hadoop-ozone/pull/263/checks?check_run_id=316855596
https://github.com/apache/hadoop-ozone/runs/316658066

happening intermittently in other PRs and post-commit builds, too.

Thanks the info. But it seems to be reproducible with this patch. (it fails all the time). Can be connected with the async pipeline creation.
Will try to debug tomorrow with pushing more commits + adjust timeouts + save the log files even in case of timeouts...

This acceptance test failed.
1 datanode is up and healthy (until now)
3 datanodes are up and registered to the scm
SCM is out of safe mode.
Safe mode is off
ozone-basic :: Smoketest ozone cluster startup
Check webui static resources                                          | PASS |
Start freon testing                                                   | FAIL |
255 != 0
ozone-basic :: Smoketest ozone cluster startup                        | FAIL |
2 critical tests, 1 passed, 1 failed
2 tests total, 1 passed, 1 failed
Output:  /tmp/smoketest/ozone/result/robot-ozone-ozone-basic-scm.xml

Hi @elek,  can this be merged now?

Hi @elek, can this be merged now?

Definitely. I am 100% +1. Just waited one day for any potential feedback from us timezone.
(As I understood @bharatviswa504 an @xiaoyuyao were happy with the patch earlier, so I will commit this patch. The us holidays might help to find any instability on the mater).
Thank you very much the patient and the continues improvement. And sorry for this complicated merge. I owe you some quick code reviews for next patches ;-)
I am doing the last local compilation and will push it.

Thanks @elek @xiaoyuyao @bharatviswa504 for the code review, especially @elek for help narrow down the test failure issues.

authors: ChenSammi adoroszlai bharatviswa504 elek xiaoyuyao

repo: apache/hadoop-ozone


text:
HBASE-24045 Support setting -Dhadoop.profile in adhoc_run_tests.sh

I think this shouldn't break the associated jenkins job, but...

FYI other folks grinding on flakeys @saintstack @jojochuang @qiaoandxiang

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 26s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 35s
Maven dependency ordering for branch

_ Patch Compile Tests _

+0 :ok:
mvndep
0m  9s
Maven dependency ordering for patch

_ Other Tests _

1m 45s

Subsystem
Report/Notes

Docker
Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1347/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/1347

Optional Tests

uname
Linux 14183806daaa 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / eed730e492

Max. process+thread count
46 (vs. ulimit of 10000)

modules
C:  U:

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1347/1/console

versions
git=2.17.1 maven=2018-06-17T18:33:14Z)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 29s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 34s
Maven dependency ordering for branch

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 10s
Maven dependency ordering for patch

_ Other Tests _

1m 52s

Subsystem
Report/Notes

Docker
Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1347/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/1347

Optional Tests

uname
Linux cfb496bc2f86 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / eed730e492

Max. process+thread count
58 (vs. ulimit of 10000)

modules
C:  U:

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1347/1/console

versions
git=2.17.1 maven=2018-06-17T18:33:14Z)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 26s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+0 :ok:
shelldocs
0m  0s
Shelldocs was not available.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+0 :ok:
mvndep
0m 34s
Maven dependency ordering for branch

_ Patch Compile Tests _

+0 :ok:
mvndep
0m  9s
Maven dependency ordering for patch

+1 :green_heart:
shellcheck
0m  0s
There were no new shellcheck issues.

+1 :green_heart:
whitespace
0m  1s
The patch has no whitespace issues.

_ Other Tests _

+0 :ok:
asflicense
0m  0s
ASF License check generated no output?

1m 56s

Subsystem
Report/Notes

Docker
Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1347/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/1347

Optional Tests
dupname asflicense shellcheck shelldocs

uname
Linux 559dfaddc85f 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / eed730e492

Max. process+thread count
47 (vs. ulimit of 10000)

modules
C:  U:

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1347/1/console

versions
git=2.17.1 maven=2018-06-17T18:33:14Z) shellcheck=0.4.6

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

No shellcheck issues at all? I find that suspect...

authors: Apache-HBase ndimiduk

repo: apache/hbase


text:
HBASE-22033 Update to maven-javadoc-plugin 3.2.0 and switch to non-forking aggregate goals

needs a forward port of HBASE-19663 on master (previously only on branch-1)
updates our maven-javadoc-plugin to latest (3.2.0)
switches all of our javadocs reports to use the non-forking version.

using mvn -B install -DskipTests && mvn -B site -DskipTests; times on my local laptop below. I'm sure the impact would be greater in docker for OS X. these single run times has noise I'm sure.
before:
hbase busbey$ grep "Total time: " ~/Downloads/HBASE-22033.before.mvn.log
19:11:32 [INFO] Total time:  05:30 min
19:27:04 [INFO] Total time:  15:29 min

after:
busbey-mba13:hbase busbey$ grep "Total time: " ~/Downloads/HBASE-22033.after.mvn.log
18:53:29 [INFO] Total time:  06:22 min
19:02:50 [INFO] Total time:  09:19 min

checked using that website generation still works using bash ./dev-support/jenkins-scripts/generate-hbase-website.sh --local-repo ~/.m2/repository .

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 40s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

_ Patch Compile Tests _

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  1s
The patch has no ill-formed XML file.

+1 :green_heart:
hadoopcheck
11m 29s
Patch does not cause any errors with Hadoop 3.1.2 3.2.1.

_ Other Tests _

+1 :green_heart:
asflicense
0m 16s
The patch does not generate ASF License warnings.

19m 13s

Subsystem
Report/Notes

Docker
Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1796/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/1796

Optional Tests
dupname asflicense hadoopcheck xml

uname
Linux 7855355898cb 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 79d56499ca

Max. process+thread count
76 (vs. ulimit of 12500)

modules
C: . U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1796/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 13s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
4m 51s
master passed

+1 :green_heart:
compile
3m  2s
master passed

+1 :green_heart:
shadedjars
6m 27s
branch has no errors when building our shaded downstream artifacts.

-0 :warning:
javadoc
0m 16s
root in master failed.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
4m 29s
the patch passed

+1 :green_heart:
compile
3m  4s
the patch passed

+1 :green_heart:
javac
3m  4s
the patch passed

+1 :green_heart:
shadedjars
6m 24s
patch has no errors when building our shaded downstream artifacts.

-0 :warning:
javadoc
0m 15s
root in the patch failed.

_ Other Tests _

+1 :green_heart:
unit
264m 35s
root in the patch passed.

298m  5s

Subsystem
Report/Notes

Docker
Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1796/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/1796

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux b038e039bdd1 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 79d56499ca

Default Java
2020-01-14

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1796/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1796/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1796/1/testReport/

Max. process+thread count
4904 (vs. ulimit of 12500)

modules
C: . U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1796/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 37s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
3m 49s
master passed

+1 :green_heart:
compile
2m 15s
master passed

+1 :green_heart:
shadedjars
5m 32s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
2m  8s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 20s
the patch passed

+1 :green_heart:
compile
2m 24s
the patch passed

+1 :green_heart:
javac
2m 24s
the patch passed

+1 :green_heart:
shadedjars
5m 30s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
2m  1s
the patch passed

_ Other Tests _

-1 :x:
unit
318m 50s
root in the patch failed.

348m 44s

Subsystem
Report/Notes

Docker
Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1796/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/1796

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux e32a3ef57438 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 79d56499ca

Default Java
1.8.0_232

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1796/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1796/1/testReport/

Max. process+thread count
6740 (vs. ulimit of 12500)

modules
C: . U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1796/1/console

versions
git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

jdk8 unit failures are unrelated; patch doesn't change anything in tests or backup implementation.

thanks everyone!

authors: Apache-HBase busbey


text:
HBASE-27421 Bump spotless plugin to 2.27.2 and reimplement the 'Remov…

…e unhelpful javadoc stubs' rule

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
3m 14s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+1 :green_heart:
mvninstall
2m 24s
master passed

+1 :green_heart:
compile
6m  4s
master passed

+1 :green_heart:
spotless
0m 39s
branch has no errors when running spotless:check.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m  2s
the patch passed

+1 :green_heart:
compile
6m  3s
the patch passed

-0 :warning:
javac
6m  3s
root generated 7 new + 696 unchanged - 7 fixed = 703 total (was 703)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  0s
The patch has no ill-formed XML file.

+1 :green_heart:
hadoopcheck
8m  7s
Patch does not cause any errors with Hadoop 3.2.4 3.3.4.

+1 :green_heart:
spotless
0m 36s
patch has no errors when running spotless:check.

_ Other Tests _

+1 :green_heart:
asflicense
0m 13s
The patch does not generate ASF License warnings.

34m 31s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4824/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/4824

Optional Tests
dupname asflicense javac hadoopcheck spotless xml compile

uname
Linux 5fff3826cf40 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 79f853ee7c

Default Java
Temurin-1.8.0_345-b01

javac
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4824/1/artifact/yetus-general-check/output/diff-compile-javac-root.txt

Max. process+thread count
139 (vs. ulimit of 30000)

modules
C: . U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4824/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 56s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
2m 45s
master passed

+1 :green_heart:
compile
1m 51s
master passed

+1 :green_heart:
shadedjars
3m 48s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
2m  3s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 23s
the patch passed

+1 :green_heart:
compile
1m 50s
the patch passed

+1 :green_heart:
javac
1m 50s
the patch passed

+1 :green_heart:
shadedjars
3m 45s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 55s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
275m 23s
root in the patch passed.

299m 31s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4824/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/4824

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux d1c40ea54829 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 79f853ee7c

Default Java
Eclipse Adoptium-11.0.16.1+1

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4824/1/testReport/

Max. process+thread count
4780 (vs. ulimit of 30000)

modules
C: . U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4824/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 23s
Docker mode activated.

-0 :warning:
yetus
0m  6s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
2m 29s
master passed

+1 :green_heart:
compile
1m 34s
master passed

+1 :green_heart:
shadedjars
3m 47s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 31s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 16s
the patch passed

+1 :green_heart:
compile
1m 33s
the patch passed

+1 :green_heart:
javac
1m 33s
the patch passed

+1 :green_heart:
shadedjars
3m 46s
patch has no errors when building our shaded downstream artifacts.

-0 :warning:
javadoc
1m 26s
root generated 13 new + 71 unchanged - 11 fixed = 84 total (was 82)

_ Other Tests _

-1 :x:
unit
324m 41s
root in the patch failed.

345m 26s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4824/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/4824

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux d8dd1a08a50e 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 79f853ee7c

Default Java
Temurin-1.8.0_345-b01

javadoc
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4824/1/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-root.txt

unit
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4824/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4824/1/testReport/

Max. process+thread count
2086 (vs. ulimit of 30000)

modules
C: . U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4824/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.


text:
HIVE-25944: Format pom.xml-s

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Checked with running xmlstarlet locally, the edits of the poms are the same with the pr, +1 from my side.

repo: apache/hive


text:
HIVE-28262:Single column use MultiDelimitSerDe parse column error

What changes were proposed in this pull request?
modify LazyStruct#findIndexes function and LazyStruct#parseMultiDelimit function, change fields.length Conditional judgment：
public void parseMultiDelimit(byte[] rawRow, byte[] fieldDelimit) {
     - if (fields.length > 1 && delimitIndexes[i - 1] != -1) {
     + if (delimitIndexes[i - 1] != -1) {
  }

 private int[] findIndexes(byte[] array, byte[] target) {
    - if (fields.length <= 1) {
    + if (fields.length < 1) {
   ...
    - for (int i = 1; i < indexes.length; i++) {
    + for (int i = 1; i <= indexes.length; i++) {
    ...
    }
    return indexes;
  }

I add an test for this fix:
@Test
    public void testParseMultiDelimit() throws Throwable {
        try {
            // single field named id
            List<String> structFieldNames = new ArrayList<>();
            structFieldNames.add("id");
            // field type is string
            List<TypeInfo> fieldTypes = new ArrayList<>();
            PrimitiveTypeInfo primitiveTypeInfo = new PrimitiveTypeInfo();
            primitiveTypeInfo.setTypeName("string");
            fieldTypes.add(primitiveTypeInfo);

            // separators + escapeChar => "|"
            byte[] separators = new byte[]{124, 2, 3, 4, 5, 6, 7, 8};

            // sequence =>"\N"
            Text sequence = new Text();
            sequence.set(new byte[]{92, 78});

            // create lazy object inspector parameters
            LazyObjectInspectorParameters lazyObjectInspectorParameters = new LazyObjectInspectorParametersImpl(false, (byte) '0',
                    false, null, separators, sequence);
            // create a lazy struct inspector
            ObjectInspector lazyStructInspector = LazyFactory.createLazyStructInspector(structFieldNames, fieldTypes, lazyObjectInspectorParameters);
            LazyStruct lazyStruct = (LazyStruct) LazyFactory.createLazyObject(lazyStructInspector);

            // origin row data
            String rowData = "1|@|";
            // row field delimiter
            String fieldDelimiter = "|@|";

            // parse row use multi delimit
            lazyStruct.parseMultiDelimit(rowData.getBytes(StandardCharsets.UTF_8),
                    fieldDelimiter.getBytes(StandardCharsets.UTF_8));

            // check the first field and second field start position index
            // before fix result: 0,1
            // after fix result: 0,2
            Assert.assertArrayEquals(new int[]{0, 2}, lazyStruct.startPosition);
        } catch (Throwable e) {
            e.printStackTrace();
            throw e;
        }
    }

Why are the changes needed?
If a table only have one column field with multidelimit，query this column data is error data.
When I use this data to do other operation(e.g cast use UDFToLong function),get result is NULL.
Does this PR introduce any user-facing change?
No
Is the change a dependency upgrade?
No
How was this patch tested?
test class: serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyStruct.java
test function: org.apache.hadoop.hive.serde2.lazy.TestLazyStruct#testParseMultiDelimit
test command: mvn test -Dtest=TestLazyStruct --pl serde

Hi, @LaughingVzr Thanks for you contribution. Failed tests seemed to be related your change, Please take a look: http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-5252/7/tests

Ok, I will check it

LGTM.  Pending some nit comments. :)
Thanks for the fix!

authors: LaughingVzr zhangbutao


text:
[SUPPORT] Huge Performance Issue With BLOOM Index On A 1.6 Billion COW Table

Describe the problem you faced
Spark Config: spark.driver.cores=1;spark.driver.memory=18g;spark.executor.cores=10;spark.executor.memory=32g;spark.driver.maxResultSize=8g;spark.default.parallelism=400;spark.sql.shuffle.partitions=400;spark.dynamicAllocation.maxExecutors=20;spark.executor.memoryOverhead=3g;spark.kryoserializer.buffer.max=1024m
But HUDI spends many time on HoodieBloomIndex.tagLocation:

Previous stages: they are rather quick.

And With GC issues:

To Reproduce
Steps to reproduce the behavior:

Create a table
CREATE TABLE temp_db.xxxxxxxxxxx (
_hoodie_is_deleted BOOLEAN,
t_pre_combine_field BIGINT,
order_type INT,
order_no INT,
profile_no INT,
profile_type STRING,
profile_cat STRING,
u_version STRING,
order_line_no INT,
profile_c STRING,
profile_i INT,
profile_f DECIMAL(20,8),
profile_d TIMESTAMP,
active STRING,
entry_datetime TIMESTAMP,
entry_id INT,
h_version INT)
USING hudi
CLUSTERED BY (order_no, profile_type, profile_no, order_type, profile_cat)
INTO 2 BUCKETS
TBLPROPERTIES (
'primaryKey' = 'order_no,profile_type,profile_no,order_type,profile_cat',
'hoodie.cleaner.policy.failed.writes' = 'LAZY',
'type' = 'cow',
'hoodie.write.lock.filesystem.expire' = '15',
'preCombineField' = 't_pre_combine_field',
'hoodie.write.lock.provider' = 'org.apache.hudi.client.transaction.lock.FileSystemBasedLockProvider',
'hoodie.write.concurrency.mode' = 'optimistic_concurrency_control',
'hoodie.index.type' = 'BLOOM'
);
2. BULK_INSERT 1.6 Billion Data, this is quick, costing 12 mins

SET spark.sql.parquet.datetimeRebaseModeInWrite = CORRECTED;
set hoodie.datasource.write.operation = bulk_insert;
SET hoodie.combine.before.insert=false;
INSERT OVERWRITE temp_db.xxxxxxxxxxxxx
SELECT FALSE, 1, * FROM ods_us.xxxxxx_source;
3. Insert 1 million data, this is the step with mentioned performance issue.
INSERT INTO temp_db.xxxxxxxxx
(
SELECT TRUE AS _hoodie_is_deleted, *  -- 0 rows
FROM ods_us.xxxxxxxx_dddd
UNION ALL
SELECT FALSE AS _hoodie_is_deleted,* -- 1 million rows
FROM ods_us.xxxxxxxxx_stage
)
Expected behavior
UPSERT quickly.
Environment Description

Hudi version : hudi-spark3.2-bundle_2.12-0.14.1.jar

Spark version : 3.2

Hive version : 3.0

Hadoop version : 3.0

Storage (HDFS/S3/GCS..) : HDFS

Running on Docker? (yes/no) : no

Additional context
I've also tried BUKCET index , but BULK_INSERT takes 1h 30 mins, and INSERT failed with this exception:
Job aborted due to stage failure: Task 8 in stage 2.0 failed 4 times, most recent failure: Lost task 8.3 in stage 2.0 (TID 41) (gsc-bissssss.org executor 1): java.lang.RuntimeException: java.lang.NumberFormatException: For input string: "f2c9f2eb"
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:46)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:224)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:352)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1498)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1408)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1472)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1295)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:384)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:335)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1491)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NumberFormatException: For input string: "f2c9f2eb"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at org.apache.hudi.index.bucket.BucketIdentifier.bucketIdFromFileId(BucketIdentifier.java:79)
at org.apache.hudi.index.bucket.HoodieSimpleBucketIndex.lambda$loadBucketIdToFileIdMappingForPartition$0(HoodieSimpleBucketIndex.java:60)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.apache.hudi.index.bucket.HoodieSimpleBucketIndex.loadBucketIdToFileIdMappingForPartition(HoodieSimpleBucketIndex.java:56)
at org.apache.hudi.index.bucket.HoodieSimpleBucketIndex$1.computeNext(HoodieSimpleBucketIndex.java:94)
at org.apache.hudi.index.bucket.HoodieSimpleBucketIndex$1.computeNext(HoodieSimpleBucketIndex.java:87)
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
... 25 more
Record Level Index failed during BULK_INSERT, also OOM issue.
Stacktrace
Add the stacktrace of the error.

Update: This Stage completed after 1.6 hours
Other stages completed all very quickly.
The performance is so bad and I cannot figure out why. Could anyone help with this? Thanks in advance.

It might be the false-positive, can you try bucket index or simple index instead.

It might be the false-positive, can you try bucket index or simple index instead.

I've also tried BUKCET index , but BULK_INSERT takes 1h 30 mins, and INSERT failed with this exception:
Job aborted due to stage failure: Task 8 in stage 2.0 failed 4 times, most recent failure: Lost task 8.3 in stage 2.0 (TID 41) (gsc-bissssss.org executor 1): java.lang.RuntimeException: java.lang.NumberFormatException: For input string: "f2c9f2eb"
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:46)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:224)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:352)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1498)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1408)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1472)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1295)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:384)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:335)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1491)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NumberFormatException: For input string: "f2c9f2eb"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at org.apache.hudi.index.bucket.BucketIdentifier.bucketIdFromFileId(BucketIdentifier.java:79)
at org.apache.hudi.index.bucket.HoodieSimpleBucketIndex.lambda$loadBucketIdToFileIdMappingForPartition$0(HoodieSimpleBucketIndex.java:60)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.apache.hudi.index.bucket.HoodieSimpleBucketIndex.loadBucketIdToFileIdMappingForPartition(HoodieSimpleBucketIndex.java:56)
at org.apache.hudi.index.bucket.HoodieSimpleBucketIndex$1.computeNext(HoodieSimpleBucketIndex.java:94)
at org.apache.hudi.index.bucket.HoodieSimpleBucketIndex$1.computeNext(HoodieSimpleBucketIndex.java:87)
at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
... 25 more
Record Level Index failed during BULK_INSERT, also OOM issue.
With SimpeIndex, it never ends running , also has OOM issue

This error indicates that bucket index does not take effect:
Caused by: java.lang.NumberFormatException: For input string: "f2c9f2eb"

This error indicates that bucket index does not take effect:
Caused by: java.lang.NumberFormatException: For input string: "f2c9f2eb"

Hi @danny0405
I think that this may be a new issue, and it is easy to reproduce:

BULK_INSERT & Overwrite into a BUCKET index table
UPSERT into this table with some data again

Then this error shows up. Maybe BULK_INSERT is not compatible with BUCKET Index?

BULK_INSERT & Overwrite into a BUCKET index table

The bucket index for Spark bulk insert is introduced since release 0.14.0, did you use that release?

BULK_INSERT & Overwrite into a BUCKET index table

The bucket index for Spark bulk insert is introduced since release 0.14.0, did you use that release?

Yes I use 0.14.1 @danny0405

Hi @nsivabalan Could you please help with this?
And I reduced the size of the base table to 100 million, UPSERT still took very long to finish (33 mins)

@silly-carbon Did we know how much file groups the job is touching. Is it possible to attach the .hoodie zip (without metadata dir) or share one commit file to look further into it.

repo: apache/hudi


text:
[HUDI-7549] Reverting spurious log block deduction with LogRecordReader

Change Logs
We added support to deduce spurious log blocks with log block reader (https://github.com/apache/hudi/pull/9545 and https://github.com/apache/hudi/pull/9611 ) in 0.14.0.
Apparently there are some cases where it could lead to data loss or data consistency issues.
https://github.com/apache/hudi/pull/9611#issuecomment-2016687160
So, we are reverting the changes for now until we have a thorough patch where we can deduce such spurious log blocks.
Impact
No data consistency issues w/ log record reader.
Risk level (write none, low medium or high below)
medium
Documentation Update
none
Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

@nsivabalan could you check the CI failures?

Azure CI is green.

authors: nsivabalan yihua


text:
[HUDI-2245] BucketAssigner generates the fileId evenly to avoid data …

…skew
Tips

Thank you very much for contributing to Apache Hudi.
Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request
(For example: This pull request adds quick-start document.)
Brief change log
(for example:)

Modify AnnotationLocation checkstyle rule in checkstyle.xml

Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end.
Added HoodieClientWriteTest to verify the change.
Manually verified the change by running a job locally.

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

+1 LGTM


text:
Exclude Python 3.9.7 due to import error in catalog module

Adjusts the supported Python versions to exclude 3.9.7, as importing the catalog module on this version results in an unexpected error. The issue does not occur in versions 3.9.6 or 3.9.8, indicating a specific incompatibility with Python 3.9.7.
Related to #500, #497

Related python issue: https://bugs.python.org/issue45081

This sounds like a great idea @ndrluis - thank you for identifying this issue and putting in the fix. Could we ask for you to rebase your PR against the current main, and run poetry lock with your proposed python version change?

@sungwy Done!

With this change, what do I see as a user with python 3.9.7? Does it show that 3.9.7 is not supported and I should use a supported version?

Yes @kevinjqliu!
This is a example

authors: kevinjqliu ndrluis sungwy

repo: apache/iceberg-python


text:
Docs for metric, system view, system properties commands

control.sh metric command.
control.sh system view command.
ignite.sh system properties command

The Contribution Checklist

[ ] There is a single JIRA ticket related to the pull request.
[ ] The web-link to the pull request is attached to the JIRA ticket.
[ ] The JIRA ticket has the Patch Available state.
[ ] The pull request body describes changes that have been made.
The description explains WHAT and WHY was made instead of HOW.
[ ] The pull request title is treated as the final commit message.
The following pattern must be used: IGNITE-XXXX Change summary where XXXX - number of JIRA issue.
[ ] A reviewer has been mentioned through the JIRA comments
(see the Maintainers list)
[ ] The pull request has been checked by the Teamcity Bot and
the green visa attached to the JIRA ticket (see TC.Bot: Check PR)

Notes

How to Contribute
Coding abbreviation rules
Coding Guidelines
Apache Ignite Teamcity Bot

If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com #ignite channel.

Hello, @Mmuzaf
Can you, please, take a look at my changes?

Helo, @dmagda
Can you, please, take a look at my changes?

Helo, @dmagda
Can you, please, take a look at my changes?

Merged to the master.
@dmagda Thanks for the review!

Merged to the master.
@dmagda Thanks for the review!

@nizhikov
Thank you. I've cherry-picked this commit to the ignite-2.10 branch.

@nizhikov
Thank you. I've cherry-picked this commit to the ignite-2.10 branch.

authors: Mmuzaf nizhikov

repo: apache/ignite


text:
[AIRFLOW-279] Apply DagRun timeouts before checking active runs

Dear Airflow Maintainers,
Please accept this PR that addresses the following issues:

https://issues.apache.org/jira/browse/AIRFLOW-279

Testing Done:

Tested on Stripe's airflow setup

[Current coverage][cc-pull] is 64.06%

Merging [#1623][cc-pull] into [master][cc-base-branch] will decrease coverage by 0.01%

@@             master      #1623   diff @@
==========================================
  Files           121        121
  Lines          8462       8462
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
- Hits           5422       5421     -1
- Misses         3040       3041     +1
  Partials          0          0

Powered by Codecov. Last updated by [518e007...8f0a433][cc-compare]
[cc-base-branch]: https://codecov.io/gh/apache/incubator-airflow/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/apache/incubator-airflow/compare/518e0073a81585b39757afb4c0d8d65d400a88e2...8f0a43363c5edd62c79d2a94a0ef8ac21a49e4d1
[cc-pull]: https://codecov.io/gh/apache/incubator-airflow/pull/1623?src=pr

This is a duplicate :-). There is another pr that includes a test for this.

Ah, thanks! I tried to check for related issues but hadn't found anything.

authors: bolkedebruin codecov-io forklady42

repo: apache/incubator-airflow

license_source: bigquery


text:
[AIRFLOW-283] Make store_to_xcom_key a templated field in GoogleCloudStorageDownloadOperator

Dear Airflow Maintainers,
Please accept this PR that addresses the following issues:

https://issues.apache.org/jira/browse/AIRFLOW-283

Testing Done:
Reminders for contributors:

Your PR's title must reference an issue on Airflow's JIRA. For example, a PR called "[AIRFLOW-1] My Amazing PR" would close JIRA issue #1. Please open a new issue if required!
Please squash your commits when possible and follow the 7 rules of good Git commits.

Please rebase. Looks like you never rebased after your merge was squashed.

authors: criccomini illop


text:
Better to avoid responsing in header_filter_by_lua and body_filter_by_lua

https://github.com/apache/incubator-apisix/blob/master/lua/apisix.lua#L142
if phase ~= "log" then
        for i = 1, #plugins, 2 do
            local phase_fun = plugins[i][phase]
            if phase_fun then
                local code, body = phase_fun(plugins[i + 1], api_ctx)
                if code or body then
                    core.response.exit(code, body)
                end
            end
        end
        return api_ctx
    end

Since ngx.print is meaningless after content_by_lua, maybe we can treat the filter phases as another log phase?

yes, you are right.
we can print an error log message in this case.

authors: membphis spacewander

repo: apache/incubator-apisix


text:
[BEAM-171] Fix JAXBCoder in the nested context

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[x] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[x] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[x] Replace "<Jira issue #>" in the title with the actual Jira issue
number, if there is one.
[x] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

JAXBCoder in the nested context is broken with any coder that writes
content after the output of the XML stream, as decode will read the
entire remaining stream and fail.
In the nested context, prepend a long representing the size of the XML
while encoding, and limit the size of the returned stream while
decoding.
Replaces #112
This should not be ported to the DataflowJavaSDK

R: @dhalperi

beam issue?

Stopped flushing, as discussed, as the marshaller must have already written its contents to the stream before closed, and the stream will be flushed and closed by the actual owner.

LGTM, merging

authors: dhalperi tgroh

repo: apache/incubator-beam