text:
Back-end websocket connection exception causes port not to be released(This is a very serious bug)

Describe the bug
Please provide details of the problem, including the version of Spring Cloud that you
are using.
<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <!--<version>2.3.7.RELEASE</version>-->
    <version>2.4.2</version>
    <relativePath/> <!-- lookup parent from repository -->
</parent>

<spring-cloud.version>2020.0.0</spring-cloud.version>

Sample
If possible, please provide a test case or sample application that reproduces
the problem. This makes it much easier for us to diagnose the problem and to verify that
we have fixed it.
Use Spring cloud gateway to proxy back-end websocket service
When used normally, the port can be released
gateway port is  8082
var ws = new WebSocket("ws://10.15.105.4:8082/demo/ws/msgws")

ws.close();

When using a websocket address that does not exist, an exception occurs and the port will not be released
netstat -anoq | findstr "4096"
  TCP    10.15.105.4:4096       10.15.105.4:8081       ESTABLISHED     5992
  TCP    10.15.105.4:8081       10.15.105.4:4096       ESTABLISHED     11132
  TCP    0.0.0.0:4096           0.0.0.0:0              BOUND           5992

netstat -anoq |findstr "4096"
TCP    0.0.0.0:4096           0.0.0.0:0              BOUND           5992

var ws = new WebSocket("ws://10.15.105.4:8082/demo/ws/msgws333")

ws.close();

gateway exception：
2021-01-19 10:45:54.015  WARN 5992 --- [ctor-http-nio-4] r.netty.http.client.HttpClientConnect    : [id: 0x1e2c000c, L:/10.15.105.4:4096 - R:/10.15.105.4:8081] The connection observed an error

io.netty.handler.codec.http.websocketx.WebSocketClientHandshakeException: Invalid handshake response getStatus: 404
	at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker13.verify(WebSocketClientHandshaker13.java:272) ~[netty-codec-http-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker.finishHandshake(WebSocketClientHandshaker.java:302) ~[netty-codec-http-4.1.58.Final.jar:4.1.58.Final]
	at reactor.netty.http.client.WebsocketClientOperations.onInboundNext(WebsocketClientOperations.java:119) ~[reactor-netty-http-1.0.3.jar:1.0.3]
	at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94) ~[reactor-netty-core-1.0.3.jar:1.0.3]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) ~[netty-codec-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) ~[netty-codec-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.58.Final.jar:4.1.58.Final]
	at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

You can expose the next bean to resolve such a case.
@Bean
    public HttpClientCustomizer terminateConnectionOnWsHandshakeErrorNettyClientCustomizer() {
        return (client) -> client.doOnResponseError((response, exception) -> {
            if (shouldDisposeWSConnection(response, exception)) {
                ((HttpOperations) response).channel().close();
            }
        });
    }

    private boolean shouldDisposeWSConnection(reactor.netty.http.client.HttpClientResponse response, Throwable exception) {
        return exception instanceof WebSocketClientHandshakeException
                && response.isWebsocket()
                && response instanceof HttpOperations;
    }

But it'll be great if it'll be resolved in https://github.com/reactor/reactor-netty

so it looks like this is a reactor-netty issue? @violetagg

@Fetsivalen @Orefa Please create an issue in Reactor Netty repo

This should be fixed with https://github.com/reactor/reactor-netty/pull/1635

Hi, can you tell me in which version of spring cloud gateway we can see the correction to the issue?
Thanks a lot
Angelo

@violetagg Could you please advise on the possible timeframe this fix will be available in spring-cloud ? Thanks in advance!

@ManojKNair Reactor Netty 0.9.21.RELEASE is available on Maven Central so that you can update it even as of today. For SCG releases I'll leave these to SCG maintainers to clarify.

We don't maintain the version of reactor netty, it comes from boot. @violetagg which version of boot picks up these changes?

Reactor Netty v0.9.x is used in Spring Boot 2.3.x series, I don't expect that Reactor Netty 0.9.21.RELEASE or any other version after 0.9.21.RELEASE to be picked up by Spring Boot 2.3.x series. Check the supported Spring Boot versions: https://github.com/spring-projects/spring-boot/wiki/Supported-Versions#released-versions
Reactor Netty v1.0.x is used in Spring Boot 2.4.x/2.5.x series, and Reactor Netty 1.0.8 is available in Spring Boot 2.4.8/2.5.2

@violetagg @spencergibb Thanks for the quick response.
We are using Reactor Netty 1.0.8 and Spring boot 2.5.2 and we have facing the same issue. (We found that this fix  reactor/reactor-netty#1635 is not there in 1.0.8).
This fix is not valid for 1.0.8 ?

@ManojKNair Can you check again? The fix is in the tag https://github.com/reactor/reactor-netty/commit/8afb8339f62cc8e1e1a180c18559c5830c7787b3 and also I checked the jar on maven central and the fix is there.

@violetagg Sorry, my bad. It is there in the version.
But we still facing the issue for web socket requests. HTTP requests are working fine. Could you please advise. Please see details below.

2021-06-29 13:28:07.618  WARN 4948 --- [ctor-http-nio-4] r.netty.http.client.HttpClientConnect    : [id:ead04e38-1, L:/10.212.134.226:51868 - R:/192.168.0.148:5555] The connection observed an error

io.netty.handler.codec.http.websocketx.WebSocketClientHandshakeException: Invalid handshake response getStatus: 404 Not Found
    at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker13.verify(WebSocketClientHandshaker13.java:272) ~[netty-codec-http-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.handler.codec.http.websocketx.WebSocketClientHandshaker.finishHandshake(WebSocketClientHandshaker.java:304) ~[netty-codec-http-4.1.65.Final.jar:4.1.65.Final]
    at reactor.netty.http.client.WebsocketClientOperations.onInboundNext(WebsocketClientOperations.java:117) ~[reactor-netty-http-1.0.8.jar:1.0.8]
    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94) [reactor-netty-core-1.0.8.jar:1.0.8]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.65.Final.jar:4.1.65.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.65.Final.jar:4.1.65.Final]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]

Application.yml file
spring:
  main:
    web-application-type: reactive
  profiles:
    active: dev
  cloud:
    gateway:
      default-filters:
        - DedupeResponseHeader=Access-Control-Allow-Credentials Access-Control-Allow-Origin
      globalcors:
        corsConfigurations:
          '[/**]':
              allowedOrigins: "*"
              allowedMethods: "*"
              allowedHeaders: "*"
      httpclient:
        pool:
          max-idle-time: 10s
      ssl:
        useInsecureTrustManager: true
      filter:
        StripPrefix: 1
        remove-non-proxy-headers:
          headers:
            - Keep-Alive
            - TE
            - Trailer
            - Transfer-Encoding
      routes:
        - id : no_op
          uri: no://op
          predicates:
            - Path=/app/tunneling/**
        - id: request_size_route
          uri: http://192.168.0.148:5555
          predicates:
            - Path=/**

Please help...

@ManojKNair Can you provide the output of netstat see https://github.com/spring-cloud/spring-cloud-gateway/issues/2115#issue-788672396? Do you see the connection opened?

@violetagg Our scenario is bit different, please see details below.
We sent request : [in web client]-> ws://domain.com/socket -> [in Cloud gateway] -> 192.168.10.10:5555 (Reverse proxied ).
The IP 192.168.10.10 is valid , still it throws 404.
Is there any other scenario where we get the same error?
Thanks in advance!

@ManojKNair Let me clarify that the fix in Reactor Netty is for the case when a connection is not closed and doesn't have anything with 404 response.

@violetagg Thanks for the response.
We had an investigation and finally got the root cause of the issue.
We found that the header of the web socket request is going as all small in netty but server only accept first letter as capital( Connection,Upgrade). We will check with Netty team .
Are you Netty maintainer ? I saw you have replied to @Jerry29. Could you please suggest a best person to contact ?

Are you Netty maintainer ? I saw you have replied to @Jerry29. Could you please suggest a best person to contact ?

@ManojKNair No I'm not Netty maintainer. Here is more info about Netty channels https://netty.io/community.html

Thank you @violetagg

authors: Fetsivalen ManojKNair Orefa manang spencergibb violetagg

repo: spring-cloud/spring-cloud-gateway


text:
Failed to invoke Feign and RestTemplate in Spring Cloud 2020's Gateway

Following code ran in Spring Cloud H version correctly, buT failed in 2020 version. Please have a look
Feign class
@FeignClient(value = "discovery-guide-service-a")
public interface GatewayFeign {
    @GetMapping(path = "/invoke/{value}")
    String invoke(@PathVariable(value = "value") String value);
}

Gateway Filter class
public class MyGatewayFilter implements GlobalFilter, Ordered {
    private static final Logger LOG = LoggerFactory.getLogger(MyGatewayFilter.class);

    @Autowired
    private GatewayFeign gatewayFeign;

    @Autowired
    private RestTemplate restTemplate;

    @Override
    public int getOrder() {
        return 10000;
    }

    @Override
    public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
        try {
            String parameter = "MyGatewayFilter";
            String feignValue = gatewayFeign.invoke(parameter);
            String restTemplateValue = restTemplate.getForEntity("http://discovery-guide-service-a/rest/" + parameter, String.class).getBody();

            LOG.info("Feign invoke result={}", feignValue);
            LOG.info("RestTemplate invoke result={}", restTemplateValue);
        } catch (RestClientException e) {
            LOG.info("Invoke failed", e);
        }

        return chain.filter(exchange);
    }
}

Exception
java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-http-nio-4
	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:83)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
	|_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ HTTP GET "/discovery-guide-service-a/invoke/gateway" [ExceptionHandlingWebHandler]
Stack trace:
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:83)
		at reactor.core.publisher.Mono.block(Mono.java:1703)
		at org.springframework.cloud.loadbalancer.blocking.client.BlockingLoadBalancerClient.choose(BlockingLoadBalancerClient.java:150)
		at org.springframework.cloud.openfeign.loadbalancer.FeignBlockingLoadBalancerClient.execute(FeignBlockingLoadBalancerClient.java:88)
		at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:119)
		at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:89)
		at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100)
		at com.sun.proxy.$Proxy122.invoke(Unknown Source)
		at com.nepxion.discovery.guide.gateway.filter.MyGatewayFilter.filter(MyGatewayFilter.java:44)
		at org.springframework.cloud.gateway.handler.FilteringWebHandler$GatewayFilterAdapter.filter(FilteringWebHandler.java:137)
		at org.springframework.cloud.gateway.filter.OrderedGatewayFilter.filter(OrderedGatewayFilter.java:44)
		at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.lambda$filter$0(FilteringWebHandler.java:117)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4046)
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:173)
		at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4046)
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:173)
		at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
		at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:73)
		at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:281)
		at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:860)
		at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)
		at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:73)
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1789)
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
		at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)
		at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:281)
		at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:860)
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)
		at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1789)
		at reactor.core.publisher.MonoFilterWhen$MonoFilterWhenMain.onNext(MonoFilterWhen.java:149)
		at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2359)
		at reactor.core.publisher.MonoFilterWhen$MonoFilterWhenMain.onSubscribe(MonoFilterWhen.java:112)
		at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:54)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4046)
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:448)
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:250)
		at reactor.core.publisher.FluxDematerialize$DematerializeSubscriber.onNext(FluxDematerialize.java:98)
		at reactor.core.publisher.FluxDematerialize$DematerializeSubscriber.onNext(FluxDematerialize.java:44)
		at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:270)
		at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:228)
		at reactor.core.publisher.FluxDematerialize$DematerializeSubscriber.request(FluxDematerialize.java:127)
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:235)
		at reactor.core.publisher.FluxDematerialize$DematerializeSubscriber.onSubscribe(FluxDematerialize.java:77)
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:164)
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86)
		at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)
		at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4046)
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:448)
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:218)
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:164)
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86)
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
		at reactor.core.publisher.Mono.subscribe(Mono.java:4046)
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:173)
		at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
		at reactor.netty.http.server.HttpServer$HttpServerHandle.onStateChange(HttpServer.java:860)
		at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:638)
		at reactor.netty.transport.ServerTransport$ChildObserver.onStateChange(ServerTransport.java:475)
		at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:525)
		at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
		at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:209)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.lang.Thread.run(Thread.java:748)

Maybe I know the root cause, in H version, i still use Ribbon, but in 2020 with webflux spring cloud loadbalancer, it does not support Feign invoke in Spring Cloud Gateway filter. Am I right?
Refer to the issue
https://github.com/spring-cloud/spring-cloud-gateway/issues/1090

Maybe I know the root cause, in H version, i still use Ribbon, but in 2020 with webflux spring cloud loadbalancer, it does not support Feign invoke in Spring Cloud Gateway filter. Am I right?
Refer to the issue
https://github.com/spring-cloud/spring-cloud-gateway/issues/1090

I have the same problem with you. How did you solve it, just down the version ?

I have the same problem too. How did you solve it, just down the version ?

See
https://github.com/spring-cloud/spring-cloud-gateway/issues/1090

I have the same problem too.

I have the same problem too.

use webclient  instead of Feign , Feign is a sync framework, but  gateway is async .

I have the same problem too.

use webclient instead of Feign , Feign is a sync framework, but gateway is async .

Thanks，use webclient instead of Feign , It's OK

I have the same problem, In spring cloud gateway, if service discovery is implemented based on "spring-cloud-loadbalancer", an error will be reported: "java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-http-nio-", if I have to use spring cloud loadbalancer for service discovery, how can I solve this problem?

Maybe I know the root cause, in H version, i still use Ribbon, but in 2020 with webflux spring cloud loadbalancer, it does not support Feign invoke in Spring Cloud Gateway filter. Am I right?
Refer to the issue
#1090

right! so. How to implements service discovery?

Maybe I know the root cause, in H version, i still use Ribbon, but in 2020 with webflux spring cloud loadbalancer, it does not support Feign invoke in Spring Cloud Gateway filter. Am I right?
Refer to the issue
#1090

right! so. How to implements service discovery?

@Autowired
private WebClient.Builder webClientBuilder;
Mono monoInfo = webClientBuilder.build().
get().uri("http://your service/api/xxxx").header(arg1,
arg2).retrieve().bodyToMono(XXXX.class);

Maybe I know the root cause, in H version, i still use Ribbon, but in 2020 with webflux spring cloud loadbalancer, it does not support Feign invoke in Spring Cloud Gateway filter. Am I right?
Refer to the issue
#1090

right! so. How to implements service discovery?

@Autowired
private WebClient.Builder webClientBuilder;
Mono monoInfo = webClientBuilder.build().
get().uri("http://your service/api/xxxx").header(arg1,
arg2).retrieve().bodyToMono(XXXX.class);

Thanks!

source: gharchive/issue

authors: HaojunRen colbertwong krislint novayoung zhouwei517

license_type: permissive

license_source: github-api


text:
Can't initialize GqlDatastoreQuery properly in multithread environment: Annotated GQL Query Method findByQuery has 18 tags but a different number of parameter values: 3

My Code:
public interface CountryCountRepository extends DatastoreRepository<CountryCountProjection, Key> {

    @Query("select count, country from TABLE where date = @date and type = @type and platform = @platform")
    List<CountryCountProjection> findByQuery(String date, BQDataType type, MobilePlatform platform);

}

Problem:
method inside GqlDatastoreQuery failed when it's initialized with several threads.
	private List<String> getOriginalParamTags() {
		if (this.originalParamTags == null) {
			this.originalParamTags = new ArrayList<>();
I got exception here:
		if (tags.size() != vals.size()) {
			throw new DatastoreDataException("Annotated GQL Query Method "
					+ this.queryMethod.getName() + " has " + tags.size()
					+ " tags but a different number of parameter values: " + vals.size());
		}

this code throw exception:
	private GqlQuery<? extends BaseEntity> bindArgsToGqlQuery(String gql,
			List<String> tags,
			List vals) {
		Builder builder = GqlQuery.newGqlQueryBuilder(gql);
		builder.setAllowLiteral(true);
		if (tags.size() != vals.size()) {
			throw new DatastoreDataException("Annotated GQL Query Method "
					+ this.queryMethod.getName() + " has " + tags.size()
					+ " tags but a different number of parameter values: " + vals.size());
		}

Because:
in debug i have:
tags = {ArrayList@10850}  size = 12
 0 = "date"
 1 = "date"
 2 = "date"
 3 = "date"
 4 = "type"
 5 = "type"
 6 = "type"
 7 = "type"
 8 = "platform"
 9 = "platform"
 10 = "platform"
 11 = "platform"
vals = {ArrayList@10851}  size = 3
 0 = "20190222"
 1 = {BQDataType@10899} "NEW_USERS"
 2 = {MobilePlatform@10900} "ANDROID"

instead of:
tags = {ArrayList@10539}  size = 3
 0 = "date"
 1 = "type"
 2 = "platform"
vals = {ArrayList@10540}  size = 3
 0 = "20190215"
 1 = {BQDataType@10581} "NEW_USERS"
 2 = {MobilePlatform@10582} "ANDROID"

@ChengyuanZhao @dmitry-s PTAL

@alexeenka
Could you tell more about how or when you're starting these threads?
I suspect the issue is in
private List<String> getOriginalParamTags()
But this method should only do its work once during initialization.
The GqlDatastoreQuery initialization happens at the startup of the application just once.

@ChengyuanZhao
yes, during initialization i need to do some computation work, i have:
@SpringBootApplication
@RestController
@EnableAsync
public class SomeApplication implements CommandLineRunner {
    @Override
    public void run(String... args) {
        LOGGER.info("APPLICATION - STARTED!");
       service.doWork();
    }
}

inside that method i have:
...
        IntStream.range(0, days).parallel().forEach(i -> {
                    List<CountryCountProjection> iData =  countryCountRepository.findByQuery(startDate.plusDays(i).format(BASIC_ISO_DATE), BQDataType.NEW_USERS, mobilePlatform);
                    }
...
        });
...

repository code:
public interface CountryCountRepository extends DatastoreRepository<CountryCountProjection, Key> {

    @Query("select count, country from TABLE where date = @date and type = @type and platform = @platform")
    List<CountryCountProjection> findByQuery(String date, BQDataType type, MobilePlatform platform);

}

@alexeenka
I'm trying to to replicate your issue by also using   IntStream.range(0, days).parallel().forEach and submitting queries from a custom repo with @Query queries taking parameters but i'm not getting this error and I'm verifying that indeed the queries are happening asynchronously.
Can you tell more about how you're creating your instance of countryCountRepository ? Are you using our Datastore starter auto-config module?

sorry, in what version that fix is apply?
i use?
<spring-cloud-gcp.version>1.1.2.RELEASE</spring-cloud-gcp.version>

authors: ChengyuanZhao alexeenka meltsufin

repo: spring-cloud/spring-cloud-gcp

license_source: bigquery


text:
Repackage spring-cloud-kubernetes-configserver so that it can also be used as a dependency

I am trying to add security to the spring-cloud-kubernetes-configserver, and the only way I see is to fork the project and add the necessary changes like dependencies and configuration.
It would be great if the spring-cloud-kubernetes-configserver was packaged in a way that it can be either directly launched as a service (the way it is today) or that it can also be used as a dependency for a custom built config server with enhanced functionality (similar to the spring-cloud-config-server on which it is based).
There seems to be a simple solution that only involves changing the classifier in the repackage goal of the build script that would allow both usages, as described here: https://stackoverflow.com/questions/40089443

Makes sense, PRs welcome if you are interested

I’d like to help because we’ll need this as well. However, I’m not entirely sure how to proceed. Could you please point me to a guideline or provide some instructions?

@pepanek would love some help!  If you form the project make the changes and submit a PR we can go from there

You just need to fork the project and that way you can push branches to your own fork
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo

authors: pepanek ryanjbaxter tspoeri

repo: spring-cloud/spring-cloud-kubernetes


text:
Some CVEs in dependencies are threatening your project!

Hi, I have noticed that some library CVEs may be related to your projects. I suggest updating their versions to increase the security of your project. Here is the details:

Vulnerable Library Version: org.hibernate : hibernate-validator : 5.2.4.Final
CVE ID: CVE-2019-10219, CVE-2017-7536
Import Path: spring-cloud-kubernetes-examples/kubernetes-reload-example/pom.xml
Suggested Safe Versions: 6.0.0.Alpha1, 6.0.0.Alpha2, 6.0.0.Beta1, 6.0.0.Beta2, 6.0.0.CR1, 6.0.0.CR2, 6.0.0.CR3, 6.0.0.Final, 6.0.1.Final, 6.0.10.Final, 6.0.11.Final, 6.0.12.Final, 6.0.13.Final, 6.0.14.Final, 6.0.15.Final, 6.0.16.Final, 6.0.17.Final, 6.0.18.Final, 6.0.2.Final, 6.0.3.Final, 6.0.4.Final, 6.0.5.Final, 6.0.6.Final, 6.0.7.Final, 6.0.8.Final, 6.0.9.Final, 6.1.0.Alpha1, 6.1.0.Alpha2, 6.1.0.Alpha3, 6.1.0.Alpha4, 6.1.0.Alpha5, 6.1.0.Alpha6, 6.1.0.Final, 6.1.1.Final, 6.1.2.Final

Thanks for trying to help. In the future, if you believe that you have found a security vulnerability and have identified a way in which it can be exploited, please report it to the Pivotal Security team.


text:
Consul Retry does not work as expected.

I have micro-service which listens on RabbitMQ message and based on that message a Feign client is been invoked for which I have to enable Spring-Retry. To enable I have added the following configurations in my application.yml
cloud:
   consul:
        retry:
          max-attempts: 5
          initial-interval: 1000
      loadbalancer:
        retry:
          enabled: false`

And I have added @EnableRetry to the component class. But I don't see retry is happening rather I'm seeing the following error
org.springframework.messaging.MessageHandlingException: nested exception is feign.RetryableException: Connection refused (Connection refused) executing GET http://flex-executionconfiguration-service/api/actionconfig/sync_smoke_uuid_1
	at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:96)
	at org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:89)
	at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109)
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127)
	at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:121)
	at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:89)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:358)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:269)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:186)
	at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:115)
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127)
	at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:121)
	at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:89)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:358)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:269)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:186)
	at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:115)
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127)
	at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:121)
	at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:89)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105)
	at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:188)
	at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter.access$1100(AmqpInboundChannelAdapter.java:56)
	at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$Listener.processMessage(AmqpInboundChannelAdapter.java:246)
	at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$Listener.onMessage(AmqpInboundChannelAdapter.java:203)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:822)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:745)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:97)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1276)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:726)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1219)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1189)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1500(SimpleMessageListenerContainer.java:97)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1421)
	at java.lang.Thread.run(Thread.java:745)
Caused by: feign.RetryableException: Connection refused (Connection refused) executing GET http://flex-executionconfiguration-service/api/actionconfig/sync_smoke_uuid_1
	at feign.FeignException.errorExecuting(FeignException.java:67)
	at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:104)
	at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76)
	at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
	at com.sun.proxy.$Proxy135.getActionConfig(Unknown Source)
	at com.jobengine.mapper.JobModelMapper.toJobRequest(JobModelMapper.java:53)
	at com.jobengine.handler.JobExecutionRequestHandler.handle(JobExecutionRequestHandler.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:113)
	at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:102)
	at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:49)
	at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:347)
	at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:88)
	at org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:131)
	at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:330)
	at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:169)
	at org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:319)
	at org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:155)
	at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:93)
	... 55 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
	at sun.net.www.http.HttpClient.New(HttpClient.java:308)
	at sun.net.www.http.HttpClient.New(HttpClient.java:326)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at feign.Client$Default.convertResponse(Client.java:152)
	at feign.Client$Default.execute(Client.java:74)
	at org.springframework.cloud.sleuth.instrument.web.client.feign.TraceFeignClient.execute(TraceFeignClient.java:90)
	at org.springframework.cloud.netflix.feign.ribbon.RetryableFeignLoadBalancer$1.doWithRetry(RetryableFeignLoadBalancer.java:91)
	at org.springframework.cloud.netflix.feign.ribbon.RetryableFeignLoadBalancer$1.doWithRetry(RetryableFeignLoadBalancer.java:76)
	at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
	at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:163)
	at org.springframework.cloud.netflix.feign.ribbon.RetryableFeignLoadBalancer.execute(RetryableFeignLoadBalancer.java:76)
	at org.springframework.cloud.netflix.feign.ribbon.RetryableFeignLoadBalancer.execute(RetryableFeignLoadBalancer.java:47)
	at com.netflix.client.AbstractLoadBalancerAwareClient$1.call(AbstractLoadBalancerAwareClient.java:109)
	at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:303)
	at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:287)
	at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:231)
	at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:228)
	at rx.Observable.unsafeSubscribe(Observable.java:10211)
	at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.drain(OnSubscribeConcatMap.java:286)
	at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.onNext(OnSubscribeConcatMap.java:144)
	at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:185)
	at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180)
	at rx.Observable.unsafeSubscribe(Observable.java:10211)
	at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94)
	at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42)
	at rx.Observable.unsafeSubscribe(Observable.java:10211)
	at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber$1.call(OperatorRetryWithPredicate.java:127)
	at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73)
	at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52)
	at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:79)
	at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:45)
	at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:276)
	at rx.Subscriber.setProducer(Subscriber.java:209)
	at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:138)
	at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:129)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	at rx.Observable.subscribe(Observable.java:10307)
	at rx.Observable.subscribe(Observable.java:10274)
	at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:445)
	at rx.observables.BlockingObservable.single(BlockingObservable.java:342)
	at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:102)
	at org.springframework.cloud.netflix.feign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:63)
	at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:97)
	... 75 common frames omitted

2017-07-06 15:59:02.503 ERROR [flex-jobengine-service,c82b6307c9de35fa,c82b6307c9de35fa,false] 3598 ---[enerContainer-1]c.o.f.j.h.JobExecutionErrorHandler       : Error while processing job execution request: Message conversion failed

org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Message conversion failed
	at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$Listener.onMessage(AmqpInboundChannelAdapter.java:223)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:822)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:745)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:97)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1276)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:726)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1219)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1189)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1500(SimpleMessageListenerContainer.java:97)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1421)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.messaging.MessageHandlingException: nested exception is org.springframework.amqp.AmqpRejectAndDontRequeueException: org.springframework.messaging.MessageHandlingException: nested exception is feign.RetryableException: Connection refused (Connection refused) executing GET http://flex-executionconfiguration-service/api/actionconfig/sync_smoke_uuid_1, failedMessage=GenericMessage [payload=JobExecutionRequest(executionId=test-execution-sync_smoke, actionConfigUuid=sync_smoke_uuid_1, jobContextVariables=null, assets=null, async=false), headers={amqp_receivedDeliveryMode=NON_PERSISTENT, amqp_deliveryTag=2, amqp_consumerQueue=flex.async.flexjob.request.queue, X-Message-Sent=true, amqp_redelivered=false, messageSent=true, spanTraceId=746c1542ebb336eb, spanId=ca46865aca9260e2, amqp_receivedRoutingKey=flex.async.flexjob.request.queue, X-B3-SpanId=ca46865aca9260e2, currentSpan=[Trace: 746c1542ebb336eb, Span: ca46865aca9260e2, Parent: 9ad2cd7f2666784a, exportable:false], X-B3-Sampled=0, X-B3-TraceId=746c1542ebb336eb, id=2006b886-6349-9c70-52f3-47f634750a8f, amqp_consumerTag=amq.ctag-z6Og-RPiqnK7p_IusivJQg, X-Current-Span=[Trace: 746c1542ebb336eb, Span: ca46865aca9260e2, Parent: 9ad2cd7f2666784a, exportable:false], spanSampled=0}]
	at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:96)
	at org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:89)
	at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109)
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127)
	at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:121)
	at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:89)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105)
	at org.springframework.integration.endpoint.MessageProducerSupport.sendErrorMessageIfNecessary(MessageProducerSupport.java:207)
	at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:191)
	at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter.access$1100(AmqpInboundChannelAdapter.java:56)
	at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$Listener.processMessage(AmqpInboundChannelAdapter.java:246)
	at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$Listener.onMessage(AmqpInboundChannelAdapter.java:203)
	... 11 common frames omitted
Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: org.springframework.messaging.MessageHandlingException: nested exception is feign.RetryableException: Connection refused (Connection refused) executing GET http://flex-executionconfiguration-service/api/actionconfig/sync_smoke_uuid_1, failedMessage=GenericMessage [payload=JobExecutionRequest(executionId=test-execution-sync_smoke, actionConfigUuid=sync_smoke_uuid_1, jobContextVariables=null, assets=null, async=false), headers={amqp_receivedDeliveryMode=NON_PERSISTENT, amqp_deliveryTag=2, amqp_consumerQueue=flex.async.flexjob.request.queue, X-Message-Sent=true, amqp_redelivered=false, messageSent=true, spanTraceId=746c1542ebb336eb, spanId=ca46865aca9260e2, amqp_receivedRoutingKey=flex.async.flexjob.request.queue, X-B3-SpanId=ca46865aca9260e2, currentSpan=[Trace: 746c1542ebb336eb, Span: ca46865aca9260e2, Parent: 9ad2cd7f2666784a, exportable:false], X-B3-Sampled=0, X-B3-TraceId=746c1542ebb336eb, id=2006b886-6349-9c70-52f3-47f634750a8f, amqp_consumerTag=amq.ctag-z6Og-RPiqnK7p_IusivJQg, X-Current-Span=[Trace: 746c1542ebb336eb, Span: ca46865aca9260e2, Parent: 9ad2cd7f2666784a, exportable:false], spanSampled=0}]
	at com.jobengine.handler.JobExecutionErrorHandler.handle(JobExecutionErrorHandler.java:35)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:113)
	at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:129)
	at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:49)
	at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:347)
	at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:88)
	at org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:131)
	at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:330)
	at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:169)
	at org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:319)
	at org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:155)
	at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:93)
	... 28 common frames omitted
Caused by: org.springframework.messaging.MessageHandlingException: nested exception is feign.RetryableException: Connection refused (Connection refused) executing GET http://flex-executionconfiguration-service/api/actionconfig/sync_smoke_uuid_1
	at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:96)
	at org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:89)
	at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109)
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127)
	at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:121)
	at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:89)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:358)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:269)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:186)
	at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:115)
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127)
	at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:121)
	at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:89)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:358)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:269)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:186)
	at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:115)
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127)
	at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:121)
	at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:89)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105)
	at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:188)
	... 14 common frames omitted
Caused by: feign.RetryableException: Connection refused (Connection refused) executing GET http://flex-executionconfiguration-service/api/actionconfig/sync_smoke_uuid_1
	at feign.FeignException.errorExecuting(FeignException.java:67)
	at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:104)
	at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76)
	at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
	at com.sun.proxy.$Proxy135.getActionConfig(Unknown Source)
	at com.jobengine.mapper.JobModelMapper.toJobRequest(JobModelMapper.java:53)
	at com.jobengine.handler.JobExecutionRequestHandler.handle(JobExecutionRequestHandler.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:113)
	at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:102)
	at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:49)
	at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:347)
	at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:88)
	at org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:131)
	at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:330)
	at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:169)
	at org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:319)
	at org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:155)
	at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:93)
	... 55 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
	at sun.net.www.http.HttpClient.New(HttpClient.java:308)
	at sun.net.www.http.HttpClient.New(HttpClient.java:326)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at feign.Client$Default.convertResponse(Client.java:152)
	at feign.Client$Default.execute(Client.java:74)
	at org.springframework.cloud.sleuth.instrument.web.client.feign.TraceFeignClient.execute(TraceFeignClient.java:90)
	at org.springframework.cloud.netflix.feign.ribbon.RetryableFeignLoadBalancer$1.doWithRetry(RetryableFeignLoadBalancer.java:91)
	at org.springframework.cloud.netflix.feign.ribbon.RetryableFeignLoadBalancer$1.doWithRetry(RetryableFeignLoadBalancer.java:76)
	at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
	at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:163)
	at org.springframework.cloud.netflix.feign.ribbon.RetryableFeignLoadBalancer.execute(RetryableFeignLoadBalancer.java:76)
	at org.springframework.cloud.netflix.feign.ribbon.RetryableFeignLoadBalancer.execute(RetryableFeignLoadBalancer.java:47)
	at com.netflix.client.AbstractLoadBalancerAwareClient$1.call(AbstractLoadBalancerAwareClient.java:109)
	at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:303)
	at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:287)
	at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:231)
	at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:228)
	at rx.Observable.unsafeSubscribe(Observable.java:10211)
	at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.drain(OnSubscribeConcatMap.java:286)
	at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.onNext(OnSubscribeConcatMap.java:144)
	at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:185)
	at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180)
	at rx.Observable.unsafeSubscribe(Observable.java:10211)
	at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94)
	at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42)
	at rx.Observable.unsafeSubscribe(Observable.java:10211)
	at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber$1.call(OperatorRetryWithPredicate.java:127)
	at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73)
	at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52)
	at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:79)
	at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:45)
	at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:276)
	at rx.Subscriber.setProducer(Subscriber.java:209)
	at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:138)
	at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:129)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	at rx.Observable.subscribe(Observable.java:10307)
	at rx.Observable.subscribe(Observable.java:10274)
	at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:445)
	at rx.observables.BlockingObservable.single(BlockingObservable.java:342)
	at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:102)
	at org.springframework.cloud.netflix.feign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:63)
	at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:97)
	... 75 common frames omitted

2017-07-06 15:59:02.506  WARN [flex-jobengine-service,,,] 3598 ---[enerContainer-1]s.a.r.l.ConditionalRejectingErrorHandler : Execution of Rabbit message listener failed.

org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener threw exception
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:915)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:825)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:745)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:97)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:189)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1276)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:726)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1219)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1189)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1500(SimpleMessageListenerContainer.java:97)
	at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1421)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.messaging.MessageHandlingException: nested exception is org.springframework.amqp.AmqpRejectAndDontRequeueException: org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Message conversion failed
	at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:96)
	at org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:89)
	at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109)
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127)
	at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:121)
	at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:89)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105)
	at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$Listener.onMessage(AmqpInboundChannelAdapter.java:223)
	at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:822)
	... 10 common frames omitted
Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Message conversion failed
	at com.jobengine.handler.JobExecutionErrorHandler.handle(JobExecutionErrorHandler.java:35)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:113)
	at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:129)
	at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:49)
	at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:347)
	at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:88)
	at org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:131)
	at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:330)
	at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:169)
	at org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:319)
	at org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:155)
	at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:93)
	... 24 common frames omitted
Caused by: org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Message conversion failed
	... 12 common frames omitted
Caused by: org.springframework.messaging.MessageHandlingException: nested exception is org.springframework.amqp.AmqpRejectAndDontRequeueException: org.springframework.messaging.MessageHandlingException: nested exception is feign.RetryableException: Connection refused (Connection refused) executing GET http://flex-executionconfiguration-service/api/actionconfig/sync_smoke_uuid_1, failedMessage=GenericMessage [payload=JobExecutionRequest(executionId=test-execution-sync_smoke, actionConfigUuid=sync_smoke_uuid_1, jobContextVariables=null, assets=null, async=false), headers={amqp_receivedDeliveryMode=NON_PERSISTENT, amqp_deliveryTag=2, amqp_consumerQueue=flex.async.flexjob.request.queue, X-Message-Sent=true, amqp_redelivered=false, messageSent=true, spanTraceId=746c1542ebb336eb, spanId=ca46865aca9260e2, amqp_receivedRoutingKey=flex.async.flexjob.request.queue, X-B3-SpanId=ca46865aca9260e2, currentSpan=[Trace: 746c1542ebb336eb, Span: ca46865aca9260e2, Parent: 9ad2cd7f2666784a, exportable:false], X-B3-Sampled=0, X-B3-TraceId=746c1542ebb336eb, id=2006b886-6349-9c70-52f3-47f634750a8f, amqp_consumerTag=amq.ctag-z6Og-RPiqnK7p_IusivJQg, X-Current-Span=[Trace: 746c1542ebb336eb, Span: ca46865aca9260e2, Parent: 9ad2cd7f2666784a, exportable:false], spanSampled=0}]
	at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:96)
	at org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:89)
	at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109)
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127)
	at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:121)
	at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:89)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105)
	at org.springframework.integration.endpoint.MessageProducerSupport.sendErrorMessageIfNecessary(MessageProducerSupport.java:207)
	at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:191)
	at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter.access$1100(AmqpInboundChannelAdapter.java:56)
	at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$Listener.processMessage(AmqpInboundChannelAdapter.java:246)
	at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$Listener.onMessage(AmqpInboundChannelAdapter.java:203)
	... 11 common frames omitted
Caused by: org.springframework.amqp.AmqpRejectAndDontRequeueException: org.springframework.messaging.MessageHandlingException: nested exception is feign.RetryableException: Connection refused (Connection refused) executing GET http://flex-executionconfiguration-service/api/actionconfig/sync_smoke_uuid_1, failedMessage=GenericMessage [payload=JobExecutionRequest(executionId=test-execution-sync_smoke, actionConfigUuid=sync_smoke_uuid_1, jobContextVariables=null, assets=null, async=false), headers={amqp_receivedDeliveryMode=NON_PERSISTENT, amqp_deliveryTag=2, amqp_consumerQueue=flex.async.flexjob.request.queue, X-Message-Sent=true, amqp_redelivered=false, messageSent=true, spanTraceId=746c1542ebb336eb, spanId=ca46865aca9260e2, amqp_receivedRoutingKey=flex.async.flexjob.request.queue, X-B3-SpanId=ca46865aca9260e2, currentSpan=[Trace: 746c1542ebb336eb, Span: ca46865aca9260e2, Parent: 9ad2cd7f2666784a, exportable:false], X-B3-Sampled=0, X-B3-TraceId=746c1542ebb336eb, id=2006b886-6349-9c70-52f3-47f634750a8f, amqp_consumerTag=amq.ctag-z6Og-RPiqnK7p_IusivJQg, X-Current-Span=[Trace: 746c1542ebb336eb, Span: ca46865aca9260e2, Parent: 9ad2cd7f2666784a, exportable:false], spanSampled=0}]
	at com.ooyala.flex.jobengine.handler.JobExecutionErrorHandler.handle(JobExecutionErrorHandler.java:35)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:113)
	at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:129)
	at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:49)
	at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:347)
	at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:88)
	at org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:131)
	at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:330)
	at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:169)
	at org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:319)
	at org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:155)
	at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:93)
	... 28 common frames omitted
Caused by: org.springframework.messaging.MessageHandlingException: nested exception is feign.RetryableException: Connection refused (Connection refused) executing GET http://flex-executionconfiguration-service/api/actionconfig/sync_smoke_uuid_1
	at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:96)
	at org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:89)
	at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:109)
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127)
	at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:121)
	at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:89)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:358)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:269)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:186)
	at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:115)
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127)
	at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:121)
	at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:89)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:358)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:269)
	at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:186)
	at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:115)
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127)
	at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:148)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:121)
	at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:89)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:423)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45)
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105)
	at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:188)
	... 14 common frames omitted
Caused by: feign.RetryableException: Connection refused (Connection refused) executing GET http://flex-executionconfiguration-service/api/actionconfig/sync_smoke_uuid_1
	at feign.FeignException.errorExecuting(FeignException.java:67)
	at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:104)
	at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76)
	at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)
	at com.sun.proxy.$Proxy135.getActionConfig(Unknown Source)
	at com.jobengine.mapper.JobModelMapper.toJobRequest(JobModelMapper.java:53)
	at com.jobengine.handler.JobExecutionRequestHandler.handle(JobExecutionRequestHandler.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:113)
	at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:102)
	at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:49)
	at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:347)
	at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:88)
	at org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:131)
	at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:330)
	at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:169)
	at org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:319)
	at org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:155)
	at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:93)
	... 55 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
	at sun.net.www.http.HttpClient.New(HttpClient.java:308)
	at sun.net.www.http.HttpClient.New(HttpClient.java:326)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at feign.Client$Default.convertResponse(Client.java:152)
	at feign.Client$Default.execute(Client.java:74)
	at org.springframework.cloud.sleuth.instrument.web.client.feign.TraceFeignClient.execute(TraceFeignClient.java:90)
	at org.springframework.cloud.netflix.feign.ribbon.RetryableFeignLoadBalancer$1.doWithRetry(RetryableFeignLoadBalancer.java:91)
	at org.springframework.cloud.netflix.feign.ribbon.RetryableFeignLoadBalancer$1.doWithRetry(RetryableFeignLoadBalancer.java:76)
	at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
	at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:163)
	at org.springframework.cloud.netflix.feign.ribbon.RetryableFeignLoadBalancer.execute(RetryableFeignLoadBalancer.java:76)
	at org.springframework.cloud.netflix.feign.ribbon.RetryableFeignLoadBalancer.execute(RetryableFeignLoadBalancer.java:47)
	at com.netflix.client.AbstractLoadBalancerAwareClient$1.call(AbstractLoadBalancerAwareClient.java:109)
	at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:303)
	at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:287)
	at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:231)
	at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:228)
	at rx.Observable.unsafeSubscribe(Observable.java:10211)
	at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.drain(OnSubscribeConcatMap.java:286)
	at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.onNext(OnSubscribeConcatMap.java:144)
	at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:185)
	at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180)
	at rx.Observable.unsafeSubscribe(Observable.java:10211)
	at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94)
	at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42)
	at rx.Observable.unsafeSubscribe(Observable.java:10211)
	at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber$1.call(OperatorRetryWithPredicate.java:127)
	at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73)
	at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52)
	at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:79)
	at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:45)
	at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:276)
	at rx.Subscriber.setProducer(Subscriber.java:209)
	at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:138)
	at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:129)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
	at rx.Observable.subscribe(Observable.java:10307)
	at rx.Observable.subscribe(Observable.java:10274)
	at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:445)
	at rx.observables.BlockingObservable.single(BlockingObservable.java:342)
	at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:102)
	at org.springframework.cloud.netflix.feign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:63)
	at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:97)
	... 75 common frames omitted

Please learn how to properly format code and logs.

I'm confused at what consul retry has to do with netflix and feign.

It's a configuration issue that I resolved. Apologies for raising this issue.

authors: spencergibb vivek-dhayalan

repo: spring-cloud/spring-cloud-netflix