text:
Traceid and spanid wrong when using Kotlin Reactor Coroutines

Hello. When using Kotlin Reactor Coroutines like the following:
  return mono {
            val text = "Greetings from Spring Boot!"
            logger.error(text)
            text
        }

then the traceId and spanId that are being included in the log statement above look like the following:
ERROR [00000000000000000000000000000000,0000000000000000] 80834 --- [atcher-worker-1] com.example.demo.HelloController         : Greetings from Spring Boot!

The traceId and spanId in the logs are always 0 (zeros) .
This happens only for log statements inside such mono {} blocks. Other log statements outside of these blocks work as expected.
I have created a sample reproducer project here:
https://github.com/akoufa/springbootissues
Just start the Spring Boot app and use the following curl request:
curl --location 'localhost:8080/'

Since I'm not an expert in Kotlin - we have a test here that proves that this is working (https://github.com/micrometer-metrics/micrometer/blob/main/micrometer-core/src/test/kotlin/io/micrometer/core/instrument/kotlin/AsContextElementKtTests.kt). How does what we do differ from what you do?

@marcingrzejszczak @shakuzen
The behaviour is very strange. I did some tests and I found the following:
 @GetMapping("/")
    suspend fun index(): String {
        logger.error("Greetings")
        mono {
            "test"
        }.awaitSingle()
        logger.error("Spring Boot!")
        return "Spring Boot"
    }

The first statement logs the trace id and span id fine. The second statements mostly has zeroed values.

Of course the difference is that we have thread switch after the mono { ... } statement from ctor-http-nio-4 to atcher-worker-1

The problem is related to setting the span to null on the scope reset when there is no other span observation to fall back to, and how OtelTracer handles this.
Basically, when coroutines being finished, it does the following:

Internally Kotlin invokes io.micrometer.core.instrument.kotlin.KotlinObservationContextElement#restoreThreadContext
If there was no prior Observation set, it will invoke io.micrometer.observation.contextpropagation.ObservationThreadLocalAccessor#reset.
This eventually gets into io.micrometer.tracing.handler.TracingObservationHandler#onScopeReset, where it ignores any passed context
It calls the current tracer's method withSpan and passes null
If the current tracer is OtelTracer and the span passed into io.micrometer.tracing.otel.bridge.OtelTracer#withSpan is null, then it initializes the current Otel context with the invalid span, which has traceId and spanId set to zeroes (as a side-effect in the OtelSpanInScope constructor)

In the end we get the current Otel context being set to the invalid span, even if before it was set to null, which results to a polluted ThreadLocal variable.
Somewhere in io.micrometer.tracing.otel.bridge.OtelSpanInScope#storedContext, perhaps, some special handling should be added for an invalid span, so it gets reset to null, instead of the invalid value.
I'm not entirely sure what is the right approach here, as I don't fully understand the difference between reset and close.

AFAIR the problem in OTel is that you can't remove a span from thread local - you can only set it to invalid. OpenTelemetry will never return a null span when there's no current span. Look at the Span interface from opentelemetry
public interface Span extends ImplicitContextKeyed {

  /**
   * Returns the {@link Span} from the current {@link Context}, falling back to a default, no-op
   * {@link Span} if there is no span in the current context.
   */
  static Span current() {
    Span span = Context.current().get(SpanContextKey.KEY);
    return span == null ? getInvalid() : span;
  }

  /**
   * Returns an invalid {@link Span}. An invalid {@link Span} is used when tracing is disabled,
   * usually because there is no OpenTelemetry SDK installed.
   */
  static Span getInvalid() {
    return PropagatedSpan.INVALID;
  }

So if we null the current span by calling Tracing's tracer.withSpan(null) the best we can do for OTel is to reset this to the default invalid span.

@marcingrzejszczak @vooft What does this mean that the library cannot be used in conjunction with Kotlin Coroutines + Reactor?

I didn't say that. I'm just replying to @vooft comment

@marcingrzejszczak I am trying to use observationRegistry.asContextElement()  but it seems to be losing my logging.. a small snippet of how I am using it
		logger.info("has context")
		CoroutineScope(Dispatchers.Default).launch(errorHandler) {
			withContext(observationRegistry.asContextElement()) {
				logger.info("has no context")
			}

		}
		logger.info("also has no context")

I am using spring boot and injecting observationRegistry, not sure if that is proper usage.
Also I can get the "has no context" logging properly if I change withContext(observationRegistry.asContextElement()) { with withContext(MDCContext(context)) { where val context = MDC.getCopyOfContextMap() is set before CoroutineScope.  However that last logger.info always loses context

Hi @marcingrzejszczak. We have the same problem as @akoufa but with coroutines router (we are using Brave insead of OTel).
If we use standard router and provide coroutine context as suggested, both logs (before&after delay) contains traceId&spanId.
@Configuration(proxyBeanMethods = false)
class RoutesConfiguration(@Autowired private val observationRegistry: ObservationRegistry) {
	private val logger = KotlinLogging.logger {}

	// traceId&spanId are logged only before delay
	@Bean
	fun notWorkingCoRouter() = coRouter {
		GET("/log1") {
			logger.info("before")
			delay(500)
			logger.info { "after" }
			ServerResponse.ok().buildAndAwait()
		}
	}

	// traceId&spanId are logged before and after delay
	@Bean
	fun workingRouter() = router {
		GET("/log2") {
			mono(observationRegistry.asContextElement()) {
				logger.info("before")
				delay(500)
				logger.info { "after" }
				ServerResponse.ok().buildAndAwait()
			}
		}
	}
}

Do you think this issue can be fixed or we should use some kind of workaround?

Is there any workaround that does not include injecting the observationRegistry to all controllers?
And is there any plan to fix this issue?

I don't have enough Kotlin knowledge. If there's anyone that can help us out with this then please be my guest and comment

@marcingrzejszczak in general the problem is that when a coroutine opens, the context might be lost in the child coroutines.
for example:
@RestController
class FooRestController {

  @GetMapping
  suspend fun foo(): Foo {
    logger.info { "before" }
    delay(500) // kotlin decide to change to coroutine#2
    logger.info { "after" } // we lose the observation context
  }

  @GetMapping
  fun foo(): Mono<Foo> {
    return mono(observationRegistry.asContextElement()) {
        logger.info { "before" }
        delay(500) // kotlin decide to change to coroutine#2
        logger.info { "after" } // we do not loss the context, because the parent coroutine was started with the observation context and its being propogated
    }
  }
}

If it was possible to somehow add the observationRegistry.asContextElement() into the place that initialize the coroutine it should be propagated downstream.
Another option is somehow to hook between the MDC context, Reactor context, and Coroutine context.
An example of how it has been done for MDC can be found with this official library: https://kotlinlang.org/api/kotlinx.coroutines/kotlinx-coroutines-slf4j/
with this library, another working example of propagating properly would be:
@RestController
class FooRestController {

  @GetMapping
  suspend fun foo(): Foo {
    logger.info { "before" }
    delay(500) // kotlin decide to change to coroutine#2
    logger.info { "after" } // we lose the observation context
  }

  @GetMapping
  suspend fun foo(): Foo {
    return withContext(MDCContext()) {
        logger.info { "before" }
        delay(500) // kotlin decide to change to coroutine#2
        logger.info { "after" } // we do not loss the context, because kotlinx-coroutines-slf4j handles the binding between MDC and the coroutine context
    }
  }
}

i am not sure where exactly it should be fixed, but in theory if we had an entrypoint to add the observationContext as a coroutine context into the place that spring initialize the coroutine scope for the controller's suspend function.. it should work as expected.

FYI, on the Spring side, for Controllers, the coroutine entry point is provided by InvocableHandlerMethod class.
https://github.com/spring-projects/spring-framework/blob/e5ee369e70e0a8ca04437f502649b11171a1c741/spring-web/src/main/java/org/springframework/web/method/support/InvocableHandlerMethod.java#L201
Its doInvoke(...) checks if the function is a Kotlin suspend function and, if so, ultimately calls CoroutinesUtils.invokeSuspendingFunction(...)
Notably, it does not give a CoroutineContext object, so the coroutine is started with no context.
For WebFlux, maybe Spring should take whatever is in Reactor Context at this point and copy it into Coroutine Context?

I think we need help from Sebastian. Let us summon our Kotlin Guru @sdeleuze!

@marcingrzejszczak Do we have any update on this or a suggestion for a temporary workaround on our side?

Be aware there are a bunch of Coroutines context issues still open on Spring Framework, and this area needs refinements to support all the use cases. See for example https://github.com/spring-projects/spring-framework/issues/26977, https://github.com/spring-projects/spring-framework/issues/27522 and https://github.com/spring-projects/spring-framework/issues/27010.
It is expected to have seemless context propagation when doing Reactor to Coroutines conversion (this is handled on Coroutines Reactor integration side) but when creating a new Coroutines context and when an existing should be reused likely requires some refinements.
We probably need to plan some related discussion with me and @marcingrzejszczak and/or @bclozel to discuss the design of such refined integration.

Any updates regarding this issue?

Hey, I am starting working on that and may ask guidance to Kotin team on the best way to fix that. This kind of issue seems to impact both Micrometer and Spring as soon as kotlinx.coroutines.reactor.mono is involved.

So the challenge was to pass the CoroutineContext despite the fact that kotlinx.coroutines.reactor.mono is not a suspending function (it cannot capture the context and requires to pass the context explicitly, as documented).
I made progresses on Spring Framework side, for now focusing on the functional router DSL:

I think I have found a way to pass the CoroutineContext defined in a filter to the suspending request handler, see this related comment with a draft commit.
I am also considering adding a fun context(provider: suspend (ServerRequest) -> CoroutineContext) method to the router DSL in Spring Framwork 6.1 to provide a custom CoroutineContext for various use cases, see https://github.com/spring-projects/spring-framework/issues/27010#issuecomment-1695796770.
The Reactive variant of awaitSingle() will be replaced by the Mono variant, see https://github.com/spring-projects/spring-framework/issues/31127

I will try to refine the annotation-based WebFlux support as well.
I am not sure what need to be done or not on Micrometer side, but hopefully those progresses on Spring Framework side will help. Feedback on those draft commits welcomed if any, otherwise I will move forward on Spring Framework side and drop a comment here when the various improvement will be available in Spring Framework 6.1.0-SNAPSHOT.

With the 3 issues above fixed, 6.1.0-SNAPSHOT Spring Framework builds should now contain everything needed from a Spring Framework POV for CoroutineContext propagation in WebFlux functional DSL.
If somebody could build and share a reproducer with Spring coRouter and Micrometer, that would allow me and Micrometer team to provide guidelines, identify potential refinements and validate that those recent Spring changes are useful and works as expected.

I have just added the last part of Spring Framework 6.1 CoroutineContext refinement: it is now possible to propagate it via CoWebFilter. https://github.com/spring-projects/spring-framework/issues/27522

I have just added the last part of Spring Framework 6.1 CoroutineContext refinement: it is now possible to propagate it via CoWebFilter. spring-projects/spring-framework#27522

Thanks for these new features.
However, I'm not sure how this will fix the fact we have to call withContext manually into each controller method like in @tomfi example:
@RestController
class FooRestController {

  @GetMapping
  suspend fun foo(): Foo {
    return withContext(MDCContext()) {
        logger.info { "before" }
        delay(500)
        logger.info { "after" }
    }
  }
}

This is not very intuitive and quite invasive. Also, this used to work in Spring Boot 2 by the way, I don't know if this might help or not.

@grassehh

However, I'm not sure how this will fix the fact we have to call withContext manually into each controller method

I updated the dependency to Spring Framework 6.1 and registered the CoWebFilter with the following code. Please note that this is not the exact code used in my project, but a simplified version. The key point is that I'm using withContext in CoWebFilter to add MDCContext(). It seems to be working as intended for now, but I'll need to take a closer look. I hope this helps.
@Bean
fun coroutineWebFilter(): WebFilter {
    return object : CoWebFilter() {
        override suspend fun filter(exchange: ServerWebExchange, chain: CoWebFilterChain)
        = withContext(MDCContext()) {
            chain.filter(exchange)
        }
    }
}

Fails also when calling a coroutine from Java, as vanilla as possible.
Servlet environment + webclient / http client called from inside a kotlin suspend function.
Hooks.enableAutomaticContextPropagation() does not seem to do anything.
WebClient/resttemplate with and without mono works without the hook or the capture.
@GetMapping("/")
public String index() {
  logger.info("==index=="); // traced
  MonoKt
    .mono((CoroutineContext) Dispatchers.getIO(), (scope, continuation) -> someSuspendFunction(continuation))  //suspend content not traced
    .doOnEach(it -> logger.info("test")) // not traced
    .contextCapture() // does not help
    .block();

  return "ok";
}

//id("org.springframework.boot") version "3.1.4"
//id("io.spring.dependency-management") version "1.1.3"

dependencies {
  implementation("org.springframework.boot:spring-boot-starter-actuator")
  implementation("org.springframework.boot:spring-boot-starter-web")
  implementation("org.springframework.boot:spring-boot-starter-webflux")
  implementation("io.micrometer:micrometer-tracing-bridge-brave")
  implementation("org.jetbrains.kotlinx:kotlinx-coroutines-reactor:1.7.3")
}

Not sure if it's the same problem.
Read around and couldn't figure out how to pass in MDCContext() or ReactorContext.
The observationRegistry.asContextElement() described above is not available for me.

@grassehh here are the changes I made to your repo - basically just upgrading versions to get to Spring Framework 6.1 snapshots and the code in https://github.com/micrometer-metrics/tracing/issues/174#issuecomment-1727691192. With that I was able to get tracing context in MDC in controllers using coroutines: https://github.com/grassehh/spring-boot-3-tracing-coroutine/compare/main...shakuzen:CoWebFilter?expand=1
I did not look more into the logbook stuff.

Can you guys have a look over #403 as it seems to still happen in calls originating from java, not sure if it's the exact issue but seems like it.
Using the latest snapshot and passing in mdccontext but still not getting the http client to forward baggage headers, logs do work though.

@grassehh here are the changes I made to your repo - basically just upgrading versions to get to Spring Framework 6.1 snapshots and the code in #174 (comment). With that I was able to get tracing context in MDC in controllers using coroutines: https://github.com/grassehh/spring-boot-3-tracing-coroutine/compare/main...shakuzen:CoWebFilter?expand=1
I did not look more into the logbook stuff.

Hello, I have tried your commit. The "CONTROLLER LOG OUTSIDE withContext()" log contains the "traceId", "spanId" and "myBaggageFilter", however it does not contain the "myBaggageController" which is created at the beginning of the controller function through the "Tracer.createBaggageInScope()" method.
So it's better than before but there is still one baggage missing. :(

@grassehh here are the changes I made to your repo - basically just upgrading versions to get to Spring Framework 6.1 snapshots and the code in #174 (comment). With that I was able to get tracing context in MDC in controllers using coroutines: https://github.com/grassehh/spring-boot-3-tracing-coroutine/compare/main...shakuzen:CoWebFilter?expand=1
I did not look more into the logbook stuff.

I confirm this works fine, but only if I use withContext(observationRegistry.asContextElement()) :)
If I use withContext(MDCContext()) the myBaggageController is not logged.

I would be interested to understand why if somebody can dig a bit more into that.

I've got an interesting situation in my sample project:

With Spring Boot 3.1.5, I need to add observationRegistry.asContextElement() within each controller. Custom baggage fields are logged as expected though.
With Spring Boot 3.2.0-SNAPSHOT I now can move observationRegistry.asContextElement() to CoWebFilter, but at the same time my custom baggage stopped showing up in the logs (only traceId is still printed).

Here is the code I'm using: LoggingControllerV1.kt
P.S. In both cases Hooks.enableAutomaticContextPropagation() still has to be called in the main function.

Not sure if that's related, but see https://github.com/spring-projects/spring-framework/issues/31706.

For me, the tracing works as expected using the CoWebFilter (basically the same code that @grassehh shared.
What I don't get:
Why doesn't this work when I use spring's coRouter instead of spring's @RestController?
I can see that the filter function of the CoWebFilter is also invoked when using the coRouter. But: The trace/mdccontext is dropped if I use delay with coRouter.
On some additional research I found the following:
https://github.com/spring-projects/spring-framework/issues/27010
So when I add the context to my coRouter, the tracing and mdcContext is working (also after delay):
@Configuration
class DemoRouter(
    private val demoHandler: DemoHandler,
    private val observationRegistry: ObservationRegistry
) {

    @Bean
    fun demoRoutes() = coRouter {
        context { observationRegistry.asContextElement() }
        GET("/demo", demoHandler::demoCall)
    }

}

What I don't get:

The CoWebFilter is also invoked and sets the context - why is this not enough and I also have to set the context on the level of the coRouter?
Is there some way to define the context for the coRouter globally -> I like it better to have it set in a central way as in CoWebFilter, otherwise I would have to add it to every coRouter.

Hey in Spring Framework 6.1 there were some fixes done related to this issue. Is there an option to check if the problem persists in Boot 3.2, Spring Framework 6.1? If that's the case can I ask for a reproducer with a README on how to reproduce it?

In my case I haven't built an example with coRouter yet (so other folks might need to test this out as well), but for the code based on @RestController which I have here, it's now enough to add withContext(observationRegistry.asContextElement()) to CoWebFilter, and this ensures traceId propagation.
One thing about CoWebFilter that I ran into was that the following code will ensure baggage is correctly attached to the trace (will appear in the logs):
withContext(observationRegistry.asContextElement()) {
    tracer.createBaggageInScope("userId", "Saruman").use {
        logger.info("Before chain.filter()")
        chain.filter(exchange)
        logger.info("After chain.filter()")
    }
}

while the following code (within CoWebFilter) doesn't work as expected:
tracer.createBaggageInScope("userId", "Saruman").use {
    withContext(observationRegistry.asContextElement()) {
        logger.info("Before chain.filter()")
        chain.filter(exchange)
        logger.info("After chain.filter()")
    }
}

This is apparently because within CoWebFilter if you try to get observationRegistry.currentObservation it's correctly pre-set already, while tracer.currentTraceContext().context() returns null. 🤷

Hey in Spring Framework 6.1 there were some fixes done related to this issue. Is there an option to check if the problem persists in Boot 3.2, Spring Framework 6.1? If that's the case can I ask for a reproducer with a README on how to reproduce it?

Not sure if we are deviating from the original author's issue @akoufa but for me context propagation with custom baggages works fine when resuming coroutine function using spring boot 3.2.0 and micrometer-tracing-bom 1.2.0.
As suggested in previous comments, declaring a CoWebFilter like in my sample here, is the trick. You can call the /coroutine route of my sample and you will see it contains the baggages.
@ilya40umov have you properly set these on application startup ?
Hooks.enableAutomaticContextPropagation()
ContextRegistry.getInstance().registerThreadLocalAccessor(ObservationAwareSpanThreadLocalAccessor(tracer));
ObservationThreadLocalAccessor.getInstance().observationRegistry = observationRegistry
Metrics.observationRegistry(observationRegistry)

@grassehh Thanks! I had Hooks.enableAutomaticContextPropagation() in my project already, but added the rest to be sure. Still the same thing that I reported. Looking at your reproducer I see that you have AppWebFilter (a separate, regular filter) adding the baggage and then also you add it in the controllers. In my case I was trying to add baggage and apply observationRegistry.asContextElement() in the same CoWebFilter, which is where I ran into this behavior.
Anyway, as you said it may be worth raising a separate issue for this particular case (just not sure on which project to open it to be honest).

@meberhard looks like the coRouter implementation does not pick up the context that is stored by CoWebFilter in an exchange attribute.
In fact I suspect that it's better to do the following:
coRouter {
    context { it.exchange().attributes[CoWebFilter.COROUTINE_CONTEXT_ATTRIBUTE] as CoroutineContext }
    GET("/demo", demoHandler::demoCall)
}

The only place in Spring Framework that is handling this attribute otherwise is this: https://github.com/spring-projects/spring-framework/blob/main/spring-webflux/src/main/java/org/springframework/web/reactive/result/method/InvocableHandlerMethod.java#L296
which is not getting called for coRouter handlers (just checked with a debugger).
@sdeleuze I've got a question on the currentCoWebFilter implementation. What happens if I actually have multiple CoWebFilter implementations in my project and they all manipulate CoroutineContext somehow?

@ilya40umov Thanks a lot for already creating the issues in the spring-project!
I had even more issues with the coRouter:

setting the context does not work with nesting
the context is not applied to the onError handler

So for now I changed to @RestController in my project.
I already asked these question in StackOverflow, but no answers so far:
https://stackoverflow.com/questions/77597922/spring-webflux-corouter-corouterfunctiondsl-coroutine-context-not-applied-to

I will have a deeper look to the 2 issues raised on Spring Framework side, and to this Stackoverflow question.

The 2 issues should be fixed in Spring Framework 6.1.2-SNAPSHOT, could you try to see how it goes before we do the release to confirm the fixes are working as expected?

@sdeleuze
That's awesome! I just tested this again.
With Spring Framework 6.1.2-SNAPSHOT, I don't have to use context in my coRouter to set the context. The CoWebFilter is applied.
The context to onError is only propagated if I am on the same thread.
If I do for example a SQL operation or just delay and the thread switches, the onError handler loses the context.
Here is a super light-weight example:
https://github.com/meberhard/corouter-context-propagation
In this example, a call to /rest/v1/demo-router will delay in the repository and then the errorHandler loses the context (trace-id empty)
08:45:07.143 [reactor-http-nio-3] INFO  com.example.demo.router.DemoHandler traceID=6577ad13c221af552e79008709c5f31b - test handler
08:45:07.144 [reactor-http-nio-3] INFO  c.example.demo.service.DemoService traceID=6577ad13c221af552e79008709c5f31b - call to test service
08:45:07.144 [reactor-http-nio-3] INFO  c.e.demo.repository.DemoRepository traceID=6577ad13c221af552e79008709c5f31b - demo repository called
08:45:08.159 [kotlinx.coroutines.DefaultExecutor] ERROR c.example.demo.router.ErrorHandler traceID= - test

The same implementation with @RestController works (context is available in the @RestControllerAdvice that handles the exception (call to /rest/v1/demo-controller:
08:45:13.150 [task-1] INFO  c.example.demo.router.DemoController traceID=6577ad199bf4e1b7bcfcdf1648e7d54a - test contoller called
08:45:13.150 [task-1] INFO  c.example.demo.service.DemoService traceID=6577ad199bf4e1b7bcfcdf1648e7d54a - call to test service
08:45:13.150 [task-1] INFO  c.e.demo.repository.DemoRepository traceID=6577ad199bf4e1b7bcfcdf1648e7d54a - demo repository called
08:45:14.177 [kotlinx.coroutines.DefaultExecutor] INFO  c.e.demo.router.ControllerAdvice traceID=6577ad199bf4e1b7bcfcdf1648e7d54a - Exception handled: test

Could it be related to the usage of ThreadLocal based infrastructure here? With Coroutines, like with Reactive, you are not supposed to use ThreadLocal but rather Reactor or Coroutines contexts.

Not sure if this has anything to do, but have you tried declaring Hooks.enableAutomaticContextPropagation() inside your WebConfig post contruct rather than after call to runApplication ?

@grassehh is correct - enableAutomaticContextPropagation() needs to be the first thing you do @meberhard

Hint: you can also use spring.reactor.context-propagation=auto property instead of calling Hooks.enableAutomaticContextPropagation() by hand since Spring Boot 3.2.0.

Could it be related to the usage of ThreadLocal based infrastructure here? With Coroutines, like with Reactive, you are not supposed to use ThreadLocal but rather Reactor or Coroutines contexts.

I tried removing the two lines
ContextRegistry.getInstance().registerThreadLocalAccessor(ObservationAwareSpanThreadLocalAccessor(tracer));
ObservationThreadLocalAccessor.getInstance().observationRegistry = observationRegistry

-> same result, no context/traceId in the onError method.
I tried it both - with the ThreadLocal stuff that you have in your example and without.
Hooks.enableAutomaticContextPropagation()
ContextRegistry.getInstance().registerThreadLocalAccessor(ObservationAwareSpanThreadLocalAccessor(tracer));
ObservationThreadLocalAccessor.getInstance().observationRegistry = observationRegistry
Metrics.observationRegistry(observationRegistry)

Hooks.enableAutomaticContextPropagation()
ContextRegistry.getInstance().registerThreadLocalAccessor(ObservationAwareSpanThreadLocalAccessor(tracer));
ObservationThreadLocalAccessor.getInstance().observationRegistry = observationRegistry
Metrics.observationRegistry(observationRegistry)

Just FYI - with @RestController only the line Hooks.enableAutomaticContextPropagation() is required - so probably we can drop the other lines also.

@meberhard Not sure if this has anything to do, but have you tried declaring Hooks.enableAutomaticContextPropagation() inside your WebConfig post contruct rather than after call to runApplication ?

Also tried that -> moved this to the WebConfig, same result (no context/traceId in the onError method).

@grassehh is correct - enableAutomaticContextPropagation() needs to be the first thing you do @meberhard

good point -> I don't even know why it worked at all with the @RestController like that. I anyway also moved this line before the run - same result: no context/traceId in the onError call.

Hint: you can also use spring.reactor.context-propagation=auto property instead of calling Hooks.enableAutomaticContextPropagation() by hand since Spring Boot 3.2.0.

Awesome! Thanks a lot for this 🙏

I have fixed the nested route + context issue via https://github.com/spring-projects/spring-framework/issues/31831, so please test and report any remaining problem with Spring Framework 6.1.2-SNAPSHOT (release planned tomorrow).

I have fixed the nested route + context issue via spring-projects/spring-framework#31831, so please test and report any remaining problem with Spring Framework 6.1.2-SNAPSHOT (release planned tomorrow).

I can confirm that the nesting now works as expected, but the context is still not propagated to the onError handler.
    @Bean
    fun demoRoutes() = coRouter {
        context { observationRegistry.asContextElement() }
        "/rest".nest {
            "v1".nest {
                "/demo-router".nest {
                    GET("", demoHandler::demoCall)
                }
            }
        }
        onError<RuntimeException> { throwable, serverRequest ->
            errorHandler.handleError(throwable, serverRequest)
        }
    }

For the route demo-router the context and trace is now available!
But if there is any error, there is still no context and trace for the call to errorHandler.

@meberhard This works for me so please double check:
@Bean
fun router() = coRouter {
    context { CoroutineName("Custom context") }
    "/nested".nest {
        GET("/") {
            throw RuntimeException()
        }
    }
    onError<RuntimeException> { _, _ ->
        ok().bodyValueAndAwait(currentCoroutineContext().toString())
    }
}

If you can reproduce, please create a Spring Framework issue with a self contained repro as an attached archive or a link to a repository.

I am not sure if this is related but in my original comment https://github.com/micrometer-metrics/tracing/issues/174#issuecomment-1515603299, when starting a new coroutine within a contexed span, it would lose all span/trace info.
Is this expected when launching a new coroutine?

We have potentially a repro on https://github.com/spring-projects/spring-framework/issues/31893, if somebody could check if the issue is on Micrometer side, or let me know if something weird is seen on Spring side, that could help moving forward.

See https://github.com/spring-projects/spring-framework/issues/31893#issuecomment-1876559400 for ideas on what to explore on Micrometer side to provide proper Coroutines support.

So, @sdeleuze and I were investigating on a different issue and talked about this one in the process.
If you'e subscribed to this issue and you have a sample application that shows this problem, could you try the following?
Upgrade to Spring Boot 3.2.2
Sebastien fixed a few things in recent Spring Framework versions, so please check this version first.
Contribute a custom CoWebFilter
If the above didn't work, please continue with this step.
Kotlin coroutines provides an SLF4J module that could help setting/restoring values in the MDC from the coroutines context.
Please try and configure this in your application and report back here:
First, add the following dependencies to your application:
    implementation 'org.jetbrains.kotlinx:kotlinx-coroutines-reactor'
    implementation 'org.jetbrains.kotlinx:kotlinx-coroutines-slf4j:1.7.3'

Add the following class to your application in a package that's scanned for components:
package com.example.kotlinschedule

import kotlinx.coroutines.slf4j.MDCContext
import kotlinx.coroutines.withContext
import org.springframework.stereotype.Component
import org.springframework.web.server.CoWebFilter
import org.springframework.web.server.CoWebFilterChain
import org.springframework.web.server.ServerWebExchange

@Component
class MDCCoWebFilter : CoWebFilter() {

    override suspend fun filter(exchange: ServerWebExchange, chain: CoWebFilterChain) {
        withContext(MDCContext()) {
            chain.filter(exchange)
        }
    }
}

Then check with one of your controllers whether the traceId and spanId information is available from logs statements in controller handlers.

@bclozel as I wrote in spring issue I think this may be what you are looking for

@antechrestos I'm not sure I understand. You're saying this filter solution works for your case then? I don't need a sample now just confirmation. For me in some cases it worked even without that filter, hence my question to the community. Thanks!

@bclozel without filter,, when I get result from a CoMongoRepository the context is lost
With a filter that get the current observation.asCoroutineContext it works
Yet my code does no delay, and so on.
I need to push a little further

The code I used
    @Bean
    fun coroutineContextFilter(observationRegistry: ObservationRegistry): CoWebFilter = object : CoWebFilter() {
        override suspend fun filter(exchange: ServerWebExchange, chain: CoWebFilterChain) {
            withContext(observationRegistry.asContextElement()){
                chain.filter(exchange)
            }
        }
    }

@bclozel I enriched my example
A CoWebFilter
    @Bean
    fun coroutineContextFilter(observationRegistry: ObservationRegistry): CoWebFilter = object : CoWebFilter() {
        override suspend fun filter(exchange: ServerWebExchange, chain: CoWebFilterChain) {
            when(exchange.request.uri.path) {
                "/api/v1/users-without-context" -> chain.filter(exchange)
                "/api/v1/users-with-mdc" -> withContext(MDCContext()){
                    chain.filter(exchange)
                }
                else -> withContext(observationRegistry.asContextElement()){
                    chain.filter(exchange)
                }
            }
        }
    }

And a controller with different scenario and same treatment
    @PostMapping("/api/v1/users-without-context")
    @ResponseStatus(HttpStatus.CREATED)
    suspend fun createUserNoContext(@RequestBody user: CreateUserDto) : UserDto {
        return launchUserCreation(user)
    }

    @PostMapping("/api/v1/users")
    @ResponseStatus(HttpStatus.CREATED)
    suspend fun createUser(@RequestBody user: CreateUserDto) : UserDto {
        return launchUserCreation(user)
    }

    @PostMapping("/api/v1/users-with-mdc")
    @ResponseStatus(HttpStatus.CREATED)
    suspend fun createUserMdc(@RequestBody user: CreateUserDto) : UserDto {
        return launchUserCreation(user)
    }

    private suspend fun launchUserCreation(user: CreateUserDto): UserDto {
        logger.debug("Start save of user {}", user)
        return userRepository.create(
            User.newUser(
                firstname = user.firstname,
                name = user.name,
            )
        )
            .let { toWeb(it) }
            .also {
                logger.debug("Wait for it")
                longTreatment(it)
                logger.debug("Created user {}", it)
            }
    }

    private suspend fun longTreatment(userDto: UserDto) {
        Mono.just(userDto)
            .flatMap { mono(MDCContext()) {
                logger.debug("Start long treatment")
                delay(Duration.ofSeconds(2).toMillis())
                logger.debug("End long treatment")
            } }
            .tap(
                Micrometer.observation(observationRegistry) { registry ->
                    val observation = Observation.createNotStarted("long-treatment", registry)
                        .contextualName("long-treatment")
                    observation
                }
            )
            .awaitSingle()
    }

Without contexts
2024-02-01T15:05:50.277+01:00 DEBUG 7296 --- [or-http-epoll-2] [c1265f5a0fe3c50e318f7cec8253067b-f4afc96976097cde] o.a.s.t.api.web.UserController           : Start save of user CreateUserDto(firstname=string, name=string)
2024-02-01T15:05:50.277+01:00 DEBUG 7296 --- [or-http-epoll-2] [c1265f5a0fe3c50e318f7cec8253067b-f4afc96976097cde] o.a.s.t.infrastructure.MongoRepository   : about to create User(id=b132c76a-53cb-40d6-87df-54c04af8ca0d, firstname=string, name=string)
2024-02-01T15:05:50.280+01:00 DEBUG 7296 --- [ntLoopGroup-3-3] [c1265f5a0fe3c50e318f7cec8253067b-f4afc96976097cde] o.a.s.t.infrastructure.MongoRepository   : user created User(id=b132c76a-53cb-40d6-87df-54c04af8ca0d, firstname=string, name=string)
2024-02-01T15:05:50.281+01:00 DEBUG 7296 --- [ntLoopGroup-3-3] [                                                 ] o.a.s.t.api.web.UserController           : Wait for it
2024-02-01T15:05:50.281+01:00 DEBUG 7296 --- [atcher-worker-1] [c1265f5a0fe3c50e318f7cec8253067b-9c4b3066d9da0f6c] o.a.s.t.api.web.UserController           : Start long treatment
2024-02-01T15:05:52.282+01:00 DEBUG 7296 --- [atcher-worker-1] [c1265f5a0fe3c50e318f7cec8253067b-9c4b3066d9da0f6c] o.a.s.t.api.web.UserController           : End long treatment
2024-02-01T15:05:52.283+01:00 DEBUG 7296 --- [atcher-worker-1] [c1265f5a0fe3c50e318f7cec8253067b-f4afc96976097cde] o.a.s.t.api.web.UserController           : Created user UserDto(id=b132c76a-53cb-40d6-87df-54c04af8ca0d, firstname=string, name=string)

We can see that starting a new observation let the SLF4JEventistener implemented instanciated by spring get notified of a scope opened. I observed the same behaviour in the issue I opened in spring.
However we see thaat soan are well corralated tracing
With context
whatever context (MDCContext or observationRegistry.asContextElement()), the result is the same, trace id are in log and span are well corralated
I prefer using observationRegistry.asContextElement as I do not need another dependency (org.jetbrains.kotlinx:kotlinx-coroutines-slf4j )

@sdeleuze why not putting a default observationRegistry.asContextElement when no context is found as spring always  starts an observation if ObservationRefistry is present?

@antechrestos Maybe we could use a variant of that combined with @bclozel findings, see https://github.com/spring-projects/spring-framework/issues/32165#issuecomment-1923350261 for more details.

@sdeleuze nice

I have created a separate issue for extending Micrometer Observation API with some more convenient methods to help using it from Kotlin code: https://github.com/micrometer-metrics/tracing/issues/597. Not sure if you folks would agree that it's necessary though.

@sdeleuze I have been trying to find a ticket in Spring Boot which would take care of adding the following 2 bits of configuration automatically (i.e. things that are needed to ensure context propagation), as otherwise pretty much every micro-service developed with Spring Boot & Coroutines needs to add these (or something similar):

ContextPropagationConfig.kt
ObservationFilter.kt

Is this something on your radar?

@ilya40umov Related issue is likely going to be https://github.com/micrometer-metrics/tracing/issues/174#issuecomment-1923352767 (we need to update the title to expand its scope) so feel free to comment there after reading my latest proposal.

@bclozel Hi, I'm trying to start coroutine from GatewayFilter. The tracer works fine till I enter into this flat map " return super.writeWith(fluxBody.buffer().flatMap { dataBuffers: List ->..."
until then I'm able to get current span and trace id. Inside this flat map tracer giving me null for current context.
During research I found I need to use this lines, to activate automatic propagation. So I having that set up
"
Hooks.enableAutomaticContextPropagation()
ContextRegistry.getInstance().registerThreadLocalAccessor(ObservationAwareSpanThreadLocalAccessor(tracer))
ObservationThreadLocalAccessor.getInstance().observationRegistry = observationRegistry
Metrics.observationRegistry(observationRegistry)
"
can someone told me how to resolve this propagation of trace id to reactive processing like here in this flat map?
https://github.com/component
class BodyModifyingFilter(
private val linkExpansionAndTranslationService: LinkExpansionAndTranslationService,
private val observationRegistry: ObservationRegistry,
private val tracer: Tracer
) :
GatewayFilter, Ordered {
override fun filter(exchange: ServerWebExchange, chain: GatewayFilterChain): Mono {
val decoratedResponse: ServerHttpResponseDecorator = object : ServerHttpResponseDecorator(exchange.response) {
override fun writeWith(body: Publisher): Mono {
if (body is Flux<*>) {
val fluxBody: Flux = Flux.from(body as Publisher)
return super.writeWith(fluxBody.buffer().flatMap { dataBuffers: List ->
val bufferFactory = exchange.response.bufferFactory()
val joined = bufferFactory.join(dataBuffers)
val bytes = ByteArray(joined.readableByteCount())
joined.read(bytes)
DataBufferUtils.release(joined) // Release the memory for the data buffer
val securityContextMono = ReactiveSecurityContextHolder.getContext()
                securityContextMono.flatMap { securityContext ->
                    mono(SecurityCoroutineContext(securityContext, tracer)) {
                        linkExpansionAndTranslationService.processBody(exchange, bytes)
                    }.flatMap { processedBytes ->
                        val buffer = bufferFactory.wrap(processedBytes)
                        exchange.response.headers.contentLength = processedBytes.size.toLong()
                        Mono.just(buffer)
                    }
                }

            })
        }
        return super.writeWith(body) // Fallback for other types of publishers
    }
}

return chain.filter(exchange.mutate().response(decoratedResponse).build())

}
}
}

authors: akoufa antechrestos bclozel boris-senapt calebdelnay chemicL grassehh ilya40umov krodyrobi marcingrzejszczak maxxis95 meberhard msosa qavid sdeleuze shakuzen styx7659 tomfi vooft

repo: micrometer-metrics/tracing


text:
[oracle-cloud] Update common files for branch master

Update common files

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

source: gharchive/pull-request

authors: CLAassistant micronaut-build

license_type: permissive

license_source: github-api


text:
[CHALLENGE SUBMISSION]

Please provide the following information for your submission:
What is your name? (First, Last)
Gwyneth Pena
Where is your GitHub Repo ?(With your challenge solution)
https://github.com/madebygps/25-days-of-serverless-2019/tree/master/day_05
What Challenge is this for? (Challenge number between 1 and 25)
5
(Optional) Anything else we should know? e.g., language used, location, blog post?
Logic apps, Cognitive Services API.

THANK YOU FOR YOUR #25DaysOfServerless SUBMISSION
We checked out your repository. We're thrilled to say your solution is now listed in our Hall of Fame
Thank you for your contributions!
This issue is now resolved! 🎉

source: gharchive/issue

authors: madebygps softchris

license: MIT


text:
Action Create a new app - app.json require changes

Hi Freddy,
I would like to ask you to update the app.json in that is created in action Create a new app.
Below are needed updates:
application:
Current value:  "application": "19.0.0.0",
Expected value:  "application": "22.0.0.0",
features:
Current value: missing
Expected value: "features": ["NoImplicitWith"]
Those two are changes that every time need to be modify in the app.json file.
Best regards,
Krzysztof

Fixed in v3.1

repo: microsoft/AL-Go


text:
Error: Could not connect to the server. Please verify that: 1. The server is started. 2. The development endpoint is enabled (DeveloperServicesEnabled server setting). 3. The service name matches the one specified in the URL. 4. The port number matches the one specified in the URL (DeveloperServicesPort server setting). 5. The protocol matches the one specified in the URL (DeveloperServicesSSLEnabled server setting), and that a trusted certificate is used in case of an SSL connection.

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps and to reproduce the behavior:

Go to '...'

AL Code to reproduce the issue
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
5. Versions:

AL Language:

Business Central:

Hi @brucecotter, can you specify which version of the extension you are using, which version of BC you are targeting, but also whether you are in Cloud or OnPrem setup?

@brucecotter please follow the template we have provided, it helps us help you :)

This issue has been closed because it needs more information and has not had recent activity. Please open a new issue if you believe this has not been resolved and reference the current issue.

authors: atoader brucecotter qutreson

repo: microsoft/AL