text:
azure-messaging-servicebus-7.0.0 - Concurrent sessions handling with single consumer

Query/Question
I am exploring the new SDK azure-messaging-servicebus-7.0.0. I have got a publisher that sends messages with an eventId and the eventId is the same across all the messages (the order is important). The publisher can also send messages with different eventId as well.
When I enable the session and sends two messages with different sessionId (eventId1, eventId2), the subscriber-only consumes the messages with session eventId1 or eventid2, but not both.
How can I make the receiver accept multiple different session messages concurrently. I see a property maxConcurrentSessions but it's not visible to the client builder.
Why is this not a Bug or a feature Request?
A clear explanation of why is this not a bug or a feature request?
Setup (please complete the following information if applicable):

OS: iOS
IDE : Intellij
Version azure-messaging-servicebus-7.0.0

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[y] Query Added
[ y] Setup information Added

What is the max number of maxConcurrentSessions allowed?

What is the max number of maxConcurrentSessions allowed?

@indvamshi
Here is sample for specifying this property if  you use Processor
https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/servicebus/azure-messaging-servicebus/src/samples/java/com/azure/messaging/servicebus/ServiceBusSessionProcessorSample.java#L26
And for receiver, you can  maxConcurrentSessions() on sessionReceiver()
https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/servicebus/azure-messaging-servicebus/src/samples/java/com/azure/messaging/servicebus/ReceiveSingleSessionAsyncSample.java#L38
What is the max number of maxConcurrentSessions allowed ? : It is fairly large number.
Do you have certain requirement of how many session you need to handle ?

@indvamshi
Here is sample for specifying this property if  you use Processor
https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/servicebus/azure-messaging-servicebus/src/samples/java/com/azure/messaging/servicebus/ServiceBusSessionProcessorSample.java#L26
And for receiver, you can  maxConcurrentSessions() on sessionReceiver()
https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/servicebus/azure-messaging-servicebus/src/samples/java/com/azure/messaging/servicebus/ReceiveSingleSessionAsyncSample.java#L38
What is the max number of maxConcurrentSessions allowed ? : It is fairly large number.
Do you have certain requirement of how many session you need to handle ?

@indvamshi
Here is sample for specifying this property if you use Processor
https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/servicebus/azure-messaging-servicebus/src/samples/java/com/azure/messaging/servicebus/ServiceBusSessionProcessorSample.java#L26
And for receiver, you can maxConcurrentSessions() on sessionReceiver()
https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/servicebus/azure-messaging-servicebus/src/samples/java/com/azure/messaging/servicebus/ReceiveSingleSessionAsyncSample.java#L38
What is the max number of maxConcurrentSessions allowed ? : It is fairly large number.
Do you have certain requirement of how many session you need to handle ?

@indvamshi
Here is sample for specifying this property if you use Processor
https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/servicebus/azure-messaging-servicebus/src/samples/java/com/azure/messaging/servicebus/ServiceBusSessionProcessorSample.java#L26
And for receiver, you can maxConcurrentSessions() on sessionReceiver()
https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/servicebus/azure-messaging-servicebus/src/samples/java/com/azure/messaging/servicebus/ReceiveSingleSessionAsyncSample.java#L38
What is the max number of maxConcurrentSessions allowed ? : It is fairly large number.
Do you have certain requirement of how many session you need to handle ?

Thank you @hemanttanwar for your quick reply.
Regarding the concurrent sessions number, I thought there is a limit. The question regarding concurrent sessions raised on session lock.
We have a requirement where a publisher puts a couple of messages onto the queue with an event Id and event state. The eventId remains the same for a group of messages but the state differs for each message. The message needs to be processed in the order based on the state. Also, the publisher sends with different eventIds and states.
To achieve message ordering we planned to go with ASB sessions. But, the problem we are facing is on the consumer the session is locked to an eventId and if I publish a message with different eventId its not consumed.
Today, I tried with concurrent sessions number as 2, and able to consume two messages. Underneath, ASB created two different session channels.
This is not the ideal solution to go forward, as the eventids can be millions and cannot create concurrent sessions for that number.  Is there a better way ASB can provide w.r.t sessions, is there a way to unlock the session and create a new session for incoming eventId.
This could be a general problem how this can be achieved with ASB.
Thanks
Vamshi

Thank you @hemanttanwar for your quick reply.
Regarding the concurrent sessions number, I thought there is a limit. The question regarding concurrent sessions raised on session lock.
We have a requirement where a publisher puts a couple of messages onto the queue with an event Id and event state. The eventId remains the same for a group of messages but the state differs for each message. The message needs to be processed in the order based on the state. Also, the publisher sends with different eventIds and states.
To achieve message ordering we planned to go with ASB sessions. But, the problem we are facing is on the consumer the session is locked to an eventId and if I publish a message with different eventId its not consumed.
Today, I tried with concurrent sessions number as 2, and able to consume two messages. Underneath, ASB created two different session channels.
This is not the ideal solution to go forward, as the eventids can be millions and cannot create concurrent sessions for that number.  Is there a better way ASB can provide w.r.t sessions, is there a way to unlock the session and create a new session for incoming eventId.
This could be a general problem how this can be achieved with ASB.
Thanks
Vamshi

@indvamshi If you want guaranteed ordering, using session queue is right choice.  Understanding session in depth will help to design your solution.
Here are some session reading you can do  https://docs.microsoft.com/en-us/azure/service-bus-messaging/message-sessions
Sending
SessionId is assigned to message when sender send a message.
When a receiver is created , it will lock that session id to ensure ordering of message. How long a session is locked based on how you have configured at the time of creating topic/queue.
Receiving
There are two ways you can receive messages 1. Using ServiceBusProcessorClient 2. Using ServiceBusSenderAsyncClient/ServiceBusSenderClient

Processor is high level and easy way to receive messages but you have less controls. When you use the processor , you can specify how many session you want to receive from. If there no messages for 60 seconds by default, it will release that session and lock next available session. You can control this time by setting retryOptions on builder (explained in code below) but this property is also used for another purpose example connection level retry, how long to wait for locking a new  session etc.
      // Example below will receive message from 10 session and if there are no messages received in 20 seconds , it will move on to next session and you will start getting message from next available session.

        ServiceBusProcessorClient processorClient = new ServiceBusClientBuilder()
                .connectionString(connectionString)
                .retryOptions(new AmqpRetryOptions().setTryTimeout(Duration.ofSeconds(20)))
                .sessionProcessor()
                .queueName(queueName)
                .maxConcurrentSessions(10)
                .processMessage(ServiceBusSessionProcessorSample::processMessage)
                .processError(ServiceBusSessionProcessorSample::processError)
                .buildProcessorClient();

In you case you want better control, when to release a session, you can use  ServiceBusSenderAsyncClient/ServiceBusSenderClient. Here is one sample .

You can control when to dispose a receiver, when you have received last message for that sessionId.  The definition of last message is specific to the application, for example in your case you use some state to determine that a message is last for that sessionId. If you use Async receiver, you can use Project reactor APIs to dispose the receiver and acceptNextSession().
For example takeUntil() operator, return true when you have received last message for that sessionId based on state.

@indvamshi If you want guaranteed ordering, using session queue is right choice.  Understanding session in depth will help to design your solution.
Here are some session reading you can do  https://docs.microsoft.com/en-us/azure/service-bus-messaging/message-sessions
Sending
SessionId is assigned to message when sender send a message.
When a receiver is created , it will lock that session id to ensure ordering of message. How long a session is locked based on how you have configured at the time of creating topic/queue.
Receiving
There are two ways you can receive messages 1. Using ServiceBusProcessorClient 2. Using ServiceBusSenderAsyncClient/ServiceBusSenderClient

Processor is high level and easy way to receive messages but you have less controls. When you use the processor , you can specify how many session you want to receive from. If there no messages for 60 seconds by default, it will release that session and lock next available session. You can control this time by setting retryOptions on builder (explained in code below) but this property is also used for another purpose example connection level retry, how long to wait for locking a new  session etc.
      // Example below will receive message from 10 session and if there are no messages received in 20 seconds , it will move on to next session and you will start getting message from next available session.

        ServiceBusProcessorClient processorClient = new ServiceBusClientBuilder()
                .connectionString(connectionString)
                .retryOptions(new AmqpRetryOptions().setTryTimeout(Duration.ofSeconds(20)))
                .sessionProcessor()
                .queueName(queueName)
                .maxConcurrentSessions(10)
                .processMessage(ServiceBusSessionProcessorSample::processMessage)
                .processError(ServiceBusSessionProcessorSample::processError)
                .buildProcessorClient();

In you case you want better control, when to release a session, you can use  ServiceBusSenderAsyncClient/ServiceBusSenderClient. Here is one sample .

You can control when to dispose a receiver, when you have received last message for that sessionId.  The definition of last message is specific to the application, for example in your case you use some state to determine that a message is last for that sessionId. If you use Async receiver, you can use Project reactor APIs to dispose the receiver and acceptNextSession().
For example takeUntil() operator, return true when you have received last message for that sessionId based on state.

Hi
Thank for you detailed response.
I have give a try with retryOptions with duration has 20seconds, This is what I observed

Published a message with a sessionId1. The message got consumed straight away.
Published another message with sessionId2 immediately, it was not consumed . After 20 seconds timeout, sessionId2 got consumed.
Published same message with sessiond2 again got consumed immediately.
Published first message with sessionId1, it was not consumed. Waited for 20 seconds still not consumed.

So once the lock for a particular session is locked and expired , am unable to retain the session back.
My understanding was when I use retry option, no matter what the sessionId is the message should be consumed similar like step 2.
Am I missing any property to retain the Sessionid1?
Thanks
Vamshi

Hi
Thank for you detailed response.
I have give a try with retryOptions with duration has 20seconds, This is what I observed

Published a message with a sessionId1. The message got consumed straight away.
Published another message with sessionId2 immediately, it was not consumed . After 20 seconds timeout, sessionId2 got consumed.
Published same message with sessiond2 again got consumed immediately.
Published first message with sessionId1, it was not consumed. Waited for 20 seconds still not consumed.

So once the lock for a particular session is locked and expired , am unable to retain the session back.
My understanding was when I use retry option, no matter what the sessionId is the message should be consumed similar like step 2.
Am I missing any property to retain the Sessionid1?
Thanks
Vamshi

Are  you using ServiceBusProcessorClient or ServiceBusSessionReceiverAsyncClient/ServiceBusSessionReceiverClient ?May be you can put your code on git somewhere, so we can help you better with your scenario.
If you use Processor, what is value maxConcurrentSessions you use in your builder ? This controls how many session you want to receive messages from. It seems like you are using one sessionId at a time to receive message from. It you set it to 2, the receiver will try to lock two sessions and you will receive messages from these two locked session.
You asked about receiving from 100/more sessionId, The current design where you use ServiceBusSessionReceiverAsyncClient/ServiceBusSessionReceiverClient,  one client instance will connect to one sessionId at a time. You can multi thread your program to create many client and each client is locked on to one sessionId.
As in client-server model, one session receiver will get a lock to one session and maintain order within that session.
Processor gives you flexibility to  lock multiple session at same time using maxConcurrentSessions but each sessionId will have its own AMQP Link establish to receive message. You have design your solution with this in mind that one receiver will lock on to one sessionId at a time.

I am using ServiceBusProcessorClient,
serviceBusClient = new ServiceBusClientBuilder()
.connectionString(host)
.sessionProcessor()
.maxConcurrentSessions(2)
.topicName(topic)
.receiveMode(ServiceBusReceiveMode.PEEK_LOCK)
.subscriptionName(subscription)
.processMessage(consumer.receiveMessageWithSession())
.processError(consumer.errorHandler())
.buildProcessorClient();
public Consumer receiveMessageWithSession() {
Consumer onMessage = context -> {
ServiceBusReceivedMessage message = context.getMessage();
//processor.processMesssage(new String(message.getBody().toBytes()));
processMessage(message);
};
return onMessage;
}
If I have got maxConcurrentSessions(2), then yes it work as expected. For each amqp link a session is assigned. What if I do not want to use that link to the same sessionId like assign to a different sessionId. If I have got maxConcurrentSessions(2) am bound to use only session id1,id2 what If I want to send messages with sesion id3,id4.
Also, as mentioned earlier if I have got maxConcurrentSessions(1), the same amqp link is shared between two sessions id1 and id2 based on the setTryTimeout. Firstly id1 is consumed, then wait for retryOption i.e 1 minutes to expire, then id2 is consumed. But, if send id1 again, its not consumed.
But, by using Microsoft sdk, ISessionHandler is able to solve the above scenario. I can see the messages getting consumed no matter what the order of the messages. I can see messages consumed with session id1,id2 ...id2,id1..etc..
The behaviour is different in Azure SDK 7.0.0 version, is it a defect or am I missing any settings?

@indvamshi
I am able to achieve this i.e. consumed with session id1,id2 ...id2,id1..etc.. , Here is sample test program in gist https://gist.github.com/hemanttanwar/b0ea9c64a76b8c51d12952832ebb16fc

Hey @indvamshi
Closing this issue as we have not heard back from you in a while. Hopefully, the sample code @hemanttanwar shared above helps.
If not, please feel free to drop a comment and we can re-open the issue

Sorry for leaving a post on a closed ticket, but did the amqp retry behavior change since this post?
.retryOptions(new AmqpRetryOptions().setTryTimeout(Duration.ofSeconds(20)))
For the life of me, I can't get the processor client to grab onto new sessions any faster than a 60 second interval.
Here is my ServiceBusClientBuilder config
return new ServiceBusClientBuilder()
                .credential(serviceBusNamespace, clientSecretCredential)
                .retryOptions(new AmqpRetryOptions()
                    .setTryTimeout(Duration.ofSeconds(5)))
                .sessionProcessor()
                .topicName(topicName)
                .subscriptionName(subscriptionName)
                .receiveMode(ServiceBusReceiveMode.PEEK_LOCK)
                .maxConcurrentSessions(10)
                .processMessage(messageHandler::processMessage)
                .processError(messageHandler::processError)
                .buildProcessorClient();

I would expect this to generate a client that would receive message from 10 session (which it does) and if there are no messages received in 5 seconds for a session, it would move on to next session and start getting message from next available session.  However, the next sessions are not processed until that 60 second loop is complete.  I can see that the duration is applied to the session and that it reports out that there are no other messages within that 5 second window, but I must be missing something in my understanding of how this should work.
Thanks.

authors: JRiggenbach hemanttanwar httr01 indvamshi ramya-rao-a


text:
Implement ShareClients

Implement ShareClients

Rolled up into #4051

source: gharchive/issue

repo: Azure/azure-sdk-for-java

license: MIT

license_type: permissive

license_source: github-api


text:
Service Bus track 2 API Review

Track API Review comments

Closed in favour of #4506

authors: conniey hemanttanwar


text:
add migration guide for sb si

Add migration guide for service bus integration library and starter library.

Close and move content to wiki

source: gharchive/pull-request


text:
tsp, openai, use latest spec commit-id, and latest typespec-java

Description
Use latest typespec-java and latest commit id from spec.
Reason for latest commit id is that the previous one "fe056966cf070be84e92dd2dc1b566bae35002cf" does not have Java configure in tsp-configure.yaml
One commit in history makes the 2 new operation @internal, hence SDK won't generate model, and the API is package private.
https://github.com/Azure/azure-rest-api-specs/pull/24546/files
Another commit fixed the enum https://github.com/Azure/azure-rest-api-specs/pull/24538/files
I didn't update CHANGELOG, as tsp in spec could still change. They may remove the LRO template, and make op "StartGenerateImage" a simple API.
All SDK Contribution checklist:

[x] The pull request does not introduce [breaking changes]
[ ] CHANGELOG is updated for new features, bug fixes or other significant changes.
[x] I have read the contribution guidelines.

General Guidelines and Best Practices

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-ai-openai

Sure. I will not run the sync job till the openai get finalized.

authors: azure-sdk weidongxu-microsoft


text:
Update CHANGELOG.md for easm sdk based on prepare-release script

Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
All SDK Contribution checklist:

[ ] The pull request does not introduce [breaking changes]
[ ] CHANGELOG is updated for new features, bug fixes or other significant changes.
[ ] I have read the contribution guidelines.

General Guidelines and Best Practices

[ ] Title of the pull request is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

/azp run java - easm - ci

authors: thang-bit weidongxu-microsoft


text:
Increment versions for mongocluster releases

Increment package versions for mongocluster releases

API change check
API changes are not detected in this pull request.


text:
Increment versions for standbypool releases

Increment package versions for standbypool releases

API change check
API changes are not detected in this pull request.


text:
[Automation] Generate Fluent Lite from Swagger cognitiveservices#package-2023-05

[Automation] Generate Fluent Lite from Swagger cognitiveservices#package-2023-05

API change check
APIView has identified API level changes in this PR and created following API reviews.
com.azure.resourcemanager:azure-resourcemanager-cognitiveservices

/azp run java - cognitiveservices - mgmt - tests

authors: XiaofeiCao azure-sdk


text:
[Service Bus] Use the term "Message" in batch related apis

In the latest v7 of the Service Bus preview package, we have the method createBatch() that takes CreateBatchOptions and returns a object that has the method tryAdd().
It is being suggested that we add the term "Message" in the above APIs to make it easy for people to find the right method to use
Since we are using the term "message" in all the apis on the sender and receiver, it is not a stretch to do the same for the batch related apis. So, lets do the below renames

createBatch -> createMessageBatch()
CreateBatchOptions -> CreateMessageBatchOptions
tryAdd() -> tryAddMessage()

PRs are welcome to add this feature.
Code Pointers:

Set up environment and run rush build
Open the sdk/servicebus/service-bus folder in your editor/IDE and start the renames

cc @JoshLove-msft, @richardpark-msft, @HarshaNalluru, @bterlson

Done with #11887
Thanks @mohsin-mehmood!

repo: Azure/azure-sdk-for-js


text:
Support setting of correlationId on Key Vault Operations

Is your feature request related to a problem? Please describe.
For debugging issues with KeyVault operations, being able to directly correlate application logs with keyvault logs would be extremely valuable.  Key Vault does support a correlationId in it's diagnostic logs, but there doesn't seem to be any way to pass this through via the SDK.
Describe the solution you'd like
Support the passing of a correlationId in requests that is used in Key Vault Diagnostic logs
Describe alternatives you've considered
None
Additional context

@marty2bell Hello! I'm Daniel, I'll be doing my best to help you.
I am currently studying what are the options to give you a proper reply. I'll be back as soon as I have something concrete.

@marty2bell Hello! I'm Daniel, I'll be doing my best to help you.
I am currently studying what are the options to give you a proper reply. I'll be back as soon as I have something concrete.

@marty2bell
Hello Mr Bell. I'm still investigating how to provide a quality response to you. In the mean time, our clients automatically send a GUID through the header  x-ms-client-request-id  on every request. You can specify your own request header x-ms-client-request-id that won't be overwritten by using a code similar to the following:
    const result = await client.setSecret(secretName, secretValue, {
      requestOptions: {
        customHeaders: {
          "x-ms-client-request-id": "CUSTOM_REQUEST_ID"
        }
      }
    });

On the responses, Azure will send back a header x-ms-request-id which is populated by a GUID created on the service side. These two headers can be used to diagnose issues with the Key Vault service.
Other header that the service supports seems to be x-ms-client-correlation-id which can be sent in a similar way on a per-request basis using the requestOptions property in the options bag.
I am still investigating about this with my team. In the mean time, I would really appreciate your thoughts so far, since your feedback will help me focus my efforts on your specific needs.
Thank you for your time so far.

@marty2bell
Hello Mr Bell. I'm still investigating how to provide a quality response to you. In the mean time, our clients automatically send a GUID through the header  x-ms-client-request-id  on every request. You can specify your own request header x-ms-client-request-id that won't be overwritten by using a code similar to the following:
    const result = await client.setSecret(secretName, secretValue, {
      requestOptions: {
        customHeaders: {
          "x-ms-client-request-id": "CUSTOM_REQUEST_ID"
        }
      }
    });

On the responses, Azure will send back a header x-ms-request-id which is populated by a GUID created on the service side. These two headers can be used to diagnose issues with the Key Vault service.
Other header that the service supports seems to be x-ms-client-correlation-id which can be sent in a similar way on a per-request basis using the requestOptions property in the options bag.
I am still investigating about this with my team. In the mean time, I would really appreciate your thoughts so far, since your feedback will help me focus my efforts on your specific needs.
Thank you for your time so far.

Hello @sadasant
We have tried setting the custom headers as you suggested. Here is the code snippet (we set all the suggested headers):
const ciphertext = await cryptographyClient.sign(SIGNING_ALGORITHM, hash,                 {                     requestOptions: {                         customHeaders: {                             'x-ms-client-request-id': '1c0cb266-28ce-47ab-ae6a-c43b049dc14d',                             'x-ms-request-id': '1c0cb266-28ce-47ab-ae6a-c43b049dc14d',                             'x-ms-client-correlation-id': '1c0cb266-28ce-47ab-ae6a-c43b049dc14d',                             'x-ms-correlation-id': '1c0cb266-28ce-47ab-ae6a-c43b049dc14d',                         },                         timeout: 2000,                     },                 });
However this is not showing up in our logs as CorrelationId. The LogsAnalytics query we used wasAzureDiagnostics  | project TimeGenerated, OperationName, ResultType, CorrelationId, requestUri_s, DurationMs, ResultSignature, httpStatusCode_d, identity_claim_appid_g, clientInfo_s | sort by TimeGenerated desc
Would you expect it to show up as CorrelationId? Or is there some other field we can access that would contain this?

Hello @sadasant
We have tried setting the custom headers as you suggested. Here is the code snippet (we set all the suggested headers):
const ciphertext = await cryptographyClient.sign(SIGNING_ALGORITHM, hash,                 {                     requestOptions: {                         customHeaders: {                             'x-ms-client-request-id': '1c0cb266-28ce-47ab-ae6a-c43b049dc14d',                             'x-ms-request-id': '1c0cb266-28ce-47ab-ae6a-c43b049dc14d',                             'x-ms-client-correlation-id': '1c0cb266-28ce-47ab-ae6a-c43b049dc14d',                             'x-ms-correlation-id': '1c0cb266-28ce-47ab-ae6a-c43b049dc14d',                         },                         timeout: 2000,                     },                 });
However this is not showing up in our logs as CorrelationId. The LogsAnalytics query we used wasAzureDiagnostics  | project TimeGenerated, OperationName, ResultType, CorrelationId, requestUri_s, DurationMs, ResultSignature, httpStatusCode_d, identity_claim_appid_g, clientInfo_s | sort by TimeGenerated desc
Would you expect it to show up as CorrelationId? Or is there some other field we can access that would contain this?

@ceramcardle
Thank you for that message! That helps me see how to focus on finding a resolution path for you. I'll reply again once I have an answer that you can reproduce via a similar query.

@ceramcardle
Thank you for that message! That helps me see how to focus on finding a resolution path for you. I'll reply again once I have an answer that you can reproduce via a similar query.

@heaths how correlationId is passed from client?

@heaths how correlationId is passed from client?

I can't say for JS for sure without digging (@sadasant?), but in .NET we generate a new one (if not passed in directly) with each request.

I can't say for JS for sure without digging (@sadasant?), but in .NET we generate a new one (if not passed in directly) with each request.

@jlichwa @heaths the field correlation ID is not part of our API, that I remember. For example, I'm unable to find mention of it on the swagger files (as an example: link). What we've settled on sending is the header x-ms-client-request-id which we're automatically sending through our cores. I could be missing something.

@jlichwa @heaths the field correlation ID is not part of our API, that I remember. For example, I'm unable to find mention of it on the swagger files (as an example: link). What we've settled on sending is the header x-ms-client-request-id which we're automatically sending through our cores. I could be missing something.

@jlichwa @heaths @sadasant
Guys, just to clarify, we have tried passing the x-ms-client-request-id custom header via the key vault SDK (see code snippet above) which has not worked.
Does the javascript SDK support this?

@jlichwa @heaths @sadasant
Guys, just to clarify, we have tried passing the x-ms-client-request-id custom header via the key vault SDK (see code snippet above) which has not worked.
Does the javascript SDK support this?

@marty2bell yes, our clients in JS create and send this header with a default value. The service team is investigating what's going on on their side. Thank you for your patience! We'll reply back as soon as we have more info.

@marty2bell yes, our clients in JS create and send this header with a default value. The service team is investigating what's going on on their side. Thank you for your patience! We'll reply back as soon as we have more info.

@marty2bell so far, we've been able to confirm that this is a bug, meaning that, as we're describing this internally, the "x-ms-client-request-id should be returned in the response header if it was passed in the request". We'll let you know as soon as we have an update on that issue. Thank you for reporting this issue and for helping us narrow down the problem!

@marty2bell so far, we've been able to confirm that this is a bug, meaning that, as we're describing this internally, the "x-ms-client-request-id should be returned in the response header if it was passed in the request". We'll let you know as soon as we have an update on that issue. Thank you for reporting this issue and for helping us narrow down the problem!

I've added the Service Attention label since we've already determined this is a service issue. Sorry for the late update! I've been heads down doing something else.

I've added the Service Attention label since we've already determined this is a service issue. Sorry for the late update! I've been heads down doing something else.

We don't support passing the values, but you can correlate your client logs using x-ms-request-id which is generated by server and returned in response.
Definition:
x-ms-request-id – is a server generated value that is returned on the response and also used in logging as correlation id.
There is also client request id:
x-ms-client-request-id - should be returned in the response header if it was passed in the request. This is purely for client correlation of the request and responses. We do log it in our internal logs, but not available in audit log.
#please-close

#please-close

@sadasant @ramya-rao-a  can you close the issue?

authors: ceramcardle heaths jlichwa marty2bell sadasant


text:
[JS] [KeyVault] Address PR feedback

Back when we did PR #3020, @schaabs mentioned several non-blocking feedback points that haven't been addressed. We'll track progress here:

~Regarding Promise<Secret> as the return value of updateSecretAttributes, when it returns this.getSecretFromSecretBundle(response): https://github.com/Azure/azure-sdk-for-js/pull/3020/files/6081d2fdf057d5a98ecaa5ed5fbffdee72953d1b#r285722703~ The return value here is correct, it can't be Promise since getSecretFromSecretBundle returns a Secret.
Regarding backupSecret returning  Promise<Uint8Array | undefined>. Why is this | undefined? https://github.com/Azure/azure-sdk-for-js/pull/3020/files/6081d2fdf057d5a98ecaa5ed5fbffdee72953d1b#r285723246
does it still make sense to have distinct options types for each of the operations?, from: https://github.com/Azure/azure-sdk-for-js/pull/3020#discussion_r285724709
Should the id property be on the ParsedKeyVaultEntityIdentifier?, from: https://github.com/Azure/azure-sdk-for-js/pull/3020#discussion_r285725517

I'm addressing the only point I know how to address, the rest is up to you, @jonathandturner :sun_with_face:

Are the last two point still relevant? @schaabs

This issue is stale. I'll close it.


text:
[Service Bus] Add a test that the version used in user agent string matches with the version being shipped

In #3017 we need to revert package.json imports. However, this brings us back to the possibility of our constant version being out of date with package.json. We can ensure this never happens with a test that reads the package.json file and compares it with the constant. In the future we can investigate other approaches to remove this manual effort.
/cc @ramya0820 @ramya-rao-a

@bterlson @AlexGhiondea @ramya-rao-a
For comparing contents of package.json with constants.ts file, adding any test would result in dependency on package.json which would break the browser tests. As per the node resolve plugin bugs uncovered as part of #3017.
Thus, it's not possible to add a regular test and we can instead accomplish this using eslint / tslint custom rules.
Since we are currently in process of moving from tslint to eslint, this approach would require coordinating with Eng Sys to publish the custom eslint plugin and would incur more efforts depending on where we are with https://github.com/Azure/azure-sdk-for-js/issues/2253
Does this approach seem acceptable?

We were thinking of using fs module to read the package.json file and not importing the package.json file in the test file
See https://github.com/Azure/azure-sdk-for-js/blob/%40azure/service-bus_1.0.2/sdk/servicebus/service-bus/.scripts/prepSamples.js for reference to using the fs module

We were thinking of using fs module to read the package.json file and not importing the package.json file in the test file

But the tests would run referencing the dist-esm source right? Then it would not find the package.json file?

We want a test case that would

use fs module to read the file at sdk/servicebus/service-bus/package.json
parse the json contents of the file, determine the name and version
import the constant packageJsonInfo from the constants.ts file
compare the 2 and fail if they don't match.

We want a test case that would

use fs module to read the file at sdk/servicebus/service-bus/package.json
parse the json contents of the file, determine the name and version
import the constant packageJsonInfo from the constants.ts file
compare the 2 and fail if they don't match.

@ramya-rao-a The concern I was trying to clarify on is that since the package.json would not be packaged it to dist / dist-esm folder the fs module may not be able to resolve to it - can still try and see if this works out if that's what we intend by having it in test case and not npm script.

Since the suggestion is to use the path to the source code i.e sdk/servicebus/service-bus/package.json, this approach does not rely on package.json file being packaged into dist/dist-esm folder

authors: bterlson ramya-rao-a ramya0820


text:
[monitor-query] Work around the double-JSON encoding bug

The Log Analytics service has a bug where it sometimes returns the batch query output encoded twice.
The issue is still under investigation (https://github.com/Azure/azure-sdk-for-js/issues/15688) but in the meantime this workaround will enable people to use the API, which is critical since we're in preview.
Mitigation for #15688

CC: @pakrym, the inspiration for this workaround.


text:
Update README.md

Fixes https://github.com/Azure/azure-sdk-for-js/issues/26656
Packages impacted by this PR
@azure/web-pubsub-express
Issues associated with this PR
https://github.com/Azure/azure-sdk-for-js/issues/26656
Describe the problem that is addressed by this PR
What are the possible designs available to address the problem? If there are more than one possible design, why was the one in this PR chosen?
Are there test cases added in this PR? (If not, why?)
Provide a list of related PRs (if any)
Command used to generate this PR:**(Applicable only to SDK release request PRs)
Checklists

[ ] Added impacted package name to the issue description
[ ] Does this PR needs any fixes in the SDK Generator?** (If so, create an Issue in the Autorest/typescript repository and link it here)
[ ] Added a changelog (if necessary)

API change check
API changes are not detected in this pull request.

authors: azure-sdk vicancy


text:
[Event Hubs] Implementation of partition manage for durable store for checkpoints using blob storage

This pr creates a separate plugin and implement partitionManager for durable store for checkpoints using Blob Storage.
TODO:

[ ] Update README
[ ] Add Sample
[ ] Update docs

I wanted to make a couple high-level comments.
We aren't storing any checkpoint data in the body of a blob, we're storing it all as metadata. When you make a listBlobsFlat call, you can specify that you want the metadata returned on each blob as well:
await containerClient.listBlobsFlat({include: ['metadata']});

This lets us get the metadata for each blob without having to make individual calls to the Storage service for each blob.
Since we're storing the blobs with a known prefix, we can also include that to the listBlobsFlat call so we only get blobs that start with our prefix:
await containerClient.listBlobsFlat({
  include: ['metadata'],
  prefix: `${eventHubName}/${consumerGroupName}/`}
);

Few things need to be updated, will create another PR for this.

authors: ShivangiReja chradek


text:
[FEATURE REQ] Get list of active sessions in the specified queue/topic/subscription

Library name
Azure.Messaging.ServiceBus
Please describe the feature.
Is it possible to get list of sessions in the specified queue/topic/subscription?
Use case: scale microservice based on sessions (there are no reasons to scale by CPU or total message count because if there are 1000 messages in single session only then 1 thread will process them all, all other instances will do nothing).
PS: Legacy WindowsAzure.ServiceBus library has such feature
https://docs.microsoft.com/ru-ru/dotnet/api/microsoft.servicebus.messaging.subscriptionclient.getmessagesessionsasync?view=azure-dotnet

Thank you for your feedback.  Tagging and routing to the team member best able to assist.

Similar request - https://github.com/Azure/azure-sdk-for-net/issues/20569

Closing this as a duplicate of https://github.com/Azure/azure-sdk-for-net/issues/20569

authors: JoshLove-msft VeselovAndrey jsquire

repo: Azure/azure-sdk-for-net


text:
Microsoft.Azure.ServiceBus.ServiceBusException: 'sender9' is closed

Library name and version
Microsoft.Azure.ServiceBus 5.1.3
Query/Question
Hi Team,
We are having application in dot net core and hosted on fargate container. This application continuously listen to queue on one service bus. On receiving any message it sends that message to another queue of same service bus. To send a message we are creating MessageSender using queue path at the start of the application itself. Also we are using AMQPWebSockets as transport type. But during this we are experiencing intermittently exception as Microsoft.Azure.ServiceBus.ServiceBusException: 'sender9' is closed.
Also on very next attempt using service bus retry it gets successfully send to that queue. We are not doing any reset or any explicit retry for this.
Exception :
Microsoft.Azure.ServiceBus.ServiceBusException: 'sender9' is closed\n   at Microsoft.Azure.ServiceBus.Core.MessageSender.OnSendAsync(IList1 messageList)\\n   at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func1 operation, TimeSpan operationTimeout)\n   at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func1 operation, TimeSpan operationTimeout)\\n   at Microsoft.Azure.ServiceBus.Core.MessageSender.SendAsync(IList1 messageList)\n   at ManagedSubscriber.Core.ServiceBusOperations.Complete(Message auditMessage, ServiceBusEntites serviceBusEntites, Message retrievedMessage, MessageProcessingReport messageProcessingReport)
Service bus package : Microsoft.Azure.ServiceBus
Version : 5.1.3
AMQP package : Microsoft.Azure.Amqp
version : 2.4.11
Dot net core : 3.1
Kindly guide us on the same.
Environment
Dot net core 3.1

@onlywithlad  It seems that the sender application took time to send the message and client instance is closed in between.
I see that you are trying to use the older version (5.1.3) of SDK.  I recommend you update to the latest version (5.2.0) and check this behavior in your environment.
Additionally, you can start capture Azure SDK logs if you are not doing it currently.  Please refer to this document for additional details.
Also, as you mentioned this is issue is happening intermittently, you may need to get it investigated through a Support ticket.
Please let me know if you have any questions.
Thanks
Saurabh

@onlywithlad Please let me know if you have any questions. Thanks.

HI @SaurabhSharma-MSFT thank you for your guidance!
But even after upgrading to latest version, issue is occurring randomly. And to capture azure sdk logs we are looking for steps to reproduce the issue. Otherwise as our application is continuously running application it will exhaust the space.

HI @SaurabhSharma-MSFT  we observed one more behavior. whenever we receive sender closed exception we are able to see one more error that is - "Microsoft.Azure.ServiceBus.ServiceBusException: The connection was inactive for more than the allowed 60000 milliseconds and is closed by container '9a467ca2-d058-4484-8c39-99e4842d3555'".
As our application is continuous running application we are creating senders at the start of application only. we are not closing or recreating inbetween.

Having the same error on Asp.Net Core 5.

Azure.Messaging.ServiceBus.ServiceBusException: 'sender179' is closed (GeneralError)
at Azure.Messaging.ServiceBus.Amqp.AmqpSender.SendBatchInternalAsync(IEnumerable1 messages, TimeSpan timeout, CancellationToken cancellationToken)    at Azure.Messaging.ServiceBus.Amqp.AmqpSender.<>c.<<SendBatchAsync>b__18_0>d.MoveNext() --- End of stack trace from previous location ---    at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.<>c__201.<b__20_0>d.MoveNext()
--- End of stack trace from previous location ---
at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)    at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1,TResult](Func4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
at Azure.Messaging.ServiceBus.ServiceBusRetryPolicy.RunOperation[T1](Func`4 operation, T1 t1, TransportConnectionScope scope, CancellationToken cancellationToken)
at Azure.Messaging.ServiceBus.Amqp.AmqpSender.SendBatchAsync(ServiceBusMessageBatch messageBatch, CancellationToken cancellationToken)
at Azure.Messaging.ServiceBus.ServiceBusSender.SendMessagesAsync(ServiceBusMessageBatch messageBatch, CancellationToken cancellationToken)

@onlywithlad

"Microsoft.Azure.ServiceBus.ServiceBusException: The connection was inactive for more than the allowed 60000 milliseconds and is closed by container '9a467ca2-d058-4484-8c39-99e4842d3555'".
I don't think this is an issue as Azure network has a timeout value of 1 min. If a connection is idle for 1 minute, then that connection will be closed.
Please refer a link to see more details.

HI @SaurabhSharma-MSFT
also is there  a way we can check if connection is idle or not? and in case of idle or sender closed issue how can we make it active again?

Hi @onlywithlad.  The Microsoft.Azure.ServiceBus library has been deprecated and is only receiving critical security and bug fixes.  Unfortunately, this means that feature requests such as adding the ability to recover from an idle connection closing are not under consideration.  It is recommended that applications using Microsoft.Azure.ServiceBus migrate to the current library, Azure.Messaging.ServiceBus which is under active development.

authors: SaurabhSharma-MSFT deivyd321 jsquire onlywithlad


text:
microsoft.Extensions.* probably the wrong place for this package

Library name and version
Microsoft.Extensions.Azure 1.1.1
Query/Question
With reference to https://docs.microsoft.com/en-us/aspnet/core/migration/50-to-60?view=aspnetcore-6.0&tabs=visual-studio#update-package-references
This package should not belong in the Microsoft.Extensions.* namespace as it does not upgrade with the .Net version, or the version should be upgraded to correspond to the .Net version that its targeting.
Microsoft.Extensions.* has a great architecture in that the series fits well with the development of .Net. so if you're developing with .net^.2, you know you need v6.2.xxxx of all the microsoft.Extensions.* libraries.  This package does not fit is with that sequence.  Could it either:

be made to fit in with this series
be put in another namespace? (Azure.Extensions., or Microsoft.Azure. seem to be good options)
The migration documentation be updated to reflect the exceptions (There are a couple of others, but this one bugs me more)

Option 3 is my least favoured option.
Sorry - its not really a question, more a bug, or design change request. For reference Migrating following the instructions is mostly very smooth, with the exception of this little bit of grid in the flow.
Environment
Migrating a number of projects from .net 5 to .net 6
Mostly Dockerised ASP.net web APIs, Azure pipelines,

Label prediction was below confidence level 0.6 for Model:ServiceLabels: 'Extensions:0.5941816,Functions:0.048598874,Azure.Core:0.032728378'

Thank you for your feedback.  Tagging and routing to the team members best able to assist.

As a GA library, any changes to the package name would constitute a breaking change to applications.   This is not something that we will be considering as a result.   Closing this out.

authors: azure-sdk grimley517 jsquire


text:
ClientModel: Integrate Azure.Core with Client Model

Scope to be decided as part of ClientModel: Decide on an approach to Azure.Core 2.0 integration with ClientModel (#42757).

We should track these customer issues as feedback related to this work.

authors: annelo-msft jsquire


text:
Produce a dependency graph during CI

Anywhere that we currently generate a dependency report we should also publish a dependency graph. #8597 and Azure/azure-sdk-tools#278 will be needed for the implementation.

https://github.com/Azure/azure-sdk-for-net/pull/9964

authors: bsiegel chidozieononiwu


text:
Dev/jtoroman/operation clients

Adding .Net clients for get, list and cancel managed server management operation API.

Here is a link to Swagger PR:
https://github.com/Azure/azure-rest-api-specs/pull/8243
I am waiting on another swagger PR (the one which enables .NET clients generation) to be merged before we merge this PR.: https://github.com/Azure/azure-rest-api-specs/pull/8558

Yes, I used autorest directly since generate.ps1 was failing.

Yes, I used autorest directly since generate.ps1 was failing.

Hi, our suggestion is to edit your generate.ps1 and change -AutoRestVersion "latest" into -AutoRestVersion "2.0.4413", and still use it to generate code (because it also generates metadata that is helpful for debug tracing).

@isra-fel I have run the generate.ps1 script as you suggested and added metadata files.


text:
Update package version for ARM

Update package version for ARM

Hi @vivsriaus, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

It looks like you're working at Microsoft (visriniv). If you're full-time, we DON'T require a contribution license agreement.

If you are a vendor, or work for Microsoft Open Technologies, DO please sign the electronic contribution license agreement. It will take 2 minutes and there's no faxing! https://cla.azure.com.

TTYL, AZPRBOT;

@azuresdkci retest this please.

authors: azurecla hovsepm vivsriaus


text:
Fixed code for authentication using AzureCredentials

Just fixed method name..

Can one of the admins verify this patch?

Hi @mholec, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no contribution license agreement is required at this point. Real humans will now evaluate your PR.

TTYL, AZPRBOT;

@azuresdkci test this please

@hovsepm Please assign appropriately to someone on the fluent team to triage

authors: azurecla azuresdkci hovsepm markcowl mholec


text:
[Event Hubs] Remove Project References

Summary
The focus of these changes is to remove the project references to the Azure.Messaging.EventHubs library.  Now that the 5.7.x line has reached a stable release, the package references can be restored.

/azp run net - eventhub - tests

API change check for Azure.Messaging.EventHubs.Experimental
API changes are not detected in this pull request for Azure.Messaging.EventHubs.Experimental

API change check for Microsoft.Azure.WebJobs.Extensions.EventHubs
API changes are not detected in this pull request for Microsoft.Azure.WebJobs.Extensions.EventHubs

authors: azure-sdk jsquire


text:
Fix management plane tests

This PR includes:

Remove the Manually attribute in mgmt plane test project so we will not miss the test failure in pipeline.
Fix some test records that are failed due to the tags change
For AppConfiguration, Communication and WebPubSub, skip the tests in live mode to ensure these tests will not block the test pipeline.

Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

API change check
API changes are not detected in this pull request.


text:
update api-version for Azure.ResourceManager.Monitor

Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.ResourceManager.Monitor

authors: ArcturusZhang azure-sdk


text:
[For Discussion] Example generated LRO

Example to illustrate what a generated LRO might look like, to illustrate this for the Azure Load Testing service.

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.Template

authors: annelo-msft azure-sdk


text:
Increment version for redisenterprise releases

Increment package version after release of Azure.ResourceManager.RedisEnterprise

API change check
API changes are not detected in this pull request.


text:
Design ATOM api integration

There are two approaches to integrating the management API.

Standalone client, as we had via control client
a getter/creator off of the core ServiceBusClient (e.g. as if it was a receiver.)

Create proposal, OK past architects, align with broader SDK family.
(To transfer mental state into this ticket, the one big concern about 2 was that you now have 2 "levels' of potentially conflicting-seeming settings, e.g. retry settings)

Settled with architects and broader SDK crew on distinct clients for GA.  If necessary an off-client creator could come later.

repo: Azure/azure-sdk-for-python


text:
[Blob] SAS token created using python did not work for azcopy

Hello Everyone
I am trying to generate a SAS token using Python code as per below doc
https://docs.microsoft.com/en-us/python/api/overview/azure/storage-blob-readme?view=azure-python
from datetime import datetime, timedelta
from azure.storage.blob import BlobServiceClient, generate_account_sas, ResourceTypes, AccountSasPermissions
sas_token = generate_account_sas(
account_name="https://accountname.blob.core.windows.net",
account_key="entered accountkey",
resource_types=ResourceTypes(service=True),
permission=AccountSasPermissions(read=True,add=True,create=True,write=True,delete=True),
expiry=datetime.utcnow() + timedelta(hours=1)
)
print (sas_token)
SAS token I received I added in below azcopy command
azcopy rm "https://accountnameb.blob.core.windows.net/containername/filename?sas_token"
I am getting below error
RequestId:0d6b21bb-101e-006a-0775-7d38d6000000
Time:2021-07-20T14:41:47.5594591Z, Details:
AuthenticationErrorDetail: Signature did not match. String to sign used was accounname
rwdac
b
s
2021-07-20T15:41:20Z
2020-06-12
Code: AuthenticationFailed
RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
Thanks

Hi @ashi4286, thank you for opening an issue! I'll tag the appropriate folks so we can take a look at this as soon as possible.

Code worked fine after making the below change
resource_types=ResourceTypes(object=True)
Please close this issue

Glad to hear that you found a solution! Closing now.


text:
Unable to control what is being logged and sent to App-insights by Azure Monitor log exporter.

I am using a Fast API app which is being started as module along with my API. There is also an Logging-config.yml file where we define what all loggers we have and what kind of handler is being used :
version: 1
disable_existing_loggers: false
loggers:
  root:
    level: WARNING  # WARNING is the default level
    handlers:
      - console
  uvicorn:
    qualname: uvicorn
    propagate: 0
    level: DEBUG
    handlers:
      - console
  my_api:
    qualname: my_api
    propagate: 0
    level: DEBUG
    handlers:
      - console

handlers:
  console:
    class: logging.StreamHandler
    formatter: standard
    stream: ext://sys.stderr

formatters:
  standard:
    format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s'
    datefmt: '%Y-%m-%d %H:%M:%S'
  otel:
    format: '%(asctime)s %(levelname)s [%(name)s] [%(filename)s:%(lineno)d] [trace_id=%(otelTraceID)s span_id=%(otelSpanID)s resource.service.name=%(otelServiceName)s trace_sampled=%(otelTraceSampled)s] - %(message)s'
    datefmt: '%Y-%m-%d %H:%M:%S'

Now this is the code I am using to export logs from my application :

configure_azure_monitor()
tracer = trace.get_tracer(__name__)

logger_provider = LoggerProvider()
set_logger_provider(logger_provider)

log_exporter = AzureMonitorLogExporter()
logger_provider.add_log_record_processor(BatchLogRecordProcessor(exporter=log_exporter))
handler = LoggingHandler()

logging.getLogger().addHandler(handler)
logging.getLogger().setLevel(logging.INFO)

logger = logging.getLogger(__name__)

FastAPIInstrumentor.instrument_app(app)

I am passing the Application Insights connection string through my environment variables. Further this is how I start my FastAPI app with uvicorn.
python -m uvicorn my_api.main:app
      --host 0.0.0.0
      --port 443
      --reload
      --log-config logging-dev.yml

As you can see, we have different logs for different modules and and if we fetch by name you get the different logger.

What logger should be used to capture logs from all the modules and further be traced within a span under a tracer ?
Currently, I did try with the root logger and I was able to send correlated logs to Application Insights, but there were duplicates of each log statements in App as well as instrumented libraries in App-Insights . Attaching the screenshot for the same. Why so ?

What should be the strategy to send the logs of all the modules specified in the startup command using the logging-config.yml ?
How to provide Opentelemetry's loggingHandler in the logging-config.yml file ? Is configuring loggers through yml file supported?

@summahto
Let's try to address your issues one by one.

I did try with the root logger and I was able to send correlated logs to Application Insights, but there were duplicates of each log statements

First off, you are using BOTH the Azure monitor Distro and Azure monitor exporter, which is why you are seeing duplicate telemetry. I see you have commented on a separate issue on when to use which and this provides a good explanation. Please choose only one of them to use.

What logger should be used to capture logs from all the modules

If you want to capture all trace type telemetry from all modules, it is recommended you create a namespaced logger for each logger that you want to capture from. This avoids the problem from this issue where you will get infinite recursion because adding the handler to the root logger will also capture the logs sent from the SDK itself. If you do not want to create a separate namespaced logger for each library and you still want to use the root logger, or if you are not in control of the libraries that you want to capture telemetry for, you can achieve something like this with the suggestion made here.

and further be traced within a span under a tracer ?

You can correlate log telemetry with your spans created under the tracer if the logs are within the context of the span that you create.
...
with tracer.start_as_current_span("foo"):
    logger.warning("WARNING: Inside of span")   <-- trace telemetry will be correlated with dependency telemetry create from above line
...

If you want to correlate your log telemetry with an instrumented library call (in your case fast api), it should be automatically be correlated if you add the logging call in your request call.

@app.get("/")
async def test():
     logger.warning("fast api req made")
    return {"message": "Hello World"}

How to provide Opentelemetry's loggingHandler in the logging-config.yml file ? Is configuring loggers through yml file supported?

The OpenTelemetry LoggingHandler is just like any other Python logging handler so any configuration that supports Python logging handlers should work for OpenTelemetry  LoggingHandler as well.

@lzchen ,
Thank you so much for the detailed explanation. I was able to fix most of the above issues.
But, had a lot of trouble understanding how opentelemetry's data model for Trace, spans, logs, metrics fits into Applications Insights data model. Do we have a document which explains how each field maps from Opentelemetry's  data model to Applications Insights Data model.
I did go through this documentation provided here, but this does not explain everything. There are so many fields which are present in Opentelemetry that have no explanation here. For example a span, how is opentelemetry's span represented in Application Insights ? There is no mention of it. What happens to a span's fields like SpanKind, Span status etc. I see a lot of such fields missing in Application Insights.
Can you please help regarding this or maybe point me in the right direction ?

@summahto
There is no commonly accessed place where the mappings from OpenTelemetry to Application Insights are exposed to the public. Is there a reason you need to know this information? If this is regarding configuration of telemetry you can inquire about specific fields and we will imform you of how to populate/configure them, or you can refer to the official documentation for the exporter or the distro

@lzchen ,

Is there a reason you need to know this information?
Reason is pretty straightforward. I want to instrument my application as much as possible initially. So I have been setting these fields when I am creating spans but I am unable to see these fields in Application Insights. Hence, there is a loss of information.
Further, It will also help us understand which fields are supported and which are not. That way, I do not try to keep looking for some field which opentelemetry supports and I have set in my application code, but is never sent to Application Insights. This will save time.

@summahto
As mentioned above, there is no central publicly available mapping. If you want to be able to configure telemetry in app insights, please refer to this and this for most common situations. If you have more complicated scenarios, you can inquire them individually here.

authors: lzchen summahto


text:
Can't manage and request Just in Time access to VM using python SDK

No support for this API endpoint:
https://docs.microsoft.com/en-us/rest/api/securitycenter/jitnetworkaccesspolicies

Actually there is support in the azure-mgmt-security package :)
https://pypi.org/project/azure-mgmt-security/
I agree however there is no reference documentation for it, like the link you pushed for the RestAPI. I keep this one open until the reference doc for Python is online.
Thanks!

Ref doc is now online:
https://docs.microsoft.com/en-us/python/api/azure-mgmt-security/azure.mgmt.security?view=azure-python
Thanks!


text:
Private IP allocation for multiple VMs

Hi,
I am using the sdk to create multiple vms under the same resource group. The code for the same is as below. Despite of choosing the method of private ip allocation as dynamic,  it somehow seems to assign the same IP  for example :'10.1.0.4' to all the VMs. That does not work for my application as it uses private IPs to communicate internally amongst the multiple nodes. Also giving the address space and subnet address prefix is mandatory so I have to pass a range in and it somehow always chooses the same one from that range. Am I missing something or do I need to specify some parameter which chooses different IPs? Or do I need to configure the network interface any differently?
print ("Creating/Updating the Virtual Network")
address_space = AddressSpace(address_prefixes=['10.1.0.0/16'])
subnet1 = Subnet(name=subnet_name, address_prefix='10.1.0.0/24')
result = network_client.virtual_networks.create_or_update(group_name, network_name, VirtualNetwork(location=region, address_space=address_space, subnets=[subnet1]))
self.sleep_until_update_operation_done(result, network_name, verbose)
subnet = network_client.subnets.get(group_name, network_name, subnet_name)

print ("Creating/Updating the Public IP Address")
ip_address = PublicIPAddress(location=region, public_ip_allocation_method=IPAllocationMethod.dynamic, idle_timeout_in_minutes=4)
result = network_client.public_ip_addresses.create_or_update(group_name,ip_name, ip_address)
self.sleep_until_update_operation_done(result, ip_name, verbose)
public_ip_address = network_client.public_ip_addresses.get(group_name, ip_name)

print ("Creating/Updating the Network Interface")
network_interface_ip_conf = NetworkInterfaceIPConfiguration(name='default', private_ip_allocation_method=IPAllocationMethod.dynamic, subnet=subnet, public_ip_address=PublicIPAddress(id=(public_ip_address.id)))
result = network_client.network_interfaces.create_or_update(group_name, interface_name, NetworkInterface(location=region,ip_configurations=[network_interface_ip_conf]))
self.sleep_until_update_operation_done(result, interface_name, verbose)

Hi @tmarballi
Could you confirm which version of the SDK you use?
Thanks!

@lmazuel The latest one and that is 2.0.0rc5 I believe!

I just retested and it works just fine, if you provide the same subnet id for all the NIC you want in the same subnet.
Could you share the code that creates two VMs? Just to be sure you use correctly the same subnet id.

So I should be using the same network interface across VMs? I think I tried that before and I got an error while creating a VM because it said that this subnet / virtual network is already in use with another resource sort of an error. The code I attached above is a method called create_network_interface and I create a new network interface for each VM. Do you have any example of how the same subnet can be used. Because I remember trying that once but I got an error like I just mentioned.
for _ in range(count):
vm_network_name = Haikunator().haikunate()
self.create_network_interface(network_client, parameters,
vm_network_name, vm_network_name, vm_network_name, vm_network_name)
network_interface = network_client.network_interfaces.get(
resource_group, vm_network_name)
self.create_virtual_machine(credentials, network_client,
network_interface.id, parameters, vm_network_name)

Not the same network interface, the same subnet id.
Create your Virtual Network and your Subnet, and then use the same "subnet" object for each Network Interface you create.

Gotcha! I will try that and let you know! Thanks for your help!

Using the same Subnet object worked perfectly! Thanks a ton! Closing this issue!

authors: lmazuel tmarballi


text:
[AutoPR] policyinsights/resource-manager

Created to accumulate context: policyinsights/resource-manager

Codecov Report

Merging #2122 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2122   +/-   ##
=======================================
  Coverage   53.08%   53.08%
=======================================
  Files        4774     4774
  Lines      119517   119517
=======================================
  Hits        63440    63440
  Misses      56077    56077

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f262d80...dce3ab7. Read the comment docs.

@lmazuel For the first PR we had in swagger repo (https://github.com/Azure/azure-rest-api-specs/pull/2523), an automatic PR was created here https://github.com/Azure/azure-sdk-for-python/pull/2026. At the time there was an issue with folders and so we had decided to close it. swagger repo PR was merged with changes but looks like no new PR was created here at the time. Now we are releasing swagger specs for our new non-preview release as part of https://github.com/Azure/azure-rest-api-specs/pull/2929 which looks like triggered this PR. Two issues I'm trying to wrap my head around: 1. I see that the files generated are for our preview specs. What will happen now? 2. Build CI seems to have failed though Details link take me to 404. What is the issue?
Could you help? I'm looking at releasing Phyton SDK for our RP, either for our preview API version, or ideally for the new API version.

(message created by the CI based on PR content)

@bulentelmaci :

CI failed because your package is new, and packaging is not autogenerated (so no packaging, no CI). I just pushed the packaging info
This PR represents correctly the state of RestAPI specs @ master
Your open PR https://github.com/Azure/azure-rest-api-specs/pull/2929 has its own SDK PR here https://github.com/Azure/azure-sdk-for-python/pull/2452. Once your PR 2929 is merged, the content of #2452 will be merged here, making this PR still the correct state of RestAPI specs @ master

@lmazuel Thank you! So this PR (which reflects the state of things in specs master) will be merged when I test the package and make sure it is OK. Is there any other steps required? And 2452 will be merged when 2929 is merged?

Correct, 2452 will be merged automatically when 2929 will be merged. If you just want to test the GA version, you can add test to PR 2452, they will eventually come here.
When this PR 2122 represent exactly what you want to release, we merge and delete the branch. Next ti,e you touch the branch, the branch will be re-created from "master" with a new diff, and so on.

@lmazuel As soon as I'm done testing this, I will let you know (by Monday I think). Is there anything else needed that I should do before this PR is approved and merged?
When merged, I assume customers can install it from this GitHub? When would it go online in PyPI and when would it be part of azure package?

@bulentelmaci For your general questions, merge and PyPI are the same thing, I only merge when it's ready to be on PyPI. So customers will be able to install from Github or PyPI.

authors: AutorestCI bulentelmaci codecov-io lmazuel


text:
[AutoPR signalr/resource-manager] update CheckNameAvailability resource to add extra location in its route

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/3004

(message created by the CI based on PR content)

Codecov Report

Merging #2523 into restapi_auto_signalr/resource-manager will decrease coverage by 0.12%.
The diff coverage is n/a.

@@                            Coverage Diff                            @@
##           restapi_auto_signalr/resource-manager    #2523      +/-   ##
=========================================================================
- Coverage                                  54.98%   54.85%   -0.13%
=========================================================================
  Files                                       5618     5552      -66
  Lines                                     130539   129079    -1460
=========================================================================
- Hits                                       71779    70810     -969
+ Misses                                     58760    58269     -491

Impacted Files
Coverage Δ

...resource/managedapplications/models/application.py
35.71% <0%> (-8.74%)
:arrow_down:

...re/mgmt/resource/managedapplications/models/sku.py
41.66% <0%> (-8.34%)
:arrow_down:

...e/mgmt/resource/managedapplications/models/plan.py
45.45% <0%> (-7.88%)
:arrow_down:

...nagedapplications/models/application_definition.py
33.33% <0%> (-6.67%)
:arrow_down:

...tions/models/application_provider_authorization.py
62.5% <0%> (-4.17%)
:arrow_down:

...e/links/v2016_09_01/models/resource_link_filter.py
71.42% <0%> (-1.3%)
:arrow_down:

...ces/v2017_05_10/operations/resources_operations.py
50.25% <0%> (-0.56%)
:arrow_down:

...applications/operations/applications_operations.py
12.07% <0%> (-0.13%)
:arrow_down:

...s/operations/application_definitions_operations.py
13.06% <0%> (-0.07%)
:arrow_down:

...s/v2017_05_10/operations/deployments_operations.py
96.44% <0%> (-0.01%)
:arrow_down:

... and 85 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update be8ed5e...10b5161. Read the comment docs.

This PR has been merged into https://github.com/Azure/azure-sdk-for-python/pull/2413

authors: AutorestCI codecov-io


text:
[AutoPR containerinstance/resource-manager] Adding June Swagger Update - Azure Container Instance

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/3170

(message created by the CI based on PR content)

@AutorestCI rebase

Rebase done and pushed to the branch

This PR has been merged into https://github.com/Azure/azure-sdk-for-python/pull/2320

authors: AutorestCI lmazuel


text:
[Azure AMQP] Remove Deprecated SSL.Wrap_Socket

Closes #25434

This PR removes the deprecated ssl.wrap_socket and replaces it with SSLContext.wrap_socket
ssl_version also now uses either PROTOCOL_TLS_CLIENT or PROTOCOL_TLS_SERVER. PROTOCOL_TLS will be deprecated in py3.10 and its preferable to use the other values
Refactored some of the custom ssl  context code to make it cleaner and be the same across sync / async.

API change check
API changes are not detected in this pull request.

/azp run python - eventhub - perf

/azp run python - servicebus - perf

/azp run python - eventhub - tests

/azp run python - servicebus - tests

/azp run python - eventhub - tests

/azp run python - servicebus - tests

/azp run python - eventhub - tests

/azp run python - servicebus - tests

/azp run python - eventhub - tests

/azp run python - servicebus - tests

/azp run python - servicebus - tests

/azp run python - eventhub - tests

/azp run python - eventhub - tests

authors: azure-sdk kashifkhan


text:
[AutoPR policyinsights/resource-manager] typo: policyinsights/resource-manager/Microsoft.PolicyInsights/policyEvents

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/4535

This PR has been merged into https://github.com/Azure/azure-sdk-for-python/pull/3309


text:
Pipeline Witness classifier for Maven artifact downloads

We see transient build failures due to downloads of Maven artifacts. The most recent example was in the Spotbugs step where it was hitting Maven Central directly instead of using our proxy. That has been addressed, however even with using the proxy we can still see periodic issues.
We need a classifier which can detect these issues regardless of where they occur (could occur in any step that triggers a Maven goal) and look through the logs for the errors that indicate artifact download problems. It is possible that mucking with the Maven Wagon settings could help this but I want to first get a sense of magnitude of the problem and establish a baseline.

/cc @JimSuplizio @alzimmermsft

Here is a link to an instance of this issue to build the logic around:
https://dev.azure.com/azure-sdk/public/_build/results?buildId=479663&view=logs&j=2714c728-d141-5cd1-e4fc-49513190fa65&t=6afb2316-c245-59b5-7ebb-f53c7bea6c91&l=608
The build has been retained for analysis.

repo: Azure/azure-sdk-tools


text:
Hosting Hub as part of Azure SignalR Service - Serverless arch

Hi Team,
We would still need Hosting Service to Host the Hub along with Azure SignalR Service, Is it possible to host the my hub without any Host. I would just specify set of server side methods that can be called by clients? In my case, the clients are Mobile Apps they only interact with SignalR Service, no other Azure Service.
If it is not possible, what is the best approach to host my Hub?

@yasin-uohyd I think the function you are looking for is already in preview.
You can set a SignalR Service to Serverless:

thanks for update @faandrei .
Not, I was looking for creating a Hub and I do not have any App Server to Host. Is there anyway, I can host my custom hub logic without using App Server?

+1
@ferronsw What exactly does this feature Flag do?

@alexjebens  I couldn't find what it does exactly. I would create a separate issue if you want to know.

Hi @yasin-uohyd , I'm the Product Manager of Azure SignalR Service, thinking you might be interested in one of our new internal previewed features for supporting raw WebSocket in serverless scenarios. If you are interested to try it out and would like to bring us some feedback, please ping me at zhshang at microsoft dot com.  Thanks. -Zhidi

authors: alexjebens ferronsw sffamily yasin-uohyd

repo: Azure/azure-signalr


text:
Replace manually crafted REST client with auto-generated one

changed files can be categorized into three groups which are committed in three separate three commit:

src/Common/REST/*: files generated by Autorest
src\Microsoft.Azure.SignalR.Common\REST: enable generated REST client to authorize with JWT token.
other files: replace the old REST client

How about exclude generated code in this PR to make it small and clean?

How about exclude generated code in this PR to make it small and clean?

Generated code commits in Add REST client generated by Autorest, you can exclude this commit to view file changes.

This is a pretty large PR, how about separating it into several pieces:

Only contains the generated code, and the script file to generate it, and the doc describing the steps
The convenience layer (logic layer) leveraging the generated code
Replace the current REST client with the new one

I think Liangying suggested you split this PR into 3 PRs as above. Exclude some commit isn't convenient for reviewers, especially for those who don't see your comment.

This is a pretty large PR, how about separating it into several pieces:

Only contains the generated code, and the script file to generate it, and the doc describing the steps
The convenience layer (logic layer) leveraging the generated code
Replace the current REST client with the new one

I think Liangying suggested you split this PR into 3 PRs as above. Exclude some commit isn't convenient for reviewers, especially for those who don't see your comment.

Thanks for pointing out my misunderstanding. The first of the 3 PRs is #1007 .

authors: Y-Sindo wanlwanl


text:
Cannot find files when AzCopy from AzureStorage using --include-pattern option

Which version of the AzCopy was used?
10.10.0
Which platform are you using? (ex: Windows, Mac, Linux)
Windows Server 2016
What command did you run?
"azcopy.exe"   copy "https://XXX.blob.core.windows.net/interfacedata/SMD_INPUT/"   "D:\10_DataTransfer\PJ04\AzureStorage\SMD_INPUT"   --recursive=false  --include-pattern   "ABCED.xlsx"
What problem was encountered?
Azcopy can not fine file, when using --include-pattern option and if the file has same name with folder at same level.
202106_AzCopy_Download_Issue.xlsx
AzCopy_log_NG.log
AzCopy_log_OK.log
How can we reproduce the problem in the simplest way?

Set file and folder at same level in AzureStorage Container.  Please check attached .xlsx file.
Set required file in folder set at 1) step.
Do azcopy command from path 1)-folder and  assign "--include-pattern" with 2)-filename.

Have you found a mitigation/solution?
Yes, we delete the file has same name with folder, we can download required files.

@nao-830713 Having Virtual directories and flles with identical names at same level is not recommended. This scenario will not be supported.

repo: Azure/azure-storage-azcopy


text:
Fix ADLSG2 intermittent failure

When a ADLSG2 folder already exists with blobs under it and we attempt to overwrite the properties on it, the transfer will fail. As a result, we opt for a best-effort overwrite.

@adreed-msft was this issue reported on Github? If so can we link it here?

No-- This was detected through #1898.

Can we write a test for this?

Let's add some tests and get prioritize it to get merged in to get our pipelines stable

authors: adreed-msft gapra-msft tasherif-msft


text:
Fix/make tier processing consistent between copy and sync

Without this, there'll be a bug in that it will impossible to, for instance, sync between a standard blob storage account and a premium block blob account (because they don't have the same access tiers).
Note that there are two commits in this PR. Only the first is technically necessary to fix the bug. But the second reduces code duplication and should improve maintainability as we add more properties in future.  The second basically just takes common.NewCopyTransfer and makes it a method of storedObject so we don't have to pass heaps of parameters invididually

Should comfortably reduce some code dupe between sync and copy.

authors: JohnRusk adreed-msft


text:
Changes to support mac/linux

Changes:

Added generateProtos.sh (mostly a copy of the batch script)
Added a new Build target for Grpc project to support mac/linux
Changed how process info start starts the application (and how Node.js described itself)

@christopheranderson,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot

Small comment. Can we also have a more descriptive commit message?

Most of these files appear to be from a different WIP PR (Matt's PR)? What is the actual proposed change in this PR?

@mathewc I messed up the history with a rebase, the important stuff is 1521b2f which I'll pull into my branch

authors: christopheranderson dnfclas fabiocav mamaso mathewc

repo: Azure/azure-webjobs-sdk-script


text:
Easy Tables binding is confusing to Mobile Apps developers

In Functions, the "Easy Tables" binding uses the URL of a mobile app, which it looks like can be any table exposed by a App Service mobile app. In Mobile Apps, an "Easy Table" is usually a Node.js backend service, since Easy Tables in the portal doesn't support .NET backend.
This makes "Easy Tables" confusingly overloaded. Instead of Easy Tables we should call it Mobile Tables.

Closing this one, see related issue in script repo. https://github.com/Azure/azure-webjobs-sdk-templates/issues/108

Reactivating. Before I port this bug to the right repo (after failing the first time), we should just decide whether we are likely to take this change.

I was just reading this and it confused the hell out of me :)

@mathewc is it my description that is confusing? I can try to be more clear.

Here's an email thread where this is discussed with Adrian and Paul:
RE Easy Tables.pdf

I've created a new issue with a "Fix to this" here: https://github.com/Azure/azure-webjobs-sdk-templates/issues/139
We'll close this one if we decide to do the rename.

authors: christopheranderson ggailey777 mathewc paulbatum

repo: Azure/azure-webjobs-sdk-templates

license: mit

license_source: bigquery


text:
Add peering gateway opts

Expose additional peering configuration options to eliminate need to manually configure peering with existing vnets.

- Add config dictionary in ['vnet']['peer'] for gateway options
- Add config fields in ['vnet']['peer'][<peer-name>] for traffic options
- Add method to delete peerings with azhpc-destroy when defined in config.json
- Default configs remain unchanged
- Update to README.md to reflect changes with example of adding peering

Good point that I meant to bring up with the PR, is there a preferred way to add tests?

Good point that I meant to bring up with the PR, is there a preferred way to add tests?

Not really. We trust your tests

Okay, I'll add some testing.

How are tests run?  I see the ci dir, various bash scripts, what looks like CI yaml config files, but I'm not sure at all how this is actually run.

@xpillons I tested in a couple of different VPNs and it works as I expect.  However, an interesting side effect is that if you create a new VNet and create a peering to the another VNet that contains a VPN gateway with an active VPN connection, it drops the connection. You can immediately reconnect, but it is not ideal.  Do you have any idea about how to eliminate behavior?

I've seen this too and I don't know how to fix that. It's good to merge for me now.

Okay, thanks for confirming the behavior is not just limited to that.  I don't have permissions to merge, so you'll have to do that.

authors: xpillons yosoyjay

repo: Azure/azurehpc