text:
"Signature did not match" with UserDelegationCredential

Bug Report

import path of package in question

github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/sas
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/service

SDK version e.g. master, latest, 18.1.0

github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.3.2

output of go version

go version go1.22.0 darwin/arm64

What happened?
I fail to do a SAS upload to a signed URL

PUT https://studcteststorage.blob.core.windows.net/studctest/sas_upload_test.txt/ --------------------------------------------------------------------------------
RESPONSE 403: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
ERROR CODE: AuthenticationFailed --------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?>
<Error>
AuthenticationFailed
<Message>
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:b304660c-d01e-003d-0cce-9f3224000000 Time:2024-05-06T16:00:21.9356046Z
</Message>
<AuthenticationErrorDetail>
Signature did not match. String to sign used was w 2024-05-07T16:00:21Z /blob/studcteststorage/studctest/sas_upload_test.txt/ <RG-B Managed Identity Object ID> <Tenant ID> 2024-05-04T16:00:21Z 2024-05-07T16:00:21Z b 2023-11-03 https 2023-11-03 b
</AuthenticationErrorDetail>
</Error>

Note: I redacted <RG-B Managed Identity Object ID> and <Tenant ID> above.

What did you expect or want to happen?
I expected the SAS PUT request to work without issue. I am able to successfully write blobs to the container by calling methods like https://github.com/Azure/azure-sdk-for-go/blob/e13ea6e16ac0a4e20f167ce788d6fc5b5f017bcc/sdk/storage/azblob/client.go#L144
after creating a client like this

url := fmt.Sprintf("https://%s.blob.core.windows.net", accountName)
opts := &azidentity.WorkloadIdentityCredentialOptions{
	TenantID: *tenantID,
	ClientID: *rgBmanagedIdentityClientID,
}
cred, err := azidentity.NewWorkloadIdentityCredential(opts)
client, err := azblob.NewClient(url, cred, nil)

_, err = client.UploadStream(ctx, containerName, key, src, &azblob.UploadStreamOptions{
	Concurrency: 1,
	BlockSize:   1 << 20, // 1mb
})

How can we reproduce it?
Here is the code that gives the 403 error:

url := fmt.Sprintf("https://%s.blob.core.windows.net", accountName)
opts := &azidentity.WorkloadIdentityCredentialOptions{
	TenantID: *tenantID,
	ClientID: *rgBmanagedIdentityClientID,
}
cred, err := azidentity.NewWorkloadIdentityCredential(opts)
client, err := azblob.NewClient(url, cred, nil)

expiry := time.Now().Add(24 * time.Hour).UTC()
info := azblobService.KeyInfo{
	Start:  to.Ptr(time.Now().UTC().Add(-48 * time.Hour).Format(sas.TimeFormat)),
	Expiry: to.Ptr(expiry.Format(sas.TimeFormat)),
}

serviceClient := client.ServiceClient()
udc, err := serviceClient.GetUserDelegationCredential(ctx, info, nil)

sigVals := sas.BlobSignatureValues{
	Protocol:      sas.ProtocolHTTPS,
	ExpiryTime:    expiry,
	Permissions:   perm.String(),
	ContainerName: containerName,
	BlobName:      path,
}

query, err := sigVals.SignWithUserDelegation(udc)

signedUrl := fmt.Sprintf(
	"https://%s.blob.core.windows.net/%s/%s?%s",
	accountName,
	containerName,
	blobKey,
	query.Encode(),
)

sasClient, err := azblob.NewClientWithNoCredential(signedUrl, nil)
resp, err := sasClient.UploadStream(ctx, "", "", strings.NewReader("SAS upload test"), &azblob.UploadStreamOptions{})

Anything we should know about your environment.

My infrastructure setup is as follows:
I have a resource group (let's call it RG-A)with a k8s cluster, a managed identity, and other resources. In another resource group (say RG-B), I have only a managed identity and a storage account with a container.
I can successfully write blobs from an app running in the k8s cluster in RG-A to the storage container in RG-B. (Only the SAS is giving me trouble.) In RG-B, I gave Storage Account Contributor and Storage Blob Data Owner roles to the RG-B managed identity. Then, I created a federated credential that allows the k8s cluster in RG-A to auth to the managed identity in RG-B, by using the following command:
az identity federated-credential create \
  --name fc-x-tenant-test \
  --identity-name rg-B-managedIdentity \
  --resource-group rg-B \
  --issuer <RG-A AKS Cluster OIDC Provider URL> \
  --subject system:serviceaccount:default:<RG-A Service Account name>

Also, I logged the query with a previous version of the code (the only difference being I hadn't set the start time to 2 days in the past yet, it was only being set to 10 seconds in the past).
Query {
	version:2023-11-03
	services:
	resourceTypes:
	protocol:https
	startTime:{wall:0 ext:0 loc:<nil>}
	expiryTime:{wall:151009920 ext:63850466819 loc:<nil>}
	snapshotTime:{wall:0 ext:0 loc:<nil>}
	ipRange:{Start:[] End:[]}
	identifier:
	resource:b
	permissions:w
	signature:bDcn0MCs1kGakVX9OOtKt/ApBllv68AJLZnwZK4g+5g=
	cacheControl: contentDisposition: contentEncoding: contentLanguage: contentType:
	signedOID:<RG-B Managed Identity Object ID>
	signedTID:<Tenant ID>
	signedStart:{wall:0 ext:63850380409 loc:<nil>}
	signedService:b
	signedExpiry:{wall:0 ext:63850466809 loc:<nil>}
	signedVersion:2023-11-03
	signedDirectoryDepth: authorizedObjectID: unauthorizedObjectID: correlationID: encryptionScope: stTimeFormat: seTimeFormat:
}

Hi,
we are currently analyzing a similar issue in our productive environment where a blobstorage upload failed. Our best guess currently is, that the node time was off by a few seconds.
import path of package in question
	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob"
	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/bloberror"
	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob"

SDK version e.g. master, latest, 18.1.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.3.2
go version in prod is 1.22.1 linux/amd64
code where the error happens:
`blockblobClient, err := createBlockBlobUploadClient(destPath)
if err != nil {
return err
}
_, err = blockblobClient.UploadStream(ctx, data, &blockblob.UploadStreamOptions{})`

Here is the error we've encountered:
PUT https://xxxxxxxxxxxxxx.blob.core.windows.net/xxxxxxxxxxxxxx/xxxxxxxxxxxxxx/xxxxxxxxxxxxxx.zip\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nERROR CODE: AuthenticationFailed\n--------------------------------------------------------------------------------\n<?xml version=\"1.0\" encoding=\"utf-8\"?><Error>AuthenticationFailed<Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:a9ba6d7b-f01e-002b-5a84-9f2a94000000\nTime:2024-05-06T07:09:35.3537024Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request '+vWptkqk4dfaIzum/0fvvQjVBwTOaEAiGlu34TY9P2k=' is not the same as any computed signature. Server used following string to sign: 'PUT\n\n\n985458\n\napplication/octet-stream\n\n\n\n\n\n\nx-ms-date:Mon, 06 May 2024 07:09:35 GMT\nx-ms-version:2023-11-03\n/xxxxxxxxxxxxxx/xxxxxxxxxxxxxx/xxxxxxxxxxxxxx.zip\nblockid:wfCx0A6XRlxSWrV4EJ7X7AAAAA4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==\ncomp:block'.</AuthenticationErrorDetail></Error>\n--------------------------------------------------------------------------------\n

Hi @johannesvietze thank you for looking into this!!
I see that your AuthenticationErrorDetail says The MAC signature found in the HTTP request '+vWptkqk4dfaIzum/0fvvQjVBwTOaEAiGlu34TY9P2k=' is not the same as any computed signature..., while my AuthenticationErrorDetail says Signature did not match. String to sign used was....
Does that mean these are different causes of the 403?
Also,

Our best guess currently is, that the node time was off by a few seconds

what do you mean by  "node time"? I thought that there might be some clock skew causing the start time window to be after the actual time of the attempted upload, so I made it the Start time be a full 2 days in the past
info := azblobService.KeyInfo{
	Start:  to.Ptr(time.Now().UTC().Add(-48 * time.Hour).Format(sas.TimeFormat)),
	Expiry: to.Ptr(expiry.Format(sas.TimeFormat)),
}

and the issue still persists. (and from Signature did not match. String to sign used was w 2024-05-07T16:00:21Z /blob/studcteststorage/studctest/sas_upload_test.txt/ <RG-B Managed Identity Object ID> <Tenant ID> 2024-05-04T16:00:21Z 2024-05-07T16:00:21Z b 2023-11-03 https 2023-11-03 b, I can verify the start time was a full 2 days in the past)

@vibhansa-msft @tanyasethi-msft my issue was that I cannot specify the blob name in the BlobSignatureValues; however, I can specify it there when I use a SharedKeyCredential and sign with SignWithSharedKey instead of using a UserDelegationCredential

Hi @levinandrew , Thanks for reporting this.
While creating BlobSignatureValues, you can specify the BlobName to generate the SAS for an Azure Storage blob.
In the code above, while creating BlobSignatureValues,
Permissions:   perm.String()
What are the permissions given?
Also,
resp, err := sasClient.UploadStream(ctx, "", "", strings.NewReader("SAS upload test"), &azblob.UploadStreamOptions{})
The blob name and container name are "", "". Is that intentional, and why is it so?

repo: Azure/azure-sdk-for-go


text:
LocalNetworkGatewaysClient.Get doesn't validate parameters

In localnetworkgateways.go, the Get function doesn't validate parameters for special cases.
For example, if the value of localNetworkGatewayName is an empty string, the REST call won't fail, it actually becomes a different request:
expected request - .../providers/Microsoft.Network/localNetworkGateways/<name>?apiVersion=... This is to return the information of the specified LNG.
actual request - .../providers/Microsoft.Network/localNetworkGateways/?apiVersion= ... This is to list all LNGs from the resource group.
So the REST call won't fail, it returns 200 OK, with unexpected JSON body. Then autorest won't be able to unmarshall the JSON value into a proper LocalNetworkGateway object, and fails silently.
The result is that the Get function returns a LocalNetworkGateway object with all but the Response field set to nil.
I believe the correct behavior of the function is not to send the REST request if the  localNetworkGatewayName parameter is empty, and return an error instead.

Yup agreed

In order to fix this, we're going to need to put constraints in the Swagger that ensure that an empty localNetworkGatewaysID is not permissible. (Here is the relevant swagger).
Further more, it's pretty clear that this is a categorical problem. We should add a requirement that variables present in the path have a minimum length of 1. I'll file follow up bugs that will pop-up below.

That's correct Martin,
this is actaully implemented in some swaggers such as this one:
https://github.com/Azure/azure-rest-api-specs/blob/master/arm-storage/2016-12-01/swagger/storage.json#L69
We can create an issue in API Specs repo and tag it to networking as an improvement. but from what I'm seeing it's a common behavior across alot of swaggers to not include this, and maybe we should be adding this as a warning in the linter. @veronicagg and @ravbhatnagar might wanna take a look here :)

Beat you to it @salameer! I have already spun up a Linter issue and submitted a PR to add that min code. Next, we have to validate that the generator actually does anything with those criteria.

Chatting with @mcardosos offline, it sounds like our generator already supports parameter minLength validation.

My attention is still on this issue. We're currently working on chasing down the correct folks to approve Azure/azure-rest-api-specs#1088.

Specs repo updated. I'll mark this bug resolved, but keep it open until we can generate a new version of the API with the validation in it.

source: gharchive/issue

authors: marstr salameer whiskeyjay

license: MIT

license_type: permissive

license_source: github-api


text:
[BUG] Azure Monitor OpenTelemetry Exporter client not exporting to Azure

I have instrumented my Spring Boot application to export OpenTelemetry traces/metrics to an Azure Application Insights resource following the documentation here: https://learn.microsoft.com/en-us/java/api/overview/azure/monitor-opentelemetry-exporter-readme?view=azure-java-preview
I am providing the full Connection String from the resource which looks like this:
InstrumentationKey=XXXXXX-XXX-XXXX-XXX-XXX;IngestionEndpoint=https://eastus2-3.in.applicationinsights.azure.com/;LiveEndpoint=https://eastus2.livediagnostics.monitor.azure.com/
This is my code snippet to configure the AzureMonitorExporterBuilder:
final AzureMonitorExporterBuilder azureMonitorExporterBuilder = new AzureMonitorExporterBuilder().connectionString("InstrumentationKey=<above instumentation key>;IngestionEndpoint=https://eastus2-3.in.applicationinsights.azure.com/;LiveEndpoint=https://eastus2.livediagnostics.monitor.azure.com/");

However, when the actual exporting is taking place I see this URL being created:
https://dc.services.visualstudio.com/v2.1/track
Debugging the code in my IDE I believe this is happening because of the following:
The AzureMonitorExporterBuilder.buildTraceExporter() method internally creates a SpanDataMapper.
https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/monitor/azure-monitor-opentelemetry-exporter/src/main/java/com/azure/monitor/opentelemetry/exporter/AzureMonitorExporterBuilder.java#L228
The 2nd parameter of the SpanDataMapper's constructor is a BiConsumer<AbstractTelemetryBuilder, Resource> telemetryInitializer, which references the populateDefaults method in the AzureMonitorExporterBuilder.
This is creating a new connectionString using only the instrumentationKey and not the other properties of the full connectionString passed into the AzureMonitorExporterBuilder. The other properties (e.g. IngestionEndpoint) get populated by defaults in ConnectionStringBuilder and the new ConnectionString object ends up being cached with the key being the InstrumentationKey and value containing the InstrumentationKey and the incorrect defaults.
The TelemetryItem that ends up being exported contains a reference to the InstrumentationKey and when the TelemetryPipeline.send method is being called it creates the URL from the cached ConnectionString and then tries to export to the incorrect URL.
I believe the code change required would be to create a connectionString where the key is the instrumentationKey and the value is all of the connection properties provided by the AzureMonitorExporterBuilder and should never reference the defaults in ConnectionStringBuilder.
https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/monitor/azure-monitor-opentelemetry-exporter/src/main/java/com/azure/monitor/opentelemetry/exporter/implementation/configuration/ConnectionStringBuilder.java#L30-L32

@trask @lmolkova could you please follow up

hi @tyocum17, thanks for reporting this!
I've sent a PR that (among other things) fixes this: #32454

this is fixed in the latest release

authors: joshfree trask tyocum17

repo: Azure/azure-sdk-for-java


text:
Release Azure.Identity Library Preview

Goal:
Release the Azure.Identity library
Integrate this library in the other client libraries.

[ ] Gap Analysis of MSAL
[ ] Provide Feedback to MSAL Team and schedule updates
[ ] Finalize Azure.Identity footprint for SDK Preview
[x] Azure.Core Updates

[x] Credential (base for Token, SAS, etc)
[x] TokenCredential
[x] CredentialPolicy

[ ] Azure.Identity

[ ] Classes

[ ] AzureCredential
[ ] ClientSecretCredential
[ ] ClientCertificateCredential
[ ] MSICredential
[ ] TokenCredentialProvider
[ ] EnvironmentCredentialProvider
[ ] MSICredentialProvider

[ ] Implementations

[ ] ClientSecretCredential
[ ] ClientCertificateCredential
[ ] ManagedIdentityCredential

[ ] Investigate other authentication schemes we need to support (user auth, auth from shared token cache)

Closing Preview 1 (July Preview) Epic

authors: AlexGhiondea joshfree


text:
Create Gitter community channel for azure-sdk-for-java repo

Someone from EngSys with appropriate permissions to https://github.com/Azure, please click the following buttons

go to https://gitter.im/home#createroom
Community select Azure
Room add azure-sdk-for-java

Then add the following to the root Readme.md in azure-sdk-for-java under ### Community
* Chat with other community members [![Join the chat at https://gitter.im/azure/azure-sdk-for-java](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/azure/azure-sdk-for-java?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
Thanks

/cc @jeffwilcox

Yeah, this appears to require Azure org (on GitHub) admin rights so @jeffwilcox is the right person to ask.

Closing as community gitter channels have been created for each of the azure/azure-sdk-for-[language] repos.

authors: joshfree kurtzeborn


text:
Follow-up on potential codegen issues

Base classes should be abstract. E.g.: Analyzer
MagnitudeScoringParameters -> Need to fix isShouldBoostBeyondRangeByConstant because it makes no sense. Same with SuggestOptions.isUseFuzzyMatching. Same with various Boolean properties of WordDelimiterTokenFilter.
Suggester -- getSearchMode and setSearchMode should not be exposed, at least at an instance level. It certainly shouldn't be mutable. Ditto for SynonymMap's format property. In C#, these are static immutable properties.

Will follow up if it needs to fit in Preview 1

Double check with @jianghaolu

Base classes should be abstract. E.g.: Analyzer
Swagger does not support the feature. If this is necessary, then we need to do it in transform layer.
@brjohnstmsft is there any special benefit to make it analyzer?

MagnitudeScoringParameters -> Need to fix isShouldBoostBeyondRangeByConstant because it makes no sense. Same with SuggestOptions.isUseFuzzyMatching. Same with various Boolean properties of WordDelimiterTokenFilter.

Auto-gen makes it by design. If the boolean field has "is" as prefix, then do nothing. If no "is" prefix, add "is" for the field.
@brjohnstmsft Two options to address this:

Rename the property which is meaningful with "is" Prefix

Use transformer to change the name.

Suggester -- getSearchMode and setSearchMode should not be exposed, at least at an instance level. It certainly shouldn't be mutable. Ditto for SynonymMap's format property. In C#, these are static immutable properties.

For make search mode immutable, we can change the field to 'read-only' in swagger.
Then auto-gen will generate getSearchMode only.

Swagger does not support static field for object, so we cannot simply make it static (only in transform). Also, we don't think it is necessary to the field static. If we want it to be immutable, then mark it as 'read-only' in swagger.

@sima-zhu I don't understand this question:

@brjohnstmsft is there any special benefit to make it analyzer?

Did you mean abstract? If so, the benefit is that customers won't attempt to instantiate the class directly, which leads to serialization errors if they send a base class instance in a request.

@sima-zhu

Auto-gen makes it by design. If the boolean field has "is" as prefix, then do nothing. If no "is" prefix, add "is" for the field.

That's a bad design. 😉
I would rather not rename the properties to work around a bad codegen design choice, when the names we chose are (IMO) very clear. I'd suggest using transforms to fix this if that's what it takes.

@sima-zhu

Suggester -- getSearchMode and setSearchMode should not be exposed, at least at an instance level. It certainly shouldn't be mutable. Ditto for SynonymMap's format property. In C#, these are static immutable properties.

For make search mode immutable, we can change the field to 'read-only' in swagger.
Then auto-gen will generate getSearchMode only.
Swagger does not support static field for object, so we cannot simply make it static (only in transform). Also, we don't think it is necessary to the field static. If we want it to be immutable, then mark it as 'read-only' in swagger.

I don't think read-only is accurate in this case, since the parameters actually do need to be sent to the REST API. The issue is that these parameters only support one value, so it's redundant to require customers to send this value every time. The C# code generator solves this problem by emitting a static read-only property. I'm not too opinionated about how the Java codegen solves this problem, as long as it's solved, because customers shouldn't have to think about these parameters (at least until we add more values to them).
I realize we should never have exposed these parameters in the REST API until they had more than one possible value, but it's too late to fix that now.

For boolean naming:

AutocompleteOptions/AutocompleteRequest/SuggestOptions/SuggestRequest: isUseFuzzyMatching
MagnitudeScoringParameters: isShouldBoostBeyondRangeByConstant
WordDelimiterTokenFilter
CommonGramTokenFilter: isUseQueryMode
DictionaryDecompounderTokenFilter: isOnlyLongestMatch
EntityRecognitionSkill: isIncludeTypelessEntities
KeywordMarkerTokenFilter: isIgnoreCase
OcrSkill: isShouldDetectOrientation
PathHierarchyTokenizer: isReverseTokenOrder
PhoneticTokenFilter: isReplaceOriginalTokens
SearchOptions: isIncludeTotalResultCount
SearchRequest: isIncludeTotalResultCount
StopwordsTokenFilter: isIgnoreCase/isRemoveTrailingStopWords
SynonymTokenFilter: isIgnoreCase/isExpend

authors: AlexGhiondea brjohnstmsft sima-zhu


text:
Pipeline Trigger Changes

This PR updates the Azure Core pipeline to trigger when the SDK parent and the client SDK parent POM files are changed and adds a pipeline to run linting when changes are made to code-quality-reports.

If you remove the code-quality-reports/ci.yml file I can approve this and that'll take care of adding our parents to the triggers for the core pipeline.

source: gharchive/pull-request

authors: alzimmermsft mitchdenny


text:
Fix DateTimeFormatter Pattern and Jacoco Coverage Configuration

This PR fixes the DateTimeFormatter pattern to use the expected yyyy instead of YYYY. Additionally, it fixes Jacoco code coverage configuration properties and cleans up build configuration that was duplicated.

/azp run java - core

/azp run java - core

/check-enforcer override

Overriding check enforcer as all pipelines are passing other than a well-known blocking pipeline which is being investigated separately in this PR, https://github.com/Azure/azure-sdk-for-java/pull/18390


text:
Sync code to spring boot 3 weekly

This is a weekly job to sync non spring code from branch main into branch feature/spring-boot-3.

API change check
API changes are not detected in this pull request.

authors: azure-sdk moarychan


text:
Increment versions for storage releases

Increment package versions for storage releases

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-storage-common


text:
Improve Spring Monitor readme

cc @mattmccleary

API change check
API changes are not detected in this pull request.

authors: azure-sdk jeanbisutti


text:
[Aggregate-reports] Link verification check failing in nightly runs

Aggregate-reports nightly test runs are failing with:

Error message:
Summary of broken links:
'file:///D:/a/_work/1/s/documentation/test-quickstart.md' has 1 broken link(s):
https://docs.microsoft.com/en-us/azure/active-directory/develop/app-objects-and-service-principals
Checked 19436 links with 1 broken link(s) found.
##[error]PowerShell exited with code '1'.

For more details check here:

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1651151&view=results

Label prediction was below confidence level 0.6 for Model:ServiceLabels: 'Docs:0.41295305,Azure.Core:0.20296557,Azure.Identity:0.038878124'

@qiaozha could you have a look?

This issue should be fixed by PR #22464  and https://github.com/Azure/azure-rest-api-specs/pull/19682

authors: azure-sdk jeremymeng qiaozha v-jiaodi

repo: Azure/azure-sdk-for-js


text:
Missing reference docs for Preview packages for Keyvault Certificates, Storage File Shares, Event Hubs

This issue is to track the missing reference docs for the below 3 libraries that are currently in preview

@azure/keyvault-certificates 4.0.0-preview.9
@azure/storage-file-share 12.0.0-preview.6
@azure/event-hubs 5.0.0-preview.6
@azure/eventhubs-checkpointstore-blob 1.0.0-preview.4
@azure/app-configuration 1.0.0-preview.7

While certificates & file-shares dont have a presence in the preview toc tree at all, the node for event hubs contains Track 1 references

The preview docs at https://docs.microsoft.com/en-us/javascript/api/overview/azure/?view=azure-node-preview now have the right nodes with the right content


text:
[Identity] update mapping for browser for azd

Packages impacted by this PR
@azure/identity
Issues associated with this PR
Bug fixes
Describe the problem that is addressed by this PR
What are the possible designs available to address the problem? If there are more than one possible design, why was the one in this PR chosen?
Are there test cases added in this PR? (If not, why?)
Provide a list of related PRs (if any)
Command used to generate this PR:**(Applicable only to SDK release request PRs)
Checklists

[ ] Added impacted package name to the issue description
[ ] Does this PR needs any fixes in the SDK Generator?** (If so, create an Issue in the Autorest/typescript repository and link it here)
[ ] Added a changelog (if necessary)

API change check
API changes are not detected in this pull request.

@tyteen4a03 Yes, I will release the official fix for it this week in 3.3.2

authors: KarishmaGhiya azure-sdk


text:
[QUESTION] Can Microsoft.Azure.EventHubs.ServiceFabricProcessor be safely use?

Query/Question
I found out something like Microsoft.Azure.EventHubs.ServiceFabricProcessor. But it seems, that documentation around it has gone, and only nuget package and source code remains.
Can I safely use it?
Any documentation anywhere?

//cc: @jamesbirdsall

Any update? :)

ServiceFabricProcessor is still officially in preview, and there are some known issues and API changes we would like to make before it goes GA. That said, some customers are using it, and if someone using it hits a stopper, we will fix it with priority.

authors: JamesBirdsall darthkurak jsquire

repo: Azure/azure-sdk-for-net


text:
Event Hubs backlog

The backlog for Event Hubs issues.

Closing as part of the migration from epics => GH projects

authors: AlexGhiondea jsquire


text:
[Documentation] Rename BoundingBox sample to make it Sample9

Due to different sample PRs being merged close to each other, two unrelated samples are numbered as Sample8. We must update the BoundingBox sample to be Sample9, and update documentation accordingly.

Fixed in PR #12971

authors: kinelski maririos


text:
Azure.Identity.CredentialUnavailableException

@abethan commented on Sun Dec 06 2020
    public Startup(IHostingEnvironment env)
    {
        var localConfig = new ConfigurationBuilder()
            .SetBasePath(env.ContentRootPath)
            .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
            .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
            .AddEnvironmentVariables();

        TokenCredential tokenCredential = new DefaultAzureCredential();

        if (env.IsDevelopment())
        {
            localConfig.AddUserSecrets(Assembly.Load(new AssemblyName(env.ApplicationName)));
            tokenCredential = new DefaultAzureCredential(new DefaultAzureCredentialOptions { VisualStudioTenantId = "<tenantId>" });
        }

        var localConfigBuild = localConfig.Build();

        Configuration = new ConfigurationBuilder()
            .AddConfiguration(localConfigBuild)
            .AddAzureAppConfiguration(options => options.Connect(new Uri(localConfigBuild["AzureConfiguration:AppConfigurationEndPoint"]), tokenCredential)
                                                        .ConfigureKeyVault(x => x.SetCredential(tokenCredential))
                                                        .ConfigureClientOptions(clientOptions => clientOptions.Retry.MaxRetries = 5)).Build();
    }

I'm getting the below exception with this code
Azure.Identity.CredentialUnavailableException: 'Process "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\b1hwzg3d.j4s\TokenService\Microsoft.Asal.TokenService.exe" has failed to get access token in 30 seconds.'
I have got 2 accounts inside the Visual Studio

But I have assigned the correct AD tenant in the Options.

Please help me to fix this issue. Thanks.

Hi @abethan.  Thank you for your feedback and we regret that you're experiencing difficulties.  Tagging and routing to the team member best able to assist.

Hi @abethan - I have a few questions:

Can you confirm which version of VS 2019 you are using?
Does it reproduce with the latest version?
Does it reproduce if you only have one account registered?

authors: christothes jsquire zhenlan


text:
Azure service bus .NET sdk raises OperationCanceledException from time to time

Hi all,
I'm working with a .NET core 3.1 worker service application.  We are referencing the Microsoft.Azure.ServiceBus nuget package version 5.1.0. The application is deployed as a pod inside a kubernetes cluster. We use Azure Kubernetes Service (AKS) to manage our cluster.
This application is meant to listen to messages published to an Azure Service Bus topic by another application. When the application starts up a subscription is created on the existing Azure Service Bus topic and a  SubscriptionClient instance is created in order to listen to incoming messages.
Our subscription client is configured as follows:

the receive mode is ReceiveMode.ReceiveAndDelete
the retry policy is RetryPolicy.Default
we register a message handler by using the SubscriptionClient.RegisterMessageHandler method
we set AutoComplete = true in the MessageHandlerOptions
we set a default value of 10 for the MaxConcurrentCalls property in the MessageHandlerOptions
we provide an exceptionReceivedHandler in the constructor of class MessageHandlerOptions which simply logs a warning log message each time an error occurs while receiving messages.

We noticed that, from time to time, our application raises some OperationCanceledException while receiving messages from the Azure Service Bus topic.
In order to better understand what's going on I followed this documentation to add additional application-level logs in order to better understand the issue.
First of all I noticed that, before throwing the OperationCanceledException, the Microsoft.Azure.ServiceBus.Receive operation works fine every time it is performed. This is a sample log that our application writes before the error appears (this is one of the additional logs that I added):

Operation "Microsoft.Azure.ServiceBus.Receive" is finished, Duration=00:00:59.9991789, Status=RanToCompletion, Id="00-9cd1f0d0190fb34381fe58b8e06191ec-00ac27ca7a505a4b-00", StartTime="2021-01-17T13:39:51.4380719Z"

Suddenly the error occurs and we get the following error log (this is one of the additional logs that I added):

Operation "Microsoft.Azure.ServiceBus.Receive" with id "00-fe45ce4ba807fb45af16386641cd86ce-2786cf7eab4ba24c-00" started at "2021-01-17T13:39:56.4920237Z" raised an error: "The operation was canceled.Reference: 790ae7c96ae64a65861fbc662626fb07_G8, 01/17/2021 13:40:54"

Microsoft.Azure.ServiceBus.ServiceBusCommunicationException: The operation was canceled.Reference: 790ae7c96ae64a65861fbc662626fb07_G8, 01/17/2021 13:40:54
---> System.OperationCanceledException: The operation was canceled.
at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.Azure.Amqp.ReceivingAmqpLink.EndReceiveMessages(IAsyncResult result, IEnumerable1& messages)    at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown ---    at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnReceiveAsync(Int32 maxMessageCount, TimeSpan serverWaitTime)    --- End of inner exception stack trace ---    at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnReceiveAsync(Int32 maxMessageCount, TimeSpan serverWaitTime)    at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<>c__DisplayClass65_0.<<ReceiveAsync>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func1 operation, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(Int32 maxMessageCount, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.MessageReceivePump.b__12_0()

The exact same error is also logged as a warning by the exceptionReceivedHandler delegate that we pass to the constructor of the MessageHandlerOptions class for our subscription client message handler:

Azure Notification Bus Message handler encountered an exception
Exception context for troubleshooting:

Endpoint: ***** REDACTED *****
Entity Path: ***** REDACTED *****
Executing Action: Receive

Microsoft.Azure.ServiceBus.ServiceBusCommunicationException: The operation was canceled.Reference: 790ae7c96ae64a65861fbc662626fb07_G8, 01/17/2021 13:40:54
---> System.OperationCanceledException: The operation was canceled.
at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.Azure.Amqp.ReceivingAmqpLink.EndReceiveMessages(IAsyncResult result, IEnumerable1& messages)    at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown ---    at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnReceiveAsync(Int32 maxMessageCount, TimeSpan serverWaitTime)    --- End of inner exception stack trace ---    at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnReceiveAsync(Int32 maxMessageCount, TimeSpan serverWaitTime)    at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<>c__DisplayClass65_0.<<ReceiveAsync>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func1 operation, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(Int32 maxMessageCount, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.MessageReceivePump.b__12_0()

At the this point we have another log, which notifies us that this time the Microsoft.Azure.ServiceBus.Receive has failed (this is one of the additional logs that I added):

Operation "Microsoft.Azure.ServiceBus.Receive" is finished, Duration=00:00:57.7255408, Status=Faulted, Id="00-fe45ce4ba807fb45af16386641cd86ce-2786cf7eab4ba24c-00", StartTime="2021-01-17T13:39:56.4920237Z"

After a couple of failed Microsoft.Azure.ServiceBus.Receive operations (both of them creates the exact logs sequence that I have described above) things work properly again. The following Microsoft.Azure.ServiceBus.Receive operations complete without errors:

Operation "Microsoft.Azure.ServiceBus.Receive" is finished, Duration=00:01:00.1531469, Status=RanToCompletion, Id="00-487f0fa5352c6545a7938da9a13c405a-4c77146b1efa4646-00", StartTime="2021-01-17T13:39:56.5046654Z"

We have noticed that this kind of errors are much more frequent when we deploy our application inside of kubernetes. For some other customers the same application is deployed as a web job inside an Azure App Service: in that case this kind of errors are much less frequent.
Usually in the .NET world OperationCanceledException means a timeout due to a canceled CancellationToken. This seems to be contradicted by the above log messages: as you can see the duration of the faulted  Microsoft.Azure.ServiceBus.Receive operation (00:00:57.7255408) is shorter than the duration of the successful one (00:00:59.9991789 and 00:01:00.1531469). This seems really strange to me.
Can you please help us to understand what is the root cause of this errors ?
Kind regards.
Enrico.

Thank you for your feedback.  Tagging and routing to the team best able to assist.

Thank you for your feedback.  Tagging and routing to the team best able to assist.

//fyi:  @JoshLove-msft

//fyi:  @JoshLove-msft

@jsquire many thanks. Let me know if you need any further detail to carry on the investigation.

@jsquire many thanks. Let me know if you need any further detail to carry on the investigation.

@EnricoMassone
This may be caused due to disruptions in the connection between the containers in Kubernetes and Azure Service Bus. This is expected behavior and the SDK is designed to retry to recover from this.

@EnricoMassone
This may be caused due to disruptions in the connection between the containers in Kubernetes and Azure Service Bus. This is expected behavior and the SDK is designed to retry to recover from this.

@EnricoMassone
This may be caused due to disruptions in the connection between the containers in Kubernetes and Azure Service Bus. This is expected behavior and the SDK is designed to retry to recover from this.

Hi and thanks for helping.
So basically we can assume that these errors are harmless, correct?
The main point for us is being sure that no messages will be lost from the worker service. What happens if a message is published during these periods of lost connectivity? Is the application able to get the message once the connectivity is recovered?
Is there anything that we can check or do at the AKS level in order to mitigate these connectivity errors?

@EnricoMassone
This may be caused due to disruptions in the connection between the containers in Kubernetes and Azure Service Bus. This is expected behavior and the SDK is designed to retry to recover from this.

Hi and thanks for helping.
So basically we can assume that these errors are harmless, correct?
The main point for us is being sure that no messages will be lost from the worker service. What happens if a message is published during these periods of lost connectivity? Is the application able to get the message once the connectivity is recovered?
Is there anything that we can check or do at the AKS level in order to mitigate these connectivity errors?

@axisc this morning I have checked the logs again and I have found another type of errors.
These errors come from a different pod of our kubernetes cluster. This component is a .NET core 2.2 console application.  Apart from the different .NET core version everything else is the same as the aforementioned worker service application: we reference the same version of the Microsoft.Azure.ServiceBus NUGET package (5.1.0), we setup a SubscriptionClient instance where we register a message handler by calling SubscriptionClient.RegisterMessageHandler (all the settings for the subscription client are the same as described above).
Here is the error message and stacktrace that we log for these new errors:

Operation "Microsoft.Azure.ServiceBus.Receive" with id "00-e8a50f7c1984f24f80b3d93fc00d2a99-51b4867f88bcb247-00" started at "2021-01-19T08:51:01.9020027Z" raised an error: "The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'fe050101-102d-4ac7-bd43-3d314ca2cbdc'.Reference: 80d5dc8c897745ff82c6687b79096042_G7, 01/19/2021 08:52:07"

Microsoft.Azure.ServiceBus.ServiceBusCommunicationException: The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'fe050101-102d-4ac7-bd43-3d314ca2cbdc'.Reference: 80d5dc8c897745ff82c6687b79096042_G7, 01/19/2021 08:52:07 ---> Microsoft.Azure.ServiceBus.ServiceBusException: The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'fe050101-102d-4ac7-bd43-3d314ca2cbdc'.
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnReceiveAsync(Int32 maxMessageCount, TimeSpan serverWaitTime)
--- End of inner exception stack trace ---
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnReceiveAsync(Int32 maxMessageCount, TimeSpan serverWaitTime)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<>c__DisplayClass65_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func1 operation, TimeSpan operationTimeout)    at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func1 operation, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(Int32 maxMessageCount, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.MessageReceivePump.b__12_0()

The exact same error is also logged as a warning by the exceptionReceivedHandler delegate that we pass to the constructor of the MessageHandlerOptions class for our subscription client message handler:

Azure Notification Bus Message handler encountered an exception
Exception context for troubleshooting:

Endpoint: *** REDACTED ***
Entity Path: *** REDACTED ***
Executing Action: Receive

Microsoft.Azure.ServiceBus.ServiceBusCommunicationException: The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'fe050101-102d-4ac7-bd43-3d314ca2cbdc'.Reference: 80d5dc8c897745ff82c6687b79096042_G7, 01/19/2021 08:52:07 ---> Microsoft.Azure.ServiceBus.ServiceBusException: The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'fe050101-102d-4ac7-bd43-3d314ca2cbdc'.
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnReceiveAsync(Int32 maxMessageCount, TimeSpan serverWaitTime)
--- End of inner exception stack trace ---
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnReceiveAsync(Int32 maxMessageCount, TimeSpan serverWaitTime)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<>c__DisplayClass65_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func1 operation, TimeSpan operationTimeout)    at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func1 operation, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(Int32 maxMessageCount, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.MessageReceivePump.b__12_0()

The pattern seems to be the same of the errors described in my original question. Before the exceptions pop up everything seems to work fine inside the kubernetes pod:

Operation "Microsoft.Azure.ServiceBus.Receive" is finished, Duration=00:01:12.3908558, Status=RanToCompletion, Id="00-33b4323ad1e4ea4196298f4f8e992e7e-506b29dfa0063242-00", StartTime="2021-01-19T08:50:53.5191260Z"

After the two error logs reported above we get an information log which notifies the failure of the Microsoft.Azure.ServiceBus.Receive operation:

Operation "Microsoft.Azure.ServiceBus.Receive" is finished, Duration=00:01:06.0124836, Status=Faulted, Id="00-e8a50f7c1984f24f80b3d93fc00d2a99-51b4867f88bcb247-00", StartTime="2021-01-19T08:51:01.9020027Z"

Again it seems that the pod is able to automatically recover the connectivity to the Azure service bus subscription. This log is written after the previous one:

Operation "Microsoft.Azure.ServiceBus.Receive" is finished, Duration=00:01:02.5989493, Status=RanToCompletion, Id="00-1a6168d50a4f4145aa619949d31851fc-96f12be1bbd1f14c-00", StartTime="2021-01-19T08:51:10.0042316Z"

@axisc are these errors of the same nature of the ones reported in my original question ? This time the error message seems to clearly indicate a timeout / communication issue between the kubernetes cluster and the Azure service bus server.

@axisc this morning I have checked the logs again and I have found another type of errors.
These errors come from a different pod of our kubernetes cluster. This component is a .NET core 2.2 console application.  Apart from the different .NET core version everything else is the same as the aforementioned worker service application: we reference the same version of the Microsoft.Azure.ServiceBus NUGET package (5.1.0), we setup a SubscriptionClient instance where we register a message handler by calling SubscriptionClient.RegisterMessageHandler (all the settings for the subscription client are the same as described above).
Here is the error message and stacktrace that we log for these new errors:

Operation "Microsoft.Azure.ServiceBus.Receive" with id "00-e8a50f7c1984f24f80b3d93fc00d2a99-51b4867f88bcb247-00" started at "2021-01-19T08:51:01.9020027Z" raised an error: "The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'fe050101-102d-4ac7-bd43-3d314ca2cbdc'.Reference: 80d5dc8c897745ff82c6687b79096042_G7, 01/19/2021 08:52:07"

Microsoft.Azure.ServiceBus.ServiceBusCommunicationException: The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'fe050101-102d-4ac7-bd43-3d314ca2cbdc'.Reference: 80d5dc8c897745ff82c6687b79096042_G7, 01/19/2021 08:52:07 ---> Microsoft.Azure.ServiceBus.ServiceBusException: The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'fe050101-102d-4ac7-bd43-3d314ca2cbdc'.
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnReceiveAsync(Int32 maxMessageCount, TimeSpan serverWaitTime)
--- End of inner exception stack trace ---
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnReceiveAsync(Int32 maxMessageCount, TimeSpan serverWaitTime)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<>c__DisplayClass65_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func1 operation, TimeSpan operationTimeout)    at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func1 operation, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(Int32 maxMessageCount, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.MessageReceivePump.b__12_0()

The exact same error is also logged as a warning by the exceptionReceivedHandler delegate that we pass to the constructor of the MessageHandlerOptions class for our subscription client message handler:

Azure Notification Bus Message handler encountered an exception
Exception context for troubleshooting:

Endpoint: *** REDACTED ***
Entity Path: *** REDACTED ***
Executing Action: Receive

Microsoft.Azure.ServiceBus.ServiceBusCommunicationException: The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'fe050101-102d-4ac7-bd43-3d314ca2cbdc'.Reference: 80d5dc8c897745ff82c6687b79096042_G7, 01/19/2021 08:52:07 ---> Microsoft.Azure.ServiceBus.ServiceBusException: The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'fe050101-102d-4ac7-bd43-3d314ca2cbdc'.
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnReceiveAsync(Int32 maxMessageCount, TimeSpan serverWaitTime)
--- End of inner exception stack trace ---
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.OnReceiveAsync(Int32 maxMessageCount, TimeSpan serverWaitTime)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.<>c__DisplayClass65_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func1 operation, TimeSpan operationTimeout)    at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func1 operation, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(Int32 maxMessageCount, TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.Core.MessageReceiver.ReceiveAsync(TimeSpan operationTimeout)
at Microsoft.Azure.ServiceBus.MessageReceivePump.b__12_0()

The pattern seems to be the same of the errors described in my original question. Before the exceptions pop up everything seems to work fine inside the kubernetes pod:

Operation "Microsoft.Azure.ServiceBus.Receive" is finished, Duration=00:01:12.3908558, Status=RanToCompletion, Id="00-33b4323ad1e4ea4196298f4f8e992e7e-506b29dfa0063242-00", StartTime="2021-01-19T08:50:53.5191260Z"

After the two error logs reported above we get an information log which notifies the failure of the Microsoft.Azure.ServiceBus.Receive operation:

Operation "Microsoft.Azure.ServiceBus.Receive" is finished, Duration=00:01:06.0124836, Status=Faulted, Id="00-e8a50f7c1984f24f80b3d93fc00d2a99-51b4867f88bcb247-00", StartTime="2021-01-19T08:51:01.9020027Z"

Again it seems that the pod is able to automatically recover the connectivity to the Azure service bus subscription. This log is written after the previous one:

Operation "Microsoft.Azure.ServiceBus.Receive" is finished, Duration=00:01:02.5989493, Status=RanToCompletion, Id="00-1a6168d50a4f4145aa619949d31851fc-96f12be1bbd1f14c-00", StartTime="2021-01-19T08:51:10.0042316Z"

@axisc are these errors of the same nature of the ones reported in my original question ? This time the error message seems to clearly indicate a timeout / communication issue between the kubernetes cluster and the Azure service bus server.

The first set of errors you saw were because of connection disruption. The messages are not "lost", because the service bus will not ack it and the client side SDK will retry the operation. The client will only proceed to the next message after the current message is successfully ACKed by Service Bus.
The second set of errors you saw are due to an idle connection. In Azure Service Bus, connections are stateful and do utilize resources, so idle connections are disconnected by Service Bus. Again there is no impact to this because the client SDK has built in retry logic.

The first set of errors you saw were because of connection disruption. The messages are not "lost", because the service bus will not ack it and the client side SDK will retry the operation. The client will only proceed to the next message after the current message is successfully ACKed by Service Bus.
The second set of errors you saw are due to an idle connection. In Azure Service Bus, connections are stateful and do utilize resources, so idle connections are disconnected by Service Bus. Again there is no impact to this because the client SDK has built in retry logic.

The first set of errors you saw were because of connection disruption. The messages are not "lost", because the service bus will not ack it and the client side SDK will retry the operation. The client will only proceed to the next message after the current message is successfully ACKed by Service Bus.
The second set of errors you saw are due to an idle connection. In Azure Service Bus, connections are stateful and do utilize resources, so idle connections are disconnected by Service Bus. Again there is no impact to this because the client SDK has built in retry logic.

@axisc thanks for all the clarifications.
Based on what you said the .NET sdk has been designed to gracefully handle these errors (via retry policies). So no changes are required at the application level and we still have the guarantee that the application will have a canche to consume all the messages published to the topic, regardless these transient errors. Is my understanding correct ?
I would like to investigate what happens at the AKS level for the first set of errors (the one caused by 'connection disruption'), in order to understand whether we can do anything at the infrastructure level to mitigate this. Do you think this is worth the effort ? I mean, should we consider these connection disruption events as expected and related to the 'business as usual' of the Azure Service Bus or, instead, should they be considered as a symptom that something is not working well in our infrastructure ?

The first set of errors you saw were because of connection disruption. The messages are not "lost", because the service bus will not ack it and the client side SDK will retry the operation. The client will only proceed to the next message after the current message is successfully ACKed by Service Bus.
The second set of errors you saw are due to an idle connection. In Azure Service Bus, connections are stateful and do utilize resources, so idle connections are disconnected by Service Bus. Again there is no impact to this because the client SDK has built in retry logic.

@axisc thanks for all the clarifications.
Based on what you said the .NET sdk has been designed to gracefully handle these errors (via retry policies). So no changes are required at the application level and we still have the guarantee that the application will have a canche to consume all the messages published to the topic, regardless these transient errors. Is my understanding correct ?
I would like to investigate what happens at the AKS level for the first set of errors (the one caused by 'connection disruption'), in order to understand whether we can do anything at the infrastructure level to mitigate this. Do you think this is worth the effort ? I mean, should we consider these connection disruption events as expected and related to the 'business as usual' of the Azure Service Bus or, instead, should they be considered as a symptom that something is not working well in our infrastructure ?

@EnricoMassone
There is no cache, it is just that every data operation is committed to storage and log before you get an ACK. So no application changes are needed. Your business logic must wait for the ACK to make sure it went through.
You might need to pull some AKS experts for the AKS specific issues. My understanding is that if your containers move across the AKS cluster, then the stateful links between that container and Service Bus will break and the retry logic will engage to rebuild the connection. Do you know if container moves are occurring frequently (or at all)?
Is it worth the effort? It really depends - you should consider them business as usual and rely on retry policy regardless for all cloud native services, but if it is a significant hindrance and you have to process the same message multiple times because the connection broke, it may help to refine the configuration.

@EnricoMassone
There is no cache, it is just that every data operation is committed to storage and log before you get an ACK. So no application changes are needed. Your business logic must wait for the ACK to make sure it went through.
You might need to pull some AKS experts for the AKS specific issues. My understanding is that if your containers move across the AKS cluster, then the stateful links between that container and Service Bus will break and the retry logic will engage to rebuild the connection. Do you know if container moves are occurring frequently (or at all)?
Is it worth the effort? It really depends - you should consider them business as usual and rely on retry policy regardless for all cloud native services, but if it is a significant hindrance and you have to process the same message multiple times because the connection broke, it may help to refine the configuration.

@EnricoMassone just checking in if you had any follow up questions. If not, I can proceed to close the issue.

@EnricoMassone just checking in if you had any follow up questions. If not, I can proceed to close the issue.

@axisc I'm going to investigate what happens at the AKS level, we have an open ticket with the Azure service desk.
Just a clarification request, above you mentioned the ack. The only ack I'm aware of, in the context of the Azure service bus messaging, is the ack done by the client when the message processing is complete, so that the Azure Service Bus can remove the message from the subscription. Are you talking about that kind of ack ? (based on my understanding, you were referring to a different idea of ack).

@axisc I'm going to investigate what happens at the AKS level, we have an open ticket with the Azure service desk.
Just a clarification request, above you mentioned the ack. The only ack I'm aware of, in the context of the Azure service bus messaging, is the ack done by the client when the message processing is complete, so that the Azure Service Bus can remove the message from the subscription. Are you talking about that kind of ack ? (based on my understanding, you were referring to a different idea of ack).

When you use receive and delete, then yes the client ACK is not needed.
However, given the rest of the discussion, I'd recommend using the PeekLock (and complete) since that will eliminate the possibility of losing messages during such disconnects/reconnects/etc.
ACK is also sent by Service Bus to the sender when the messages are accepted.

When you use receive and delete, then yes the client ACK is not needed.
However, given the rest of the discussion, I'd recommend using the PeekLock (and complete) since that will eliminate the possibility of losing messages during such disconnects/reconnects/etc.
ACK is also sent by Service Bus to the sender when the messages are accepted.

When you use receive and delete, then yes the client ACK is not needed.
However, given the rest of the discussion, I'd recommend using the PeekLock (and complete) since that will eliminate the possibility of losing messages during such disconnects/reconnects/etc.
ACK is also sent by Service Bus to the sender when the messages are accepted.

Thanks for the clarification. About the receive and delete you are right: that's not the right strategy and we should change that. Based on my understanding, if there is a network error while the message is dispatched from the service bus to the client, that message will be lost forever. So, considering our transient connectivity issues, we should prevent that with a PeekLock strategy.

When you use receive and delete, then yes the client ACK is not needed.
However, given the rest of the discussion, I'd recommend using the PeekLock (and complete) since that will eliminate the possibility of losing messages during such disconnects/reconnects/etc.
ACK is also sent by Service Bus to the sender when the messages are accepted.

Thanks for the clarification. About the receive and delete you are right: that's not the right strategy and we should change that. Based on my understanding, if there is a network error while the message is dispatched from the service bus to the client, that message will be lost forever. So, considering our transient connectivity issues, we should prevent that with a PeekLock strategy.

@EnricoMassone  spot on!
Do let me know if there are follow up questions. If not, I can go ahead and close the issue.

@EnricoMassone  spot on!
Do let me know if there are follow up questions. If not, I can go ahead and close the issue.

@EnricoMassone spot on!
Do let me know if there are follow up questions. If not, I can go ahead and close the issue.

@axisc thanks for all the answers. I can proceed to close this issue. In case of further troubles I'll reopen this one.

@EnricoMassone spot on!
Do let me know if there are follow up questions. If not, I can go ahead and close the issue.

@axisc thanks for all the answers. I can proceed to close this issue. In case of further troubles I'll reopen this one.

We are also seeing this error from time to time. It happens in both AKS cluster and in on-prem data center connected to Azure via Express route. If this error is meant to be automatically retried by client, then why it shows up in the error log?
We use 5.1.0 version of the SDK.
Message: The operation was canceled.Reference: 37edd876e8554bbfa0dd202cd38097b8_G14, 04/08/2021 07:07:16

Microsoft.Azure.ServiceBus.Core.MessageReceiver+<OnReceiveAsync>d__88.MoveNext():662
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():12
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):40

Microsoft.Azure.ServiceBus.Core.MessageReceiver+<>c__DisplayClass65_0+<<ReceiveAsync>b__0>d.MoveNext():125
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():12
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):40

Microsoft.Azure.ServiceBus.RetryPolicy+<RunOperation>d__19.MoveNext():449
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():12
Microsoft.Azure.ServiceBus.RetryPolicy+<RunOperation>d__19.MoveNext():177
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():12
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):40

Microsoft.Azure.ServiceBus.Core.MessageReceiver+<ReceiveAsync>d__65.MoveNext():482
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():12
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):40

Microsoft.Azure.ServiceBus.Core.MessageReceiver+<ReceiveAsync>d__63.MoveNext():118
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():12
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):40

-- Application-specific call stack removed

Message: The operation was canceled.
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():12
Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result):245
Microsoft.Azure.Amqp.ReceivingAmqpLink.EndReceiveMessages(IAsyncResult result, IEnumerable`1& messages):8
System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization):15
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():12
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task):38
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task):40

Microsoft.Azure.ServiceBus.Core.MessageReceiver+<OnReceiveAsync>d__88.MoveNext():472

@sksk571 hi, we have solved the issue by adoting the new Azure.Messaging.ServiceBus SDK as a replacement for Microsoft.Azure.ServiceBus version 5.1.0.
By adopting the Azure.Messaging.ServiceBus package we get rid of all the errors.

authors: EnricoMassone axisc jsquire sksk571


text:
LiveOnly tests should run in Live mode locally even if AZURE_TEST_MODE is not live

Or there should at least be a way to enable this behavior for a given test class. If a test can only be run live, that doesn't necessarily mean that you don't want to run it when you are running all your other tests in Playback, it just means that the test cannot run in Playback. In other words, you should be able to run all your tests, and have the ones that support playback run in playback, and the others run live.

This was addressed with the alwaysRunLocally parameter - https://github.com/Azure/azure-sdk-for-net/blob/02a2e93ba7ad204f7c76038c06c7ad92f51bc068/sdk/core/Azure.Core.TestFramework/src/LiveOnlyAttribute.cs#L24


text:
[BUG] Form Recognizer Studio - Cross page tables fail when generating models

Library name and version
2022-06-30-preview
Describe the bug
I have created a custom neural model using Form Recognizer Studio using the API Version 2022-06-30-preview. In the release notes it says "Tabular fields support cross page tables by default". I have uploaded 5 documents and created label data that consists of a number of dynamic tables. These tables often extend across pages and I have tagged the rows as instructed. I then click the Train button and view the model from the Models menu item. It shows the status as failed. On clicking on the model it says 👍

ModelBuildError: Could not build the model: One or more errors occurred. (Only 1 valid input document(s) were found. Please provide at least 5 input documents. Labels file Title register (EGL177127) - 06_10_2016.pdf is invalid for the following reason(s): 'Contains FieldLabel in Title register (EGL177127) - 06_10_2016.pdf with label regions across multiple pages.' Labels file Official Copy (Register of Title and Title Plan) - WT436868 - 17_02_2021.pdf is invalid for the following reason(s): 'Contains FieldLabel in Official Copy (Register of Title and Title Plan) - WT436868 - 17_02_2021.pdf with label regions across multiple pages.' Labels file Title register-K834188_1.pdf is invalid for the following reason(s): 'Contains FieldLabel in Title register-K834188_1.pdf with label regions across multiple pages.' Labels file B267430001 Office copy register of title - NK451411.pdf.pdf is invalid for the following reason(s): 'Contains FieldLabel in B267430001 Office copy register of title - NK451411.pdf.pdf with label regions across multiple pages.')

The error suggests there are FieldLabels that are across multiple pages. Whilst there are Field Labels, they are all on a single page in the document, so I am unsure what this message is referring to.
Expected behavior
I expected the Model to train successfully with Tabular fields support across page tables
Actual behavior
See description
Reproduction Steps

Create Custom Neural model in Form Recognizer Studio
Upload 5 documents with tabular data that spans pages
Tag the data
Select Train to start training the model
View the errored status on the model

Environment
No response

@jsquire thanks for the swift response. Is there a repo for the Form Recognizer Studio? There used to be for its predecessor, but this has gone stale: https://github.com/microsoft/OCR-Form-Tools


text:
Assembly conflict in Azure in process function v4

Library name and version
Azure.Data.AppConfiguration
Describe the bug
Azure.Data.AppConfiguration: Method 'AsPages' in type 'FuncAsyncPageable`1' from assembly 'Azure.Data.AppConfiguration, Version=1.2.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8' does not have an implementation.
Expected behavior
We are converting a .net3.1 inprocess function to .net6 function v4.
The assembly is getting resolved properly on the localhost, but when it is deployed to Azure fails with the following error
Azure.Data.AppConfiguration: Method 'AsPages' in type 'FuncAsyncPageable`1' from assembly 'Azure.Data.AppConfiguration, Version=1.2.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8' does not have an implementation.
Actual behavior
Azure.Data.AppConfiguration: Method 'AsPages' in type 'FuncAsyncPageable`1' from assembly 'Azure.Data.AppConfiguration, Version=1.2.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8' does not have an implementation.
Reproduction Steps
These are the libraries referenced in the project file:
"Azure.Identity" Version="1.4.1"
"AzureExtensions.Swashbuckle" Version="3.3.2"
"AspNetCore.HealthChecks.CosmosDb" Version="3.1.2"
"Elevate.CacheServices.Client" Version="1.1.1"
"FluentValidation" Version="10.3.3"
"Microsoft.ApplicationInsights" Version="2.20.0"
"Microsoft.Azure.Cosmos" Version="3.22.1"
"Microsoft.Azure.WebJobs.Extensions.CosmosDB" Version="3.0.9"
"Microsoft.Extensions.Configuration.AzureAppConfiguration" Version="5.2.0"
"Microsoft.Extensions.Logging" Version="6.0.0"
"Microsoft.Extensions.Options.DataAnnotations" Version="6.0.0"
"Microsoft.NET.Sdk.Functions" Version="4.1.1"
"Newtonsoft.Json" Version="13.0.1"
Environment
No response

Thank you for your feedback.  Tagging and routing to the team member best able to assist.

I found a StackOverflow post that describes this exact scenario and recommends adding an explicit dependency for Microsoft.Extensions.Configuration.Abstractions to 6.0.0, as the issue appears to be due to an incompatible with 7.x which is pulled in transitively via your AppConfig reference. Would you be able to try this workaround?
@fabiocav Is there any other docs we can point customers to for this issue? Is the above workaround accurate to your knowledge? I can also move this issue over to the functions host repo.

I tried as per your suggestion, but I am getting the same error:
Azure.Data.AppConfiguration: Method 'AsPages' in type 'FuncAsyncPageable`1' from assembly 'Azure.Data.AppConfiguration, Version=1.2.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8' does not have an implementation.

@xztmp1 could you share a minimal repo?

Sorry I won't be able to share the code as it has proprietary information. This is a simple Http trigger function

Given that this appears to be an issue with the dependency resolution happening on the function host, and not locally, I will go ahead and transfer this issue to the functions host repo. The functions team should be able to debug further including having access to any server logs that may help them track down the issue.

authors: JoshLove-msft jsquire xztmp1


text:
Create samples and snippets

This tracks the work for generating the samples and snippets.

Fixed by #10568

authors: AlexGhiondea tg-msft


text:
use autorest version "V2" for mgmt sdk

The PR is to prevent the conflict between track 1 generator and autorest v3.
"V2" is the same as "2.0.4413".

/azp run net - mgmt - ci


text:
Added support for Application Gateway Web Application Firewall

Added a field to the application gateway object to hold the configuration for the web application firewall.
Added the new SKUs.
Bumped up the version number.
Updated the test code and mock recording (hardcoded location to "West Central US" as this is a region that gets updated early, and resource provider rollout is very time consuming these days).

Can one of the admins verify this patch?

Hi @allanbomsft, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

It looks like you're working at Microsoft (allanbo). If you're full-time, we DON'T require a contribution license agreement.

If you are a vendor, DO please sign the electronic contribution license agreement. It will take 2 minutes and there's no faxing! https://cla.azure.com.

TTYL, AZPRBOT;

@azuresdkci retest this please

@azuresdkci test this please

@azuresdkci add to whitelist

authors: allanbomsft azurecla azuresdkci cormacpayne shahabhijeet


text:
[Storage] Update to version 6.1.0-preview with feature file encryption

Description

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] I have read the contribution guidelines.
[x] The pull request does not introduce breaking changes.

General Guidelines

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[x] Pull request includes test coverage for the included changes.

SDK Generation Guidelines

[x] If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
[x] The generate.cmd file for the SDK has been updated with the version of AutoRest, as well as a link to the swagger spec, used to generate the code.
[x] The project.json and AssemblyInfo.cs files have been updated with the new version of the SDK.

Related to swagger update:
https://github.com/Azure/azure-rest-api-specs/pull/872

PR to swagger updating is merged:
https://github.com/Azure/azure-rest-api-specs/pull/872

@markcowl
Comments are addressed.


text:
Address arch board review comments

Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.ResourceManager.Hci

authors: azure-sdk fengzhou-msft


text:
[Storage][DataMovement] Refactor/fix support for CreationPreference

This change includes a couple of changes to how we handle the CreationPreference option:

Update checkpoint file to store StorageResourceCreationPreference instead of just a bool
Allow CreationPreference to be overridden on resume. This also meant we needed to add a new Default option to the enum
Refactor how we populate options on resume to allow for overriding. (Includes some changes to how I previosuly handled transfer sizes for consistency).

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.Storage.DataMovement

authors: azure-sdk jalauzon-msft


text:
Update Changelog for exporter

Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

closing this. Sorry @vishweshbankwar I was having merge conflicts and git wasn't cooperating. Thanks for getting this started for me.

authors: TimothyMothra vishweshbankwar


text:
Updating ADF SDK version and release notes before publishing

Description

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] Please add REST spec PR link to the SDK PR
[ ] I have read the contribution guidelines.
[ ] The pull request does not introduce breaking changes.

General Guidelines

[ ] Title of the pull request is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

SDK Generation Guidelines

[ ] If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
[ ] The generate.cmd file for the SDK has been updated with the version of AutoRest, as well as the commitid of your swagger spec or link to the swagger spec, used to generate the code.
[ ] The *.csproj and AssemblyInfo.cs files have been updated with the new version of the SDK.

@azuresdkci Retest this please

@azuresdkci retest

authors: dsgouda hvermis


text:
Enable-WriteCore-feature-for-signalr

Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.ResourceManager.SignalR

authors: azure-sdk welovej


text:
Override DisposeAsync in StorageWriteStream for asynchronous flushing during dispose

Should fix #35548.
Note I opted to place the public DisposeAsync() next to protected Dispose(bool disposing) to have related methods close together in the source code.

API change check
API changes are not detected in this pull request.

I believe this PR is ready for review.

authors: agehrke azure-sdk


text:
azure-mgmt-subscription broken: Subscription Creation Functionality Removed With No Replacement

azure-mgmt-subscription:
0.7.0:
All:
3:

Describe the bug
A breaking change was committed, merged, and released with no justification and no alternative functionality provided.  This change broke our automation used for creating new subscriptions and is forcing us to pin an old version of this package in our automation, which is less than ideal for many reasons.
Relevant commit, pull, and release:
https://github.com/Azure/azure-sdk-for-python/commit/380ae625102aeb2ff1fa3d788583ad8077713836
https://github.com/Azure/azure-sdk-for-python/pull/14426
https://github.com/Azure/azure-sdk-for-python/releases/tag/azure-mgmt-subscription_0.7.0
To Reproduce
Steps to reproduce the behavior:

Install latest azure-mgmt-subscription
Attempt to use any of these functions, which were removed without an alternative being provided:

Removed operation SubscriptionOperations.create_subscription_in_enrollment_account
Removed operation SubscriptionOperations.create_subscription
Removed operation SubscriptionOperations.create_csp_subscription
Removed operation group SubscriptionOperationOperations

Expected behavior
Expect to be able to use the aforementioned functions for subscription creation or have an alternative function provided.  Removing functionality without warning and providing no alternative is not expected behavior for any SDK.

Hi service team, the breaking changes come from this PR: https://github.com/Azure/azure-rest-api-specs/pull/10321
Would you help to check whether it's by design?

we have released our GA version for subscription creation APIs which is to use New-AzSubscriptionAlias cmdlet for subscription creation automations. You can see this documented in detail at https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/programmatically-create-subscription-enterprise-agreement?tabs=rest-getEnrollments%2Crest-EA.
We are asking everyone to move to adopt to this new cmdlet which is GA. We will keep the preview cmdlet around (only in older versions) until folks have had the chance to migrate and dont have a timeline when we will stop supporting preview versions.

we have released our GA version for subscription creation APIs which is to use New-AzSubscriptionAlias cmdlet for subscription creation automations. You can see this documented in detail at https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/programmatically-create-subscription-enterprise-agreement?tabs=rest-getEnrollments%2Crest-EA.
We are asking everyone to move to adopt to this new cmdlet which is GA. We will keep the preview cmdlet around (only in older versions) until folks have had the chance to migrate and dont have a timeline when we will stop supporting preview versions.

Please re-activate if this doesnt address your issue.

Please re-activate if this doesnt address your issue.

authors: anuragdalmia changlong-liu sas-adstut

repo: Azure/azure-sdk-for-python


text:
get_api_version and set_immutability_policy fails to typecheck with Mypy

Mypy fails to typecheck the functions in the title
Affected files:
azure.storage.blob._serialize
azure.storage.blob._blob_client

@tasherif-msft can you please take a look?

Actually, @tasherif-msft it looks like @jmg-duarte opened a PR to fix this (thank you!). Please see:
https://github.com/Azure/azure-sdk-for-python/issues/22741

@tjprescott @jmg-duarte thanks for opening the PR - I have merged it in!

authors: jmg-duarte tasherif-msft tjprescott


text:
Cannot get InteractiveBrowserCredential() to cache credentials

Package Name:  azure-identity
Package Version: 1.7.0
Operating System: Windows 11
Python Version: 3.8.0

Describe the bug
Using InteractiveBrowserCredential() with enable_persistent_cache=True does not seem to cache the credentials over subsequent runs (even for as little as 30 seconds).  When I also specify the authenication_record parameter, it still prompts for credentials in the browse on each run.
To Reproduce
Steps to reproduce the behavior:
Run the following python program:
# auth_with_cach_test.py: test out ability to get credentials without having to authenticate each time
import os
from azure.identity import InteractiveBrowserCredential, AuthenticationRecord, TokenCachePersistenceOptions

fn = "cred_cache.txt"
deserialized_record = None

found_cc = os.path.exists(fn)
if found_cc:
    with open(fn, "rt") as infile:
        cred_json = infile.read()
        deserialized_record = AuthenticationRecord.deserialize(cred_json)
        print("record read from:", fn)

credential = InteractiveBrowserCredential(enable_persistent_cache=True, authenication_record=deserialized_record)
record = credential.authenticate()

if not found_cc:
    # serialize to our cache
    cred_json = record.serialize()
    with open(fn, "wt") as outfile:
        outfile.write(cred_json)

    print("record written to:", fn)

Expected behavior
The first time the program is run, I expect it to prompt for credentials in the browser.  The second time the program is run, I expect it to run to completion without prompting for credentials in the browser, but it always prompts.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
See original issue: #9744.  Note: I am looking for the simplest way to get an "auth" to pass to the Azure Workspace.get() method, since it seems to require one with for certain Singularity workspaces.  I would prefer, in the following order:

to just log-in using my Azure portal login cached creds - is there a way to do this yet?
to have Azure cache my credentials itself (in the cloud, or on my local computer) - is there a way to do this yet?
to be able to cache some bit of information that I can use to get the "auth" without having to make the user authenticate again
The above test program represents the suggested code from #9744, as I interpret it.  Did I miss something?

Thanks for the feedback. Here is sample code to use token cache:
cred = InteractiveBrowserCredential(tenant_id=tenant_id, cache_persistence_options=TokenCachePersistenceOptions())
token = cred.get_token(YOUR_SCOPE)
auth_record = cred.authenticate()
cred1 = InteractiveBrowserCredential(tenant_id=tenant_id,
                                         cache_persistence_options=TokenCachePersistenceOptions(),
                                         authentication_record=auth_record)
token1 = cred1.get_token(YOUR_SCOPE)

Does this work for you?

No, that still requires browser authentication every time I run it.  BTW, is this new code supposed to have Azure or the user do the credential persistence?
Here is the complete version of your new code that I tested:
# auth_with_cache_test2.py: test ability to get Azure creds without always authenticating
import os
from azure.identity import InteractiveBrowserCredential, AuthenticationRecord, TokenCachePersistenceOptions

scopes = ""
tenant_id = "72f988bf-86f1-41af-91ab-2d7cd011db47"

cred = InteractiveBrowserCredential(tenant_id=tenant_id, cache_persistence_options=TokenCachePersistenceOptions())
token = cred.get_token(scopes)
auth_record = cred.authenticate()
cred1 = InteractiveBrowserCredential(tenant_id=tenant_id,
                                         cache_persistence_options=TokenCachePersistenceOptions(),
                                         authentication_record=auth_record)
token1 = cred1.get_token(scopes)
print("token1=", token1)

I doubt "" is a valid scope.
Could you try something like
"https://management.core.windows.net/.default"?
We expect to see a browser pop up in the first request
token = cred.get_token(scopes)
And on the second one
token1 = cred1.get_token(scopes)
It should come from the cache so there should no browser pop up.

Hi @xiangyan99 - using the above URL for scopes, it still authenticates in the browser, on each run of the test app.  Are you able to repro the problem with my program?

No. I cannot repro it.
Here is my app:
from azure.identity import InteractiveBrowserCredential, TokenCachePersistenceOptions

def main():
    tenant_id = 'YOUR_TENANT_ID'
    cred = InteractiveBrowserCredential(tenant_id=tenant_id, cache_persistence_options=TokenCachePersistenceOptions())
    token = cred.get_token("https://management.core.windows.net/.default")
    auth_record = cred.authenticate()
    cred1 = InteractiveBrowserCredential(tenant_id=tenant_id,
                                         cache_persistence_options=TokenCachePersistenceOptions(),
                                         authentication_record=auth_record)
    token1 = cred1.get_token("https://management.core.windows.net/.default")
    print("pass")

if __name__ == '__main__':
    main()

The browser only pops up once.

I just ran your program 4 times - each time, the browser got focus for authentication.  One thing that might be different about my environment - I don't run "az" (Azure CLI).   Is this design somehow dependent on it being installed?

Here is my program:
sing_test2.py: test out ability to create a workspace object for a singularity workspace
import os
from azureml.core.authentication import TokenAuthentication
from azure.identity import InteractiveBrowserCredential, AuthenticationRecord, TokenCachePersistenceOptions
fn = "cred_cache.txt"
tenant_id = None
deserialized_record = None
scope = https://management.core.windows.net/.default
tenant_id = "72f988bf-86f1-41af-91ab-2d7cd011db47"
if os.path.exists(fn):
# read serialized cred from disk
with open(fn, "rt") as infile:
cred_json = infile.read()
deserialized_record = AuthenticationRecord.deserialize(cred_json)
#tenant_id = deserialized_record.tenant_id
cpo = TokenCachePersistenceOptions()
credential = InteractiveBrowserCredential(tenant_id=tenant_id,
#enable_persistent_cache=True,
cache_persistence_options=cpo,
authenication_record=deserialized_record)
if not os.path.exists(fn):
# serialize to our cred to disk
record = credential.authenticate()
cred_json = record.serialize()
with open(fn, "wt") as outfile:
    outfile.write(cred_json)

prove the credential is valid
token = credential.get_token(scope)
print(token)
From: Xiang Yan @.>
Sent: Tuesday, March 29, 2022 3:53 PM
To: Azure/azure-sdk-for-python @.>
Cc: Roland Fernandez @.>; Mention @.>
Subject: Re: [Azure/azure-sdk-for-python] Cannot get InteractiveBrowserCredential() to cache credentials (Issue #23721)
No. I cannot repro it.
Here is my app:
from azure.identity import InteractiveBrowserCredential, TokenCachePersistenceOptions
def main():
tenant_id = 'YOUR_TENANT_ID'

cred = InteractiveBrowserCredential(tenant_id=tenant_id, cache_persistence_options=TokenCachePersistenceOptions())

token = cred.get_token("https://management.core.windows.net/.default")

auth_record = cred.authenticate()

cred1 = InteractiveBrowserCredential(tenant_id=tenant_id,

                                     cache_persistence_options=TokenCachePersistenceOptions(),

                                     authentication_record=auth_record)

token1 = cred1.get_token("https://management.core.windows.net/.default")

print("pass")

if name == 'main':
main()

The browser only pops up once.

Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-sdk-for-python%2Fissues%2F23721%23issuecomment-1082446307&data=04|01|rfernand%40microsoft.com|eaabed7e3362491a99ea08da11d6dd34|72f988bf86f141af91ab2d7cd011db47|1|0|637841911790030875|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=4ZUqRzdC1Je0zp%2FLoZuW%2FFSnG8iwrNsCUiVRjE6ouYo%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABAY3XQBY6LNKHI54KENDMTVCOCUPANCNFSM5R45AFKA&data=04|01|rfernand%40microsoft.com|eaabed7e3362491a99ea08da11d6dd34|72f988bf86f141af91ab2d7cd011db47|1|0|637841911790030875|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000&sdata=3FOWLdvFp7My%2Fa7rAyE7OfkCCJLzOY5SMUemlnUlEqI%3D&reserved=0.
You are receiving this because you were mentioned.Message ID: @.***>

Thanks to help from @xiangyan99, the below, updated program now works correctly (Azure authentication in browser is only done on first run):
# creds_cache_test.py: test out ability to use cached creds to avoid authenticating on every run
import os
from azure.identity import InteractiveBrowserCredential, AuthenticationRecord, TokenCachePersistenceOptions

# actual credential caching is handling by azure, but our app must supply the authentication record
# which identifies the tentant_id, username, etc.
fn = "cred_cache.txt"
deserialized_record = None

if os.path.exists(fn):
    # read serialized authentication_record from local file
    with open(fn, "rt") as infile:
        cred_json = infile.read()
        deserialized_record = AuthenticationRecord.deserialize(cred_json)

cpo = TokenCachePersistenceOptions()
credential = InteractiveBrowserCredential(cache_persistence_options=cpo,
    authentication_record=deserialized_record)

if not os.path.exists(fn):
    # serialize authentication_record to local file
    record = credential.authenticate()
    cred_json = record.serialize()

    with open(fn, "wt") as outfile:
        outfile.write(cred_json)

# prove the credential is valid
scope = "https://management.core.windows.net/.default"
token = credential.get_token(scope)
print(token)

Why is this closed?  Shouldn't InteractiveBrowserCredential try to load from the cache first?

Yes. And this is the code snippet how to achieve it.
https://github.com/Azure/azure-sdk-for-python/issues/23721#issuecomment-1083539872

Maybe I don't understand the design decision here.  When I look at SharedTokenCacheCredential, if I don't pass an authentication_record parameter, it figures it out for me.  Why isn't this the case for InteractiveBrowserCredential?
My work-around for this was to use a chained credential of (SharedTokenCacheCredential, InteractiveBrowserCredential) so that they loaded.  But it's silly that InteractiveBrowserCredential can't figure out how to use the cache without an authentication_record when SharedTokenCacheCredential can.

If you don't mind, could you share your code snippets how you use those credentials?
My guess is you did not specify tenant_id for them. If it is not specified, SharedTokenCacheCredential gives a default one but InteractiveBrowserCredential cannot give a default one hence it prompts you to specify.


text:
[multiapi combiner] Fix the breaking change in model between different api-version

Take model RunCommandResult of azure-mgmt-compute as example, multiapi_combiner use model of latest 2023-03-01 as default model. However, it is very different with 2017-12-01.

When use 2017-12-01 to call service, we get response like here: https://github.com/Azure/azure-rest-api-specs/blob/b646a42aa3b7a0ce488d05f1724827ea41d12cf1/specification/compute/resource-manager/Microsoft.Compute/ComputeRP/stable/2017-12-01/examples/VirtualMachineRunCommand.json#L15-L37
When use 2023-03-01 to call service, we get response like here: https://github.com/Azure/azure-rest-api-specs/blob/b646a42aa3b7a0ce488d05f1724827ea41d12cf1/specification/compute/resource-manager/Microsoft.Compute/ComputeRP/stable/2023-03-01/examples/runCommandExamples/VirtualMachineRunCommand.json#L13-L31

The response is very different so if we use 2023-03-01 model to deserialize response of 2017-12-01, we can't get expected output:

Here is code to reproduce:
from azure.mgmt.compute import ComputeManagementClient
from azure.identity import DefaultAzureCredential

response_data = {
    "startTime": "2017-06-16T16:01:37.8958419-07:00",
    "endTime": "2017-06-16T16:10:11.2897717-07:00",
    "status": "Succeeded",
    "properties": {
        "output": [
            {
                "code": "ComponentStatus/StdOut/succeeded",
                "level": "Info",
                "displayStatus": "Provisioning succeeded",
                "message": "This is a sample script with parameters value1 value2",
            },
            {
                "code": "ComponentStatus/StdErr/succeeded",
                "level": "Info",
                "displayStatus": "Provisioning succeeded",
                "message": "",
            },
        ]
    },
    "name": "289dbc84-3c84-4a86-9e40-bbd4d61edcaf",
}

# use model of 2017-12-01: OK
client = ComputeManagementClient(credential=DefaultAzureCredential(), subscription_id="000", api_version="2017-12-01")
result = client.virtual_machines._deserialize._deserialize("RunCommandResult", response_data)
print(result.serialize())

# use model of 2023-03-01: NO
client = ComputeManagementClient(credential=DefaultAzureCredential(), subscription_id="000")
result = client.virtual_machines._deserialize._deserialize("RunCommandResult", response_data)
print(result.serialize())

Here is updated code to reproduce the error:
import os
from azure.identity import DefaultAzureCredential
from azure.mgmt.compute import ComputeManagementClient
from dotenv import load_dotenv

load_dotenv()

client = ComputeManagementClient(
    credential=DefaultAzureCredential(),
    subscription_id=os.getenv("AZURE_SUBSCRIPTION_ID"),
)

response = client.virtual_machines.begin_run_command(
    resource_group_name="testgroupx",
    vm_name="temp-vm-for-test",
    parameters={
        'commandId': 'RunShellScript',
        'script': [
            'ls /tmp',
        ]
    },
    api_version="2017-12-01",
).result()
print(response.as_dict())

Here is the workaround: https://github.com/Azure/azure-sdk-for-python/pull/30681/commits/3f06864cbad53fb6084cd73924ef1d589474fe83

Here is the workaround: https://github.com/Azure/azure-sdk-for-python/pull/30681/commits/3f06864cbad53fb6084cd73924ef1d589474fe83
But there is still breakings for result since we use latest model to deserialize the response of old api-version:

result before:
{'name': '911566c1-8af5-4989-a30c-448a2efd9673', 'status': 'Succeeded', 'start_time': '2023-08-25T05:59:20.834945Z', 'end_time': '2023-08-25T05:59:41.084726Z', 'output': [{'code': 'ProvisioningState/succeeded', 'level': 'Info', 'displayStatus': 'Provisioning succeeded', 'message': 'Enable succeeded: \n[stdout]\n10e511004cae46237916a29b58af8ef1bff158ea92e43b02edf0b20f23d25c38\nPipelineAgentDebugOutput\nStateDir\nasatenant.conf3004157894\nasatenant.conf309059580\nsnap-private-tmp\nsystemd-private-906917e844464f0fafbcd196cf14bd80-ModemManager.service-sC6IKf\nsystemd-private-906917e844464f0fafbcd196cf14bd80-chrony.service-93N1wh\nsystemd-private-906917e844464f0fafbcd196cf14bd80-systemd-logind.service-A6iZvf\nsystemd-private-906917e844464f0fafbcd196cf14bd80-systemd-resolved.service-YJir6e\n\n[stderr]\n'}]}
result now
{'value': [{'code': 'ProvisioningState/succeeded', 'level': 'Info', 'display_status': 'Provisioning succeeded', 'message': 'Enable succeeded: \n[stdout]\n10e511004cae46237916a29b58af8ef1bff158ea92e43b02edf0b20f23d25c38\nPipelineAgentDebugOutput\nStateDir\nasatenant.conf3004157894\nasatenant.conf309059580\nsnap-private-tmp\nsystemd-private-906917e844464f0fafbcd196cf14bd80-ModemManager.service-sC6IKf\nsystemd-private-906917e844464f0fafbcd196cf14bd80-chrony.service-93N1wh\nsystemd-private-906917e844464f0fafbcd196cf14bd80-systemd-logind.service-A6iZvf\nsystemd-private-906917e844464f0fafbcd196cf14bd80-systemd-resolved.service-YJir6e\n\n[stderr]\n'}]}

Another workaround which doesn't need change generated SDK code:
def get_long_running_output(pipeline_response):
    body_as_json = pipeline_response.context["deserialized_data"]
    if isinstance(body_as_json, list):
        body_as_json = {"value": body_as_json}
    return RunCommandResult.deserialize(body_as_json)

class RunCommandPolling(ARMPolling):
    def initialize(
        self,
        client,
        initial_response,
        _,
    ):
        super().initialize(client, initial_response, get_long_running_output)

result = compute_client.virtual_machines.begin_run_command(
    resource_group_name="lmazuel-vm-test",
    vm_name="lmazuel-vm-test",
    parameters={
        'commandId': 'RunShellScript',
        'script': [
            'ls /tmp',
        ]
    },
    api_version="2017-12-01",
    polling=RunCommandPolling(lro_options={"final-state-via": "location"}),
).result()

print(result.as_dict())

B is very special since it can't be applied to common operation and paging operation. If operation can accept another callback for pipeline_response which takes effect before deserialization, it is possible to adopt for all kinds of operation. Here is workaround C

FYI:
But there is still breakings for result since we use latest model to deserialize the response of old api-version:

result before:
{'name': '911566c1-8af5-4989-a30c-448a2efd9673', 'status': 'Succeeded', 'start_time': '2023-08-25T05:59:20.834945Z', 'end_time': '2023-08-25T05:59:41.084726Z', 'output': [{'code': 'ProvisioningState/succeeded', 'level': 'Info', 'displayStatus': 'Provisioning succeeded', 'message': 'Enable succeeded: \n[stdout]\n10e511004cae46237916a29b58af8ef1bff158ea92e43b02edf0b20f23d25c38\nPipelineAgentDebugOutput\nStateDir\nasatenant.conf3004157894\nasatenant.conf309059580\nsnap-private-tmp\nsystemd-private-906917e844464f0fafbcd196cf14bd80-ModemManager.service-sC6IKf\nsystemd-private-906917e844464f0fafbcd196cf14bd80-chrony.service-93N1wh\nsystemd-private-906917e844464f0fafbcd196cf14bd80-systemd-logind.service-A6iZvf\nsystemd-private-906917e844464f0fafbcd196cf14bd80-systemd-resolved.service-YJir6e\n\n[stderr]\n'}]}
result now
{'value': [{'code': 'ProvisioningState/succeeded', 'level': 'Info', 'display_status': 'Provisioning succeeded', 'message': 'Enable succeeded: \n[stdout]\n10e511004cae46237916a29b58af8ef1bff158ea92e43b02edf0b20f23d25c38\nPipelineAgentDebugOutput\nStateDir\nasatenant.conf3004157894\nasatenant.conf309059580\nsnap-private-tmp\nsystemd-private-906917e844464f0fafbcd196cf14bd80-ModemManager.service-sC6IKf\nsystemd-private-906917e844464f0fafbcd196cf14bd80-chrony.service-93N1wh\nsystemd-private-906917e844464f0fafbcd196cf14bd80-systemd-logind.service-A6iZvf\nsystemd-private-906917e844464f0fafbcd196cf14bd80-systemd-resolved.service-YJir6e\n\n[stderr]\n'}]}


text:
[Document Intelligence] Incremental training on Studio & Python SDK

Will there be a release for incremental training of custom models in the studio and python sdk?

Thanks for reaching out, we’ll investigate asap.

There are currently no plans to support this.  One option is to retain the original training data and train a new custom with the combined data.  You can now increase max_training_hours to more effectively utilize larger training set size.

authors: bojunehsu galvangoh xiangyan99