text:
[BUG] token retrieval intermittently stuck in BearerTokenAuthenticationPolicy in Azure.Core 1.40.0

Library name and version
Azure.Core 1.40.0
Describe the bug
Same as #44817. However, the "fix" in #44882 only works if a CancellationToken is passed on and cancellation is requested.
We're using ASP.NET Core Data Protection with Azure.Extensions.AspNetCore.DataProtection.Blobs. This is building an Azure.Core.HttpMessage with default cancellation token. So, it is never cancelled and requests hang indefinitely.
I know Azure.Core 1.42.0 is the latest version. But it seems the fix in #44882 would not work in this scenario. The workaround using an CancellationToken does not solve the real issue. There seems to be a specific scenario in which the CurrentTokenTcs never gets a result.
Multiple of our applications deadlock in this situation:

[Managed to Native Transition]
System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout = -1, System.Threading.CancellationToken cancellationToken) Line 264	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 2386	C#
System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalWaitCore(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) Line 2354	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task = Id = 50, Status = WaitingForActivation, Method = "{null}", Result = "{Not yet computed}") Line 51	C#
System.Private.CoreLib.dll!System.Runtime.CompilerServices.TaskAwaiter<Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.AuthHeaderValueInfo>.GetResult() Line 173	C#
[Waiting on Async Operation, double-click or press enter to view Async Call Stacks]
Azure.Core.dll!Azure.Core.Pipeline.TaskExtensions.EnsureCompleted<Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.AuthHeaderValueInfo>(System.Threading.Tasks.Task<Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.AuthHeaderValueInfo> task = Id = 50, Status = WaitingForActivation, Method = "{null}", Result = "{Not yet computed}") Line 33	C#
Azure.Core.dll!Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.TokenRequestState.GetCurrentHeaderValue(bool async = false, bool checkForCompletion = false, System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Line 438	C#
[Resuming Async Method]
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start<Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.TokenRequestState.d__19>(ref Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.TokenRequestState.d__19 stateMachine = {Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.TokenRequestState.d__19}) Line 55	C#
Azure.Core.dll!Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.GetAuthHeaderValueAsync(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, Azure.Core.TokenRequestContext context = {Azure.Core.TokenRequestContext}, bool async = false) Line 207	C#
Azure.Core.dll!Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AuthenticateAndAuthorizeRequest(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, Azure.Core.TokenRequestContext context = {Azure.Core.TokenRequestContext}) Line 172	C#
Azure.Storage.Blobs.dll!Azure.Storage.StorageBearerTokenChallengeAuthorizationPolicy.AuthorizeRequestInternal(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, bool async = false) Line 69	C#
Azure.Storage.Blobs.dll!Azure.Storage.StorageBearerTokenChallengeAuthorizationPolicy.AuthorizeRequest(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}) Line 52	C#
Azure.Core.dll!Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[4]", bool async = false) Line 127	C#
Azure.Core.dll!Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[4]") Line 61	C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(Azure.Core.HttpMessage message, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline) Line 47	C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[5]") Line 40	C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(Azure.Core.HttpMessage message, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline) Line 47	C#
Azure.Core.dll!Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[6]", bool async = false) Line 50	C#
Azure.Core.dll!Azure.Core.Pipeline.RedirectPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[6]") Line 198	C#
Azure.Core.dll!Azure.Core.Pipeline.RetryPolicy.ProcessAsync(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[7]", bool async = false) Line 85	C#
Azure.Core.dll!Azure.Core.Pipeline.RetryPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[7]") Line 59	C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(Azure.Core.HttpMessage message, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline) Line 47	C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[8]") Line 40	C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(Azure.Core.HttpMessage message, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline) Line 47	C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[9]") Line 40	C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(Azure.Core.HttpMessage message, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline) Line 47	C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[10]") Line 40	C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(Azure.Core.HttpMessage message, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline) Line 47	C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.ReadOnlyMemory<Azure.Core.Pipeline.HttpPipelinePolicy> pipeline = "System.ReadOnlyMemory[11]") Line 40	C#
Azure.Core.dll!Azure.Core.Pipeline.HttpPipeline.Send(Azure.Core.HttpMessage message = {Azure.Core.HttpMessage}, System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Line 174	C#
Azure.Storage.Blobs.dll!Azure.Storage.Blobs.BlobRestClient.Download(string snapshot = null, string versionId = null, int? timeout = null, string range = "bytes=0-268435455", string leaseId = null, bool? rangeGetContentMD5 = null, bool? rangeGetContentCRC64 = null, string encryptionKey = null, string encryptionKeySha256 = null, Azure.Storage.Blobs.Models.EncryptionAlgorithmTypeInternal? encryptionAlgorithm = null, System.DateTimeOffset? ifModifiedSince = null, System.DateTimeOffset? ifUnmodifiedSince = null, string ifMatch = null, string ifNoneMatch = "0x8DC8A29E41EA7D4", string ifTags = null, System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Line 175	C#
Azure.Storage.Blobs.dll!Azure.Storage.Blobs.Specialized.BlobBaseClient.StartDownloadAsync(Azure.HttpRange range = {Azure.HttpRange}, Azure.Storage.Blobs.Models.BlobRequestConditions conditions = {Azure.Storage.Blobs.Models.BlobRequestConditions}, Azure.Storage.DownloadTransferValidationOptions validationOptions = {Azure.Storage.DownloadTransferValidationOptions}, long startOffset = 0, bool async = false, System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Line 1737	C#
Azure.Storage.Blobs.dll!Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadStreamingInternal(Azure.HttpRange range = {Azure.HttpRange}, Azure.Storage.Blobs.Models.BlobRequestConditions conditions = {Azure.Storage.Blobs.Models.BlobRequestConditions}, Azure.Storage.DownloadTransferValidationOptions transferValidationOverride = {Azure.Storage.DownloadTransferValidationOptions}, System.IProgress progressHandler = null, string operationName = "BlobBaseClient.DownloadStreaming", bool async = false, System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Line 1561	C#
Azure.Storage.Blobs.dll!Azure.Storage.Blobs.PartitionedDownloader.DownloadTo(System.IO.Stream destination = {System.IO.MemoryStream}, Azure.Storage.Blobs.Models.BlobRequestConditions conditions = {Azure.Storage.Blobs.Models.BlobRequestConditions}, System.Threading.CancellationToken cancellationToken) Line 307	C#
Azure.Storage.Blobs.dll!Azure.Storage.Blobs.Specialized.BlobBaseClient.StagedDownloadAsync(System.IO.Stream destination = {System.IO.MemoryStream}, Azure.Storage.Blobs.Models.BlobRequestConditions conditions = {Azure.Storage.Blobs.Models.BlobRequestConditions}, System.IProgress progressHandler = null, Azure.Storage.StorageTransferOptions transferOptions = {Azure.Storage.StorageTransferOptions}, Azure.Storage.DownloadTransferValidationOptions transferValidationOverride = null, bool async = false, System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Line 2893	C#
Azure.Storage.Blobs.dll!Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadTo(System.IO.Stream destination = {System.IO.MemoryStream}, Azure.Storage.Blobs.Models.BlobRequestConditions conditions = {Azure.Storage.Blobs.Models.BlobRequestConditions}, Azure.Storage.StorageTransferOptions transferOptions = {Azure.Storage.StorageTransferOptions}, System.Threading.CancellationToken cancellationToken = IsCancellationRequested = false) Line 2677	C#
Azure.Extensions.AspNetCore.DataProtection.Blobs.dll!Azure.Extensions.AspNetCore.DataProtection.Blobs.AzureBlobXmlRepository.GetLatestData() Line 199	C#
Azure.Extensions.AspNetCore.DataProtection.Blobs.dll!Azure.Extensions.AspNetCore.DataProtection.Blobs.AzureBlobXmlRepository.GetAllElements() Line 57	C#
Microsoft.AspNetCore.DataProtection.dll!Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.GetAllKeys()	Unknown
Microsoft.AspNetCore.DataProtection.dll!Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore(System.DateTimeOffset now = {System.DateTimeOffset}, Microsoft.AspNetCore.DataProtection.KeyManagement.IKey keyJustAdded = null)	Unknown
Microsoft.AspNetCore.DataProtection.dll!Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.ICacheableKeyRingProvider.GetCacheableKeyRing(System.DateTimeOffset now = {System.DateTimeOffset})	Unknown
Microsoft.AspNetCore.DataProtection.dll!Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(System.DateTime utcNow, bool forceRefresh)	Unknown
Microsoft.AspNetCore.DataProtection.dll!Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(byte[] plaintext = {byte[949538]})	Unknown

Expected behavior
Blob storage call is executed
Actual behavior
The call is stuck and never completes
Reproduction Steps
Hard to reproduce
Environment

@christothes:  Would you please take a look and offer your thoughts?

@christothes : Maybe it's due to the fact there's a single BlobRestClient' with a single HttpPipeline containing the same BearerTokenAuthenticationPolicy instance? I can see 2 concurrent Download calls on the same BlobRestClient instance.
I'm not too familiar with this code, but is thread safe? Concurrent calls on https://github.com/christothes/azure-sdk-for-net/blob/6c76e7d2374473c7be189f7cf35d641024a0d164/sdk/core/Azure.Core/src/Pipeline/BearerTokenAuthenticationPolicy.cs#L256 could modify the same state. Although with the lock, it would be one after the other. However, does https://github.com/christothes/azure-sdk-for-net/blob/6c76e7d2374473c7be189f7cf35d641024a0d164/sdk/core/Azure.Core/src/Pipeline/BearerTokenAuthenticationPolicy.cs#L203-L205 still work if the 2nd calll also changed the state before this code is executed?

#45223

My bad, i missed this subtle change: https://github.com/Azure/azure-sdk-for-net/commit/93512b14ca1b6d40dde499bfb1e74440779dae5f?diff=split&w=0#diff-f6d09d34c9aed3acf5957c27473cfdc4a1d52f0219b8a25339e0abb206a89209R412.
This issue can be closed as I've only seen it with Azure.Core 1.40.0. So it might be already be fixed.

authors: MarcWils jsquire timaiv


text:
Either update or remove use of LoggingExtensions

Throughout the Storage API, we call logging extensions such as LogMethodEnter, LogMethodExit, LogException, etc. Currently these extensions, don't do anything as they are behind a Conditional compilation attribute that isn't enabled, e.g.:
        [Conditional("EnableLoggingHelpers")]
        public static void LogMethodExit(
            this HttpPipeline pipeline,
            string className,
            [CallerMemberName] string member = default,
            string message = "")
        => LogTrace(pipeline, $"EXIT METHOD {className} {member}\n{message}");

If we want this logging, we should define EnableLoggingHelpers and do any other updates that are needed. If we don't need this, we can delete this file and remove all calls to these methods.

I vote we remove the LoggingExtensions.

Hi @JoshLove-msft, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.

source: gharchive/issue

authors: JimSuplizio JoshLove-msft seanmcc-msft

repo: Azure/azure-sdk-for-net

license: MIT

license_type: permissive

license_source: github-api


text:
Performed Code Cleanup

Removed redundant code
Standardized naming and style
Fixed comments
Fixed warnings

Can one of the admins verify this patch?

As the owners of this version of the client library, @nemakam, @shankarsama and team would be the authoritative voice for feedback here.

@axisc and @shankarsama - Would you please be so kind as to provide feedback to @tstepanski and advise if these changes are something that you'd like to consider or if we should look to close out the PR?

Hi @tstepanski.  Thank you for your contribution, and I'm sorry that you haven't received any feedback.  Unfortunately, it does not look as if the Service Bus team would like to consider these changes at this point in time.  I'm going to close this out, since there hasn't been any recent activity or engagement.  Please feel free to reopen if you'd like to continue working on these changes.

source: gharchive/pull-request

authors: azuresdkci jsquire tstepanski


text:
Swallow NotImplementedException in EventSource name deduplication logic

Fixes: https://github.com/Azure/azure-sdk-for-net/issues/24055

Hi.
I am part of the now closed report #24055 submitted by Muhammet Sahin and we fins ourselves blocked from publishing our xamarin based mobile app to Appstore due to this.
All worked/works fine for some reason while testing the same build of the app from AppCenter, but as we now move to the next phase and add it to Appstore and Testflight, this issue emerged...
In what timeframe can we get access to a fix for testing in our app?
Best regards
Thomas Odell Balkeståhl

In addition, there is a nightly feed: pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json you can get the latest set of packages and test the fix.

Hi.
We have tried to downgrade, but it got even worse, now the app won't even start at all. We get 2-3 crashes at start and then it is dead.
Do you know if the deployment via Testflight/Appstore affects this in any way? Anything that 'manipulates' the versions of the packages?
Most likely, it is as you say, that the bug was introduced in aug, but we had a working app in AppCenter(Microsoft) and the exact same build published on Testflight triggered the bug.

Update:
We have now managed to get our app running. It was due to the bug in azure.storage/azure.core, but also in relation to the experimental flags in Xamarin.
https://docs.microsoft.com/en-us/xamarin/xamarin-forms/internals/experimental-flags
(With 'we' I'm referring to our big hero @muhammetsahin who managed to solve it with no blame to himself)

authors: Candelit pakrym


text:
Add EntityId to secrets, keys, and certificates

Fixes #28564

API change check for Azure.Security.KeyVault.Certificates
API changes have been detected in Azure.Security.KeyVault.Certificates. You can review API changes here
API changes
+         public string EntityId { get; }

API change check for Azure.Security.KeyVault.Keys
API changes have been detected in Azure.Security.KeyVault.Keys. You can review API changes here
API changes
+         public string EntityId { get; }

API change check for Azure.Security.KeyVault.Secrets
API changes have been detected in Azure.Security.KeyVault.Secrets. You can review API changes here
API changes
+         public string EntityId { get; }

Waiting for 7.4-preview.1 to deploy so I can record tests and write assertions.

authors: azure-sdk heaths


text:
Use ArrayBackedPropertyBag for PipelineRequest

Benchmarks:
|                    Categories | count |            Method |        Mean |    Error |   StdDev | Ratio |  Gen 0 | Allocated |
|------------------------------ |------ |------------------ |------------:|---------:|---------:|------:|-------:|----------:|
|      CreateHttpRequestMessage |     2 | HttpRequestHeader |    656.1 ns |  2.51 ns |  2.35 ns |  1.00 | 0.0048 |     496 B |
|      CreateHttpRequestMessage |     2 | ArrayBackedHeader |    470.6 ns |  0.74 ns |  0.58 ns |  0.72 | 0.0048 |     496 B |
|                               |       |                   |             |          |          |       |        |           |
|      CreateHttpRequestMessage |     3 | HttpRequestHeader |    727.0 ns |  2.69 ns |  2.52 ns |  1.00 | 0.0048 |     496 B |
|      CreateHttpRequestMessage |     3 | ArrayBackedHeader |    608.6 ns |  2.65 ns |  2.35 ns |  0.84 | 0.0076 |     776 B |
|                               |       |                   |             |          |          |       |        |           |
|      CreateHttpRequestMessage |     8 | HttpRequestHeader |  1,287.3 ns |  4.97 ns |  4.40 ns |  1.00 | 0.0114 |   1,104 B |
|      CreateHttpRequestMessage |     8 | ArrayBackedHeader |  1,180.9 ns | 14.22 ns | 13.30 ns |  0.92 | 0.0134 |   1,384 B |
|                               |       |                   |             |          |          |       |        |           |
|      CreateHttpRequestMessage |    16 | HttpRequestHeader |  2,490.9 ns |  6.50 ns |  5.43 ns |  1.00 | 0.0267 |   2,736 B |
|      CreateHttpRequestMessage |    16 | ArrayBackedHeader |  2,504.4 ns |  6.96 ns |  5.81 ns |  1.01 | 0.0305 |   3,016 B |
|                               |       |                   |             |          |          |       |        |           |
|      CreateHttpRequestMessage |    32 | HttpRequestHeader |  4,374.9 ns | 12.06 ns | 10.07 ns |  1.00 | 0.0381 |   4,120 B |
|      CreateHttpRequestMessage |    32 | ArrayBackedHeader |  5,410.5 ns | 21.57 ns | 19.12 ns |  1.24 | 0.0458 |   4,656 B |
|                               |       |                   |             |          |          |       |        |           |
| CreateHttpRequestMessageTwice |     2 | HttpRequestHeader |  1,324.0 ns |  5.69 ns |  5.04 ns |  1.00 | 0.0114 |   1,256 B |
| CreateHttpRequestMessageTwice |     2 | ArrayBackedHeader |    950.8 ns |  4.91 ns |  4.35 ns |  0.72 | 0.0095 |   1,032 B |
|                               |       |                   |             |          |          |       |        |           |
| CreateHttpRequestMessageTwice |     3 | HttpRequestHeader |  1,506.9 ns |  5.77 ns |  5.40 ns |  1.00 | 0.0134 |   1,336 B |
| CreateHttpRequestMessageTwice |     3 | ArrayBackedHeader |  1,137.6 ns |  0.91 ns |  0.80 ns |  0.76 | 0.0134 |   1,312 B |
|                               |       |                   |             |          |          |       |        |           |
| CreateHttpRequestMessageTwice |     8 | HttpRequestHeader |  2,855.4 ns | 15.61 ns | 14.60 ns |  1.00 | 0.0305 |   2,888 B |
| CreateHttpRequestMessageTwice |     8 | ArrayBackedHeader |  2,187.8 ns |  2.24 ns |  1.75 ns |  0.77 | 0.0267 |   2,528 B |
|                               |       |                   |             |          |          |       |        |           |
| CreateHttpRequestMessageTwice |    16 | HttpRequestHeader |  5,452.2 ns | 15.40 ns | 12.86 ns |  1.00 | 0.0687 |   6,792 B |
| CreateHttpRequestMessageTwice |    16 | ArrayBackedHeader |  4,419.1 ns | 29.19 ns | 27.31 ns |  0.81 | 0.0610 |   5,792 B |
|                               |       |                   |             |          |          |       |        |           |
| CreateHttpRequestMessageTwice |    32 | HttpRequestHeader |  9,906.1 ns | 11.67 ns |  9.11 ns |  1.00 | 0.1068 |  10,840 B |
| CreateHttpRequestMessageTwice |    32 | ArrayBackedHeader |  8,834.6 ns | 20.36 ns | 15.90 ns |  0.89 | 0.0916 |   8,816 B |
|                               |       |                   |             |          |          |       |        |           |
|                 MultipleReads |     8 | HttpRequestHeader |  2,976.5 ns | 11.14 ns |  9.30 ns |  1.00 | 0.0229 |   2,472 B |
|                 MultipleReads |     8 | ArrayBackedHeader |  1,447.9 ns |  2.72 ns |  2.55 ns |  0.49 | 0.0153 |   1,472 B |
|                               |       |                   |             |          |          |       |        |           |
|                 MultipleReads |    16 | HttpRequestHeader |  5,971.1 ns | 19.34 ns | 17.14 ns |  1.00 | 0.0534 |   5,448 B |
|                 MultipleReads |    16 | ArrayBackedHeader |  3,195.2 ns | 24.38 ns | 22.80 ns |  0.54 | 0.0305 |   3,168 B |
|                               |       |                   |             |          |          |       |        |           |
|                 MultipleReads |    32 | HttpRequestHeader | 11,225.7 ns | 40.46 ns | 35.87 ns |  1.00 | 0.0916 |   9,520 B |
|                 MultipleReads |    32 | ArrayBackedHeader |  7,953.4 ns | 37.65 ns | 35.22 ns |  0.71 | 0.0458 |   4,936 B |

First scenario - CreateHttpRequestMessage - is the base one, when we create a request and send it directly to the socket. In case of less than 16 headers, ArrayBackedPropertyBag is faster
Second scenario - CreateHttpRequestMessageTwice simulates the retry case. Here, even with 32 headers the benefit is about 10%
Third scenario - MultipleReads - simulates Azure.Storage case when headers are used to create a signature

API change check
API changes are not detected in this pull request.

It would be good if this PR description started with a note explaining about what this PR does, as opposed to starting with a benchmark table :-)
But I am not surprised this improves perf. The headers collection is pretty inefficient. We should send this benchmark and the scenarios we have (changing header values) that make the BCL headers collection suboptimal. Maybe this can be fixed in the BCL so that we don't have to write code like in this PR.

This PR appears to have caused test flakiness in the .NET Live tests pipeline - Beginning on 2/17, we have seen intermittent test failures in our live batch tests - https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2203247&view=ms.vss-test-web.build-test-results-tab&runId=39465973&resultId=100295&paneView=debug
Based on the timing, it appears it was caused by this commit.

@AlexanderSher @amnguye

authors: AlexanderSher KrzysztofCwalina azure-sdk seanmcc-msft


text:
Increment version for signalr releases

Increment package version after release of Azure.ResourceManager.SignalR

API change check
API changes are not detected in this pull request.


text:
[KeyVault] [Do Not Merge] Adding ECC key support

Description
These changes introduce ECC key support to the Key Vault SDK.  They correspond to the swagger update https://github.com/Azure/azure-rest-api-specs/pull/1724.

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] I have read the contribution guidelines.
[x] The pull request does not introduce breaking changes.

General Guidelines

[ ] Title of the pull request is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

SDK Generation Guidelines

[ ] If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
[ ] The generate.cmd file for the SDK has been updated with the version of AutoRest, as well as the commitid of your swagger spec or link to the swagger spec, used to generate the code.
[ ] The *.csproj and AssemblyInfo.cs files have been updated with the new version of the SDK.

closing since this will be merged into the KvDev branch instead. https://github.com/Azure/azure-sdk-for-net/pull/3815


text:
[AzureMonitorExporter] resolve AOT warnings

Related: #37734.
This PR mitigates AOT warnings in the Azure.Monitor.OpenTelemetry.Exporter library.
Azure.Monitor.OpenTelemetry.Exporter has 14 totals warnings

AzureMonitorExporterEventSource.cs

IL2026:RequiresUnreferencedCodeAttribute

Using member 'System.Diagnostics.Tracing.EventSource.WriteEvent(Int32,Object[])' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. EventSource will serialize the whole object graph. Trimmer will not safely handle this case because properties may be trimmed. This can be suppressed if the object is a primitive type.

The fix is to decorate these methods with [UnconditionalSuppressMessage("ReflectionAnalysis", "IL2026:RequiresUnreferencedCode", Justification = "Parameters to this method are primitive and are trimmer safe.")]

AzureMonitorStatsbeat.GetVmMetadataResponse & IngestionResponseHelper.GetErrorsFromResponse

IL2026:RequiresUnreferencedCodeAttribute

Using member 'System.Text.Json.JsonSerializer.Deserialize(String,JsonSerializerOptions)' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. JSON serialization and deserialization might require types that cannot be statically analyzed. Use the overload that takes a JsonTypeInfo or JsonSerializerContext, or make sure all of the required types are preserved.

IL3050:RequiresDynamicCodeAttribute

Using member 'System.Text.Json.JsonSerializer.Deserialize(String,JsonSerializerOptions)' which has 'RequiresDynamicCodeAttribute' can break functionality when AOT compiling. JSON serialization and deserialization might require types that cannot be statically analyzed and might need runtime code generation. Use System.Text.Json source generation for native AOT applications.

The fix is to use source generation.

Models\StackFrame

IL2026:RequiresUnreferencedCodeAttribute

Using member 'System.Diagnostics.StackFrame.GetMethod()' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. Metadata for the method might be incomplete or removed.

The fix is to decorate with UnconditionalSuppressMessage . GetMethod() may return null. In this case we will fall back to ToString().

LogsHelper.GetProblemId

IL2026:RequiresUnreferencedCodeAttribute

Using member 'System.Diagnostics.StackFrame.GetMethod()' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. Metadata for the method might be incomplete or removed.

The fix is to decorate with UnconditionalSuppressMessage . GetMethod() may return null. In this case we will fall back to ToString().

API change check
API changes are not detected in this pull request.

@vitek-karas, @Yun-Ting, @m-redding Please help with this review :)

authors: TimothyMothra azure-sdk


text:
Azure Deployment Manager is being decommissioned. Remove its NET SDK through this PR

Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

@ArthurMa1978 I remember there is a decommission process to follow?

@rohantagaru can you provide official announcement of this deprecation?

API change check
API changes are not detected in this pull request.

From @rohantagaru, this rp has been removed from cli last year, https://github.com/Azure/azure-cli-extensions/pull/4653

authors: ArthurMa1978 archerzz azure-sdk rohantagaru


text:
Add support for schema create or update for Swagger, WSDL and OpenApi

Fix based on REST Spec https://github.com/Azure/azure-rest-api-specs/pull/5824

@dsgouda the build failure is related to an EventHub test.

@weshaggard Please take a look at the failures.

@dsgouda  I wanted to add an extension to one of the models for better usability. Give me an hour.

@dsgouda I re-queued the failing test leg but I don't believe it is related to the changes in this PR so if it fails again merge anyway.
@jsquire looks like another EventHubs test reliability failure
Failed   Microsoft.Azure.EventHubs.Tests.ServiceFabricProcessor.OptionsTests.RuntimeInformationTest
Error Message:
 Assert.True() Failure
Expected: True
Actual:   False
Stack Trace:
   at Microsoft.Azure.EventHubs.Tests.ServiceFabricProcessor.OptionsTests.RuntimeInformationTest() in D:\a\1\s\sdk\eventhub\Microsoft.Azure.EventHubs\tests\ServiceFabricProcessor\OptionsTests.cs:line 191

I added a note about this test in issue https://github.com/Azure/azure-sdk-for-net/issues/5995, and if we see it keep failing then I'll disable it.

@dsgouda I have pushed my changes. Feel free to merge it as soon as CI passes or fails with EventHub test failure.

@dsgouda can you merge this.

authors: dsgouda solankisamir weshaggard


text:
[Schema Registry] Cross language design review

Ongoing discussion:

Design: API

what should the method return?

  - Schema:
      - content
      - SchemaProperties
 - SchemaProperties
     - id
     - version
     - ...

Design: API: SRAvroSerializer.serialize(..., schema):

what's the expected type of the parameter schema?
if string/bytes, then it's the SDK's duty to do normalize -- remove space from the input (\n\t, etc.) or the service would handle it
Laurent: good with bytes and string for p1

Design: Naming

response type(object)

SchemaId: in JS it's called SchemaIdResponse, probably we should consider a better/different/ name? what's naming convention here
Schema: same to the question above

content/string/schema

Option: all return the same object type e.g.: SchemaProperties

parameter name

Design: SchemaProperties dict mixin support?
Impl: Encoding

big/small endian problem to id/format identifier
Shall we use struct.pack/unpack to construct payload?
Ask service team/other languages how they impl this

Impl: Dependency

which packages are required for user (dependency)?

Laurent: postpone aio implementation later, only do the sync avro serializer now

Others

generate schema from class/type? the input being a object, is it pythonic?
future discussion, not now, but protobuf probably need to support this

Eng: Doc

Doc auto generation, need to ask Scott
Samples and sample readme release to official ms website
API reference

Finished discussion:

Impl: Parsing in sr and avsr

should we remove all the space (regular expression "\s") in the schema string user passes into our sdk?
it's the service's duty

Design: Typing

serialization type: string vs enum vs both
class a(str, Enum)

auto register schema for SR Serializer?

data collected and moved into onenote page, will spawn separate issues for each task

repo: Azure/azure-sdk-for-python


text:
Chat Samples are not able to run in the pipeline

Package Name:  azure-communication-chat

Describe the bug
We want to make the Python ACS samples run in the pipeline to have an extra security layer when we do our releases. We refactored our TNM, Identity and SMS samples to make them able to run in the pipeline, however, chat samples are not being able to work because of some special environment variables they need in order to run successfully.
We are seeing references to an AZURE_COMMUNICATION_SERVICE_ENDPOINT env variable in the samples. This env variable doesn't exist in the pipeline so it should be removed fron the samples or added to the key vault from the resource we use to test to avoid any inconsistencies with the env variables the Chat Client needs to initiate.
@juancamilor

This is the PR I opened to add this feature https://github.com/Azure/azure-sdk-for-python/pull/18234
If you go to the pipeline logs you can see exactly where are the errors that need to be addressed.

@LuChen-Microsoft  FYI

authors: jbeauregardb juancamilor


text:
[Test Proxy] Remove custom default matcher setup in proxy_startup

Per https://github.com/Azure/azure-sdk-for-python/pull/23148, we now call set_custom_default_matcher within proxy_startup.py in order to preserve backwards compatibility and ignore headers that we now omit from recordings. Eventually, once recordings are free from these headers, we should remove this call and use the default matcher upon startup.
Note: at this point, we should also revert any set_custom_default_matcher calls that set bodiless matching to set_bodiless_matcher. The linked PR has details about this change as well.

This is tracked by https://github.com/Azure/azure-sdk-for-python/issues/34897.


text:
Broken links in Azure Resources libraries for Python

The links to the packages are broken.
For example, clicking azure.mgmt.resources.features takes you to
https://docs.microsoft.com/en-us/python/api/azure.mgmt.resource.features
instead of
https://docs.microsoft.com/en-us/python/api/azure-mgmt-resource/azure.mgmt.resource.features

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e19ddc50-3cc0-eded-49ed-e1f2552a421c
Version Independent ID: 391c4753-337c-8a9f-fa58-2c6fc95cb6df
Content: Azure Resources libraries for Python
Content Source: docs-ref-services/latest/azure.mgmt.resource.md
Service: resources
Product: azure
Technology: azure
GitHub Login: @lisawong19
Microsoft Alias: ramyar

Hi @scbedd could you help merge the fix PR or address proper person to merge it? Thanks!

@scbedd can you review this when you get a chance?

Github auto-closed the issue when I merged the PR. Re-opening until the change is actually visible on docs.ms.

The issue was fixed already.

authors: msyyc rguptar scbedd


text:
No module named 'azure.storage'

Hello, I have tried the instruction to install, reinstall the azure package.
However, all the time I am still getting the ModuleNotFoundError: No module named 'azure.storage'.
following are the results of: pip freeze
azure==4.0.0
azure-applicationinsights==0.1.0
azure-batch==4.1.3
azure-common==1.1.16
azure-cosmosdb-nspkg==2.0.2
azure-cosmosdb-table==1.0.5
azure-datalake-store==0.0.34
azure-eventgrid==1.2.0
azure-graphrbac==0.40.0
azure-keyvault==1.1.0
azure-loganalytics==0.1.0
azure-mgmt==4.0.0
azure-mgmt-advisor==1.0.1
azure-mgmt-applicationinsights==0.1.1
azure-mgmt-authorization==0.50.0
azure-mgmt-batch==5.0.1
azure-mgmt-batchai==2.0.0
azure-mgmt-billing==0.2.0
azure-mgmt-cdn==3.0.0
azure-mgmt-cognitiveservices==3.0.0
azure-mgmt-commerce==1.0.1
azure-mgmt-compute==4.3.1
azure-mgmt-consumption==2.0.0
azure-mgmt-containerinstance==1.2.0
azure-mgmt-containerregistry==2.2.0
azure-mgmt-containerservice==4.2.2
azure-mgmt-cosmosdb==0.4.1
azure-mgmt-datafactory==0.6.0
azure-mgmt-datalake-analytics==0.6.0
azure-mgmt-datalake-nspkg==2.0.0
azure-mgmt-datalake-store==0.5.0
azure-mgmt-datamigration==1.0.0
azure-mgmt-devspaces==0.1.0
azure-mgmt-devtestlabs==2.2.0
azure-mgmt-dns==2.1.0
azure-mgmt-eventgrid==1.0.0
azure-mgmt-eventhub==2.1.0
azure-mgmt-hanaonazure==0.1.1
azure-mgmt-iotcentral==0.1.0
azure-mgmt-iothub==0.5.0
azure-mgmt-iothubprovisioningservices==0.2.0
azure-mgmt-keyvault==1.1.0
azure-mgmt-loganalytics==0.2.0
azure-mgmt-logic==3.0.0
azure-mgmt-machinelearningcompute==0.4.1
azure-mgmt-managementgroups==0.1.0
azure-mgmt-managementpartner==0.1.0
azure-mgmt-maps==0.1.0
azure-mgmt-marketplaceordering==0.1.0
azure-mgmt-media==1.0.0
azure-mgmt-monitor==0.5.2
azure-mgmt-msi==0.2.0
azure-mgmt-network==2.2.1
azure-mgmt-notificationhubs==2.0.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-policyinsights==0.1.0
azure-mgmt-powerbiembedded==2.0.0
azure-mgmt-rdbms==1.4.0
azure-mgmt-recoveryservices==0.3.0
azure-mgmt-recoveryservicesbackup==0.3.0
azure-mgmt-redis==5.0.0
azure-mgmt-relay==0.1.0
azure-mgmt-reservations==0.2.1
azure-mgmt-resource==2.0.0
azure-mgmt-scheduler==2.0.0
azure-mgmt-search==2.0.0
azure-mgmt-servicebus==0.5.2
azure-mgmt-servicefabric==0.2.0
azure-mgmt-signalr==0.1.1
azure-mgmt-sql==0.9.1
azure-mgmt-storage==2.0.0
azure-mgmt-subscription==0.2.0
azure-mgmt-trafficmanager==0.50.0
azure-mgmt-web==0.35.0
azure-nspkg==3.0.2
azure-servicebus==0.21.1
azure-servicefabric==6.3.0.0
azure-servicemanagement-legacy==0.20.6
azure-storage==0.33.0
azure-storage-blob==1.3.1
azure-storage-common==1.3.0
azure-storage-file==1.3.1
azure-storage-nspkg==3.0.0
azure-storage-queue==1.3.0
Could  anyone help on this?
Thanks a lot!

Hi @irisava
Could you confirm version of Python, version of pip, platform (Windws, Ubuntu, etc.) and exact command used to install.
Thanks you

Hello @lmazuel
Thank you for the reply!
Following is the info of my working environment:
Python 3.6.5
pip 18.1 from ...\appdata\local\programs\python\python36-32\lib\site-packages\pip (python 3.6)
Windows 10
Last command used in cmd: pip install azure-storage

azure-storage and azure-storage-blob/file/queue are incompatible and cannot work together. azure-storage is actually the deprecated old version of the three packages azure-storage-blob/file/queue
Please juste use azure-storage-blob/file/queue or just use azure-storage but not both. azure-storage-blob/file/queue is recommended if you don't have existing code base.
Thank you,

Closing for inactivity, since I believe I addressed the initial question. If this is still a problem, feel to open a new issue in the storage repo:
https://github.com/Azure/azure-storage-python
Thanks,

authors: irisava lmazuel


text:
which program will work?　（Program which can upload a text file to vector index of Azure AI search ）

I'm looking for a basic python program that will upload a text file to the vector index of Azure AI search, but
everything gives me an error, and I can't find the version number of the azure search documents package or a working ptyhon program anywhere.
Everything gives me an error.
I guess it's probably because the development is so hard that they haven't been able to check the operation of the developed program, but which program will work?
GitHub
Azure/azure-search-vector-samples
Azure/azure-sdk-for-python

Hi @TaisukeIto! Sorry to hear about your experience with the AI Search library - it's a rapidly growing service so sometimes the API changes on newer versions and samples become outdated fast.
One of our most popular samples uses the AI Search library and should have accurate behavior - here's what I found for a quick search of the SearchClient.upload_files() method: link. It looks like this sample is on 11.6.0b1.
The API reference is also here if you're curious about the details.
If you're still running into errors, please post the specific error you're running into.

authors: TaisukeIto rohit-ganguly


text:
Hey! I've been using Cash App to send money and spend using the Cash App Card. Try it using my code and we’ll each get $5. GQ4N8C8
https://cash.app/app/GQ4N8C8

Hello


text:
Initial Container Instance

FYI @yolo3301 @derekbekoe
Didn't look at the diff in details yet at the time of the PR, but naming and packaging should be ok.

Codecov Report

Merging #1330 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #1330      +/-   ##
==========================================
+ Coverage   56.04%   56.04%   +<.01%
==========================================
  Files        2691     2692       +1
  Lines       71246    71247       +1
==========================================
+ Hits        39932    39933       +1
  Misses      31314    31314

Impacted Files
Coverage Δ

...zure-mgmt-containerinstance/azure/mgmt/__init__.py
100% <100%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f477734...3bf348f. Read the comment docs.

authors: codecov-io lmazuel


text:
Revert "Communication identity api redesign (#16420)"

This reverts commit 30b917b2b377e7fadbd66c478209b3ab7427ca78.

/azp run python - communication - tests


text:
[Videoanalyzer] Fixed cspell typos in videoanalyzer

Description
Fix https://github.com/Azure/azure-sdk-for-python/issues/22681
All SDK Contribution checklist:

[X] The pull request does not introduce [breaking changes]
[X] CHANGELOG is updated for new features, bug fixes or other significant changes.
[X] I have read the contribution guidelines.

General Guidelines and Best Practices

[X] Title of the pull request is clear and informative.
[X] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[X] Pull request includes test coverage for the included changes.

CI check here: https://dev.azure.com/azure-sdk/public/_build/results?buildId=1847868&view=results
I'll merge as soon as it's green. Thanks again! 😸

/check-enforcer override

authors: kristapratico syso-jxx


text:
Added Sampler Factory

Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
All SDK Contribution checklist:

[ ] The pull request does not introduce [breaking changes]
[ ] CHANGELOG is updated for new features, bug fixes or other significant changes.
[ ] I have read the contribution guidelines.

General Guidelines and Best Practices

[ ] Title of the pull request is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

Make sure to add a changelog entry

authors: jeremydvoss lzchen


text:
[AutoRelease] t2-cognitiveservices-2023-01-23-44734(can only be merged by SDK owner)

https://github.com/Azure/sdk-release-request/issues/3679
Live test success
https://dev.azure.com/azure-sdk/internal/_build?definitionId=976
BuildTargetingString
azure-mgmt-cognitiveservices
Skip.CreateApiReview
true

issue link:https://github.com/Azure/sdk-release-request/issues/3679


text:
Black 22.3.0 Azure-Core

Nothing fancy, just use the latest black, so the Typing PRs don't fail on black, while not making those PRs full of uninteresting lines.

Ok, so we don't need this PR, it happened I didn't see we have a black config file, it's why I had so many diff


text:
[AutoPR keyvault/resource-manager] KV multiapi Readme

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/3416

This PR has been merged into https://github.com/Azure/azure-sdk-for-python/pull/2927


text:
[ml] Update docstrings to meet guidelines and fix example paths

…ines
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
All SDK Contribution checklist:

[ ] The pull request does not introduce [breaking changes]
[ ] CHANGELOG is updated for new features, bug fixes or other significant changes.
[ ] I have read the contribution guidelines.

General Guidelines and Best Practices

[ ] Title of the pull request is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-ai-ml

authors: azure-sdk diondrapeck


text:
vnext issue creator script

Resolves https://github.com/Azure/azure-sdk-for-python/issues/29344
We pin the versions of type checkers/linters in this repo so that we don't see any surprises in CI when new versions are released. To keep up with the latest, we periodically bump the pinned version of these checkers and then go through the process of getting all libraries clean for that version. We would like to improve this process by 1) giving early notice of when a version bump will happen and what errors need to be fixed in a given library and 2) standardize when we do version bumps (e.g., quarterly, the Monday after release week).
The idea behind this PR is to give library owners an early heads up of what checks are failing with the next version of the type checkers/linters and provide a deadline / merge date for when that version will be merged. It adds a script which will create GH issues if a client library is failing a vnext check for pylint, mypy, or pyright and will run as part of the test-weekly pipeline. If a library fails a vnext check, the script will either create an issue (if one doesn't exist) or update the issue with the latest dates/links to builds.
Example issue: https://github.com/Azure/azure-sdk-for-python/issues/31463

@kristapratico No complaints about the code of this PR at all!
From a strategery point of view, I'm trying to get our common code under azure-sdk-tools instead of adding new scripts/code to tox folder.
Reason being:

Makes it super easy to re-use the code here that submits a new issue

Has a place to run tests out of if you add them

Yes you could put a test file right alongside this under tox/, but that would get awkward pretty quick 😂

That being said, I'm not going to block on that.

@scbedd Ah thanks for pointing that out, I meant to mention that I wasn't sure if this was a great place for the script to live. Were you thinking under tools/azure-sdk-tools/ci_tools? I'm happy to move it in this PR.

@kristapratico absolutely have some suggestions!
tools/azure-sdk-tools/ci_tools/gh <-- access through ci_tools.gh
or
tools/azure-sdk-tools/gh_tools/ <-- would need to create top levelm so would probably just be gh_tools or whatever you come up with.
Both work. Arguably creating an issue isn't tightly bound to CI, so there are arguments for making them their own namespace.

authors: kristapratico scbedd


text:
[Key Vault] Add support for pre-backup and pre-restore operations

Description
Resolves https://github.com/Azure/azure-sdk-for-python/issues/35252. This adds client-facing support for pre-backup and pre-restore methods, for checking whether a full backup or full restore operation can be performed.
As a draft, this PR doesn't include tests because of default feature unavailability in the service. Once the feature can be easily enabled, tests and samples will be added.
All SDK Contribution checklist:

[x] The pull request does not introduce [breaking changes]
[x] CHANGELOG is updated for new features, bug fixes or other significant changes.
[x] I have read the contribution guidelines.

General Guidelines and Best Practices

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-keyvault-administration

authors: azure-sdk mccoyp


text:
fixed warning for aio and get call function tools for stream within t…

…he SDK
Description
Get rid of warning in aio AgentOperation.    TO do that, I copied the AgentsOperation from sync to async/aio and modify accordingly.
Call functions within SDK for streaming instead of asking developers to call in their code.   I will do this for non-streaming.
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
All SDK Contribution checklist:

[ ] The pull request does not introduce [breaking changes]
[ ] CHANGELOG is updated for new features, bug fixes or other significant changes.
[ ] I have read the contribution guidelines.

General Guidelines and Best Practices

[ ] Title of the pull request is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

Give some time to review :)

authors: howieleung jhakulin


text:
Management namespace approval improvements for new UI

Today the management namespace approval is a manual process.  Teams have to create a GitHub issue and follow the guidance on this wiki page to initiate the process.  https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/821/Naming-for-new-initial-management-or-client-libraries-(new-SDKs)
Questions:

Should a dedicated team of architects be responsible for approving the management plane namespaces?
Should the archboard use APIView to approve the namespaces?  This is how it is done for data plane.
Are teams blocked from releasing new initial SDKs if they do not have approval for namespace for management plane?  This is implemented for data plane.
If we are going to continue with the current process, then we should create the GitHub issue for the user in the Release Planner.  The user can enter the suggested names of the namespaces and we have all of the other information needed to create the GitHub issue for them using the template - https://github.com/Azure/azure-sdk-pr/issues/new?assignees=kyle-patterson%2C+ronniegeraghty&labels=architecture%2C+board-review%2C+mgmt-namespace-review&projects=&template=adp_mgmt_namespace_review.md&title=Board+Review%3A+Management+Plane+Namespace+Review+<client+library+name>

All of the questions have been covered either in docs or already implemented in the SDK release app
The only reminaing one is

If we are going to continue with the current process, then we should create the GitHub issue for the user in the Release Planner. The user can enter the suggested names of the namespaces and we have all of the other information needed to create the GitHub issue for them using the template - https://github.com/Azure/azure-sdk-pr/issues/new?assignees=kyle-patterson%2C+ronniegeraghty&labels=architecture%2C+board-review%2C+mgmt-namespace-review&projects=&template=adp_mgmt_namespace_review.md&title=Board+Review%3A+Management+Plane+Namespace+Review+<client+library+name>

which will be covered by https://github.com/Azure/azure-sdk-tools/issues/4601

authors: ladonnaq maririos

repo: Azure/azure-sdk-tools


text:
Board Review: Event Grid System Events

Thank you for submitting this review request. Thorough review of your client library ensures that your APIs are consistent with the guidelines and the consumers of your client library have a consistently good experience when using Azure.
The Architecture Board reviews Track 2 libraries only. If your library does not meet this requirement, please reach out to Architecture Board before creating the issue.
Please reference our review process guidelines to understand what is being asked for in the issue template.
To ensure consistency, all Tier-1 languages (C#, TypeScript, Java, Python) will generally be reviewed together.  In expansive libraries, we will pair dynamic languages (Python, TypeScript) together, and strongly typed languages (C#, Java) together in separate meetings.
For Tier-2 languages (C, C++, Go, Android, iOS), the review will be on an as-needed basis.
Before submitting, ensure you adjust the title of the issue appropriately.
Note that the required material must be included before a meeting can be scheduled.
Contacts and Timeline

Responsible service team: API Management, DataBox
Main contacts: @JoshLove-msft
Expected code complete date: 1/6/23
Expected release date: 1/13/23

About the Service

Link to the service REST APIs:
https://github.com/Azure/azure-rest-api-specs/pull/21771
https://github.com/Azure/azure-rest-api-specs/pull/21945

.NET

APIView Link: To be added

Java

APIView Link: To be added

Python

APIView Link: To be added

TypeScript

APIView Link: To be added

Scheduled for Jan 12th, from 2:05PM - 4PM PST.

As per email, I cannot attend this meeting  since I'm based in Belgium.
Can we move this to 10 PM CET / 1AM PT please? I'm OK to stay up for the meeting then.

Even I can't attend this meeting since I'm based in India.
Can we move it to 12 AM IST / 10:30 AM PST ?

@tomkerkhove & @aakash049 the review session time can be moved up, just let me get an agreed upon time. Would 10:05AM - 12PM PST work for you both?

I can check but allocating 2h in the evening is a bit much as it feels like the discussion will not take that much given it's 2 different topics. Can we split them or have some indication?
I can potentially do 10:30-11:30 but I think this is still too late for @aakash049 who is in India.

From the context in the issue description it looks like the two topics of this meeting will be 11 new events for API Management and an Event Grid system topic for DataBox. @aakash049 & @tomkerkhove, can you let me know which part you're interested in, and I'll add info to the review session stating which topic should go first and second. Since @aakash049 is in India and it will be latest for them, we can arrange it, so their topic is covered from 10-11AM PST and @tomkerkhove's topic is covered from 11AM-12PM PST. Could that work for you both?

It's not ideal because that is my 8 PM but I'll make it work :) I'm joining for the 11 new events for API Management

I'll be discussing about Event Grid System topic for Databox, 10-11 AM PST works for me.

Okay, thanks for being flexible. I'll speak with the architects now to confirm the time. There is a chance they could do 9AM-11AM, but the normal morning time slot is 10-12. I'll keep you posted.

Scheduled for 1/12 from 9:05AM - 11AM PST

Thanks! I'll join at 10 AM PST to represent APIM

@ronniegeraghty is this scheduled for 10-12 or 9-11. It sounds like both service reps are available from 10 onward?

Adding a note here to reflect the update in the meeting invite.
The review session is scheduled to take place between 9:05AM - 11AM PST.
@aakash049 will be going first for the DataBox related topic from 9:05AM - 9:35AM PST.
Then, @tomkerkhove  will be going for the API Management related topic from 9:35 - 10:05AM PST

Recording (MS INTERNAL ONLY)

authors: JoshLove-msft aakash049 ronniegeraghty tg-msft tomkerkhove

repo: Azure/azure-sdk


text:
Documentation - Indexes Should Account for Renamed Packages

Right now, we generate github.io landing pages purely based off of what is present within the repo.
We know that packages will be renamed (most recent example being azure-storage-fileshare), so we may need to account for this. Depending on the level of investment to github.io docs, this may be important.
Most straightforward way I can think of is to always include locations in the index for packages that we've published to blob storage before.\
CC @kaerm

This is super rare and we don't lose any history/data related to docs without this.  Cutting this feature since we'll never prioritize it enough to do it.  The amount of work needed doesn't match the benefit given how rare this will occur.

authors: kurtzeborn scbedd


text:
Add blog for JS abort controller

Let me know any comments. Edits allowed from maintainers, so feel free to fix any minor issues you like 😀

All feedback addressed, let me know if there are more suggestions!

Was the second file "how q" added by accident?  Might want to remove it from the PR.

that extra file is very strange! I'll slice it out of the history.

Well clearly I messed up my filter-branch.

Recreating PR.

@jongio Thanks so much for the feedback, I addressed most of this feedback over in #1240.
I'm open to changing the intro paragraph to a bullet list and also interested in how to improve the point about separation of concerns between signal and controller.

authors: adrianhall bterlson


text:
Need -whatif flag for Azcopy Sync tool

What command did you run?
Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.
NA - feature request
What problem was encountered?
Please add -whatif flag for Azcopy Sync tool to estimate exact the outcome of the command. Especially useful when we are dealing with large number of files and having --delete-destination and --recursive flags.
How can we reproduce the problem in the simplest way?
NA
Have you found a mitigation/solution?
No

This sounds like our --dry-run flag, which doesn't match this exact functionality at this point.

repo: Azure/azure-storage-azcopy


text:
Allow direct Load webjobs dll

Resolves https://github.com/Azure/azure-webjobs-sdk-script/issues/1508
Allow Functions to directly load and consume a WebJobs DLL that may come from precompiled tooling.
The function.json has a new "configurationSource" : "attributes" flag in it.
This builds on several previous fixes:

This skips the InvokerBase and ILGeneration path. This  builds on some previous changes to move non-invocation responsibility (logging, metrics, return values, etc) out of the invoker path.
Recent fix to billing: https://github.com/Azure/azure-webjobs-sdk-script/issues/578
It builds on [FunctionName] and Return value support from the SDK.

Can we add something to the docs on this change. Seems pretty significant.

authors: MikeStall dallancarr

repo: Azure/azure-webjobs-sdk-script


text:
[Feature Request]: add module for ADF Linked Services

Description
currently there is no module for ADF Linked Services resource https://docs.microsoft.com/en-us/azure/templates/microsoft.datafactory/factories/linkedservices?pivots=deployment-language-bicep

Hey @clintgrove,
I just migrated this issue over from CARML. Please take a look and triage if still relevant :)

I am working on this, due to raise PR today or tomorrow 25th june 2024

authors: AlexanderSehr clintgrove tyconsulting

repo: Azure/bicep-registry-modules


text:
[AVM Module Issue]: Virtual Network Gateway - WAF and APRL alignment

Check for previous/existing GitHub issues

[x] I have checked for previous/existing GitHub issues

Issue Type?
Feature Request
Module Name
avm/res/network/virtual-network-gateway
(Optional) Module Version
No response
Description
We've been asked to ensure module defaults alignment for WAF and APRL for several modules. For the Virtual Network Gateway module can we please update the following default.
For Public IP's used by the gateway, set zone configuration to all zones [1,2,3] as the default value

Superseding #3247

(Optional) Correlation Id
No response

Hey @fabmas,
Please triage this issue when you get the chance 🙂


text:
AzureML: Datastore created with no subscription_id and resource_group

Bicep version
Bicep CLI version 0.4.1318 (ee0d808f35)
Describe the bug
Deploying a datastore through bicep results in a datastore without subscription_id and resource group setted in the azureml workspace. The workspace is correctly working but doesn't have the direct link to the blob storage.

To Reproduce
simply create a datastore resource with
resource datastore 'Microsoft.MachineLearningServices/workspaces/datastores@2021-03-01-preview`

Can you share the full code sample that you deployed?
@stan-sz - do you happen to know anything about this one?

The code that I deployed is something like the following code:
resource datastore 'Microsoft.MachineLearningServices/workspaces/datastores@2021-03-01-preview' = {
  name: '${workspace_name}/dstr_preproc'
  properties: {
    contents: {
      contentsType: 'AzureBlob'
      accountName: ext_storage_reference.name
      containerName: 'bscont-preproc'
      credentials: {
        credentialsType: 'AccountKey'
        secrets: {
          key: listKeys(ext_storage_reference.id, '2019-06-01').keys[0].value
          secretsType: 'AccountKey'
        }
      }
      endpoint: environment().suffixes.storage // https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-functions-deployment
      protocol: 'https'
    }
  }
}

The issue is still there also with the stable version of 2022-05-01
resource pipeDatastore 'Microsoft.MachineLearningServices/workspaces/datastores@2022-05-01' = {
  name: '${workspace_name}/dstr_preproc'
  properties: {
    datastoreType: 'AzureBlob'
    accountName: ext_storage_reference_tmp.name
    containerName: 'bscont--preproc'
    credentials: {
      credentialsType: 'AccountKey'
      secrets: {
        key: listKeys(ext_storage_reference_tmp.id, '2019-06-01').keys[0].value
        secretsType: 'AccountKey'
      }
    }
    endpoint: environment().suffixes.storage // https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-functions-deployment
    protocol: 'https'
  }
}

using the latest api 2022-06-01-preview  everything works due to the fact that now we can specify subscriptionId and resourceGroup
resource tmpPipeDatastore 'Microsoft.MachineLearningServices/workspaces/datastores@2022-06-01-preview' = {
  name: '${workspace_name}/dstr_preproc'
  properties: {
    datastoreType: 'AzureBlob'
    accountName: ext_storage_reference_tmp.name
    containerName: 'bscont-preproc'
    subscriptionId: env.subcription_id
    resourceGroup: rg_name
    credentials: {
      credentialsType: 'AccountKey'
      secrets: {
        key: listKeys(ext_storage_reference_tmp.id, '2019-06-01').keys[0].value
        secretsType: 'AccountKey'
      }
    }
    endpoint: environment().suffixes.storage // https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-functions-deployment
    protocol: 'https'
  }
}

The links are created correctly but they end up in a strange "not found" error

Here for example i clicked on the container link

This is an issue with the Azure ML Resource Provider. Can you open an Azure support case, so this can be routed to the Azure ML team?

authors: VELCpro alex-frankel

repo: Azure/bicep


text:
Azure Spring Apps / App Deployment : API missing to get relativePath

Bicep version
az bicep version

Bicep CLI version 0.11.1 (030248df55)

Describe the bug
I have a Bicep snipet to create an Azure Spring Apss / App Deployment :
// https://learn.microsoft.com/en-us/azure/templates/microsoft.appplatform/2022-11-01-preview/spring/apps/deployments?pivots=deployment-language-bicep#usersourceinfo-objects
resource adminserverappdeployment 'Microsoft.AppPlatform/Spring/apps/deployments@2022-11-01-preview' = {
  name: 'default'
  parent: adminserverapp
  sku: {
    name: azureSpringAppsSkuName
  }
  ….
    source: {
      version: deploymentVersion
      type: 'Jar' // Jar, Container or Source [https://learn.microsoft.com/en-us/azure/templates/microsoft.appplatform/2022-11-01-preview/spring/apps/deployments?pivots=deployment-language-bicep#usersourceinfo](https://learn.microsoft.com/en-us/azure/templates/microsoft.appplatform/2022-11-01-preview/spring/apps/deployments?pivots=deployment-language-bicep#usersourceinfo)
      jvmOptions: '-Xms512m -Xmx1024m -Dspring.profiles.active=mysql,key-vault,cloud'
      // [https://learn.microsoft.com/en-us/rest/api/azurespringapps/apps/get-resource-upload-url?tabs=HTTP#code-try-0](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Frest%2Fapi%2Fazurespringapps%2Fapps%2Fget-resource-upload-url%3Ftabs%3DHTTP%23code-try-0&data=05%7C01%7CSteve.Pincaud%40microsoft.com%7C6ba1c341743548429ec708daf35eb3dd%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638089885656486803%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hZBkaW6MJfx0XQkTVt9cmn0hveGmDpqwACN6ZsxKyIc%3D&reserved=0)
      // should be a link to a BLOB storage
      relativePath: 'https://stasapetcliasa.blob.core.windows.net/petcliasa-blob/asa-spring-petclinic-admin-server-2.6.6.jar'
      runtimeVersion: 'Java_11'
    }
  }
}

There is this API which result provides the relativePah field , but how to get that result in Bicep ?
Without this value, it looks like there is no way to create a Deployment with Bicep which is asked by my customer.
To Reproduce
See snippet above
Additional context
This is a show stopper for my customer who wants to use Bicep only WITHOUT any extra steps in a script
Ask: this get-resource-upload-url API should be callable through Bicep
@alex-frankel

The AppPlatform/Spring RP team is looking into this

Hi @ezYakaEagle442
If you need to create a new deployment, you can fill the the relativePath with a placeholder <default>.
@description('The instance name of the Azure Spring Cloud resource')
param springCloudInstanceName string

param location string = resourceGroup().location

resource springCloudInstance 'Microsoft.AppPlatform/Spring@2022-11-01-preview' = {
  name: springCloudInstanceName
  location: location
  sku: {
    name: 'S0'
    tier: 'Standard'
  }
  properties: {
  }
}

resource apiGatewayApp 'Microsoft.AppPlatform/Spring/apps@2022-11-01-preview' = {
  name: 'api-gateway'
  parent: springCloudInstance
}

resource apiGatewayDeploymentApp 'Microsoft.AppPlatform/Spring/apps/deployments@2022-11-01-preview' = {
  name: 'default'
  parent: apiGatewayApp
  sku: {
    name: 'S0'
  }
  properties: {
    active: true
    source: {
      relativePath: '<default>'
      type: 'Jar'
    }
    deploymentSettings: {
      resourceRequests: {
        cpu: '1'
        memory: '2Gi'
      }
    }
  }
}

If you need a real storage location that can be used to upload artifacts and pass to the deployment, you need a POST call to the app's getResourceUploadUrl action. You need to leverage the Deployment Script support in bicep to do this.

create a user assigned identity
assign Contributor role for the identity to the target resource group that contains the Azure Spring Apps instance
Add the following snippet to get the URL.

resource getUploadUrl 'Microsoft.Resources/deploymentScripts@2020-10-01' = {
  name: 'get-upload-url'
  location: location
  kind: 'AzureCLI'
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      // replace the ??xxx?? placeholder below with your identity properties
      '${resourceId('??your-identity-group??', 'Microsoft.ManagedIdentity/userAssignedIdentities', '??your identity name??')}': {}
    }
  }
  properties: {
    forceUpdateTag: utcValue
    azCliVersion: '2.40.0'
    timeout: 'PT30M'
    scriptContent: 'az rest --method post --url ${apiGatewayApp.id}/getResourceUploadUrl?api-version=2022-11-01-preview'
    retentionInterval: 'P1D'
  }
}

// you can get the url and path using the following assignment
var relativePath = getUploadUrl.properties.outputs.relativePath
var uploadUrl = getUploadUrl.properties.outputs.uploadUrl

However, if you want to do real deployment using bicep (upload JAR and then patch deployment), it's not a good idea IMO as bicep is more ARM Template oriented. You will need to write further deployment scripts to call curl to upload your JAR. Reference: https://blog.soft-cor.com/uploading-large-files-to-an-azure-file-share-using-a-shell-script-and-standard-linux-commands/

authors: allxiao ezYakaEagle442 stephaniezyen


text:
Warning: Unable to fetch all az cli versions

Warning: Unable to fetch all az cli versions, please report it as an issue on https://github.com/Azure/CLI/issues. Output: ***
"name": "azure-cli",
"tags": [
"0.10.0",
"0.10.1",
"0.10.10",
"0.10.11",
"0.10.12",
"0.10.13",
"0.10.14",
"0.10.2",
"0.10.3",
"0.10.4",
"0.10.5",
"0.10.6",
"0.10.7",
"0.10.8",
"0.9.10",
"0.9.13",
"0.9.14",
"0.9.15",
"0.9.16",
"0.9.17",
"0.9.18",
"0.9.19",
"0.9.2",
"0.9.20",
"0.9.4",
"0.9.5",
"0.9.6",
"0.9.7",
"0.9.8",
"0.9.9",
"2.0.24",
"2.0.26",
"2.0.27",
"2.0.28",
"2.0.29",
"2.0.31",
"2.0.32",
"2.0.34",
"2.0.37",
"2.0.38",
"2.0.41",
"2.0.42",
"2.0.43",
"2.0.44",
"2.0.45",
"2.0.46",
"2.0.47",
"2.0.49",
"2.0.50",
"2.0.51",
"2.0.52",
"2.0.53",
"2.0.54",
"2.0.55",
"2.0.56",
"2.0.57",
"2.0.58",
"2.0.59",
"2.0.60",
"2.0.61",
"2.0.62",
"2.0

Hi @vn0siris, this is a duplicate issue of #153, which has been fixed in #154. You can point to master branch as a temporary workaround. I will inform you once the new version is released.

repo: Azure/cli


text:
Handle expires_on in int format

Unmarshal the value into an interface{} and perform the proper
conversion depending on the underlying type.
Thank you for your contribution to Go-AutoRest! We will triage and review it as soon as we can.
As part of submitting, please make sure you can make the following assertions:

[ ] I've tested my changes, adding unit tests if applicable.
[ ] I've added Apache 2.0 Headers to the top of any new source files.

Fixes https://github.com/Azure/go-autorest/issues/696

Looking at the related issue, it looks to me that expires_on, at least in that example, isn't the number of seconds from now but probably from the Unix epoch.  I need to take a closer look.

Everywhere except App Service, expires_on is epoch seconds, either as a number or a string.

And App Service is as a time-stamp correct?

OK I did a little digging.  Token.Expires() already treats ExpiresOn as Unix time.  It does mean though that our handling of expires_on in date-time format is incorrect at present.

authors: chlowell jhendrixMSFT

repo: Azure/go-autorest

license_source: bigquery


text:
Signed version of kubelogin.exe

Can we get a signed version of kubelogin.exe, we are only allowed to use signed binaries and exe in our production systems.

ack. I think the upcoming publishing update should be able to address it

authors: kurian-dm weinong

repo: Azure/kubelogin


text:
Disabling PR Workflow for Forked PRs

Using trigger pull_request_target allows workflows from forked repos to get access to the secrets and GitHub tokens of this repository, as specified here.
Until testing is fixed, we are removing the capability to run tests on forked PRs to resolve a security issue. After testing is enabled for the GitHub Action, we will enable the workflow accordingly.

Note: Support for running this workflow on forked branches will be added after proper investigation. There were 2 approaches of fixing this:

Either support running only on non forked branches. E.x. functions GitHub Action.
Make the workflow work explicitly E.x. SQL Deploy GitHub Action.

We have picked the first approach, and will investigate more for the second approach

repo: Azure/load-testing


text:
Try running MMLSPARK in YARN mode

Hi, We are trying to use mmlspark in a Cloudera environment using pyspark through terminal [1] and Cloudera Data Science Workbench (CDSW) [2].
All our efforts have failed and we wonder if this option is possible. The only way we've got it working is to use pyspark without yarn and even if it runs we got another error [3].
*We also tried to run in a Google Cloud Dataproc cluster with same error [4]
[1] Terminal
pyspark2 --master local --deploy-mode yarn--packages Azure:mmlspark:0.14,com.microsoft.ml.lightgbm:lightgbmlib:2.1.250,com.jcraft:jsch:0.1.54,com.microsoft.cntk:cntk:2.4,io.spray:spray-json_2.11:1.3.2,org.openpnp:opencv:3.4.2-0

[2] CDSW
from pyspark.sql import SparkSession warehouseLocation = "/prod/bcp/edv/mesapymesh/datain" jarsLocation = "/home/cdsw/" spark = SparkSession\ .builder.appName("SparkML")\ .config("spark.sql.warehouse.dir", warehouseLocation)\ .config("spark.jars.ivy", jarsLocation)\ .config("spark.jars.packages", "Azure:mmlspark:0.14,com.microsoft.ml.lightgbm:lightgbmlib:2.1.250,com.jcraft:jsch:0.1.54,com.microsoft.cntk:cntk:2.4,io.spray:spray-json_2.11:1.3.2,org.openpnp:opencv:3.4.2-0")\ .enableHiveSupport()\ .getOrCreate()

[3] Error running in local mode

[4] Google Cloud Data Data Proc error
Code
pyspark --packages Azure:mmlspark:0.14
Log
`Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/10/04 17:31:53 WARN org.apache.spark.deploy.yarn.Client: Same path resource file:/home/cticbigdata/.ivy2/jars/Azure_mmlspark-0.14.jar added multiple times to distributed cache.
18/10/04 17:31:53 WARN org.apache.spark.deploy.yarn.Client: Same path resource file:/home/cticbigdata/.ivy2/jars/io.spray_spray-json_2.11-1.3.2.jar added multiple times to distributed cache.
18/10/04 17:31:53 WARN org.apache.spark.deploy.yarn.Client: Same path resource file:/home/cticbigdata/.ivy2/jars/com.microsoft.cntk_cntk-2.4.jar added multiple times to distributed cache.
18/10/04 17:31:53 WARN org.apache.spark.deploy.yarn.Client: Same path resource file:/home/cticbigdata/.ivy2/jars/org.openpnp_opencv-3.2.0-1.jar added multiple times to distributed cache.
18/10/04 17:31:53 WARN org.apache.spark.deploy.yarn.Client: Same path resource file:/home/cticbigdata/.ivy2/jars/com.jcraft_jsch-0.1.54.jar added multiple times to distributed cache.
18/10/04 17:31:53 WARN org.apache.spark.deploy.yarn.Client: Same path resource file:/home/cticbigdata/.ivy2/jars/com.microsoft.ml.lightgbm_lightgbmlib-2.1.250.jar added multiple times to distributed cache.
ivysettings.xml file not found in HIVE_HOME or HIVE_CONF_DIR,/etc/hive/conf.dist/ivysettings.xml will be
...
Using Python version 2.7.9 (default, Sep 25 2018 20:42:16)
SparkSession available as 'spark'.

import mmlspark
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named mmlspark
`

[5] Notes

CDH 5.12
Spark 2.2.0 CDH / Spark 2.2.1 GCD
Python 2.7.13 / 3.6

Thanks in advance for your help. If you need more information or something else I will be checking for news.

Hi,
You should only need the option
"--packages Azure:mmlspark:0.14"
The other options shouldn't matter.  MMLSpark should work anywhere where spark is deployed, it shouldn't matter what cluster you are using.  Having said that, I've only tested it on Azure Databricks and HDInsight.  If you want to meet over skype I could try and debug it with you, but I don't have access to a cloudera workbench unfortunately :(.
Thank you, Ilya

@antoniocachuan
Can you also try on spark 2.3?  We only support spark 2.3 now (older versions support 2.2).
You can also send me an email at mmlspark-support@microsoft.com if you want to diagnose your issue.

@imatiach-msft
Thanks for your answer, In this moment is not possible to test it in Spark 2.3. Also I tried with "--packages Azure:mmlspark:0.14" in Google Cloud Data Proc cluster with the same results.
PD: I really appreciate your help, just emailed you.
Regards,
Antonio C.

@antoniocachuan the strange thing is, I don't see any errors anywhere.  It looks like you retrieved the jar so you would think that it would just work.  I'm not quite sure what the problem might be.  We could try and add the python files manually from the zip to see if anything fails.  Otherwise, when using --packages it should just pick up the python files and import them.  My guess is something in the import step if failing, but that might not be the case because I don't see an error anywhere.

@imatiach-msft  I tried in CDH running --package using the Scala API and It works, now I am getting a error related to the issue #335  also I could test adding manually the python files.
spark2-shell --master yarn --packages Azure:mmlspark:0.14,com.microsoft.ml.lightgbm:lightgbmlib:2.1.250

Error #335
Caused by: java.lang.UnsatisfiedLinkError: /data/06/yarn/nm/usercache/s16746/appcache/application_1538195866523_1970/container_e100_1538195866523_1970_01_000002/tmp/mml-natives2115945537512894448/lib_lightgbm.so: /lib64/libm.so.6: version GLIBC_2.23 not found (required by /data/06/yarn/nm/usercache/s16746/appcache/application_1538195866523_1970/container_e100_1538195866523_1970_01_000002/tmp/mml-natives2115945537512894448/lib_lightgbm.so)

@antoniocachuan  I also encountered the similar problem of ‘No module named mmlspark’，but I compile the source code of mmlspark-0.15, i install 'mmlspark-0.15-py2.py3-none-any.whl' package to my ubuntu16.04 envrionment，this problem is gone！

@antoniocachuan hello, have you solved this question? can you give me some advise ,thanks

authors: antoniocachuan imatiach-msft kunguang vinglogn

repo: Azure/mmlspark


text:
Support arrays in property definitions

The Plug and Play documentation states that the definition of arrays in properties is not supported yet (https://docs.microsoft.com/en-us/azure/iot-develop/concepts-modeling-guide).
Since it is now finally possible to have arrays in the device twin tough (https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-device-twins#tags-and-properties-format), it should also be possible to define arrays in component properties.

this is something we are targeting for DTDL v3. Stay tuned for upcoming updates.

DTDL v3 has been published as preview, with support for arrays in properties.
Can you close this this issue?

repo: Azure/opendigitaltwins-dtdl


text:
Bump sidecar image

Bump sidecar image to resolve CVEs

/azp run pr-e2e-arc

repo: Azure/osm-azure


text:
Should be portalfx in stead of portalf in the breaking changes link

Should be portalfx in stead of portalf in the breaking changes link

Docs Build status updates of commit e65372f:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

portal-sdk/generated/downloads.md
:bulb:Suggestion
View
Details

portal-sdk/generated/downloads.md

[Suggestion] Missing attribute: author. Add the current author's GitHub ID.
[Suggestion] Missing attribute: title. Add a title string to show in search engine results.

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Docs Build status updates of commit e65372f:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

portal-sdk/generated/downloads.md
:bulb:Suggestion
View
Details

portal-sdk/generated/downloads.md

[Suggestion] Missing attribute: author. Add the current author's GitHub ID.
[Suggestion] Missing attribute: title. Add a title string to show in search engine results.

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

@nickharris please take a look

My bad I only changed display link :S

authors: mikekinsman ppgovekar

repo: Azure/portaldocs


text:
feat(#60): include CodeQL@v2 workflow

Include CodeQL workflow to security scan

Changes are added to resolve the conflicts, so closing it here.

authors: BALAGA-GAYATRI jbenaventem

repo: Azure/powershell


text:
Possibility to inject key-vault values as environment variables with or without k8s secrets

Describe the solution you'd like
Many applications have a native ability to read options/parameters from environment variables like e.g. dot net core. Right now to get that behavior we need to:

Mount the files that won't be used later
Add secrets as k8s secrets that won't be used later
Map k8s secrets to env.variables

For many applications there is not much need for all of these and it would be a nice feature to just allow injection of variables without the need to configure either mounts or secrets.
Anything else you would like to add:
It would be a cool additional feature if injecting as env. variable can update k8s secrets (like what is happening now during the mount if configured) with enabled auto-rotation that will restart pods to inject new environment variables with renewed secrets if the secret has changed.
Or perhaps it's possible to evaluate the injected variables instead of creating a k8s secret to determine if pod should be restarted.
That would allow for completely pain-free secret update, like e.g. database password update.

Are we able to use Deployment env.valueFrom.secretKeyRef along with SecretProviderClass secretObjects already?

Mount the files that won't be used later

Even with the ability to use env.valueFrom.secretKeyRef, there is still a requirement to mount the files, otherwise the synced secrets aren't created. That's a bit unfortunate; it makes it more complicated to build a chart that can use regular secrets OR the CSI driver in different environments. Each deployment needs to be modified to mount the volume from the CSI driver.

Is there anyway to expose vault secret directly as pod's env variables.  My client does not allow you to use k8s secret due to base64 encode, which is still clear text. For e.g some db and sensitive data store credentials really need to be hidden from K8s admins/developers.

authors: ilya-git kpkool ltouro ms1111


text:
Nextjs + Authjs v5 Callback Error with Duende Identity Server provider in a Azure Static Web App

Describe the bug
After the user is successfully logged in on my duende identity server v6, it tries to redirect the user to the specified callback url (following the Auth.js documentation here).
Here's an example redirect url:

https://mydomain/api/auth/callback/duende-identity-service?code=3D96EBB5721191CD18CDBBFE5FFD818017E4C59F09214552AE74636913DB21B6-1&scope=openid profile email&session_state=G2eqKlEXsiOGOl0W5zNmDk7MloXu18w1M3YapSqv7qI.E986B1AED0AC76CFE51406215F9A08F6&iss=myidentityserver
My SWA instead of retrieving the session and redirecting back the user to the "dashboard" it returns a 302 status code with this weird url as location. https://1dd4069c374e:8080/api/auth/error?error=Configuration
On localhost the redirect works as expected.
To Reproduce
Can't give much details of the code as it's private. But will try to create a mock application to replicate this behaviour.
auth.config.ts
export default {
  providers: [
    DuendeIDS6Provider({
      id: 'duende-identity-service', // default id duende-identityserver6!!
      name: 'Duende Identity Service',
      clientId: process.env.AUTH_DUENDE_IDENTITY_SERVER6_ID!,
      clientSecret: process.env.AUTH_DUENDE_IDENTITY_SERVER6_SECRET!,
      issuer: process.env.AUTH_DUENDE_IDENTITY_SERVER6_ISSUER,
    }),
  ],
} satisfies NextAuthConfig;

middleware.ts
const intlMiddleware = createMiddleware({
  defaultLocale,
  localePrefix,
  locales,
  pathnames,
});

const authMiddleware = auth(
  (req: NextRequest & { auth: Session | null }): Response | void => {
    const session = req.auth;

    // Handle session

    return intlMiddleware(req);
  },
);

const middleware = (req: NextRequest) => {
  // some validations

  if (isAuthPage) {
    return (authMiddleware as any)(req);
  }

  if (isPublicPage) {
    return intlMiddleware(req);
  }
  return (authMiddleware as any)(req);
};

export const config = {
  matcher: ['/((?!api|_next/static|_next/image|favicon.ico|.*.swa).*)/'],
};

export default middleware;

staticwebapp.config.json
{
  "forwardingGateway": {
    "allowedForwardedHosts": [
      "mydomain"
    ]
  }
}

Expected behavior
Location should be https://mydomain/dashboard
Actual response:

Device info (if applicable):

OS: Windows
Browsers: Brave, Firefox, Chrome, Edge
Version: Latest

@OsoThevenin This was happening to me too. I spent quite a while working through the code and realised the way AuthJS was setting the hostname was a bit odd. The weird url is actually the HOST of the server.
I can't recall exactly helped me work around the issue but it was either setting the AUTH_URL or the AUTH_REDIRECT_PROXY_URL to the actual domain i.e. "https:///api/auth"
See this issue https://github.com/nextauthjs/next-auth/issues/10928#issuecomment-2121092912

@OsoThevenin This was happening to me too. I spent quite a while working through the code and realised the way AuthJS was setting the hostname was a bit odd. The weird url is actually the HOST of the server.
I can't recall exactly how I worked around the issue but it was either setting the AUTH_URL or the AUTH_REDIRECT_PROXY_URL to the actual domain i.e. "https:///api/auth"
See this issue nextauthjs/next-auth#10928 (comment)

Definetly this helped fix the issue. Thanks a lot ❤

authors: OsoThevenin alasdairmackenzie

repo: Azure/static-web-apps


text:
AzureStaticWebApp step fails immediately

For the life of me, I can't get this AzureStaticWebApp pipeline step to succeed.
I have a react app that I created through the standard create-react-app npx command. I tried following a bunch of official and unofficial tutorials, but nothing I've tried has worked. I've seen many tutorials get you to use the GitHub Actions flow, but I want to get an Azure DevOps CI/CD pipeline setup for this react project.
The one thing that I suspect might be causing some issues is that I am using a self-hosted Agent (vsts-agent-win-x64-2.188.3) to run the pipelines, since I dont have access to the hosted parallelism
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

I run my pipeline
AzureStaticWebApp step fails immediately

GitHub Actions or Azure Pipelines workflow YAML file

trigger:
  - main

pool:
  name: Default

steps:
  - checkout: self
    submodules: true

  - task: Npm@1
    displayName: 'npm install'
    inputs:
      verbose: false

  - task: Npm@1
    displayName: 'npm run build'
    inputs:
      command: custom
      verbose: false
      customCommand: 'run build'

  - task: PublishBuildArtifacts@1
    displayName: 'Publish Artifact: drop'
    inputs:
      PathtoPublish: build

  - task: AzureStaticWebApp@0
    inputs:
      app_location: '/build'
      azure_static_web_apps_api_token: '$(deployment_token)'

Output of AzureStaticWebApp step

2021-07-01T04:36:39.8899695Z ##[section]Starting: AzureStaticWebApp
2021-07-01T04:36:39.9036032Z ==============================================================================
2021-07-01T04:36:39.9036303Z Task         : Deploy Azure Static Web App
2021-07-01T04:36:39.9036657Z Description  : [PREVIEW] Build and deploy an Azure Static Web App
2021-07-01T04:36:39.9036859Z Version      : 0.187.1
2021-07-01T04:36:39.9037027Z Author       : Microsoft Corporation
2021-07-01T04:36:39.9037214Z Help         : https://aka.ms/swadocs
2021-07-01T04:36:39.9037436Z ==============================================================================
2021-07-01T04:36:40.1961223Z ##[section]Finishing: AzureStaticWebApp

staticwebapp.config.json file

{
  "navigationFallback": {
    "rewrite": "/index.html"
  }
}

Expected behavior
The AzureStaticWebApps step completes and deploys my react project to my Azure Static Web App
Screenshots
Here is what my entire pipeline looks like

Any help would be greatly appreciated! Thanks

Is the VM running the pipeline a Windows machine? We've seen this in the past if the VM is not capable of running the task startup script.

Hi miwebst,
I am running the Agent on my own local machine, which is Windows 10.
Did the issue that you are referring to get resolved? If so, how did they resolve it?
Thanks

I FINALLY FIXED MY ISSUE!!!!
When you mentioned that it might be a Windows problem, I used my Ubuntu VM:
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal
Then I had to install npm and docker.io:
sudo apt install npm
sudo apt-get install docker.io
Then I had to setup docker for my user on the machine:
https://www.digitalocean.com/community/questions/how-to-fix-docker-got-permission-denied-while-trying-to-connect-to-the-docker-daemon-socket
After that I was able to run my pipeline with the following YAML:
trigger:
  - main

pool:
  name: Default

steps:
  - checkout: self
    submodules: true

  - task: AzureStaticWebApp@0
    inputs:
      app_location: '/'
      output_location: 'build'
      azure_static_web_apps_api_token: '$(deployment_token)'

I hope this helps some people out :)

I would have to double check, but I think installing docker would fix my Windows issue. I will reply with my results

authors: kevinprescottwong-Dev miwebst


text:
chore: repository governance

Repository governance update
This PR was automatically created by the AVM Team hive-mind using the grept governance tool.
We have detected that some files need updating to meet the AVM governance standards.
Please review and merge with alacrity.
Grept config source: git::https://github.com/Azure/Azure-Verified-Modules-Grept.git//terraform
Thanks! The AVM team :heart:

Supersceeded by #81


text:
[AVM Module Issue]: "Missing required argument" that is not declared as 'required'.

Check for previous/existing GitHub issues

[X] I have checked for previous/existing GitHub issues

Issue Type?
I'm not sure
(Optional) Module Version
No response
(Optional) Correlation Id
No response
Description
This simple code uses only the parameters that are declared as "Required" in the documentation, but Terraform (plan operation) still shows an error that it's not enough. I expected the module to generate admin password for me and provide it as output. I have no intention in using key vaults, it's a simple test scenario.
module "avm-res-compute-virtualmachine" {
  source = "Azure/avm-res-compute-virtualmachine/azurerm"

  name                    = module.naming.windows_virtual_machine.name_unique
  resource_group_name     = azurerm_resource_group.this.name
  location                = azurerm_resource_group.this.location
  virtualmachine_sku_size = "Standard_A2_v2"
  zone                    = 1
}

The error (and follow-up errors):
Error: Missing required argument
  with module.avm-res-compute-virtualmachine.azurerm_key_vault_secret.admin_password[0],
  on .terraform\modules\avm-res-compute-virtualmachine\main.authentication.tf line 35, in resource "azurerm_key_vault_secret" "admin_password":
35:   key_vault_id    = var.admin_credential_key_vault_resource_id
The argument "key_vault_id" is required, but no definition was found.

Error: Attempt to get attribute from null value
  on .terraform\modules\avm-res-compute-virtualmachine\main.windows_vm.tf line 135, in resource "azurerm_windows_virtual_machine" "this":
135:       offer     = local.source_image_reference.offer
This value is null, so it does not have any attributes.

Error: Attempt to get attribute from null value
	on .terraform\modules\avm-res-compute-virtualmachine\main.windows_vm.tf line 136, in resource "azurerm_windows_virtual_machine" "this":
136:       publisher = local.source_image_reference.publisher
	local.source_image_reference is null
This value is null, so it does not have any attributes.

Error: Attempt to get attribute from null value
	on .terraform\modules\avm-res-compute-virtualmachine\main.windows_vm.tf line 137, in resource "azurerm_windows_virtual_machine" "this":
137:       sku       = local.source_image_reference.sku
	local.source_image_reference is null
This value is null, so it does not have any attributes.

Error: Attempt to get attribute from null value.
	on .terraform\modules\avm-res-compute-virtualmachine\main.windows_vm.tf line 138, in resource "azurerm_windows_virtual_machine" "this":
138:       version   = local.source_image_reference.version
	local.source_image_reference is null
This value is null, so it does not have any attributes.

I encountered this issue when providing UN + PW for the admin.
Referencing a KV seems tightly coupled to the parameter generate_admin_password_or_ssh_key which is defaulted to true. Since I'm providing UN+PW I disable the value, and the KV requirement is avoided.
module "avm-onprem-mgmt-vm" {
  source  = "Azure/avm-res-compute-virtualmachine/azurerm"
  name = module.on_prem_naming.virtual_machine.name
  location = azurerm_resource_group.onprem.location
  resource_group_name = azurerm_resource_group.onprem.name
  admin_username = var.username
  admin_password = var.password
  generate_admin_password_or_ssh_key = false
  virtualmachine_sku_size = var.vmsize
  zone = null
  virtualmachine_os_type = "Windows"
  source_image_reference = {
    publisher = "MicrosoftWindowsServer"
    offer     = "WindowsServer"
    sku       = "2022-datacenter-azure-edition-hotpatch"
    version   = "latest"
  }
  network_interfaces = {
    mgmt_nic = {
      name = module.on_prem_naming.network_interface.name
      location = azurerm_resource_group.onprem.location
      resource_group_name = azurerm_resource_group.onprem.name
      ip_configurations = {
        mgmt_ipconfig = {
          name = "mgmt-ipconfig"
          subnet_id = module.avm-onprem-mgmt-subnet.resource_id
          private_ip_address_allocation = "Dynamic"
          public_ip_address_id = null
          primary = true
        }
      }
    }
  }
}

https://github.com/Azure/terraform-azurerm-avm-res-compute-virtualmachine/blob/20917bef881cc8e346864c8b159d4546d27dccb2/main.authentication.tf#L32

@wplj and @eamreyes - Release 0.15.0 removes the requirement for the key vault id, and moves the id to a single interface for all of the generated secret (password or ssh key) configuration items.  (Also deprecates the old inputs for removal in a future release).  It also cleans up the inputs so that it can be deployed with only required inputs.  This is now also tested in the minimal example. Finally, please be aware there are breaking changes in the release so please review the release notes when you move to 0.15.

authors: eamreyes jchancellor-ms wplj


text:
chore: repository governance

Repository governance update
This PR was automatically created by the AVM Team hive-mind using the grept governance tool.
We have detected that some files need updating to meet the AVM governance standards.
Please review and merge with alacrity.
Grept config source: git::https://github.com/Azure/Azure-Verified-Modules-Grept.git//terraform
Thanks! The AVM team :heart:

Supersceeded by #23


text:
Broken Wiki links

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Versions

terraform: 1.7.0
azure provider: 3.65.0
module: 6.1.0
Description
Describe the bug
Wiki links for connectivity with custom settings are still broken on some sub pages. Further to the issue I previously raised (#1094) which was closed due to this PR the following pages contain links to the same which need fixing:
Examples.md
[Examples]-Deploy-Connectivity-Resources.md
[Examples]-Deploy-Multi-Region-Networking-With-Custom-Settings.md
[Examples]-Deploy-Virtual-WAN-Multi-Region-With-Custom-Settings.md
[Examples]-Deploy-Virtual-WAN-Resources.md
[Examples]-Deploy-using-multiple-module-declarations-with-orchestration.md
[Examples]-Deploy-using-multiple-module-declarations-with-remote-state.md
Broken links are for both the hub and spoke and VWAN custom settings pages.
Steps to Reproduce

Navigate to the pages above
click links for connectivity with custom settings
get redirected to 'Home'

Screenshots

Additional context

Would really like to be able to share a PR with the fixes for this but still unable to contribute - is it possible to be accepted as a contributor to raise PRs?

Hey @sissonsrob,
You can indeed submit a PR by forking this repo and then making your changes and submitting via a pull request. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/getting-started/about-collaborative-development-models#fork-and-pull-model

Thanks @jtracey93 - unsure why I couldn't raise the PR last time but I have now raised this PR which fixes the other pages affected by this.
Note - the fork was from my other account
Hope it helps