text:
SSL: CERTIFICATE_VERIFY_FAILED when build with pyinstaller + latest dependencies

Context

OS and version used: EmbeddedLinx kernel 5.10.65 armv7l GNU/Linux
Python version: python3.7
pip version: 19.0.3
list of installed packages::
altgraph==0.17.2
azure-iot-device==2.11.0
certifi==2021.10.8
charset-normalizer==2.0.12
deprecation==2.1.0
idna==3.3
importlib-metadata==4.11.3
janus==1.0.0
packaging==21.3
paho-mqtt==1.6.1
pyinstaller==5.0
pyinstaller-hooks-contrib==2022.4
pyparsing==3.0.8
PySocks==1.7.1
requests==2.27.1
requests-unixsocket==0.3.0
typing-extensions==4.2.0
urllib3==1.26.9
zipp==3.8.0

Description of the issue
We are using this SDK in our application, which is build with pyinstaller running om embedded linux.
After upgrading all our requirements we ran into SSL: CERTIFACATE_VERIFY_FAILED error when build it with pyinstaller (and his latest dependencies) and running it on our target.
After some digging I found that a newer version of a dependency of pyinstaller caused the issue, being pyinstaller-hooks-contrib.
pyinstaller-hooks-contrib==2021.3 is still ok, but starting from pyinstaller-hooks-contrib==2021.4 things go south.
If you check the changelog of this pip module:
https://github.com/pyinstaller/pyinstaller-hooks-contrib/blob/master/CHANGELOG.rst#updated-hooks-5
There they mention:
Removed the certifi run-time hook because it was not required for certifi to function in a frozen application. It was sometimes setting the SSL_CERT_FILE environment variable which causes applications to behave differently when frozen. In particular the SSLContext.set_default_verify_paths() method loads the certificates from certifi when the SSL_CERT_FILE environment variable is set. (https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/335)
Then found this related ticket:
https://github.com/pyinstaller/pyinstaller/issues/6667
The maintainer suggest setting SSL_CERT_FILE. --> https://github.com/pyinstaller/pyinstaller/issues/6667#issuecomment-1064316393
But this certificate comes from/with the SDK? So the SDK should point to it via this env variable?
However, we can continue by setting this pip module fixed to 2021.3, but this is just a temporary fix.
Going forward I think a solution must be found in order to keep using this SDK together with pyinstaller (and his latest dependencies) on our (and other) embedded IoT-devices.
Code sample exhibiting the issue
await iot_hub_device_client.connect()
Console log of the issue
2022-04-27T10:41:37.464Z azure.iot.device.common.pipeline.pipeline_thread DEBUG pipeline_thread(104): wrapper: Starting run_op in pipeline thread
2022-04-27T10:41:37.466Z azure.iot.device.common.pipeline.pipeline_thread DEBUG pipeline_thread(82): _get_named_executor: Creating pipeline executor
2022-04-27T10:41:37.471Z azure.iot.device.common.pipeline.pipeline_stages_http DEBUG pipeline_stages_http(50): _run_op: Gateway Hostname not present. Setting Hostname to: renson-iot-dev-iotcloudcore-ioth.azure-devices.net
2022-04-27T10:41:37.475Z azure.iot.device.common.pipeline.pipeline_stages_http DEBUG pipeline_stages_http(56): _run_op: HTTPTransportStage(InitializePipelineOperation): got connection args
2022-04-27T10:41:37.478Z azure.iot.device.common.http_transport DEBUG http_transport(67): _create_ssl_context: creating a SSL context
2022-04-27T10:41:37.486Z azure.iot.device.common.pipeline.pipeline_ops_base DEBUG pipeline_ops_base(111): complete: InitializePipelineOperation: completing without error
2022-04-27T10:41:37.488Z azure.iot.device.common.pipeline.pipeline_thread DEBUG pipeline_thread(104): wrapper: Starting <azure.iot.device.common.evented_callback.EventedCallback object at 0xb5bfee90> in callback thread
2022-04-27T10:41:37.492Z azure.iot.device.common.pipeline.pipeline_thread DEBUG pipeline_thread(82): _get_named_executor: Creating callback executor
2022-04-27T10:41:37.496Z azure.iot.device.common.evented_callback DEBUG evented_callback(50): wrapping_callback: Callback completed with result None
2022-04-27T10:41:37.502Z azure.iot.device.common.pipeline.pipeline_thread DEBUG pipeline_thread(104): wrapper: Starting run_op in pipeline thread
2022-04-27T10:41:37.506Z azure.iot.device.common.pipeline.pipeline_stages_base DEBUG pipeline_stages_base(460): _start_token_update_alarm: SasTokenStage: Scheduling automatic SAS Token renewal at epoch time: 1651052377
2022-04-27T10:41:37.514Z azure.iot.device.common.pipeline.pipeline_stages_mqtt DEBUG pipeline_stages_mqtt(126): _run_op: Gateway Hostname not present. Setting Hostname to: renson-iot-dev-iotcloudcore-ioth.azure-devices.net
2022-04-27T10:41:37.516Z azure.iot.device.common.pipeline.pipeline_stages_mqtt DEBUG pipeline_stages_mqtt(132): _run_op: MQTTTransportStage(InitializePipelineOperation): got connection args
2022-04-27T10:41:37.519Z azure.iot.device.common.mqtt_transport DEBUG mqtt_transport(136): _create_mqtt_client: creating mqtt client
2022-04-27T10:41:37.521Z azure.iot.device.common.mqtt_transport INFO mqtt_transport(149): _create_mqtt_client: Creating client for connecting using MQTT over TCP
2022-04-27T10:41:37.524Z azure.iot.device.common.mqtt_transport DEBUG mqtt_transport(320): _create_ssl_context: creating a SSL context
2022-04-27T10:41:37.528Z azure.iot.device.common.mqtt_transport DEBUG mqtt_transport(327): _create_ssl_context: configuring SSL context with default certs
2022-04-27T10:41:37.531Z azure.iot.device.common.mqtt_transport DEBUG mqtt_transport(277): _create_mqtt_client: Created MQTT protocol client, assigned callbacks
2022-04-27T10:41:37.533Z azure.iot.device.common.pipeline.pipeline_ops_base DEBUG pipeline_ops_base(111): complete: InitializePipelineOperation: completing without error
2022-04-27T10:41:37.536Z azure.iot.device.common.pipeline.pipeline_thread DEBUG pipeline_thread(104): wrapper: Starting <azure.iot.device.common.evented_callback.EventedCallback object at 0xb5bd4230> in callback thread
2022-04-27T10:41:37.539Z azure.iot.device.common.evented_callback DEBUG evented_callback(50): wrapping_callback: Callback completed with result None
2022-04-27T10:41:37.543Z azure.iot.device.iothub.aio.loop_management DEBUG loop_management(38): _make_new_loop: Creating new event loop - CLIENT_INTERNAL_LOOP
2022-04-27T10:41:37.546Z asyncio DEBUG selector_events(53): init: Using selector: EpollSelector
2022-04-27T10:41:37.568Z root INFO conn_test(59): main: after create_from_connection_string
2022-04-27T10:41:37.571Z root INFO conn_test(63): main: Run forever!
2022-04-27T10:41:37.574Z azure.iot.device.iothub.aio.async_clients INFO async_clients(226): connect: Connecting to Hub...
2022-04-27T10:41:37.578Z azure.iot.device.iothub.pipeline.mqtt_pipeline DEBUG mqtt_pipeline(254): connect: Starting ConnectOperation on the pipeline
2022-04-27T10:41:37.582Z azure.iot.device.common.pipeline.pipeline_thread DEBUG pipeline_thread(104): wrapper: Starting run_op in pipeline thread
2022-04-27T10:41:37.586Z azure.iot.device.common.pipeline.pipeline_stages_base DEBUG pipeline_stages_base(1131): _run_op: ReconnectStage(ConnectOperation): State changes DISCONNECTED -> CONNECTING. Sending op down
2022-04-27T10:41:37.590Z azure.iot.device.common.pipeline.pipeline_stages_base DEBUG pipeline_stages_base(695): _block: ConnectionLockStage(ConnectOperation): blocking
2022-04-27T10:41:37.593Z azure.iot.device.common.pipeline.pipeline_stages_mqtt DEBUG pipeline_stages_mqtt(177): _run_op: MQTTTransportStage(ConnectOperation): connecting
2022-04-27T10:41:37.601Z azure.iot.device.common.pipeline.pipeline_stages_mqtt DEBUG pipeline_stages_mqtt(67): _start_connection_watchdog: MQTTTransportStage(ConnectOperation): Starting watchdog
2022-04-27T10:41:37.606Z azure.iot.device.common.mqtt_transport DEBUG mqtt_transport(378): connect: connecting to mqtt broker
2022-04-27T10:41:37.609Z azure.iot.device.common.mqtt_transport INFO mqtt_transport(389): connect: Connect using port 8883 (TCP)
2022-04-27T10:41:37.659Z azure.iot.device.common.mqtt_transport INFO mqtt_transport(292): _force_transport_disconnect_and_cleanup: Forcing paho disconnect to prevent it from automatically reconnecting
2022-04-27T10:41:37.663Z azure.iot.device.common.mqtt_transport DEBUG mqtt_transport(314): _force_transport_disconnect_and_cleanup: Done forcing paho disconnect
2022-04-27T10:41:37.665Z azure.iot.device.common.pipeline.pipeline_stages_mqtt INFO pipeline_stages_mqtt(191): _run_op: transport.connect raised error
2022-04-27T10:41:37.673Z azure.iot.device.common.pipeline.pipeline_stages_mqtt INFO pipeline_stages_mqtt(192): _run_op: Traceback (most recent call last):
File "azure/iot/device/common/mqtt_transport.py", line 391, in connect
File "paho/mqtt/client.py", line 914, in connect
File "paho/mqtt/client.py", line 1073, in reconnect
File "ssl.py", line 1117, in do_handshake
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "azure/iot/device/common/pipeline/pipeline_stages_mqtt.py", line 189, in _run_op
File "azure/iot/device/common/mqtt_transport.py", line 403, in connect
azure.iot.device.common.transport_exceptions.TlsExchangeAuthError
2022-04-27T10:41:37.676Z azure.iot.device.common.pipeline.pipeline_stages_mqtt DEBUG pipeline_stages_mqtt(104): _cancel_connection_watchdog: MQTTTransportStage(ConnectOperation): cancelling watchdog
2022-04-27T10:41:37.678Z azure.iot.device.common.pipeline.pipeline_ops_base DEBUG pipeline_ops_base(109): complete: ConnectOperation: completing with error
2022-04-27T10:41:37.682Z azure.iot.device.common.pipeline.pipeline_stages_base DEBUG pipeline_stages_base(674): on_operation_complete: ConnectionLockStage(ConnectOperation): op failed.  Unblocking queue with error:
2022-04-27T10:41:37.684Z azure.iot.device.common.pipeline.pipeline_stages_base DEBUG pipeline_stages_base(704): _unblock: ConnectionLockStage(ConnectOperation): unblocking and releasing queued ops.
2022-04-27T10:41:37.687Z azure.iot.device.common.pipeline.pipeline_stages_base DEBUG pipeline_stages_base(708): _unblock: ConnectionLockStage(ConnectOperation): processing 0 items in queue for error=
2022-04-27T10:41:37.690Z azure.iot.device.common.pipeline.pipeline_stages_base DEBUG pipeline_stages_base(1345): on_complete: ReconnectStage(ConnectOperation): failed, state change CONNECTING -> DISCONNECTED
2022-04-27T10:41:37.693Z azure.iot.device.common.pipeline.pipeline_thread DEBUG pipeline_thread(104): wrapper: Starting on_complete in callback thread
2022-04-27T10:41:37.697Z azure.iot.device.common.async_adapter INFO async_adapter(72): wrapping_callback: Callback completed with error
2022-04-27T10:41:37.704Z azure.iot.device.common.async_adapter INFO async_adapter(73): wrapping_callback: ['azure.iot.device.common.transport_exceptions.TlsExchangeAuthError\n']
2022-04-27T10:41:37.708Z root INFO conn_test(38): init: ClientError: Error in the IoTHub client due to TLS exchanges.
Traceback (most recent call last):
File "azure/iot/device/common/mqtt_transport.py", line 391, in connect
File "paho/mqtt/client.py", line 914, in connect
File "paho/mqtt/client.py", line 1073, in reconnect
File "ssl.py", line 1117, in do_handshake
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "azure/iot/device/iothub/aio/async_clients.py", line 33, in handle_result
File "azure/iot/device/common/async_adapter.py", line 91, in completion
File "azure/iot/device/common/pipeline/pipeline_stages_mqtt.py", line 189, in _run_op
File "azure/iot/device/common/mqtt_transport.py", line 403, in connect
azure.iot.device.common.transport_exceptions.TlsExchangeAuthError
The above exception was the direct cause of the following exception:

I get a similar error when connecting to a (transparent) gateway running IoT Edge Runtime 1.2. Everything works if I connect to runtime 1.1 hub.
Here's my stacktrace:
INFO:azure.iot.device.common.mqtt_transport:Creating client for connecting using MQTT over TCP
INFO:azure.iot.device.iothub.aio.async_clients:Connecting to Hub...
INFO:azure.iot.device.common.mqtt_transport:Connect using port 8883 (TCP)
INFO:azure.iot.device.common.mqtt_transport:Forcing paho disconnect to prevent it from automatically reconnecting
INFO:azure.iot.device.common.pipeline.pipeline_stages_mqtt:transport.connect raised error
INFO:azure.iot.device.common.pipeline.pipeline_stages_mqtt:Traceback (most recent call last):
  File "/.../.venv/lib/python3.10/site-packages/azure/iot/device/common/mqtt_transport.py", line 390, in connect
    rc = self._mqtt_client.connect(
  File "/.../.venv/lib/python3.10/site-packages/paho/mqtt/client.py", line 914, in connect
    return self.reconnect()
  File "/.../.venv/lib/python3.10/site-packages/paho/mqtt/client.py", line 1073, in reconnect
    sock.do_handshake()
  File "/.../.pyenv/versions/3.10.2/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ssl.py", line 1341, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for '<hostname>'. (_ssl.c:997)

The certificates are fine and I can verify them using openssl.

@robbinvandamme
I've taken a look into the issue with PyInstaller - it appears the change they made is correct - what was happening before was that PyInstaller was (incorrectly) always using the certifi certificates, even when it wasn't supposed to. The correct behavior is to use the default system certificates rather than the ones in certifi.
My guess is that if your application no longer works now that PyInstaller has fixed this bug, that would mean that your system's certificate stores are not correctly configured. Can you check your system configuration?
We would expect that certificates would load upon calling SSLContext.load_default_certs() from the ssl module in Python. Please try the following code in a Python session to validate that your certificates are loadable:
import ssl

ctx = ssl.SSLContext()
ctx.load_default_certs()
print(ctx.get_ca_certs())

If this prints out an empty list, then your certs have not been loaded correctly.

@cartertinney
I tried your sample code to load the certificates on:

our embedded target
a stock beaglebone
an ubuntu20 laptop
an ubuntu22 laptop
All with the same result, an empty list.

I also create an executable for ARM and x86.
The x86 one works on my laptop with the latest pip packages.
The ARM one works on a stock beaglebone, but not on our embedded target.
On our embedded target I can only get it to work with the older version of the pyinstaller-hooks-contrib.
I you have any ideas, I'm glad to hear them, think I need to dive deeper in our embedded device and check why its not working with that latest pip module.

@cartertinney
I printed the default ca certs from inside the sdk mqtt_transport and even that returns an empty list.
But connection to IoTHub is working since its on my laptop.

So can you give me some more info about how the sdk uses or handles the certificates?
On my embedded device, I see that (the SDK?) downloads a certificate. Is it that one that is used by the SDK?

@cartertinney I found the problem.
I was missing a symlink /usr/lib/ssl/certs that points to /etc/ssl/certs where the certificates are installed.
After adding that, connection with IoTHub using latest pyinstaller was ok.
So when using the latest pyinstaller, the certifi certs are not used anymore by default and it looks at the system certificates.
If we want to use the certifi cacert.pem, it we should set the environment variable our-self like this.
    cert_file = None
    if getattr(sys, 'frozen', False) and hasattr(sys, "_MEIPASS"):
        print(sys._MEIPASS)
        cert_file = sys._MEIPASS + "/certifi/cacert.pem"
        os.environ["SSL_CERT_FILE"] = cert_file
        print("cert_file: %s" % cert_file)

I've haven't decided what the best option is for us.
Whats your opinion about this? Set SSL_CERT_FILE and use certifi cacert.pem or not set it and rely on system installed certificates?

@cartertinney
After some small tests I also think requests always uses the certifi certs by default. Because if I remove the /tmp/_MEIGGpP4m/certifi/cacert.pem while the application (build with pyinstaller) is running, it fails with error that it doesn't find it anymore.
If I run it without creating a pyinstaller package, it uses: ~/venv/lib/python3.8/site-packages/certifi/cacert.pem
Your SDK or our libssl was not able to find our system certs in /etc/ssl/certs by default, we had to create a symlink from /usr/lib/ssl/certs to it. After that everything seems to be fine.
In the past your SDK (or our application) indeed probably was working by accident on our system because SSL_CERT_FILE was set to the certifi cacert.pem by pyinstaller-hooks-contrib, and now that's not the case anymore.
If your SDK by default wants to use the system certs, I guess then it's better to just leave it this way instead of me forcing the SDK to use the certifi cacert.pem? In the end, all I had to fix was one symlink.
What's the reason the SDK is using the system certs by default and not the certifi certs? If the request module is doing it, there must be some kind of reason for it maybe? And the cacert.pem will always be there because requests is a dependency of the SDK en certifi is a dependency for requests.

Yes, that's what I was saying. The requests library, by default, will use certifi certs. However, our azure-iot-device library does not. This behavior of using the system store certs is universal across languages of the Azure IoT Device SDK. requests is currently a dependency of our SDK, but is only used for a very small number of tasks, and ideally may eventually be removed. Additionally where it is used in our SDK, it is configured to use the system store rather than certifi

@cartertinney this part is not clear to me:
"Additionally where it is used in our SDK, it is configured to use the system store rather than certifi"
I created a venv and installed your sdk.
That comes with "./venv_cert_test/lib/python3.7/site-packages/certifi/cacert.pem".
If I remove that cacert.pem and try my sample app or in fact just do "from azure.iot.device.aio import IoTHubDeviceClient", it fails...
So is still requires ./certifi/cacert.pem to be there.
Maybe it's not used after (re-)configuring the SDK or the way requests should use certificates, but it seems that it needs to be there on starting the application or importing all the modules?
I attached the traceback in a file. traceback.txt
Probably expected behavior?
Apart from that no further questions anymore :-D.
Big thanks for your time and explanation!


text:
Question: Best Practice when cancellation during Pull Operations from Server

According to this issue (https://github.com/Azure/azure-mobile-apps-net-client/issues/116) Azure App Service will not support transactions.
What is the best way to handle the following situation under these circumstances?
We have a Xamarin.Forms App which uses Azure App Service with Offline Sync. We have a very complex database design which many  interdependent tables. So many Pull Operations a needed to retrieve all tables.
Due to various circumstances it can happen that the sync aborts and several tables are not completely downloaded. In this case i have incomplete inconsistent data in the local store. Transactions would be helpful here, so a rollback would be possible. Unfortunately Azure App Service does not offer this.
What is the recommended procedure for this situation? One way would be to forbid working with the app until a successful sync has been done, but I don't think that's a good idea.

Just my thoughts, transactional support for the syncing is less important (it will catch up during the next sync), and i guess you can prevent access to the app until it has fully synced. But I think your app will have to be able to handle partly-synced hierarchical data (normally should sort itself out after later syncs).
Bigger issue is that sqlite on the local phone does not support transactions. So if you're halfway through a larger data update on local the phone and it crashes or gets disrupted somehow then you'll sit with fractured data on the device, and during the next sync it will push the broken data up to the server, from which it will propagate to all other devices too :)
Afaik there is no workaround for the lack of transactional support on the mobile device (this is a sqlite restriction).

I think it depends on the use case. Transactions on the local store are less important for us, because we insert only a few data recors on the client.
In our use case the app has to be fully functional after every sync. It is an app for field managers for retail stores. So when a field manager stands in front of an retail store an the sync fail, we cannot access to the app because some data is missing. The app is a mission critical tool for our field managers. A algorithm which checks every data record if it is complete and prevent access only to data recorcs which are incomplite would be very complex (in the case of our database design).
Our solution so few: We create a new local store after every sync.

Create a new local store
Download all records in the new local store
If sync completes without errors replace the old local store with the new one

This works but it doesn't feel right to delete and recreate the local store every time. Also this way an incremental sync is impossible.

Hi @OPunktSchmidt,
Without actual transaction support, I doubt that I can give you generic solution that will work for all use cases. I guess the most of workarounds here is about data-structure design.
However, I see the way how to improve your workaround:

~Create a new local store~ Copy existing database file
~Download all records in the new local store~ Do incremental sync
If sync completes without errors replace the old local store with the new one

Since the local db is just a SQLite file, theoretically it should work.

Hi @MatkovIvan
many thanks for this simple but brilliant idea. This works in our use case.
This has given me a headache. At some point I thought too complicated instead of considering such a simple solution. Thanks!

source: gharchive/issue

authors: MatkovIvan OPunktSchmidt ltaljaard

repo: Azure/azure-mobile-apps-net-client

license_type: permissive

license_source: bigquery


text:
Resource group name doesn't satisfy the constraint pattern /^[-\w._()]+$/

We have created an Azure resource group from the portal, named Instruktører (notice the special character ø). When we try and perform a GET call using azure-sdk-for-js (https://github.com/Azure/azure-sdk-for-js/blob/master/sdk/resources/arm-resources/src/operations/resourceGroups.ts#L115) on this resource group, it throws an error:
(node:57737) UnhandledPromiseRejectionWarning: Error: "resourceGroupName" with value "Instruktører" should satisfy the constraint "Pattern": /^[-\w\._\(\)]+$/.
    at failValidation (/Azure/node_modules/@azure/arm-resources/node_modules/@azure/ms-rest-js/dist/msRest.node.js:461:19)

Expected behaviour:
Since the portal accepts the resource group name Instruktører, we'd expect the SDK to accept it as well and not throw the above error.
Please check if the swagger definitions needs to be updated: https://github.com/Azure/azure-rest-api-specs/blob/0cc5a06b6712e644fc5d4fec5005a3c945d18a7a/specification/resources/resource-manager/Microsoft.Resources/stable/2020-06-01/resources.json#L2545

Related issue: https://github.com/Azure/azure-sdk-for-js/issues/13247

@snaheth can you help check this or point to correct owner to help fix this? thanks!

Fized here https://github.com/Azure/azure-rest-api-specs/pull/13492

authors: SumitPopat tjegbejimba zhenglaizhang

repo: Azure/azure-rest-api-specs

license: MIT

license_source: github-api


text:
Microsoft.Insights workbooks & workbooktemplates declared with incorrect ARM URI

I noticed that the newly-added Microsoft.Insights workbooks & workbooktemplates resource are using "/resourceGroup/" instead of "/resourceGroups/" in the "path" declarations. This is tripping up the https://github.com/Azure/autorest.azureresourceschema extension, and I presume generating SDKs which won't actually work with ARM.
@ericc1103 / @avidhanju since you recently authored changes, would you be able to take a look?

@ericc1103 / @avidhanju
Any update?

I can take a look at this in the second week of January when I get back from vacation.
Why was this not part of the various checks that ran when I last merged Workbook Templates?
#https://github.com/Azure/azure-rest-api-specs/pull/7611/checks


text:
Adding Java configuration for devspaces

This checklist is used to make sure that common issues in a pull request are addressed. This will expedite the process of getting your pull request merged and avoid extra work on your part to fix issues discovered during the review process.
PR information

[ ] The title of the PR is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
[ ] Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
[ ] If applicable, the PR references the bug/issue that it fixes.
[ ] Swagger files are correctly named (e.g. the api-version in the path should match the api-version in the spec).

Quality of Swagger

[ ] I have read the contribution guidelines.
[ ] My spec meets the review criteria:

[ ] The spec conforms to the Swagger 2.0 specification.
[ ] The spec follows the guidelines described in the Swagger checklist document.
[ ] Validation tools were run on swagger spec(s) and have all been fixed in this PR.

Automation for azure-sdk-for-python

Automation for azure-sdk-for-ruby

Automation for azure-sdk-for-go

Automation for azure-sdk-for-node

Automation for azure-sdk-for-java

@AutorestCI regenerate azure-sdk-for-java

@AutorestCI regenerate azure-sdk-for-java

source: gharchive/pull-request

authors: AutorestCI jianghaolu lenala


text:
chore: jsonfmt azure-kusto

Ran jsonfmt -w "specification/azure-kusto/**/*.json"

Can one of the admins verify this patch?

Automation for azure-sdk-for-ruby

Automation for azure-sdk-for-js

Automation for azure-sdk-for-python

Automation for azure-sdk-for-go

Automation for azure-sdk-for-java

authors: AutorestCI azuresdkci nschonni


text:
luis authoring api updates for nested children and required features

Latest improvements:
MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Contribution checklist:

[ ] I have reviewed the documentation for the workflow.
[ ] Validation tools were run on swagger spec(s) and have all been fixed in this PR.
[ ] The OpenAPI Hub was used for checking validation status and next steps.

ARM API Review Checklist

[ ] Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
adding/removing APIs.
adding/removing properties.
adding/removing API-version.
adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

[ ] If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

You don't have permission to trigger SDK Automation.
Please add yourself to Azure group from opensource portal if you are MSFT employee,
or please ask reviewer to add comment *** /openapibot sdkautomation ***.
Please ask tih@microsoft.com (or NullMDR in github) for additional help.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: a-noyass sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Can one of the admins verify this patch?

authors: AutorestCI a-noyass azuresdkci msftclas


text:
Clean up / Ignore unnecessary CI artifacts, Issue #1422

This is my first PR with an "official" repo, plz go easy. Thanks!

Thanks for submitting a PR @Azure/azure-sdk-team-signalr
Just wondering if we could get a bit more detail on the specific issue you are looking to solve with this PR. Are there files being created in tools which are making your commit experience difficult? Also with regard to deleting the grepResults file, I don't think it will have any impact since the file removal is performed after any artifacts are uploaded, and since the VM running the build is destroyed after each run its likely an unnecessary change.

Just wondering if we could get a bit more detail on the specific issue you are looking to solve with this PR. Are there files being created in tools which are making your commit experience difficult? Also with regard to deleting the grepResults file, I don't think it will have any impact since the file removal is performed after any artifacts are uploaded, and since the VM running the build is destroyed after each run its likely an unnecessary change.

Hi Mitch,
This PR was in regards to the issue found here: https://github.com/Azure/azure-sdk-for-c/issues/1422
It was submitted in an attempt to satisfy the requirements located within that issue, but I didn't realize that the grepResults file will be deleted anyway. I was following the instructions but if the issue needs to be closed without any changes, I understand that too. Thank you, and thank you for the tip.
Taren

Fixes https://github.com/Azure/azure-sdk-for-c/issues/1422

Are there files being created in tools which are making your commit experience difficult?

If we generate code coverage locally, the tools folder gets created which needs to be ignored. Like other .gitignore files (such as DoxygenWarningLog.txt and **/[Dd]oc/generated_doxygen_docs), we don't want them accidentally pushed as part of a commit.

Also with regard to deleting the grepResults file, I don't think it will have any impact since the file removal is performed after any artifacts are uploaded, and since the VM running the build is destroyed after each run its likely an unnecessary change.

I understand it isn't strictly necessary, but we are creating a file on disk to validate encoding, so it is good to clean up that file after we are done with it. I see no harm in doing so.

LGTM then.

This is my first PR with an "official" repo

Congrats! Keep it up ;)

Congrats! Keep it up ;)
Thank you Ahsonkhan! I appreciate it. I am definitely looking for other ways to contribute and help in any way I can.

Nice. If you are interested in C and C++, here are some help wanted issues:
https://github.com/Azure/azure-sdk-for-cpp/issues?q=is%3Aissue+is%3Aopen+label%3A"help+wanted"

For example: https://github.com/Azure/azure-sdk-for-cpp/issues/898
https://github.com/Azure/azure-sdk-for-c/issues?q=is%3Aissue+is%3Aopen+label%3A"help+wanted"
For example: https://github.com/Azure/azure-sdk-for-c/issues/1263

Let me know if you are interested in other programming languages instead. You can look at good first issue labels.

Awesome, I will start working on issue https://github.com/Azure/azure-sdk-for-cpp/issues/898 . I am interested in primarily C,C#, C++, and outside C languages possibly Python.
Thanks for the resources!

@TarenLewis - shall I put you down for https://github.com/Azure/azure-sdk-for-cpp/issues/898 and remove the help wanted label? It isn't super urgent, but we'd like to get that fixed before end-of-year.

@ahsonkhan I was planning on writing a python script to handle it, but I won't likely be able to start working on it for a couple weeks. I'd like to leave it open just in case someone else gets to it before me, but plan on working on it once I start my break. (I'm in finals week at university right now)
Thanks!

Sounds great. Good luck.

Gotcha, thanks for the suggestion. Probably a much better solution.

authors: TarenLewis ahsonkhan mitchdenny

repo: Azure/azure-sdk-for-c


text:
Identity: apply Jeff's concurrent token refresh logic

We currently have one, but Jeff's invention is better.
Also, we wanted to extract identity into a module, this might be a good task to combine with.

@antkmsft Can you elaborate on this item.  Is this work necessary now that we have token caching?

@RickWinter, this have been so long ago. We've made multiple improvements since then. We are good in terms of concurrency, and identity has been extracted into a separate module long ago. We should close this item, it is completed.

authors: RickWinter antkmsft

repo: Azure/azure-sdk-for-cpp


text:
Use std::vector instead of std::list for KeyOperation arrays.

Addressing the API review feedback:
https://apiview.dev/Assemblies/Review/da8d0ecdf87c49238b75fee069413029#3508f13ee9d3483ea2e4223592ed08df

Unless you don't have the full surface area for Keys (except Crypto) that .NET does, this seems to be missing a few changes.

Looks like I caught them all, or they were already std::vector.

/azp run cpp - keyvault


text:
Remove CopyFromUri

I'll remove this API from current release as it hasn't been approved by Arch Board. I'll add it back in next release.
Pull Request Checklist
Please leverage this checklist as a reminder to address commonly occurring feedback when submitting a pull request to make sure your PR can be reviewed quickly:
See the detailed list in the contributing guide.

[x] C++ Guidelines
[x] Doxygen docs
[x] Unit tests
[x] No unwanted commits/changes
[x] Descriptive title/description

[x] PR is single purpose
[x] Related issue listed

[x] Comments in source
[x] No typos
[x] Update changelog
[x] Not work-in-progress
[x] External references or docs updated
[x] Self review of PR done
[x] Any breaking changes?

/azp run cpp - storage


text:
docV1

Doc about the perf test
Pull Request Checklist
Please leverage this checklist as a reminder to address commonly occurring feedback when submitting a pull request to make sure your PR can be reviewed quickly:
See the detailed list in the contributing guide.

[x] C++ Guidelines
[x] Doxygen docs
[x] Unit tests
[x] No unwanted commits/changes
[x] Descriptive title/description

[x] PR is single purpose
[x] Related issue listed

[x] Comments in source
[x] No typos
[x] Update changelog
[x] Not work-in-progress
[x] External references or docs updated
[x] Self review of PR done
[x] Any breaking changes?

@LarryOsterman  for your viewing pleasure , V2

@LarryOsterman for your rereviewing pleasure

/azp run checkenforcer

/check-enforcer evaluate

check enfoercer check passed somehow it is not reflected here

/check-enforcer override


text:
graphrbac: support the replyUrlsWithType property for Applications

Feature Request
Looking at the Azure Portal when configuring reply/redirect URLs for applications, it makes use of the replyUrlsWithType property of the application resource, available with v2.0 of the API. This property is undocumented.
How can we get support for this property, as it's seemingly the only way to create different types of reply/redirect URLs and will fill a large gap in support for different types of applications?
Implementing this would be a huge enabler, as it is currently the canonical way to set the new redirect URIs for applications, given that MS Graph support is currently lagging behind here.
Relevant portal blade

Add a platform

Add an SPA redirect

Resulting API query

Payload
PATCH /myorganization/applications/0ee8ac26-20f5-4da4-abd8-4b504785782b?api-version=2.0

{"id":"0ee8ac26-20f5-4da4-abd8-4b504785782b","replyUrlsWithType":[{"type":"Web","url":"http://unittest.hashicorptest.com"},{"type":"Spa","url":"https://blah-spa"},{"type":"InstalledClient","url":"https://blahnative"}],"logoutUrl":null,"oauth2AllowImplicitFlow":false,"oauth2AllowIdTokenImplicitFlow":false}

Highly interested in this since its preventing anyone from using terraform to automate the creation & management of single paged AD applications that use the PKCE oauth2 flow.

@armleads-azure any updates?

Bump

@armleads-azure @ArcturusZhang Any updates on this?

@armleads-azure @ArcturusZhang Any updates on this?

#bump
Our Azure Fabric Explorer is secured using AD. As described in the following page : https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-creation-setup-aad
We got the whole setup in terraform, but miss out on one thing: replyUrlsWithType as we need it to define
$nativeApp = @{     publicClient = "true"     displayName = $NativeClientApplicationName     replyUrls = @("urn:ietf:wg:oauth:2.0:oob")     requiredResourceAccess = $nativeAppResourceAccess }
Currently Terraform azuread_application doesn't allow non 'https' url's in the replyUrls property.
Not even sure the type "Mobile and desktop applications" is supported azuread_application

authors: CaptainStealthy FreFre autom8ter lilyjma manicminer

repo: Azure/azure-sdk-for-go


text:
azidentity: import cycle not allowed`

I follow the manual and get error:
package command-line-arguments imports github.com/Azure/azure-sdk-for-go/sdk/azidentity imports github.com/Azure/azure-sdk-for-go/sdk/azidentity: import cycle not allowed
What may help me to fix this?
Code:
package main import ( "context" "fmt" azidentity "github.com/Azure/azure-sdk-for-go/sdk/azidentity" a "github.com/microsoft/kiota-authentication-azure-go" )

Hi @jeruly, I can't reproduce this - this is my code:
package main

import (
	"fmt"

        // btw, this re-aliasing shouldn't be needed - the package name for this is already `azidentity`
	azidentity "github.com/Azure/azure-sdk-for-go/sdk/azidentity"
	a "github.com/microsoft/kiota-authentication-azure-go"
)

func main() {
	cred, err := azidentity.NewDeviceCodeCredential(nil)

	if err != nil {
		panic(err)
	}

	authProvider, err := a.NewAzureIdentityAuthenticationProviderWithScopes(cred, []string{"User.Read"})

	if err != nil {
		panic(err)
	}

	fmt.Printf("%#v\n", authProvider)
}

and my go.mod:
module kiotatest

go 1.19

require (
	github.com/Azure/azure-sdk-for-go/sdk/azcore v1.1.3 // indirect
	github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.1.0 // indirect
	github.com/Azure/azure-sdk-for-go/sdk/internal v1.0.0 // indirect
	github.com/AzureAD/microsoft-authentication-library-for-go v0.5.1 // indirect
	github.com/cjlapao/common-go v0.0.25 // indirect
	github.com/golang-jwt/jwt v3.2.1+incompatible // indirect
	github.com/google/uuid v1.3.0 // indirect
	github.com/kylelemons/godebug v1.1.0 // indirect
	github.com/microsoft/kiota-abstractions-go v0.9.1 // indirect
	github.com/microsoft/kiota-authentication-azure-go v0.4.1 // indirect
	github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4 // indirect
	github.com/yosida95/uritemplate/v3 v3.0.2 // indirect
	golang.org/x/crypto v0.0.0-20220511200225-c6db032c6c88 // indirect
	golang.org/x/net v0.0.0-20220826154423-83b083e8dc8b // indirect
	golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10 // indirect
	golang.org/x/text v0.3.7 // indirect
)

From the error message it looks like the command-line-arguments package might be involved - is that a public package?

Thanks a lot . The issue was in mod file . Module name was set as "github.com/Azure/azure-sdk-for-go/sdk/azidentity". So this caused and issue

Nothing wrong with an easy solution, glad you were able to figure it out. :)

authors: jeruly richardpark-msft


text:
[BUG] Cannot install extension on VM

Describe the bug
I'm trying to install a CustomScript extension using the sdk but It gives an error saying that the virtual machine does not support updates to the purchase plan.
If I use the azure cli or the website works fine.
The image that I'm using on the vm is bitnami wordpress.
Exception or Stack Trace

2020-08-26 11:55:18.437 ERROR 1116 --- [nio-8081-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.microsoft.azure.CloudException: The resource '/subscriptions/ae01d003-8509-415b-9c84-bf526ddd7d56/resourceGroups/storesResourceGroup/providers/Microsoft.Compute/virtualMachines/Loja07' of type 'Microsoft.Compute/virtualMachines' does not support updates to the purchase plan.: The resource '/subscriptions/ae01d003-8509-415b-9c84-bf526ddd7d56/resourceGroups/storesResourceGroup/providers/Microsoft.Compute/virtualMachines/Loja07' of type 'Microsoft.Compute/virtualMachines' does not support updates to the purchase plan.] with root cause
rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: retrofit2.Response.class
at rx.exceptions.OnErrorThrowable.addValueAsLastCause(OnErrorThrowable.java:118) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:73) ~[rxjava-1.3.8.jar:1.3.8]
at retrofit2.adapter.rxjava.CallArbiter.deliverResponse(CallArbiter.java:120) ~[adapter-rxjava-2.6.4.jar:na]
at retrofit2.adapter.rxjava.CallArbiter.emitResponse(CallArbiter.java:102) ~[adapter-rxjava-2.6.4.jar:na]
at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:46) ~[adapter-rxjava-2.6.4.jar:na]
at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:24) ~[adapter-rxjava-2.6.4.jar:na]
at rx.Observable.unsafeSubscribe(Observable.java:10327) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[rxjava-1.3.8.jar:1.3.8]
at rx.Observable.unsafeSubscribe(Observable.java:10327) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeSingle.call(OnSubscribeSingle.java:81) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeSingle.call(OnSubscribeSingle.java:27) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.SingleToObservable.call(SingleToObservable.java:39) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.SingleToObservable.call(SingleToObservable.java:27) ~[rxjava-1.3.8.jar:1.3.8]
at rx.Observable.unsafeSubscribe(Observable.java:10327) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.3.8.jar:1.3.8]
at rx.Observable.unsafeSubscribe(Observable.java:10327) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.DeferredScalarSubscriber.subscribeTo(DeferredScalarSubscriber.java:153) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:32) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:22) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[rxjava-1.3.8.jar:1.3.8]
at rx.Observable.unsafeSubscribe(Observable.java:10327) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[rxjava-1.3.8.jar:1.3.8]
at rx.Observable.unsafeSubscribe(Observable.java:10327) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[rxjava-1.3.8.jar:1.3.8]
at rx.Observable.unsafeSubscribe(Observable.java:10327) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[rxjava-1.3.8.jar:1.3.8]
at rx.Observable.unsafeSubscribe(Observable.java:10327) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.operators.OperatorSubscribeOn$SubscribeOnSubscriber.call(OperatorSubscribeOn.java:100) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.schedulers.CachedThreadScheduler$EventLoopWorker$1.call(CachedThreadScheduler.java:230) ~[rxjava-1.3.8.jar:1.3.8]
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) ~[rxjava-1.3.8.jar:1.3.8]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

To Reproduce
1 - Create a VM and using the Wordpress image of bitnami.
2 - Try to install a extension using the following code.
3 - Error occurs.
Code Snippet
        final String linuxCustomScriptExtensionName = "CustomScript";
        final String linuxCustomScriptExtensionPublisherName = "Microsoft.Azure.Extensions";
        final String linuxCustomScriptExtensionTypeName = "CustomScript";
        final String linuxCustomScriptExtensionVersionName = "2.0";
        final List<String> linuxScriptFileUris = new ArrayList<>();
        linuxScriptFileUris.add("...");

            VirtualMachine vm = azure.virtualMachines().getByResourceGroup("storesResourceGroup",
                    storeName);

            vm.update()
                    .defineNewExtension(linuxCustomScriptExtensionName)
                    .withPublisher(linuxCustomScriptExtensionPublisherName)
                    .withType(linuxCustomScriptExtensionTypeName)
                    .withVersion(linuxCustomScriptExtensionVersionName)
                    .withMinorVersionAutoUpgrade()
                    .withPublicSetting("fileUris", linuxScriptFileUris)
                    .withPublicSetting("commandToExecute", "bash install.sh " + info.getUrl() + " " + info.getTitle() + " " + info.getLanguage())
                    .attach()
                    .apply();

Expected behavior
Should install the extension with success.
Setup (please complete the following information):

OS: Windows 10
IDE : Intellij
Version of the Library used: 1.36.1

Additional context
Add any other context about the problem here.
Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[x] Bug Description Added
[x] Repro Steps Added
[x] Setup information Added

https://github.com/Azure/azure-libraries-for-java/issues/632

hi @xccc-msft could you pls help to have a look? thanks

@esfomeado You may try with below code:
VirtualMachineExtensionInner inner = new VirtualMachineExtensionInner()
                .withPublisher(linuxCustomScriptExtensionPublisherName)
                .withVirtualMachineExtensionType(linuxCustomScriptExtensionTypeName)
                .withTypeHandlerVersion(linuxCustomScriptExtensionVersionName)
                .withAutoUpgradeMinorVersion(true)
                .withSettings(settingMap);
computeManager.inner().virtualMachineExtensions().createOrUpdate("rgName", "vmName", linuxCustomScriptExtensionName, inner);

@xccc-msft Sorry I don't have deep knowledge about the Azure SDK and I haven't found many examples.
I did the following:
ApplicationTokenCredentials credentials = new ApplicationTokenCredentials(
		"xxx", "xxx", "xxx", AzureEnvironment.AZURE
);

ComputeManager computeManager = ComputeManager.authenticate(new RestClient.Builder()
		.withBaseUrl(credentials.environment(), AzureEnvironment.Endpoint.RESOURCE_MANAGER)
		.withResponseBuilderFactory(new AzureResponseBuilder.Factory())
		.withSerializerAdapter(new AzureJacksonAdapter())
		.withCredentials(credentials)
		.build(), "xxx");

Map<String, String> settingsMap = new HashMap<>();
settingsMap.put("commandToExecute", "ls");

VirtualMachineExtensionInner inner = new VirtualMachineExtensionInner()
		.withPublisher("Microsoft.Azure.Extensions")
		.withVirtualMachineExtensionType("CustomScript")
		.withTypeHandlerVersion("2.1")
		.withAutoUpgradeMinorVersion(true)
		.withSettings(settingsMap);

computeManager.inner().virtualMachineExtensions().createOrUpdate("storesResourceGroup", "xxx",
		"CustomScript", inner);

I get this error:

java.lang.IllegalArgumentException: location is required and cannot be null.

If I try to add the .withLocation() to the VirtualMachineExtensionInner it returns a Resource instead of a VirtualMachineExtensionInner. Not sure of how to proceed or if you have any other example.
Any input will be welcomed.
Thanks

@esfomeado You can add below line before calling createOrUpdate()
inner.withLocation(Region.US_WEST.name());

Works fine now.
Thanks

@esfomeado As per the warning info, you may consider to report here. Thanks.

authors: esfomeado xccc-msft yungezz

repo: Azure/azure-sdk-for-java


text:
Could not connect Cosmos DB after upgrading SDK 4.19.1

Query/Question
Error connecting to Cosmos DB
Why is this not a Bug or a feature Request?
Trying to find the documentation to upgrade the SDK
Setup (please complete the following information if applicable):
,"type":"WARN","logLine":97,"className":"io.netty.channel.ChannelInitializer","methodName":"exceptionCaught","log":"Failed to initialize a channel. Closing: [id: 0x3e1044df]","stackTrace":"java.util.NoSuchElementException: reactor.left.sslHandler\n\tat io.netty.channel.DefaultChannelPipeline.getContextOrDie(DefaultChannelPipeline.java:1073)\n\tat io.netty.channel.DefaultChannelPipeline.addAfter(DefaultChannelPipeline.java:302)\n\tat io.netty.channel.DefaultChannelPipeline.addAfter(DefaultChannelPipeline.java:290)\n\tat reactor.netty.tcp.SslProvider.addSslReadHandler(SslProvider.java:600)\n\tat reactor.netty.tcp.SslProvider.addSslHandler(SslProvider.java:562)\n\tat reactor.netty.http.client.HttpClientConfig$HttpClientChannelInitializer.onChannelInit(HttpClientConfig.java:824)\n\tat reactor.netty.transport.TransportConfig$TransportChannelInitializer.initChannel(TransportConfig.java:394)\n\tat io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)\n\tat io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)\n\tat io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:938)\n\tat io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)\n\tat io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:181)\n\tat io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:358)\n\tat io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:339)\n\tat reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnectionAllocator$PooledConnectionInitializer.initChannel(DefaultPooledConnectionProvider.java:542)\n\tat io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)\n\tat io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)\n\tat io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:938)\n\tat io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)\n\tat io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)\n\tat io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)\n\tat io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)\n\tat io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)\n\tat io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:502)\n\tat io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:417)\n\tat io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:474)\n\tat io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)\n\tat io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:293)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.lang.Thread.run(Thread.java:748)\n"}
"className":"com.azure.cosmos.implementation.RxGatewayStoreModel","methodName":"lambda$toDocumentServiceResponse$3","log":"Network failure","stackTrace":"java.net.UnknownHostException: \n\tat java.net.InetAddress.getAllByName0(InetAddress.java:1281)\n\tat java.net.InetAddress.getAllByName(InetAddress.java:1193)\n\tat java.net.InetAddress.getAllByName(InetAddress.java:1127)\n\tat io.netty.util.internal.SocketUtils$9.run(SocketUtils.java:169)\n\tat io.netty.util.internal.SocketUtils$9.run(SocketUtils.java:166)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat io.netty.util.internal.SocketUtils.allAddressesByName(SocketUtils.java:166)\n\tat io.netty.resolver.DefaultNameResolver.doResolveAll(DefaultNameResolver.java:50)\n\tat io.netty.resolver.SimpleNameResolver.resolveAll(SimpleNameResolver.java:79)\n\tat io.netty.resolver.SimpleNameResolver.resolveAll(SimpleNameResolver.java:71)\n\tat io.netty.resolver.InetSocketAddressResolver.doResolveAll(InetSocketAddressResolver.java:73)\n\tat io.netty.resolver.InetSocketAddressResolver.doResolveAll(InetSocketAddressResolver.java:31)\n\tat io.netty.resolver.AbstractAddressResolver.resolveAll(AbstractAddressResolver.java:158)\n\tat reactor.netty.transport.TransportConnector.doResolveAndConnect(TransportConnector.java:283)\n\tat reactor.netty.transport.TransportConnector.lambda$connect$6(TransportConnector.java:110)\n\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125)\n\tat reactor.netty.transport.TransportConnector$MonoChannelPromise._subscribe(TransportConnector.java:579)\n\tat reactor.netty.transport.TransportConnector$MonoChannelPromise.lambda$subscribe$0(TransportConnector.java:499)\n\tat io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)\n\tat io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:293)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.lang.Thread.run(Thread.java:748)\n"}
Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[x] Query Added
[x] Setup information Added

This has been fixed, @PKWM pls test on the latest version.

authors: PKWM kushagraThapar


text:
Azure Cosmos SDK - Retriable Writes

Currently the azure-cosmos SDK only automatically issues retries when a retry is guaranteed to maintain idempotency. A “createItem” request would for example only automatically be retried if the request was never sent on the wire – because otherwise the result could be a 409 (because the initial request actually made it to the service and was processed) instead of the idempotent 201. Today this leaks the responsibility for retries around writes etc. into the application.
To simplify the experience for application developers the new SDK APIs should allow the application developer to include automatic retries – even when that means non-idempotent results.
Application developers should not have to worry about implementing retries for every write operation – but they might have to consider the consequences of possibly different responses – like being able to tolerate a 409 due to retries of a previously successful request. We will discuss this further in the design phase – there are ways to also shield application developers form having to deal with these possibly new/unexpected response status code – but that would require the new APIs to diverge more form existing APIs and it is a trade-off between making it simple for application developers to adopt to the new SDK APIs or hiding possibly unexpected status codes.

This is done.


text:
[BUG]AddressRefresh diagnostics also shows up in GatewayDiagnostics

Issue:
Simulate address refresh request timeout, the address refresh request details is included in addressResolutionStatistics, but also shows up in gatewayDiagnosticsList which can be confusing.  (Can use faultInjectionServerErrorRuleTests_AddressRefresh_ResponseDelay test to get a sample cosmosDiagnostics)

@xinlian12 - I think this has been addressed a while ago?


text:
Fix record test

Cannot fix:
DnsZoneRecordSetETagTests.CreateRecordSetsWithDefaultETag: deprecated, need to use private dns
graphrbac.UpdateUser: 403, not very sure the reason, currently there is only MS graph API docs https://docs.microsoft.com/en-us/graph/api/user-update?view=graph-rest-1.0&tabs=http
GettingSqlServerMetrics: due to azure-core encode bug https://github.com/Azure/azure-sdk-for-java/issues/11451
still fix:
appservice: hostname binding error and other error
Special Fix:
network watcher: need storage with V2
storage: cannot use import operation with others
sql: cannot create US_WEST sql in microsoft tenant
sql: import need to specify operation import
cosmos: update will not start immediately and some location will be automatically deleted (don't know why)
cosmos: data-plane key could not end with slash

/check-enforcer evaluate


text:
Increment version for template releases

Increment version for template releases

Can one of the admins verify this patch?

authors: azure-sdk azuresdkci


text:
Fixed single source was empty error in direct mode

Fixed single source was empty error in direct mode
Added test case to cover both direct and gateway mode.

Can you please add some more description , it is hard to relate test case with the code change

Can you please add some more description , it is hard to relate test case with the code change

@simplynaveen20  - added more description to the PR, thanks for asking.

/azp run java - cosmos - tests

Closing this PR in lieu of : https://github.com/Azure/azure-sdk-for-java/pull/13805


text:
Fixed concurrency issue in buffered upload that caused large files to…

… hang
Resolves https://github.com/Azure/azure-sdk-for-java/issues/20946

/azp run java - storage - tests

/azp run java - storage - tests

/azp run java - storage - tests

/azp run java - storage - tests


text:
Standardize indention and publishing

cc @Azure/azure-sdk-eng

Can one of the admins verify this patch?

authors: azuresdkci weshaggard


text:
Fixing dependencies to not include betas

Description
Previous patch releases included betas in the dependency updates section which is incorrect. Fixed so betas are not mentioned in the dependency updates section.

API change check
API changes are not detected in this pull request.

authors: azure-sdk ibrahimrabab


text:
Increment versions for hybridkubernetes releases

Increment package versions for hybridkubernetes releases

API change check
API changes are not detected in this pull request.


text:
Regions Contacted on Multi Region Diagnostics Fix

Description
The regionsContacted field in the diagnostics was displaying the wrong region for multi-region accounts that don't specify a preferred regions. We were returning the first location in a map of the location endpoints and the regions, so not necessarily the location related to the default endpoint. Now it should return the first region in the available write locations list. https://github.com/Azure/azure-sdk-for-java/issues/40696

API change check
API changes are not detected in this pull request.

/azp run java - cosmos - tests

/azp run java - cosmos - tests

/azp run java - cosmos - tests


text:
Azure JSON convenience APIs and Azure Core Serializer implementations

Description
Adds convenience overloads to JsonSerializable to simplify scenarios such as writing it to a String, OutputStream, or Writer and deserializing from a byte[], String, InputStream, or Reader.
Has the Azure Core serializer packages implement subtypes of JsonObject and JsonArray based on Jackson and GSON equivalents to simplify interoperability and reduce the need to convert JSON representation from one object type to another.
All SDK Contribution checklist:

[x] The pull request does not introduce [breaking changes]
[x] CHANGELOG is updated for new features, bug fixes or other significant changes.
[x] I have read the contribution guidelines.

General Guidelines and Best Practices

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[x] Pull request includes test coverage for the included changes.

API change check
APIView has identified API level changes in this PR and created following API reviews.
com.azure:azure-json

/check-enforcer override

authors: alzimmermsft azure-sdk


text:
Increment versions for cosmos releases

Increment package versions for cosmos releases

API change check
API changes are not detected in this pull request.

/check-enforcer override

authors: azure-sdk kushagraThapar


text:
[Automation] Generate Fluent Lite from Swagger workloads#package-2023-04

[Automation] Generate Fluent Lite from Swagger workloads#package-2023-04

API change check
APIView has identified API level changes in this PR and created following API reviews.
com.azure.resourcemanager:azure-resourcemanager-workloads


text:
@azure/openai 2.0.0-beta2 publishes declaration maps pointing to unpublished sources

Package Name: @azure/openai
Package Version: 2.0.0-beta.2
Operating system: Mac OS 14.2.1
[X] nodejs

version: 22.9.0

[ ] browser

name/version: Not Applicable

[X] typescript

version: 5.6.3

Is the bug related to documentation in

[ ] README.md
[ ] source code documentation
[ ] SDK API docs on https://learn.microsoft.com

Describe the bug
All current versions of @azure/openai published on NPM in the 2.0.0 branch (all alpha- and beta builds) include declaration maps (.d.ts.map) files. These are intended to be used in packages that also include their source code in the published package, such that consuming code editors can be taken directly to the original source declaration by via the TypeScript language service. However, @azure/openai 2.0.0-beta2 does not include the src folder that the declaration maps map the sources to.
To Reproduce
Steps to reproduce the behavior:

Install @azure/openai (or decompress and inspect the .tar archive packed by NPM)
Observe that declaration maps (.d.ts.map files) are shipped, pointing to source that are not published along with the package.

Expected behavior
The package should either not include declaration maps, or include the sources.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.

Hi @wessberg, thank you for reaching out! Yes, @azure/openai is a companion library to openai and is not exporting runtime JS definitions at the moment but it might in the future. Please checkout the current samples folder for examples of how to use it alongside openai: https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/openai/openai/samples/v2-beta

The package should include actual executable code.

Could you share more about your use case and why is this a concern?

@deyaaeldeen
Sure! As described in the original issue:

As a consequence, the section from the Migration guide from v1 to v2 about how to import and construct an AzureKeyCredential with an api token is currently non-functional, both for JavaScript runtimes, but also in the type layer, as @azure/openai does not export any typings either (only via @azure/openai/types, which the migration guide does not describe).

A reported issue that is related to this is: https://github.com/Azure/azure-sdk-for-js/issues/31170

Or, am I reading the migration guide incorrectly? Are you there describing how AzureKeyCredentials were created in the past, and that environment variables should be used going forward?

Are you there describing how AzureKeyCredentials were created in the past, and that environment variables should be used going forward

Yes.

OK. In that case, let me rephrase the original issue to only cover that the package ships declaration maps pointing to sources not shipped alongside the published package.

The package should either not include declaration maps, or include the sources.

For context, the package provides partial declaration maps for exclusive features in Azure OpenAI. Importing @azure/openai/types in your code will light up those features in your intellisense using module augmentation. The actual code in the OpenAI client library doesn't need to do anything special to handle requests and responses for those features so no need to ship actual definitions.
I hope this explanation helps clarify the organizational structure of the code in the package. If you have specific needs or questions about your use case, I would be glad to review them to determine if adjustments are necessary. Meanwhile, I encourage you to explore the provided samples.
I am closing this issue for now, but please do not hesitate to reopen it or create new ones if you encounter further questions or issues.

@deyaaeldeen I understand that. I'm not refering to the declaration files, (.d.ts), I'm refering to the declaration maps (.d.ts.map) files, which are published by your library, because your tsconfig is configured to generate them. But, they are pointing to sources that are not bundled along with the package. In your case, declaration maps are really not relevant, as the sources and dist are effectively the same, considering they're all working with ambient declarations in the type layer.

Sorry I missed that earlier; you're right. The maps are great for us to check our code, but they might not mean much to our customers without the src folder included. I'll circle back to you on this soon. Meanwhile, could you let me know if this is causing any problems for you? Thanks for your patience!

No worries! And no, it's only a minor thing. The Typescript Language Service will still fall back to the .d.ts file for Go To Definition actions if the referenced source file could not be resolved. So really you can just backlog this, or close it again if it provides you value internally.

authors: deyaaeldeen wessberg

repo: Azure/azure-sdk-for-js


text:
[Service Bus] Debug date type support for SQLParam

Issue - Part 1 of #9850

Closing.
Will be done as part of #10460


text:
Add purview-sharing rest client

Packages impacted by this PR
Issues associated with this PR
Describe the problem that is addressed by this PR
Adding rest client package for Sharing service under Microsoft Purview product group.
What are the possible designs available to address the problem? If there are more than one possible design, why was the one in this PR chosen?
Are there test cases added in this PR? (If not, why?)
Yes
Provide a list of related PRs (if any)
Command used to generate this PR:**(Applicable only to SDK release request PRs)
Checklists

[ ] Added impacted package name to the issue description
[ ] Does this PR needs any fixes in the SDK Generator?** (If so, create an Issue in the Autorest/typescript repository and link it here)
[x] Added a changelog (if necessary)

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-rest-purview-sharing

authors: anniruddh azure-sdk


text:
Update Azure Quantum Jobs readme, tests and samples to use Quantum Computing sample

Packages impacted by this PR
quantum-jobs
Issues associated with this PR
none
Describe the problem that is addressed by this PR
This PR updates the Azure.Quantum.Jobs Readme, Sample and Tests to use a Quantum Computing job instead of the recently deprecated Quantum Inspired Optimization (QIO) job.
The SDK itself has not changed and a new release is not needed.
What are the possible designs available to address the problem? If there are more than one possible design, why was the one in this PR chosen?
Azure Quantum QIO job type was deprecated, so we had to update our readme, samples and tests.
Are there test cases added in this PR? (If not, why?)
Yes, Tests were updated.
Provide a list of related PRs (if any)

https://github.com/Azure/azure-sdk-for-java/pull/36153
https://github.com/Azure/azure-sdk-for-net/pull/37892

Command used to generate this PR:**(Applicable only to SDK release request PRs)
Checklists

[x] Added impacted package name to the issue description
[ ] Does this PR needs any fixes in the SDK Generator?** (If so, create an Issue in the Autorest/typescript repository and link it here)
[ ] Added a changelog (if necessary)

API change check
API changes are not detected in this pull request.

Replaced by #26527

authors: azure-sdk vxfield


text:
[AutoPR subscription/resource-manager] Subscription new Py path

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/6053

This PR has been merged into https://github.com/Azure/azure-sdk-for-js/pull/3097


text:
[QUERY] XML docs for DefaultAzureCredential.GetTokenAsync

Query/Question
Thanks for the super helpful library 🙇♂
The XML docs for DefaultAzureCredential.GetTokenAsync mention the following:

This method is called by Azure SDK clients. It isn't intended for use in application code.

While I understand the main use of TokenCredential.GetTokenAsync is for the various Azure SDK clients, we found ourselves calling it manually to get tokens for Azure SQL. See this yet to be published blog post 👉 https://github.com/mderriey/mderriey.github.io/blob/azure-sql-managed-identity/_posts/2020-07-11-connect-to-azure-sql-with-aad-and-managed-identities.md#connecting-to-azure-sql-using-azure-active-directory-authentication
The portion of the XML docs aforementioned got people on my team asking if using this class directly was something we should be doing.
If you think it's fine, could we look at updating the XML docs?
I we shouldn't be using it directly, would you mind pointing us in the right direction?
Environment:

Name and version of the Library package used: Azure.Identity 1.2.0-preview.5
Hosting platform or OS and .NET runtime version (dotnet --info output for .NET Core projects): Not relevant
IDE and version: Not relevant

Nice blog! :) Let me know when you publish it.

Thanks @Petermarcu, it's now published over at https://mderriey.com/2020/07/17/connect-to-azure-sql-with-aad-and-managed-identities/

@mderriey Thanks for reaching out with this question, and thanks for the great blog post! Your use of the GetTokenAsync method here is perfectly valid. The statement from the XML docs you mentioned above, was meant to discourage people from calling GetTokenAsync directly in the most common case that they are using a client from our SDK which supports authenticating using a TokenCredential. However, it's probably too strongly worded, as we didn't intend to discourage usages such as what you demonstrate in your blog post. I think we should probably re-word this a bit. Did you have any suggestions in mind?

Hey @schaabs, thanks for getting back to me on this!
Right now, the docs read:

This method is called by Azure SDK clients. It isn't intended for use in application code.

Here are a few suggestions:

This method is called by Azure SDK clients. It isn't intended for use in application code when using them.
When using Azure SDK clients, this method is automatically called through TokenCredential and doesn't need to be used in application code.
Application code shouldn't use this method when using Azure SDK clients as they internally call it to acquire tokens.

Naming Phrasing is hard, right? 😂

I'm also considering to use the chaining logic from ManagedIdentityCredential + ClientSecretCredential from this library instead of ConfidentialClientApplication (from Microsoft.Identity.Client) to get a token for connecting to a Azure AD protected WebApi.
However does the ManagedIdentityCredential and ConfidentialClientApplication also cache the token?

@StefH this is good question. As of our 1.2.0 release the ClientSecretCredential does cache acquired tokens, but the ManagedidentityCredential does not. We do have plans to update the ManagedIdentityCredential so that it also caches acquired tokens, but this requires some changes from Microsoft.Identity.Client so it probably won't happen until the end of this year, early next year.
In general if you are calling GetTokenAsync on an implementation of TokenCredential directly, you shouldn't assume it caches the token. This is why the method returns the AccessToken structure which not only contains the Token but also the ExpiresOn time.

@schaabs Thanks for explaining. In that case I'll cache the tokens using IMemoryCache.
My code looks like:
internal class AccessTokenService : IAccessTokenService
{
    private const string Key = "StefAccessTokenService_AccessToken";

    private readonly Lazy<TokenRequestContext> _tokenRequestContext;
    private readonly Lazy<TokenCredential> _tokenCredential;
    private readonly ILogger<AccessTokenService> _logger;
    private readonly IMemoryCache _cache;

    public AccessTokenService(ILogger<AccessTokenService> logger, IOptions<AuditClientMicrosoftIdentityClientOptions> options, IMemoryCache cache)
    {
        _logger = logger;
        _cache = cache;

        _tokenRequestContext = new Lazy<TokenRequestContext>(() => new TokenRequestContext(new[] { $"{options.Value.Resource}/.default" }));
        _tokenCredential = new Lazy<TokenCredential>(() => CreateTokenCredential(options.Value));
    }

    public async Task<string> GetTokenAsync(CancellationToken cancellationToken)
    {
        var cachedAccessToken = await
            _cache.GetOrCreateAsync(Key, async entry =>
            {
                _logger.LogInformation("getting new token");
                var accessToken = await _tokenCredential.Value.GetTokenAsync(_tokenRequestContext.Value, cancellationToken).ConfigureAwait(false);

                entry.AbsoluteExpiration = accessToken.ExpiresOn;
                return accessToken;
            }).ConfigureAwait(false);

        _logger.LogInformation("token  : " + cachedAccessToken.Token);
        _logger.LogInformation("expire : " + cachedAccessToken.ExpiresOn.ToLocalTime());

        return cachedAccessToken.Token;
    }

    private TokenCredential CreateTokenCredential(AuditClientMicrosoftIdentityClientOptions optionsValue)
    {
        return new ChainedTokenCredential(
            new ManagedIdentityCredential(optionsValue.ClientId),
            new ClientSecretCredential(optionsValue.TenantId, optionsValue.ClientId, optionsValue.ClientSecret)
        );
    }
}

Fixed by https://github.com/Azure/azure-sdk-for-net/pull/15696

authors: Petermarcu StefH mderriey schaabs

repo: Azure/azure-sdk-for-net


text:
Implement better error messaging when ulong values are used in TableEntity custom models

Library name
Azure.Tables
Please describe the feature.
Table does not support ulong. However, in the SDK source code, we have several places trying to treat ulong as long, which is misleading and may cause overflow problem and get rejected by the Azure Storage Table without meaningful error message.
e.g.:
https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/tables/Azure.Data.Tables/src/Extensions/DictionaryTableExtensions.cs#L40
https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/tables/Azure.Data.Tables/src/Extensions/TablesTypeBinder.cs#L39
#25968

Still valid, but low pri


text:
[BUG] Azure GPT-4-Turbo JSON mode response generation breaks after 1024 tokens

Library name and version
openai==1.11.1
Describe the bug
Continuing this post.
I'm running a data extraction tasks on documents and I'm trying to take advantage of the 128k context window that gpt-4-turbo offers as well as the JSON mode setting. I'm experiencing a bug where the generation breaks at token length 1024 during generation. Afaik, the output length limit should be 4096. Even then, it looks like the bugs are similar to an issue that the docs are mentioning, but the existing solutions / recommendations are insufficient and don't explain this specific observation I'm making.
Reproduction Steps Note
As I was writing this post, I was creating code to reproduce this error arbitrarily and got an "error message" I never got before during my data extraction tasks.
{
    "status": "failed",
    "reason": "The current configuration of the AI model does not support generating a response that exceeds 1024 tokens. Counting to 1500 in JSON would go beyond this token limit and is therefore not possible within a single response."
}

This is NOT a real error messages from the endpoint. It's the GPT generated response from the chat completion. This is confusing me even more but leading me to believe there's some config setting that's disabled or not working as intended, or the off chance that OpenAI as a whole forgot to enable 4096 output token length.
Expected behavior
A JSON mode response that goes beyond 1024 tokens
Actual behavior
Infinite Stream of Blank Characters

When using JSON mode, always instruct the model to produce JSON via some message in the conversation, for example via your system message. If you don't include an explicit instruction to generate JSON, the model may generate an unending stream of whitespace and the request may run continually until it reaches the token limit. To help ensure you don't forget, the API will throw an error if the string "JSON" does not appear somewhere in the context.

My prompts do include the JSON mode instruction, and even then, I observe that when the response generation reaches token length 1024, it'll begin generating an infinite stream of blank characters until content_filter is triggered. Below, I've stripped the blank space characters to indicate the token length.

Premature Stop and Malformed JSON Response

The JSON in the message the model returns may be partial (i.e. cut off) if finish_reason is length, which indicates the generation exceeded max_tokens or the conversation exceeded the token limit. To guard against this, check finish_reason before parsing the response.

The response returned was a result of the length stop condition. I noted that the token length of the returned response was 1058 tokens; however, on further inspection I actually saw that the JSON response was malformed (beyond being incomplete) and it began at exactly where the token length reached 1024!

Reproduction Steps
Just create some messages and make a chat completion that'll likely generate a response token_length > 1024. Here's some code below to get started
client = AzureOpenAI(
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_key=os.environ["AZURE_OPENAI_API_KEY"],
    api_version="2023-05-15",
)

initial_messages = [
    {
        "role": self.SYSTEM,
        "content": "You are an AI Assistant. You will follow the user instruction. You will write a JSON response.",
    },
    {
        "role": self.USER,
        "content": "I want you to use GPT to generate the counting to 1500. Do not create code. Respond in JSON only. I do not care about reasonableness. You will count to 1500. You will write each number individually, and your output length max is 4096 tokens.",
    },
]

response_stream = client.chat.completions(
    model="gpt-4-turbo",
    response_format={"type": "json_object"},
    messages=initial_messages,
    max_tokens=4096,
    stream=True,
)

response_from_stream = ""

blank_space_threshold = 125
for chat_completion_chunk in response_stream:
    if (
        hasattr(chat_completion_chunk, "choices")
        and chat_completion_chunk.choices
    ):
        if (
            hasattr(chat_completion_chunk.choices[0], "delta")
            and chat_completion_chunk.choices[0].delta
            and hasattr(
                chat_completion_chunk.choices[0].delta, "content"
            )
        ):
            content = chat_completion_chunk.choices[0].delta.
            if content:
                response_from_stream += content

                if not content.isspace():
                    consecutive_blank_space_count = 0
                else:
                    consecutive_blank_space_count += len(content)
                    if (
                        consecutive_blank_space_count
                        > blank_space_threshold
                    ):
                        raise Exception(
                            "Encountered blank space bug."
                            + "\n"
                            + "Response from Stream Token Length: "
                            + str(
                                self.num_tokens_from_response(
                                    response_from_stream,
                                    model=self.chatgpt_model,
                                )
                            )
                            + "\n"
                            + "Response from Stream: "
                            + response_from_stream
                        )

Environment
No response

Troubleshooting
Confirm max_tokens and model deployment
Can confirm this runs, and increasing max_tokens beyond 4096 will cause a model error as expected.
response_stream = client.chat.completions(
    model="gpt-4-turbo",
    response_format={"type": "json_object"},
    messages=initial_messages,
    max_tokens=4096,
    stream=True,
)

Test random settings and prompts
Tried different prompts, temperature values, etc. Included more and less "JSON" instructions. Never generated more than 1024 tokens.
JSON mode on vs. off
Generation is not capped by the observed 1024 token limit when JSON mode is OFF. It appears to be a specific bug that occurs when JSON mode is on.


text:
Search SDK: Ensuring that all URL paths are in OData format

Azure Search allows normal path syntax for keys, but sometimes this can lead to ambiguity between operation names and resource names, like index names. For example, naming an index "stats" would cause problems because "stats" is a reserved action name in the REST API. We're making our routing smarter in the API to make such clashes less likely, but we're updating the SDK just to be on the safe side.
We are not bumping the NuGet package version yet because we're going to batch up this change with other changes before pushing a new version.

@azuresdkci retest this please

authors: brjohnstmsft markcowl


text:
Bumped AssemblyFileVersion in AssemblyInfo.cs for DeviceUpdate

All SDK Contribution checklist:
This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] Please open PR in Draft mode if it is:

Work in progress or not intended to be merged.
Encountering multiple pipeline failures and working on fixes.

[ ] If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
[x] I have read the contribution guidelines.
[x] The pull request does not introduce breaking changes.

General Guidelines and Best Practices

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

SDK Generation Guidelines

[ ] The generate.cmd file for the SDK has been updated with the version of AutoRest, as well as the commitid of your swagger spec or link to the swagger spec, used to generate the code. (Track 2 only)
[x] The *.csproj and AssemblyInfo.cs files have been updated with the new version of the SDK. Please double check nuget.org current release version.

Additional management plane SDK specific contribution checklist:
Note: Only applies to Microsoft.Azure.Management.[RP] or Azure.ResourceManager.[RP]

[ ] Include updated management metadata.
[ ] Update AzureRP.props to add/remove version info to maintain up to date API versions.

Management plane SDK Troubleshooting

If this is very first SDK for a services and you are adding new service folders directly under /SDK, please add new service label and/or contact assigned reviewer.
If the check fails at the Verify Code Generation step, please ensure:

Do not modify any code in generated folders.
Do not selectively include/remove generated files in the PR.
Do use generate.ps1/cmd to generate this PR instead of calling autorest directly.
Please pay attention to the @microsoft.csharp version output after running generate.ps1. If it is lower than current released version (2.3.82), please run it again as it should pull down the latest version,

Old outstanding PR cleanup
Please note:
If PRs (including draft) has been out for more than 60 days and there are no responses from our query or followups, they will be closed to maintain a concise list for our reviewers.

@YalinLi0312 please approve. I eventually touched only 1 attribute, that has fixed failing build checks.


text:
[Storage] Support 2021-02-01 and upgrade to 20.0.0

All SDK Contribution checklist:
swagger PR merged: https://github.com/Azure/azure-rest-api-specs/pull/13550
This checklist is used to make sure that common guidelines for a pull request are followed.

[x] Please open PR in Draft mode if it is:

Work in progress or not intended to be merged.
Encountering multiple pipeline failures and working on fixes.

[x] If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
[x] I have read the contribution guidelines.
[x] The pull request does not introduce breaking changes.

General Guidelines and Best Practices

[x] Title of the pull request is clear and informative.
[x] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[x] Pull request includes test coverage for the included changes.

SDK Generation Guidelines

[x] The generate.cmd file for the SDK has been updated with the version of AutoRest, as well as the commitid of your swagger spec or link to the swagger spec, used to generate the code. (Track 2 only)
[x] The *.csproj and AssemblyInfo.cs files have been updated with the new version of the SDK. Please double check nuget.org current release version.

Additional management plane SDK specific contribution checklist:
Note: Only applies to Microsoft.Azure.Management.[RP] or Azure.ResourceManager.[RP]

[x] Include updated management metadata.
[x] Update AzureRP.props to add/remove version info to maintain up to date API versions.

Management plane SDK Troubleshooting

If this is very first SDK for a services and you are adding new service folders directly under /SDK, please add new service label and/or contact assigned reviewer.
If the check fails at the Verify Code Generation step, please ensure:

Do not modify any code in generated folders.
Do not selectively include/remove generated files in the PR.
Do use generate.ps1/cmd to generate this PR instead of calling autorest directly.
Please pay attention to the @microsoft.csharp version output after running generate.ps1. If it is lower than current released version (2.3.82), please run it again as it should pull down the latest version,

Old outstanding PR cleanup
Please note:
If PRs (including draft) has been out for more than 60 days and there are no responses from our query or followups, they will be closed to maintain a concise list for our reviewers.

Hi @blueww , could you fix the errors in ci checks?

The error looks like a network error on download nuget package in build.
Will rerun the CI

/Users/runner/hostedtoolcache/dotnet/sdk/5.0.200/NuGet.targets(131,5): error : Failed to download package 'Microsoft.AspNetCore.Authorization.Policy.2.1.0' from 'https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.authorization.policy/2.1.0/microsoft.aspnetcore.authorization.policy.2.1.0.nupkg'. [/Users/runner/work/1/s/eng/service.proj]
/Users/runner/hostedtoolcache/dotnet/sdk/5.0.200/NuGet.targets(131,5): error : The HTTP request to 'GET https://api.nuget.org/v3-flatcontainer/microsoft.aspnetcore.authorization.policy/2.1.0/microsoft.aspnetcore.authorization.policy.2.1.0.nupkg' has timed out after 100000ms. [/Users/runner/work/1/s/eng/service.proj]
/Users/runner/hostedtoolcache/dotnet/sdk/5.0.200/NuGet.targets(131,5): error : The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'Microsoft.AspNetCore.Authorization.Policy.2.1.0' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again. [/Users/runner/work/1/s/eng/service.proj]
/Users/runner/hostedtoolcache/dotnet/sdk/5.0.200/NuGet.targets(131,5): error :   Unable to find package 'Microsoft.AspNetCore.Authorization.Policy.2.1.0'. [/Users/runner/work/1/s/eng/service.proj]

/azp run

The only CI failure is not valid:
The failure is on verify generated code, error is "Discrepancy detected between generated code in PR and reference generation." But I have not change the code after generate it.
Since autorest issue https://github.com/Azure/autorest.csharp/issues/885, (.net SDK code should skip validate optional properties if it's null), generate .net SRP SDK code must use --use:@microsoft.azure/autorest.csharp@2.3.90 now, and I have added it into generate.ps1.
But the code generated by CI will always check the optional properties no matter it's null or not, so the CI generated code seems not right. It seems CI don't generate SDK code use the correct autorest version, so it might not use generate.ps1 correctly.
Besides that, the failure also occur in the my last PR which is already merged.
Please help to review and merged the PR if not other comments.
And please help to fix the CI failure by mail CI use correct generate.ps1, to avoid it fail again in the future PR.


text:
Increment version for remoterendering releases

Increment package version after release of Azure.MixedReality.RemoteRendering

/check-enforcer evaluate

authors: azure-sdk jsquire


text:
Increment version for webpubsub releases

Increment package version after release of Azure.ResourceManager.WebPubSub

API change check
API changes are not detected in this pull request.


text:
ACR 3.0 SDK (support ACR Build/Task GA Service)

Description

Swagger: https://github.com/Azure/azure-rest-api-specs/tree/master/specification/containerregistry/resource-manager
Swagger PR: https://github.com/Azure/azure-rest-api-specs/pull/3617

This checklist is used to make sure that common guidelines for a pull request are followed.

[X] Please add REST spec PR link to the SDK PR
[X] I have read the contribution guidelines.
[] The pull request does not introduce breaking changes.

General Guidelines

[X] Title of the pull request is clear and informative.
[X] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[X] Pull request includes test coverage for the included changes.

SDK Generation Guidelines

[X] If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
[X] The generate.cmd file for the SDK has been updated with the version of AutoRest, as well as the commitid of your swagger spec or link to the swagger spec, used to generate the code.
[X] The *.csproj and AssemblyInfo.cs files have been updated with the new version of the SDK.

@djyou

@northtyphoon do you need any other approvals or is this good to merge

@dsgouda, it's ready to merge.

Will merge once CIs pass

authors: dsgouda northtyphoon


text:
[Compute] Adding 'Spot' value to 'Priority' Enum.

SDK Tests related to swagger update: https://github.com/Azure/azure-rest-api-specs/pull/7748.
Refactored the test class "VMScaleSetPriorityTests"

/azp run

Now, this PR also includes https://github.com/Azure/azure-sdk-for-net/pull/8915

@isra-fel  Hi, @rakeshakkera updated the version number and included the .txt file.  (Also, regenerated from the latest Swagger master branch)

authors: hyonholee rakeshakkera


text:
Support Cosmos partial update

Support Cosmos DB partial document update
I suggest supporting both strings and SDK-provided enumerations for the values of operations' op (eg {"op": "set", ...} and {"op": azure.cosmos.PartialUpdateOperation.set, ...})

Hi! Thanks for flagging this. This is on our roadmap. Happy to provide updates on this whenever you want to check in the future.

@gahl-levy  has there been any progress on this / when can we expect this feature to be ready?

Hi @angusrw. This feature is on our roadmap, but we don't have an ETA just yet unfortunately.

Does this feature have a timeline yet?

@clrogers2 - this is currently planned in the ongoing semester, and is scheduled to be done by March 2023.

Is this still planned for March '23?  My team has plans to update our product with this functionality as soon as available.

Is there any update on when this will be finished/live?

Is there any update on this feature?

This feature is still in our roadmap and we are targeting for July'23

Version 4.4.0b1 is out and addresses this, thank you! https://pypi.org/project/azure-cosmos/4.4.0b1/

authors: AbhinavDAIICT EpicWink PrithviKambhampati aldc-seanogrady angusrw clrogers2 gahl-levy kushagraThapar simorenoh

repo: Azure/azure-sdk-for-python


text:
Add offline managed identity tests

i.e. validate the client creates the correct request for each platform:

[ ] App Service (2017-09-01)
[ ] Arc
[ ] Cloud Shell
[ ] IMDS
[x] Service Fabric

ZenHub! I shake my fist.


text:
wrong documentation link for azure-mgmt-privatedns

The azure-mgmt-privatedns/README.rst file says that code examples may be found at https://docs.microsoft.com/python/api/overview/azure/ but there are no Private DNS code examples there.

Hi @qwrrty
The link is right to the place the sample should be, but the sample doesn't exist yet. Our template system to build the Readme assumes the sample will be written soon enough that we can put the link in place, I'm sorry in this particular situation it was not the case :(
Tagging @muwaqar from the team, who might be able to provide C# samples or some examples in the meantime, to give some clues.
Thanks,

Tagging @dijyotir since I am currently OOF.
On Sat, Apr 6, 2019, 7:24 AM Laurent Mazuel notifications@github.com
wrote:

Hi @qwrrty https://github.com/qwrrty
The link is right to the place the sample should be, but the sample
doesn't exist yet. Our template system to build the Readme assumes the
sample will be written soon enough that we can put the link in place, I'm
sorry in this particular situation it was not the case :(
Tagging @muwaqar https://github.com/muwaqar from the team, who might be
able to provide C# samples or some examples in the meantime, to give some
clues.
Thanks,
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-sdk-for-python/issues/4790#issuecomment-480451078,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAXp6KAu02LJ8cfQ-G4RkUsg-2ENx025ks5vd9tIgaJpZM4cfmZE
.

authors: lmazuel muwaqar qwrrty


text:
[AutoPR cognitiveservices/data-plane/TextAnalytics] Update CognitiveServices/Text-Analytics Swagger with new /entities endpoint

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/2985

(message created by the CI based on PR content)

Codecov Report

Merging #2503 into restapi_auto_cognitiveservices/data-plane/TextAnalytics will increase coverage by 0.01%.
The diff coverage is 65.08%.

@@                                     Coverage Diff                                     @@
##           restapi_auto_cognitiveservices/data-plane/TextAnalytics    #2503      +/-   ##
===========================================================================================
+ Coverage                                                    55.12%   55.13%   +0.01%
===========================================================================================
  Files                                                         6369     6391      +22
  Lines                                                       141224   141450     +226
===========================================================================================
+ Hits                                                         77848    77995     +147
- Misses                                                       63376    63455      +79

Impacted Files
Coverage Δ

...ervices/language/textanalytics/models/input_py3.py
100% <100%> (ø)

...e/textanalytics/models/text_analytics_api_enums.py
100% <100%> (ø)
:arrow_up:

.../textanalytics/models/language_batch_result_py3.py
100% <100%> (ø)

...uage/textanalytics/models/language_batch_result.py
100% <100%> (ø)
:arrow_up:

...vices/language/textanalytics/models/batch_input.py
100% <100%> (ø)
:arrow_up:

...iveservices/language/textanalytics/models/input.py
100% <100%> (ø)
:arrow_up:

...ognitiveservices/language/textanalytics/version.py
100% <100%> (ø)
:arrow_up:

...language/textanalytics/models/detected_language.py
100% <100%> (ø)
:arrow_up:

...analytics/models/language_batch_result_item_py3.py
100% <100%> (ø)

...s/language/textanalytics/models/batch_input_py3.py
100% <100%> (ø)

... and 52 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 215a44f...c521c4a. Read the comment docs.

This PR has been merged into https://github.com/Azure/azure-sdk-for-python/pull/2522

authors: AutorestCI codecov-io