text:
create_from_edge_environment does not send messages to IoT Hub

Hello,
I've been facing a weird issue using:
iotedge 1.2.10
edgeAgent/edgeHub 1.2.10.55800403
azure-iot-device 2.11.0
When I create the client module using create_from_edge_environment function within my IoT Edge Module, the module is able to correctly push and pull twin, but any message sent is never received on the hub.
If I use the from_connection_string and use a connection string within the same Docker, it works...
This is really strange, here some information I got from logs:
2022-06-08 09:13:18 azure.iot.device.common.mqtt_transport _create_mqtt_client 149 [INFO]: Creating client for connecting using MQTT over TCP
2022-06-08 09:13:18 azure.iot.device.common.http_transport request 108 [INFO]: sending https POST request to devices/Gateway-1200723-01004/files .
2022-06-08 09:13:19 azure.iot.device.iothub.aio.async_clients get_storage_info_for_blob 572 [INFO]: Successfully retrieved storage_info
2022-06-08 09:13:20 azure.iot.device.common.http_transport request 108 [INFO]: sending https POST request to devices/Gateway-1200723-01004/files/notifications .
2022-06-08 09:13:20 azure.iot.device.iothub.aio.async_clients notify_blob_upload_status 598 [INFO]: Successfully notified blob upload status
2022-06-08 09:13:20 azure.iot.device.iothub.aio.async_clients send_message 349 [INFO]: Sending message to Hub...
2022-06-08 09:13:20 azure.iot.device.common.mqtt_transport publish 528 [INFO]: publishing on devices/Gateway-1200723-01004/modules/GWApp/messages/events/
2022-06-08 09:13:20 azure.iot.device.common.mqtt_transport on_publish 243 [INFO]: payload published for 8
2022-06-08 09:13:20 azure.iot.device.iothub.aio.async_clients send_message 356 [INFO]: Successfully sent message to Hub

Jun 08 09:13:05 iot-gate-imx8 aziot-edged[21652]: 2022-06-08T09:13:05Z [INFO] - [mgmt] - - - [2022-06-08 09:13:05.748824404 UTC] "GET /modules?api-version=2021-12-07 HTTP/1.1" 200 OK 1952 "-" "-" auth_id(-)
Jun 08 09:13:10 iot-gate-imx8 aziot-edged[21652]: 2022-06-08T09:13:10Z [INFO] - [mgmt] - - - [2022-06-08 09:13:10.787328141 UTC] "GET /modules?api-version=2021-12-07 HTTP/1.1" 200 OK 1952 "-" "-" auth_id(-)
Jun 08 09:13:15 iot-gate-imx8 aziot-edged[21652]: 2022-06-08T09:13:15Z [INFO] - [mgmt] - - - [2022-06-08 09:13:15.820943495 UTC] "GET /modules?api-version=2021-12-07 HTTP/1.1" 200 OK 1952 "-" "-" auth_id(-)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (74 bytes)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - <-- POST /key?api-version=2020-09-01 {"content-length": "74", "content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] flushed 377 bytes
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] read eof
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (2599 bytes)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - <-- POST /decrypt?api-version=2020-09-01 {"content-length": "2599", "content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] flushed 2286 bytes
Jun 08 09:13:20 iot-gate-imx8 aziot-edged[21652]: 2022-06-08T09:13:20Z [INFO] - [work] - - - [2022-06-08 09:13:20.484697762 UTC] "POST /modules/%24edgeHub/genid/637800092105391013/decrypt?api-version=2019-01-30 HTTP/1.1" 200 OK 2176 "-" "-" auth_id(-)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] read eof
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (74 bytes)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - <-- POST /key?api-version=2020-09-01 {"content-length": "74", "content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] flushed 377 bytes
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] read eof
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (2574 bytes)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - <-- POST /encrypt?api-version=2020-09-01 {"content-length": "2574", "content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] flushed 2311 bytes
Jun 08 09:13:20 iot-gate-imx8 aziot-edged[21652]: 2022-06-08T09:13:20Z [INFO] - [work] - - - [2022-06-08 09:13:20.564417345 UTC] "POST /modules/%24edgeHub/genid/637800092105391013/encrypt?api-version=2019-01-30 HTTP/1.1" 200 OK 2201 "-" "-" auth_id(-)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] read eof
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (74 bytes)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - <-- POST /key?api-version=2020-09-01 {"content-length": "74", "content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] flushed 377 bytes
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] read eof
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (2599 bytes)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - <-- POST /decrypt?api-version=2020-09-01 {"content-length": "2599", "content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] flushed 2286 bytes
Jun 08 09:13:20 iot-gate-imx8 aziot-edged[21652]: 2022-06-08T09:13:20Z [INFO] - [work] - - - [2022-06-08 09:13:20.640107437 UTC] "POST /modules/%24edgeHub/genid/637800092105391013/decrypt?api-version=2019-01-30 HTTP/1.1" 200 OK 2176 "-" "-" auth_id(-)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] read eof
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (74 bytes)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - <-- POST /key?api-version=2020-09-01 {"content-length": "74", "content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] flushed 377 bytes
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] read eof
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (3614 bytes)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - <-- POST /encrypt?api-version=2020-09-01 {"content-length": "3614", "content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] flushed 3351 bytes
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] read eof
Jun 08 09:13:20 iot-gate-imx8 aziot-edged[21652]: 2022-06-08T09:13:20Z [INFO] - [work] - - - [2022-06-08 09:13:20.717824087 UTC] "POST /modules/%24edgeHub/genid/637800092105391013/encrypt?api-version=2019-01-30 HTTP/1.1" 200 OK 3241 "-" "-" auth_id(-)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (74 bytes)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - <-- POST /key?api-version=2020-09-01 {"content-length": "74", "content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] flushed 377 bytes
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] read eof
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (3639 bytes)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - <-- POST /decrypt?api-version=2020-09-01 {"content-length": "3639", "content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] flushed 3326 bytes
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] read eof
Jun 08 09:13:20 iot-gate-imx8 aziot-edged[21652]: 2022-06-08T09:13:20Z [INFO] - [work] - - - [2022-06-08 09:13:20.806012386 UTC] "POST /modules/%24edgeHub/genid/637800092105391013/decrypt?api-version=2019-01-30 HTTP/1.1" 200 OK 3216 "-" "-" auth_id(-)
Jun 08 09:13:20 iot-gate-imx8 aziot-edged[21652]: 2022-06-08T09:13:20Z [INFO] - [mgmt] - - - [2022-06-08 09:13:20.855975590 UTC] "GET /modules?api-version=2021-12-07 HTTP/1.1" 200 OK 1952 "-" "-" auth_id(-)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (74 bytes)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - <-- POST /key?api-version=2020-09-01 {"content-length": "74", "content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] flushed 377 bytes
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] read eof
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] parsed 2 headers
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body is content-length (3639 bytes)
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - <-- POST /decrypt?api-version=2020-09-01 {"content-length": "3639", "content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] incoming body completed
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [INFO] - --> 200 {"content-type": "application/json"}
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::io] flushed 3326 bytes
Jun 08 09:13:20 iot-gate-imx8 aziot-keyd[21666]: 2022-06-08T09:13:20Z [DBUG] - [hyper::proto::h1::conn] read eof
Jun 08 09:13:20 iot-gate-imx8 aziot-edged[21652]: 2022-06-08T09:13:20Z [INFO] - [work] - - - [2022-06-08 09:13:20.879883540 UTC] "POST /modules/%24edgeHub/genid/637800092105391013/decrypt?api-version=2019-01-30 HTTP/1.1" 200 OK 3216 "-" "-" auth_id(-)

As you can see, no message is received on the IoT Hub... I normally have a function app that will use the fallback route to receive messages sent.

Any way to investiguate it further on IoT Hub side ? Or some additional logs to investigate on python side ?
Thanks

@MrVhek have u raised this issue in the Edge repo?

To get more debug logs add:
import logging
logging.basicConfig(level=logging.DEBUG)
To this at the beginning of your application

@MrVhek have u raised this issue in the Edge repo?

Not yet, not sure if this can be related to your SDK or to Azure IoT Edge directly ?

To get more debug logs add:
import logging logging.basicConfig(level=logging.DEBUG)
At the beginning of your application.

Thanks, I've added it but don't see more python logs than those that I posted above.
I need to see if I can have more logs on Azure side.

i am facing same problem

@MrVhek can you try reverting to IoT Edge 1.1? There are known issues with using Edge 1.2 with the SDK.

Sorry for the late reply, I finally found the issue, it was related to the routing of send_message.

Be sure to not have an output after your module name (here GWApp or ToolsApp).

authors: MrVhek anthonyvercolano cartertinney raowaqas72

repo: Azure/azure-iot-sdk-python


text:
chore: Updated language classifiers

Formally indicated support for Python 3.9 and 3.10

Formally indicated support for Python 3.9 and 3.10

source: gharchive/pull-request

license: MIT

license_type: permissive

license_source: github-api


text:
Container Instances should support environment variable names containing dot '.'

System Info
OS: Windows 10
Shell: Ubuntu 20.04 on Windows
Azure CLI: 2.36.0

Expected Behavior
Executing the following Azure CLI command should create a container instance running Elasticsearch:
az container create --image elasticsearch:7.3.2 --name es -g test-rg --memory 2 --cpu 1 --ports 9200 --environment-variables "discovery.type=single-node"
Using environment variable names containing dot '.' should work.
Current Behavior
Executing the Azure CLI command exit with an error:
(InvalidContainerEnvironmentVariable) The environment variable name in container 'es' of container group 'es' is invalid. A valid environment variable name must start with alphabetic character or '_', followed by a string of alphanumeric characters or '_' (e.g. 'my_name',  or 'MY_NAME',  or 'MyName').
Code: InvalidContainerEnvironmentVariable
Message: The environment variable name in container 'es' of container group 'es' is invalid. A valid environment variable name must start with alphabetic character or '_', followed by a string of alphanumeric characters or '_' (e.g. 'my_name',  or 'MY_NAME',  or 'MyName').

@stephanwehr Thanks for reaching out to us and sharing this feedback. I was able to reproduce this issue from AzCLI as well as Powershell. Post isolating this issue, I found that this issue isn't related to the Container Instance but the underlying image OS type.
The bash manual mentions this clearly (here)

name
A word consisting solely of letters, numbers, and underscores, and beginning with a letter or underscore. Names are used as shell variable and function names. Also referred to as an identifier.

The elasticSearch image seems to be using Linux OSType. So the fix / workaround has to come from the OS side and not from the AzCLI / REST API Spec. Hope this helps.

@navba-MSFT you are correct, Bash can't but Elasticsearch can.
With the "env" command it is possible to set environment variables containing special characters.
Here is an example:
# env -i "cluster.initial_master_nodes=elelastico" "my:test:var=value" /bin/bash -c "env"
my:test:var=value
cluster.initial_master_nodes=elelastico
PWD=/root
SHLVL=0
_=/usr/bin/env

In Docker compose the environment can be set like this:
version: '2.2'

services:

  elasticsearch:
    image: elasticsearch:7.3.2
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.type=single-node
    ports:
      - 9200:9200

I've dug a little bit deeper on this issues and it seems Linux executes bash and other programs with execve().
With environ() an initial environment can be set.
How Linux execute programs can be seen using "strace":
$ strace env -i "my:test:var=some text" /bin/bash -c ":" 2>&1 | grep "execve"
execve("/usr/bin/env", ["env", "-i", "my:test:var=some text", "/bin/bash", "-c", ":"], 0x7ffd31dcb018 /* 21 vars */) = 0
execve("/bin/bash", ["/bin/bash", "-c", ":"], 0x5577895c20f0 /* 1 var */) = 0

As I understand, on Linux container environments like Docker for example set the initial environment before executing the 1st program inside the container. There is a process environment and bash environment which both can be retrieved using commands like "env" or "printenv".
I assume referring to this error message that the API performs a syntax check on variable names which should not be there:
{"error":{"code":"InvalidContainerEnvironmentVariable","message":"The environment variable name in container 'es' of container group 'es' is invalid. A valid environment variable name must start with alphabetic character or '_', followed by a string of alphanumeric characters or '_' (e.g. 'my_name',  or 'MY_NAME',  or 'MyName')."}}

@stephanwehr I just wanted to confirm if you are able to get this working with elasticsearch image with Linux OStype ? Awaiting your reply.

@navba-MSFT I'm not able to get Elasticsearch to work with Linux OStype because environment variables containing dot '.' are not supported.

@stephanwehr AFAIK, this ask is something which violates the IEEE standard for environment variables.

Environment variable names used by the utilities in the Shell and Utilities volume of IEEE Std 1003.1-2001 consist solely of uppercase letters, digits, and the '_' (underscore)

Refer this.

@navba-MSFT as stated this applies to the Shell and Utilities and not necessarily to other programs like Elasticsearch.
There is no hard requirement under Linux or inside a Docker container to use a shell to execute programs.
Linux itself allows variable names of type char:
       #include <stdlib.h>

       int setenv(const char *name, const char *value, int overwrite);
       int unsetenv(const char *name);

Reference

@stephanwehr Thanks for clarifying. I have added the Service Team to look into this feature request ask.

I've tried hosting ElasticSearch on ACI and got this issue.
Are there any updates?

Same for me tried to use ElasticSearch with ACI and I get the same error with dotted env variable names. Why is this closed? There is no fix or workaround

source: gharchive/issue

authors: iwate justinpenguin45 navba-MSFT stephanwehr

repo: Azure/azure-rest-api-specs


text:
webapp:New-AzResourceGroupDeployment for web site with slot fails if template contains "cloningInfo"

Description
I'm trying to deploy a web app with two slots (production, deploytarget). Everything works fine the first time, but subsequent runs with the web app already present fail with an error indicating the site already exists. I was under the impression that ARM templates should always be able to run even if the resources already exist? If not is there a quick way to tell ARM that it should not touch the slot if it exists?
The problem goes away if I remove the "cloningInfo" options from the template. It appears there is a problem with cloning onto an existing app service.
"cloningInfo": {
  "sourceWebAppId": "[resourceId('Microsoft.Web/sites', variables('webAppName'))]",
  "overwrite": true
}

Script/Steps for Reproduction
New-AzResourceGroupDeployment # with provided template

This is the template that is used:
{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "environment": {
      "type": "string",
      "allowedValues": [
        "devtest",
        "test",
        "stag",
        "prod"
      ]
    },

    "skuName": {
      "type": "string",
      "defaultValue": "S1"
    },
    "skuCapacity": {
      "type": "int",
      "defaultValue": 1
    },

    "nodeVersion": {
      "type": "string",
      "defaultValue": "10.6.0"
    },

    "location": {
      "type": "string",
      "defaultValue": "[resourceGroup().location]"
    },
    "deployTargetSlotName": {
      "type": "string",
      "defaultValue": "deploytarget"
    }
  },
  "variables": {
    "appServiceName": "[concat('TestAppServiceName-ASP-', parameters('environment'))]",
    "webAppName": "[concat('TestAppServiceName-APP-', parameters('environment'))]"
  },
  "resources": [
    {
      "name": "[variables('appServiceName')]",
      "type": "Microsoft.Web/serverfarms",
      "kind": "app",
      "sku": {
        "name": "[parameters('skuName')]",
        "capacity": "[parameters('skuCapacity')]"
      },
      "apiVersion": "2016-09-01",
      "location": "[parameters('location')]",
      "scale": null,
      "properties": {
          "name": "[variables('appServiceName')]"
      }
    },
    {
      "name": "[variables('webAppName')]",
      "type": "Microsoft.Web/sites",
      "apiVersion": "2018-02-01",
      "location": "[parameters('location')]",
      "tags": {
        "displayName": "[variables('webAppName')]"
      },
      "properties": {
        "name": "[variables('webAppName')]",
        "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('appServiceName'))]",
        "clientAffinityEnabled": false,
        "httpsOnly": true,
        "siteConfig": {
          "alwaysOn": true,
          "phpVersion": ""
        }
      },
      "dependsOn": [
        "[variables('appServiceName')]"
      ],
      "resources": [
        {
          "name": "[parameters('deployTargetSlotName')]",
          "type": "slots",
          "apiVersion": "2018-02-01",
          "dependsOn": [
            "[variables('webAppName')]"
          ],
          "location": "[parameters('location')]",
          "properties": {
            "enabled": false,
            "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('appServiceName'))]",
            "clientAffinityEnabled": false,
            "httpsOnly": true,
            "cloningInfo": {
              "sourceWebAppId": "[resourceId('Microsoft.Web/sites', variables('webAppName'))]",
              "overwrite": true
            }
          }
        }
      ]
    }
  ],
  "outputs": {
    "webappName": {
      "type": "string",
      "value": "[variables('webAppName')]"
    }
  }
}

Module Version
ModuleType Version    Name                                ExportedCommands
---------- -------    ----                                ----------------
Script     0.6.1      Az.Profile                          {Add-AzEnvironment, Clear-AzContext, Clear-AzDefault, Connect-AzAccount...}
Script     0.6.1      Az.Storage                          {Add-AzRmStorageContainerLegalHold, Add-AzStorageAccountNetworkRule, Disable-AzStorageDeleteRetentionPolicy, Enable-AzStorageDeleteRetentionPolicy...}
Manifest   3.1.0.0    Microsoft.PowerShell.Management     {Add-Computer, Add-Content, Checkpoint-Computer, Clear-Content...}
Manifest   3.0.0.0    Microsoft.PowerShell.Security       {ConvertFrom-SecureString, ConvertTo-SecureString, Get-Acl, Get-AuthenticodeSignature...}
Manifest   3.1.0.0    Microsoft.PowerShell.Utility        {Add-Member, Add-Type, Clear-Variable, Compare-Object...}
Manifest   3.0.0.0    Microsoft.WSMan.Management          {Connect-WSMan, Disable-WSManCredSSP, Disconnect-WSMan, Enable-WSManCredSSP...}
Script     1.2        PSReadline                          {Get-PSReadlineKeyHandler, Get-PSReadlineOption, Remove-PSReadlineKeyHandler, Set-PSReadlineKeyHandler...}

Environment Data
Name                           Value
----                           -----
PSVersion                      5.1.17134.407
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.17134.407
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Debug Output
13:23:59 - Resource Microsoft.Web/sites/slots 'TestAppServiceName-ASP-devtest/deploytarget' failed with message '{
  "status": "Failed",
  "error": {
    "code": "ResourceDeploymentFailure",
    "message": "The resource operation completed with terminal provisioning state 'Failed'.",
    "details": [
      {
        "message": "Conflict\r\nDetail: Site with name TestAppServiceName-ASP-devtest(deploytarget) already exists.\r\nExtendedCode: 04008\r\n   at Microsoft.Web.Hosting.Administration.Csm.Common.Clients.GeoClientBase.GetResponse[T](HttpWebRequest request, Boolean reThrowOnHostUnavailability, HttpStatusCode[]
supressLogForHttpCodes)\r\n   at Microsoft.Web.Hosting.Administration.GeoScale.Sql.WebSiteCloneManager.StartSiteRestore(SiteClone clone, Operation cloneOperation)\r\n"
      }
    ]
  }
}'

Is there any update in regards to this issue?

@panchagnula so its been over 2 years now that this bug was reported. You reassigned this over a year ago, but nothing happened.
@zikalino any update?
@maggiepint you maybe?
Can anyone from microsoft please give a status update?

@rudfoss did you ever find a workaround?

this issue seems already fixed

Is this issue ever going to be fixed?  It is impossible to redeploy a template when using cloningInfo unless you know ahead of time if the resource already exists and set the overwrite property accordingly.  This goes against the whole premise of ARM.

authors: PaulVrugt m1dst rudfoss simonethreechillies


text:
Fixing swagger correctness items RSV

MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Changelog
Add a changelog entry for this PR by answering the following questions:

What's the purpose of the update?

[ ] new service onboarding
[ ] new API version
[ ] update existing version for new feature
[ ] update existing version to fix swagger quality issue in s360
[ ] Other, please clarify

When are you targeting to deploy the new service/feature to public regions? Please provide the date or, if the date is not yet available, the month.
When do you expect to publish the swagger? Please provide date or, the the date is not yet available, the month.
If updating an existing version, please select the specific language SDKs and CLIs that must be refreshed after the swagger is published.

[ ] SDK of .NET (need service team to ensure code readiness)
[ ] SDK of Python
[ ] SDK of Java
[ ] SDK of Js
[ ] SDK of Go
[ ] PowerShell
[ ] CLI
[ ] Terraform
[ ] No refresh required for updates in this PR

Contribution checklist:

[ ] I commit to follow the Breaking Change Policy of "no breaking changes"
[ ] I have reviewed the documentation for the workflow.
[ ] Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.
ARM API Review Checklist

Applicability: :warning:
If your changes encompass only the following scenarios, you should SKIP this section, as these scenarios do not require ARM review.

Change to data plane APIs
Adding new properties
All removals

Otherwise your PR may be subject to ARM review requirements. Complete the following:

[ ] Check this box if any of the following apply to the PR so that label "WaitForARMFeedback" will be added automatically to begin ARM API Review. Failure to comply may result in delays to the manifest.

Adding a new service
Adding new API(s)
Adding a new API version
-[ ] To review changes efficiently, ensure you are using OpenAPIHub to initialize the PR for adding a new version. More details, refer to the wiki.

[ ] Ensure you've reviewed following guidelines including ARM resource provider contract and REST guidelines. Estimated time (4 hours). This is required before you can request review from ARM API Review board.

[ ] If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist
If any of the following scenarios apply to the PR, request approval from the Breaking Change Review Board as defined in the Breaking Change Policy.

[ ] Removing API(s) in a stable version
[ ] Removing properties in a stable version
[ ] Removing API version(s) in a stable version
[ ] Updating API in a stable or public preview version with Breaking Change Validation errors
[ ] Updating API(s) in public preview over 1 year (refer to Retirement of Previews)

Action: to initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Addition details on the process and office hours are on the Breaking change Wiki.
Please follow the link to find more details on PR review process.

Hi @pratimaupadhyay02, please add the new properties to new version and fix the linter errors.


text:
Copy QuestionAnswering definitions to common.json and refer them

MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Changelog
Add a changelog entry for this PR by answering the following questions:

What's the purpose of the update?

[ ] new service onboarding
[ ] new API version
[ ] update existing version for new feature
[ ] update existing version to fix swagger quality issue in s360
[ ] Other, please clarify

When are you targeting to deploy the new service/feature to public regions? Please provide the date or, if the date is not yet available, the month.
When do you expect to publish the swagger? Please provide date or, the the date is not yet available, the month.
If updating an existing version, please select the specific language SDKs and CLIs that must be refreshed after the swagger is published.

[ ] SDK of .NET (need service team to ensure code readiness)
[ ] SDK of Python
[ ] SDK of Java
[ ] SDK of Js
[ ] SDK of Go
[ ] PowerShell
[ ] CLI
[ ] Terraform
[ ] No refresh required for updates in this PR

Contribution checklist:

[ ] I commit to follow the Breaking Change Policy of "no breaking changes"
[ ] I have reviewed the documentation for the workflow.
[ ] Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.
ARM API Review Checklist

Applicability: :warning:
If your changes encompass only the following scenarios, you should SKIP this section, as these scenarios do not require ARM review.

Change to data plane APIs
Adding new properties
All removals

Otherwise your PR may be subject to ARM review requirements. Complete the following:

[ ] Check this box if any of the following appy to the PR so that the label "ARMReview" and "WaitForARMFeedback" will be added by bot to kick off ARM API Review.  Missing to check this box in the following scenario may result in delays to the ARM manifest review and deployment.

Adding a new service
Adding new API(s)
Adding a new API version
-[ ] To review changes efficiently, ensure you are using OpenAPIHub to initialize the PR for adding a new version. More details, refer to the wiki.

[ ] Ensure you've reviewed following guidelines including ARM resource provider contract and REST guidelines. Estimated time (4 hours). This is required before you can request review from ARM API Review board.

[ ] If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist
If any of the following scenarios apply to the PR, request approval from the Breaking Change Review Board as defined in the Breaking Change Policy.

[ ] Removing API(s) in a stable version
[ ] Removing properties in a stable version
[ ] Removing API version(s) in a stable version
[ ] Updating API in a stable or public preview version with Breaking Change Validation errors
[ ] Updating API(s) in public preview over 1 year (refer to Retirement of Previews)

Action: to initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Addition details on the process and office hours are on the Breaking change Wiki.
Please follow the link to find more details on PR review process.

@lmazuel, the only breaking change is our old friend the now-required error property. Could you merge regardless?

Hopefully once this GAs to main this specific issue will be a thing of the past.

@lmazuel, the only breaking change is our old friend the now-required error property. Could you merge regardless?

Hopefully once this GAs to main this specific issue will be a thing of the past.

Hi @lmazuel , can you help us merge it? Thanks!

authors: ChongTang heaths


text:
[Batch] Switch contentLength from int64 to string

This is a breaking change, but accurately reflects what the Batch service has always returned for this property.
Data Plane API Specification Update Pull Request

[!TIP]
Overwhelmed by all this guidance? See the Getting help section at the bottom of this PR description.

PR review workflow diagram
Please understand this diagram before proceeding. It explains how to get your PR approved & merged.

API Info: The Basics
Most of the information about your service should be captured in the issue that serves as your API Spec engagement record.

Link to API Spec engagement record issue:

Is this review for (select one):

[ ] a private preview
[x] a public preview
[ ] GA release

Change Scope
This section will help us focus on the specific parts of your API that are new or have been modified. Please share a link to the design document for the new APIs, a link to the previous API Spec document (if applicable), and the root paths that have been updated.

Design Document:
Previous API Spec Doc:
Updated paths:

Viewing API changes
For convenient view of the API changes made by this PR, refer to the URLs provided in the table
in the Generated ApiView comment added to this PR. You can use ApiView to show API versions diff.
Suppressing failures
If one or multiple validation error/warning suppression(s) is detected in your PR, please follow the
Swagger-Suppression-Process
to get approval.
❔Got questions? Need additional info?? We are here to help!

 Contact us!
The Azure API Review Board is dedicated to helping you create amazing APIs. You can read about our mission and learn more about our process on our wiki.

💬 Teams Channel
💌 email

Click here for links to tools, specs, guidelines & other good stuff
Tooling

Open API validation tools were run on this PR. Go here to see how to fix errors
Spectral Linting

Guidelines & Specifications

Azure REST API Guidelines
OpenAPI Style Guidelines
Azure Breaking Change Policy

Helpful Links

Schedule a data plane REST API spec review

Getting help

First, please carefully read through this PR description, from top to bottom.
If you don't have permissions to remove or add labels to the PR, request write access per aka.ms/azsdk/access#request-access-to-rest-api-or-sdk-repositories
To understand what you must do next to merge this PR, see the Next Steps to Merge comment. It will appear within few minutes of submitting this PR and will continue to be up-to-date with current PR state.
For guidance on fixing this PR CI check failures, see the hyperlinks provided in given failure
and https://aka.ms/ci-fix.
If the PR CI checks appear to be stuck in queued state, please add a comment with contents /azp run.
This should result in a new comment denoting a PR validation pipeline has started and the checks should be updated after few minutes.
If the help provided by the previous points is not enough, post to https://aka.ms/azsdk/support/specreview-channel and link to this PR.

Note that this TypeSpec is currently in a preview state.

/azp run

@raych1 is there anything blocking this from being merged? I'm not sure what the VersionReviewRequired tag is, but if it's regarding making changes to an existing API version, I think we should be fine. The TypeSpec in this PR is in a public preview state, not GA.

/azp run

/azp run


text:
typo: consumption/resource-manager/Microsoft.Consumption

chages -> charges
Cylce -> Cycle

Can one of the admins verify this patch?

Automation for azure-sdk-for-js

Automation for azure-sdk-for-python

Automation for azure-sdk-for-ruby

Automation for azure-sdk-for-node

Automation for azure-sdk-for-go

Automation for azure-sdk-for-java

@sandeepkatumalla could you take a look since you worked on this version last?

I think the build failure is actually a message issue https://github.com/Azure/avocado/issues/11

authors: AutorestCI azuresdkci dsgouda nschonni


text:
Updated ContentMD5 parameter in DataLake swagger

Latest improvements:
MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Contribution checklist:

[ ] I have reviewed the documentation for the workflow.
[ ] Validation tools were run on swagger spec(s) and have all been fixed in this PR.
[ ] The OpenAPI Hub was used for checking validation status and next steps.

ARM API Review Checklist

[ ] Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
adding/removing APIs.
adding/removing properties.
adding/removing API-version.
adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

[ ] If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

Can one of the admins verify this patch?

Automation for azure-sdk-for-python

Automation for azure-sdk-for-go

authors: AutorestCI azuresdkci seanmcc-msft


text:
Managed Instance failover swagger

Latest improvements:
MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Contribution checklist:

[x] I have reviewed the documentation for the workflow.
[x] Validation tools were run on swagger spec(s) and have all been fixed in this PR.
[ ] The OpenAPI Hub was used for checking validation status and next steps.

ARM API Review Checklist

[x] Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
adding/removing APIs.
adding/removing properties.
adding/removing API-version.
adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

[ ] If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

/azp run automation - sdk

Can one of the admins verify this patch?

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run

@MilanBrkicFON Any update on this PR?

@NullMDR Last week changes landed in production, but I didn't have time to update this pr.
Now is ready for review!

/azp run

/azp run automation - sdk

Hi @NullMDR are you ok with the changes? Can we merge this PR?

/azp run

@MilanBrkicFON You need to wait for ARM review. I've emailed them.

/azp run automation - sdk

/azp run

@pilor @allenjzhang Can I have this PR reviewed please?

@pilor @allenjzhang Can I have this PR reviewed please?

API reviews are a weekly rotation described here.  Please reach out to this week or next week's reviewer

@MilanBrkicFON, pretty straightforward PR, though I have noticed consistent pattern issues with SQL swagger files. Please take a look at these easy fixes. Will loop back to work with you to cleanup if these are manually created. If these are generated, please incorporate feedback and make appropriate changes to generate higher quality swagger in the future.
BTW, this is SDK team review. As per policy, ARM folks need to sign off for new APIs. Giving the low complexity of this PR, it should be a quick sign off. Please engage ARM folks per @pilor's instruction.

/azp run automation - sdk

/azp run automation - sdk

@allenjzhang Thanks for the review. I think I've addressed all of your comments. Can you please check again?
I will also update internal docs on ARM development to warn engineers what we are doing wrong and to avoid same mistakes in the future, but I think the right solution here would be working auto-generation script.

/azp run automation - sdk

/azp run

/azp run automation - sdk

/azp run automation - sdk

@pilor @allenjzhang @NullMDR Can someone merge this PR please? I am not authorized to do it.

authors: AutorestCI MilanBrkicFON NullMDR allenjzhang azuresdkci pilor


text:
Investigate telemetry possibly not getting in?

From @kyle-patterson:
"[...] I can see that for storage up through -beta.9, but there's no new data coming in from libraries since then. And for keyvault, I can only see two instances of "azsdk-cpp-keyvault/7.2 {os info...}" , where I would have expected keyvault-keys (and more data coming through...) [...]
It almost seems like there was a change in core in the june or july release, but I haven't found anything yet that seems related...
I did confirm that keyvault is passing the expected value when creating the ttp pipeline: https://github.com/Azure/azure-sdk-for-cpp/blob/83295c69edc5d2a6594c09fcd06eae7073753171/sdk/keyvault/azure-security-keyvault-keys/src/key_client.cpp#L28
it's not hyper-critical, but I'd like to get it resolved in the next release so that we can start getting data again..."

I wrote a simple app that invokes key creation API.
In order to avoid setting up HTTPS decryption on my local machine, I modified the SDK to send requests to a different server on the internet instead of Azure, and read headers there. I compiled with both libcurl and WinHTTP transport adapters. Both have azsdk-cpp-keyvault-keys/7.2 (Windows 10 Enterprise 6.3 19043 19041.1.amd64fre.vb_release.191206-1406) as User-Agent header. (Note that the version does not match the package version, I opened a bug for that - https://github.com/Azure/azure-sdk-for-cpp/issues/2765). But other than that, it look like everything works as it supposed to.

@kyle-patterson can see the telemetry now, I won't post much detail here (not that there's something secret-worthy or sensational was found). Kyle, please reactivate or talk to me if you see problems.

repo: Azure/azure-sdk-for-cpp


text:
Feature request: Core SDK should support HTTP request with request body and non-buffered(stream) response

Currently we have three constructor overloads:

with request body, buffered response
without request body, non-buffered response
without request body, buffered response

Storage SDK needs HTTP request with request body and non-buffered(stream) response to implement Query Blob Content

reopen since it was reverted in https://github.com/Azure/azure-sdk-for-cpp/pull/3033

@vhvb1989 Can you share a timeline for fixing this issue?

@vhvb1989 Can you share a timeline for fixing this issue?

I did in the past with: https://github.com/Azure/azure-sdk-for-cpp/pull/3002
However, I reverted it as it was not yet required from storage. So, feel free to re-apply the changes from that PR together with the Storage feature that it requires it.

OK, that works for me.


text:
mint storage beta 9

Pull Request Checklist
Please leverage this checklist as a reminder to address commonly occurring feedback when submitting a pull request to make sure your PR can be reviewed quickly:
See the detailed list in the contributing guide.

[x] C++ Guidelines
[x] Doxygen docs
[x] Unit tests
[x] No unwanted commits/changes
[x] Descriptive title/description

[x] PR is single purpose
[x] Related issue listed

[x] Comments in source
[x] No typos
[x] Update changelog
[x] Not work-in-progress
[x] External references or docs updated
[x] Self review of PR done
[x] Any breaking changes?

/azp run cpp - storage

/azp run cpp - storage

/azp run cpp - storage


text:
Removed version>= fields for openssl in vcpkg.json files

Remove version>= fields from vcpkg.json files.
Fixes #3703
These fields are broken because:

These fields are illegal in the vcpkg schema because version>= only works with semver version numbers and 1.1.1n is not a legal semver version number.
We rely on the vcpkg baseline to determine which version of openssl we use, currently 3.0.2.

Pull Request Checklist
Please leverage this checklist as a reminder to address commonly occurring feedback when submitting a pull request to make sure your PR can be reviewed quickly:
See the detailed list in the contributing guide.

[X] C++ Guidelines
[X] Doxygen docs
[X] Unit tests
[X] No unwanted commits/changes
[X] Descriptive title/description

[X] PR is single purpose
[X] Related issue listed

[X] Comments in source
[X] No typos
[X] Update changelog
[X] Not work-in-progress
[X] External references or docs updated
[X] Self review of PR done
[X] Any breaking changes?

Would it be best to have version >= 3.0.2 to ensure its at least of that version?

Would it be best to have version >= 3.0.2 to ensure its at least of that version?

A great question, not 100% sure to be honest. We define the version we take from our vcpkg baseline (eng\vcpkg-commit.txt) and we'll pick the version of whatever vcpkg has at that point. As I understand it, the version overrides are there if we specifically require functionality which might be different from the baseline (typically older than the baseline).
If we just want to use whatever the baseline has, we should leave it blank.

authors: LarryOsterman RickWinter


text:
package should be named "avs", not "vmware"

The package should be named "avs", not "vmware". #10454 was merged before I had a chance to review. Please rename the package. See also https://github.com/Azure/sdk-release-request/issues/496.
Current location:
https://github.com/Azure/azure-sdk-for-go/tree/master/services/preview/vmware/mgmt/2019-08-09-preview/vmware
Should be:
https://github.com/Azure/azure-sdk-for-go/tree/master/services/preview/avs/mgmt/2019-08-09-preview/avs

Hi @ctaggart sorry for that... I will change it, release it and remove the wrong-named package in the next major version release (at the end of this month)

Should be resolved in v44.0.0

thank you @ArcturusZhang ! ❤

authors: ArcturusZhang ctaggart

repo: Azure/azure-sdk-for-go


text:
[Tables] Shared Access Signature Capabilities

Adding SAS capabilities to the Tables SDK

[ ] Add Shared Access Signature credential
[ ] SharedAccessSignaturePolicy for signing requests properly
[ ] generate_account_sas method
[ ] generate_table_sas method
[ ] TableSasPermission object
[ ] AccountSasPermissions object
[ ] ResourceTypes object
[ ] AccountSasPermissions object

Python implementation: https://github.com/Azure/azure-sdk-for-python/pull/15946


text:
Sync eng/common directory with azure-sdk-tools for PR 1562

Sync eng/common directory with azure-sdk-tools for PR https://github.com/Azure/azure-sdk-tools/pull/1562  See eng/common workflow

/check-enforcer reset

authors: azure-sdk weshaggard


text:
Fixed bugs in Resource Group demo code

[ ] The purpose of this PR is explained in this or a referenced issue.
[ ] The PR does not update generated files.

These files are managed by the codegen framework at Azure/autorest.go.

[ ] Tests are included and/or updated for code changes.
[ ] Updates to CHANGELOG.md are included.
[ ] MIT license headers are included in each file.

@RickWinter I believe I fixed the indentation.


text:
route table / route / subnet association

Is there a path for managing route table / routes and their subnet association in beta3 ? The SDK points for route table creation, route management and subnet association in beta4 are great and are working perfectly. However, since I am currently required to use released or final code as dependencies, is there any way to manage these same things via beta3 or interaction with an ARM template (don't believe ARM templates allow for update only deploy)

not easily - there may be a workaround via the use of .inner() of the various involved objects, but that approach will get very hairy very quickly....
beta4 is coming out in about 1-2 weeks though, so better just wait for that.

Ahh beta4 is the answer I was hoping for and needed to know a timeline but was not sure where to ask. Thank you very much for the prompt respond, and I believe an ARM template can be used temporarily in the mean time.

authors: martinsawicki paulojohnj

repo: Azure/azure-sdk-for-java


text:
Decouple PagedIterable and PagedFlux to Prevent Background Page Requests

Fixes #15575
This PR decouples some ContinuablePagedIterable implementations from ContinuablePagedFlux when the ContinuablePagedFlux implementation is an instanceof ContinuablePagedFluxCore. This change allows for finer control of page enumeration by removing our dependency on Reactor's toIterable and toStream implementations on a Flux.
Previously, when our implementation used Reactor's toIterable and toStream to implement ContinuablePagedIterable's functionality we would see additional pages getting requested. Calling into these methods we passed a batchSize of 1 which indicated to the backing Flux to only request one element, paged, from upstream at time. But this also served a dual purpose of internal tracking for the backing enumerable to determine when it needed to make additional requests to upstream, so on every next iteration it would hit its internal limit of 1 and make another page request. So, calling ContinuablePagedIterable.streamByPage().findFirst() or ContinuablePagedIterable.iterableByPage().iterator().next() would result in two page requests. Additionally, due to the reactive, event loop, driven nature of the backing enumerable these additional page requests could happen after the execution of the mentioned calling patterns completed.
Now, during construction of ContinuablePagedIterable we will check the instanceof the backing ContinuablePagedFlux and if it is ContinuablePagedFluxCore the PageRetriever and batchSize configuration will be taken from the object and ContinuablePagedIterable will handle page enumeration with finer grain controls. To maintain current functionality that Flux.toStream() has a page will be eagerly requested when a Iterator or Stream is created from the ContinuablePagedIterable. Internally, these will be backed by one of two Iterable implementations, ContinuablePagedByItemIterable or ContinuablePagedByPageIterable. The implementations will only make additional page requests when needed, for page item iterators it will be when the most recently retrieved page has no additional elements and for page iterators it will be once the next page is requested. The page requests will be blocking which is functionally equivalent to the previous experience, where both implementations would throw if called from within a non-blocking reactive thread.
In the future we may investigate completely decoupling ContinuablePagedIterable and ContinuablePagedFlux by having both be constructed with PagedRetriever and handle their own enumeration of pages.

/azp run java - appconfiguration - tests


text:
Added support for encryption algorithms for symmetric keys

Fixes #14805.
Tests are pending.

A few changes required yet, but I'm signing off to unblock. Also, LocalKeyCryptographyClient->LocalCryptographyClient if you haven't GA'd it yet.

We have already GA'd a public LocalCryptographyClient. In the case of LocalKeyCryptographyClient, it's just an abstract class that we extend from for our different internal clients for EC, RSA and AES key operations.


text:
Make the dependency versions the same with SDK Bom managed

As title.

API changes have been detected in com.azure:azure-core. You can review API changes here
API changes
+     public enum ErrorOptions {
+         THROW,
+         NO_THROW;
+     }
+         public RequestOptions setErrorOptions(EnumSet<ErrorOptions> errorOptions)

We decided not to use Azure SDK Bom first, so I close this PR now.

authors: azure-sdk moarychan


text:
STG88 Features

Description
This PR introduces service version support for 2023-01-03, and the following STG88 features:

High Throughput Append Blob for AppendBlobClient
File Share List Handles Access Rights
Add Owner, Group, and Permissions to PathProperties for DataLake

API change check
API changes are not detected in this pull request.

/azp run java - storage - tests

/check-enforcer override

/azp run java - storage - tests

/check-enforcer override

authors: alzimmermsft azure-sdk ibrahimrabab


text:
[AutoPR graphrbac/data-plane] graph: fix invalid types of accountEnabled

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/5873

This PR has been merged into https://github.com/Azure/azure-sdk-for-java/pull/3099


text:
fix429EscapeE2ETimeout

Changes included:

Fixed Issue https://github.com/Azure/azure-sdk-for-java/issues/37419
Issue
429 escaped e2e timeout gating.
Root cause
Currently, the E2ETimeout for point operations is not sitting on top of ClientRetryPolicy, which means for each retry from ClientRetryPolicy will reset the e2e timeout timer.
Fix
For point operations, E2E timeout should be gated for all retries by client retry policy

Fixed Issue https://github.com/Azure/azure-sdk-for-java/issues/37589
Issue
E2E timeout is not being applied for readMany (Query cases)
Fix
Apply E2E timeout for readMany query as well

API change check
API changes are not detected in this pull request.

/azp run java - cosmos - tests


text:
[Automation] Generate Fluent Lite from Swagger mariadb#package-2020-01-01

[Automation] Generate Fluent Lite from Swagger mariadb#package-2020-01-01

/azp run java - mariadb - ci

API change check
APIView has identified API level changes in this PR and created following API reviews.
com.azure.resourcemanager:azure-resourcemanager-mariadb

authors: XiaofeiCao azure-sdk


text:
Release changes for 3.4.1

Changefeed bug fixes (includes all the bug fixes released in 3.3.3)
Disaster Recovery related bug fixes
Exception when Cosmos DB HTTP response header is larger than 8192 bytes: https://github.com/Azure/azure-sdk-for-java/issues/6069
Vulnerability through dependency in the SDK v3.4.0: https://github.com/Azure/azure-sdk-for-java/issues/6433
CosmosSyncScripts null pointer exception in azure-cosmos: https://github.com/Azure/azure-sdk-for-java/issues/6281
Default consistency level parsing for Bounded Staleness and Consistent Prefix: https://github.com/Azure/azure-sdk-for-java/issues/6707
Null Value Holder change: https://github.com/Azure/azure-sdk-for-java/issues/6307

Closing this PR as there are breaking changes with azure-data-sdk-parent v1.2.0 with latest azure-cosmos pom files.


text:
[Cosmos] When are we getting Deno drivers?

Are Cosmos drivers for Deno in the works?  We now have MongoDB https://github.com/denodrivers/deno_mongo and all the others -- but no Cosmos.  Why don't I see any mention of this?   Where are the drivers to use Azure blob storage?  How can one use Deno within Azure without any hooks into the services layer?

#13281 is the issue that we are using to track all discussions around deno support in Azure SDKs
@zfoster, @southpolesteve If you have nothing to add here that will be specific to cosmos, I would recommend moving this conversation to #13281
cc @bterlson, @xirzec, @chradek

Yep that sounds good, thanks! nothing I can think of so closing this one

authors: aadamsx ramya-rao-a zfoster

repo: Azure/azure-sdk-for-js


text:
Sorting arm-consumption 'usage details' by resource group

I am using the arm-consumption package to get the usage details for a subscription and finding it difficult to sort the results in a meaningful way.  I need to get them by resource group, but 1) it does not appear to be possible to filter the details by resource group when getting the data from azure and 2) the 'resource group' property on the usage details items does not correspond directly to the resource group, i.e., if the resource group is called 'ExampleGroup' (as per arm-resources or portal.azure.com), the usage details property 'resource group' is a longer string which appears to begin with the first few letters of the resource group but not in a consistent enough way to use to search.  (There are a larger number of 'resource groups' appearing in arm-consumption than in arm-resources, and their names do not appear to correlate in a consistent, searchable way).
I would like to either be able to use a resource group filter when fetching the usage details, and/or have a property in the usage details that gives the resource group of the item as defined in arm-resources.
I have tried getting all the usage details data and sorting it myself by resource group, however then I encounter the problem where the resource group name that I need to sort by is not a property of the usage details items.
I have also tried looking to see if there is any existing API that provides a map between the arm-resources resource groups names and the arm-consumption resource group names, but have not so far found one.

@BasiaMH I think you can pass a scope parameter to get all the usage details within one resource group https://github.com/Azure/azure-rest-api-specs/blob/main/specification/consumption/resource-manager/Microsoft.Consumption/stable/2021-10-01/examples/UsageDetailsListByManagementGroup.json#L6 . let me know if that works for you ? Thanks

Thanks,
I can pass subscriptions/{subscriptionId} in the scope, which returns all
usage details for our subscription, but if I try to pass the
resourceGroups/{resourceGroupName} I get no results.
What is returned is an array with one item, containing a url
https://costmanagement.trafficmanager.net/subscriptions/c9e88f4c-b9f3-4941-b51e-c9cbbfb9742e/resourcegroups/dicomweb-group/providers/Microsoft.Consumption/usagedetails/Export
And when I navigate to the url it just gives an error message:
Either: "The resource you are looking for has been removed, had its name
changed, or is temporarily unavailable."
or:
{"message":"No HTTP resource was found that matches the request URI
'https://costmanagement.trafficmanager.net/subscriptions/c9e88f4c-b9f3-4941-b51e-c9cbbfb9742e/resourcegroups/dicomweb-group/providers/Microsoft.Consumption/usagedetails/Export'.","messageDetail":"No
route providing a controller name was found to match request URI
'https://costmanagement.trafficmanager.net/subscriptions/c9e88f4c-b9f3-4941-b51e-c9cbbfb9742e/resourcegroups/dicomweb-group/providers/Microsoft.Consumption/usagedetails/Export'"}
Basia
On Fri, Jan 14, 2022 at 3:22 AM Qiaoqiao Zhang @.***>
wrote:

@BasiaMH https://github.com/BasiaMH I think you can pass a scope
parameter to get all the usage details within one resource group
https://github.com/Azure/azure-rest-api-specs/blob/main/specification/consumption/resource-manager/Microsoft.Consumption/stable/2021-10-01/examples/UsageDetailsListByManagementGroup.json#L6
. let me know if that works for you ? Thanks
—
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-sdk-for-js/issues/19835#issuecomment-1012906456,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFJXFDSC4B3G53GBYR3R4MDUV7MLNANCNFSM5L343HJQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID:
@.***>

@BasiaMH Did you pass the scope like /subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName} ?

That's exactly what I tried, yes.
(Before that I tried including resourceGroups in a filter, but that just
returned all the data unfiltered.)
On Sun., Jan. 16, 2022, 8:52 p.m. Qiaoqiao Zhang, @.***>
wrote:

@BasiaMH https://github.com/BasiaMH Did you pass the scope like
/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName} ?
—
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-sdk-for-js/issues/19835#issuecomment-1014065383,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFJXFDWW4TZVRKOVU5UWZFLUWNY4DANCNFSM5L343HJQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID:
@.***>

Looks like we need some help from service team.
@mitagarg Is it possible that you could help here, as I see you are the author of this PR https://github.com/Azure/azure-rest-api-specs/pull/17013

Hi, is there any update or further information on this problem?  Or a different workaround someone could suggest to somehow get the usage and cost per resource or per resource group?
Thanks

@BasiaMH Hi, Since this is probably an issue on the service side, Could you open a ticket here https://docs.microsoft.com/en-us/answers/topics/azure-cost-management.html so that you can get direct support from the service team ?

authors: BasiaMH qiaozha


text:
[Event Hubs] Re-visit all parameter validations

Relevant sections from General guidelines:

The client library will have a client object with several methods that call methods on the service. Service parameters are directly passed across the wire to an Azure service. Client parameters are not passed directly to the service, but used within the SDK to fulfill the request. Examples of client parameters include values that are used to construct a URI, or a file that needs to be uploaded to storage.
✅ DO validate client parameters.
⛔ DO NOT validate service parameters. This includes null checks, empty strings, and other common validating conditions. Let the service validate any request parameters.
✅ DO validate the developer experience when the service parameters are invalid to ensure appropriate error messages are generated by the service. If the developer experience is compromised due to service-side error messages, work with the service team to correct prior to release.

Relevant sections from Typescript guidelines:

YOU SHOULD coerce incorrect types into an appropriate type, if possible. JavaScript users expect some amount of fuzziness with parameters as the standard library tends to coerce types if possible. TypeScript users should get pedantic types as they have opted in to types and expect errors.

Relevant work done in Service Bus regarding this: https://github.com/Azure/azure-sdk-for-js/issues/1145#issuecomment-481893308
Please use the parameter validations done in Service Bus as a reference

Main points to be kept in mind for parameter validations

Missing mandatory arguments should throw an error.
If expected type for an argument is string, use String() to convert given value to string. This is to ensure that we do our best to coerce given input to string for our Javascript users
If expected type for an argument is an array, given value is not an array, then make an array out of the given value
If given value doesnt match expected type and this results in us not being able to form a request to the service, then throw error.

For reference, see what was done for Service Bus

Done with #3652


text:
Propagate changes to work around DNS resolution issue in Linux

See: https://github.com/Azure/azure-sdk-for-java/pull/10576/files
We've seen issues where DNS resolution fails in Ubuntu 18. This workaround fixes that problem. This only needs to run on Ubuntu 18 instances.

@chradek, @richardpark-msft Any chance this can solve our dns woes with linux in SB and EH?

Closing this issue. We already have this integrated with a step to bypass local DNS using common template.


text:
Bertong sms impl

This draft PR is to solicit feedback, primarily on whether what i'm testing is appropriate and consistent with what we hve done

/azp run python - communication - tests

/azp run js - communication - tests

/check-enforcer reset

/azp run js - communication-sms - tests

/azp run js - communication-sms - tests

/azp run js - communication-sms - tests

/azp run js - communication-sms - tests

/azp run js - communication-sms - tests

/azp run js - communication-sms - tests


text:
push communication-email recordings

fix https://github.com/Azure/azure-sdk-for-js/issues/28656

API change check
API changes are not detected in this pull request.


text:
Post release automated changes for databoundaries releases

Post release automated changes for azure-arm-databoundaries

API change check
API changes are not detected in this pull request.


text:
[core-amqp] Fixed a build error on master

Seems like a fitting solution for a problem we're having on master.
The error is visible here: https://dev.azure.com/azure-sdk/public/_build/results?buildId=255040&view=logs&j=23e2e6de-b7c3-5918-7121-f16b46172e49&t=9c960e92-0110-52f6-5b54-96f3b7fea4ea&l=697

This isn't necessary. My master was outdated.


text:
[BUG] Memory Leaking When SDK Combined With Application Insights

Describe the bug
When using the Azure.Storage.Queues (v 12.3.0) NuGet package in combination with Microsoft.ApplicationInsights.AspNetCore (v 2.13.1) in a .Net core 3.1 application that is constantly polling queues for new messages, we find that the number of objects built up in memory is steadily increasing, specifically:
Microsoft.ApplicationInsights.DependencyCollector.Implementation.AzureSdkDiagnosticsEventHandler - creating about 5000 objects a minute and the garbage collector is not disposing them.
Expected behavior
That the AI objects would be cleaned up by the GC
Actual behavior (include Exception or Stack Trace)
Eventually the app will start throwing OutOfMemoryExceptions.
To Reproduce

Have the following in your app startup:

        public void ConfigureServices(IServiceCollection services)
        {
            services.AddApplicationInsightsTelemetry();
        }

Add a hosted service:

            var host = new WebHostBuilder()
                .UseKestrel()
                .ConfigureServices(c => c.AddHostedService<QueueWorker>())

Have the worker constantly new up a QueueClient to check for queue messages.

Environment:

Azure.Storage.Queues 12.3.0
.NET runtime version 3.1.101
IDE and version : Visual Studio Professional v 16.4.5

For step 3, why not have the worker re-use the same QueueClient?  The clients are thread-safe.

That has proven to be the workaround we have gone with - storing any queue clients that we have created in a static dictionary that remains for the lifetime of the app (we have a web app that uses this shared code and would be newing up the clients pretty heavily to create queue messages).
I will leave it open and for others to decide if this is a bug that needs fixing or if it was simply of a case of us "using it wrong" in which case it can be closed.

We're tracking getting this documented for all clients in this issue. https://github.com/Azure/azure-sdk-for-net/issues/8941.

Hi there
We understand the recommended way is to use a single client though isn't still a bug having a memory leak because many clients are created?
In our specific case, we created a lot of clients for an hour or so at night and the memory is never release even days after.
We will use a single client in the future, so thanks to @speedy-ms
Thanks
Phil

authors: Petermarcu jesuissur seanmcc-msft speedy-ms

repo: Azure/azure-sdk-for-net


text:
[QUERY] Azure .NET SDK for web app/function app creation and deployment

Library name and version
Microsoft.Azure.Management.AppService
Query/Question
Hi,
I would like to do the below commands from the CLI in the SDK instead.

$'az functionapp create -p {_appPlanName} --name {_globalFunctionName} --resource-group {_resourceGroupName} --runtime dotnet --storage-account {_storageAccountName} --functions-version 4'

$'az functionapp deployment source config-zip -g {_resourceGroupName} -n {_globalFunctionName} --src {_zipName}'

Looks like there's SDK Microsoft.Azure.Management.AppService could create and deploy web app, but there's no example for this. Could you please share some example for C# SDK creation of Azure function, SDK deploy of Azure function and SDK deploy of web app
Thanks
Environment
No response

function app is one kind of web site.
Here is an example of web site creation. Need to set Kind as functionapp to create a function app, and you might experiment more if you can fit the settings for function app into WebSiteData
I did not find any SDK regarding the deployment though.

It requires details of the service itself, so it would be best to involve the service team to provide a sample of the usage.

@jsquire Could you please help route this issue to service team for further assistance?

@live1206 : Service teams do not generally provide samples based on our SDKs.  This is something that our team would own and coordinate with the service team directly, if needed.    Please work with @ArthurMa1978 to determine next steps.

Hi @Arvinorange.  Unfortunately, this repository is focused on the Azure SDK for .NET, service teams do not monitor these issues.  To involve the service team, your best path forward would be to open an Azure support request.

@live1206 : Service teams do not generally provide samples based on our SDKs. This is something that our team would own and coordinate with the service team directly, if needed. Please work with @ArthurMa1978 to determine next steps.

@jsquire AFAIK, The sample of how to use SDK for a specific scenario is auto-generated based on examples in Spec repo.
The service team need to provide us a detailed example of the request and response how to create a function app.
If we take a look at the example of CreateOrUpdateStaticSiteBuildFunctionAppSettings, it generated a sample CreateOrUpdateAppSettings_CreatesOrUpdatesTheFunctionAppSettingsOfAStaticSiteBuild, which does not provide any details regarding how to create a function app with .NET SDK.
I have checked with @ArthurMa1978 we can't do anything here if the service team doesn't provide the scenario example with details.

authors: Arvinorange jsquire live1206


text:
[BUG] Azure.Core.ResponseOfT contains disposable Response

Describe the bug
Response<T> contains a disposable Response property. There is no way to dispose of the inner Response object. This is a memory leak and will cause additional overhead for garbage collection.
Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[x] Bug Description Added
[x] Repro Steps Added
[x] Setup information Added

@pakrym this is related to the recent PR

Not disposing a reference doesn't cause a memory leak as long as it doesn't hold any common/unmanaged resources without having a finalizer.
In the case of the Response the only part that holds onto shared resource is the content stream. It also depends if response buffering is enabled:

In the case where buffering is enabled the process of buffering itself would dispose the underlying stream and avoid resource leak, buffered stream is just a memory stream and can be collected.
In the case where buffering is disabled, it's clients responsibility to either dispose Response it got from the pipeline or return Response where T is IDisposable and consumer of the client SDK has to dispose the value.

In addition not disposing something doesn't add any GC overhead because no matter if an object is disposed GC still has to walk the graph and collect it.

Response ContentStream is a Stream. There is no guarantee that it is a MemoryStream. Does Response need to be updated to be MemoryStream instead of Stream?
How will users know that there is inner property that needs to be disposed of?
Would a better solution be to use Memory or Span instead of Stream? It would remove all the dispose logic and it would give better performance.

The response should be able to handle both buffered and streamed responses, Memory or MemoryStream would only work for buffered responses.
We know that Response.ContentStream would be MemoryStream for buffered responses because we control the buffering logic and create the stream.

Any reason not to make ChannelEnumerableSubscription itself IAsyncEnumerator ?

My issues with this is it is dependent on a implementation detail, and not the contract. If someone changes that implementation in the future then this will be broken. There is also no way for users to know if they need to dispose of the inner stream or not to dispose of it.
What scenario requires supporting stream? Would users actually get better use out of always having a buffered Memory object?

My issues with this is it is dependent on a implementation detail, and not the contract. If someone changes that implementation in the future then this will be broken. There is also no way for users to know if they need to dispose of the inner stream or not to dispose of it.

Users never have to dispose the inner stream. Users only need to dispose something when the return type is Response then they have to dispose the response.Value.

What scenario requires supporting stream? Would users actually get better use out of always having a buffered Memory object?

Downloading large files from blob storage, for example.

Users never have to dispose the inner stream. Users only need to dispose something when the return type is Response then they have to dispose the response.Value.

I'm a little confused here. Response<T>  has an inner Response object that is disposable. If Response  contains a network or some other Stream type that needs to be disposed of how is it getting disposed of?

I'm a little confused here. Response has an inner Response object that is disposable. If Response contains a network stream or some other stream type that needs to be disposed of how is it getting disposed of?

We are controlling the clients and the pipeline and wouldn't return Response that has a network stream in it.


text:
Enable a way to Unregister Message Handler and Session Handler

Currently our SDK doesn't support a way to unregister message handler and session handler so customers are getting ObjectDisposedException exceptions when receivers are closed due to connection closed caused by regular application upgrades etc. Exposing this functionality to unregister handler and await for pending receives and message handling operations to finish to allow for a graceful unregister on the handler they previously registered. Customer are expected to call this function before they close down their handler. Customer can register again after unregister as they are independent operations.

Is there an issue that discusses this change? Given this is a new feature I'd expect to see some transparency about it.
/cc @JoshLove-msft @jsquire

@SeanFeldman Hi Sean! Thanks for your comments. This is only a draft PR. This is a feature request from Azure Functions team. I will create an issue to discuss this to allow more transparency in design.

I have validated a version of these changes (commit "fbd31d0") with Azure Functions Service Bus extension and did not find any issues.  Tested the following methods:

QueueClient.UnregisterSessionHandlerAsync(TimeSpan)
SubscriptionClient.UnregisterSessionHandlerAsync(TimeSpan)
3 MessageReceiver.UnregisterMessageHandlerAsync(TimeSpan)

I also attempted to call SubscriptionClient.RegisterSessionHandler() while SubscriptionClient.UnregisterSessionHandlerAsync(TimeSpan) was processing and got an exception as expected.  Calling RegisterSessionHandler() after UnregisterSessionHandlerAsync(TimeSpan) completed worked normally as expected.
Nice work @DorothySun216!  Approving from my side.

used version: 4.2.1
I want to use UnregisterMessageHandler to "pause" the ISubscriptionClient. aka stop processing messages from the Service Bus for a while. The only problem I encounter, is that after I unregister and try to re-register a handler, the runningTaskCancellationTokenSource in MessageReceiver.cs does not get a new instance, but keeps using the disposed one. That is why I get the following exception:
System.ObjectDisposedException: The CancellationTokenSource has been disposed.
Suggestion to also re-instantiate the runningTaskCancellationTokenSource just like the receivePumpCancellationTokenSource in MessageReceiver.cs (line 1333).

used version: 4.2.1
I want to use UnregisterMessageHandler to "pause" the ISubscriptionClient. aka stop processing messages from the Service Bus for a while. The only problem I encounter, is that after I unregister and try to re-register a handler, the runningTaskCancellationTokenSource in MessageReceiver.cs does not get a new instance, but keeps using the disposed one. That is why I get the following exception:
System.ObjectDisposedException: The CancellationTokenSource has been disposed.
Suggestion to also re-instantiate the runningTaskCancellationTokenSource just like the receivePumpCancellationTokenSource in MessageReceiver.cs (line 1333).

@RicoBakels thanks for reaching out. Are you using our service bus SDK directly or using Azure Functions?

@RicoBakels thanks for reaching out. Are you using our service bus SDK directly or using Azure Functions?

@DorothySun216 I am using the service bus SDK

@DorothySun216 I am using the service bus SDK

@RicoBakels I see. I will work on a repro. So you just register a handler and then unregister and then register again, and during this time the exception occurred? Can you show a code snippet for your use case if it is convenient?

@RicoBakels I see. I will work on a repro. So you just register a handler and then unregister and then register again, and during this time the exception occurred? Can you show a code snippet for your use case if it is convenient?

@DorothySun216  Thanks for the fast reply. Here is a code snipped where I use the ISubscriptionClient in my class.
At startup of the (Service Fabric) service, The message handler will get registered with the OpenAsync method. When for some reason the service needs to stop handling messages from the Service Bus for a while, the service needs to pause the handling of messages by calling the Pause method.
Then when the service needs to start handling messages again, I call the OpenAsync method again to register the handler.
// (re-)register message handler.
public Task<string> OpenAsync(CancellationToken cancellationToken)
{
    _logger.LogInformation("AzureServiceBusReaderService starting");
    var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandlerAsync)
    {
        MaxConcurrentCalls = 1,
        AutoComplete = false
    };
    _subscriptionClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
    return Task.FromResult(_subscriptionClient.SubscriptionName);
}

// Unregister message handler so messages won't get picket up from Service Bus.
public void Pause() => _subscriptionClient.UnregisterMessageHandlerAsync(TimeSpan.FromSeconds(1));

@DorothySun216  Thanks for the fast reply. Here is a code snipped where I use the ISubscriptionClient in my class.
At startup of the (Service Fabric) service, The message handler will get registered with the OpenAsync method. When for some reason the service needs to stop handling messages from the Service Bus for a while, the service needs to pause the handling of messages by calling the Pause method.
Then when the service needs to start handling messages again, I call the OpenAsync method again to register the handler.
// (re-)register message handler.
public Task<string> OpenAsync(CancellationToken cancellationToken)
{
    _logger.LogInformation("AzureServiceBusReaderService starting");
    var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandlerAsync)
    {
        MaxConcurrentCalls = 1,
        AutoComplete = false
    };
    _subscriptionClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
    return Task.FromResult(_subscriptionClient.SubscriptionName);
}

// Unregister message handler so messages won't get picket up from Service Bus.
public void Pause() => _subscriptionClient.UnregisterMessageHandlerAsync(TimeSpan.FromSeconds(1));

authors: DorothySun216 RicoBakels SeanFeldman sidkri


text:
Created wrapper class which contains information to rehydrate

Moved around dependencies, job plan models to create Checkpointer internally as well.

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.Storage.DataMovement

authors: amnguye azure-sdk


text:
Increment version for storage releases

Increment package version after release of Azure.Storage.Common

API change check
API changes are not detected in this pull request.


text:
[ACS][CallAutomation]Move SourceCallerIdNumber from Answer to Transfer api

Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

Run autorest to updated a lot models include media teams work
In CallMedia.cs, StartHoldMusicAsync and StartHoldMusic is updated to pass the build and left comment for media team to work on the StartHoldMusicOptions.Loop and operationCallbackUri
Move SourceCallerIdNumber from Answer to Transfer api

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.Communication.CallAutomation


text:
[HDInsight on AKS] Api version 2024-05-01 support

Contributing to the Azure SDK
Support api version 2024-05-01.
Preview to Stable.
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.ResourceManager.HDInsight.Containers

The GA plan has changed. Stop this thread. Thanks.

authors: azure-sdk yuc-Li


text:
AZD-CM integration

Added the ability to provision CDK CM infrastructure using azd.
Steps to try it:

Execute the CloudMachineTests.Configure (this will create bicep files from the CM CDK)
In the test's bin folder, run azd init
pick "Minimal" template
you can now do azd provision

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.Provisioning.CloudMachine

authors: KrzysztofCwalina azure-sdk


text:
[event hubs] support receiver redirect

Background
The Event Hubs service supports redirects for amqp receiver links. When the client creates an amqp receiver link with redirects enabled, the service will respond with an amqp:link:redirect error if it also supports redirects. The error contains the information needed to create a new connection to the service.
The main benefit of receiver redirect is it allows a more direct connection to the host actually sending events to the client which should reduce latency.
API changes
The Event Hub clients responsible for consuming events (e.g. EventHubConsumerClient) should accept a new option to enable redirect that has a default value of false.
Proposal:
  { enableRedirect: true } // new option 'enableRedirect'

Reasoning
It is important to make this option opt-in for 2 reasons:

Currently 1 client creates 1 connection. This changes this assumption to 1 client can create many connections. This has implications since there is a limit to the number of connections that can be open at one time to an Event hub.
By default the SDK speaks AMQP over ports 5671/5672. With redirect enabled, new connections may be made on ports 104xx. Users may need to update their firewall rules to support redirect:
https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-protocol-guide#amqp-outbound-port-requirements

Details
Redirect walkthrough

User instantiates a consumer client with redirect enabled.
User makes call to receive events (this causes a receiver link to be created using customer-provided credential/host info).
Connection/cbs session/receiver link are created.

The receiver link must have amqp:link:redirect included in the list of desired_capabilities if redirect is enabled.
This must be omitted if redirect is not enabled.

The receiver link recieves a LinkRedirectError.
The SDK parses the hostname/port/address info from the error.
A new amqp connection is created using the hostname/port from the error.
When the cbs session is created, the applicationProperties.name value should be the address extracted from the LinkRedirectError.
The SDK creates the receiver link on the new connection and begins receiving events.

Considerations
It is possible for every receiver link to have different hostname/port/address values from one another in the LinkRedirectError. You may create a new connection for every LinkRedirectError. Optionally, you may use a connection for multiple receiver links if the links share a hostname/port.
Creating a new connection for each redirect is simpler to implement because it is easier to know when the connection should be closed, but does potentially increase connection density.
When to re-use the original connection?
It is possible that the server node used for receiving events can change. When the SDK sees an amqp:connection:forced error from the service, or the connection is closed, then the original connection should be used to create a new receiver link. At this point a new LinkRedirectError should be received and the redirect walkthrough can be followed.
Otherwise, for transient issues the connection created for the link after a LinkRedirectError can be re-used.
Reference
There is a draft PR for the JavaScript SDK that has the basics implemented for this feature.
https://github.com/Azure/azure-sdk-for-js/pull/7782

goal for April: design

authors: chradek yunhaoling

repo: Azure/azure-sdk-for-python


text:
azure.storage.blob.BlobClient.from_blob_url() parses incorrect blob name and container name

Package Name: azure-storage-blob
Package Version: 12.3.2
Operating System: Windows 10 Version 10.0.19041 Build 19041, running WSL 1
Python Version: 3.8.3

Describe the bug
azure.storage.blob.BlobClient.get_blob_properties() returns incorrect "name" and "container" properties
To Reproduce
Steps to reproduce the behavior:
Code
from azure.storage.blob import BlobClient

PUBLIC_BLOB_URI = 'https://lilablobssc.blob.core.windows.net/nacti-unzipped/part0/sub000/2010_Unit150_Ivan097_img0003.jpg'

blob_client = BlobClient.from_blob_url(PUBLIC_BLOB_URI)
properties = blob_client.get_blob_properties()
print(properties['name'])
print(properties['container'])

Running Code in Python REPL:
>>> from azure.storage.blob import BlobClient
>>>
>>> PUBLIC_BLOB_URI = 'https://lilablobssc.blob.core.windows.net/nacti-unzipped/part0/sub000/2010_Unit150_Ivan097_img0003.jpg'
>>>
>>> blob_client = BlobClient.from_blob_url(PUBLIC_BLOB_URI)
>>> properties = blob_client.get_blob_properties()
>>> print(properties['name'])
2010_Unit150_Ivan097_img0003.jpg
>>> print(properties['container'])
sub000

Expected behavior
A clear and concise description of what you expected to happen.
The blob name should be "part0/sub000/2010_Unit150_Ivan097_img0003.jpg" instead of "2010_Unit150_Ivan097_img0003.jpg" and the container should be "nacti-unzipped" instead of "sub000".

Update: the issue is actually in azure.storage.blob.BlobClient.from_blob_url().
from azure.storage.blob import BlobClient

PUBLIC_BLOB_URI = 'https://lilablobssc.blob.core.windows.net/nacti-unzipped/part0/sub000/2010_Unit150_Ivan097_img0003.jpg'

blob_client = BlobClient.from_blob_url(PUBLIC_BLOB_URI)
print(blob_client.blob_name)
print(blob_client.container_name)

Running Code in Python REPL:
>>> from azure.storage.blob import BlobClient
>>>
>>> PUBLIC_BLOB_URI = 'https://lilablobssc.blob.core.windows.net/nacti-unzipped/part0/sub000/2010_Unit150_Ivan097_img0003.jpg'
>>>
>>> blob_client = BlobClient.from_blob_url(PUBLIC_BLOB_URI)
>>> print(blob_client.blob_name)
'2010_Unit150_Ivan097_img0003.jpg'
>>> print(blob_client.container_name)
'sub000'

Hi @chrisyeh96
Thanks for reporting this and post that pr. It's the SDKs problem with parsing blob url with sub directories! Sorry for the inconvenience.

the fix has been released, you can do pip install azure-storage-blob==12.4.0 to use it.


text:
[API Management Management] Impossible to create new API using sdk

Package Name: Azure management Api Management
Package Version: 1.0.0
Operating System: Windows
Python Version: 3.7

Describe the bug
It seems as the api management client has a bug when trying to create or update an API. The package tries to poll for a result but includes uneccessary data in the request which throws an error in the aiohttp package
To Reproduce
Steps to reproduce the behavior:

Create a client for interacting with API management

creds = DefaultAzureCredential()
client = ApiManagementClient(
    credential=creds, subscription_id=settings.subscription_id
)

create a version set

api_version_set=client.api_version_set.create_or_update(
    resource_group_name=settings.resource_group_name,
    service_name=settings.api_management_name,
    version_set_id="anystuff",
    parameters=ApiVersionSetContract(
        display_name="title",
        versioning_scheme="Segment",
        description="Version configuration",
    ),
)

And then create any form of API.

api=client.api.begin_create_or_update(
    resource_group_name=settings.resource_group_name,
    service_name=settings.api_management_name,
    api_id="wss-mp-test-order",
    parameters=get_parameters(oai, api_version_set.id),
)

This throws an error:
return _RequestContextManager(self._request(method, url, **kwargs))
TypeError: _request() got an unexpected keyword argument 'path_format_arguments'

which, as far as I can see, results from these lines in the sdk:
https://github.com/Azure/azure-sdk-for-python/blob/79a64bb9d40610d1696819a2595a49d18d7a7ad6/sdk/apimanagement/azure-mgmt-apimanagement/azure/mgmt/apimanagement/aio/operations/_api_operations.py#L420-L427
where the path_format_arguments are interpreted in the end as parameters to the GET request polling for a result.
Expected behavior
A created API and successful response in the code
Screenshots
N/A
Additional context
I had to upgrade from 0.2.0 because of a different set of bugs (not reported yet, but at least one still persist). Now everything broke, even the things that I had a workaround for before.

Hi @ludokriss, this version of SDK is using a preview-version of API.
It seems that something have changed in service side.
We will update it in the near future.

@00Kai0 any updates?


text:
Incorrect requirement for azure.mgmt.compute

It seems that the version requirement for azure-common~=1.1 in azure.mgmt.compute is too broad: new versions of the package reference azure.profiles, which was introduced in azure-common 1.1.9.
For example, if I try to install the latest version (4.0.0rc2) of azure.mgmt.common in an environment which already has a previous, older version of azure-common, the installation completes successfully but I can't import anything from the new package.
Steps to reproduce:

From a clean environment, install an older version of azure-common, then install the latest azure.mgmt.compute from source.

pip install "azure-common<=1.1.8"
git clone https://github.com/Azure/azure-sdk-for-python.git
pip install azure-sdk-for-python/azure-mgmt-compute

In a Python console, import the new package.

>>> import azure.mgmt.compute
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ageorgou/anaconda/envs/test-azure/lib/python3.6/site-packages/azure/mgmt/compute/__init__.py", line 8, in <module>
    from .compute_management_client import ComputeManagementClient
  File "/Users/ageorgou/anaconda/envs/test-azure/lib/python3.6/site-packages/azure/mgmt/compute/compute_management_client.py", line 16, in <module>
    from azure.profiles import KnownProfiles, ProfileDefinition
ModuleNotFoundError: No module named 'azure.profiles'

Expected behaviour is for the import to work, as it successfully does with newer (>=1.1.9) versions of azure-common.

Hi @ageorgou
Yes you're right, I should write ~=1.1;>=1.1.9. I didn't notice since I do fresh env each time, but on update it won't work. Will fix it asap. Thanks!

Hi @lmazuel,
Yes, I only noticed this when updating an existing installation.
I was looking at this again and realised that it happens in other packages too. A quick search (grep "azure.profiles") shows:

azure-mgmt-compute
azure-mgmt-resource
azure-mgmt-storage
azure-mgmt-network
azure-mgmt-containerregistry

The last two already require specific versions. I have submitted a PR for the other three in case that makes things easier, but please feel free to ignore it if not!

Hi @ageorgou
I released storage this morning including the fix, so your PR conflicts :/. If you could remove storage from your PR, I'll merge it.
Thanks!

Hi @lmazuel, done now.

authors: ageorgou lmazuel