text:
Stack Self Drifts

Describe the bug
The template has a Lambda Backed Custom Resource (CustomControlTowerDeploymentLambda) that modifies the attribute of the S3 Bucket (CustomControlTowerPipelineS3Bucket) created by the same stack. Thus, causing a configuration drift and potential alerts and bigger issues. Is there any way to avoid that? Perhaps avoid custom resources that configure stack managed resources? Thx
To Reproduce
Launch the Stack, check drift.
Expected behavior
Expected behaviour is to have no configuration drifts after stack creation. This could cause other issues if we had any mechanisms to revert drifts automatically.
Please complete the following information about the solution:

[x] Version: v2.2.0
[x] Region: eu-west-1
[x] Was the solution modified from the version published on this repository? NO
[x] If the answer to the previous question was yes, are the changes available on GitHub? NO
[x] Have you checked your service quotas for the sevices this solution uses? N/A
[x] Were there any errors in the CloudWatch Logs? N/A

Screenshots
If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).

Additional context
Add any other context about the problem here.

Hey @felipeamarante, thanks for bringing this to our attention, I've opened a backlog with the team to tackle this issue.

authors: balltrev felipeamarante


text:
Fixed bucket does not exist error

Fixed issue #103
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

hi @codejamninja this will break a lot of deployments :D
The ~AWS_REGION that you removed is required as the WAF needs to be deployed in every region you want to use it when you want to use it with ALBs, Regional APIs.

Close the old ticket. It doesn't apply to the latest version any more.

source: gharchive/pull-request

authors: aijunpeng aldegoeij clayrisser

license_type: permissive

license_source: github-api


text:
Error: Downgrade or same version not allowed

Hello everyone,
I've been working in this for some weeks, I've followed the official AWS OTA Update tutorial and AWS Getting Started with FreeRTOS. I succeeded and my ESP32 was working with the OTA Demo running on it, but the problem comes when executing an OTA job.
To update the firmware, I do the following steps:

Change code in aws_iot_ota_update_demo.c (for example a comment or the MQTT topic to publish to, definitely not relevant lines that could involve malfunction in the OTA Agent)
Increment variable APP_VERSION_BUILD in file aws_application_version.h
Generate build files again using

cmake -DVENDOR=espressif -DBOARD=esp32_wrover_kit -DCOMPILER=xtensa-esp32 -S . -B build/

Build files inside build/ directory

make all -j4

Upload newly generated aws_demos.bin inside build folder to my AWS S3 bucket
Execute OTA job from CLI using the following JSON file (changing the version of the S3 object with the new one):

aws iot create-ota-update --cli-input-json file://cli_ota_update_job.json

{
    "otaUpdateId": "otajob1",
    "description": "This is a test job",
    "targets": [
      "arn:aws:iot:eu-west-1:XXXXXXXXXXXX:thing/Node2"
    ],
    "protocols": [
      "MQTT"
    ],
    "targetSelection": "SNAPSHOT",
    "awsJobExecutionsRolloutConfig": {
        "maximumPerMinute": 1000
    },
    "awsJobPresignedUrlConfig": { },
    "files": [
      {
        "fileName": "/",
        "fileLocation": {
            "s3Location": {
                "bucket": "ota-updates-tfm",
                "key": "aws_demos.bin",
                "version": NEW_VERSION_STRING
            }
        },
        "codeSigning": {
            "startSigningJobParameter": {
                "signingProfileParameter": {},
                "signingProfileName": "profile_esp32_2",
                "destination": {
                    "s3Destination": {
                        "bucket": "ota-updates-tfm",
                        "prefix": "SignedImages/"
                    }
                }
            }
        }
      }
    ],
    "roleArn": "arn:aws:iam::XXXXXXXXXXXX:role/OTA_Service_Role"
}

The ESP32 gets the job correctly, but when updating I get the following error message before rebooting to previous version:

53 1220 [OTA Agent Task] [prvValidateUpdateVersion] We rebooted and the version is still the same.
54 1220 [OTA Agent Task] [prvParseJobDoc] Downgrade or same version not allowed, rejecting the update & rebooting.
55 1224 [OTA Agent Task] [prvPublishStatusMessage] Msg: {"status":"FAILED","statusDetails":{"reason":"rejected: 0x23000000"}}

System information

Hardware board: ESP32-DevKitC
Operating System: Ubuntu 20.04
Code version: FreeRTOS version 202012.00, downloaded directly from AWS FreeRTOS console
Project/Demo: ota demo, but modified using this code mentioned in issue #2569 as I wanted to use OTA Agent demo and publish to MQTT topic using same connection

Expected behavior
I expected to have my ESP32 updated with the new firmware as I updated the version in the aws_application_version.h file.
Additional context
I don't know if it would be relevant, but if I flash directly to the board, the new version is recognized as expected and do not reboot.

Hello @Carfer10 ,
Can you please check in the logs after the image is received and the device reboots what the version number is ? It will be great if you can share the logs.

Hi @pvyawaha,
I can't share it right now, but the last time I checked the version received and the version when the device reboots. Both are the same.
This is not the case when I flash the device directly, the version received is the new one and it doesn't reboot.
Tomorrow I'll be able to share the logs

Hello @Carfer10 ,
Can you please check in the logs after the image is received and the device reboots what the version number is ? It will be great if you can share the logs.

This is the change log after the ota update is received and until the device reboots:
4826 95722 [OTA Agent Task] [prvIngestDataBlock] Received file block 273, size 4096
4827 95722 [OTA Agent Task] [prvIngestDataBlock] Remaining: 1
4828 95722 [OTA Agent Task] [prvExecuteHandler] Called handler. Current State [WaitingForFileBlock] Event [ReceivedFileBlock] New state [WaitingForFileBlock]
4829 95724 [OTA Agent Task] [INFO ][MQTT][957240] (MQTT connection 0x3ffb9cd0) MQTT PUBLISH operation queued.
4830 95724 [OTA Agent Task] [prvRequestFileBlock_Mqtt] OK: $aws/things/Node2/streams/AFR_OTA-e4480498-43e8-4d6d-8315-77db3a813e24/get/cbor
4831 95724 [OTA Agent Task] [prvExecuteHandler] Called handler. Current State [WaitingForFileBlock] Event [RequestFileBlock] New state [WaitingForFileBlock]
4832 95742 [OTA Agent Task] [prvIngestDataBlock] Received file block 274, size 3824
4833 95742 [OTA Agent Task] [prvIngestDataBlock] Received final expected block of file.
4834 95742 [OTA Agent Task] [prvStopRequestTimer] Stopping request timer.
I (957497) ota_pal: No such certificate file: /. Using aws_ota_codesigner_certificate.h.

4835 95776 [iot_thread] State: WaitingForFileBlock  Received: 277   Queued: 0   Processed: 0   Dropped: 0
4836 95779 [iot_thread] [INFO ][MQTT][957790] (MQTT connection 0x3ffb9cd0) MQTT PUBLISH operation queued.
4837 95779 [iot_thread] [INFO ][MQTT][957790] (MQTT connection 0x3ffb9cd0, PUBLISH operation 0x3ffba01c) Waiting for operation completion.
4838 95916 [OTA Agent Task] [prvIngestDataBlock] File receive complete and signature is valid.
4839 95916 [OTA Agent Task] [prvStopRequestTimer] Stopping request timer.
4840 95916 [OTA Agent Task] [prvPublishStatusMessage] Msg: {"status":"IN_PROGRESS","statusDetails":{"self_test":"ready","updatedBy":"0x90012"}}
4841 95917 [OTA Agent Task] [INFO ][MQTT][959170] (MQTT connection 0x3ffb9cd0) MQTT PUBLISH operation queued.
4842 95917 [OTA Agent Task] [INFO ][MQTT][959170] (MQTT connection 0x3ffb9cd0, PUBLISH operation 0x3ffba190) Waiting for operation completion.
4843 95921 [iot_thread] [INFO ][MQTT][959210] (MQTT connection 0x3ffb9cd0, PUBLISH operation 0x3ffba01c) Wait complete with result SUCCESS.
4844 95936 [OTA Agent Task] [INFO ][MQTT][959360] (MQTT connection 0x3ffb9cd0, PUBLISH operation 0x3ffba190) Wait complete with result SUCCESS.
4845 95936 [OTA Agent Task] [prvPublishStatusMessage] 'IN_PROGRESS' to $aws/things/Node2/jobs/AFR_OTA-pruebacliexterna33/update
4846 95936 [OTA Agent Task] Received eOTA_JobEvent_Activate callback from OTA Agent.
4847 95936 [OTA Agent Task] [INFO ][MQTT][959360] (MQTT connection 0x3ffb9cd0) Disconnecting connection.
4848 95936 [OTA Agent Task] [INFO] [MQTT] [core_mqtt.c:2149] 4849 95936 [OTA Agent Task] Disconnected from the broker.4850 95936 [OTA Agent Task]
4851 95936 [OTA Agent Task] [INFO ][MQTT][959360] (MQTT connection 0x3ffb9cd0) Network connection closed.
I (959417) boot_comm: chip revision: 1, min. application chip revision: 0
I (959477) esp_image: segment 0: paddr=0x001a0020 vaddr=0x3f400020 size=0x32ab8 (207544) map
I (959647) esp_image: segment 1: paddr=0x001d2ae0 vaddr=0x3ffbdb60 size=0x039d0 ( 14800)
I (959667) esp_image: segment 2: paddr=0x001d64b8 vaddr=0x40080000 size=0x00400 (  1024)
0x40080000: _WindowOverflow4 at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/freertos_kernel/portable/ThirdParty/GCC/Xtensa_ESP32_IDF3/xtensa_vectors.S:1713

I (959667) esp_image: segment 3: paddr=0x001d68c0 vaddr=0x40080400 size=0x09750 ( 38736)
I (959707) esp_image: segment 4: paddr=0x001e0018 vaddr=0x400d0018 size=0xc0780 (788352) map
0x400d0018: _flash_cache_start at ??:?

/xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/freertos_kernel/queue.c:1477 (xQueueSemaphoreTake)- assert failed!
abort() was called at PC 0x4008f609 on4852 96021 [iot_thread] State: WaitingForFileBlock  Receive core 0
0x4008f609: xQueueSemaphoreTake at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/freertos_kernel/queue.c:2429

ELF file SHA256: e6199796c6985075

Backtrace: 0x4008d1d0:0x3ffb3c70 0x4008d401:0x3ffb3c90 0x4008f609:0x3ffb3cb0 0x401514d3:0x3ffb3cf0 0x401514ed:0x3ffb3d10 0x40146c50:0x3ffb3d30 0x40148660:0x3ffb3d60 0x4014704e:0x3ffb3d80 0x40147a0c:0x3ffb3db0 0x400d3091:0x3ffb3de0 0x400d31b2:0x3ffb3e20 0x400d3291:0x3ffb3e80 0x400d2e0e:0x3ffb3ea0 0x401513ab:0x3ffb3ed0
0x4008d1d0: invoke_abort at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/vendors/espressif/esp-idf/components/esp32/panic.c:715

0x4008d401: abort at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/vendors/espressif/esp-idf/components/esp32/panic.c:715

0x4008f609: xQueueSemaphoreTake at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/freertos_kernel/queue.c:2429

0x401514d3: prIotMutexTimedLock at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/libraries/abstractions/platform/freertos/iot_threads_freertos.c:318

0x401514ed: IotMutex_Lock at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/libraries/abstractions/platform/freertos/iot_threads_freertos.c:318

0x40146c50: _IotMqtt_IncrementConnectionReferences at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/libraries/c_sdk/standard/mqtt/src/iot_mqtt_api.c:758

0x40148660: _IotMqtt_CreateOperation at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/libraries/c_sdk/standard/mqtt/src/iot_mqtt_operation.c:355

0x4014704e: IotMqtt_Publish at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/libraries/c_sdk/standard/mqtt/src/iot_mqtt_api.c:1670

0x40147a0c: IotMqtt_TimedPublish at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/libraries/c_sdk/standard/mqtt/src/iot_mqtt_api.c:1820

0x400d3091: prvPublishMessage at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/demos/ota/aws_iot_ota_update_demo.c:227

0x400d31b2: vRunOTAUpdateDemo at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/demos/ota/aws_iot_ota_update_demo.c:290

0x400d3291: vStartOTAUpdateDemoTask at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/demos/ota/aws_iot_ota_update_demo.c:433

0x400d2e0e: runDemoTask at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/demos/demo_runner/iot_demo_freertos.c:314

0x401513ab: _threadRoutineWrapper at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/libraries/abstractions/platform/freertos/iot_threads_freertos.c:318

Rebooting...
ets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:7280
ho 0 tail 12 room 4
load:0x40078000,len:12652
load:0x40080400,len:6708
entry 0x40080778
I (76) boot: Chip Revision: 1
I (76) boot_comm: chip revision: 1, min. bootloader chip revision: 0
I (41) boot: ESP-IDF HEAD-HASH-NOTFOUND 2nd stage bootloader
I (41) boot: compile time 09:27:35
I (41) boot: Enabling RNG early entropy source...
I (47) boot: SPI Speed      : 40MHz
I (51) boot: SPI Mode       : DIO
I (55) boot: SPI Flash Size : 4MB
I (59) boot: Partition Table:
I (63) boot: ## Label            Usage          Type ST Offset   Length
I (70) boot:  0 nvs              WiFi data        01 02 00010000 00006000
I (78) boot:  1 otadata          OTA data         01 00 00016000 00002000
I (85) boot:  2 phy_init         RF data          01 01 00018000 00001000
I (93) boot:  3 ota_0            OTA app          00 10 00020000 00177000
I (100) boot:  4 ota_1            OTA app          00 11 001a0000 00177000
I (108) boot:  5 storage          WiFi data        01 02 00317000 00010000
I (115) boot: End of partition table
I (120) boot: ota rollback check done
I (124) boot_comm: chip revision: 1, min. application chip revision: 0
I (131) esp_image: segment 0: paddr=0x00020020 vaddr=0x3f400020 size=0x32ab8 (207544) map
I (213) esp_image: segment 1: paddr=0x00052ae0 vaddr=0x3ffbdb60 size=0x039d0 ( 14800) load
I (219) esp_image: segment 2: paddr=0x000564b8 vaddr=0x40080000 size=0x00400 (  1024) load
0x40080000: _WindowOverflow4 at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/freertos_kernel/portable/ThirdParty/GCC/Xtensa_ESP32_IDF3/xtensa_vectors.S:1713

I (220) esp_image: segment 3: paddr=0x000568c0 vaddr=0x40080400 size=0x09750 ( 38736) load
I (245) esp_image: segment 4: paddr=0x00060018 vaddr=0x400d0018 size=0xc0780 (788352) map
0x400d0018: _flash_cache_start at ??:?

I (522) esp_image: segment 5: paddr=0x001207a0 vaddr=0x40089b50 size=0x12728 ( 75560) load
0x40089b50: r_lld_evt_restart at ??:?

I (569) boot: Loaded app from partition at offset 0x20000
I (569) boot: Disabling RNG early entropy source...
I (570) cpu_start: Pro cpu up.
I (574) cpu_start: Application information:
I (578) cpu_start: Project name:     esp-idf
I (583) cpu_start: App version:      1
I (588) cpu_start: Compile time:     Jun 19 2021 09:26:54
I (594) cpu_start: ELF file SHA256:  e6199796c6985075...
I (600) cpu_start: ESP-IDF:          HEAD-HASH-NOTFOUND
I (606) cpu_start: Single core mode
I (610) heap_init: Initializing. RAM available for dynamic allocation:
I (617) heap_init: At 3FFAFF10 len 000000F0 (0 KiB): DRAM
I (623) heap_init: At 3FFB6388 len 00001C78 (7 KiB): DRAM
I (629) heap_init: At 3FFB9A20 len 00004108 (16 KiB): DRAM
I (635) heap_init: At 3FFBDB5C len 00000004 (0 KiB): DRAM
I (641) heap_init: At 3FFD1020 len 0000EFE0 (59 KiB): DRAM
I (648) heap_init: At 3FFE0440 len 0001FBC0 (126 KiB): D/IRAM
I (654) heap_init: At 40078000 len 00008000 (32 KiB): IRAM
I (660) heap_init: At 4009C278 len 00003D88 (15 KiB): IRAM
I (666) cpu_start: Pro cpu start user code
I (13) cpu_start: Starting scheduler on PRO CPU.
E (24) uart: uart_set_hw_flow_ctrl(268): rx flow thresh error
0 1 [main] Initializing lwIP TCP stack
I (34) PKCS11: Initializing NVS partition: "storage"
E (64) PKCS11: failed nvs get file size 4354 0
1 4 [main] Write certificate...
2 7 [iot_thread] [INFO ][DEMO][70] ---------STARTING DEMO---------

I (104) BTDM_INIT: BT controller compile version [0fff744]
3 8 [iot_thread] [INFO ][INIT][80] SDK successfully initialized.
I (104) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (214) phy: phy_version: 4180, cb3948e, Sep 12 2019, 16:39:13, 0, 0
GAP procedure initiated: advertise; disc_mode=2 adv_channel_map=0 own_addr_type=1 adv_filter_policy=0 adv_itvl_min=187 adv_itvl_max=375
I (534) wifi:wifi driver task: 3ffdd438, prio:23, stack:3584, core=0
I (534) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (534) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (554) wifi:wifi firmware version: 1afdae0
I (554) wifi:config NVS flash: enabled
I (554) wifi:config nano formating: disabled
I (554) wifi:Init dynamic tx buffer num: 32
I (554) wifi:Init data frame dynamic rx buffer num: 32
I (564) wifi:Init management frame dynamic rx buffer num: 32
I (574) wifi:Init management short buffer num: 32
I (574) wifi:Init static rx buffer size: 1600
I (574) wifi:Init static rx buffer num: 10
I (584) wifi:Init dynamic rx buffer num: 32
I (594) wifi:mode : sta (80:7d:3a:bb:5e:a8)
I (594) WIFI: SYSTEM_EVENT_STA_START
I (2764) wifi:new:<9,0>, old:<1,0>, ap:<255,255>, sta:<9,0>, prof:1
I (3754) wifi:state: init -> auth (b0)
I (3764) wifi:state: auth -> assoc (0)
I (3774) wifi:state: assoc -> run (10)
I (3824) wifi:connected with WIFI_SSID, aid = 2, channel 9, BW20, bssid = BSSID
I (3824) wifi:security type: 4, phy: bgn, rssi: -44
I (3824) wifi:pm start, type: 1

I (3834) WIFI: SYSTEM_EVENT_STA_CONNECTED
I (3894) wifi:AP's beacon interval = 102400 us, DTIM period = 1
I (6714) event: sta ip: 192.168.100.125, mask: 255.255.255.0, gw: 192.168.100.1
I (6714) WIFI: SYSTEM_EVENT_STA_GOT_IP
4 669 [eventTask] [INFO ][DEMO][6690] Connected to WiFi access point, ip address: 192.168.100.125.
5 670 [iot_thread] [INFO ][DEMO][6700] Successfully initialized the demo. Network type for the demo: 1
6 670 [iot_thread] [INFO ][MQTT][6700] MQTT library successfully initialized.
GAP procedure initiated: stop advertising.
GAP procedure initiated: stop advertising.
7 673 [iot_thread] OTA demo version 0.9.18
8 673 [iot_thread] Creating MQTT Client...
E (6894) PKCS11: failed nvs get file size 4354 0
9 980 [iot_thread] Connecting to broker...
10 981 [iot_thread] [INFO ][MQTT][9800] Establishing new MQTT connection.
11 981 [iot_thread] [INFO ][MQTT][9810] (MQTT connection 0x3ffb9a3c, CONNECT operation 0x3ffb7aac) Waiting for operation completion.
12 992 [NetRecv] [INFO] [MQTT] [core_mqtt_serializer.c:970] 13 992 [NetRecv] CONNACK session present bit not set.14 992 [NetRecv]
15 992 [NetRecv] [INFO] [MQTT] [core_mqtt_serializer.c:912] 16 992 [NetRecv] Connection accepted.17 992 [NetRecv]
18 993 [iot_thread] [INFO ][MQTT][9930] (MQTT connection 0x3ffb9a3c, CONNECT operation 0x3ffb7aac) Wait complete with result SUCCESS.
19 993 [iot_thread] [INFO ][MQTT][9930] New MQTT connection 0x3ffbdc5c established.
20 993 [iot_thread] Connected to broker.
21 994 [iot_thread] [OTA_AgentInit_internal] OTA Task is Ready.
22 995 [iot_thread] [INFO ][MQTT][9940] (MQTT connection 0x3ffb9a3c) MQTT PUBLISH operation queued.
23 995 [iot_thread] [INFO ][MQTT][9950] (MQTT connection 0x3ffb9a3c, PUBLISH operation 0x3ffd640c) Waiting for operation completion.
I (9964) ota_pal: prvPAL_GetPlatformImageState
I (10014) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (10014) esp_ota_ops: [0] aflags/seq:0x2/0x1, pflags/seq:0xffffffff/0x0
24 1000 [OTA Agent Task] [prvExecuteHandler] Called handler. Current State [Ready] Event [Start] New state [RequestingJob]
25 1001 [OTA Agent Task] [INFO ][MQTT][10010] (MQTT connection 0x3ffb9a3c) SUBSCRIBE operation scheduled.
26 1001 [OTA Agent Task] [INFO ][MQTT][10010] (MQTT connection 0x3ffb9a3c, SUBSCRIBE operation 0x3ffb9dd8) Waiting for operation completion.
27 1004 [iot_thread] [INFO ][MQTT][10040] (MQTT connection 0x3ffb9a3c, PUBLISH operation 0x3ffd640c) Wait complete with result SUCCESS.
28 1010 [OTA Agent Task] [INFO ][MQTT][10100] (MQTT connection 0x3ffb9a3c, SUBSCRIBE operation 0x3ffb9dd8) Wait complete with result SUCCESS.
29 1010 [OTA Agent Task] [prvSubscribeToJobNotificationTopics] OK: $aws/things/Node2/jobs/$next/get/accepted
30 1012 [OTA Agent Task] [INFO ][MQTT][10110] (MQTT connection 0x3ffb9a3c) SUBSCRIBE operation scheduled.
31 1012 [OTA Agent Task] [INFO ][MQTT][10120] (MQTT connection 0x3ffb9a3c, SUBSCRIBE operation 0x3ffb9dd8) Waiting for operation completion.
32 1023 [OTA Agent Task] [INFO ][MQTT][10230] (MQTT connection 0x3ffb9a3c, SUBSCRIBE operation 0x3ffb9dd8) Wait complete with result SUCCESS.
33 1023 [OTA Agent Task] [prvSubscribeToJobNotificationTopics] OK: $aws/things/Node2/jobs/notify-next
34 1023 [OTA Agent Task] [prvRequestJob_Mqtt] Request #0
35 1024 [OTA Agent Task] [INFO ][MQTT][10240] (MQTT connection 0x3ffb9a3c) MQTT PUBLISH operation queued.
36 1024 [OTA Agent Task] [INFO ][MQTT][10240] (MQTT connection 0x3ffb9a3c, PUBLISH operation 0x3ffd640c) Waiting for operation completion.
37 1039 [OTA Agent Task] [INFO ][MQTT][10390] (MQTT connection 0x3ffb9a3c, PUBLISH operation 0x3ffd640c) Wait complete with result SUCCESS.
38 1039 [OTA Agent Task] [prvExecuteHandler] Called handler. Current State [RequestingJob] Event [RequestJobDocument] New state [WaitingForJob]
39 1041 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ clientToken: 0:Node2 ]
40 1041 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ timestamp: 1624372566 ]
41 1041 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ jobId: AFR_OTA-pruebacliexterna33 ]
42 1041 [OTA Agent Task] [prvParseJSONbyModel] Identified parameter [ self_test ]
43 1041 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ updatedBy: 589842 ]
44 1041 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ protocols: ["MQTT"] ]
45 1041 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ streamname: AFR_OTA-e4480498-43e8-4d6d-8315-77db3a813e24 ]
46 1041 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ filepath: / ]
47 1041 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ filesize: 1126128 ]
48 1041 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ fileid: 0 ]
49 1041 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ certfile: / ]
50 1041 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ sig-sha256-ecdsa: MEUCIQC/pBdLED9qFCfiC1weHbAbsU9i... ]
51 1041 [OTA Agent Task] [prvParseJobDoc] In self test mode.
52 1041 [OTA Agent Task] [prvValidateUpdateVersion] We rebooted and the version is still the same.
53 1041 [OTA Agent Task] [prvParseJobDoc] Downgrade or same version not allowed, rejecting the update & rebooting.
I (10434) ota_pal: prvPAL_SetPlatformImageState, 3
W (10554) ota_pal: Set image as invalid!
I (10564) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (10574) esp_ota_ops: [0] aflags/seq:0x2/0x1, pflags/seq:0xffffffff/0x0
W (10574) ota_pal: Image not in self test mode 2
I (10584) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (10584) esp_ota_ops: [0] aflags/seq:0x2/0x1, pflags/seq:0xffffffff/0x0
54 1057 [OTA Agent Task] [prvPublishStatusMessage] Msg: {"status":"FAILED","statusDetails":{"reason":"rejected: 0x23000000"}}
55 1058 [OTA Agent Task] [INFO ][MQTT][10580] (MQTT connection 0x3ffb9a3c) MQTT PUBLISH operation queued.
56 1058 [OTA Agent Task] [INFO ][MQTT][10580] (MQTT connection 0x3ffb9a3c, PUBLISH operation 0x3ffb9d3c) Waiting for operation completion.
57 1080 [OTA Agent Task] [INFO ][MQTT][10800] (MQTT connection 0x3ffb9a3c, PUBLISH operation 0x3ffb9d3c) Wait complete with result SUCCESS.
58 1080 [OTA Agent Task] [prvPublishStatusMessage] 'FAILED' to $aws/things/Node2/jobs/AFR_OTA-pruebacliexterna33/update
59 1080 [OTA Agent Task] [prvResetDevice] Attempting forced reset of device...
60 1104 [iot_thread] State: Ready  Received: 1   Queued: 0   Processed: 0   Dropped: 0
61 1105 [iot_thread] [INFO ][MQTT][11050] (MQTT connection 0x3ffb9a3c) MQTT PUBLISH operation queued.
62 1105 [iot_thread] [INFO ][MQTT][11050] (MQTT connection 0x3ffb9a3c, PUBLISH operation 0x3ffb9d3c) Waiting for operation completion.
63 1120 [iot_thread] [INFO ][MQTT][11200] (MQTT connection 0x3ffb9a3c, PUBLISH operation 0x3ffb9d3c) Wait complete with result SUCCESS.
I (11324) wifi:state: run -> init (0)
I (11324) wifi:pm stop, total sleep time: 4848359 us / 7494707 us

I (11324) wifi:new:<9,0>, old:<9,0>, ap:<255,255>, sta:<9,0>, prof:1
I (11334) WIFI: SYSTEM_EVENT_STA_DISCONNECTED: 8
64 1131 [eventTask] [INFO ][DEMO][11310] Disconnected from WiFi access point, reason code: 0.
I (11334) BTDM_INIT: BT controller compile version [0fff744]
I (11344) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (11614) wifi:flush txq
I (11614) wifi:stop sw txq
I (11614) wifi:lmac stop hw txq
ets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:7280
ho 0 tail 12 room 4
load:0x40078000,len:12652
load:0x40080400,len:6708
entry 0x40080778
I (76) boot: Chip Revision: 1
I (76) boot_comm: chip revision: 1, min. bootloader chip revision: 0
I (41) boot: ESP-IDF HEAD-HASH-NOTFOUND 2nd stage bootloader
I (41) boot: compile time 09:27:35
I (41) boot: Enabling RNG early entropy source...
I (47) boot: SPI Speed      : 40MHz
I (51) boot: SPI Mode       : DIO
I (55) boot: SPI Flash Size : 4MB
I (59) boot: Partition Table:
I (63) boot: ## Label            Usage          Type ST Offset   Length
I (70) boot:  0 nvs              WiFi data        01 02 00010000 00006000
I (78) boot:  1 otadata          OTA data         01 00 00016000 00002000
I (85) boot:  2 phy_init         RF data          01 01 00018000 00001000
I (93) boot:  3 ota_0            OTA app          00 10 00020000 00177000
I (100) boot:  4 ota_1            OTA app          00 11 001a0000 00177000
I (108) boot:  5 storage          WiFi data        01 02 00317000 00010000
I (115) boot: End of partition table
I (120) boot: ota rollback check done
I (124) boot_comm: chip revision: 1, min. application chip revision: 0
I (131) esp_image: segment 0: paddr=0x00020020 vaddr=0x3f400020 size=0x32ab8 (207544) map
I (213) esp_image: segment 1: paddr=0x00052ae0 vaddr=0x3ffbdb60 size=0x039d0 ( 14800) load
I (219) esp_image: segment 2: paddr=0x000564b8 vaddr=0x40080000 size=0x00400 (  1024) load
0x40080000: _WindowOverflow4 at /xxxxxxxxxxxxxxxxxxxxxxxxx/FreeRTOS-Repo-Node2/FreeRTOS/freertos_kernel/portable/ThirdParty/GCC/Xtensa_ESP32_IDF3/xtensa_vectors.S:1713

I (220) esp_image: segment 3: paddr=0x000568c0 vaddr=0x40080400 size=0x09750 ( 38736) load
I (245) esp_image: segment 4: paddr=0x00060018 vaddr=0x400d0018 size=0xc0780 (788352) map
0x400d0018: _flash_cache_start at ??:?

I (522) esp_image: segment 5: paddr=0x001207a0 vaddr=0x40089b50 size=0x12728 ( 75560) load
0x40089b50: r_lld_evt_restart at ??:?

I (569) boot: Loaded app from partition at offset 0x20000
I (569) boot: Disabling RNG early entropy source...
I (570) cpu_start: Pro cpu up.
I (574) cpu_start: Application information:
I (578) cpu_start: Project name:     esp-idf
I (583) cpu_start: App version:      1
I (588) cpu_start: Compile time:     Jun 19 2021 09:26:54
I (594) cpu_start: ELF file SHA256:  e6199796c6985075...
I (600) cpu_start: ESP-IDF:          HEAD-HASH-NOTFOUND
I (606) cpu_start: Single core mode
I (610) heap_init: Initializing. RAM available for dynamic allocation:
I (617) heap_init: At 3FFAFF10 len 000000F0 (0 KiB): DRAM
I (623) heap_init: At 3FFB6388 len 00001C78 (7 KiB): DRAM
I (629) heap_init: At 3FFB9A20 len 00004108 (16 KiB): DRAM
I (635) heap_init: At 3FFBDB5C len 00000004 (0 KiB): DRAM
I (641) heap_init: At 3FFD1020 len 0000EFE0 (59 KiB): DRAM
I (648) heap_init: At 3FFE0440 len 0001FBC0 (126 KiB): D/IRAM
I (654) heap_init: At 40078000 len 00008000 (32 KiB): IRAM
I (660) heap_init: At 4009C278 len 00003D88 (15 KiB): IRAM
I (666) cpu_start: Pro cpu start user code
I (13) cpu_start: Starting scheduler on PRO CPU.
E (24) uart: uart_set_hw_flow_ctrl(268): rx flow thresh error
0 1 [main] Initializing lwIP TCP stack
I (34) PKCS11: Initializing NVS partition: "storage"
E (74) PKCS11: failed nvs get file size 4354 0
1 5 [main] Write certificate...
2 12 [iot_thread] [INFO ][DEMO][120] ---------STARTING DEMO---------

I (144) BTDM_INIT: BT controller compile version [0fff744]
3 12 [iot_thread] [INFO ][INIT][120] SDK successfully initialized.
I (154) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (264) phy: phy_version: 4180, cb3948e, Sep 12 2019, 16:39:13, 0, 0
GAP procedure initiated: advertise; disc_mode=2 adv_channel_map=0 own_addr_type=1 adv_filter_policy=0 adv_itvl_min=187 adv_itvl_max=375
I (594) wifi:wifi driver task: 3ffdd60c, prio:23, stack:3584, core=0
I (594) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (594) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (614) wifi:wifi firmware version: 1afdae0
I (614) wifi:config NVS flash: enabled
I (614) wifi:config nano formating: disabled
I (614) wifi:Init dynamic tx buffer num: 32
I (614) wifi:Init data frame dynamic rx buffer num: 32
I (624) wifi:Init management frame dynamic rx buffer num: 32
I (634) wifi:Init management short buffer num: 32
I (634) wifi:Init static rx buffer size: 1600
I (634) wifi:Init static rx buffer num: 10
I (644) wifi:Init dynamic rx buffer num: 32
I (654) wifi:mode : sta (80:7d:3a:bb:5e:a8)
I (654) WIFI: SYSTEM_EVENT_STA_START
I (2824) wifi:new:<9,0>, old:<1,0>, ap:<255,255>, sta:<9,0>, prof:1
I (3814) wifi:state: init -> auth (b0)
I (3814) wifi:state: auth -> assoc (0)
I (3824) wifi:state: assoc -> run (10)
I (3884) wifi:connected with WIFI_SSID, aid = 2, channel 9, BW20, bssid = BSSID
I (3894) wifi:security type: 4, phy: bgn, rssi: -43
I (3894) wifi:pm start, type: 1

I (3894) WIFI: SYSTEM_EVENT_STA_CONNECTED
I (3904) wifi:AP's beacon interval = 102400 us, DTIM period = 1
I (7154) event: sta ip: 192.168.100.125, mask: 255.255.255.0, gw: 192.168.100.1
I (7154) WIFI: SYSTEM_EVENT_STA_GOT_IP
4 713 [eventTask] [INFO ][DEMO][7130] Connected to WiFi access point, ip address: 192.168.100.125.
5 714 [iot_thread] [INFO ][DEMO][7140] Successfully initialized the demo. Network type for the demo: 1
6 714 [iot_thread] [INFO ][MQTT][7140] MQTT library successfully initialized.
GAP procedure initiated: stop advertising.
GAP procedure initiated: stop advertising.
7 717 [iot_thread] OTA demo version 0.9.18
8 717 [iot_thread] Creating MQTT Client...
E (7304) PKCS11: failed nvs get file size 4354 0
9 1020 [iot_thread] Connecting to broker...
10 1020 [iot_thread] [INFO ][MQTT][10200] Establishing new MQTT connection.
11 1020 [iot_thread] [INFO ][MQTT][10200] (MQTT connection 0x3ffb9cd0, CONNECT operation 0x3ffb7ab8) Waiting for operation completion.
12 1033 [NetRecv] [INFO] [MQTT] [core_mqtt_serializer.c:970] 13 1034 [NetRecv] CONNACK session present bit not set.14 1034 [NetRecv]
15 1034 [NetRecv] [INFO] [MQTT] [core_mqtt_serializer.c:912] 16 1034 [NetRecv] Connection accepted.17 1034 [NetRecv]
18 1035 [iot_thread] [INFO ][MQTT][10350] (MQTT connection 0x3ffb9cd0, CONNECT operation 0x3ffb7ab8) Wait complete with result SUCCESS.
19 1035 [iot_thread] [INFO ][MQTT][10350] New MQTT connection 0x3ffbdc5c established.
20 1035 [iot_thread] Connected to broker.
21 1036 [iot_thread] [OTA_AgentInit_internal] OTA Task is Ready.
22 1037 [iot_thread] [INFO ][MQTT][10370] (MQTT connection 0x3ffb9cd0) MQTT PUBLISH operation queued.
23 1037 [iot_thread] [INFO ][MQTT][10370] (MQTT connection 0x3ffb9cd0, PUBLISH operation 0x3ffd3524) Waiting for operation completion.
I (10384) ota_pal: prvPAL_GetPlatformImageState
I (10424) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (10434) esp_ota_ops: [0] aflags/seq:0x2/0x1, pflags/seq:0xffffffff/0x0
24 1042 [OTA Agent Task] [prvExecuteHandler] Called handler. Current State [Ready] Event [Start] New state [RequestingJob]
25 1042 [OTA Agent Task] [INFO ][MQTT][10420] (MQTT connection 0x3ffb9cd0) SUBSCRIBE operation scheduled.
26 1042 [OTA Agent Task] [INFO ][MQTT][10420] (MQTT connection 0x3ffb9cd0, SUBSCRIBE operation 0x3ffba0e0) Waiting for operation completion.
27 1046 [iot_thread] [INFO ][MQTT][10460] (MQTT connection 0x3ffb9cd0, PUBLISH operation 0x3ffd3524) Wait complete with result SUCCESS.
28 1064 [OTA Agent Task] [INFO ][MQTT][10640] (MQTT connection 0x3ffb9cd0, SUBSCRIBE operation 0x3ffba0e0) Wait complete with result SUCCESS.
29 1064 [OTA Agent Task] [prvSubscribeToJobNotificationTopics] OK: $aws/things/Node2/jobs/$next/get/accepted
30 1065 [OTA Agent Task] [INFO ][MQTT][10650] (MQTT connection 0x3ffb9cd0) SUBSCRIBE operation scheduled.
31 1065 [OTA Agent Task] [INFO ][MQTT][10650] (MQTT connection 0x3ffb9cd0, SUBSCRIBE operation 0x3ffba0e0) Waiting for operation completion.
32 1075 [OTA Agent Task] [INFO ][MQTT][10740] (MQTT connection 0x3ffb9cd0, SUBSCRIBE operation 0x3ffba0e0) Wait complete with result SUCCESS.
33 1075 [OTA Agent Task] [prvSubscribeToJobNotificationTopics] OK: $aws/things/Node2/jobs/notify-next
34 1076 [OTA Agent Task] [prvRequestJob_Mqtt] Request #0
35 1076 [OTA Agent Task] [INFO ][MQTT][10760] (MQTT connection 0x3ffb9cd0) MQTT PUBLISH operation queued.
36 1076 [OTA Agent Task] [INFO ][MQTT][10760] (MQTT connection 0x3ffb9cd0, PUBLISH operation 0x3ffb9a3c) Waiting for operation completion.
37 1101 [OTA Agent Task] [INFO ][MQTT][11010] (MQTT connection 0x3ffb9cd0, PUBLISH operation 0x3ffb9a3c) Wait complete with result SUCCESS.
38 1101 [OTA Agent Task] [prvExecuteHandler] Called handler. Current State [RequestingJob] Event [RequestJobDocument] New state [WaitingForJob]
I (11054) ota_pal: prvPAL_SetPlatformImageState, 4
W (11054) ota_pal: Set image as aborted!
I (11054) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (11064) esp_ota_ops: [0] aflags/seq:0x2/0x1, pflags/seq:0xffffffff/0x0
W (11074) ota_pal: Image not in self test mode 2
I (11074) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (11084) esp_ota_ops: [0] aflags/seq:0x2/0x1, pflags/seq:0xffffffff/0x0
39 1102 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ clientToken: 0:Node2 ]
40 1103 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ timestamp: 1624372579 ]
41 1103 [OTA Agent Task] [prvParseJSONbyModel] parameter not present: execution
42 1103 [OTA Agent Task] [prvParseJSONbyModel] parameter not present: jobId
43 1103 [OTA Agent Task] [prvParseJSONbyModel] parameter not present: jobDocument
44 1103 [OTA Agent Task] [prvParseJSONbyModel] parameter not present: afr_ota
45 1103 [OTA Agent Task] [prvParseJSONbyModel] parameter not present: protocols
46 1103 [OTA Agent Task] [prvParseJSONbyModel] parameter not present: files
47 1103 [OTA Agent Task] [prvParseJSONbyModel] parameter not present: filepath
48 1103 [OTA Agent Task] [prvParseJSONbyModel] parameter not present: filesize
49 1103 [OTA Agent Task] [prvParseJSONbyModel] parameter not present: fileid
50 1103 [OTA Agent Task] [prvParseJSONbyModel] parameter not present: certfile
51 1103 [OTA Agent Task] [prvParseJSONbyModel] parameter not present: sig-sha256-ecdsa
52 1103 [OTA Agent Task] [prvParseJobDoc] No active jobs available in the service for execution.
53 1103 [OTA Agent Task] [prvParseJobDoc] Ignoring job without ID.
54 1106 [OTA Agent Task] [prvExecuteHandler] Handler failed. Current State [WaitingForJob] Event  [ReceivedJobDocument] Error Code [603979776]
55 1146 [iot_thread] State: Ready  Received: 1   Queued: 0   Processed: 0   Dropped: 0
56 1147 [iot_thread] [INFO ][MQTT][11470] (MQTT connection 0x3ffb9cd0) MQTT PUBLISH operation queued.
57 1147 [iot_thread] [INFO ][MQTT][11470] (MQTT connection 0x3ffb9cd0, PUBLISH operation 0x3ffb9a78) Waiting for operation completion.
58 1161 [iot_thread] [INFO ][MQTT][11610] (MQTT connection 0x3ffb9cd0, PUBLISH operation 0x3ffb9a78) Wait complete with result SUCCESS.
59 1261 [iot_thread] State: WaitingForJob  Received: 1   Queued: 0   Processed: 0   Dropped: 0

What I've found until now, it's that this error happen after trying to send an ota update to more than one node with the same job and the same ".bin" file.
This update to more than one node fails (with same error as mentioned) and, afterwards, those nodes stopped updating correctly as mentioned above. It seems like they get bugged after this ota job.

Hello @Carfer10 ,
When you are sending the same .bin file to multiple Things are you storing the credentials somewhere in non volatile memory and reading from it ? If it is still in the header file then the same .bin will have same credentials.

Thanks for your response @pvyawaha
I'm not storing the credentials in non volatile memory, how should I proceed to update an N number of nodes with same firmware image?

Hi @Carfer10,
Did the above suggestion help resolve the issue ?
Additionally I see that demo uses an old MQTT library version which is no longer actively maintained. Is it possible for you to upgrade to the new MQTT and OTA library so that you can stay up to date with the latest features and fixes. Please take a look at the OTA demo using latest MQTT and OTA library (LTS) versions  here

Hello @Carfer10 ,
Closing this ticket now , please reopen if you have questions on the same topic or create a new one for other topics.

source: gharchive/issue

authors: Carfer10 pvyawaha ravibhagavandas

repo: aws/amazon-freertos

license: MIT


text:
Put coreMQTT-Agent submodule back to version 1.0.0

Description

Set coreMQTT-Agent submodule to version 1.0.0, so that aws/amazon-freertos is not ahead of FreeRTOS/FreeRTOS.

/bot run checks


text:
How to use redshift_connector from Glue?

Driver version
redshift_connector-2.0.882-py3-none-any.whl
Redshift version
PostgreSQL 8.0.2 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.4.2 20041017 (Red Hat 3.4.2-6.fc3), Redshift 1.0.27747
Client Operating System
Glue 1.0
Python version
Python3 Glue version 1.0
Problem description

Expected behaviour:
Connect to the database

Actual behaviour:
CalledProcessError: Command '['/usr/local/bin/python', '-m', 'pip', 'install', '--target=/glue/lib/installation', '/tmp/glue-python-libs-fj9rfah5/redshift_connector-2.0.882-py3-none-any.whl']' returned non-zero exit status 1.

Any other details that can be helpful:
The script is running on my PC but doesn't tun on Glue

Python Driver trace logs

Reproduction code
import redshift_connector
Connects to Redshift cluster using AWS credentials
conn = redshift_connector.connect(
host='xyz.redshift.amazonaws.com',
database='xyz',
user='xyz',
password='xyz',
port = 5439
)
cursor: redshift_connector.Cursor = conn.cursor()
cursor.execute("SELECT version()")
return_value = cursor.fetchone()[0]
print(return_value)
conn.close

Hi @dburtsev ,
Thanks for reaching out. The team will try to reproduce.

Change job type from 'Python shell' to Spark 2.4 / Python 3
New error: ModuleNotFoundError: No module named 'scramp'

ModuleNotFoundError: No module named 'asn1crypto'
I load scramp-1.4.0-py3-none-any.whl and asn1crypto-1.4.0-py2.py3-none-any.whl into s3 and add references to the job python library path.
New error: InterfaceError: ('communication error', NotADirectoryError(20, 'Not a directory'))

Here is one option for importing redshift_connector to use with Glue.
If this option is not available to you , the following code block can be run on your development machine to download redshift_connector, and it's dependencies. These can be placed in an S3 bucket referenced by the Glue Python library path.
pip3 download redshift_connector

For example, the following dependencies were downloaded to my machine.
asn1crypto-1.4.0-py2.py3-none-any.whl       idna-2.10-py2.py3-none-any.whl              requests-2.25.1-py2.py3-none-any.whl
beautifulsoup4-4.9.3-py3-none-any.whl       jmespath-0.10.0-py2.py3-none-any.whl        s3transfer-0.5.0-py3-none-any.whl
boto3-1.18.1-py3-none-any.whl               lxml-4.6.3-cp39-cp39-macosx_10_9_x86_64.whl scramp-1.4.0-py3-none-any.whl
botocore-1.21.1-py3-none-any.whl            python_dateutil-2.8.2-py2.py3-none-any.whl  six-1.16.0-py2.py3-none-any.whl
certifi-2021.5.30-py2.py3-none-any.whl      pytz-2021.1-py2.py3-none-any.whl            soupsieve-2.2.1-py3-none-any.whl
chardet-4.0.0-py2.py3-none-any.whl          redshift_connector-2.0.882-py3-none-any.whl urllib3-1.26.6-py2.py3-none-any.whl

If the second option is appropriate for your use case, zipping the dependencies and placing them in an S3 bucket should work.

I have confirmed importing redshift_connector works using a Spark 2.4 / Python 3 job. The approach detailed below was easiest for me to get working.
If you are working from the AWS Console.

Click to edit the Glue job
Under the Security configuration, script libraries, and job parameters tab scroll to the Job parameters section
Set the key to --additional-python-modules and the value to redshift_connector
Run the job

Example job
print("pre-import print stmt")

import redshift_connector

print("import worked!")

I add --additional-python-modules key
Got new error: Command failed with exit code 1
How many libraries do you have in Python library path? I have only redshift_connector, scramp and asn1crypto

It works now.
I misspell library and got "Command failed with exit code 1" - fixes
I got "NotADirectoryError: [Errno 20] Not a directory" - fixed by adding SSL = False in redshift_connector.connect

authors: Brooke-white dburtsev

repo: aws/amazon-redshift-python-driver


text:
Update Hub's example for auth event tracking

Previous exemple was 1) long and confusing, 2) didn't work out of the box, 3) was set to "debug" mode so it didn't appear in the console, 4) built a class for no apparent reason, 5) had misleading variable names
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@a69c29e). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff            @@
##             master    #938   +/-   ##
========================================
  Coverage          ?   87.5%
========================================
  Files             ?      78
  Lines             ?    3706
  Branches          ?     684
========================================
  Hits              ?    3243
  Misses            ?     441
  Partials          ?      22

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a69c29e...0ccffde. Read the comment docs.

authors: codecov-io mtrelis

repo: aws/aws-amplify


text:
Refactor event handling and remove service reconciler

What type of PR is this?
refactoring/enhancement
Which issue does this PR fix:
#359
What does this PR do / Why do we need it:
We have multiple reconcilers that do the same thing which causes bug #359, and it is really hard to understand what is going on event handler. I did a refactoring to help clarify which resource events are mapped to which other resources.

(Almost) deleted service reconciler, now it only removes finalizer for keeping compatibility. Its main purpose was watching endpoint event, so now it is watched on serviceexport / route reconcilers instead.
Removed and unified event handlers. Now service event handler covers all service related events such as policy and endpoints.
Added mapper.go under event handler package dedicated to resource mapping using k8s client api calls
Added a few unit tests on event handler

If an issue # is not available please add repro steps and logs from aws-gateway-controller showing the issue:
Testing done on this change:
e2etest, also manually tested #359 case by updating existing service endpoints.
Automation added to e2e:

Will this PR introduce any new dependencies?:

Will this break upgrades or downgrades. Has updating a running cluster been tested?:
Does this PR introduce any user-facing change?:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Pull Request Test Coverage Report for Build 5993051159

131 of 183 (71.58%)  changed or added relevant lines in 5 files are covered.
3 unchanged lines in 2 files lost coverage.
Overall coverage increased (+1.7%) to 39.191%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

controllers/serviceexport_controller.go
0
3
0.0%

controllers/eventhandlers/serviceimport.go
14
18
77.78%

controllers/eventhandlers/service.go
34
41
82.93%

controllers/route_controller.go
0
8
0.0%

controllers/eventhandlers/mapper.go
83
113
73.45%

Files with Coverage Reduction
New Missed Lines
%

controllers/service_controller.go
1
0.0%

controllers/eventhandlers/service.go
2
81.25%

Totals

Change from base Build 5992865652:
1.7%

Covered Lines:
4003

Relevant Lines:
10214

💛  - Coveralls

Pull Request Test Coverage Report for Build 5993051159

131 of 183 (71.58%)  changed or added relevant lines in 5 files are covered.
3 unchanged lines in 2 files lost coverage.
Overall coverage increased (+1.7%) to 39.191%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

controllers/serviceexport_controller.go
0
3
0.0%

controllers/eventhandlers/serviceimport.go
14
18
77.78%

controllers/eventhandlers/service.go
34
41
82.93%

controllers/route_controller.go
0
8
0.0%

controllers/eventhandlers/mapper.go
83
113
73.45%

Files with Coverage Reduction
New Missed Lines
%

controllers/service_controller.go
1
0.0%

controllers/eventhandlers/service.go
2
81.25%

Totals

Change from base Build 5992865652:
1.7%

Covered Lines:
4003

Relevant Lines:
10214

💛  - Coveralls

Pull Request Test Coverage Report for Build 5993051159

131 of 183 (71.58%)  changed or added relevant lines in 5 files are covered.
3 unchanged lines in 2 files lost coverage.
Overall coverage increased (+1.7%) to 39.191%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

controllers/serviceexport_controller.go
0
3
0.0%

controllers/eventhandlers/serviceimport.go
14
18
77.78%

controllers/eventhandlers/service.go
34
41
82.93%

controllers/route_controller.go
0
8
0.0%

controllers/eventhandlers/mapper.go
83
113
73.45%

Files with Coverage Reduction
New Missed Lines
%

controllers/service_controller.go
1
0.0%

controllers/eventhandlers/service.go
2
81.25%

Totals

Change from base Build 5992865652:
1.7%

Covered Lines:
4003

Relevant Lines:
10214

💛  - Coveralls

authors: coveralls solmonk


text:
Support Python 3

This is trivial these days.

So, this is not happening, I guess?

This project has been archived. Please use CloudFormation’s native support for Custom Resources as documented at https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html

authors: hyandell nikolay

repo: aws/aws-cfn-resource-bridge

license_source: bigquery


text:
IAM RBAC to Schema Registry?

Can I use this jar to give certain role/arn access to certain subjects in schema registry, similar to https://docs.confluent.io/platform/current/schema-registry/security/rbac-schema-registry.html ?

The aws-msk-iam-auth library is only for interactions with the Kafka cluster and providing users or roles access to Kafka resources such as topics or consumer groups. The aws-glue-schema-registry handles the interactions with resources (registries, schemas) in glue schema registry.
The two can be used together.
However, if you have questions about authn/z with the glue schema registry could you open an issue with that library? You will get answers from the experts on the glue schema registry library.

@sayantacC  I'm interested in confluent registry not glue one

repo: aws/aws-msk-iam-auth


text:
Not able to administer my SGE

Environment:

AWS ParallelCluster 2.3.1
OS: ubuntu 14
Scheduler: SGE
Master instance type: C4
Compute instance type: C4

Bug description and how to reproduce:
I am trying to run this command to enable my queue
qmod -e all.q@*

And when I change to sgeadmin user, I still have permission issues
(base) ubuntu@ip-xxxxx:~$ sudo su sgeadmin
sgeadmin@ip-xxxx:/home/ubuntu$ qmod -e all.q@*
ubuntu - you have no permission to modify queue "all.q@ip-xxxx.us-west-2.compute.internal"

I tried to add ubuntu as a manager
sgeadmin@ip-xxxx:/home/ubuntu$ qconf -am ubuntu
denied: client (ip-xxx.us-west-2.compute.internal/qconf/1) uses old GDI version 6.2u5 while qmaster uses newer version 8.1.9
denied: "ubuntu" must be manager for this operation

My all.q seems to be not enabled and I am stuck not being able to edit my SGE cluster created by ParallelCLuster

Try as root sudo su

@sean-smith  but it is still throwing : denied: ubuntu must be manager for this operation

Hi @radaniba I was able to get this working as root. I only needed to set SGE_ROOT.
ubuntu@ip-172-31-38-29:~$ sudo su
root@ip-172-31-38-29:/home/ubuntu# export SGE_ROOT=/opt/sge
root@ip-172-31-38-29:/home/ubuntu# /opt/sge/bin/lx-amd64/qmod -e all.q@*
Queue instance "all.q@ip-172-31-38-249.ec2.internal" is already in the specified state: enabled
root@ip-172-31-38-29:/home/ubuntu#

Can I ask why you’re using Ubuntu1404 and not Ubuntu1604?
On Jun 27, 2019, at 12:35 PM, Sean Smith <notifications@github.commailto:notifications@github.com> wrote:
Hi @radanibahttps://github.com/radaniba I was able to get this working as root. I only needed to set SGE_ROOT.
ubuntu@ip-172-31-38-29:~$ sudo su
root@ip-172-31-38-29:/home/ubuntu# export SGE_ROOT=/opt/sge
root@ip-172-31-38-29:/home/ubuntu# /opt/sge/bin/lx-amd64/qmod -e all.q@*
Queue instance "all.q@ip-172-31-38-249.ec2.internalmailto:all.q@ip-172-31-38-249.ec2.internal" is already in the specified state: enabled
root@ip-172-31-38-29:/home/ubuntu#
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/aws/aws-parallelcluster/issues/1159?email_source=notifications&email_token=ABKJ77CAWQLDZIAKPLZMSXLP4SJUZA5CNFSM4H3LJ7BKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYWWNZQ#issuecomment-506291942, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABKJ77HST5QSJUK7BCJP7TLP4SJUZANCNFSM4H3LJ7BA.

Thank you that solved my issue after exporting SGE_ROOT, thank you so much and yes I am using ubuntu 14

authors: radaniba sean-smith

repo: aws/aws-parallelcluster


text:
Removes @timeout decorator.

Description of changes:
The Wrapt package has requirements that temporarily install versions of python that are not compatible with a strict 3.6 or 3.7 Python install. This issue was noticed when attempting to create an AWS Lambda layer from ParallelCluster's testing framework. Many other timeout mechanisms exist and the existing package is only used in a single case.
The Wrapt requirements can be seen at https://github.com/bitranox/wrapt_timeout_decorator/blob/master/requirements.txt.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #1765 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #1765   +/-   ##
========================================
  Coverage    52.61%   52.61%
========================================
  Files           24       24
  Lines         3959     3959
========================================
  Hits          2083     2083
  Misses        1876     1876

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b4d9854...0da77a4. Read the comment docs.

authors: codecov-io patemotter


text:
Linker error when trying to link with S3 (Ubuntu 16.04 gcc 5.3.1)

I'm trying to write test program with aws-sdk-cpp for S3 file download. I'm trying to connect to my private cloud server that I've setup. But before I do that, the compilation fails in the linker stage.
What i've done so far is copy the AWS C++ S3 example from the blog and trying to modify it. Here's the source code for my program.
#include <iostream>
#include <aws/s3/S3Client.h>
#include <aws/core/client/ClientConfiguration.h>
#include <aws/core/auth/AWSCredentialsProvider.h>
#include <aws/core/utils/ratelimiter/DefaultRateLimiter.h>
#include <aws/core/utils/Outcome.h>
#include <aws/core/utils/DateTime.h>
#include <aws/core/http/HttpClient.h>
#include <aws/core/http/HttpClientFactory.h>
#include <aws/s3/model/GetObjectRequest.h>
#include <fstream>

using namespace Aws::Auth;
using namespace Aws::Http;
using namespace Aws::Client;
using namespace Aws::S3;
using namespace Aws::S3::Model;
using namespace Aws::Utils;

static const char* ALLOCATION_TAG = "AwsDownload";    // example allocation tag
static const char* ENDPOINT = "s3-region1.testserver.io";  // example test server
static const char* BUCKET_NAME = "bucket01";
static const char* KEY = "test.txt";
static const char* DOWNLOADED_FILENAME = "test.txt";
static const char* ACCESS_KEY_ID = "05945f8dded91fcd0b9b"; // sample access id
static const char* SECRET_KEY = "6zyK4A8eh8TsOuaVdBhvZ0qVgVUXauA++z05fsC"; // sample secret key

int main()
{
   std::shared_ptr<S3Client> Client;
   std::shared_ptr<HttpClientFactory> ClientFactory;
   std::shared_ptr<Aws::Utils::RateLimits::RateLimiterInterface> Limiter;
   Aws::String TimeStamp;

   // initialize the variables
   TimeStamp = DateTime::CalculateGmtTimestampAsString("%Y%m%dt%H%M%Sz");
   Limiter = Aws::MakeShared<Aws::Utils::RateLimits::DefaultRateLimiter<>>(ALLOCATION_TAG, 50000000);
   //ClientFactory = Aws::MakeShared<HttpClientFactory>(ALLOCATION_TAG);

   // create a client configuration
   ClientConfiguration config;
   config.scheme = Scheme::HTTP;
   config.endpointOverride = static_cast<Aws::String>(ENDPOINT);
   // create the aws client
   Client = Aws::MakeShared<S3Client>(
      ALLOCATION_TAG,
      AWSCredentials(Aws::String(ACCESS_KEY_ID), Aws::String(SECRET_KEY)),
      config
   );

   // Do a download
   GetObjectRequest objreq;
   objreq.SetBucket(BUCKET_NAME);
   objreq.SetKey(KEY);

   auto outcome = Client->GetObject(objreq);
   if (outcome.IsSuccess())
   {
      std::cout << "File download from S3 to location: " << DOWNLOADED_FILENAME << std::endl;
   }
   else
   {
      std::cout << "File download failed from s3 with error: " << outcome.GetError().GetMessage() << std::endl;
   }

   return 0;
}

When I compile, here's what I get:
g++  -I/usr/local/include -L/usr/local/lib -Wwrite-strings -std=c++11 -Wl,-rpath=/usr/local/lib -c main.cpp -o main.o
g++  -I/usr/local/include -L/usr/local/lib -Wwrite-strings -std=c++11 -Wl,-rpath=/usr/local/lib \
	main.o -lpthread -laws-cpp-sdk-s3 -laws-cpp-sdk-core  -o awstest
main.o:(.rodata._ZTIN3Aws2S39S3RequestE[_ZTIN3Aws2S39S3RequestE]+0x10): undefined reference to `typeinfo for Aws::AmazonSerializableWebServiceRequest'
collect2: error: ld returned 1 exit status
Makefile:68: recipe for target 'awstest' failed
make: *** [awstest] Error 1

Anyone have any ideas why this linker error is occurring? I've also tested this on Ubuntu 14.04 (gcc 4.8.4) and I do get the same linker error. What am I missing?

You may be running into the no-rtti issue that is currently outstanding.  As a test, try compiling with -fno-rtti and see if the error disappears.  The fix for that problem should be out sometime today or tomorrow.

Try
GetObjectRequest *objreq = Aws::New(ALLOCATION_TAG);
instead of
GetObjectRequest objreq;

@bretambrose Looks like the -fno-rtti worked! So is this a clone of which bug?

https://github.com/aws/aws-sdk-cpp/issues/191

Add this:
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -750,7 +750,7 @@ if(PLATFORM_WINDOWS)
     endif()
 elseif(PLATFORM_LINUX OR PLATFORM_ANDROID OR PLATFORM_APPLE)
     # max warning level, warnings are errors, turn off unused private field. We have one for an empty class.
-    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Werror -pedantic -Wextra")
+    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Werror -pedantic -Wextra -fno-rtti")

     if(CMAKE_CXX_COMPILER_ID MATCHES "Clang")
         set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-unused-private-field")

solves the problem. The build is now successful. Thanks!

authors: RudolfVonKrugstein bretambrose minaltolpadi thedevilfish

repo: aws/aws-sdk-cpp


text:
Default value confusion in ReceiveMessageRequest (SQS)

Hi,
I'm using SQS, and i found WaitTimeSeconds property confusing.
In the POV of the developer, this property should be int, which means that its default value is 0.
Where under the surface, you are using int? and if no one set this property it gets of course null.
Both have different behavior of the long polling settings.
I suggest to reflect outside that the type of the property is nullable.
So developers, can infer from the code, that there is difference between 0 and null, and expect different behavior.
https://github.com/aws/aws-sdk-net/blob/master/sdk/src/Services/SQS/Generated/Model/ReceiveMessageRequest.cs
(I think same goes to other properties as well)
Thanks.

Yes, I agree.  However, changing the type of the member would be a backward incompatible change and will break other developer's code that relies on this value being a primitive int instead of a nullable int. In the next major version of the SDK, we'll consider making this change.

authors: omerkorenn sstevenkang

repo: aws/aws-sdk-net


text:
Fix typo in fastify logger type declaration

Issue #, if available: #587
Description of changes: Fixes typo in import and type of FastifyLoggerInstance in aws-xray-sdk-fastify package
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #589 (90efaa7) into master (eebda75) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 90efaa7 differs from pull request most recent head ae73cda. Consider uploading reports for the commit ae73cda to get more accurate results

@@           Coverage Diff           @@
##           master     #589   +/-   ##
=======================================
  Coverage   83.38%   83.38%
=======================================
  Files          37       37
  Lines        1794     1794
=======================================
  Hits         1496     1496
  Misses        298      298

authors: carolabadeer codecov-commenter

repo: aws/aws-xray-sdk-node


text:
Expose etcd metrics

What would you like to be added:
After installation of EKS-A, etcd only listens on localhost for metrics which prevents us to scrape/monitor etcd with e.g. prometheus:
# grep listen-metrics-urls /etc/kubernetes/manifests/etcd.yaml
    - --listen-metrics-urls=http://127.0.0.1:2381

Please modify this parameter so, etcd listens for metrics on the control plane's IP.
Why is this needed:
To be able to monitor/scrape etcd which is a critical component of the cluster.

Thank you for this feedback. We are working on it and targeting to release this feature in Q1 2024.

Hi, guys!
Is there any changes for this feature?
Can you suggest any date of the release of this feature?

Any update on this?

authors: Cajga GEEugene d8660091 macintoshme

repo: aws/eks-anywhere


text:
Adding 1.26 template env for tinkerbell

Description of changes:
Adding required ENV var for 1.26 template for tinkerbell.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve

/lgtm

/lgtm
/woof
/approve

authors: abhay-krishna pokearu taneyland


text:
Cloudstack reconciler validator

Issue #, if available:

Cloudstack validator#1055
Validate Datacenter Config#1259

Description of changes:
Building validator in controller to use cmk to run validations against Cloudstack
https://github.com/aws/eks-anywhere/pull/5452
Testing (if applicable):
manually tested:

manually add new secret to create new wl cluster
successfully validate dc
use invalid secret (invalid api-key) to create new wl cluster

"Reconciler error" err=<  getting domain info - 🙈 Error: failed to decode response

use unexistent secret name to create cluster

cloudstack_datacenter_controller.go:73] "Failed to reconcile CloudStackDatacenterConfig" err="Secret "secret2" not found" controller="cloudstackdatacenterconfig"
Reconciler error" err="Secret "secret2" not found"

fail to validate dc stops reconciliation

CloudStackDatacenterConfig hasn't been validated yet"

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve


text:
update go to v1.21

Issue #, if available:
Description of changes:
Testing (if applicable):
Documentation added/planned (if applicable):
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/test all

The purpose of this was to address to failing govulncheck presubmit, which seems to have been addressed already here


text:
Cilium reconciler to use provider namespace in generating network policy

Description of changes:
Cilium reconciler when generating manifests were not using the provider namespaces when generating cilium manifests. This is not a problem for the most scenarios except when running cilium in "always" policy enforcement mode, where cilium expects explicit network policies to enable routing between any pods/services. This change enables the cilium reconciler to generate network policy for the provider's namespace that allows routing between the provider's controller and other services like kubernetes.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve

/lgtm

authors: abhay-krishna vignesh-goutham


text:
Get latest Git tag by semver instead of chronological order

This PR fixes the EKS-A version script by getting latest Git tag by semver, as intended, instead of chronological order.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve
/cherrypick release-0.20

/cherrypick release-0.21

/lgtm


text:
builder-base: use go 1.13/1.15/1.16 from al2 yum repo

Issue #, if available:
Description of changes:
AL2 maintains a version of golang a bit longer then upstream based on the version of golang they use to build docker in AL2.  These versions tend to have CVE patches backported.  This PR switches the builder-base to use these AL2 provided versions, even for older unsupported versions like 1.13 and 1.15, instead of upstreams
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/lgtm

authors: abhay-krishna jaxesn

repo: aws/eks-distro-build-tooling


text:
Added file release/1-24/production/RELEASE

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/lgtm

/unhold
/approve

repo: aws/eks-distro


text:
fix(docs): fixed SDTIN

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

I tried fixing tiny typo on my issue

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: 92eb0002d4c369a2168ab2587771b84863e63649
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

authors: aws-cdk-automation seiyashima

repo: aws/jsii