text:
[General] Not able to build AWS OTA Demo for ESP32

Briefly summarize the issue being raised
When I try to run the following command:
idf.py -DVENDOR=espressif -DBOARD=esp32_devkitc -DCOMPILER=xtensa-esp32 build

I get the following error:
/Users/jagaty/esp/esp-idf/components/newlib/platform_include/time.h:28:19: error: unknown type name 'clockid_t'; did you mean '_lock_t'?
 int clock_settime(clockid_t clock_id, const struct timespec *tp);
                   ^~~~~~~~~
                   _lock_t
/Users/jagaty/esp/esp-idf/components/newlib/platform_include/time.h:29:19: error: unknown type name 'clockid_t'; did you mean '_lock_t'?
 int clock_gettime(clockid_t clock_id, struct timespec *tp);
                   ^~~~~~~~~
                   _lock_t
/Users/jagaty/esp/esp-idf/components/newlib/platform_include/time.h:30:18: error: unknown type name 'clockid_t'; did you mean '_lock_t'?
 int clock_getres(clockid_t clock_id, struct timespec *res);
                  ^~~~~~~~~
                  _lock_t
In file included from /Users/jagaty/Documents/Experiments/OTA_Test/FreeRTOS/libraries/3rdparty/mbedtls/include/mbedtls/platform.h:68,
                 from /Users/jagaty/Documents/Experiments/OTA_Test/FreeRTOS/vendors/espressif/esp-idf/components/mbedtls/port/mbedtls_debug.c:18:
/Users/jagaty/Documents/Experiments/OTA_Test/FreeRTOS/libraries/3rdparty/mbedtls/include/mbedtls/platform_time.h:78:9: error: unknown type name 'time_t'
 typedef time_t mbedtls_time_t;

Describe the desired outcome
Ideally I should be able to able to build this and then flash onto my device.
System information

Hardware board: [https://shop.m5stack.com/products/esp32-psram-timer-camera-x-ov3660](ESP32 DevkitC)
IDE used: VS Code
Operating System: MacOS
Code version: latest - downloaded from AWS Console
Project/Demo: OTA

References
error.txt
Code/Steps to reproduce the current status

Download FreeRTOS with OTA demo enabled from AWS console onto the MAC.
ESP32 specific setup
Complete the prerequisites
Run idf.py -DVENDOR=espressif -DBOARD=esp32_devkitc -DCOMPILER=xtensa-esp32 build from the FreeRTOS directory.

Additional context
Similar issue
But I cannot find any solution mentioned to the issue.
Would appreciate any help!

Hello @pratyushjagaty!
Thank you for reporting this error. We are looking into this and will get back to you.

IDF Version:
idf.py --version
ESP-IDF v4.2.2-197-gf4a2c5b52

Executed fullclean with the result:
idf.py fullclean
Executing action: fullclean
Done

Finally run: idf.py -DVENDOR=espressif -DBOARD=esp32_devkitc -DCOMPILER=xtensa-esp32 build
The errors I face are the same. The exact stack trace is in errors.txt

Thank you for those details @pratyushjagaty.
Would you mind printing out the environment variables as well (you can type env to print all environment variables)? I think that fullclean does not remove the environment variables - this is what I got from the ESP website:
idf.py fullclean will delete the entire “build” directory contents. This includes all CMake configuration output. The next time the project is built, CMake will configure it from scratch. Note that this option recursively deletes all files in the build directory, so use with care. Project configuration is not deleted.
You should delete/clear the (old) environment variables from the list before starting anew.
Once you clean the environment variables, you can add them back to your terminal's path with this command (as copied from the getting started guide): source vendors/espressif/esp-idf/export.sh

Env variables after a clean and restart
TERM_SESSION_ID=w0t0p0:BA42DD65-D486-42F5-93AF-3DC6B03CA3A7
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.n0CM5K3R1J/Listeners
LC_TERMINAL_VERSION=3.4.8
COLORFGBG=15;0
ITERM_PROFILE=Default
XPC_FLAGS=0x0
PWD=/Users/jagaty/Desktop/FreeRTOS_Testing/FreeRTOS
SHELL=/bin/zsh
__CFBundleIdentifier=com.googlecode.iterm2
LC_CTYPE=UTF-8
TERM_PROGRAM_VERSION=3.4.8
TERM_PROGRAM=iTerm.app
PATH=/apollo/env/SDETools/bin:/Users/jagaty/.toolbox/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/jagaty/Library/Python/2.7/bin:/Library/Apple/usr/bin
LC_TERMINAL=iTerm2
COLORTERM=truecolor
COMMAND_MODE=unix2003
TERM=xterm-256color
HOME=/Users/jagaty
TMPDIR=/var/folders/xt/qc1497916t14xxc0b2rbhqwsq_xh4j/T/
USER=jagaty
XPC_SERVICE_NAME=0
LOGNAME=jagaty
ITERM_SESSION_ID=w0t0p0:BA42DD65-D486-42F5-93AF-3DC6B03CA3A7
__CF_USER_TEXT_ENCODING=0x0:0:0
SHLVL=1
OLDPWD=/Users/jagaty
JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_231.jdk/Contents/Home
P9K_TTY=old
_P9K_TTY=/dev/ttys000
ZSH=/Users/jagaty/.oh-my-zsh
PAGER=less
LESS=-R
LSCOLORS=Gxfxcxdxbxegedabagacad
P9K_SSH=0
_=/usr/bin/env

Post this I run: . $HOME/esp/esp-idf/export.sh
And the env variables are :
TERM_SESSION_ID=w0t0p0:BA42DD65-D486-42F5-93AF-3DC6B03CA3A7
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.n0CM5K3R1J/Listeners
LC_TERMINAL_VERSION=3.4.8
COLORFGBG=15;0
ITERM_PROFILE=Default
XPC_FLAGS=0x0
PWD=/Users/jagaty/Desktop/FreeRTOS_Testing/FreeRTOS
SHELL=/bin/zsh
__CFBundleIdentifier=com.googlecode.iterm2
LC_CTYPE=UTF-8
TERM_PROGRAM_VERSION=3.4.8
TERM_PROGRAM=iTerm.app
PATH=/Users/jagaty/esp/esp-idf/components/esptool_py/esptool:/Users/jagaty/esp/esp-idf/components/espcoredump:/Users/jagaty/esp/esp-idf/components/partition_table:/Users/jagaty/esp/esp-idf/components/app_update:/Users/jagaty/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin:/Users/jagaty/.espressif/tools/xtensa-esp32s2-elf/esp-2020r3-8.4.0/xtensa-esp32s2-elf/bin:/Users/jagaty/.espressif/tools/esp32ulp-elf/2.28.51-esp-20191205/esp32ulp-elf-binutils/bin:/Users/jagaty/.espressif/tools/esp32s2ulp-elf/2.28.51-esp-20191205/esp32s2ulp-elf-binutils/bin:/Users/jagaty/.espressif/tools/openocd-esp32/v0.10.0-esp32-20200709/openocd-esp32/bin:/Users/jagaty/.espressif/python_env/idf4.2_py3.7_env/bin:/Users/jagaty/esp/esp-idf/tools:/apollo/env/SDETools/bin:/Users/jagaty/.toolbox/bin:/Library/Frameworks/Python.framework/Versions/3.7/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/jagaty/Library/Python/2.7/bin:/Library/Apple/usr/bin
LC_TERMINAL=iTerm2
COLORTERM=truecolor
COMMAND_MODE=unix2003
TERM=xterm-256color
HOME=/Users/jagaty
TMPDIR=/var/folders/xt/qc1497916t14xxc0b2rbhqwsq_xh4j/T/
USER=jagaty
XPC_SERVICE_NAME=0
LOGNAME=jagaty
ITERM_SESSION_ID=w0t0p0:BA42DD65-D486-42F5-93AF-3DC6B03CA3A7
__CF_USER_TEXT_ENCODING=0x0:0:0
SHLVL=1
OLDPWD=/Users/jagaty
JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_231.jdk/Contents/Home
P9K_TTY=old
_P9K_TTY=/dev/ttys000
ZSH=/Users/jagaty/.oh-my-zsh
PAGER=less
LESS=-R
LSCOLORS=Gxfxcxdxbxegedabagacad
P9K_SSH=0
IDF_PATH=/Users/jagaty/esp/esp-idf
IDF_TOOLS_EXPORT_CMD=/Users/jagaty/esp/esp-idf/export.sh
IDF_TOOLS_INSTALL_CMD=/Users/jagaty/esp/esp-idf/install.sh
OPENOCD_SCRIPTS=/Users/jagaty/.espressif/tools/openocd-esp32/v0.10.0-esp32-20200709/openocd-esp32/share/openocd/scripts
IDF_PYTHON_ENV_PATH=/Users/jagaty/.espressif/python_env/idf4.2_py3.7_env
_=/usr/bin/env

And finally I have run:
idf.py -DVENDOR=espressif -DBOARD=esp32_devkitc -DCOMPILER=xtensa-esp32 build
I get the same error. (Linked in errors.txt above)

Hello @pratyushjagaty, I am sorry to hear that. We are trying to debug the issue you are having.
Also, while we debug the issue, it might be fruitful to parallelly ask this question on the ESP forum as they have more knowledge about the tool.

Yeah, I have gone ahead and done it. It's pending approval from a moderator. Will keep you posted.

Hello @pratyushjagaty,
Taking a look at the environment variables it seems the IDF_PATH, IDF_TOOLS_EXPORT_CMD etc.. are set to an esp idf path outside of amazon freertos. This could be an issue as the code in this repo uses a customized version of esp-idf which is in vendors/espressif/esp-idf.
As mentioned in doc here, you need to only download the ESP tool chain externally but use the IDF from within this repo under vendors/espressif/esp-idf. You can do so by first opening a new session to clear all current env variables and executing vendors/espressif/esp-idf/install.sh and source vendors/espressif/esp-idf/export.sh.

This solved the issue. Had to do what you said and do a chmod on the idf.py inside the tools directory. I am closing this now!
Thanks!


text:
Failing to successfully 'link table' with this driver version with MS Access 365.

Driver version
2.00.00.01
Redshift version
PostgreSQL 8.0.2 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.4.2 20041017 (Red Hat 3.4.2-6.fc3), Redshift 1.0.40182
Client Operating System
Winbdows 2011 64 bit
Table schema
All tables & vies
Problem description
Failing to successfully 'link table' with this driver version with MS Access 365.
Notable errors from the odbc driver connection log:
Jul 21 07:42:11.386 TRACE 26864 Simba::ODBC::Connection::SQLGetConnectAttr: +++++ enter +++++
Jul 21 07:42:11.386 INFO  26864 Simba::ODBC::Connection::SQLGetConnectAttr: Attribute: Unknown Attribute (30002)
Jul 21 07:42:11.399 ERROR 26864 Simba::ODBC::Connection::SQLGetConnectAttr: [Amazon][ODBC] (10210) Attribute identifier invalid or not supported: 30002
Our prior installed version driver (1.04.56.1000) does return this error.
->

Thanks for reporting issue. Do you mean 1.4.56 also has same issue?

Thanks for reporting issue. Do you mean 1.4.56 also has same issue?

I updated my comments above: Our prior installed version driver (1.04.56.1000) does not return this error.

1.4.56 does not have this issue.
Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: ilesh garish @.>
Sent: Tuesday, July 26, 2022 12:25:31 AM
To: aws/amazon-redshift-odbc-driver @.>
Cc: Sorensen, Joanna (EOTSS) @.>; Author @.>
Subject: Re: [aws/amazon-redshift-odbc-driver] Failing to successfully 'link table' with this driver version with MS Access 365. (Issue #1)
CAUTION: This email originated from a sender outside of the Commonwealth of Massachusetts mail system.  Do not click on links or open attachments unless you recognize the sender and know the content is safe.
Thanks for reporting issue. Do you mean 1.4.56 also has same issue?
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/aws/amazon-redshift-odbc-driver/issues/1*issuecomment-1194984608__;Iw!!CUhgQOZqV7M!jJwcP8yhxRDoF4dZLaRbN05kDbQ5kUA6xLvJKIwTqsgM1lzpCA2OJ9prACLpi9lXJanEvOkW-J2OF5xMeOeRj2TNTdd-aQ$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AZGGVNRQNLDHIZSG3YBYDWDVV5SDXANCNFSM54SXH7VA__;!!CUhgQOZqV7M!jJwcP8yhxRDoF4dZLaRbN05kDbQ5kUA6xLvJKIwTqsgM1lzpCA2OJ9prACLpi9lXJanEvOkW-J2OF5xMeOeRj2Qs7iGcsA$.
You are receiving this because you authored the thread.Message ID: @.***>

source: gharchive/issue

authors: EOTSSJSorensen iggarish

repo: aws/amazon-redshift-odbc-driver

license_type: permissive

license_source: github-api


text:
[e2e test]. Need to automate creating "non-default" namespace

Today, to run e2e test,  user requires to create a "non-default" namespace in the cluster.  This needs to be automated.

The e2e tests have been updated to only create objects in the non-default namespace (referencing a common constant). I think it might be better to create a new namespace for each test run and delete it afterward.

authors: erikfuller liwenwu-amazon


text:
bug: Target not automatically registered in Target Group

I am seeing errors like this in gateway api controller logs and the target is not automatically registered in target group
E0511 21:55:34.435754       1 request.go:539] Failed request: VpcLattice/ListTargets, Payload: {  TargetGroupIdentifier: ""}, Error: InvalidParameter: 1 validation error(s) found.
93
- minimum field size of 17, ListTargetsInput.TargetGroupIdentifier.

Please advise, will provide more info upon request.
P/S: I've already created a support ticket in AWS (Case ID 12717202271), posting this here for visibility.

Can you provide us more details (like config yaml) so that we can reproduce this issue? More logs would be helpful as well.

Here's the full logs
About the config, I didn't do much custom configuration other than:

Putting all of the related resources in "vpc-lattice" namespace
Setting service account to use the role I've created
Creating a gateway in default namespace, then allowing all namespace to access the gateway

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  annotations:
    application-networking.k8s.aws/lattice-vpc-association: 'true'
  labels:
    app.kubernetes.io/instance: stag-eks-2-vpc-lattice
  name: lattice-gateway
  namespace: default
spec:
  gatewayClassName: amazon-vpc-lattice
  listeners:
    - allowedRoutes:
        namespaces:
          from: All
      name: http
      port: 80
      protocol: HTTP

We are looking into namespace issue that might be related to this. Can you also provide debug level log with GATEWAY_API_CONTROLLER_LOGLEVEL=debug?

Here's the debug logs, hope this helps!

Here's the output
I hope it's okay for me to just the relevant yaml, the full output will need some sanitisation before sharing out in the public

Can confirm that explicitly specifying namespace solves the issue.

authors: solmonk zhuoyang


text:
reading file twice (in PutObject)

Apparently AWS SDK needs to calculate some sort of signature before sending bytes over wire (in PutObject/etc) -- related stream gets read and then rewound to start and read again. Is there any way to avoid this double "read file from disk" besides using streambuf that keeps entire file in memory?

The default ctor args for s3 turn the payload hashing off of you are using tls. Also, setting the content length field will prevent the seek operations to figure it out. By controlling both of these settings, the file will only be read once.
Sent from my iPhone

On Oct 30, 2017, at 9:27 AM, crusader-mike notifications@github.com wrote:
Apparently AWS SDK needs to calculate some sort of signature before sending bytes over wire (in PutObject/etc) -- related stream gets read and then rewound to start and read again. Is there any way to avoid this double "read file from disk" besides using streambuf that keeps entire file in memory?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Huh... I am using https and reading file (in my own implementation of streambuf) only in underflow() -- according to your criteria SDK shouldn't scan my file twice. I'll double check tonight

After examining my (relatively recent) SDK sources I came to a conclusion that it is impossible to switch off "payload signing" without overriding request class methods -- all requests get payload signed if <request-class>::SignBody() returns true. Similar flag passed by S3Client into aws signer class gets used only with payload-less requests -- i.e. not used at all. The only way to avoid payload being signed is to derive from requess-class, override SignBody() and return false.
I dug in this a bit more and found that Amazon S3 (as of now) allows unsigned payloads, but you can configure your bucket to deny such requests.  Is it a prelude to total ban on unsigned payloads service wide?
In any case -- it would be very nice to have a switch in S3Client (or library-wide?) that would forcefully disable/enable payload signing (regardless of HTTP/HTTPS flag). Please :)
Until then I believe I have only these options:

live with 'double read'
load entire file into memory -- not end of the world, since I use multipart upload for large files anyway
override SignBody() of each request I use and hope users don't use HTTP urls in my program

Sounds like option #2 is the least of all evil right now.

This is a bug. We need to fix it.

Thank you. Would be nice to be able to skip payload signature in HTTP case too...

A note about option N2 -- multipart upload is limited to 10'000 chunks. Which means for files bigger than 50GB  chunk size will start growing and at some point they won't fit into memory (especially if you upload them in parallel). I.e. you'll have to limit memory buffer and you'll end up reading entire file twice.

Hi, you can avoid reading file twice by skipping payload signature with the latest update(ver 1.3.0) now:
auto s3Client = Aws::MakeShared<S3Client>(ALLOCATION_TAG, credentialsProvider, config, AWSAuthV4Signer::PayloadSigningPolicy::Never /*signPayloads*/, true /*useVirtualAddressing*/);

in which AWSAuthV4Signer::PayloadSigningPolicy::Never will overrule SignBody(), even though it's true, then the payload will never be signed.
Please consult this commit for more details: https://github.com/aws/aws-sdk-cpp/commit/b9ccf4d0ded1dccd1fe66fbba6b5c9c306fbb249

Thanks, Pushen!

@singku @wps132230 I've discovered that with HTTP endpoint my source files/objects are still being read twice.
Culprit is check for request.GetUri().GetScheme() != Http::Scheme::HTTPS check at this line.
Can someone explain why SDK insists on calculating payload hash on non-HTTPS endpoints even if I explicitly asked it not to "sign boody"?
Is there an easy way to switch it off without substituting request signer object?

authors: JonathanHenson crusader-mike marcomagdy wps132230

repo: aws/aws-sdk-cpp


text:
404 returned on any non-Home Controllers after deployment

Expected Behavior
Deploying to Elastic Beanstalk via  AWS via Visual Studio 2017 should make the application available at each controller that is defined.
Current Behavior
I am able to access pages from the Home controller, but all other controllers return 404.
Steps to Reproduce (for bugs)
Unsure what aspect is causing the issue.
Context
I am trying to deploy, as usual, to AWS Elastic Beanstalk.
Your Environment
AWSSDK 3.3.25 and AWSSDK 3.3.27.3
Localhost is running on Windows10 and works fine.
Configuration: IIS 10.0 running on 64bit Windows Server 2016/1.2.0
EC2 instance type: t1.micro
EC2 image ID: ami-19e8f179
Have you attached logs?: Yes
Visual Studio 2017
.NET Core 2.0 and .NET Core 2.1
.NET Core Info

.NET Core version used for development:
.NET Core 2.0 and .NET Core 2.1

.NET Core version installed in the environment where application runs: :
.NET Core 2.0 and .NET Core 2.1

dotnet --info Output
.NET Core SDK (reflecting any global.json):
Version:   2.1.300-preview2-008533
Commit:    996eb6c92c
Runtime Environment:
OS Name:     Windows
OS Version:  10.0.17134
OS Platform: Windows
RID:         win10-x64
Base Path:   C:\Program Files\dotnet\sdk\2.1.300-preview2-008533\
Host (useful for support):
Version: 2.1.1
Commit:  6985b9f684
.NET Core SDKs installed:
1.0.4 [C:\Program Files\dotnet\sdk]
1.1.0 [C:\Program Files\dotnet\sdk]
2.0.0 [C:\Program Files\dotnet\sdk]
2.0.2 [C:\Program Files\dotnet\sdk]
2.1.102 [C:\Program Files\dotnet\sdk]
2.1.201 [C:\Program Files\dotnet\sdk]
2.1.300-preview2-008533 [C:\Program Files\dotnet\sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.0-preview2-final [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.0-preview2-final [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 1.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 1.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.0-preview2-26406-04 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

Hi @Neuroforge, I see that you created three issues for this. I'm going to close this one as a duplicate, and we'll try to help you out on the one that remains.

authors: Neuroforge klaytaybai

repo: aws/aws-sdk-net


text:
Unexpected behaviour when picking up credentials

Describe the bug
When you do this:
services.AddDefaultAWSOptions(configuration.GetAWSOptions());
And you have this in your appsettings file:
"AWS": {
  "Profile": "my_profile"
}

You expect it to look up credentials in .aws\credentials under 'my_profile'. However, that is not what it does. It only looks up 'default' in that file, and settles on that. This is highly unexpected behaviour and leads to https://github.com/aws/aws-sdk-net/issues/499 and numerous of frustrated posts on stackoverflow.
Strictly speaking, it follows the behaviour described on https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/creds-assign.html, but it is highly unexpected, and should be changed. Many do not have 'default' credentials, and if they do, might end up using unintended credentials.
Kind regards, Emil
AWS .NET SDK and/or Package version used

Targeted .NET Platform
.NET 7
Operating System and version
Windows

Hi!
I can see that the Profile property is set, so that's not the problem. The problem is that it is only reading 'default' from the credentials file. I don't need to set the ProfilesLocation. It is reading the correct path (it is reading [default] from the correct file). Not using IIS.
Steps to reproduce:

Add [default] and [my_profile] to credentials file in user directory with access keys.
Set "Profile": "my_profile"
Run the application and observe that it uses the [default] credentials and not [my_profile] credentials.

Kind regards, Emil

"AWS": {
"Profile": "my_profile"
}

@emilmuller Good afternoon. I'm unsure what your setup looks like, however, I'm unable to reproduce the issue. For reproduction, I used the below steps:

Added the below test profile in credentials file (in addition to default profile.

[default]
aws_access_key_id = ASIA3MBXXXX
aws_secret_access_key = mVXXXX

[testprofile]
aws_access_key_id=testaccesskey
aws_secret_access_key=testsecretkey

Created a new ASP.NET Core Web Application (.NET 7).
Added the NuGet package references for AWSSDK.Extensions.NETCore.Setup and AWSSDK.S3, using the latest versions. Here's how the .csproj file looks like:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net7.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="3.7.7" />
    <PackageReference Include="AWSSDK.S3" Version="3.7.106" />
  </ItemGroup>

</Project>

Added the AWS key in appsettings.json:

{
  "AWS": {
    "Profile": "testprofile"
  },
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*"
}

Added the following in Program.cs:

builder.Services.AddDefaultAWSOptions(builder.Configuration.GetAWSOptions());
builder.Services.AddAWSService<IAmazonS3>();

In Index.cshtml.cs, added dependency on IAmazonS3 in constructor.

using Amazon.S3;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.RazorPages;

namespace TestNet7WebApp.Pages
{
    public class IndexModel : PageModel
    {
        private readonly ILogger<IndexModel> _logger;

        IAmazonS3 S3Client { get; set; }

        public IndexModel(ILogger<IndexModel> logger, IAmazonS3 s3Client)
        {
            _logger = logger;
            this.S3Client = s3Client;
        }

        public void OnGet()
        {

        }
    }
}

Setup breakpoint in Index.cshtml.cs and debug the application in VS 2022. When the breakpoint is hit, inspect the Credentials property of AmazonS3Client object.

RESULT:
AmazonS3Client is using the the credentials setup in testprofile in user's credentials file as shown below:

authors: ashishdhingra emilmuller


text:
read_parquet function takes up a lot of memory even before it returns the iterable object

Describe the bug
The documentation of read_parquet function suggests that using the 'chunked' argument makes the function memory friendly as it will return an iterable of dataframes instead of a regular dataframe. However, when tested with a 500MB parquet file, with chunked = 1, the function takes up more than 7GB memory even before returning the iterable object. That indicates the function is doing something underneath (possibly loading the file in memory) before being able to give back a streamable object.
If this an expected behavior, then the function unfortunately cannot be considered as memory-friendly as it ends up taking up a lot of memory, and the documentation should explicitly specify that so that the users would know what to expect. If it is not the expected behavior, then it is possibly a bug.
Sharing our code below:

def get_table_chunks_from_s3_file(app_configs: dict, sqs_values_dict: dict):

    bucket = sqs_values_dict["BucketName"]
    key = sqs_values_dict["ObjectKey"]
    boto3_session = app_configs["boto3_session"]

    file_path = "s3://" + bucket + "/" + key

    # Below function call takes up high memory before being able to return the dataframes object.
    dataframes = wr.s3.read_parquet(
        path=file_path, chunked=1, boto3_session=boto3_session
    )

    for dataframe in dataframes:
        yield pyarrow.table(dataframe)

Note that in the above, I have added a comment to share which part of the code takes up a lot of memory.
How to Reproduce
Run the read_parquet function for a relatively large parquet file in S3 and check how much memory it consumes (through a memory profiler) before giving back an iterable object.
Expected behavior
The function (as the documentation suggests) should not be taking up so much memory while trying to return an iterable of dataframes.
Your project
No response
Screenshots
No response
OS
Ubuntu 22.04
Python version
3.10.12
AWS SDK for pandas version
3.9.1
Additional context
Support Case ID: 172918156100319

Hi @geetparekh, note the rows in the parquet datasets are organized in row groups, and rows within a row group must be read in one go. Thus assumption that chunked=1 will result in lower memory footprint is not correct.
import boto3
import awswrangler as wr
import pyarrow as pa

# Path to a public parquet dataset 440.6MB file
path = "s3://ursa-labs-taxi-data/2009/01/data.parquet"
session = boto3.Session()

def get_table_chunks_from_s3_file(path, chunked, session):
	dataframes = wr.s3.read_parquet(path=path, chunked=chunked, boto3_session=session)
	yield from dataframes

# this line runs immediately and returns a generator as expected
g = get_table_chunks_from_s3_file(path, 1000000, session)

# this line consumes the first item from the generator and loads 1000000 records
next(g)

The parquet file used for the test has 14092413 in 216 row groups:
> parquet-tools inspect data.parquet

############ file meta data ############
created_by: parquet-cpp version 1.5.1-SNAPSHOT
num_columns: 18
num_rows: 14092413
num_row_groups: 216
format_version: 1.0
serialized_size: 324078

...

Reading with chunked=1000000 memory never peaked above 1G... Given Parquet's efficient compression, this is expected.
I will continue doing some tests to reproduce your issue and keep you updated.
Possibly related to

https://github.com/aws/aws-sdk-pandas/issues/2831
https://github.com/aws/aws-sdk-pandas/issues/2086

Thank you @kukushking for explaining that. Our parquet file only has one row group. So, that kind of explains that the function would try to load the whole file in memory in our case.
Would definitely suggest to explicitly add that in the documentation so that the users would know about it.

Thank you @kukushking for explaining that. Our parquet file only has one row group. So, that kind of explains that the function would try to load the whole file in memory in our case.
Would definitely suggest to explicitly add that in the documentation so that the users would know about it.

On that note, you can possibly also consider including an example in the documentation explaining how much memory the function would end up consuming given an example of number of row groups in a given parquet file and the chunk size requested.
For example, if a parquet file has 100K records with 100 row groups (for simplicity), each row group would roughly have 1000 rows. If the requested chunk size is 5000, the function will end up loading 5 row groups in memory before being an able to return the iterable object?

authors: geetparekh kukushking

repo: aws/aws-sdk-pandas


text:
Support for AWS::CloudFront::Function and FunctionAssociations in CloudFormation template editor

This is a feature request to update the Toolkit's CloudFormation autocompletion/schema to work with AWS::CloudFront::Function and FunctionAssociations

We've updated the definitions used by the Toolkit. The next time you start Visual Studio, the Toolkit should download the updated definitions file, and these fields should no longer be classified as an error.

repo: aws/aws-toolkit-visual-studio