text:
ci-cmd assumes that Athena is deployed to the same region with identity region

During CUDOS Dashboard  deploy with cid-cmd deploy In the case where user has configured  identity region in different region with QuickSight region.
e.g. I have my identity in eu-central-1 but Athena is in us-east-1
Following error will be thrown during QuickSight dataset creation
An error occurred (AccessDeniedException) when calling the ListUsers operation: Operation is being called from endpoint us-east-1, but your identity region is eu-central-1. Please use the eu-central-1 endpoint.
extend log
There are still 5 datasets missing: compute_savings_plan_eligible_spend, customer_all, ec2_running_cost, s3_view, summary_view
? Select AWS Athena database to use athenacurcfn_vmindru_cudos
unable to create, missing permissions: An error occurred (AccessDeniedException) when calling the ListUsers operation: Operation is being called from endpoint us-east-1, but your identity region is eu-central-1. Please use the eu-central-1 endpoint.
Creating dataset: customer_all...Checking if CUR is enabled and available...
        Athena table: vmindru_cudos
        Resource IDs: yes
        SavingsPlans: no
        Reserved Instances: no
done
unable to create, missing permissions: An error occurred (AccessDeniedException) when calling the ListUsers operation: Operation is being called from endpoint us-east-1, but your identity region is eu-central-1. Please use the eu-central-1 endpoint.
Creating dataset: ec2_running_cost...unable to create, missing permissions: An error occurred (AccessDeniedException) when calling the ListUsers operation: Operation is being called from endpoint us-east-1, but your identity region is eu-central-1. Please use the eu-central-1 endpoint.
Creating dataset: s3_view...unable to create, missing permissions: An error occurred (AccessDeniedException) when calling the ListUsers operation: Operation is being called from endpoint us-east-1, but your identity region is eu-central-1. Please use the eu-central-1 endpoint.
Creating dataset: summary_view...unable to create, missing permissions: An error occurred (AccessDeniedException) when calling the ListUsers operation: Operation is being called from endpoint us-east-1, but your identity region is eu-central-1. Please use the eu-central-1 endpoint.

Checking AWS environment...
        profile name: default
        accountId: 167515602781
        AWS userId: assumed-role/Admin/vmindru-Isengard
        Region: us-east-1
done

For now maybe just add a point to  "Before you start, stating that all elements, S3 bucket, Athena, QuickSight, IdentityProvider have to be in the same region" :)

Fixed in #231


text:
Small tweaks

A few updates to wording to fix typos or incorrect statements.  Plus a note on local login.

cc @yanivbos

source: gharchive/pull-request

license_type: permissive

license_source: github-api


text:
PostgreSQL SingleUser rotation, isn't working with RDS-Proxy

Recently, we connected an RDS proxy to our RDS PostgreSQL cluster.
Then we changed the host url in the secret from the cluster endpoint to the RDS endpoint.
We noticed that the secrets were not rotated then.
I then copied the code of the function and added some debug messages.
The following errors came to light:
[ERROR]	SQL-CONNECTION-ERROR: SCRAM authentication requires libpq version 10 or above
[ERROR]	SQL-CONNECTION-ERROR: FATAL:  This RDS Proxy requires TLS connections

It would be great if the function could also handle the proxy.

EDIT:
The SCRAM and TLS problems are gone, by updating the lambda.
Now the problem is, that during the rotation the new generated secret has been created as "pending".
Then the password from "pending", will be set in the database and the function checks, if the connection will work, with the new password.
At this point the function fails, because the rds proxy, will only read the credentials for the user, from the "current" version from the secret.
Maybe you can find a good practice for that, that the function can handle rotations via rds proxy.

Thank you for the report, we will look into the issue.

source: gharchive/issue

authors: fffelixdittrich jbct


text:
Fix/issue 802 - FW routes

I only add the suffix for the second route created so that it does not break current installation. See comment below :
    // Since we want to continue to support old installation, we only add the suffix for the second routes and onward since
    // they probably don't exist (see https://github.com/aws-samples/aws-secure-environment-accelerator/issues/802)
    // If we fixed the bug for all routes, the stack would fail on first run because it would try to create the
    // new route without deleting the old one first and it is not possible to have multiple routes with the same destination

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Thank you for contributing to ASEA.   Note that customers should look at using Landing Zone Accelerator on AWS (LZA)  We're currently only merging security and critical bug fixes.  However all ASEA PRs and Issues are noted by the LZA team while developing that solution.  Thank you!

authors: archikierstead fredbonin


text:
CloudFormationCreate buttons don't work

Context
When clicking on the cloudformation create button, it doesn't work properly.
We get this error
S3 error: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

Thank you for raising this. Cloudformation create buttons have now been removed in this PR: https://github.com/aws-samples/ecs-service-connect-yelb-sample-app/pull/11


text:
ユースケースビルダーで作成したユースケースでは、クリアを押してもコンテキストが残る

バグの説明
ユースケースビルダーを利用しているユーザーとして、「クリア」を押したら入力欄がクリアされるためコンテキストもクリアされて最初からやり取りを始められると感じるが、実際はテキストエリアの内容がクリアされるだけでコンテキストは残っている。
例えば、「入力テキスト」から「メール文」を作成するユースケースを作成し、一度実行した後「クリア」を押して、再度入力テキストを入力すると 2 回目の「メール文」は①前回の入力テキスト+②前回のメール文+③再度の入力テキストをコンテキストとして作成される。
再現手順
その振る舞いを再現する手順:

ユースケースビルダーに移動する
適当なユースケースを作成する
ユースケースに入力を与え出力を得る
クリアボタンを押す
再度ニュースケースに入力を与え出力を得る
ビルダーモードを中断し、会話履歴を参照すると 5 の出力が 3 の入出力を与えたうえで出されたものであることが確認できる。

想定される動作
クリアを押したら新規チャットとして処理される。あるいは、「最初からやり直す」という入力欄のクリアとは異なる機能を持つボタンを新設する。
(コードを見たうえでの所感)
useChat には clear 以外に init が存在するが (新規 id でチャットをやり直す?) 、ユースケースビルダーでは clear のみが使用される = 一度採番された id でやり取りが続くように見える。
https://github.com/aws-samples/generative-ai-use-cases-jp/blob/b8632d1dd429ca4e674ec5308a8289ead1ddd3ed/packages/web/src/components/useCaseBuilder/UseCaseBuilderView.tsx#L95

ご報告ありがとうございます！
本件は「クリアボタンを押さずに、手動で入力欄をクリアした場合」の事象ではないでしょうか？
手元の環境でクリアボタンの挙動を確認しましたところ、別の会話で履歴が作成されました。
少し細かい話になりますが、クリアボタンが押されたタイミングでuseChatのclearを実行しており、ここで会話が初期化されます。
https://github.com/aws-samples/generative-ai-use-cases-jp/blob/b8632d1dd429ca4e674ec5308a8289ead1ddd3ed/packages/web/src/components/useCaseBuilder/UseCaseBuilderView.tsx#L94
https://github.com/aws-samples/generative-ai-use-cases-jp/blob/4b44220159552d72c5067bf71fca18ae386d5f5b/packages/web/src/components/useCaseBuilder/UseCaseBuilderView.tsx#L159-L162
（Undocumentedな仕様で恐縮なのですが、）会話履歴上は会話が続いているように見えますが、ignoreHistory=trueでpostChatを実行しているため、会話履歴が無視され一問一答の形式で回答が生成される動きとなります。
ユースケースビルダーに限らず、標準の機能（翻訳など）もこちらの方法で実装をしています。
https://github.com/aws-samples/generative-ai-use-cases-jp/blob/4b44220159552d72c5067bf71fca18ae386d5f5b/packages/web/src/components/useCaseBuilder/UseCaseBuilderView.tsx#L152
https://github.com/aws-samples/generative-ai-use-cases-jp/blob/4b44220159552d72c5067bf71fca18ae386d5f5b/packages/web/src/hooks/useChat.ts#L365-L369

今確認したところ、クリアボタンを押せばリセットされることを確認しました！(お手間をかけて申し訳ない・・・)
ご確認ありがとうございます。

repo: aws-samples/generative-ai-use-cases-jp


text:
PermissionError when ./setup/ros_setup.bash at first

my IAM accont has admin permission, when I execute in Cloud9,
git clone https://github.com/aws-samples/multi-robot-fleet-sample-application.git
cd multi-robot-fleet-sample-application
./setup/ros_setup.bash
it has error:
File "/usr/local/lib/python3.6/dist-packages/colcon_core/command.py", line 528, in verb_main
rc = context.args.main(context=context)
File "/usr/local/lib/python3.6/dist-packages/colcon_bundle/verb/bundle.py", line 143, in main
dependencies_changed)
File "/usr/local/lib/python3.6/dist-packages/colcon_bundle/verb/_archive_generators.py", line 103, in generate_archive_v2
workspace_tar_gz_path)
File "/usr/local/lib/python3.6/dist-packages/colcon_bundle/verb/_overlay_utilities.py", line 32, in create_workspace_overlay
_CONTEXT_VAR_BASH
File "/usr/local/lib/python3.6/dist-packages/colcon_bundle/verb/_overlay_utilities.py", line 156, in _generate_template
with open(script_location, 'w') as file:
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/colcon_bundle/verb/assets/v2_workspace_setup.bash'

I found /usr/local/lib/python3.6/dist-packages/colcon_bundle/verb/assets/v2_workspace_setup.bash is not exist

This issue should be resolved with https://github.com/aws-samples/multi-robot-fleet-sample-application/pull/11
Can you confirm @qqhzg

Confirmed from @qqhzg  that this issue is resolved. closing issue.

authors: konduri qqhzg


text:
Ruby Code Snippet for Lambda SQS Integration

Issue #, if available: https://github.com/aws-samples/serverless-snippets/issues/93
Description of changes: Ruby Code Snippet for Lambda SQS Integration
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Hey David, Yes I have provided the python code previously. And now the Ruby.
Girish B provided the Java code. There was a typo mistake previously as I got the template from him.
I corrected it in this request.

@boyney123 Can you give guidance on the $.snippets[0].codeTabs[].snippets[].language value mapping for Ruby?
Looks like many languages are using the file extension (eg, Python has "language":"py"), but .NET files with .cs extensions have "language":"dotnet".  Is there a documented schema with an expected enum variant that we should be following?
I just approved a PR for Ruby with "language":"ruby", since syntax-highlighting for Markdown code-blocks use
  ```ruby

as the label.

Hey @dpruessner
Yeah ruby should be fine, we use this list here https://prismjs.com/#supported-languages

@ssnskar and @dpruessner are we good to merge this now? Had the ruby code been reviewed?

@ssnskar and @dpruessner are we good to merge this now? Had the ruby code been reviewed?

Yep; good-to-go. Updating Asana

authors: boyney123 dpruessner ssnskar

repo: aws-samples/serverless-snippets


text:
Document how to build using Cloud 9

Customer requested instructions on how to build the solution using Cloud9. Include region-specific instructions.

This seems out-of-scope. Closing for now.


text:
Support multiple auto stop hour configurations

The current solution applies the default of 1 hour when moving workspaces to auto stop mode, requesting that this be moved to a tagged value as we have some workspaces that run tasks with no interactivity and would like them to auto stop after 8 hours, where other workspaces can auto stop at the default of 1 hour.

Thank you for adding this to the backlog!
Correct, the issue is with the blanket number of hours for all workspaces.  I.E. I'm assuming this is the AutoStopTimeoutHours parameter I see defined in the task definition with a value of 1.  The request would be to se this on a dynamic basis using a tag on the instances themselves, or based on bundle/directory ID which complicates things more in my opinion.
I.E. Workspace ID's ws-11111111 and ws-22222222 belong to bundle Developer_Bundle and an AutoStop time of 1 hours works great.  Workspace ID's ws-33333333 and ws-44444444 belong to bundle Automation_Bundle are used to run some automation where users do not show as active (They login, start the task and the window sits idle), so we would want to set their AutoStop time to a longer interval such as 8 hours.
Hope this helps!

Thanks for sharing more details. Much appreciated.

authors: CingularIT groverlalit


text:
can I use dynamic allocation for MQTT?

Hi, Amazon FreeRTOS expert
The buffer allocation scheme uses a static method. In my use case, I need use Shadow_Get API to get the full document from the cloud and its size is quite large (close to 8KB), and if the block number assigned is too small, I will get the message drop hint. So I have to assign 20 x 8KB = 160KB buffer dedicated to MQTT, it wastes a lot for the most time.
Can I use replace the GET and FREE pointer by pvPortMalloc() and vPortFree()? So that I can use the memory more efficient. Are there any risks? (alignment requirement or fragmentation?)
BTW, I'm using the heap_4 algorithm.

Hi @xiongyu0523,
Yes, you can safely replace the current statically-allocated buffer functions with implementations using pvPortMalloc and vPortFree.  I don't see any risks in doing this.

repo: aws/amazon-freertos

license: MIT


text:
Linux simulator support?

I was curious as to what the plans were regarding the Linux simulator, now that FreeRTOS is under AWS leadership.
The Linux simulator was a community port, and was released under the old FreeRTOS GPLv2 license. However, now that FreeRTOS is MIT licensed, this makes using the Linux port somewhat difficult, as from what I understand, you cannot use GPL-licensed code in a MIT-licensed project without relicensing your code.
Are there plans on officially adding support for a Linux port/simulator? Or is there a way to run the Windows port on Linux?
Thanks!

Hi there,
Unfortunately we don't have an officially-supported Linux simulator right now, but thank you for the suggestion and feedback; it helps us prioritize our work internally.

that sucks!

A Linux simulator is very useful.

Hi @elagon
Thanks for your interest in FreeRTOS!
We actually do have a linux simulator!
https://github.com/FreeRTOS/FreeRTOS/pull/37
https://github.com/FreeRTOS/FreeRTOS-Kernel/pull/30
Thanks

authors: WRansohoff alfred2g elagon karlding


text:
Rename aws_iot_network_config.h files to iot_network_config.h

Rename aws_iot_network_config.h files to iot_network_config.h
Description
Rename aws_iot_network_config.h files which are not aws-specific.
Checklist:

[ ] I have tested my changes. No regression in existing tests.
[ ] My code is Linted.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/bot run checks


text:
fix: make agc logs adapter handle Toil

Issue #, if available: #430
Description of Changes
This changes agc logs adapter so that it doesn't try to fetch anything for a Toil context, and instead tells the user that there's no adapter.
Description of how you validated changes
I added a test, and I also tried them on the command line:
[anovak@courtyard demo-cwl-project]$ agc logs adapter -c myContext
Error: Context does not use an adapter because it is using the Toil engine

Checklist

[X] If this change would make any existing documentation invalid, I have included those updates within this PR
[X] I have added unit tests that prove my fix is effective or that my feature works
[X] I have linted my code before raising the PR
[X] Title of this Pull Request follows Conventional Commits standard: https://www.conventionalcommits.org/en/v1.0.0/

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

OK, I set this up to use a UsesWesAdapter map, but I had to tie it in with the information about what engines need the adapter that was already in environment/environment.go to avoid duplicating this information. And to do that I had to standardize on the lowercase names for everything from constants/engines.go instead of the upper-case image keys that had been in use in some places.
I think I have it all set up in a sensible way now; I've thrown out the dependency resolution logic I had before in favor of just checking the do-we-need-an-adapter flag. The only downside is that UsesWesAdapter is coming from environment instead of some kind of engines package. Should it maybe move?

repo: aws/amazon-genomics-cli


text:
fix: add vite coverage and resolve various lints

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Codecov Report
Attention: Patch coverage is 40.38462% with 31 lines in your changes missing coverage. Please review.

Project coverage is 15.66%. Comparing base (9ef3f57) to head (9bf82d9).

Files with missing lines
Patch %
Lines

packages/eslint-config/src/common.ts
0.00%
13 Missing :warning:

packages/autocomplete-parser/src/parseArguments.ts
0.00%
4 Missing :warning:

packages/autocomplete/src/hooks/keypress.ts
0.00%
3 Missing :warning:

packages/autocomplete/src/fig/themes.ts
33.33%
2 Missing :warning:

packages/api-bindings/src/event.ts
0.00%
1 Missing :warning:

packages/autocomplete-parser/src/loadSpec.ts
75.00%
1 Missing :warning:

...ackages/autocomplete/src/components/Suggestion.tsx
0.00%
1 Missing :warning:

...ges/autocomplete/src/components/SuggestionIcon.tsx
0.00%
1 Missing :warning:

...ete/src/generators/templateSuggestionsGenerator.ts
0.00%
1 Missing :warning:

packages/autocomplete/src/history/index.ts
0.00%
1 Missing :warning:

... and 3 more

Additional details and impacted files
@@             Coverage Diff             @@
##             main     #209       +/-   ##
===========================================
+ Coverage   15.51%   15.66%    +0.14%
===========================================
  Files        1502      187     -1315
  Lines      134029    16755   -117274
  Branches   134029      670   -133359
===========================================
- Hits        20795     2624    -18171
+ Misses     112037    14131    -97906
+ Partials     1197        0     -1197

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

repo: aws/amazon-q-developer-cli


text:
Close socket connection after curl handle is released to handle pool

My application is multithreading application and upload/download many files at the same time. After it goes into idle state, I can see it still keep many socket connections at "CLOSE_WAIT".
Look at the SDK source code, it's clear that SDK will only call curl_easy_cleanup during its shutdown.
I suggest SDK should call curl_easy_cleanup once a curl handle is going to be returned to pool.

Cleaning up curl handles after every use would substantially degrade the SDK's throughput when hitting the same endpoint, since every request would be made from scratch with no TLS caching.  If you want, you can certainly reduce/limit the size of the connection pool to keep the number of sockets used to be as small as you'd like (see ClientConfiguration::maxConnections).

I understand the intention that SDK keeps a pool of curl handles. However, from technical point of view, a long time "CLOSE_WAIT" socket sometimes is explained as a programming error. From end user's point of view, a system administrator may ask why your application keep so many sockets in "CLOSE_WAIT" state.
Reducing max connection is not an option as my application is a multithreading application and can be very busy sometimes (not always).
On the other hand, I am curious about if there is other way to reduce the performance impact if we close socket every time. I have no much experience with libcurl. But I guess you can implement a usage monitor logic: allocate (and keep) more handles when busy, release (and close) handle when idle.
You mentioned DNS caching. But I guess if that really takes effect. AWS S3 service seems to be a DNS load balance system. We will get different IP address for every time we try to resolve the DNS name. Testing result also shows that the SDK use many different IP address.

A TLS connection is three round trips to the server just to setup a socket. This would be a heavy penalty to pay for the application. Just about every http application maintains a connection pool under the hood. I don't see why this is a problem.
Sent from my iPhone

On Jan 4, 2017, at 7:04 PM, mikewang2016 notifications@github.com wrote:
I understand the intention that SDK keeps a pool of curl handles. However, from technical point of view, a long time "CLOSE_WAIT" socket sometimes is explained as a programming error. From end user's point of view, a system administrator may ask why your application keep so many sockets in "CLOSE_WAIT" state.
Reducing max connection is not an option as my application is a multithreading application and can be very busy sometimes (not always).
On the other hand, I am curious about if there is other way to reduce the performance impact if we close socket every time. I have no much experience with libcurl. But I guess you can implement a usage monitor logic: allocate (and keep) more handles when busy, release (and close) handle when idle.
You mentioned DNS caching. But I guess if that really takes effect. AWS S3 service seems to be a DNS load balance system. We will get different IP address for every time we try to resolve the DNS name. Testing result also shows that the SDK use many different IP address.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

and the sockets should only be staying open as long as the keep alive from the server specifies which is in keeping with the spec.
Sent from my iPhone

On Jan 4, 2017, at 7:04 PM, mikewang2016 notifications@github.com wrote:
I understand the intention that SDK keeps a pool of curl handles. However, from technical point of view, a long time "CLOSE_WAIT" socket sometimes is explained as a programming error. From end user's point of view, a system administrator may ask why your application keep so many sockets in "CLOSE_WAIT" state.
Reducing max connection is not an option as my application is a multithreading application and can be very busy sometimes (not always).
On the other hand, I am curious about if there is other way to reduce the performance impact if we close socket every time. I have no much experience with libcurl. But I guess you can implement a usage monitor logic: allocate (and keep) more handles when busy, release (and close) handle when idle.
You mentioned DNS caching. But I guess if that really takes effect. AWS S3 service seems to be a DNS load balance system. We will get different IP address for every time we try to resolve the DNS name. Testing result also shows that the SDK use many different IP address.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Many CLOSE_WAIT connections hanging is a bug, which it is a base knowledge, very clear. It is because client doesn't execute close when read return code is 0. See the answers below, we cannot see any investigations, but perfunctoriness. This is the consistent attitude when Amazon/AWS employees face to problems/questions/bugs. I get use to it.

authors: JonathanHenson bretambrose earnestzhao mikewang2016

repo: aws/aws-sdk-cpp


text:
Build test dependencies only if testing is enabled.

Issue #, if available:
#632
Description of changes:
I have noticed that test dependencies are always being built, even if I configure with -DENABLE_TESTING=OFF. This PR fixes this.
Check all that applies:

[X] Did a review by yourself.
[ ] Added proper tests to cover this PR. (If tests are not applicable, explain.)
[X] Checked if this PR is a breaking (APIs have been changed) change.
[X] Checked if this PR will not introduce cross-platform inconsistent behavior.
[ ] Checked if this PR would require a ReadMe/Wiki update.

Check which platforms you have built SDK on to verify the correctness of this PR.

[ ] Linux
[x] Windows
[ ] Android
[ ] MacOS
[ ] IOS
[ ] Other Platforms

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Thank you for noticing and fixing this!

authors: SergeyRyabinin teo-tsirpanis


text:
Amazon.Runtime.AmazonUnmarshallingException

The Amazon.Runtime.AmazonUnmarshallingException exception is being thrown whenever I execute the following code snippet.
        public List<string> GetCacheable() {
            List<string> objects = new List<string>();
            ListObjectsRequest request = new ListObjectsRequest {
                BucketName = Zoftware.Bucket,
                MaxKeys = 1000
            };
            var counter = 0;
            while (true) {
                ListObjectsResponse response = this.client.ListObjects(request);
                foreach (var o in response.S3Objects) {
                    if (o.Key.EndsWith(".mp3") || o.Key.EndsWith("album_art")) {
                        objects.Add(o.Key);
                        counter++;
                        Console.WriteLine("Counter: " + counter);
                        Console.WriteLine("Key: " + o.Key);
                    }
                }
                if (response.IsTruncated)
                    request.Marker = response.NextMarker;
                else
                    break;
            }
            return objects;
        }

The exception is thrown after about 5800 objects have been fetched and listed already.
This is the response body:
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>rzmedia</Name><Prefix></Prefix><Marker>music/Igge Scoce/Chillout Jazz Bar/med/Cat in the Kitchen.mp3</Marker><MaxKeys>1000</MaxKeys><IsTruncated>true</IsTruncated><Contents><Key>music/Iggy Azalea/The New Classic/album_art</Key><LastModified>2015-03-06T17:29:55.000Z</LastModified><ETag>"f3be59923760289cbfd7e6be010c3411"</ETag><Size>150472</Size><Owner><ID>a5f69b070192f392962aeaf6c494c6dda8f08ec7a04a3b32dbe17af6eca1913f</ID><DisplayName>locao.org</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>music/Iggy Azalea/The New Classic/med/Black Widow.mp3</Key><LastModified>2015-03-06T17:28:53.000Z</LastModified><ETag>"b2ff8f32b953beee4bb8087c8f6e447d"</ETag><Size>3504128</Size><Owner><ID>a5f69b070192f392962aeaf6c494c6dda8f08ec7a04a3b32dbe17af6eca1913f</ID><DisplayName>locao.org</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents

And this is the stack trace:
Amazon.Runtime.AmazonUnmarshallingException was unhandled
  HResult=-2146233088
  Message=Error unmarshalling response back from AWS. Response Body: <?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>rzmedia</Name><Prefix></Prefix><Marker>music/Igge Scoce/Chillout Jazz Bar/med/Cat in the Kitchen.mp3</Marker><MaxKeys>1000</MaxKeys><IsTruncated>true</IsTruncated><Contents><Key>music/Iggy Azalea/The New Classic/album_art</Key><LastModified>2015-03-06T17:29:55.000Z</LastModified><ETag>"f3be59923760289cbfd7e6be010c3411"</ETag><Size>150472</Size><Owner><ID>a5f69b070192f392962aeaf6c494c6dda8f08ec7a04a3b32dbe17af6eca1913f</ID><DisplayName>locao.org</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>music/Iggy Azalea/The New Classic/med/Black Widow.mp3</Key><LastModified>2015-03-06T17:28:53.000Z</LastModified><ETag>"b2ff8f32b953beee4bb8087c8f6e447d"</ETag><Size>3504128</Size><Owner><ID>a5f69b070192f392962aeaf6c494c6dda8f08ec7a04a3b32dbe17af6eca1913f</ID><DisplayName>locao.org</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents
  Source=AWSSDK
  RequestId=""
  ResponseBody=<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>rzmedia</Name><Prefix></Prefix><Marker>music/Igge Scoce/Chillout Jazz Bar/med/Cat in the Kitchen.mp3</Marker><MaxKeys>1000</MaxKeys><IsTruncated>true</IsTruncated><Contents><Key>music/Iggy Azalea/The New Classic/album_art</Key><LastModified>2015-03-06T17:29:55.000Z</LastModified><ETag>"f3be59923760289cbfd7e6be010c3411"</ETag><Size>150472</Size><Owner><ID>a5f69b070192f392962aeaf6c494c6dda8f08ec7a04a3b32dbe17af6eca1913f</ID><DisplayName>locao.org</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>music/Iggy Azalea/The New Classic/med/Black Widow.mp3</Key><LastModified>2015-03-06T17:28:53.000Z</LastModified><ETag>"b2ff8f32b953beee4bb8087c8f6e447d"</ETag><Size>3504128</Size><Owner><ID>a5f69b070192f392962aeaf6c494c6dda8f08ec7a04a3b32dbe17af6eca1913f</ID><DisplayName>locao.org</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents
  StackTrace:
       em Amazon.Runtime.Internal.Unmarshaller.Unmarshall(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\Handlers\Unmarshaller.cs:linha 149
       em Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\Handlers\Unmarshaller.cs:linha 53
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.S3\Internal\AmazonS3ResponseHandler.cs:linha 41
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\ErrorHandler\ErrorHandler.cs:linha 79
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\Handlers\CallbackHandler.cs:linha 45
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.Runtime.Internal.Signer.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\Handlers\Signer.cs:linha 38
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.Runtime.Internal.CredentialsRetriever.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\Handlers\CredentialsRetriever.cs:linha 72
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\RetryHandler\RetryHandler.cs:linha 84
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\Handlers\CallbackHandler.cs:linha 45
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.S3.Internal.AmazonS3KmsHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.S3\Internal\AmazonS3KmsHandler.cs:linha 34
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.Runtime.Internal.EndpointResolver.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\Handlers\EndpointResolver.cs:linha 34
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.S3.Internal.AmazonS3PostMarshallHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.S3\Internal\AmazonS3PostMarshallHandler.cs:linha 43
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.Runtime.Internal.Marshaller.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\Handlers\Marshaller.cs:linha 37
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.S3.Internal.AmazonS3PreMarshallHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.S3\Internal\AmazonS3PreMarshallHandler.cs:linha 45
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\Handlers\CallbackHandler.cs:linha 45
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.S3\Internal\AmazonS3ExceptionHandler.cs:linha 39
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\Handlers\ErrorCallbackHandler.cs:linha 44
       em Amazon.Runtime.Internal.PipelineHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\PipelineHandler.cs:linha 57
       em Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\Handlers\MetricsHandler.cs:linha 40
       em Amazon.Runtime.Internal.RuntimePipeline.InvokeSync(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\RuntimePipeline.cs:linha 135
       em Amazon.Runtime.AmazonServiceClient.Invoke[TRequest,TResponse](TRequest request, IMarshaller`2 marshaller, ResponseUnmarshaller unmarshaller) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\AmazonServiceClient.cs:linha 130
       em Amazon.S3.AmazonS3Client.ListObjects(ListObjectsRequest request) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet45\Amazon.S3\AmazonS3Client.cs:linha 1947
       em RadioZeroPlayer.Cache.GetCacheable() na c:\Users\Martin\Desktop\RadioZero\player\RadioZero Player\Cache.cs:linha 25
       em RadioZeroPlayer.UpdateCacheForm..ctor(Button controller) na c:\Users\Martin\Desktop\RadioZero\player\RadioZero Player\UpdateCacheForm.cs:linha 28
       em RadioZeroPlayer.FormRadioPlayerMain.button1_Click(Object sender, EventArgs e) na c:\Users\Martin\Desktop\RadioZero\player\RadioZero Player\FormRadioPlayerMain.cs:linha 326
       em System.Windows.Forms.Control.OnClick(EventArgs e)
       em System.Windows.Forms.Button.OnClick(EventArgs e)
       em System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
       em System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
       em System.Windows.Forms.Control.WndProc(Message& m)
       em System.Windows.Forms.ButtonBase.WndProc(Message& m)
       em System.Windows.Forms.Button.WndProc(Message& m)
       em System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
       em System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
       em System.Windows.Forms.NativeWindow.DebuggableCallback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
       em System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
       em System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData)
       em System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
       em System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
       em System.Windows.Forms.Application.Run(Form mainForm)
       em RadioZeroPlayer.Program.Main() na c:\Users\Martin\Desktop\RadioZero\player\RadioZero Player\Program.cs:linha 10
       em System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
       em System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       em Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       em System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       em System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       em System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       em System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       em System.Threading.ThreadHelper.ThreadStart()
  InnerException: System.Xml.XmlException
       HResult=-2146232000
       Message=' ', valor hexadecimal 0x01, é um caractere inválido. Linha 2, posição 271616.
       Source=System.Xml
       LineNumber=2
       LinePosition=271616
       SourceUri=""
       StackTrace:
            em System.Xml.XmlTextReaderImpl.Throw(Exception e)
            em System.Xml.XmlTextReaderImpl.Throw(String res, String[] args)
            em System.Xml.XmlTextReaderImpl.Throw(Int32 pos, String res, String[] args)
            em System.Xml.XmlTextReaderImpl.ParseNumericCharRefInline(Int32 startPos, Boolean expand, StringBuilder internalSubsetBuilder, Int32& charCount, EntityType& entityType)
            em System.Xml.XmlTextReaderImpl.ParseCharRefInline(Int32 startPos, Int32& charCount, EntityType& entityType)
            em System.Xml.XmlTextReaderImpl.ParseText(Int32& startPos, Int32& endPos, Int32& outOrChars)
            em System.Xml.XmlTextReaderImpl.ParseText()
            em System.Xml.XmlTextReaderImpl.ParseElementContent()
            em System.Xml.XmlTextReaderImpl.Read()
            em Amazon.Runtime.Internal.Transform.XmlUnmarshallerContext.ReadElement() na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Internal\Transform\UnmarshallerContext.cs:linha 480
            em Amazon.Runtime.Internal.Transform.XmlUnmarshallerContext.Read() na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Internal\Transform\UnmarshallerContext.cs:linha 387
            em Amazon.Runtime.Internal.Transform.S3UnmarshallerContext.Read() na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Internal\Transform\UnmarshallerContext.cs:linha 580
            em Amazon.S3.Model.Internal.MarshallTransformations.ContentsItemUnmarshaller.Unmarshall(XmlUnmarshallerContext context) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.S3\Model\Internal\MarshallTransformations\ContentsItemUnmarshaller.cs:linha 36
            em Amazon.S3.Model.Internal.MarshallTransformations.ListObjectsResponseUnmarshaller.UnmarshallResult(XmlUnmarshallerContext context, ListObjectsResponse response) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.S3\Model\Internal\MarshallTransformations\ListObjectsResponseUnmarshaller.cs:linha 74
            em Amazon.S3.Model.Internal.MarshallTransformations.ListObjectsResponseUnmarshaller.Unmarshall(XmlUnmarshallerContext context) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.S3\Model\Internal\MarshallTransformations\ListObjectsResponseUnmarshaller.cs:linha 38
            em Amazon.Runtime.Internal.Transform.XmlResponseUnmarshaller.Unmarshall(UnmarshallerContext input) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Internal\Transform\ResponseUnmarshallers.cs:linha 94
            em Amazon.Runtime.Internal.Transform.S3ReponseUnmarshaller.Unmarshall(UnmarshallerContext input) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Internal\Transform\ResponseUnmarshallers.cs:linha 179
            em Amazon.Runtime.Internal.Transform.ResponseUnmarshaller.UnmarshallResponse(UnmarshallerContext context) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Internal\Transform\ResponseUnmarshallers.cs:linha 62
            em Amazon.Runtime.Internal.Unmarshaller.UnmarshallResponse(UnmarshallerContext context, IRequestContext requestContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\Handlers\Unmarshaller.cs:linha 208
            em Amazon.Runtime.Internal.Unmarshaller.Unmarshall(IExecutionContext executionContext) na d:\Jenkins\jobs\build-sdkandtools-release\workspace\sdk\src\AWSSDK_DotNet35\Amazon.Runtime\Pipeline\Handlers\Unmarshaller.cs:linha 134
       InnerException:

A related blog post may be found here: http://www.deplication.net/2015/01/aws-tip-of-day-working-around-s3-failed.html
My question is, in short, what's the easiest way to get around this issue? I tried what the blog post suggested, but I can't reuse the URL-encoded keys on subsequent GetObjectRequest calls.
Thanks in advance.

Nevermind. I fixed the problem by doing this:
public List<string> GetCacheable() {
            List<string> objects = new List<string>();
            ListObjectsRequest request = new ListObjectsRequest {
                BucketName = Zoftware.Bucket,
                MaxKeys = 1000,
                Encoding = new EncodingType("url")
            };
            while (true) {
                try {
                    ListObjectsResponse response = this.client.ListObjects(request);
                    foreach (var o in response.S3Objects) {
                        if (o.Key.EndsWith(".mp3") || o.Key.EndsWith("album_art")) {
                            string key = HttpUtility.UrlDecode(o.Key);
                            objects.Add(key);
                            Console.WriteLine("Key: " + key);
                        }
                    }
                    if (response.IsTruncated)
                        request.Marker = response.NextMarker;
                    else
                        break;
                } catch (Exception) {
                    return objects;
                }
            }
            return objects;
        }

repo: aws/aws-sdk-net


text:
DisableFetchingTableMetadata leads to key exception if PropertyConverter is used

Describe the bug
DisableFetchingTableMetadata does not account for PropertyConverters. See repro.
Expected Behavior
It shouldn't fail trying to save item with a strange message.
Current Behavior
System.InvalidOperationException: 'Key attribute Type must be of type Numeric'
Reproduction Steps

Repro
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.DataModel;
using Amazon.DynamoDBv2.DocumentModel;
using Amazon.DynamoDBv2.Model;
using Amazon.Runtime;

var config = new AmazonDynamoDBConfig { ServiceURL = $"http://localhost:8000" };
var credentials = new BasicAWSCredentials("access", "secret");
var dynamoDbClient = new AmazonDynamoDBClient(credentials, config);
var context = new DynamoDBContext(dynamoDbClient, new DynamoDBContextConfig
{
    DisableFetchingTableMetadata = true
});

var tables = await dynamoDbClient.ListTablesAsync();
if (!tables.TableNames.Contains("TestTable"))
{
    await dynamoDbClient.CreateTableAsync(new CreateTableRequest
    {
        TableName = "TestTable",
        AttributeDefinitions =
        [
            new AttributeDefinition
            {
                AttributeName = "Type",
                AttributeType = ScalarAttributeType.S
            },
            new AttributeDefinition
            {
                AttributeName = "Id",
                AttributeType = ScalarAttributeType.S
            }
        ],
        KeySchema =
        [
            new KeySchemaElement
            {
                AttributeName = "Type",
                KeyType = KeyType.HASH
            },
            new KeySchemaElement
            {
                AttributeName = "Id",
                KeyType = KeyType.RANGE
            }
        ],
        ProvisionedThroughput = new ProvisionedThroughput
        {
            ReadCapacityUnits = 1,
            WriteCapacityUnits = 1
        }
    });

    Console.WriteLine("Table created");
}

await context.SaveAsync(new TestType
{
    Id = "1",
    Type = TypeEnum.Value2,
    Name = "Test",
});
Console.WriteLine("Item saved.");

var item = await context.LoadAsync<TestType>(TypeEnum.Value2, "1");
Console.WriteLine($"Item loaded: {item}");

[DynamoDBTable("TestTable")]
record class TestType
{
    [DynamoDBHashKey]
    [DynamoDBProperty(Converter = typeof(EnumAsStringConverter<TypeEnum>))]
    public TypeEnum Type { get; set; }

    [DynamoDBRangeKey]
    public string Id { get; set; }

    public string Name { get; set; }
}

enum TypeEnum
{
    Value1,
    Value2
}

public class EnumAsStringConverter<T> : IPropertyConverter where T : struct
{
    public object FromEntry(DynamoDBEntry entry) => Enum.Parse<T>(entry.AsString());
    public DynamoDBEntry ToEntry(object value) => new Primitive(value.ToString());
}

The code abode fails with an exception:
System.InvalidOperationException: 'Key attribute Type must be of type Numeric'
It works fine if I set DisableFetchingTableMetadata to false.
Possible Solution
No response
Additional Information/Context
No response
AWS .NET SDK and/or Package version used
Amazon.Lambda.DynamoDBEvents 3.7.302.19
Targeted .NET Platform
.NET 8
Operating System and version
AmazonLinuix

Needs review with the team.

authors: Dreamescaper ashishdhingra


text:
Add thread safety to FallbackCredentialsFactory

Description
This PR adds locking around the cached credentials and refrains from saving temporary credential state in a static variable to avoid race conditions.
Motivation and Context
If called from multiple threads, the cached credentials in FallbackCredentialsFactory can be set to null temporarily for failing credential generators, and this can override successfully retrieved credentials from the other threads, causing false negative errors in retrieving credentials.
Similarly, if the factory cache is reset while getting credentials, the GetCredentials method might throw an exception even when it has an actual credential.
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[x] My code follows the code style of this project
[ ] My change requires a change to the documentation
[x] I have updated the documentation accordingly
[x] I have read the README document
[ ] I have added tests to cover my changes
[x] All new and existing tests passed

License

[x] I confirm that this pull request can be released under the Apache 2 license

Is this resolving the underlying issue causing this https://github.com/aws/aws-sdk-net/issues/1945 ?

Is this resolving the underlying issue causing this https://github.com/aws/aws-sdk-net/issues/1945 ?

That issue was resolved by https://github.com/aws/aws-sdk-net/pull/2624. However this is resolving a really similar issue.

Did a full dry run and it passed. Do you mind squashing your commits before we do the release? It will go out in our next manual release.

Squashed and ready to go.

PR has been released as part of version 3.7.201.9 of AWSSDK.Core

authors: alitas djepson-rms normj


text:
Allow retry delay cancellation

Description
Using the cancelation token to allow the Task.Delay to be canceled
Motivation and Context

Testing

Screenshots (if appropriate)
Types of changes

[X] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[x] My code follows the code style of this project
[ ] My change requires a change to the documentation
[ ] I have updated the documentation accordingly
[x] I have read the README document
[ ] I have added tests to cover my changes
[x] All new and existing tests passed

License

[X] I confirm that this pull request can be released under the Apache 2 license

Ping

@normj @sstevenkang @vellozzi Rebased


text:
fix: SQL injection

Feature or Bugfix

Bugfix

Detail

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: feee7f14ee1c1de15736660f7e8e12548b1ae4c6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

authors: LeonLuttenberger malachi-constant

repo: aws/aws-sdk-pandas


text:
chore: Replace bump2version with bump-my-version

Detail

bump2version is no longer being maintained. This has resulted in the fork bump-my-version, which is being maintained
The configuration for bumping versions has now been been moved to toml
The new configuration allows us to reconfigure our version bumping logic to be less brittle, as we no longer need to define each file separately

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 6f335a810c1a57ca302e5a5cdd05fdac14cfc3df
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
Commit ID: 6f335a810c1a57ca302e5a5cdd05fdac14cfc3df
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 1785483ab6c3cfef64d2891ef05b09f648ada9a3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
Commit ID: 1785483ab6c3cfef64d2891ef05b09f648ada9a3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: f7a57ca665d7177afed5dda1766b395dff0cf40b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
Commit ID: f7a57ca665d7177afed5dda1766b395dff0cf40b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jaidisido I have validated - works all right.

AWS CodeBuild CI Report

CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
Commit ID: 68cc34757a2d1ec5c754a02f3414cc717f7b11ca
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository


text:
Upgrade trivy and harbor-scanner-trivy for harbor v2.11.0

Description of changes:
This PR

Updates trivy to v0.51.2
Updates harbor-scanner-trivy to v0.31.2

Upstream source of truth:
https://github.com/goharbor/harbor/blob/v2.11.0/Makefile#L107-L108
Testing
make build
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/lgtm

/approve

repo: aws/eks-anywhere-build-tooling


text:
Install emulator only for architecture not matching with host

If the binfmt-misc image is used to install the emulator corresponding to the host architecture itself, it throws an error.
On AMD64 machines
installing: amd64 cannot register "/usr/bin/qemu-x86_64" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: no such file or directory

On ARM64 machines
installing: arm64 cannot register "/usr/bin/qemu-aarch64" to /proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register: no such file or directory

So we are changing the script to not install the emulator for the host architecture.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/cherrypick release-0.21

/cherrypick release-0.20

/approve

/override troubleshoot-tooling-presubmit


text:
controller-runtime version compatibility with dependencies

What happened:
Dependencies of EKS Anywhere that use newer versions of https://github.com/kubernetes-sigs/controller-runtime cannot be used. For example Rufio, uses v0.15.0. This causes issues across the code base and we cannot use the latest version of Rufio.
Also, controller-runtime's latest release is v0.16.2. We should upgrade to it.
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:

EKS Anywhere Release:
EKS Distro Release:

I have discussed this a bit internally and for the Rufio changes I'm wanting to make i will investigate copying the API/types over instead of dealing with versions of controller-runtime. I am working on this in this branch https://github.com/jacobweinstock/eks-anywhere/tree/rufio-apis will open a PR when i have things working.

PR opened: https://github.com/aws/eks-anywhere/pull/6721

PR merged

repo: aws/eks-anywhere


text:
renaming github e2e tests

Issue #, if available:
Description of changes:
Updating Tinkerbell E2E test names to ensure Flux Github tests match the setupFluxEnv() and setupFluxGitEnv() regex patterns.
Testing (if applicable):
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approved


text:
fix iam kubeconfig generation in workload clusters

Issue #, if available:
Description of changes:
Testing (if applicable):
Documentation added/planned (if applicable):
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve

/cherry-pick release-0.21


text:
fix: Fix pointer mutation causing incorrect maxPods values

Fixes #7011
Fixes #6890
Fixes #6979
Description
This PR fixes a bug where both GetKubeletConfigurationWithNodeClaim and GetKubeletConfigurationWithNodePool were returning the nodeClass.Spec.Kubelet back to the caller when Kubelet was specified and there was no Kubelet compatibility annotation.
This would be fine; however, this call returned back the original pointer for the Kubelet data -- meaning that any caller could then mutate the original data for other callers without knowing it. In this case, the launch template resolver was inadvertently mutating the maxPods data across instance type combinations, leading to incorrect maxPods values being specified sometimes.
How was this change tested?

make presubmit
Print-lining and validating that we are no longer seeing inconsistent maxPods when using the production data.

Does this change impact docs?

[ ] Yes, PR includes docs updates
[ ] Yes, issue opened: #
[x] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Pull Request Test Coverage Report for Build 10887984482
Details

8 of 8 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.02%) to 79.396%

Totals

Change from base Build 10838271515:
0.02%

Covered Lines:
5942

Relevant Lines:
7484

💛  - Coveralls

In which release can we expect this fix?

repo: aws/karpenter-provider-aws


text:
Drift detection is flaky

Version
__Karpenter Version:__ v0.25.0
__Kubernetes Version:__ v1.23.17

We are not running on the latest version of Karpenter. Reading through the CHANGELOGs, I don't believe versions through 0.27.1 address our issue, but if you have reason to believe otherwise please let me know.
We have enabled drift detection in some of our clusters via the following:
❯ kubectl --namespace karpenter \
    get configmap karpenter-global-settings \
    --output yaml \
  | yq .data \
  | grep drift

featureGates.driftEnabled: "true"

This feature seems extremely brittle. Sometimes it works beautifully, other times it silently does nothing. At times where it's quietly not working, even manually annotating nodes does not cause karpenter to start deprovisioning. Again, nothing relevant is in the logs.
❯ kubectl annotate node "${node}" 'karpenter.sh/voluntary-disruption=drifted'

The only pattern we've noticed is that in working pods, we see the following logs (with some noise removed for clarity):
controller.deprovisioning	controller/singleton.go:95	starting controller
controller.deprovisioning	cloudprovider/instancetypes.go:237	discovered EC2 instance types	{"instance-type-count": 626}
controller.deprovisioning	cloudprovider/instancetypes.go:198	discovered EC2 instance types zonal offerings for subnets	{"subnet-selector": "{\"aws-ids\":\"REDACTED,REDACTED,REDACTED\"}"}
...
controller.deprovisioning	deprovisioning/controller.go:145	deprovisioning via drift delete, terminating 1 nodes REDACTED/c6a.8xlarge/on-demand
# nodes continue to deprovision as expected

On pods that aren't deprovisioning nodes, we see:
controller.deprovisioning	controller/singleton.go:95	starting controller
controller.deprovisioning	cloudprovider/instancetypes.go:237	discovered EC2 instance types	{"instance-type-count": 626}
controller.node-state	cloudprovider/instancetypes.go:198	discovered EC2 instance types zonal offerings for subnets	{"node": REDACTED, "subnet-selector": "{\"aws-ids\":\"REDACTED,REDACTED,REDACTED\"}"}
...
# nothing further

Note that in the pod that's detecting drift successfully, discovering EC2 instance types zonal offerings is performed by the deprovisioning controller while in the pod that's stalled, it's being done by the node-state controller. They also produce slightly different log output. I have no idea if this is a clue or a red herring, but it seemed significant enough to warrant mention given the lack of other information I have about this.
Note that both of these pods are configured exactly the same, with the same configmap/karpenter-global-settings:
❯ kubectl --namespace karpenter \
    get configmap karpenter-global-settings \
    --output yaml \
  | yq .data

aws.clusterEndpoint: REDACTED
aws.clusterName: REDACTED
aws.defaultInstanceProfile: REDACTED
aws.enableENILimitedPodDensity: "true"
aws.enablePodENI: "false"
aws.interruptionQueueName: ""
aws.isolatedVPC: "true"
aws.nodeNameConvention: ip-name
aws.vmMemoryOverheadPercent: "0.075"
batchIdleDuration: 1s
batchMaxDuration: 10s
featureGates.driftEnabled: "true"

Thanks for investigating!
Expected Behavior
Drift detection should cause nodes that have drifted to be annotated with karpenter.sh/voluntary-disruption=drifted, and notes with that annotation should be deprovisioned.
Actual Behavior
Nodes that have drifted are not annotated, nor are annotated nodes deprovisioned.
Steps to Reproduce the Problem
¯\(ツ)/¯
For us, it's intermittent. Sometimes pods perform correctly right off the bat. Sometimes they don't. Sometimes they stop after awhile of working successfully.
Resource Specs and Logs
Provider:
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: normal
spec:
  kubeletConfiguration:
    containerRuntime: dockerd
  labels:
    workload: normal
  limits:
    resources:
      cpu: "9600"
      memory: 19200Gi
  providerRef:
    name: normal
  requirements:
  - key: node.kubernetes.io/instance-type
    operator: In
    values:
    - c6a.8xlarge
    - c6i.8xlarge
    - c5a.8xlarge
    - c5.9xlarge
  - key: kubernetes.io/os
    operator: In
    values:
    - linux
  - key: kubernetes.io/arch
    operator: In
    values:
    - amd64
  - key: karpenter.sh/capacity-type
    operator: In
    values:
    - on-demand
  taints:
  - effect: NoSchedule
    key: workload
    value: normal
  ttlSecondsUntilExpired: 604800

AWSNodeTemplate:
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: normal
spec:
  amiFamily: AL2
  blockDeviceMappings:
  - deviceName: /dev/xvda
    ebs:
      deleteOnTermination: true
      encrypted: true
      volumeSize: 500Gi
      volumeType: gp3
  securityGroupSelector:
    aws-ids: REDACTED
  subnetSelector:
    aws-ids: REDACTED
  userData: REDACTED
  securityGroups:
  - id: REDACTED
  - id: REDACTED
  subnets:
  - id: REDACTED
    zone: us-east-1d
  - id: REDACTED
    zone: us-east-1a
  - id: REDACTED
    zone: us-east-1c

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Are you noticing nodes are annotated with karpenter.sh/voluntary-disrupted and aren't deprovisioning or are you just not seeing the nodes get annotated at all?

Hey @stouset, so to confirm, you're seeing nodes that are annotated with karpenter.sh/voluntary-disruption: drifted not be deprovisioned? Due to how deprovisioning serial nature, you may not see these nodes be deprovisioned immediately, but it should happen eventually if there are no blocking pods. If there is a reason a node that you want to be deprovisioned cannot, you should be able to find some DeprovisioningBlocked kube-events that describe what may be happening.
Otherwise, it looks like you aren't using an AMISelector, which means that any time a new AL2 EKS Optimized AMI is released, your nodes could be drifted. If a new release happened, you should have seen nodes be annotated and then drifted. Can you confirm this at all?
It looks like the most recent release was on 3/29, which was a week ago. (CLI command referenced below)
➜ aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.25/amazon-linux-2/recommended/image_id
{
    "Parameter": {
        "Name": "/aws/service/eks/optimized-ami/1.25/amazon-linux-2/recommended/image_id",
        "Type": "String",
        "Value": "ami-09979d7131171bc02",
        "Version": 4,
        "LastModifiedDate": "2023-03-29T13:58:56.375000-07:00",
        "ARN": "arn:aws:ssm:us-west-2::parameter/aws/service/eks/optimized-ami/1.25/amazon-linux-2/recommended/image_id",
        "DataType": "text"
    }
}

so to confirm, you're seeing nodes that are annotated with karpenter.sh/voluntary-disruption: drifted not be deprovisioned?

We are seeing neither drifted nodes get automatically annotated with this, nor manually-annotated nodes being deprovisioned.

Due to how deprovisioning serial nature, you may not see these nodes be deprovisioned immediately, but it should happen eventually if there are no blocking pods.

We have waited hours and see no further deprovisions, despite dozens of out-of-spec nodes remaining in the cluster. We also do not have any pods annotated with karpenter.sh/do-not-evict: "true", although we do have some pods with PodDisruptionBudgets that stop them from being moved off nodes. We do see karpenter pods with functioning drift detection log when this occurs, after it hard cordoned the node and started moving work off of it. We do not see any of this behavior when karpenter's drift detection becomes wedged.

If a new release happened, you should have seen nodes be annotated and then drifted. Can you confirm this at all?

During cluster upgrades, we sometimes see karpenter pods successfully recognize the new AMI and begin automatic rotation via drift detection. We more frequently see no attempts from karpenter to either annotate or deprovision annotated nodes, despite the AMI being out of spec. This issue persists even after a deployment restart. Sometimes the issue suddenly afflicts a running karpenter pod which had previously been successfully rotating nodes.
Karpenter does detect the new AMI successfully 100% of time. We have not found an underlying cause that triggers a particular karpenter pod to either fail to perform drift detection on startup, nor have we found a trigger for a functioning karpenter pod to suddenly stop performing drift detection. We have observed both of these behaviors.
This issue occurs independent of if we use amiFamily or amiSelector. In both scenarios, we often see karpenter pods both failing to annotate drifted nodes and failing to deprovision annotated nodes.

When a karpenter pod enters this "wedged" state (which is a strong majority of the time)

Sorry if I missed this, but what is this "wedged" state? Are you saying the Karpenter pod is crashing or restarting a lot? It sounds like you're saying you run multiple Karpenter pods. How many are you running if you don't mind me asking? I run with two replicas most of the time.
If you're running Karpenter with many replicas, is there a chance you're watching the logs of the Karpenter pod that isn't leader elected?
Lastly, just to dot my i's and cross my t's, do you check if a node should be drifted by looking at the AMI label on the node: karpenter.k8s.aws/instance-ami-id: ami-id?
Additionally, you can use kubectl to get the nodes that aren't on the AMI that you want. Can you run this and make sure that it's what you expect the output to be? Something's definitely wrong if the nodes returned from the below command do not have the karpenter.sh/voluntary-disruption annotation.
    kubectl get node -l karpenter.k8s.aws/instance-ami-id!=${ami-id-you-want}

Sorry if I missed this, but what is this "wedged" state?

This is the entire point of the bug report. Most of the time, karpenter pods sit idly and do perform neither drift detection nor decommissioning of drifted nodes, despite a cluster containing nodes that are out of spec. It emits no logs that would indicate why it is no longer performing these tasks.
Restarting it repeatedly and/or waiting 8-12 hours sometimes results in it resuming its duties.

It sounds like you're saying you run multiple Karpenter pods. How many are you running if you don't mind me asking?

Two.
❯ kubectl --namespace karpenter get pods
NAME                         READY   STATUS    RESTARTS   AGE
karpenter-7c789fc9c4-g9txt   1/1     Running   0          4h32m
karpenter-7c789fc9c4-n4hxl   1/1     Running   0          4h32m

I'm seeing this as well in my cluster and the only time I was able to get some drift action was when I delete PDBs (we have some bad ones). Is it possible to have the drift detection controller emit logs for when it can't perform a drift delete because of x reason?
Karpenter Version: 0.27.0
Kubernetes Version: 1.23

Thanks @stouset for the comprehensive report! I'm really curious what could get Karpenter stuck here on any of the logic, this definitely shouldn't be happening.
So there are two controllers that are of importance, and I'd like to get down to the problem and see if it's both controllers or just one that's being flaky.

Deprovisioning Controller - deprovisions annotated nodes
Node Controller - annotates the node as drifted

For the Deprovisioning Controller, you said:

The deprovisioner does continue to perform its task of rotating nodes for other reasons such as expiration past their ttlSecondsUntilExpired window.

Which makes me believe that the deprovisioning controller isn't wedged.
For the Node controller:
It's tough to know how often the node controller is working if you've mixed in annotating some of them yourself. Since it's flaky, it's tough to know by only the information you've given as to when it's worked correctly.
For both the Deprovisioning and Node controller to not work, I'm thinking there might be some problems with how your settings are persisting, but you said that the issue persists after deployment restarts. We don't load in new settings dynamically, so any time you change the configmap, you'll need to restart your deployment for those configmap changes to take effect.
The previous release for that region and version was ami-0f5fb1060cc8f7202 (see output below). Can you de-annotate any of the manually annotated nodes and set your AWSNodeTemplate to the following and wait 10 minutes?
amiFamily: AL2
amiSelector:
    aws-ids: ami-0f5fb1060cc8f7202

If you can run the same command to get the AMIs and annotations before changing the AWSNodeTemplate and after 10 minutes, without touching anything, I'd like to see if it works. If not, If you can do kubectl describe pods karpenter_pod_name -n karpenter as well.
        {
            "Name": "/aws/service/eks/optimized-ami/1.23/amazon-linux-2/amazon-eks-node-1.23-v20230304",
            "Type": "String",
            "Value": "{\"schema_version\":\"2\",\"image_id\":\"ami-0f5fb1060cc8f7202\",\"image_name\":\"amazon-eks-node-1.23-v20230304\",\"release_version\":\"1.23.16-20230304\"}",
            "Version": 1,
            "LastModifiedDate": "2023-03-09T13:47:35.169000-08:00",
            "ARN": "arn:aws:ssm:us-west-2::parameter/aws/service/eks/optimized-ami/1.23/amazon-linux-2/amazon-eks-node-1.23-v20230304",
            "DataType": "text"
        },

Sorry to ask so much, but it's tough to debug flakes since they're especially hard to reproduce.

Which makes me believe that the deprovisioning controller isn't wedged.

What makes me dispute this is the fact that manually-annotated nodes aren't being deprovisioned, but we'll run this test and see!

The previous release for that region and version was ami-0f5fb1060cc8f7202 (see output below). Can you de-annotate any of the manually annotated nodes and set your AWSNodeTemplate to the following and wait 10 minutes?

❯ kubectl get nodes --no-headers | awk '{print $1}' | while read node
    kubectl annotate node $node 'karpenter.sh/voluntary-disruption-'
end

❯ kubectl get nodes \
    --no-headers \
    --output jsonpath='{range .items[*]}{.metadata.labels.karpenter\.k8s\.aws/instance-ami-id}{" "}{.metadata.annotations.karpenter\.sh/voluntary-disruption}{"\n"}{end}' \
  | sort \
  | uniq -c
  26 ami-0a88006b97dc624e7

I'm experimenting in us-west-2, so I'll use AMI ami-0e968e66a05cef9f9, which is one previous from the current one in this region.
❯ kubectl --namespace karpenter get awsnodetemplates --output json | jq '.spec | {amiFamily, amiSelector}'
{
  "amiFamily": "AL2",
  "amiSelector": {
    "aws-ids": "ami-0e968e66a05cef9f9"
  }
}

❯ kubectl --namespace karpenter rollout restart deployment karpenter
deployment.apps/karpenter restarted

Immediately afterward, most of the nodes are detected as drifted, and one new node has been launched.
❯ kubectl get nodes \
    --no-headers \
    --output jsonpath='{range .items[*]}{.metadata.labels.karpenter\.k8s\.aws/instance-ami-id}{" "}{.metadata.annotations.karpenter\.sh/voluntary-disruption}{"\n"}{end}' \
  | sort \
  | uniq -c
   4 ami-0a88006b97dc624e7
  22 ami-0a88006b97dc624e7 drifted
   1 ami-0e968e66a05cef9f9

This is consistent with what I've seen. We'll see whether or not the remaining ami-0a88006b97dc624e7 nodes are annotated with drift, or whether all of the ami-0a88006b97dc624e7 are rotated. I predict no :)

If you can run the same command to get the AMIs and annotations before changing the AWSNodeTemplate and after 10 minutes, without touching anything, I'd like to see if it works. If not, If you can do kubectl describe pods karpenter_pod_name -n karpenter as well.

What would you like to see from this output?

Sorry to ask so much, but it's tough to debug flakes since they're especially hard to reproduce.

Ask everything you need in order to help identify this issue!

We discovered an issue where Karpenter would start to backoff upon successful deprovisioning.  I believe that this is what you are seeing due to:
 kubectl --namespace karpenter rollout restart deployment karpenter
deployment.apps/karpenter restarted

Immediately afterward, most of the nodes are detected as drifted, and one new node has been launched.

This restart of Karpenter resets the backoff, where it starts to replace the drifted nodes and backoff starts to accumulate again.

Can you explain in a little further detail?

For the bug that we're currently aware of, restarting should always resolve it. However it will immediately reoccur (e.g. replace a few nodes with more time between each replacement appearing to eventually stop).

This is potentially part of the problem, as I have noticed karpenters rotate until grinding to a halt. But there may be another issue as well, as restarting does not always prompt karpenter to continue.

This label is deprecated, btw: https://github.com/aws/karpenter/blob/main/pkg/apis/v1alpha1/register.go#L83. Looking at the code, it has no bearing on drift.

We suspect that this is root caused by https://github.com/aws/karpenter-core/pull/279/files. Though, the fact that restart doesn't solve it in all cases is concerning. Will be coming out next release..

Great. We'll update once this is released and can pursue further if there continue to be issues.

This should be resolved in v0.27.2, please re-open or file a new issue if you still experience this.

This issue still seems to be happening. However, I now see this in the logs:
2023-04-14T19:19:37.349Z	DEBUG	controller.deprovisioning	waiting on cluster sync	{"commit": "d01ea11-dirty"}

No further controller.deprovisioning messages occur. In this cluster, the out-of-spec nodes have been annotated as drifted already.
❯ kubectl get nodes \
    -ojsonpath='{range .items[*]}{.metadata.labels.workload}{" "}{.metadata.labels.ami_id}{.metadata.labels.karpenter\.k8s\.aws/instance-ami-id}{" "}{.status.nodeInfo.kubeletVersion}{" "}{.metadata.annotations.karpenter\.sh/voluntary-disruption}{"\n"}{end}' \
  | sort \
  | uniq -c

   3 gpu-nvidia-v1 ami-02c5235360fb8957c v1.23.17-eks-a59e1f0
  16 normal ami-04c545ca749cfe57b v1.23.17-eks-a59e1f0 drifted
 122 normal ami-0559f43d314aa2e7e v1.23.17-eks-a59e1f0
   5 pci ami-0559f43d314aa2e7e v1.23.17-eks-a59e1f0
   9 utility ami-0559f43d314aa2e7e v1.23.17-eks-a59e1f0

If you describe the node, are there events on it?

Closing again due to #3764

authors: ellistarn jonathan-innis njtran stouset tzneal visokoo

repo: aws/karpenter


text:
feat: supports karpenter.sh/instance-price node label

Fixes #1289 and Fixes #3237
Description
Adds a new node label to all provisioned nodes with the hourly price, in USD, that was used when provisioning this node.
How was this change tested?

make test

Does this change impact docs?

[x] Yes, PR includes docs updates
[ ] Yes, issue opened: #
[ ] No

Release Note

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

I chatted with the team about adding this and there were several concerns:

This is a pricing estimate at launch which may not be correct in a lot of situations (savings plans, RIs, stale pricing data, etc).
The price can change for spot each hour
Adding this as a label may make some user want to use this in their requirements, which I don't think is a great idea. It's also not possible as a float value.

Would adding the pricing data in the Karpenter logs when the node is provisioned be enough information?

I chatted with the team about adding this and there were several concerns:

This is a pricing estimate at launch which may not be correct in a lot of situations (savings plans, RIs, stale pricing data, etc).

I agree that this doesn't take into account those pieces, but this was intended to expose some kind of pricing information to give an estimate on costs, not be a cost accounting system.

The price can change for spot each hour

I was unaware of this.

Adding this as a label may make some user want to use this in their requirements, which I don't think is a great idea. It's also not possible as a float value.

I would agree that using it in the requirements isn't a good idea.

Would adding the pricing data in the Karpenter logs when the node is provisioned be enough information?

This would be a great start, but we'd love to be able to get this in a prometheus metric so we can do some rough pod cost calculations.
Would renaming the label to instance-price-estimate reduce the concerns?
My other idea on this was to expose the pricing.Provider on demand/spot pricing as prometheus metrics instead. This way we could use promql joins from node metrics to pricing metrics as karpenter knew at the given time. This would help with issue 2 above where spot prices can change hourly, and with issue 3 where you can't use them as requirements. It would not help with issue 1, but perhaps with clever metrics names, such as including estimate it would be clear?

My other idea on this was to expose the pricing.Provider on demand/spot pricing as prometheus metrics instead. This way we could use promql joins from node metrics to pricing metrics as karpenter knew at the given time. This would help with issue 2 above where spot prices can change hourly, and with issue 3 where you can't use them as requirements. It would not help with issue 1, but perhaps with clever metrics names, such as including estimate it would be clear?

I'd prefer to go with the prometheus metric and/or including the price in the log when the instance is launched for now.   And yes, I think naming it as price-estimate is enough information. Documentation on the metrics page can also explain why it may not be accurate.

Closing in favor of https://github.com/aws/karpenter/pull/3613


text:
ClusterEndpointsRefreshAgent does not handle lambda errors

I am using the lambda proxy to discover my cluster topology with ClusterEndpointRefreshAgent. Sometimes when invoking the lambda during its initialization period, it will timeout. When this happens, the refresh agent throws an error which seems to indicate that it is not properly handling/parsing the error response from the lambda service. There is no indication of what the error actually was in the client logs. Determining that it was a timeout required looking into the CloudWatch log stream for the lambda proxy.
Software versions used

Neptune Endpoints Info Lambda: neptune-endpoints-info-lambda-2.0.5.jar
Neptune Gremlin Client: software.amazon.neptune:gremlin-client:2.0.5

Expected behavior

Server-side lambda errors (of any type, not just the timeout error I observed) are reported/logged on the client
Timeout errors are retried

Actual behavior

The client fails to handle the response and the observed error does not indicate the error reported by the server. It appears this is because the refresh agent is not checking for an error response. It immediately tries to parse any response into a NeptuneClusterMetadata object without examining InvokeResult.statusCode or InvokeResult.functionError
It does not look like the Neptune Gremlin client retry configuration would retry on a timeout. The lambda documentation suggests that the AWS SDK may retry this so it may not be necessary for the Neptune Gremlin client to also retry. It was unclear from just the client logs whether any retry occurred - the server-side CloudWatch logs did not show another request until 15s after the timed out request ended.

It's not clear what error is actually being returned. From looking at documentation it looks like a 408 RequestTimeoutException is a possibility.

Hi @derekguist
Thanks for reporting this. I've added a check in the client to catch the timeout and retry. (Timeouts still return 200 OK, so I had to check the functionError and payload for the details.) See:
https://github.com/aws/neptune-gremlin-client/commit/852843ec48b2ec6652bd6307ff4c086e35a9edfd
I'd be interested to know what was causing the Lambda to timeout during its initialization period. Do you have any insights into what might be causing the timeout?
Thanks
ian

@iansrobinson That's great to hear, thanks for the quick response.
Unfortunately I don't have any insight into the cause of the timeout. I kind of assumed that it may just be typical when a request comes in to a lambda that needs to be initialized before responding. I can say that when we first created the refresh lambda, we missed the timeout value (15s) in the recommended CloudFormation template. We thus ended up with the default lambda timeout of 3s. When we updated to the 15s timeout, we observed this error much less frequently (just three occurrences in the past several days) and this fix looks like it will eliminate the impact of even the few remaining timeouts.
Here is what we saw in our lambda logs in CloudWatch - not much detail:
2024-04-14T22:21:57.525Z f72b4258-adaa-4fea-8644-527d6142effe Task timed out after 15.02 seconds

And this was always associated with:
INIT_START Runtime Version: java:11.v36 Runtime Version ARN: arn:aws:lambda:xxx

authors: derekguist iansrobinson

repo: aws/neptune-gremlin-client


text:
Correct suffix for destination bucket in prereq instructions

Issue #, if available:
n/a
Description of changes:
Correct suffix for destination bucket in prereq instructions
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Thank you for the correction.

repo: awsdocs/aws-lambda-developer-guide


text:
Pretty print ipamd JSON log for log collector

Issue #, if available:
Description of changes:
/v1/enis output with compact JSON output is hard to read when we are doing troubleshooting.
With pretty print ipamd log output with simple python3 -m json.tool
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Testing Done
Output for /v1/enis maybe too large for demo, so let's take /v1/networkutils-env-settings as sample output.
Before change
$ curl --max-time 3 --silent http://localhost:61679/v1/networkutils-env-settings
{"AWS_VPC_CNI_NODE_PORT_SUPPORT":true,"AWS_VPC_ENI_MTU":9001,"AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER":false,"AWS_VPC_K8S_CNI_CONNMARK":128,"AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS":null,"AWS_VPC_K8S_CNI_EXTERNALSNAT":false,"AWS_VPC_K8S_CNI_RANDOMIZESNAT":2,"AWS_VPC_K8S_CNI_VETHPREFIX":"eni"}

After change
$ curl --max-time 3 --silent http://localhost:61679/v1/networkutils-env-settings | $(which python3) -m json.tool
{
    "AWS_VPC_CNI_NODE_PORT_SUPPORT": true,
    "AWS_VPC_ENI_MTU": 9001,
    "AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER": false,
    "AWS_VPC_K8S_CNI_CONNMARK": 128,
    "AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS": null,
    "AWS_VPC_K8S_CNI_EXTERNALSNAT": false,
    "AWS_VPC_K8S_CNI_RANDOMIZESNAT": 2,
    "AWS_VPC_K8S_CNI_VETHPREFIX": "eni"
}

See this guide for recommended testing for PRs. Some tests may not apply. Completing tests and providing additional validation steps are not required, but it is recommended and may reduce review time and time to merge.

@cartermckinnon If you have time, I'd like to hear your opinion for the change here.
What you would prefer to prettify JSON? python -m json.tool? or jq ?

repo: awslabs/amazon-eks-ami


text:
build ARM AMIs with m6g.large instance type

Description of changes:
m6g.large instance types are available in all AWS partitions while a1.large instance type is not. Use m6g.large instance type as the default when building ARM AMIs.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

LGTM

LGTM

authors: heybronson mmerkes


text:
RPI Install problem

I'm having an issue installing the produced SDK on my RPI4. I followed the instructions in the "Install Steps for Ubuntu 17.x and Raspbian Stretch using apt-get" and used "./min-install-script".
I'm able to create a stream using "kinesis_video_gstreamer_sample_app ...", but I can't create a stream using "gst-launch-1.0.........". I get the error "WARNING: erroneous pipeline: no element "kvssink""
I assume my gstreamer plugin cannot be found, and when I check the "/amazon-kinesis-video-streams-producer-sdk-cpp/kinesis-video-native-build/downloads/local/lib" folder, there's no gstreamer related files there.
It's not clear to me where the gstreamer plugin was installed, given that I used the mini-install-script. How can I check this? Is there a way for force install-script to work on my RPI, instead of having to use mini-install-script.
I'd greatly appreciate your help to debug this problem.

Did you find a solution to this? I am also having the same problem.

@perronj2 the min-install-script does not "install" the plugin. It builds the binary but you need to follow the steps in the readme to set the paths to the plugin.

@perronj2 If my memory serves me correctly, I used mini-install-script, and the issue I had was the setting of LD_LIBRARY_PATH and GST_PLUGIN_PATH
I used:
export LD_LIBRARY_PATH=/kinesis-video-native-build
export GST_PLUGIN_PATH=/kinesis-video-native-build
This is slightly different than the documentation. I feel like this suggestion was made by the AWS support team, but I can't find the GitHub issue or Email, so I'm not 100 sure. @MushMal does this sound right? I feel like you (or @chehefen ) and I discussed this at some point.

@istranic yup that fixed it. I thought I had set the environment variables correctly but I guess not. That finally worked. I've notice a some serious documentation flaws and spending considerable amount of time searching for solutions in random posts. Thanks for the help!

Thanks for the info guys. We absolutely would appreciate any contribution to the codebase, including the documentation fixes. Please feel free to submit PRs

authors: MushMal istranic perronj2


text:
FailoverConnectionPlugin SessionStateTransferCallable capability

Describe the bug
Converting this discussion into a ticket https://github.com/awslabs/aws-advanced-jdbc-wrapper/discussions/766
We are trying to validate the SessionStateTransferCallable configuration
It looks like SessionStateTransferCallable is called only by switchCurrentConnectionTo(), which requires a non-null value for this.pluginService.getCurrentConnection() in order to call transferSessionState() to copy properties from the old connection to the new.
But switchCurrentConnectionTo() is only actually called when this.pluginService.getCurrentConnection() is null.
This, I think, means switchCurrentConnectionTo() will never call transferSessionState() to copy properties from the old connection to the new.
As a result the failover just always goes through failoverReader() or failoverWriter(), which functionally does recover, but the switchCurrentConnectionTo() path is never taken, as far as I can tell --
Am I reading this correctly?
https://github.com/awslabs/aws-advanced-jdbc-wrapper/blob/main/wrapper/src/main/java/software/amazon/jdbc/plugin/failover/FailoverConnectionPlugin.java#L863
More specifically in FailoverConnectionPlugin.java:

SessionStateTransferCallable() is invoked only by transferSessionState(), and only if this.pluginService.getCurrentConnection() is not null

transferSessionState() is invoked only by switchCurrentConnectionTo()

switchCurrentConnectionTo() is invoked only by connectTo()

connectTo() is invoked only by pickNewConnection, and only if this.pluginService.getCurrentConnection() is null

I believe this means that SessionStateTransferCallable() can never be invoked, which seems to align with our tests so far where we do not observe it being called.  Unless there is another reason we are not observing SessionStateTransferCallable being invoked during failovers.
Expected Behavior
We were expecting SessionStateTransferCallable() to be invoked to copy connection properties during a failover.
What plugins are used? What other connection properties were set?
aws wrapper driver 2.3.1 , mysql connect driver 8.0.33
Current Behavior
SessionStateTransferCallable is not invoked when an RDS failover is triggered during testing
Reproduction Steps
Start an application pointing to an RDS database with both a reader and a writer, with aws wrapper driver 2.3.1 and mysql jdbc connect driver 8.0.33, then trigger a failover in the RDS mysql database, and observe that SessionStateTransferCallable is never invoked and that the driver fails over in the normal manner of letting the connection fall out of the pool and reconnect in the normal manner.
Possible Solution
No response
Additional Information/Context
No response
The AWS Advanced JDBC Driver version used
2.3.1
JDK version used
jdk17
Operating System and version
aws

Hello @scott-long-adp,
As you can see in the issue timeline, we have recently merged a quick change to address the issue.
Could you kindly checkout our snapshot build and let us know if the issue still persists?
Thank you!

Hi @brunos-bq -- thank you -- we will pull down a snapshot and get it into a test environment where we can fail over -- thanks again -- will report back --

Hi @scott-long-adp
Heads up that the fix is now available on the latest version of the AWS Advanced JDBC Driver.
Also, since there have not been updates on this ticket for some time, I'll be closing this ticket. However, if you require further assistance, you can reopen this issue, or create a new one.
Thank you!

repo: awslabs/aws-advanced-jdbc-wrapper


text:
fix(lambda) support adding deep clones of layers to a Function

In order to keep my Lambda functions declarations DRY, I clone (using lodash) default Lambda properties and reuse them:
const defaultLayer = lambda.LayerVersion.fromLayerVersionArn(this, 'DefaultLayer', 'some arn');
const extraLayer = lambda.LayerVersion.fromLayerVersionArn(this, 'ExtraLayer', 'another arn');

const defaultLambaProps = {
  timeout: 30,
  runtime: lambda.Runtime.Python37,
  handler: 'index.handler',
  layers: [
    defaultLayer,
  ],
};

const func = new lambda.Function(this, 'Func',
  _.merge(_.cloneDeep(defaultLambaProps), {
    code: lambda.Code.asset(`funcCode`),
    layers: defaultLambaProps.layers.concat([
      extraLayer,
    ])
  })
);

However, since 0.31.0, this strategy fails:
Error: This lambda function uses a runtime that is incompatible with this layer (python3.7 is not in [nodejs, nodejs4.3, nodejs6.10, nodejs8.10, python2.7, python3.6, python3.7, java8, dotnetcore1.0, dotnetcore2.0, dotnetcore2.1, go1.x, ruby2.5, provided])
    at Function.addLayer (/Users/spgingras/entity-data-pipeline/services/cdk-app/node_modules/@aws-cdk/aws-lambda/lib/function.ts:501:13)
    at new Function (/Users/spgingras/entity-data-pipeline/services/cdk-app/node_modules/@aws-cdk/aws-lambda/lib/function.ts:445:12)
    ...

Note that the error message is confusing: python3.7 is not in [..., python3.7, ...].
I realized that after cloning occured, defaultLayer and extraLayer did not share the same Runtime.All array, resulting in layer.compatibleRuntimes.indexOf(this.runtime) returning -1.
I fixed this problem by implementing deep equality on Runtime (see Runtime.equals(other)).

Pull Request Checklist

[x] Testing

Unit test added (prefer not to modify an existing test, otherwise, it's probably a breaking change)
CLI change?: coordinate update of integration tests with team
cdk-init template change?: coordinated update of integration tests with team

[ ] Docs

jsdocs: All public APIs documented
README: README and/or documentation topic updated
Design: For significant features, design document added to design folder

[x] Title and Description

Change type: title prefixed with fix, feat and module name in parens, which will appear in changelog
Title: use lower-case and doesn't end with a period
Breaking?: last paragraph: "BREAKING CHANGE: <describe what changed + link for details>"
Issues: Indicate issues fixed via: "Fixes #xxx" or "Closes #xxx"

[ ] Sensitive Modules (requires 2 PR approvers)

IAM Policy Document (in @aws-cdk/aws-iam)
EC2 Security Groups and ACLs (in @aws-cdk/aws-ec2)
Grant APIs (only if not based on official documentation with a reference)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license.

Not sure what is happening with CodeBuild!

repo: awslabs/aws-cdk


text:
docs(home): add discord invitation link

Issue number: #154
Summary
Changes

Please provide a summary of what's being changed

Replace occurrences of Slack invitation links with links to the new Discord space.
User experience

Please share what the user experience looks like before and after this change

N/A
Checklist
Please leave checklist items unchecked if they do not apply to your change.

[x] Meets tenets criteria
[x] I have performed a self-review of this change
[x] Changes have been tested
[x] Changes are documented
[x] PR title follows conventional commit semantics

Is this a breaking change?
RFC issue number:
Checklist:

[ ] Migration process documented
[ ] Implement warnings (if it can live side by side)

Acknowledgment
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

View rendered README.md
View rendered docs/index.md

@sliedig do you want the second link next to the invitation link or do you want to replace the invitation link? I think the channel link will only work, if you have an account already, corret?

@sliedig do you want the second link next to the invitation link or do you want to replace the invitation link? I think the channel link will only work, if you have an account already, corret?

Ah, yeah fair enough. Maybe we'll leave it for now.

repo: awslabs/aws-lambda-powertools-dotnet