text:
S3 Encryption Client not using credentials of wrapped client

Problem:
I have two AWS accounts, A and B. In my ~/.aws/credentials file, I have credentials for account B. While creating the wrapped clients to pass into S3EC, I am passing in credentials for account A. I am accessing a bucket in account A. Code:
  S3Client s3Client = S3Client.builder().region(Region.EU_WEST_1)
        .credentialsProvider(StaticCredentialsProvider.create(
            AwsBasicCredentials.create("xxxx", "xxxx")
        ))
        .build();

    S3AsyncClient s3AsyncClient = S3AsyncClient.builder().region(Region.EU_WEST_1)
        .credentialsProvider(StaticCredentialsProvider.create(
            AwsBasicCredentials.create("xxxx", "xxxx")
        ))
        .build();

    S3Client s3ECClient = S3EncryptionClient.builder()
             .kmsKeyId("xxxxx")
            .wrappedClient(s3Client)
            .wrappedAsyncClient(s3AsyncClient)
           .enableLegacyUnauthenticatedModes(true)
           .build();

    ResponseInputStream<GetObjectResponse> inputStream = s3ECClient.getObject(
        GetObjectRequest.builder()
        .bucket("xxxx")
        .key("xxxx").build());

This gives
software.amazon.encryption.s3.S3EncryptionClientException: User: arn:aws:iam::<ACCOUNT B USER> is not authorized to perform: kms:Decrypt on the resource associated with this ciphertext because the resource does not exist in this Region, no resource-based policies allow access, or a resource-based policy explicitly denies access

If I comment out the credentials in my ~/.aws/credentials, I get software.amazon.encryption.s3.S3EncryptionClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain
It works fine if put credentials of account A in my ~/.aws/credentials.
Solution:
Looks like S3EC is ignoring credentials passed in the wrapped clients and picking up credentials from the default credentials resolution chain. It should use credentials of the wrapped clients.

Hello Ahmar,
Since the error is coming from KMS, this is happening because the KMS keyring instantiated by default simply uses the default credential provider chain (and any other associated client configuration). In order to configure the KMS client to use the explicitly configured credentials, you need to instantiate the KMS keyring separately and pass a configured KMS client to its builder. For example:
        KmsClient kmsClient = KmsClient.builder()
                /* KMS client configuration goes here */
                .build();

        KmsKeyring keyring = KmsKeyring.builder()
                .kmsClient(kmsClient)
                .wrappingKeyId(KMS_KEY_ID).build();

        S3AsyncClient v3Client = S3AsyncEncryptionClient.builder()
            .keyring(keyring)
            .build();

This should resolve your issue.
Other customers have ran into this issue, so I have opened a separate issue for a feature request to add a top-level configuration parameter to the S3EC which applies to all of its wrapped clients to avoid duplication of code and provide an experience similar to v1/v2 (https://github.com/aws/amazon-s3-encryption-client-java/issues/226). I will close this issue in favor of the feature request, but do not hesitate to re-open this issue if the above guidance does not fix your issue. Thank you!

authors: ahmarsuhail justplaz


text:
Iphone screen displaying white bar at top with custom Authenticator UI

I noticed if I am trying to implement custom UI Authenticator, that there is a white bar that is present on the iphone screen. Is the authenticator not hiding itself completely? I have attached a screenshot and here is my code for reference:

index.js
class App extends Component {

handleAuthStateChange(state) {
  if (state === 'signedIn') {
    return ( // go To Entry point of app
      <ApolloProvider store={store} client={client}>
      <AppWithNavigationState />
      </ApolloProvider>
    )
  }
}

    render() {
        return (
         <Authenticator {...this.props} hideDefault={true} onStateChange={this.handleAuthStateChange}>
        <Login/>
        </Authenticator>

      );
    }

}
export default App = codePush(App);

Login.js
export default class Login extends Component {

    render() {

        return (<View style={styles.container}>

            <View style={styles.backgroundContainer}>
            <Image source={images.icons.LoginImage} style={styles.backgroundImage} />
            </View>
            <View style={styles.overlay}>
            <Button iconLeft light rounded style={styles.facebookLoginButton}>
            <Icon style={styles.facebookIcon} name='logo-facebook' />
            <Text style={styles.facebookButtonText}>Login with Facebook</Text>
            </Button>
            <Button rounded bordered light style={styles.loginLaterButton}
            onPress={() => alert('hi')}>
            <Text style={styles.buttonText}>Create Account</Text>
            </Button>

            </View>

            </View>

        );
      }
    }

thanks this solved my issue!

source: gharchive/issue

repo: aws/aws-amplify

license_type: permissive

license_source: github-api


text:
Added nullptr initialization for frame context.

Description of changes: If a WebSocketConnection was deleted without calling connect it resulted in a crash in dlfree. Added a initialisation of p_wslay_frame_Context_ as nullptr within the WebSocketConnection constructor to fix this crash.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Hi @Fischiii,
Thank you for your pull request.  We will merge it into master after testing the changes internally.
Varun

source: gharchive/pull-request

authors: Fischiii vareddy

repo: aws/aws-iot-device-sdk-cpp


text:
fix: correctly generate sts client endpoint override

Issue #, if available: https://github.com/aws/aws-msk-iam-auth/issues/165
Description of changes:
aws-sdk-v2 now needs the scheme of the url, use correct api to generate the sts endpoint override
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

I am also facing this same error in my setup. @sidyag is it possible to get this PR merged?

🆙 @hhkkxxx133 @sidyag

Thanks for your contribution!

@hhkkxxx133 can we get a release?

authors: KKcorps dlabey hhkkxxx133 jvdadda

repo: aws/aws-msk-iam-auth


text:
Entropy seed call does not have intended effect

To seed the operating system's entropy pool, I wrote a standalone tool that calls aws_nitro_enclaves_library_seed_entropy() to gather randomness. As recommended here, I run this tool before my main application starts and it keeps requesting entropy every ten seconds.
Here's the tool:
#include <stdio.h>
#include <unistd.h>
#include <aws/nitro_enclaves/nitro_enclaves.h>

int main(void) {
        printf("Initializing Nitro library.\n");
        while (1) {
                aws_nitro_enclaves_library_init(NULL);
                printf("Using NSM to seed entropy pool.\n");
                AWS_ASSERT(aws_nitro_enclaves_library_seed_entropy(1024) == AWS_OP_SUCCESS);
                sleep(10);
        }
        return 0;
}

Unfortunately, running the tool does not have the intended effect and the operating system's entropy pool remains empty, and my main application blocks when attempting to fetch randomness. My startup script looks as follows:
...
echo "[+] Entropy available after boot: $(cat /proc/sys/kernel/random/entropy_avail)"
/usr/bin/entropy-init &
sleep 5
echo "[+] Entropy available after entropy initialization tool: $(cat /proc/sys/kernel/random/entropy_avail)"
...

The entropy pool is 0 before and after the tool started running:
[+] Entropy available after boot: 0
Initializing Nitro library.
[    1.010870] random: entropy-init: uninitialized urandom read (32 bytes read)
[    1.012014] random: entropy-init: uninitialized urandom read (32 bytes read)
Using NSM to seed entropy pool.
[+] Entropy available after entropy initialization tool: 0

If I use rngd instead, by running rngd -r /dev/urandom, the entropy pool is initialized successfully.
Do you have any insight into what could be going wrong here?

I've tested this out on a fresh instance with the latest release of the tools and it seems to work for me. My code is slightly different, but it should have the same overall effect:
#include <aws/nitro_enclaves/nitro_enclaves.h>

#include <aws/common/error.h>

#include <errno.h>
#include <unistd.h>

int main(void) {
    int rc = 0;
    FILE *f = NULL;
    char entropy_avail[100];
    char *str_r;

    /* Initialize the SDK */
    aws_nitro_enclaves_library_init(NULL);

    while (1) {
	sleep(5);
    	rc = aws_nitro_enclaves_library_seed_entropy(1024);
	if (rc != AWS_OP_SUCCESS) {
		fprintf(stderr, "Could not seed entropy");
		continue;
	}

	f = fopen("/proc/sys/kernel/random/entropy_avail", "r");
	if (f == NULL) {
		fprintf(stderr, "Could not open entropy file");
		continue;
	}

	str_r = fgets(entropy_avail, sizeof(entropy_avail), f);
	if (str_r == NULL) {
		fprintf(stderr, "Could not read entropy file");
		continue;
	}

	printf("entropy_avail = %s\n", str_r);

	rc = fclose(f);
	if (rc != 0) {
		fprintf(stderr, "Could not close entropy file");
		continue;
	}

    }

    aws_nitro_enclaves_library_clean_up();
    return 0;
}

The output of this is:
...
[    0.352115] nsm: loading out-of-tree module taints kernel.
[    0.352668] nsm: module verification failed: signature and/or required key missing - tainting kernel
[    0.357644] random: enclave_rngd: uninitialized urandom read (32 bytes read)
[    0.358342] random: enclave_rngd: uninitialized urandom read (32 bytes read)
[    5.359597] random: crng init done
entropy_avail = 3343

entropy_avail = 3826

entropy_avail = 3910

entropy_avail = 3922

entropy_avail = 3924

entropy_avail = 3925

entropy_avail = 3925

entropy_avail = 4029

entropy_avail = 4085

^C

This is a c5.xlarge instance in use1-az2. What AZ is your instance in? What size?
Does the code above still report 0 entropy?

I've pushed the code here: https://github.com/petreeftime/aws-nitro-enclaves-sdk-c/tree/enclave-rngd

Thanks @petreeftime, your version works. The issue with my version was AWS_ASSERT, which was defined but empty.

authors: NullHypothesis petreeftime

repo: aws/aws-nitro-enclaves-sdk-c


text:
How can I add a custom header value to a create bucket request?

It is possible to add a custom header to an object like:
Aws::S3::Model::PutObjectRequest object_request;
...
object_request.AddMetadata("x-emc-retention-period", QVariant(QDateTime::currentDateTimeUtc().secsTo(retentionTime)).toString().toLatin1().data());
How is it possible to do this for a CreateBucketRequest as well?
...
Aws::S3::Model::CreateBucketRequest request;	request.SetCreateBucketConfiguration(Aws::S3::Model::CreateBucketConfiguration().WithLocationConstraint(getBucketLocationContraint()));
request.SetBucket(m_bucket.toLatin1().data());
// how to set a custom header like "x-emc-retention-period"
...

Hi @heylaender ,
The s3 buckets don't have "metadata" or custom headers, this is an api limitation and not an sdk one though so it's not something we would be able to change.
That being said, maybe I can help you find a workaround to what you're trying to do, can you give me a bit more context on what you're trying to achieve?

I'm trying to archive on DELL ECS, which has both bucket and object level retention:
http://doc.isilon.com/ECS/3.1/DataAccessGuide/ecs_c_s3_reterntion.html
Object level retention works perfect with Aws::S3::Model::PutObjectRequest::AddMetadata.
But the customer's default is bucket level retention and this is the challenge. I know that in both cases x-emc-retention-period header can be used to send it to the archiving system. But it seems that the SDK doesn't support it for buckets.

@heylaender ,
sorry to say but the answer kind of remains the same, even the documentation you provided states : "This is an ECS extension and is not available in the standard S3 API. "
The sdk only supports what the service api supports. This kind of feature must be implemented by the service first... so yea, sorry I can't help you with this one.

I had a similar problem and added a patchfile to my local build via vcpkg.
With that I could successfully set the retention on buckets and files by setting a custom header for the request objects.
CustomHeader.txt
It is in no way pretty, or complete, but it works for me.

CreateBucketRequest req;
req.SetBucket("your_bucket_name");
req.SetAdditionalCustomHeaderValue("X-Custom-Header-Key", "your_value");

authors: F-Haun KaibaLopez aesophor heylaender

repo: aws/aws-sdk-cpp


text:
a question about aarch64-linux compile SDK fail, it seems shared-lib not found but I can't find any problem.

What platform/OS are you using?
OS Ubuntu 16.04
What compiler are you using? what version?
cmake 3.5
curl 7.61.1
openssl 1.1.1
zlib 1.2.11
What's your CMake arguments?
At first, I add these in CMakeList.txt:
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_C_COMPILER "/usr/local/linaro-aarch64-2017.08-gcc7.1/bin/aarch64-linux-gnu-gcc")
set(CMAKE_CXX_COMPILER "/usr/local/linaro-aarch64-2017.08-gcc7.1/bin/aarch64-linux-gnu-g++")
set(CMAKE_FIND_ROOT_PATH  "/home/payne/aarch-aws-env") // this path I install curl / openssl / zlib .
And Cmake argument:
cmake /home/payne/Downloads/aws-sdk-cpp-master -DBUILD_ONLY="s3;transfer;core" -DCMAKE_PREFIX_PATH=/home/payne/aws-install
-- Found Git: /usr/bin/git (found version "2.7.4")
-- TARGET_ARCH not specified; inferring host OS to be platform compilation target
-- Building AWS libraries as shared objects
-- Generating linux build config
fatal: Not a git repository: '/home/payne/Downloads/aws-sdk-cpp-master/.git'
-- Building project version: 1.6.18
-- The CXX compiler identification is GNU 7.1.1
-- Check for working CXX compiler: /usr/local/linaro-aarch64-2017.08-gcc7.1/bin/aarch64-linux-gnu-g++
-- Check for working CXX compiler: /usr/local/linaro-aarch64-2017.08-gcc7.1/bin/aarch64-linux-gnu-g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found ZLIB: /home/payne/aarch-aws-env/lib/libz.so (found version "1.2.11")
--   Zlib include directory: /home/payne/aarch-aws-env/include
--   Zlib library: /home/payne/aarch-aws-env/lib/libz.so
-- Encryption: Openssl
-- Found OpenSSL: /home/payne/aarch-aws-env/lib/libssl.so;/home/payne/aarch-aws-env/lib/libcrypto.so (found version "1.1.1")
--   Openssl include directory: /home/payne/aarch-aws-env/include
--   Openssl library: /home/payne/aarch-aws-env/lib/libssl.so;/home/payne/aarch-aws-env/lib/libcrypto.so
-- Http client: Curl
-- Found CURL: /home/payne/aarch-aws-env/lib/libcurl.so (found version "7.61.1")
--   Curl include directory: /home/payne/aarch-aws-env/include
--   Curl library: /home/payne/aarch-aws-env/lib/libcurl.so
-- Performing Test HAVE_ATOMICS_WITHOUT_LIBATOMIC
-- Performing Test HAVE_ATOMICS_WITHOUT_LIBATOMIC - Success
-- Considering s3
-- Considering transfer
-- Considering core
-- The C compiler identification is GNU 7.1.1
-- Check for working C compiler: /usr/local/linaro-aarch64-2017.08-gcc7.1/bin/aarch64-linux-gnu-gcc
-- Check for working C compiler: /usr/local/linaro-aarch64-2017.08-gcc7.1/bin/aarch64-linux-gnu-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Updating version info to 1.6.18
-- Custom memory management enabled; stl objects now using custom allocators
-- Configuring done
-- Generating done
-- Build files have been written to: /home/payne/aws_make
payne@38EN1590:~/aws_make$ make install
Scanning dependencies of target aws-cpp-sdk-core
[  0%] Building CXX object aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/source/Aws.cpp.o
[  0%] Building CXX object aws-cpp-sdk-core/CMakeFiles/aws-cpp-sdk-core.dir/source/AmazonSerializableWebServiceRequest.cpp.o
Can you provide a TRACE level log? (sanitize any sensitive information)
.......
[ 89%] Building CXX object testing-resources/CMakeFiles/testing-resources.dir/source/TestingEnvironment.cpp.o
[ 89%] Building CXX object testing-resources/CMakeFiles/testing-resources.dir/source/MemoryTesting.cpp.o
[ 89%] Building CXX object testing-resources/CMakeFiles/testing-resources.dir/source/external/gtest-all.cc.o
[ 89%] Building CXX object testing-resources/CMakeFiles/testing-resources.dir/source/platform/linux-shared/PlatformTesting.cpp.o
[ 90%] Linking CXX shared library libtesting-resources.so
[ 90%] Built target testing-resources
Scanning dependencies of target aws-cpp-sdk-s3-integration-tests
[ 90%] Building CXX object aws-cpp-sdk-s3-integration-tests/CMakeFiles/aws-cpp-sdk-s3-integration-tests.dir/BucketAndObjectOperationTest.cpp.o
[ 91%] Building CXX object aws-cpp-sdk-s3-integration-tests/CMakeFiles/aws-cpp-sdk-s3-integration-tests.dir/RunTests.cpp.o
[ 91%] Linking CXX executable aws-cpp-sdk-s3-integration-tests
/usr/local/linaro-aarch64-2017.08-gcc7.1/bin/../lib/gcc/aarch64-linux-gnu/7.1.1/../../../../aarch64-linux-gnu/bin/ld: warning: libcurl.so.4, needed by ../aws-cpp-sdk-core/libaws-cpp-sdk-core.so, not found (try using -rpath or -rpath-link)
/usr/local/linaro-aarch64-2017.08-gcc7.1/bin/../lib/gcc/aarch64-linux-gnu/7.1.1/../../../../aarch64-linux-gnu/bin/ld: warning: libssl.so.1.1, needed by ../aws-cpp-sdk-core/libaws-cpp-sdk-core.so, not found (try using -rpath or -rpath-link)
/usr/local/linaro-aarch64-2017.08-gcc7.1/bin/../lib/gcc/aarch64-linux-gnu/7.1.1/../../../../aarch64-linux-gnu/bin/ld: warning: libcrypto.so.1.1, needed by ../aws-cpp-sdk-core/libaws-cpp-sdk-core.so, not found (try using -rpath or -rpath-link)
/usr/local/linaro-aarch64-2017.08-gcc7.1/bin/../lib/gcc/aarch64-linux-gnu/7.1.1/../../../../aarch64-linux-gnu/bin/ld: warning: libz.so.1, needed by ../aws-cpp-sdk-core/libaws-cpp-sdk-core.so, not found (try using -rpath or -rpath-link)
../aws-cpp-sdk-core/libaws-cpp-sdk-core.so: undefined reference to EVP_CIPHER_CTX_ctrl@OPENSSL_1_1_0' ../aws-cpp-sdk-core/libaws-cpp-sdk-core.so: undefined reference to curl_easy_reset'
../aws-cpp-sdk-core/libaws-cpp-sdk-core.so: undefined reference to EVP_md5@OPENSSL_1_1_0' ../aws-cpp-sdk-core/libaws-cpp-sdk-core.so: undefined reference to EVP_sha256@OPENSSL_1_1_0'
../aws-cpp-sdk-core/libaws-cpp-sdk-core.so: undefined reference to ERR_get_error@OPENSSL_1_1_0' ../aws-cpp-sdk-core/libaws-cpp-sdk-core.so: undefined reference to EVP_DecryptUpdate@OPENSSL_1_1_0'
../aws-cpp-sdk-core/libaws-cpp-sdk-core.so: undefined reference to `EVP_CIPHER_CTX_copy@OPENSSL_1_1_0'
........

I found it is the issue that compile integration test will meet , and maybe some problem of my tool-chain, build shared-library no problem. so I will close it.

Hi @shenhuaz
I'm facing the same problem. Could you please let us know the steps you used to fix this?

Hello @abhijitnathwani
Integration tests build failed had no influence for me, I can still use cpp sdk shared libs when I meet this error because the shared libs already created. In addition,  I put those not found libs (curl, openssl....and so on) into my toolchain/lib dir, then I can build success, maybe it can help you.

Hi @shenhuaz
That still does not help.
If I ignore the integration-tests failure, i cannot run make install. It throws errors.
Alternatively, I also added the .so files in toolchain/lib but the error still remains.
Do you have any exact steps I could use? That'd be helpful.

hi @abhijitnathwani
maybe you can use argument -DENABLE_TESTING=OFF when cmake, it can disable Integration Test.
BTW, I don't use make install and I just find .so in build folder and copy it to my install folder(create include and lib by myself).

@shenhuaz
Thank you! That works well.

authors: abhijitnathwani shenhuaz


text:
Update Default Provider Chain to include ProcessCredentialsProvider

Issue #, if available:
1388
Description of changes:
Add the ProcessCredentialsProvider to the default chain. At least Boto and Go SDKs appear to include it in their default chains.
Check all that applies:

[X] Did a review by yourself.
[ ] Added proper tests to cover this PR. (If tests are not applicable, explain.)
[ ] Checked if this PR is a breaking (APIs have been changed) change.
[ ] Checked if this PR will not introduce cross-platform inconsistent behavior.
[ ] Checked if this PR would require a ReadMe/Wiki update.

Check which platforms you have built SDK on to verify the correctness of this PR.

[X] Linux
[ ] Windows
[ ] Android
[ ] MacOS
[ ] IOS
[ ] Other Platforms

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Super small change, not sure if more is needed. My C++ skills are also rather lacking if there is more that needs to be done.

this pr has been merged, Thanks again for the contribution!


text:
Verification codes other than the first one are not working

Describe the bug
I'm not actually sure whether this is SDK or Cognito problem or maybe I just missed something, but any help would be much appreciated.
I have a user pool set up in AWS Cognito. I've created a user and made it confirmed be changing the password via RespondToNewPasswordRequiredAsync.
I'm trying to implement 'Forgot Password' feature in my system. After making a 'ForgotPasswordAsync' I successfully receive a verification code.
However, I also try to implement a 'Resend verification code' feature. And whilst I do receive a new code, I get an 'Invalid verification code' exception. And this happens for every code other than the first one.

I thought that maybe there is some time that enables new codes, but I've tried to wait 1, 5 and 10 minutes and no success was accomplished.
As I said, the first one received is working with no issues whatsoever.

Expected Behavior
Any received code should work or at least the latest one.
Current Behavior
Only the first code received works
Reproduction Steps

Create a user pool in AWS Cognito.
Create a new user with verified email.
Change this user's password so it becomes 'Confirmed'.
Make a 'Forgot Password' request with SDK.
Receive a code but ignore it
Make a 'Resend confirmation code' request with SDK.
Make a 'Confirm Forgot Password' request with the latest code received

Possible Solution
No solutions were found
Additional Information/Context
No response
AWS .NET SDK and/or Package version used
Amazon.Extensions.CognitoAuthentication: 2.5.2
AWSSDK.CognitoIdentity: 3.7.300.10
AWSSDK.CognitoIdentityProvider: 3.7.300.10
AWSSDK.Core: 3.7.300.10
Targeted .NET Platform
.NET 6
Operating System and version
Windows 10, AmazonLinux (docker container)

Needs investigation.

@MixaKonan Good afternoon. Most likely this is a Cognito service issue if it is reproducible. The ConfirmForgotPassword is the service API operation invoked by CognitoUser.ConfirmForgotPasswordAsync(). I guess you might be resending verification code via CognitoUser.ResendConfirmationCodeAsync(), which invokes service API operation ResendConfirmationCode. So, somehow, there might be a sync issue where Cognito service is unable to verify the code other than the 1st code that was sent. Are you sure you are not using the stale verification code in you application?
Thanks,
Ashish

@ashishdhingra Good afternoon to you too. Indeed you were correct in your assumptions and I was actually using incorrect flow. I missed the difference between 'confirmation code' and 'password reset code' and it's actually stated in the documentation that 'ResendConfirmationCode' is used for registration only.

Resends the confirmation (for confirmation of registration) to a specific user in the user pool.

The issue is resolved and I will close it. Thank you very much for your support and the time wasted.

authors: MixaKonan ashishdhingra

repo: aws/aws-sdk-net-extensions-cognito


text:
S3 Windows phone 8.1 silverlight issue - throwing exception: The request signature we calculated does not match the signature you provided. Check your key and signing method.

I'm getting the following exception being thrown when trying to retrieve an S3 file, however the key/bucket and credentials data worked fine in all the previous versions of the aws sdk we've used (AWS SDk V2 and AWS-SDK-Xamarin beta)
Exception thrown: The request signature we calculated does not match the signature you provided. Check your key and signing method.
throws an exception when used in Windows Phone 8.1 Silverlight project using the latest stable PCL versions from nuget - 3.1 . I presume its something to do with the uri being generated for the S3 object under the hood and the use of the \ in our keys (vx_folder/1.0\datafiles\file.json)
#if __MOBILE__ && WINDOWS_PHONE
    public static Task DownloadToAsync(this S3Link linker, IStorageFile storageFile, CancellationToken cancellationToken = default(CancellationToken))
    {
        return linker.DownloadToAsync(storageFile.Path, cancellationToken);
    }
#endif

public static Task DownloadToAsync(this S3Link linker, string storageFile, CancellationToken cancellationToken = default(CancellationToken))
    {
        var request = new Amazon.S3.Transfer.TransferUtilityDownloadRequest();
        request.BucketName = "sbtfilesbucket"; //linker.BucketName;
        request.Key = "vx_folder/1.0\\datafiles\\file.json"; //linker.Key;
        request.FilePath = "C:\\Data\\Users\\DefApps\\APPDATA\\Local\\Packages\\xxxx\\LocalState\\file.json"; // storageFile;
        TransferUtility util = new TransferUtility("AAAXXXXXXXXXX", "7x7YYYYYYYYYYYYYYY", Amazon.RegionEndpoint.EUWest1);
        return util.DownloadAsync(request, cancellationToken);
    }

Exactly the same App/code and data  (BucketName/Key plus credentials same in all 3 platforms) using the same code Works in iOS/Andriod Xamarin forms, but not within our Windows Phone 8.1 Silverlight project, I've also tried to use the S3.GetObjectAsync and again works for all the other projects but fails for the Windows Phone 8.1 silverlight project.
Also it would be good if the latest version had implemented S3Link.DownloadToAsync which worked and we used (AWS SDK Xamarin Beta)  - will this be added back into the portable libraries?
Any help with this issue/bug would be appreciated as its holding up our release.
Thanks
Paul

It does appear that the slashes are causing this issue. If the key is "vx_folder/1.0/datafiles/file.json", the same code works without issue. Will continue looking into this.

Hi Pavel, has this been investigated any further yet, as its holding up our release to the customer, the issue is the file name and format we're using from version 2 of the SDK worked and we now have thousands of files/entries in this format which creates top level S3 folders and a file name that contains the original folder in the name e.g. "vx_folder/1.0\datafiles\file.json" therefore changing the format isn't an option for us.

Sorry about the radio silence.
Can you try using version 3.1.3 of AWSSDK.S3 or earlier? It looks like this could be a regression introduced with version 3.1.1.1 of AWSSDK.Core. As long as you pick up a version of S3 (and other packages that you are using) that has a dependency on a version of Core lower than 3.1.1.1, you should not encounter this issue.

Hi Pavel thanks for the response and info,
with version 3.1.1.1 of AWSSDK core you're correct and I can confirm the first original issue is resolved, our file/stream is retrieved from S3 which is a step forward,but now under windows phone 8 silverlight the save functions throw an exception  - tested using TransferUtility.DownloadAsync and GetObjectResponse.WriteResponseStreamToFileAsync and both functions return the same error which is a File Exception (see call stack below)
I can however save the S3 response Stream manually to the same file path passed into the DownloadAsync/WriteResponseStreamToFileAsync functions but  using IStorage,OpenStreamForWriteAsync without any issues or exceptions - but it would be better to use a valid/working up to date sdk version - currently we're still releasing our live apps using the beta AWS Mobile SDK for Xamarin as its the only version which works for us on all mobile platforms.
System.IO.FileNotFoundException occurred
HResult=-2147024773
Message=The filename, directory name, or volume label syntax is incorrect. (Exception from HRESULT: 0x8007007B)
Source=mscorlib
StackTrace:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()        at PCLStorage.WinRTFolder.<CreateFolderAsync>d__12.MoveNext()     --- End of stack trace from previous location where exception was thrown ---        at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)        at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at Amazon.S3.Model.GetObjectResponse.d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
at Amazon.S3.Model.GetObjectResponse.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
at Amazon.S3.Transfer.Internal.DownloadCommand.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at GenericCompanyTemplate.Helpers.SaveAndLoad.d__1e.MoveNext()
InnerException:

Glad to hear that the initial issue is resolved. We are working on a long-term fix, but have no estimate at this time.
The issue you are encountering with DownloadAsync/WriteResponseStreamToFileAsync has to do with the fact that the .NET SDK uses PCLStorage to work with storage for PCL assemblies. This means that you need to specify a relative path instead of absolute. For instance, given a full path, you can run the following code (which uses PCLStorage) to determine a relative path, which can then be passed into the SDK:
var relativePath = fullPath
    .Replace(PCLStorage.FileSystem.Current.LocalStorage.Path, "")
    .Trim('\\', '/');

The fix for the signature error has been merged. The latest NuGet packages for S3 and Core have been updated with the fix as well.

repo: aws/aws-sdk-net


text:
EC2ThrottledException on aws lambda call

Hi,
I received EC2ThrottledException with message:

Lambda was throttled while using the Lambda Execution Role to set up for the Lambda function.

when using lambda client. Tried to investigate that error but i have no clue what limit i am hitting and what's the reason of throttling.
Edit:
just found that link https://docs.aws.amazon.com/lambda/latest/dg/scaling.html#scaling-behavior
But i have no idea what they mean (what EC2 rate limit?):

Because Lambda depends on Amazon EC2 to provide Elastic Network Interfaces for VPC-enabled Lambda functions, these functions are also subject to Amazon EC2's rate limits as they scale. If your Amazon EC2 rate limits prevent VPC-enabled functions from adding 500 concurrent invocations per minute, please request a limit increase by following the instructions in the preceding section under To request a limit increase for concurrent executions.
Beyond this rate (i.e. for applications taking advantage of the full Immediate concurrency increase), your application should handle Amazon EC2 throttling (502 EC2ThrottledException) through client-side retry and backoff. For more details, see Error Retries and Exponential Backoff in AWS.

This is not an issue with the SDK and the error response is from the Lambda service. I think the mention of EC2 is a read herring.  Please look at this page here.

authors: pawepaw sstevenkang


text:
[aws-xray-recorder-sdk-apache-http] Don't return a null target to avoid possible NPE

Issue:
The Apache HTTP instrumentation is prone to throwing a NullPointerException by returning a null from the determineTarget method. Since a user is not expected to handle NPE, it will break the application.
Description of changes:
The method doesn't need to check for URL absoluteness and bail out early if it is not. The URIUtils.extractHost(requestUri) method does this check implicitly and we handle it's null return value by throwing a ClientProtocolException which a user will handle in their code.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Just want to confirm: The test case threw a NPE before the fix, and now just throws a ClientProtocolException?

Yes, it did. Running the same unit test without the fix:
TracedHttpClientTest > testExceptionOnRelativeUrl FAILED
    java.lang.AssertionError: unexpected exception type thrown; expected:<org.apache.http.client.ClientProtocolException> but was:<java.lang.NullPointerException>
        at org.junit.Assert.assertThrows(Assert.java:1020)
        at org.junit.Assert.assertThrows(Assert.java:981)
        at com.amazonaws.xray.proxies.apache.http.TracedHttpClientTest.testExceptionOnRelativeUrl(TracedHttpClientTest.java:124)
        Caused by:
        java.lang.NullPointerException
            at com.amazonaws.xray.proxies.apache.http.TracedHttpClient.execute(TracedHttpClient.java:180)
            at com.amazonaws.xray.proxies.apache.http.TracedHttpClientTest.lambda$testExceptionOnRelativeUrl$0(TracedHttpClientTest.java:125)
            at org.junit.Assert.assertThrows(Assert.java:1001)
            ... 2 more

repo: aws/aws-xray-sdk-java


text:
Build eks-a with path trimmed from stack traces for release builds

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/cherrypick release-0.7

authors: abhay-krishna g-gaston

repo: aws/eks-anywhere


text:
chore: attempt to fix integration test

It is currently broken because the latest version of yargs (16.0.3)
introduced an exports map in it's package.json that prevents access
to it's package.json. This is used by decdk when trying to load the
type system, and currently fails.
Since the aws-cdk codebase does not currently have this problem, it
means the version of yargs that is being run with does not match the
one in the aws-cdk lockfile. This might be cause of the use of npm install to install the "packages under test", which does not honor the
yarn.lock file and consequently upgrades yargs unexpectedly. This
switches to using yarn add instead, which will hopefully not affect
the other yargs dependencies that are locked.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: 9c920dd063af782c75653103be2fd18ebad534a1
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: a3f949a29c0b4300181ae0d632c44cb293c5307f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: a6bde9540ab879260180082be2338fb5dcde206e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: d0992aca7f780dd55800382bd223b033bf0c3145
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: 46f290d177007b9df93a6a6a3ec50716c3fa06d5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: ae09bdf3102b048f771c323121248207b7cb885c
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: d1ee2d86b90de26ee40ed51f6d13d0d43ae64919
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

authors: RomainMuller aws-cdk-automation

repo: aws/jsii


text:
Karpenter chooses the provisioner with the lower weight

Version
Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.14-eks-18ef993", GitCommit:"ac73613dfd25370c18cbbbc6bfc65449397b35c7", GitTreeState:"clean", BuildDate:"2022-07-06T18:06:50Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Expected Behavior
Karpenter should favor the provisioner with the higher weight.
Actual Behavior
Karpenter chooses the provisioner with the lower weight.
Steps to Reproduce the Problem
Create the provisioner and pod according to the yamls attached below.
Resource Specs and Logs
First provisioner -
    apiVersion: karpenter.sh/v1alpha5
    kind: Provisioner
    metadata:
      name: small-gpu-v100-compute
    spec:
      weight: 100
      kubeletConfiguration:
        containerRuntime: containerd
      providerRef:
        name: bottlerocket-template
        - key: karpenter.k8s.aws/instance-gpu-name
          operator: In
          values:
            - v100
        - key: karpenter.k8s.aws/instance-gpu-manufacturer
          operator: In
          values:
            - nvidia
        - key: node.kubernetes.io/instance-type
          operator: NotIn
          values:
            - p3.16xlarge
            - p3dn.24xlarge
      taints:
        - effect: NoSchedule
          key: type
          value: small-gpu
      ttlSecondsAfterEmpty: 1800

Second provisioner -
    apiVersion: karpenter.sh/v1alpha5
    kind: Provisioner
    metadata:
      name: small-gpu-t4-compute
    spec:
      weight: 50
      kubeletConfiguration:
        containerRuntime: containerd
      providerRef:
        name: bottlerocket-template
      requirements:
        - key: karpenter.k8s.aws/instance-gpu-name
          operator: In
          values:
            - t4
        - key: karpenter.k8s.aws/instance-gpu-manufacturer
          operator: In
          values:
            - nvidia
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values:
            - g4dn
      taints:
        - effect: NoSchedule
          key: type
          value: small-gpu
      ttlSecondsAfterEmpty: 1800

Pod spec -
spec:
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: REDACTED
    - name: ephemeral
      emptyDir: {}
  containers:
    - name: REDACTED
      image: >-
        REDACTED
      env:
        - name: REDACTED
          value: REDACTED
      resources:
        limits:
          cpu: '2'
          memory: 4G
          nvidia.com/gpu: '1'
        requests:
          cpu: '2'
          memory: 4G
          nvidia.com/gpu: '1'
      volumeMounts:
        - name: data
          mountPath: REDACTED
        - name: ephemeral
          mountPath: /ephemeral
      securityContext:
        capabilities:
          drop:
            - ALL
        allowPrivilegeEscalation: false
  securityContext:
    runAsUser: 65532
    runAsGroup: 65532
    supplementalGroups:
      - 1
    fsGroup: 65532
  tolerations:
    - key: type
      operator: Equal
      value: small-gpu
      effect: NoSchedule
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule

Karpenter log -
2022-09-11T09:22:16.142Z	INFO	Successfully created the logger.
2022-09-11T09:22:16.142Z	INFO	Logging level set to: debug
{"level":"info","ts":1662888136.146486,"logger":"fallback","caller":"injection/injection.go:61","msg":"Starting informers..."}
2022-09-11T09:22:16.146Z	INFO	controller	Initializing with version v0.16.1	{"commit": "b157d45"}
2022-09-11T09:22:16.146Z	INFO	controller	Setting GC memory limit to 966367641, container limit = 1073741824	{"commit": "b157d45"}
2022-09-11T09:22:16.171Z	DEBUG	controller.aws	Using AWS region us-east-1	{"commit": "b157d45"}
2022-09-11T09:22:16.354Z	DEBUG	controller.aws	Discovered caBundle, length 1066	{"commit": "b157d45"}
2022-09-11T09:22:16.354Z	INFO	controller	loading config from karpenter/karpenter-global-settings	{"commit": "b157d45"}
I0911 09:22:16.461763       1 leaderelection.go:243] attempting to acquire leader lease karpenter/karpenter-leader-election...
2022-09-11T09:22:16.462Z	INFO	controller	starting metrics server	{"commit": "b157d45", "path": "/metrics"}
2022-09-11T09:22:16.817Z	INFO	controller.aws.pricing	updated spot pricing with 558 instance types and 2629 offerings	{"commit": "b157d45"}
2022-09-11T09:22:18.207Z	INFO	controller.aws.pricing	updated on-demand pricing with 558 instance types	{"commit": "b157d45"}
......
2022-09-13T19:05:11.994Z	INFO	controller.provisioning	Found 1 provisionable pod(s)	{"commit": "b157d45"}
2022-09-13T19:05:11.994Z	INFO	controller.provisioning	Computed 1 new node(s) will fit 1 pod(s)	{"commit": "b157d45"}
2022-09-13T19:05:11.999Z	INFO	controller.provisioning	Launching node with 1 pods requesting {"cpu":"2145m","memory":"3939018Ki","nvidia.com/gpu":"1","pods":"8"} from types g4dn.xlarge, g4dn.2xlarge, g4dn.4xlarge, g4dn.8xlarge, g4dn.12xlarge and 2 other(s)	{"commit": "b157d45", "provisioner": "small-gpu-t4-compute"}
2022-09-13T19:05:12.139Z	DEBUG	controller.provisioning.cloudprovider	Discovered security groups: [sg-00ddcf93ec1802d03]	{"commit": "b157d45", "provisioner": "small-gpu-t4-compute"}
2022-09-13T19:05:12.140Z	DEBUG	controller.provisioning.cloudprovider	Discovered kubernetes version 1.21	{"commit": "b157d45", "provisioner": "small-gpu-t4-compute"}
2022-09-13T19:05:12.579Z	DEBUG	controller.provisioning.cloudprovider	Created launch template, Karpenter-hadc-2043405-eks-8779073898232307813	{"commit": "b157d45", "provisioner": "small-gpu-t4-compute"}
2022-09-13T19:05:14.597Z	INFO	controller.provisioning.cloudprovider	Launched instance: i-08d8fbf7926bc15ea, hostname: ip-10-0-1-236.ec2.internal, type: g4dn.xlarge, zone: us-east-1f, capacityType: on-demand	{"commit": "b157d45", "provisioner": "small-gpu-t4-compute"}

Hi @ozahavi , looking at the capacity pools, I can see that there is no availability for the two instance types that would be able to be selected by the first provisioner, which is why we ended up falling back to the next one with lower weight.
We are talking about ways that we can improve the observability of this so that it's easy for users to see what is available at a given time and also if a provisioner was considered during the scheduling loop.

I re-ran the same test and still got a g4dn.xlarge instance, while at the same exact moment I was able to create a p3.2xlarge on the same AZ. So IIUC it doesn't look like it is an instance availability issue.

If you just have the single provisioner that is requesting the p3 instances, is it able to schedule the pod or does it fail to schedule?

@ozahavi If you just have the single provisioner that is requesting the p3 instances, is it able to schedule the pod or does it fail to schedule?

Yes. If needed I can replicate that and attach the Karpenter logs.

@ozahavi Yes, can you please share the logs?

When using both provisioners small-gpu-v100-compute and small-gpu-t4-compute -
022-10-03T07:30:30.962Z	INFO	controller.provisioning	Launching node with 1 pods requesting {"cpu":"145m","memory":"57376Mi","nvidia.com/gpu":"1","pods":"8"} from types g4dn.4xlarge, g4dn.8xlarge, g4dn.12xlarge, g4dn.16xlarge, g4dn.metal	{"commit": "b157d45", "provisioner": "small-gpu-t4-compute"}
2022-10-03T07:30:31.260Z	DEBUG	controller.provisioning.cloudprovider	Discovered security groups: [sg-00ddcf93ec1802d03]	{"commit": "b157d45", "provisioner": "small-gpu-t4-compute"}
2022-10-03T07:30:31.262Z	DEBUG	controller.provisioning.cloudprovider	Discovered kubernetes version 1.21	{"commit": "b157d45", "provisioner": "small-gpu-t4-compute"}
2022-10-03T07:30:31.322Z	DEBUG	controller.provisioning.cloudprovider	Discovered ami-07d5c13b0aab6494c for query "/aws/service/bottlerocket/aws-k8s-1.21-nvidia/x86_64/latest/image_id"	{"commit": "b157d45", "provisioner": "small-gpu-t4-compute"}
2022-10-03T07:30:31.517Z	DEBUG	controller.provisioning.cloudprovider	Created launch template, Karpenter-hadc-2043405-eks-8779073898232307813	{"commit": "b157d45", "provisioner": "small-gpu-t4-compute"}
2022-10-03T07:30:34.612Z	INFO	controller.provisioning.cloudprovider	Launched instance: i-01f4d48cb26000a71, hostname: ip-10-0-21-214.ec2.internal, type: g4dn.4xlarge, zone: us-east-1b, capacityType: on-demand	{"commit": "b157d45", "provisioner": "small-gpu-t4-compute"}
2022-10-03T07:32:07.626Z	DEBUG	controller.aws.launchtemplate	Deleted launch template Karpenter-hadc-2043405-eks-8779073898232307813 (lt-0b3a99d5b386d563a)	{"commit": "b157d45"}

After deleting small-gpu-t4-compute -
2022-10-03T07:41:45.841Z	INFO	controller.provisioning	Found 1 provisionable pod(s)	{"commit": "b157d45"}
2022-10-03T07:41:45.841Z	INFO	controller.provisioning	Computed 1 new node(s) will fit 1 pod(s)	{"commit": "b157d45"}
2022-10-03T07:41:45.841Z	INFO	controller.provisioning	Launching node with 1 pods requesting {"cpu":"145m","memory":"57376Mi","nvidia.com/gpu":"1","pods":"8"} from types p3.8xlarge	{"commit": "b157d45", "provisioner": "small-gpu-v100-compute"}
2022-10-03T07:41:46.217Z	DEBUG	controller.provisioning.cloudprovider	Created launch template, Karpenter-hadc-2043405-eks-8556837692309183061	{"commit": "b157d45", "provisioner": "small-gpu-v100-compute"}
2022-10-03T07:41:48.471Z	INFO	controller.provisioning.cloudprovider	Launched instance: i-092c2a3308a02e447, hostname: ip-10-0-15-73.ec2.internal, type: p3.8xlarge, zone: us-east-1f, capacityType: on-demand	{"commit": "b157d45", "provisioner": "small-gpu-v100-compute"}
2022-10-03T07:42:07.611Z	DEBUG	controller.aws.launchtemplate	Deleted launch template Karpenter-hadc-2043405-eks-8779073898232307813 (lt-09a0d816d1001d7e8)	{"commit": "b157d45"}

@jonathan-innis any ideas on how to find the root cause for this behavior?

I Updated Karpenter to latest and checked this again and it isn't reproducing. Closing issue.

authors: jonathan-innis ozahavi

repo: aws/karpenter


text:
Docs search should return newer docs versions first

Is an existing page relevant?
No response
What karpenter features are relevant?
N/A
How should the docs be improved?
Newer versions should be returned in search first. On top of this, we should have more accurate search results.
i.e. When I search for a doc by name, I should receive that doc as the first search result. The example I've included shows me searching for "Settings" and not getting the settings doc returned in any of the search results.

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Ideally there should be a way to search only a specific version docs too (i.e. version that is currently being viewed), it is quite tedious shifting through search results as is.

authors: jonathan-innis slobo


text:
fix(s2n-quic-platform): Fix compilation in non-pktinfo platforms

Clearly this was meant to be pktinfo_size

I'm curious to know what platform you're using so we can add it to the CI. Ideally we would be testing for this 😄.

I'm actually on quite a strange platform: gramine.
Basically they LD_PRELOAD glibc and friends and implement a subset of Linux syscalls: https://gramine.readthedocs.io/en/latest/devel/features.html#tcp-ip-and-udp-ip-sockets
Since s2n-quic's feature detection in quic/s2n-quic-platform/build.rs is compilation and target_os-based, all the Linux features (sockopts) were enabled, which means it didn't run in gramine.
I had to patch build.rs like so:
diff --git a/quic/s2n-quic-platform/build.rs b/quic/s2n-quic-platform/build.rs
index 0777e6cf..0b65275f 100644
--- a/quic/s2n-quic-platform/build.rs
+++ b/quic/s2n-quic-platform/build.rs
@@ -6,6 +6,13 @@ use std::{fs::read_dir, io::Error, path::Path, process::Command};
 fn main() -> Result<(), Error> {
     let env = Env::new();

+    const SIMPLE_ENV_VAR: &str = "S2N_SIMPLE_ONLY";
+    println!("cargo:rerun-if-env-changed={}", SIMPLE_ENV_VAR);
+    let is_simple = std::env::var(SIMPLE_ENV_VAR).is_ok();
+    if is_simple {
+        return Ok(())
+    }
+
     for feature in read_dir("features")? {
         let path = feature?.path();
         if let Some(name) = path.file_stem() {

In order to turn off the features.
Thankfully the pktinfo_size change was all that was needed afterwards to make things compile and work.
Not sure if my build.rs patch is good but perhaps it'd make sense to have some flag that turns off all platform features to test the simple modules in your CI?

Not sure if my build.rs patch is good but perhaps it'd make sense to have some flag that turns off all platform features to test the simple modules in your CI?

Yeah I think it would be a good idea to have some way to disable all of the features to prevent issues like this. Would you be able to open an issue for it and we can discuss it there?

Sure!

Opened: #1850

authors: camshaft dongcarl


text:
Include EFA Drivers

What would you like to be added:
EFA drivers included in the AMI.
Why is this needed:
Need to run an involved installation process in user-data to install EFA drivers today.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start.html#efa-start-enable
Additional Info:
The EFA installer supports a minimal install which only includes the kernel module and rdma-core or a minimal install which includes openmpi and libfabric.
The disk space required for the install options are:
drivers + libfabric + openmpi: 608.45 MiB
minimal (only drivers): 495.42 MiB
Sample Install  Script
## Setup
EFA_VERSION="1.25.1" # or latest
mkdir -p /tmp/efa-installer
cd /tmp/efa-installer

## Download Installer
curl -O https://efa-installer.amazonaws.com/aws-efa-installer-${EFA_VERSION}.tar.gz

## Signature verification
curl -O https://efa-installer.amazonaws.com/aws-efa-installer.key
curl -O https://efa-installer.amazonaws.com/aws-efa-installer-${EFA_VERSION}.tar.gz.sig
KEY=$(gpg --import aws-efa-installer.key 2>&1 | grep -o 'key [0-9A-Z]\+' | cut -d' ' -f2)
EXPECTED_FP=$(gpg --fingerprint "${KEY}" 2>&1 | grep -o 'Key fingerprint = [0-9A-Z ]\+' | cut -d'=' -f2 | tr -d " ")
gpg --verify ./aws-efa-installer-${EFA_VERSION}.tar.gz.sig &> siginfo
if [[ $? -ne 0 ]]; then
    echo "EFA Installer signature failed verification!"
    exit 2
fi
ACTUAL_FP=$(cat siginfo  | grep -o 'Primary key fingerprint: [0-9A-Z ]\+' | cut -d':' -f2 | tr -d " ")
if [[ "${EXPECTED_FP}" != "${ACTUAL_FP}" ]]; then
    echo "EFA Installer fingerprints do not match!"
    exit 2
fi

## Extract and Install
tar -xf aws-efa-installer-${EFA_VERSION}.tar.gz && cd aws-efa-installer
sudo ./efa_installer.sh --yes --no-verify

## Cleanup
cd /
rm -rf /tmp/efa-installer

curl -O https://efa-installer.amazonaws.com/aws-efa-installer-${EFA_VERSION}.tar.gz
Will this be the unified URL to get this package in all partitions? Especially would like to check it for MVP regions.

repo: awslabs/amazon-eks-ami


text:
Tabular: Improved logging clarity for positive_class

Issue #, if available:
Description of changes:

Improved logging clarity for positive_class

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Job PR-974-4 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-974/4/index.html

authors: Innixma szha

repo: awslabs/autogluon


text:
Deploy jar-with-dependencies to Maven Central and in GitHub releases

As of now, only the "regular" jar is deployed to Maven Central: http://search.maven.org/#artifactdetails|com.amazonaws|aws-apigateway-importer|1.0.1|jar
By deploying the built uber-jar, you enable easier distribution and usage.

:+1:

Hi, we're working on getting the latest updates released. ETA is by end of week.

:+1: This would be really useful for us, too.

Swagger/OpenAPI import is now generally available in the API Gateway REST API, the AWS CLI and all AWS SDKs. You can also import and export Swagger definitions using the API Gateway console. This release addresses many of the open issues and feedback in this repository.
I would encourage you to migrate your workflow to the standard AWS tools. aws-apigateway-importer will receive minimal support from the API Gateway team going forward.
Any feedback or issues with the new importer should be directed to the official API Gateway forums.
Thanks,
Ryan

authors: jackrk jjmartin moshen petemounce rpgreen

repo: awslabs/aws-apigateway-importer

license_source: bigquery


text:
ApplicationLoadBalancer adds unexpected ingress rules to security group

Describe the bug
I'm trying to create an ApplicationLoadBalancer with predefined security group rules to allow traffic only from specific IP addresses.
To Reproduce
    const securityGroup = new ec2.SecurityGroup(this, `Rule`, {
      vpc,
    });
    securityGroup.addIngressRule(new ec2.CidrIPv4('1.1.1.1/27'), new ec2.TcpPort(443), 'IP address');
    const loadBalancer = new elbv2.ApplicationLoadBalancer(this, `LoadBalancer`, {
      vpc,
      internetFacing: true,
      securityGroup,
    });

The generated code seems to incorrectly allow traffic from any IP address for some reason:
        - CidrIp: 0.0.0.0/0
          Description: Allow from anyone on port 443
          FromPort: 443
          IpProtocol: tcp
          ToPort: 443

Expected behavior
I would expect the traffic to be allowed only from the specified addresses.
Version:

OS: macOS
Programming Language: Typescript
CDK Version: 0.31.0

Can you try adding open: false in the listener props?

@jogold thanks for noticing! I wasn't aware of this property as it wasn't explicitly required.
    const listener = loadBalancer
      .addListener(`HttpsListener`, {
        port: 443,
        certificateArns: [certificate.certificateArn],
        open: false, // This will do the trick
      });

authors: jogold markusl

repo: awslabs/aws-cdk


text:
Pass 'read_dictionary' kwarg to pyarrow

Thanks for the great package! The Athena CTAS method is much faster than more robust than other packages/methods.
It appears the underlying pyarrow package memory usage spikes when using wr.s3.read_parquet() and since it uses it, wr.athena.read_sql_query().
With 16 GB RAM, my python kernel is crashing due to a memory error (checking free in separate shell during execution shows this).
Looking for work arounds I found pyarrow has the kwarg read_dicitonary in pyarrow.parquet.ParquetDataset which reduces memory use and speeds up read by directly reading string columns as categories. Pandas's read_parquet passes kwargs to the underlying engine (pyarrow).
To check this I downloaded the parquet files the Athena query generated:
Line #    Mem usage    Increment   Line Contents
================================================
     4     76.1 MiB     76.1 MiB   @profile
     5                             def read_parquet_files(files):
     6     76.1 MiB      0.0 MiB       dfs=[]
     7   5248.9 MiB      0.0 MiB       for file in files:
     8   5248.9 MiB   3701.6 MiB           df = pd.read_parquet(file,read_dictionary=['a','b'])
     9   5248.9 MiB      0.0 MiB           dfs.append(df)
    10   4694.2 MiB      0.0 MiB       total_df = pd.concat(dfs,ignore_index=True,copy=False)

real	0m15.570s
user	0m11.107s
sys	0m7.833s

Ultimately this leaves the categorized columns as strings. They can be kept as categories, and further reducing memory and improving speed by using pandas.api.types.union_categoricals()
Line #    Mem usage    Increment   Line Contents
================================================
     5     76.5 MiB     76.5 MiB   @profile
     6                             def read_parquet_files(files):
     7     76.5 MiB      0.0 MiB       dfs = []
     8     76.5 MiB      0.0 MiB       a_categories = []
     9     76.5 MiB      0.0 MiB       b_categories = []
    10   5279.5 MiB      0.0 MiB       for file in files:
    11   5279.5 MiB   3800.8 MiB           df = pd.read_parquet(file,read_dictionary=['a','b'])
    12
    13   5279.5 MiB      0.0 MiB           a_categories.append(df['a'])
    14   5279.5 MiB      0.0 MiB           b_categories.append(df['b'])
    15
    16   5279.5 MiB      0.0 MiB           dfs.append(df)
    17   2885.0 MiB      0.0 MiB      total_a_categories = union_categoricals(a_cats).categories
    18   2046.0 MiB      0.0 MiB       total_b_categories = union_categoricals(b_cats).categories
    19   2046.0 MiB      0.0 MiB       for i in range(len(dfs)):
    20   2046.0 MiB      0.0 MiB           dfs[i]['a'].cat.set_categories(total_a_categories,inplace=True)
    21   2046.0 MiB      0.0 MiB           dfs[i]['b'].cat.set_categories(total_b_categories,inplace=True)
    22   3538.4 MiB   1492.5 MiB       total_df = pd.concat(dfs,ignore_index=True,copy=False)

real	0m6.288s
user	0m8.040s
sys	0m4.487s

A possible simple enhancement to awswrangler would being able to pass kwargs to pandas/pyarrow.parquet.ParquetDataset.
A more involved enhancement would be combining categoricals when read_dictionary is not None.

@sapientderek thanks! Really cool analyses!!! 👏👏👏
I'm messing around with your suggestion, let's see what I will be able to adapt on the next release.

@sapientderek feature available on version 1.0.1. Thanks!

authors: igorborgest sapientderek

repo: awslabs/aws-data-wrangler


text:
I specify a start time only, but the running period is ~23:59.

I specify a start time only, but the running period is ~23:59.
(I am not fixed the issue yet -https://github.com/awslabs/aws-instance-scheduler/pull/19)

Implementation Guide said:
--endtime
The time when the running period stops. If begintime and endtime are not specified,
the running period is 00:00 – 23:59.
...
If you specify a start time only, the instance must be stopped manually. Similarly,
if you specify a stop time only, the instance must be started manually. If you don’t specify
either time, the solution uses the days of the week, days of the month, or months rules to start
and stop instances.
https://s3.amazonaws.com/solutions-reference/aws-instance-scheduler/latest/instance-scheduler.pdf
and would it be helpful to add a "override_status=running" field in this situation?

Bob,
Are you using any day or weekday settings in the period?

Yes, period settings :
weekday:[1], begintime:”15:30”
Umm..If the weekly schedule is set, Is it only running until the day of the week?

Exactly.
All criteria that are defined for a period are used (AND) to determine the desired state. So if you use one, or more, of the date/day criteria thse must all match for that period. If a schedule has multple periods then the instance will run if at least one (OR) has a desired running state. The last desired state is stored, and the scheduler will take action if the new desired state differs from the last desired state. This allows you to manually start/stop, without the scheduler stop/start it again if the instance state differs from the desired state. It will as not use the actual instance state. If you use the enforced flag on a schedule then it will compare the actual instance state with the desired state, so the scheduler will always try to bring the instance back again into the desired schedule state.
If you use a start time only, the state before the start time will be any (basically leave status as is) and running if time >= start time. It will stop at 23:59 if you use a day/date setting for that period and no other period in that schudule has a desired running state.
Hope this expains the logic and ideas behind the scheduler logic
Regards
Arie

Then...
periods1  = weekdays:{"0"}, endtime:”23:59”
periods2  = weekdays:{"1"}, begintime:”15:30”
periods3  = weekdays:{"2-6"}, endtime:”23:59”
schedule = [periods1, periods2, periods3]
This will  try to start at 15:30 PM on Tuesday and will be stopped manually on the other day.
is it correct?

Closing due to time elapsed since last correspondence. Please re-open this if you are still having an issue.

authors: arieleeuw espozbob shsenior

repo: awslabs/aws-instance-scheduler


text:
fix(docs): indentation in tutorial snippets

Extra indent is triggering: "Unexpected indentation" for python from the sample docs. Fixed the indent as per previous samples.
Issue #, if available:
Description of changes:

Checklist

[x] Meet tenets criteria
[ ] Update tests
[x] Update docs
[x] PR title follows conventional commit semantics

Breaking change checklist

RFC issue #:

[ ] Migration process documented
[ ] Implement warnings (if it can live side by side)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

View rendered docs/tutorial/index.md

Codecov Report

Merging #988 (67ccb5e) into develop (0c35e5a) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop     #988   +/-   ##
========================================
  Coverage    99.96%   99.96%
========================================
  Files          119      119
  Lines         5335     5335
  Branches       607      607
========================================
  Hits          5333     5333
  Partials         2        2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0c35e5a...67ccb5e. Read the comment docs.

Hey @am1ru1  thank you SO MUCH for helping us improve the docs.
I'll rebuild it tomorrow so anyone coming for the latest will have it fixed without triggering a new release.

authors: am1ru1 codecov-commenter heitorlessa

repo: awslabs/aws-lambda-powertools-python


text:
design: sam-app-config

a configuration that provides parameter pass throughs for
sam cli commands

[x] Write Design Document (Do I need to write a design document?)

[ ] Write unit tests

[ ] Write/update functional tests

[ ] Write/update integration tests

[ ] make pr passes

[ ] Write documentation

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

open to modification of --identifier. it could be environment, mode, config.

repo: awslabs/aws-sam-cli


text:
Documentation specifies defaults but they are not actually used

Describe the bug
Documentation explicitly specifies that the following fields have defaults:

But they are actually not applied. If the builder doesn't explicitly specify them, they will instead be explicitly be set to None

This makes the request fail with error:
ValidationException: The provided Content Type is invalid or not supported for this model

Setting those explicitly on InvokeModelFluentBuilder avoids the error:
builder
    .accept("application/json")
    .content_type("application/json")

Links
https://docs.rs/aws-sdk-bedrockruntime/1.18.0/aws_sdk_bedrockruntime/operation/invoke_model/builders/struct.InvokeModelFluentBuilder.html
https://docs.rs/aws-sdk-bedrockruntime/1.18.0/aws_sdk_bedrockruntime/operation/invoke_model/struct.InvokeModelInput.html
Version
1.18.0

Hi @Ten0, thank you for reporting this! This is a bug in a Smithy model for bedrockruntime (from which aws_sdk_bedrockruntime is code generated), where the doc (which also comes from the model) mentions defaults but they are not actually used as fallback; the requests fail if nothing is specified for accept and content_type.
@jmklix Could you help bring this up with bedrockruntime so that the team is aware?

repo: awslabs/aws-sdk-rust


text:
S3 operations do not return a response

Describe the bug
S3 operations return empty responses
Expected Behavior
S3 operations should parse data in their response
Current Behavior
S3 operations return response objects with no data
Reproduction Steps
Perform ListObjectsV2 with Swift SDK for a bucket with objects in it
Observe that contents in the response (as well as all other fields) is nil
Possible Solution
Data is not available when deserializing a S3 response
Additional Information/Context
No response
AWS SWIFT SDK version used
main branch
Compiler and Version used
Xcode 15.2
Operating System and version
macOS 14.2.1

Shipped in AWS SDK for Swift 0.34.0

repo: awslabs/aws-sdk-swift


text:
chore(release): 1.117.0

See CHANGELOG

AWS CodeBuild CI Report

CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
Commit ID: 8efaf7f252d30dd2ba4d6af975ac71ba3018320f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
Commit ID: 362b4d7c512b5638089d811c580cb3433dffdfd7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
Commit ID: ac834c1b9e002cee23d8b0fac1dd81a74ce081ef
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

repo: awslabs/aws-solutions-constructs


text:
Remove Python version dependency in cfncluster-ebsnvme-id.

CentOS 6 ships with Python 2.6, while the ebsnvme-id script expects Python 2.7.
The script works with Python 2.7 as well, so making it use
whichever interpreter is available by default on the system.
Signed-off-by: Raghu Raja craghun@amazon.com

Tested on CentOS 6 running Python 2.6, as well as CentOS 7 running Python 2.7.5.

Will merge this PR after a quick test with Ubuntu AMIs too.

repo: awslabs/cfncluster-cookbook


text:
release: clickstream Swift 0.12.2

Description

release: clickstream Swift 0.12.2

General Checklist

[x] Added new tests to cover change, if needed
[x] Build succeeds using Swift Package Manager
[x] All unit tests pass
[x] Documentation update for the change if required
[x] PR title conforms to conventional commit style
[x] If breaking change, documentation/changelog update with migration instructions

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 96.84%. Comparing base (1e91c07) to head (f290d6d).

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #67   +/-   ##
=======================================
  Coverage   96.84%   96.84%
=======================================
  Files          35       35
  Lines        1681     1681
=======================================
  Hits         1628     1628
  Misses         53       53

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: codecov-commenter zhu-xiaowei

repo: awslabs/clickstream-swift


text:
Installing via services like Jamf

Hello,
I'm wondering if there is a way to install this via a service like Jamf? When I try to create a policy with the specified command, I get the massage that commands cannot run as 'root' which is the way that Jamf deploys this script. I've tried other commands like sudo -u but nothing seems to work.
Basically we already have brew install so it's just a matter of running the brew install for git-secrets but we want to do it without user interaction.

Sorry I dont know, just trying to get my stuff back right now

repo: awslabs/git-secrets


text:
Fix dataset writer tool.

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Please tag this pr with at least one of these labels to make our release process faster: BREAKING, new feature, bug fix, other change, dev setup

Is this a bugfix?

Yes, two for the price of one!

authors: jaheba lostella

repo: awslabs/gluon-ts


text:
small fix for binned distribution

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Job PR-352/1 is complete.
Docs are uploaded to http://gluon-ts-staging.s3-accelerate.dualstack.amazonaws.com/PR-352/1/index.html

Codecov Report

Merging #352 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #352      +/-   ##
==========================================
+ Coverage   80.16%   80.16%   +<.01%
==========================================
  Files         146      146
  Lines        8324     8325       +1
==========================================
+ Hits         6673     6674       +1
  Misses       1651     1651

Impacted Files
Coverage Δ

src/gluonts/distribution/binned.py
96.22% <100%> (+0.03%)
:arrow_up:

Job PR-352/2 is complete.
Docs are uploaded to http://gluon-ts-staging.s3-accelerate.dualstack.amazonaws.com/PR-352/2/index.html

authors: codecov-io szha vafl


text:
Fix style and type issues

Description of changes: Mypy checks just got stricter it seems, and rightfully so!
Edit: fixed some style issues as well
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Please tag this pr with at least one of these labels to make our release process faster: BREAKING, new feature, bug fix, other change, dev setup

Made this breaking because of the change to MetaData

repo: awslabs/gluonts


text:
Optimize the PRF hashes

At the moment s2n computes eight hashes of all of the handshake messages in a connection. This could be optimized to at most two hashes per connection by:

Using one hash object for each hash type. In client_done and server_done we can clone the hash state before computing a digest. This should be an easy win.
As soon as the server hello is ready to be sent (when acting as a server) or received (when acting as a client) we know what hash algorithm will be used. There's no need to keep computing the other ones. In the worst case we need both an MD5 and a SHA1 for the SSLv3 mixed PRF hash.
This could be further optimized by moving the hash computation to after a handler is called. So on the server side, there would be zero unnecessary hashing.

Supporting client auth makes this slightly more complicated. When client auth is configured, we'll need to maintain all of the hashes for our supported sigalgs until the certificate verify message.

A general way of solving this may be to maintain a bitset of "required" hashes for every connection. After every handshake message, we can remove the hashes from this set that are no longer required.

Gave this a shot in https://github.com/raycoll/s2n/commit/052266caaac2d224b78f62650e7e4b85fc25e09d . Needs some tweaks. Will probably hold onto it until we get the mutual auth changes merged.

authors: colmmacc raycoll


text:
add: README for aws-config

Motivation and Context

@rcoh mentioned we were missing this so I copied over the one from here
Description

This adds a README for the aws-config crate
Checklist

[x] I have updated CHANGELOG.md
[x] I have updated aws/SDK_CHANGELOG.md if applicable

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

The aws-config and aws-sdk-config crates are actually two completely different things with unfortunate naming. Whereas one is used for configuring the AWS SDK, the other one is an actual AWS service called Config: https://aws.amazon.com/config/
So the readme won't directly transfer over. That said, you could use this as a good starting point, delete the parts related to the Config service, and fill in those gaps with information about configuring the SDK.

The aws-config and aws-sdk-config crates are actually two completely different things with unfortunate naming. Whereas one is used for configuring the AWS SDK, the other one is an actual AWS service called Config: https://aws.amazon.com/config/
So the readme won't directly transfer over. That said, you could use this as a good starting point, delete the parts related to the Config service, and fill in those gaps with information about configuring the SDK.

How foolish of me! I'll update it.

authors: Velfi jdisanti

repo: awslabs/smithy-rs


text:
Fix query parameter values in SSDK protocol tests

Description of changes:
In other commits, we have adjusted the expectation to be that HttpRequest will
contain encoded paths and query parameter keys and values, and the generated
protocol deserializers will perform deserialization. The protocol tests still
expected the HttpRequest to hold decoded values, which was causing SSDK test
failures.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Nevermind. This is wrong. Neither the SDKs nor APIGateway use encoded query string values in the request -- just paths.

repo: awslabs/smithy-typescript