text:
Create a REST api that can be read by guests and CRUD by users

Category
REST API
Amplify CLI Version
4.32.0
AWS Services

Cognito
DynamoDB

Description
I would like to create a DynmoDB table and access it through a REST API. I want this API to be read by guests, but only CRUD by users. I believe I walked though the CLI to set this up. I added a specific use case to the lambda function:
// blogLambda/src/app.js
...
/**
 * Getting the basic blog details.
 *
 * This is called to get the basics of the blog's details. This returns the
 * number of users and posts.
 */
app.get( `/blog`, function ( request, response ) {
  let params = {
    TableName: tableName,
    Key: {
      'PK': `#BLOG`,
      'SK': `#BLOG`
    }
  }
  dynamodb.get( params, ( error, result ) => {
    if ( error ) {
      response.json( { statusCode: 500, error: error.message } )
    } else {
      response.json(
        {
          statusCode: 200,
          url: request.url,
          body: result.Item
        }
      )
    }
  } )
} )

When I attempt to call this api using the SDK,
API.get( `blogAPI`, `/blog/`, {} )
  .then( blogResponse => console.log( `response`, { blogResponse } ) )
  .catch( error => console.log( `error`, { error } ) )

I get the following errors:
Origin http://localhost:8000 is not allowed by Access-Control-Allow-Origin.
XMLHttpRequest cannot load https://<endpoint>/blog/ due to access control checks.

It works correctly when the post api is called. I can even use Postman to GET the data:

What do I need to do in order to get the GET to work through the front end? I've added the Amplify CLI steps I took to produce this. I'd also like to change the auth attributes used, but I'd like to tackle this one first.
Process
tylernorlund@Tylers-MacBook-Pro Blog % amplify add api
? Please select from one of the below mentioned services: REST
? Provide a friendly name for your resource to be used as a label for this category in the project: blogAPI
? Provide a path (e.g., /book/{isbn}): /
? Choose a Lambda source Create a new Lambda function
? Provide a friendly name for your resource to be used as a label for this category in the project: blogLambda
? Provide the AWS Lambda function name: blogLambda
? Choose the runtime that you want to use: NodeJS
? Choose the function template that you want to use: CRUD function for DynamoDB (Integration with API Gateway)
? Choose a DynamoDB data source option Create a new DynamoDB table

Welcome to the NoSQL DynamoDB database wizard
This wizard asks you a series of questions to help determine how to set up your NoSQL database table.

? Please provide a friendly name for your resource that will be used to label this category in the project: blogDB
? Please provide table name: blogDB

... <adding columns and defining PK/SK/GSI>

? Do you want to add a Lambda Trigger for your Table? Yes
? Select from the following options Create a new function
Successfully added resource blogDBTrigger731a3221 locally
Successfully updated resource blogDBTrigger731a3221 locally
? Do you want to edit the local blogDBTrigger731a3221 lambda function now? No
Successfully added DynamoDb table locally
? Do you want to access other resources in this project from your Lambda function? Yes
? Select the category api, auth, function, storage
No resources found for api
No resources found for auth
Function category has a resource called blogDBTrigger731a3221
? Select the operations you want to permit for blogDBTrigger731a3221 create, read, update, delete
Storage category has a resource called blogDB
? Select the operations you want to permit for blogDB create, read, update, delete

You can access the following resource attributes as environment variables from your Lambda function
        ENV
        FUNCTION_BLOGDBTRIGGER731A3221_NAME
        REGION
        STORAGE_BLOGDB_ARN
        STORAGE_BLOGDB_NAME
? Do you want to invoke this function on a recurring schedule? No
? Do you want to configure Lambda layers for this function? No
? Do you want to edit the local lambda function now? No

So it looks like when this is created, it sets the authorization to AWS_IAM. Is this correct?


text:
chore: fix amplify uninstall command

Description of changes
When v8.0.0 was released, the behavior for running amplify uninstall became... different.
The amplify uninstall command was intended for users of the packaged CLI binary. It essentially causes the binary to self-destruct.
Previously, if amplify was installed via NPM, the cli would do nothing except emit a message instructing the caller to run npm uninstall -g @aws-amplify/cli instead of running amplify uninstall.
Now in NPM installations, since all cli commands are proxied to a packaged binary, run amplify uninstall causes the underlying binary to be destroyed. However, subsequent CLI commands will detect that the binary is missing and reinstall it.
npm i -g @aws-amplify/cli
amplify uninstall # success! amplify is removed
amplify version # One sec while I download the binary... OK version is 8.0.0

This pull request updates the node shim with special case logic for amplify uninstall that automatically also shells out a process to run npm uninstall -g @aws-amplify/cli on behalf of the caller.
npm i -g @aws-amplify/cli
amplify uninstall # success! amplify is removed
amplify version # Command not found!

Issue #, if available
n/a

Description of how you validated changes

tested locally

Checklist

[x] PR description included
[x] yarn test passes

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #10232 (8b2ed51) into master (17b5df0) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #10232   +/-   ##
=======================================
  Coverage   54.28%   54.28%
=======================================
  Files         850      850
  Lines       46948    46948
  Branches    10002    10002
=======================================
  Hits        25487    25487
  Misses      19445    19445
  Partials     2016     2016

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

source: gharchive/pull-request

authors: codecov-commenter johnpc

repo: aws-amplify/amplify-cli

license_type: permissive

license_source: github-api


text:
Unable to deploy artifacts to console, Retrieving deployment details...

Describe the bug
amplify publish is just staying on
Deploying build artifacts to the Amplify Console..
To Reproduce
Steps to reproduce the behavior:
run amplify publish
Expected behavior
It to deploy the code
AWS Amplify Console -> App -> Retrieving deployment details... does not go away

I have tried deploying manually and it just sits at Pending

I have tried deploying manually and it just sits at Pending

opened a support ticket on AWS

opened a support ticket on AWS

source: gharchive/issue

repo: aws-amplify/amplify-console


text:
E3001 AWS::EC2::VPCEndpointService Invalid resource in us-west-2

cfn-lint version: 0.18.1
E3001 on AWS::EC2::VPCEndpointService in us-west-2
The type AWS::EC2::VPCEndpointService is giving an E3001 error.
This is a newer / valid resource https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpcendpointservice.html
But I could not find it in the spec.

The only spec that I'm finding this in is eu-north-1 which is interesting.
We can patch this as a short term fix but we'll need to dig into why this isn't the actual spec file.
Thanks for providing this feedback.

Fixed with release v0.20.0

authors: kddejong kmoschkau

repo: aws-cloudformation/cfn-python-lint


text:
Updates ACK runtime to v0.14.1

No other functional changes included other than runtime update.
Signed-off-by: Jay Pipes jaypipes@gmail.com
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license.

Please remove the now deprecated primary_identifier_field_name entries: https://github.com/aws-controllers-k8s/rds-controller/pull/34/files#diff-da5b4f2a10753269ee491b631d0505f6129d9843a5555a02430dc1428a6412e3R61-R72

nice catch. done!

/test rds-kind-e2e

/retest

The test is failing because the my-aurora-mysql DB cluster already exists from some previous run. I need to clean up the old resources...

rds-kind-e2e

/rds-kind-e2e

/test rds-kind-e2e

authors: A-Hilaly RedbackThomson jaypipes vijtrip2


text:
PRometheus on cluster to Managed Prometheus - what is the required inputs/filters to parse - important aws account(environment), kubernetes cluster, namespace, application pod name and route the monitoring information to central AWS Managed Prometheus? We noticed that the information collected from the bluenrpints prometheus exporter is not able to collect in PRometheus with required minimum information.

[ ] ✋ I have searched the open/closed issues and my issue is not listed.

Please describe your question here

Provide a link to the example/module related to the question

Additional context

Hey, please make sure to correct this issue by placing a correct title and putting the relevant details in the issue details and not in the title itself. Thanks.

authors: Zvikan raviaappala

repo: aws-ia/terraform-aws-eks-blueprints


text:
Issue running with LangChain later version.

I was getting below error when I used latest version of langchain (0.0.178.0).
kendra_chat_flan_xl.py", line 42, in build_chain
llm=SagemakerEndpoint(
File "pydantic/main.py", line 341, in pydantic.main.BaseModel.init
pydantic.error_wrappers.ValidationError: 1 validation error for SagemakerEndpoint
content_handler
instance of LLMContentHandler expected (type=type_error.arbitrary_type; expected_arbitrary_type=LLMContentHandler)
After rolling back to langchain 0.0.137.0, it started working. Is this expected?

Updated to latest version of langchain


text:
Development branch - Alexa instructions - Interaction model

Refers to button below, but there is no button on this page. (Buttons seem to appear on next page)
Also, maybe use 'courier' / monospaced font instead on inverted color for text to be entered.
Suggest revisit of Alexa instructions to ensure instructions match developer console steps, page by page.

fixed in v2

authors: JohnCalhoun rstrahan

repo: aws-samples/aws-ai-qna-bot


text:
Error in latest version of Athena Datasource is preventing deployment of Athena Datasource with ec2 role

https://github.com/grafana/athena-datasource/issues/338

Confirmed 12.6.2 version released and corrects the error


text:
Filter out null and empty TagValues

Issue #, if available: https://github.com/aws-samples/coast-grafana-cost-intelligence-dashboards/issues/38
Description of changes: Change TagValue query in every dashboard and in the standards doc to filter out empty and null values.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Maybe we shouldn't filter out these values, but instead inform 'empty' or 'undefined'.


text:
Fix the ability to specify volume size on compute instances

There is an option to change the root volume size of the head node but not on the compute instances.
This issue is to add the option to specify compute instance root volume size.

Hi @mhuguesaws  -- thanks for your interest and the feature request. This should be relatively easy to add and should be addressed in the next few days.
Cheers!

The following PR should add this support when merged and deployed.
https://github.com/aws-samples/pcluster-manager/pull/117

repo: aws-samples/pcluster-manager


text:
not authorized to perform:                                                                                lambda:GetLayerVersion

The following resource(s) failed to create:
                                                                                [SnapshotFunction]. Rollback requested by
                                                                                user.
CREATE_FAILED                                   AWS::Lambda::Function                           SnapshotFunction                                User: arn:aws:iam::99999999999:user/user is
                                                                                not authorized to perform:
                                                                                lambda:GetLayerVersion on resource:
                                                                                arn:aws:lambda:us-
                                                                                west-2:764866452798:layer:chrome-aws-
                                                                                lambda:22 (Service: AWSLambdaInternal; Status
                                                                                Code: 403; Error Code: AccessDeniedException;
                                                                                Request ID:
                                                                                2a446e2e-eaab-4187-97f3-9e7cfc79742a; Proxy:
                                                                                null)

this is us-west-2 region.

I can confirm that this is still a bug, however since it's just us who have experienced it and there has been no response from Amazon, I am doubtful we will get any help.

authors: ironmanCCT philihp


text:
Queries to services do not return all resources because paging is not being used

Describe the bug
When working on accounts with large numbers of resources in a given service, only a portion of the resources are returned in the query. For example, if we have more than 100 IAM roles, the query will return only the first 100, as this is the default limit of the list_roles method page of the iam service in boto3. (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/iam/client/list_roles.html). This goes for other resources like IAM Policies, secrets in Secrets Manager, KMS Keys, SNS Topics, etc...
To Reproduce
Take an account over 100 IAM roles and start a Resource-Based Policies scan. You will only see the first 100 IAM roles.
Expected behavior
View all IAM roles in the account
Please complete the following information about the solution:

[v1.0.2] Version:

To get the version of the solution, you can look at the description of the created CloudFormation stack.
For example, "(SO0217) - The AWS CloudFormation hub template for deployment of the Account Assessment for AWS Organisations, Version: v1.0.0".

[us-east-1] Region: [e.g. us-east-1]
[no] Was the solution modified from the version published on this repository?
[n/a] If the answer to the previous question was yes, are the changes available on GitHub?
[n/a] Have you checked your service quotas for the sevices this solution uses?
[no] Were there any errors in the CloudWatch Logs?

Screenshots
If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).
Additional context
Add any other context about the problem here.
I collected the information I needed using a personal script and applied paging whenever the resource allowed. e.g:
#Get IAM roles
...
    resource_list = []
    paginator = client.get_paginator('list_roles')
    page_iterator = paginator.paginate()
    for page in page_iterator:
      for item in page.get('Roles'):
        if item is not None:
          resource_list.append(item)
....
if resource_list is not None:
    try:
      for resource in resource_list:
....

My bad! This issue was opened erroneously. The solution is working as expected, the test was done wrong.


text:
bump gradlew version to 7.2

Summary
bump gradlew version to 7.2
Description
Related Issue
https://bitquill.atlassian.net/browse/AD-464
Additional Reviewers
@affonsoBQ
@andiem-bq
@alexey-temnikov
@birschick-bq

close for now

repo: aws/amazon-documentdb-jdbc-driver


text:
It is not possible to expand the database tree for external schemas in Redshift Serverless when authenticating with IAM

Driver version
2.1.0.7
Redshift version
PostgreSQL 8.0.2 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.4.2 20041017 (Red Hat 3.4.2-6.fc3), Redshift 1.0.38431
Client Operating System
Windows 10
JAVA/JVM version
11.0.12
Table schema
N/A
Problem description
When authenticating to Redshift Serverless using IAM and connecting via a NLB, it is not possible to expand the database tree to view the tables and views in an external schema created from a datashare.

Expected behaviour:
a. Open DBeaver or Coginiti Pro and create the connection e.g. jdbc:redshift:iam://{nlb-host}:5439/dev?isServerless=true&Profile=my-profile
b. Establish a connection
c. Expand the database tree
d. See which tables and views are in an external schema created from a datashare

Actual behaviour:
Steps A and B work (except the limitations described in issue 54 and issue 55). Steps C and D do not work

Error message/stack trace:
none

Any other details that can be helpful:
N/A

Maneesh work with you and it's not the driver issue but privilege's issue. So closing the issue as we have internal tickets to track for server side issues.

authors: iggarish tobyayre

repo: aws/amazon-redshift-jdbc-driver

license_source: bigquery


text:
feat: Centralize leaked ENI cleanup

Issue #, if available:
N/A
Description of changes:
Merge feature branch eni-cleanup into master for release.
Centralize leaked ENI cleanup in the controller to delete leaked ENIs provisioned by the controller and VPC-CNI:

Added finalizer in CNINode: when kubernetes deletes an object that has finalizers specified for it, the Kubernetes API marks the object for deletion by populating .metadata.deletionTimestamp, and returns a 202 status code (HTTP "Accepted"). The target object remains in a terminating state while the control plane, or other components, take the actions defined by the finalizers. After these actions are complete, the controller removes the relevant finalizers from the target object. When the metadata.finalizers field is empty, Kubernetes considers the deletion complete and deletes the object.
Add a new field in Spec to add Tag: This is currently used to specify tags that need to added to network interfaces provisioned by the controller and VPC-CNI. VPC-CNI will read the CNINode to get the tags.

The CNINode reconciler will add the finalizer and tag if missing on existing CNINode objects and handles the finalizer routine to clean up leaked resources.

Re-create CNINode if deleted by user: The controller will re-create CNINode if deleted by user when node object is still present.
Added metrics to count number of available ENIs found and number of leaked ENIs that failed to be deleted

Tests:
Webhook
Ran 11 of 11 Specs in 107.573 seconds
SUCCESS! -- 11 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

Ginkgo ran 1 suite in 1m49.904100803s
Test Suite Passed

SGP
Ran 18 of 22 Specs in 1183.287 seconds
SUCCESS! -- 18 Passed | 0 Failed | 0 Pending | 4 Skipped
PASS

Ginkgo ran 1 suite in 19m46.06109991s
Test Suite Passed

CNINode
Ran 6 of 6 Specs in 491.560 seconds
SUCCESS! -- 6 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

Ginkgo ran 1 suite in 8m14.030002297s
Test Suite Passed

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Merged #374 from dev branch.

authors: haouc sushrk


text:
Getting IdToken

We would like to be able to get the user's SessionToken.IdToken so we can use it call API Gateway for RESTful API calls.
I can get it by "authenticating twice", using the same method as we do in a desktop app as shown below as a modification to the sample asp.net Core app's Login page - OnPostAsync.
Is there a better way to do this?
`            try
{
returnUrl = returnUrl ?? Url.Content("~/Index");
            if (ModelState.IsValid)
            {
                var result = await _signInManager.PasswordSignInAsync(Input.UserName, Input.Password, Input.RememberMe, lockoutOnFailure: false);
                // begin modification to sample
                var provider = new AmazonCognitoIdentityProviderClient(new AnonymousAWSCredentials(), RegionEndpoint.USEast1);

                var clientId = _configuration["AWS:UserPoolClientId"];
                var clientSecret = _configuration["AWS:UserPoolClientSecret"];
                var userPool = new CognitoUserPool(_configuration["AWS:UserPoolId"], clientId, provider, clientSecret);
                CognitoUser user = new CognitoUser(Input.UserName, clientId, userPool,provider,clientSecret);
                var response = await user.StartWithSrpAuthAsync(new InitiateSrpAuthRequest() {Password = Input.Password}).ConfigureAwait(false);
                Debug.Write(user.SessionTokens.IdToken);
                // end modification to sample

`

Here is a snippet recently wrote for a the demo I'm trying to together for re:Invent. You can ignore _authenticationStateProvider. Basically user is the ClaimsPrincipal you can find from the HttpContext object.
private async Task<HttpClient> ConstructHttpClient()
{
    var authState = await this._authenticationStateProvider.GetAuthenticationStateAsync();
    var user = authState.User;
    if(!user.Identity.IsAuthenticated)
        throw new NoIdTokenException();

    var userId = this._cognitoUserManager.GetUserId(user);
    if (string.IsNullOrEmpty(userId))
        throw new NoIdTokenException();

    var cognitoUser = await this._cognitoUserManager.FindByIdAsync(userId);
    if (string.IsNullOrEmpty(cognitoUser?.SessionTokens.IdToken))
        throw new NoIdTokenException();

    var httpClient = new HttpClient();
    httpClient.DefaultRequestHeaders.Authorization = AuthenticationHeaderValue.Parse($"bearer {cognitoUser.SessionTokens.IdToken}");

    return httpClient;
}


text:
fix: Use real private field/method

Use real private class field/methods instead of typescripts public/private stuff

Resolving as TypeScript is no longer used.

authors: jimmywarting krk


text:
Wrong duration for DB transaction event

[Steps to reproduce the issue]

Create a Ruby on rails application with the X-Ray SDK for Ruby middleware [3] to send requests to the Database.

Create a Model and a corresponding Controller.
Define routing URL pattern corresponding to the Controller.

Start the X-Ray daemon.
Send a request to the created application.
Confirm the the time duration for the trace in the X-Ray console.

As a result, the start_time of a subsegment during a DB operation was set to a time in the distant future.
[Investigation]

aws-xray-sdk-ruby subscribes to transaction events and uses the start time of the event as the start_time of the subsegment.
https://github.com/aws/aws-xray-sdk-ruby/blob/master/lib/aws-xray-sdk/facets/rails/active_record.rb

However, while the subsegment start_time is assumed to be in seconds, the event start_time is recorded in milliseconds. In other words, it is 1000 times larger than the assumed value.
https://github.com/rails/rails/blob/0c09ae546fae39333068177b28767abbc6c8726e/activesupport/lib/active_support/notifications/ instrumenter.rb#L106

Digging deeper into this and it seems In Rails version 6, the ActiveSupport::Notifications module used Concurrent.monotonic_time to record times, which effectively reported times in seconds. In Rails 7 and higher they shifted to use Process.clock_gettime(Process::CLOCK_MONOTONIC, :float_millisecond), which provides time measurements in milliseconds for more granular and precise timing of events.
https://github.com/rails/rails/blob/6-1-stable/activesupport/lib/active_support/notifications/instrumenter.rb#L129C10-L129C36
vs
https://github.com/rails/rails/blob/f12ae4448477a57c6fdac39039ac9fd60ba878bf/activesupport/lib/active_support/notifications/instrumenter.rb#L154
This issue is causing any DB related traces to show incorrect durations in the XRay console.

@wangzlei @jj22ee It looks like Rails actually backported a fix to 7.1.4 now https://github.com/rails/rails/releases/tag/v7.1.4

Fix ActiveSupport::Notifications.publish_eventto preserve units.
This solves the incorrect reporting of time spent running Active Record
asynchronous queries (by a factor 1000).
Jean Boussier

authors: duckworth wangzlei

repo: aws/aws-xray-sdk-ruby


text:
Update Rufio dependency:

This has fixes for runtime panics.
https://github.com/tinkerbell/rufio/pull/77
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve

/lgtm

authors: jacobweinstock mitalipaygude

repo: aws/eks-anywhere-build-tooling


text:
Automated cherry pick of #60: bump to latest eks-d release

Cherry pick of #60 on release-0.1.
#60: bump to latest eks-d release
For details on the cherry pick process, see the cherry pick requests page.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: jaxesn
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [jaxesn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: jaxesn, TerryHowe
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [TerryHowe,jaxesn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

authors: abhay-krishna jaxesn


text:
Bumps to latest ekd-s releases

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/cherrypick release-0.10

/lgtm
/approve


text:
Feat: (low prio) Logs are double reporting timestamps

Here's some output of kubectl:
uc807dda342eb5f:~/run/eksa$ k logs --since=30m -n eksa-packages eks-anywhere-packages-5b47488b49-k5d9z
1.6608493109306107e+09	INFO	PackageBundle.PackageBundle	Add/ProcessBundle:	{"bundle": "eksa-packages/v1-23-60"}
1.6608493531004937e+09	INFO	PackageBundle.PackageBundle	Add/ProcessBundle:	{"bundle": "eksa-packages/v1-23-60"}
uc807dda342eb5f:~/run/eksa$ k logs --timestamps=true --since=30m -n eksa-packages eks-anywhere-packages-5b47488b49-k5d9z
2022-08-18T19:01:50.930723793Z 1.6608493109306107e+09	INFO	PackageBundle.PackageBundle	Add/ProcessBundle:	{"bundle": "eksa-packages/v1-23-60"}
2022-08-18T19:02:33.100605706Z 1.6608493531004937e+09	INFO	PackageBundle.PackageBundle	Add/ProcessBundle:	{"bundle": "eksa-packages/v1-23-60"}

There are, for whatever reason, two timestamps in our logging output, There's really no need for this, it's just wasted space. Plus 1.6608493109306107e+09 is not a useful way to display a timestamp, but fwiw:
$ date -d @$(printf %f "1.6608493531004937e+09")

There's probably an easier way, but that was what jumped to mind.

Also note that kubectl logs provides a --timestamp switch so even printing the timestamp once might not be relevant. We should consider using structured logs.

authors: ewollesen lewisdiamond

repo: aws/eks-anywhere-packages


text:
Add CloudStack into eksa controller manager

Issue #, if available:
Description of changes:
This is part of PRs that merge from embargo to public repo. This PR is about to add CloudStack into eks-a controller.
Testing (if applicable):
Manually create Cloudstack cluster and upgrade kubernetes versions, and scale-up worker/controlplane nodes.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/override eks-anywhere-release-tooling-presubmit
Test will fail until we have build artifacts for cloudstack branch. Overriding for now.

/override eks-anywhere-e2e-presubmit
/override eks-anywhere-release-tooling-presubmit
/lgtm

authors: vignesh-goutham wanyufe

repo: aws/eks-anywhere


text:
Add doc for bring your own DHCP:

This is needed to provide guidance for environments where layer 2 functionality is not desired.
Issue #, if available:
Description of changes:
Testing (if applicable):
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve

/lgtm

/lgtm

/retest

/cherry-pick release-0.15

/lgtm

authors: chrisnegus jacobweinstock mimcdevitt


text:
Use controlplane endpoint IPs outside DHCP ranges for e2e

Issue #, if available:
Description of changes:
Currently, the tests generate random IPs for cp endpoint within the entire subnet range, which can conflict with the DHCP range. This can result in a situation where the DHCP can assign the same IP to other machines which can cause problems.
In this PR, I'm restricting the CIDR range for generating random IPs to avoid conflicts with DHCP.
CIDR x.x.128.0/17 corresponds to IP range x.x.128.0-x.x.255.255 which is outside the DHCP ranges for both networks
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: abhinavmpandey08
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [abhinavmpandey08]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: abhinavmpandey08, g-gaston
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [abhinavmpandey08,g-gaston]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


text:
Refactor management upgrade tests to use fake client

Issue #, if available:
Description of changes:
This PR refactoring the management upgrade tests to use fake client. This is in response to the PR comment here. This will make things a bit simpler when making more changes.
Testing (if applicable):
Documentation added/planned (if applicable):
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/lgtm

Fix some linting issues post approval, so re-added lgtm


text:
Add junit test report to docker e2e test script

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/test eks-anywhere-e2e-presubmit

/test eks-anywhere-e2e-presubmit

authors: abhay-krishna g-gaston


text:
chore: npm-check-updates && yarn upgrade

Ran npm-check-updates and yarn upgrade to keep the yarn.lock file up-to-date.

AWS CodeBuild CI Report

CodeBuild project: AutoBuildProject6AEA49D1-Blkkw9bQFn8A
Commit ID: 3775c1eabda23d812872d71f5d9687f75c0282fd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

repo: aws/jsii


text:
chrore: Add CredentialProvider to Bottlerocket settings

Fixes #5507
Description
This PR adds support for settings.kubernetes.credential-providers for Bottlerocket.
How was this change tested?
By building a dev version of Karpenter with the changes of this PR, pushing the image to a private ECR repository and switched the image URL of Karpenter with the dev image on an existing EKS cluster which was already using Karpenter.
When Karpenter was deployed, it spun up a new Bottlerocket instance with the example configuration of the original issue.
After going into the node with SSM I was able to confirm the configuration was updated as expected (which wasn't the case before).
Does this change impact docs?

[ ] Yes, PR includes docs updates
[ ] Yes, issue opened: #
[x] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Pull Request Test Coverage Report for Build 7657158492

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.02%) to 82.615%

Totals

Change from base Build 7649891648:
0.02%

Covered Lines:
5004

Relevant Lines:
6057

💛  - Coveralls

authors: coveralls johanvandeweerd

repo: aws/karpenter-provider-aws


text:
Could not schedule pod, incompatible with provisioner "default"

Version

Karpenter: v0.16.1

Kubernetes: v1.22
Expected Behavior

Karpenter is active. Create some pods using a deployment.
Actual Behavior

Deploy pod to auto provision node
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 0
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.2
          resources:
            requests:
              cpu: 1
EOF
kubectl scale deployment inflate --replicas 5
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller

Steps to Reproduce the Problem

I read through the document at https://karpenter.sh/v0.16.1/getting-started/getting-started-with-terraform/#install-karpenter-helm-chart. I got errors below:
kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller
ERROR	controller.provisioning	Could not schedule pod, incompatible with provisioner "default", incompatible requirements, key karpenter.sh/provisioner-name, karpenter.sh/provisioner-name DoesNotExist not in karpenter.sh/provisioner-name In [default]	{"commit": "b157d45", "pod": "karpenter/karpenter-5755bb5b54-rh65t"}
Below is the code:
cat main.tf
terraform {
  required_version = "~> 1.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.5"
    }
    kubectl = {
      source  = "gavinbunney/kubectl"
      version = "~> 1.14"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

locals {
  cluster_name = "karpenter-demo"

  # Used to determine correct partition (i.e. - `aws`, `aws-gov`, `aws-cn`, etc.)
  partition = data.aws_partition.current.partition
}

data "aws_partition" "current" {}

module "vpc" {
  # https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest
  source  = "terraform-aws-modules/vpc/aws"
  version = "3.14.4"

  name = local.cluster_name
  cidr = "10.0.0.0/16"

  azs             = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway     = true
  single_nat_gateway     = true
  one_nat_gateway_per_az = false

  public_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                      = 1
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"             = 1
  }
}

module "eks" {
  # https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest
  source  = "terraform-aws-modules/eks/aws"
  version = "18.29.0"

  cluster_name    = local.cluster_name
  cluster_version = "1.22"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  # Required for Karpenter role below
  enable_irsa = true

  node_security_group_additional_rules = {
    ingress_nodes_karpenter_port = {
      description                   = "Cluster API to Node group for Karpenter webhook"
      protocol                      = "tcp"
      from_port                     = 8443
      to_port                       = 8443
      type                          = "ingress"
      source_cluster_security_group = true
    }
  }

  node_security_group_tags = {
    # NOTE - if creating multiple security groups with this module, only tag the
    # security group that Karpenter should utilize with the following tag
    # (i.e. - at most, only one security group should have this tag in your account)
    "karpenter.sh/discovery/${local.cluster_name}" = local.cluster_name
  }

  # Only need one node to get Karpenter up and running.
  # This ensures core services such as VPC CNI, CoreDNS, etc. are up and running
  # so that Karpenter can be deployed and start managing compute capacity as required
  eks_managed_node_groups = {
    initial = {
      instance_types = ["m5.large"]
      # Not required nor used - avoid tagging two security groups with same tag as well
      create_security_group = false

      min_size     = 1
      max_size     = 1
      desired_size = 1

      iam_role_additional_policies = [
        "arn:${local.partition}:iam::aws:policy/AmazonSSMManagedInstanceCore", # Required by Karpenter
        "arn:${local.partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy",
        "arn:${local.partition}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly", #for access to ECR images
        "arn:${local.partition}:iam::aws:policy/CloudWatchAgentServerPolicy"
      ]

      tags = {
        # This will tag the launch template created for use by Karpenter
        "karpenter.sh/discovery/${local.cluster_name}" = local.cluster_name
      }
    }
  }
}

#The EKS module creates an IAM role for the EKS managed node group nodes. We’ll use that for Karpenter.
#We need to create an instance profile we can reference.
#Karpenter can use this instance profile to launch new EC2 instances and those instances will be able to connect to your cluster.
resource "aws_iam_instance_profile" "karpenter" {
  name = "KarpenterNodeInstanceProfile-${local.cluster_name}"
  role = module.eks.eks_managed_node_groups["initial"].iam_role_name
}

#Create the KarpenterController IAM Role
#Karpenter requires permissions like launching instances, which means it needs an IAM role that grants it access. The config
#below will create an AWS IAM Role, attach a policy, and authorize the Service Account to assume the role using IRSA. We will
#create the ServiceAccount and connect it to this role during the Helm chart install.

module "karpenter_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "5.3.3"

  role_name                          = "karpenter-controller-${local.cluster_name}"
  attach_karpenter_controller_policy = true

  karpenter_tag_key               = "karpenter.sh/discovery/${local.cluster_name}"
  karpenter_controller_cluster_id = module.eks.cluster_id
  karpenter_controller_node_iam_role_arns = [
    module.eks.eks_managed_node_groups["initial"].iam_role_arn
  ]

  oidc_providers = {
    ex = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["karpenter:karpenter"]
    }
  }
}

#Install Karpenter Helm Chart
#Use helm to deploy Karpenter to the cluster. We are going to use the helm_release Terraform resource to do the deploy and pass in the
#cluster details and IAM role Karpenter needs to assume.

provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)

    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", local.cluster_name]
    }
  }
}

resource "helm_release" "karpenter" {
  namespace        = "karpenter"
  create_namespace = true

  name       = "karpenter"
  repository = "https://charts.karpenter.sh"
  chart      = "karpenter"
  version    = "v0.16.1"

  set {
    name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = module.karpenter_irsa.iam_role_arn
  }

  set {
    name  = "clusterName"
    value = module.eks.cluster_id
  }

  set {
    name  = "clusterEndpoint"
    value = module.eks.cluster_endpoint
  }

  set {
    name  = "aws.defaultInstanceProfile"
    value = aws_iam_instance_profile.karpenter.name
  }
}

#Provisioner
#Create a default provisioner using the command below. This provisioner configures instances to connect to your cluster’s endpoint and
#discovers resources like subnets and security groups using the cluster’s name.
#This provisioner will create capacity as long as the sum of all created capacity is less than the specified limit.
provider "kubectl" {
  apply_retry_count      = 5
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  load_config_file       = false

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", module.eks.cluster_id]
  }
}

resource "kubectl_manifest" "karpenter_provisioner" {
  yaml_body = <<-YAML
  apiVersion: karpenter.sh/v1alpha5
  kind: Provisioner
  metadata:
    name: default
  spec:
    requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["spot"]
    limits:
      resources:
        cpu: 1000
    provider:
      subnetSelector:
        Name: "*private*"
      securityGroupSelector:
        karpenter.sh/discovery/${module.eks.cluster_id}: ${module.eks.cluster_id}
      tags:
        karpenter.sh/discovery/${module.eks.cluster_id}: ${module.eks.cluster_id}
    ttlSecondsAfterEmpty: 30
  YAML

  depends_on = [
    helm_release.karpenter
  ]
}

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 0
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.2
          resources:
            requests:
              cpu: 1
EOF

kubectl scale deployment inflate --replicas 5

Resource Specs and Logs

kubectl logs -f -n karpenter -l app.kubernetes.io/name=karpenter -c controller
DEBUG	controller.provisioning	Relaxing soft constraints for pod since it previously failed to schedule, removing: spec.topologySpreadConstraints = {"maxSkew":1,"topologyKey":"topology.kubernetes.io/zone","whenUnsatisfiable":"ScheduleAnyway","labelSelector":{"matchLabels":{"app.kubernetes.io/instance":"karpenter","app.kubernetes.io/name":"karpenter"}}}	{"commit": "b157d45", "pod": "karpenter/karpenter-5755bb5b54-rh65t"}
2022-09-10T00:13:13.122Z
ERROR	controller.provisioning	Could not schedule pod, incompatible with provisioner "default", incompatible requirements, key karpenter.sh/provisioner-name, karpenter.sh/provisioner-name DoesNotExist not in karpenter.sh/provisioner-name In [default]	{"commit": "b157d45", "pod": "karpenter/karpenter-5755bb5b54-rh65t"}

This bug looks related to #2311 where we expanded the default configuration of the controller deployment to bring up two replicas. Since each of the pods is requesting the below resources
resources:
  requests:
    cpu: 1
    memory: 1Gi

it can't schedule on a single m5.large. I'll follow-up with a docs update to remedy this but the short of it is that the desired_size/min_size/max_size of the MNG needs to be set to 2 now.

Partial docs update was performed in #2387

repo: aws/karpenter


text:
Add Running Pods task to docs

1. Issue, if available:
2. Description of changes:
Added a new "Tasks" nav to the docs, along with a new Tasks section called "Running Pods" that describes how application developers can set pod constraints that are consumed by Karpenter
3. Does this change impact docs?

[X] Yes, PR includes docs updates
[ ] Yes, issue opened: link to issue
[ ] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

changing my mind to remove approval and "request changes" because of the outstanding question included under the storage heading.

The decision from engineering is to remove the storage section for now.


text:
Value should be of "String" type for iot-thing-name

Logging
gst-launch-1.0 v4l2src do-timestamp=TRUE device=/dev/video0 ! videoconvert ! video/x-raw,format=I420,width=640,height=480,framerate=30/1 ! omxh264enc control-rate=1 target-bitrate=1024000 periodicity-idr=30 inline-header=FALSE ! h264parse ! video/x-h264,stream-format=avc,alignment=au,width=640,height=480,framerate=30/1,profile=baseline ! kvssink stream-name="2230000001" iot-certificate="iot-certificate,endpoint=secret-endpointiot.us-east-2.amazonaws.com,cert-path=/home/pi/device.pem,key-path=/home/pi/device.key,ca-path=/home/pi/cacert.pem,role-aliases=KvsCameraIoTRoleAlias, iot-thing-name=2230000001" aws-region="us-east-2"
Describe the bug
A clear and concise description of what the bug is.
ERROR - Value should be of "String" type for iot-thing-name
Also found G_VALUE_HOLDS_STRING which is used against iot-thing-name, so it looks like the check is inside your sdk.
I do not think you should do that check here, if that is possible would be nice to open PR with the change. I think I could do that if all agrees it is redundant. C lib is not verifying that property.
To Reproduce
Steps to reproduce the behavior:
Try to run above command with iot-thing-name as a number

@disa6302  if you try to create a device 111111 in IoT core that will be created without a problem, you should either case that INT into the string or let the numbers there.

Have you tried adding it in double quotes? From the line you pasted, it does not seem like your iot thing name is in double quotes

We tried singular, double nothing works because of that check which you have  G_VALUE_HOLDS_STRING this functions checks very well

Ok. I see what you mean. Let me run some checks and verify and create a PR. :) Thank you.

@Coldplayer1995 ,
I missed suggesting this, but can you try type casting the thing name with (string) in the command line? That should fix your issue. Like this:
gst-launch-1.0 v4l2src do-timestamp=TRUE device=/dev/video0 ! videoconvert ! video/x-raw,format=I420,width=640,height=480,framerate=30/1 ! omxh264enc control-rate=1 target-bitrate=1024000 periodicity-idr=30 inline-header=FALSE ! h264parse ! video/x-h264,stream-format=avc,alignment=au,width=640,height=480,framerate=30/1,profile=baseline ! kvssink stream-name="2230000001" iot-certificate="iot-certificate,endpoint=secret-endpointiot.us-east-2.amazonaws.com,cert-path=/home/pi/device.pem,key-path=/home/pi/device.key,ca-path=/home/pi/cacert.pem,role-aliases=KvsCameraIoTRoleAlias, iot-thing-name=(string) 2230000001" aws-region="us-east-2"

@Coldplayer1995 ,
I tested the pipeline by typecasting the value with (string) and that works. I am closing this assuming answered. Please feel free to reach out if this does not help you.


text:
Tabular: Fixed rare FI crash

Issue #, if available:
Description of changes:

Fixed rare FI crash caused when a single feature is requested to get FI and that feature was ignored by the model in training.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Job PR-1229-1 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-1229/1/index.html

authors: Innixma szha

repo: awslabs/autogluon


text:
Non-uppercase environment variables

It's surprising to see non-uppercase environment variables. What I observed in thousands of other scripts from different sources is uppercase with underscores.
https://github.com/awslabs/aws-centralized-logging/blob/0cb0dcd1ecce6e3e0851b271d792dace2b1719e9/source/services/indexing/index.js#L28

This has been addressed, closing issue.

There are search results:
https://github.com/awslabs/aws-centralized-logging/search?q=DomainEndpoint&unscoped_q=DomainEndpoint
Did not dive into it but it seems that at least https://github.com/awslabs/aws-centralized-logging/blob/2a396e4633e07fccafb1937ed7564dca5c2b1120/source/services/indexing/package.json#L31 is relevant

repo: awslabs/aws-centralized-logging


text:
Customer Request : Elastic Search domain if public should have access Policy attached with Source IP condition.

Check if the ElasticSearch Domain is having Public Access. If yes then an Resource based policy should be attached with the Source IP condition in it. In that Source IP the IP should belong to its Company IP address.

Noted, you can start with https://docs.aws.amazon.com/config/latest/developerguide/elasticsearch-in-vpc-only.html even if not exactly the use case.

repo: awslabs/aws-config-rules


text:
RFC: Update, view and remove the correlation id

Key information

RFC PR: 516
Related issue(s), if known:
Area: Logger
Meet tenets: Yes
Approved by: @heitorlessa
Reviewed by: @pankajagrawal16 @saragerion

Summary
When using the correlation feature in conjunction with the SQS you may want to update the correlation id for each message in the incoming batch. This allows you to correlate log lines that are being processed in a batch with the lines in an other log group using the CloudWatch Logs Insights.
Further more next to lines that are related to the message there are also lines that are related to the processing of the batch itself. So an easy way to remove the correlation id would be beneficial to have.
Optionally, it might be useful to fetch the current correlation id for displaying purposes.
Motivation
The following code "works" but it breaks the typing as logger.set_correlation_id expects a str and not a Optional[str]:
from aws_lambda_powertools import Logger
from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools.utilities.batch import sqs_batch_processor
from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord
logger = Logger(service="callback")

def record_handler(record: dict) -> None:
    record = SQSRecord(record)
    # Get the correlation id from the message attributes
    correlation_id = record.message_attributes["correlation_id"].string_value
    logger.set_correlation_id(correlation_id)
    logger.info(f"Processing message with {correlation_id} as correlation_id")

@sqs_batch_processor(record_handler=record_handler)
def lambda_handler(event: dict, context: LambdaContext) -> None:
    logger.set_correlation_id(None)
    logger.info(f"Received a SQSEvent with {len(list(event.records))} records")

If you want to display the current correlation id you need to track it in your own logic, for example:
from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools.utilities.data_classes import (
    event_source,
    APIGatewayProxyEvent,
)
logger = Logger(service="api")

@event_source(data_class=APIGatewayProxyEvent)
@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
def lambda_handler(event: APIGatewayProxyEvent, context: LambdaContext) -> dict:
    logger.info(
        f"Received request using {event.request_context.request_id} as correlation id"
    )

Here you see that there are 2 things that you need to maintain the correlation_id_path=correlation_paths.API_GATEWAY_REST and the event.request_context.request_id both need to be updated if you want to change it.
Proposal
Adding Optional[str] typing or a logger.remove_correlation_id() method would solve the removal of the correlation id.
A logger.get_correlation_id() method would help but from the discussions on slack that might not be so straight forward. @heitorlessa could you add some context?
If this feature should be available in other runtimes (e.g. Java, Typescript), how would this look like to ensure consistency? The interface should be the same but tailored to the language standards.
User Experience
Get the current correlation id:
@event_source(data_class=APIGatewayProxyEvent)
@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
def lambda_handler(event: APIGatewayProxyEvent, _: LambdaContext) -> dict:
    logger.info(
        f"Received request using {logger.get_correlation_id()} as correlation id"
    )

Remove the correlation id:
logger.set_correlation_id(None)
# or
logger.remove_correlation_id()

Drawbacks
Not that we know of until now
Rationale and alternatives

What other designs have been considered? Why not them? TBD
What is the impact of not doing this? It breaks the typing interface, python will not break on it but it's not nice

Unresolved questions

Optional, stash area for topics that need further development e.g. TBD

Released in 1.18.0

repo: awslabs/aws-lambda-powertools-roadmap


text:
Formation CREATE_FAILED due to invalid policy JSON in template

Attempt to provision aws-perspective into additional region failed.
invalid.template.zip located here: https://aws-perspective-amplifystoragebucket-14lu9llypry0f.s3.amazonaws.com/public/cfn/import-templates/regions/e1f925c485684f555c87a2fd5b792d05ecbd2bd8.template?X-Amz-Algorithm
valid.template.zip
Editing template resolves the issue. See attachments for reference.

What version are you using? This issue should have been fix in the new release.

How do I see it and how do I upgrade?

Unfortunately the only upgrade strategy currently to uninstall and reinstall. However, we are working on a script that will allow in place upgrades. If you'd like I can put a note in this issue for you when it's done, it shouldn't be too long now.

#86

Fixed in v1.0.1

authors: ConnorKirk mohsanjaffery navado svozza

repo: awslabs/aws-perspective


text:
Log4j2 Incompatibility with SAM

Hi Guys,
I'm facing very strange issue that when I'm adding log4j2 configuration my functions stops to work correctly.
I have a project that contains dependencies to https://github.com/aws/aws-lambda-java-libs/tree/master/aws-lambda-java-log4j2 and when everything is done according to instructions My functions stops to work correctly and returns:

Function returned an invalid response (must include one of: body, headers or statusCode in the response object): invalid character '-' after top-level

When I copy log4j2.xml to src/main/resources in maven function works fine but as expected log4j2 configuration can't be found

START RequestId: aa632e55-8cd9-4b7e-9d3f-196cdab74f9e Version: $LATEST
Save started...ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging.

.....
End everything is green.
After playing with this issue it looks like that even simple sysout with text breaks overal execution and Error is returned:
Function returned an invalid response (must include one of: body, headers or statusCode in the response object): invalid character '-' after top-level
It looks like if I put simple sysout in any place of code that will be executed AWS Sam will crash.
e.g.:
Such function works fine:

public class Function implements RequestStreamHandler {
@Override
public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context) throws IOException
//My Code
}}

And such code crashes:

public class Function implements RequestStreamHandler {
@Override
public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context) throws IOException
System.out.println("Hi");
//My Code
}}

This should work in latest version of SAM CLI v0.3.0. Can you check and re-open if it doesn't?

authors: sanathkr volodymyr-f

repo: awslabs/aws-sam-cli


text:
NullPointerException if calling lambda without http headers

Hi Stefano, As per our conversation over the chime, here is the Issue for the NPE

Framework version: 1.3
Implementations: Spring Boot

Scenario
I was recently trying out to develop a microservice using Java based lambda using spring boot. After following all the instruction, my requests continuously failed giving NullPointerException while testing the lambda endpoint using Test console of API Gateway. The API Gateway is using Lambda Proxy integration to route the endpoint params to lambda.
Expected behavior
It shouldnt give NPE
Actual behavior
Giving below exception
java.lang.NullPointerException
at com.amazonaws.serverless.proxy.internal.servlet.AwsProxyHttpServletRequestReader.readRequest(AwsProxyHttpServletRequestReader.java:39)
at com.amazonaws.serverless.proxy.internal.servlet.AwsProxyHttpServletRequestReader.readRequest(AwsProxyHttpServletRequestReader.java:28)
at com.amazonaws.serverless.proxy.internal.LambdaContainerHandler.proxy(LambdaContainerHandler.java:174)
at com.amazonaws.serverless.proxy.internal.LambdaContainerHandler.proxyStream(LambdaContainerHandler.java:209)
at com.amazon.sa.SpringLambdaHandler.handleRequest(SpringLambdaHandler.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at lambdainternal.EventHandlerLoader$StreamMethodRequestHandler.handleRequest(EventHandlerLoader.java:350)
at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:888)
at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:293)
at lambdainternal.AWSLambda.<clinit>(AWSLambda.java:64)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:104)
END RequestId: c6ca76ec-db10-454a-8ffe-449b84c3ac74

Steps to reproduce

create a lambda spring boot handler as per the given instructions in this guide.
Expose this function to API Gateway using Lambda Proxy Integration
Test the API from API Gateway console
Keep the Headers empty as default
Get above NPE

Full log output
After reviewing your code of AwsProxyHttpServletRequestReader.java, I think as a best practice, we should write NPE free code as line below in highlighted in proned to NPE in case the Content-type header is not sent in the request. Although very rare to be null but still to have check on NPE in the code and avoid failing the requests.
if (request.getMultiValueHeaders().getFirst(HttpHeaders.CONTENT_TYPE) != null) {
            String contentType = request.getMultiValueHeaders().getFirst(HttpHeaders.CONTENT_TYPE);
            // put single as we always expect to have one and only one content type in a request.
            request.getMultiValueHeaders().putSingle(HttpHeaders.CONTENT_TYPE, getContentTypeWithCharset(contentType, config));
        }

Any updates on this?
Same problem occurs if one adds to CloudFormation template event to run function periodically:
        CheckWebsiteScheduledEvent:
          Type: Schedule
          Properties:
            Schedule: rate(5 minutes)

@sapessi  I can fix it if you like and you can review the code. Would that be fine?

The issues are not related. Working to replicate and push a fix for this now. It will go out as a patch release.

Closing since version 1.3.1 is out.

@sapessi  Thanks very much!!!

authors: dhawalkp huksley sapessi

repo: awslabs/aws-serverless-java-container


text:
change regex search string for linux distros

To detect the Linux distribution type, setup.py checks /etc/*-release
files which matches two files on Amazon Linux (new and old versions):
- /etc/os-release
- /etc/system-release

On Amazon Linux 2, "system-release" contains the single line
- Amazon Linux 2
While on Amazon Linux < v2 "system-release" returns
- Amazon Linux AMI release 2018.03
On Amazon Linux 2, "os-release" looks like the following:
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
``
On Amazon Linux < v2, "os-release" looks like:

NAME="Amazon Linux AMI"
VERSION="2018.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2018.03"
PRETTY_NAME="Amazon Linux AMI 2018.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"

In `setup.py` the regex current matches the string
    - "Amazon Linux AMI"
But it this is shortened to just
    - "Amazon Linux"

both old and new versions of Amazon Linux will be matched by
the regex allowing the collectd-cloudwatch plugin to be
installed.

Oops! I didn't see that someone else has opened a pull request for the same change! Please disregard this request...

repo: awslabs/collectd-cloudwatch

license: MIT


text:
Problem loading Pytorch model

Question
Hi everyone. I am trying to use this library in order to serve deep neural network models for a web service. I'm trying to load a custom pytorch model for landmark detection. I have follow the method to convert the model (https://github.com/awslabs/djl/blob/master/docs/pytorch/how_to_convert_your_model_to_torchscript.md), then I've used the example code to load a model from a local file (https://github.com/awslabs/djl/blob/master/docs/load_model.md). I have encountered the error  "No deep learning engine found.". However I have checked my dependencies on the pom.xml and I have not found the problem (djl.version is also defined in the pom.xml file):

    <dependency>
        <groupId>ai.djl.pytorch</groupId>
        <artifactId>pytorch-engine</artifactId>
        <version>${djl.version}</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/ai.djl.pytorch/pytorch-native-auto -->
    <dependency>
        <groupId>ai.djl.pytorch</groupId>
        <artifactId>pytorch-native-auto</artifactId>
        <version>1.5.0</version>
        <scope>runtime</scope>
    </dependency>

Moreover I have debugged the code and the problem is that the code fails in modelZoo.java in loadModel method on line:
Set supportedEngine = zoo.getSupportedEngines();
var zoo is of type "DefaultModelZoo" and modelLoaders array is empty. Anyone can help me with this issue?
Thank you in advance.

Thanks for reporting the issue.
Does it work if add "-Dai.djl.default_engine=PyTorch"

In here you can add 0.6.0 to replace the djl.version.

@eusebioaguilera
Would you please share the full stacktrace?
And can you enable logging and change log level to debug?
Since you are using maven, can should also print out the project dependency tree:
mvn dependency:tree

Thanks for reporting the issue.
Does it work if adding "-Dai.djl.default_engine=PyTorch"

No, the same error appears using this property.

In here you can add 0.6.0 to replace the djl.version.
You were having the dependencies missing in your environment, to proceed further, could you please provide the following information:
1. Your `pom.xml`

2. Your OS

3. Your Java version

Hi, I have used the version 0.6.0 instead of the djl.version defined, and there is no difference.

The main content of the pom.xml file is:

${project.build.directory}/endorsed
0.6.0
PyTorch

javax.servlet
javax.servlet-api

 <!-- Deep Java Library -->
 <!-- https://mvnrepository.com/artifact/ai.djl/api -->
 <dependency>
     <groupId>ai.djl</groupId>
     <artifactId>api</artifactId>
     <version>${djl.version}</version>
     <scope>compile</scope>
 </dependency>

 <!-- https://mvnrepository.com/artifact/ai.djl/model-zoo -->
 <dependency>
     <groupId>ai.djl</groupId>
     <artifactId>model-zoo</artifactId>
     <version>${djl.version}</version>
 </dependency>

 <!-- https://mvnrepository.com/artifact/ai.djl.pytorch/pytorch-model-zoo -->
 <dependency>
     <groupId>ai.djl.pytorch</groupId>
     <artifactId>pytorch-model-zoo</artifactId>
     <version>${djl.version}</version>
 </dependency>

 <!-- https://mvnrepository.com/artifact/ai.djl.pytorch/pytorch-engine -->
 <dependency>
     <groupId>ai.djl.pytorch</groupId>
     <artifactId>pytorch-engine</artifactId>
     <version>${djl.version}</version>
 </dependency>

 <!-- https://mvnrepository.com/artifact/ai.djl.pytorch/pytorch-native-auto -->
 <dependency>
     <groupId>ai.djl.pytorch</groupId>
     <artifactId>pytorch-native-auto</artifactId>
     <version>1.5.0</version>
     <scope>runtime</scope>
 </dependency>

${project.artifactId}

I am using Windows 10 as the operating system.

I am using Java 11 version.

@eusebioaguilera
Would you please share the full stacktrace?
And can you enable logging and change log level to debug?
Since you are using maven, can should also print out the project dependency tree:
mvn dependency:tree

The full stacktrace is:
jul. 09, 2020 9:26:06 A. M. com.example.machinelearning.service.utils.BaseImageDJLProcessor process
INFO: D:/example/java/JavaProjects/WebProjects/MachineLearningService/target/test-classes/com/example/machinelearning/service/processors/facialLandmarks/model
jul. 09, 2020 9:26:06 A. M. com.example.machinelearning.service.utils.BaseImageDJLProcessor process
INFO: Looking for model D:\example\java\JavaProjects\WebProjects\MachineLearningService\target\test-classes\com\example\machinelearning\service\processors\facialLandmarks\model\cpm_vgg16-epoch-049-050.pt
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 27.417 s <<< FAILURE! - in com.example.machinelearning.service.processors.ImageProcessorTest
facialLocateLandmarksTest(com.example.machinelearning.service.processors.ImageProcessorTest)  Time elapsed: 27.34 s  <<< ERROR!
ai.djl.engine.EngineException:
No deep learning engine found.
Please refer to https://github.com/awslabs/djl/blob/master/docs/development/troubleshooting.md for more details.
at com.example.machinelearning.service.processors.ImageProcessorTest.lambda$facialLocateLandmarksTest$4(ImageProcessorTest.java:145)
at com.example.machinelearning.service.processors.ImageProcessorTest.runTestProcessorDJL(ImageProcessorTest.java:69)
at com.example.machinelearning.service.processors.ImageProcessorTest.testProcessorDJL(ImageProcessorTest.java:82)
at com.example.machinelearning.service.processors.ImageProcessorTest.testProcessorDJL(ImageProcessorTest.java:75)
at com.example.machinelearning.service.processors.ImageProcessorTest.facialLocateLandmarksTest(ImageProcessorTest.java:143)
The dependency tree is:
com.example:MachineLearningService:war:1.0.0-SNAPSHOT
+- javax.servlet:javax.servlet-api:jar:3.0.1:provided
+- ai.djl:api:jar:0.6.0:compile
|  +- net.java.dev.jna:jna:jar:5.3.0:compile
|  +- org.apache.commons:commons-compress:jar:1.20:compile
|  - org.slf4j:slf4j-api:jar:1.7.21:compile
+- ai.djl:model-zoo:jar:0.6.0:compile
+- ai.djl.pytorch:pytorch-model-zoo:jar:0.6.0:compile
+- ai.djl.pytorch:pytorch-engine:jar:0.6.0:compile
+- ai.djl.pytorch:pytorch-native-auto:jar:1.5.0:runtime
- junit:junit:jar:4.12:test
- org.hamcrest:hamcrest-core:jar:1.3:test

Will take a look and back to you tomorrow

@eusebioaguilera
Would you please enable debug log of your project, see: https://github.com/awslabs/djl/blob/master/docs/development/configure_logging.md#djl-logging-configuration
With debug log, we can see more engine initialization logs.

@eusebioaguilera
Would you please enable debug log of your project, see: https://github.com/awslabs/djl/blob/master/docs/development/configure_logging.md#djl-logging-configuration
With debug log, we can see more engine initialization logs.
And can you checkout our DJL master branch (use latest code) and run following command:
gradlew :examples:run -Dai.djl.default_engine=PyTorch -Dai.djl.logging.level=debug

Hi, I have enabled debug log and the problem seems to be that the native pytorch library is not loaded.
[main] DEBUG ai.djl.repository.zoo.ModelZoo - Searching model in zoo provider: ai.djl.repository.zoo.DefaultZooProvider
[main] DEBUG ai.djl.repository.zoo.DefaultModelZoo - Scanning models in repo: class ai.djl.repository.RemoteRepository, D:/example/java/JavaProjects/WebProjects/MachineLearningService/target/test-classes/com/example/machinelearning/service/processors/model
[main] DEBUG ai.djl.util.cuda.CudaUtils - Found cudart: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudart64_100.dll
[main] DEBUG ai.djl.pytorch.jni.LibUtils - Using cache dir: C:\Users\NemoDev.pytorch\cache
[main] WARN ai.djl.pytorch.engine.PtEngine - Failed to load PyTorch native library
java.lang.NullPointerException: inStream parameter is null
at java.base/java.util.Objects.requireNonNull(Objects.java:246)
at java.base/java.util.Properties.load(Properties.java:403)
at ai.djl.pytorch.jni.LibUtils.copyJniLibraryFromClasspath(LibUtils.java:167)
at ai.djl.pytorch.jni.LibUtils.loadLibrary(LibUtils.java:71)
at ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:42)
at ai.djl.pytorch.engine.PtEngineProvider.(PtEngineProvider.java:21)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at java.base/java.util.ServiceLoader$ProviderImpl.newInstance(ServiceLoader.java:779)
at java.base/java.util.ServiceLoader$ProviderImpl.get(ServiceLoader.java:721)
at java.base/java.util.ServiceLoader$3.next(ServiceLoader.java:1394)
at ai.djl.engine.Engine.initEngine(Engine.java:46)
at ai.djl.engine.Engine.(Engine.java:41)
at ai.djl.repository.zoo.DefaultModelZoo.getSupportedEngines(DefaultModelZoo.java:71)
at ai.djl.repository.zoo.ModelZoo.loadModel(ModelZoo.java:131)
at com.example.machinelearning.service.utils.BaseImageDJLProcessor.process(BaseImageDJLProcessor.java:121)
at com.example.machinelearning.service.processors.FacialLandmarkPositioningProcessor.calcMaxProbability(FacialLandmarkPositioningProcessor.java:51)
at com.example.machinelearning.service.processors.ImageProcessorTest.lambda$facialLocateLandmarksTest$4(ImageProcessorTest.java:145)
at com.example.machinelearning.service.processors.ImageProcessorTest.runTestProcessorDJL(ImageProcessorTest.java:69)
at com.example.machinelearning.service.processors.ImageProcessorTest.testProcessorDJL(ImageProcessorTest.java:82)
at com.example.machinelearning.service.processors.ImageProcessorTest.testProcessorDJL(ImageProcessorTest.java:75)
at com.example.machinelearning.service.processors.ImageProcessorTest.facialLocateLandmarksTest(ImageProcessorTest.java:143)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
[main] WARN ai.djl.engine.Engine - Failed to load engine from: ai.djl.pytorch.engine.PtEngineProvider
[main] DEBUG ai.djl.engine.Engine - No engine found from EngineProvider
However the native pytorch component is supplied by the dependency defined in the pom.xml file (artifactId pytorch-native-auto), right? Have I to supply the path of the pytorch library or another extra configuration to solve this issue?
Thank you in advance.

Currently pytorch support CUDA 9.2 10.1 and 10.2 only. 10 will not work due to the missing matched binary. You can try to install a different CUDA version to solve this issue

@lanking520 @eusebioaguilera This seems a bug. We will look into it.
If you use pytorch-native-auto, it should fallback on CPU if cuda version is not supported.

Currently pytorch support CUDA 9.2 10.1 and 10.2 only. 10 will not work due to the missing matched binary. You can try to install a different CUDA version to solve this issue

Yes, if I install cuda 10.2, then the library starts downloading a set of dll and fails for another reason. The question is, Do I need a suitable version of cuda installed on my system to use the library? In my case I want to use the model for inference using the pytorch-native-cpu artifact.

Currently it seemed to be a bug that didn’t fall back to use CPU. As an alternative, try to replace auto by using this:
http://docs.djl.ai/pytorch/pytorch-engine/index.html#windows-cpu

@eusebioaguilera
You don't have to install matching CUDA. DJL should automatically switch to cpu if no matching cuda version is installed.
However, it seems something is wrong in your system. The downloaded .jar file seems corrupted. Would you please clear your gradle cache folder and rerun the example.
What's the error message when you using CUDA 10.2?

Currently it seemed to be a bug that didn’t fall back to use CPU. As an alternative, try to replace auto by using this:
http://docs.djl.ai/pytorch/pytorch-engine/index.html#windows-cpu

I have uninstalled cuda 10.2 and try to use the artifact for the cpu version (pytorch-native-cpu) and I get the same error "No deep learning engine found". This error disappear when cuda 10.2 is installed.

@eusebioaguilera
You don't have to install matching CUDA. DJL should automatically switch to cpu if no matching cuda version is installed.
However, it seems something is wrong in your system. The downloaded .jar file seems corrupted. Would you please clear your gradle cache folder and rerun the example.
What's the error message when you using CUDA 10.2?

AsIa

@eusebioaguilera
You don't have to install matching CUDA. DJL should automatically switch to cpu if no matching cuda version is installed.
However, it seems something is wrong in your system. The downloaded .jar file seems corrupted. Would you please clear your gradle cache folder and rerun the example.
What's the error message when you using CUDA 10.2?

I have clear my maven cache dir and try to use the cpu version of the djl for pytorch (pytorch-native-cpu). However the same error came out. This problem disappear if cuda 10.2 is reinstalled.
The next is not about the djl model loading but the model input/output. When I can load the pytorch engine and try use the model by providing the input image I get the error "No matching model with specified Input/Output type found" and I suppose that the problem is that the model provided does not fit with criteria that I defined to load it.

@eusebioaguilera
ModelNotFoundException usually is the criteria doesn't match the model.
You can use following command to list models:
cd examples
gradlew listmodel -Dai.djl.default_engine=PyTorch -Dai.djl.repository.zoo.location=file:///c:/mymodels

file:///

Hi, I have executed the command and seems that the library does not recognize it because the output is:
[INFO ] - CV.IMAGE_CLASSIFICATION ai.djl.zoo:resnet:0.0.1 {"layers":"50","flavor":"v1","dataset":"cifar10"}
[INFO ] - CV.IMAGE_CLASSIFICATION ai.djl.zoo:mlp:0.0.2 {"dataset":"mnist"}
[INFO ] - CV.IMAGE_CLASSIFICATION ai.djl.pytorch:resnet:0.0.1 {"layers":"50","dataset":"imagenet"}
[INFO ] - CV.IMAGE_CLASSIFICATION ai.djl.pytorch:resnet:0.0.1 {"layers":"18","dataset":"imagenet"}
[INFO ] - CV.OBJECT_DETECTION ai.djl.zoo:ssd:0.0.1 {"flavor":"tiny","dataset":"pikachu"}
[INFO ] - CV.OBJECT_DETECTION ai.djl.pytorch:ssd:0.0.1 {"size":"300","backbone":"resnet50","dataset":"coco"}
[INFO ] - NLP.QUESTION_ANSWER ai.djl.pytorch:bertqa:0.0.1 {"backbone":"bert","dataset":"SQuAD"}
[INFO ] - NLP.QUESTION_ANSWER ai.djl.pytorch:bertqa:0.0.1 {"backbone":"bert","dataset":"SQuAD","cased":"true"}
[INFO ] - NLP.QUESTION_ANSWER ai.djl.pytorch:bertqa:0.0.1 {"backbone":"distilbert","dataset":"SQuAD","cased":"true"}
[INFO ] - UNDEFINED ai.djl.localmodelzoo:model:N/A {}
The pytorch model is in the TorchScript format.
I don't know what the problem is. Any idea?

@eusebioaguilera
listmodels shows that you point your local repo to a folder named model, you should be able to load this model using following code:
Criteria<Image, Classifications> criteria = Criteria.builder()
        .setTypes(NDList.class, NDList.class) // defines input and output data type
        .optModelUrls("file:///var/models/model") // search models in specified path
        .build();

ZooModel<NDList, NDList> model = ModelZoo.loadModel(criteria);

Or you can use built-in ImageTranslator:
Criteria<Image, Classifications> criteria = Criteria.builder()
        .setTypes(Image.class, Classifications.class) // defines input and output data type
        .optTranslator(ImageClassificationTranslator.builder().setSynsetArtifactName("synset.txt").build())
        .optModelUrls("file:///var/model")
        .build();

@eusebioaguilera
listmodels shows that you point your local repo to a folder named model, you should be able to load this model using following code:
Criteria<Image, Classifications> criteria = Criteria.builder()
        .setTypes(NDList.class, NDList.class) // defines input and output data type
        .optModelUrls("file:///var/models/model") // search models in specified path
        .build();

ZooModel<NDList, NDList> model = ModelZoo.loadModel(criteria);

Or you can use built-in ImageTranslator:
Criteria<Image, Classifications> criteria = Criteria.builder()
        .setTypes(Image.class, Classifications.class) // defines input and output data type
        .optTranslator(ImageClassificationTranslator.builder().setSynsetArtifactName("synset.txt").build())
        .optModelUrls("file:///var/model")
        .build();

Hi, finally I have managed to load the model. I have changed the 'optArtifactId("ai.djl.localmodelzoo:file_model_name")' for the 'optModelName("file_model_name")' in the criteria and then the model was loaded without any problem. Moreover, I have written a translator based on the SimplePoseTranslator and works like a charm. Thank you all of you for the advices and the help provided. Do you need any extra test for the cuda/cpu issue?

@eusebioaguilera I don't have Windows 10 with me. I only have Windows 2019 server, I'm waiting for my Windows 10 to try it out. I will ping you once I get the environment setup.

@eusebioaguilera I don't have Windows 10 with me. I only have Windows 2019 server, I'm waiting for my Windows 10 to try it out. I will ping you once I get the environment setup.

Ok.

@eusebioaguilera I'm not able to reproduce your issue.
I installed a fresh Windows 10 (19041.264), and only installed following package:

chocolatey
git (choco install git)
jdk8 (choco install jdk8)

@eusebioaguilera finally I reproduced this issue. I will work on a fix.

Ok. If you need any help, please contact me.

@eusebioaguilera feel free to reopen this issue if the fix doesn't work for you.

@eusebioaguilera feel free to reopen this issue if the fix doesn't work for you.

Ok. I will try the bugfix. Thank you.

I works great after installing the Visual Studio 2019 and its redistributables.
Awesome work! Thanks!

@carlosuc3m  0.7.0 is out. You can try it out now.

It seems that the same error keeps happening in version 0.7.0

@carlosuc3m
The original issue was related to Windows, it seems you are using Linux. Would you mind create a new issue and add some more detail about your issue there:
Please check out DJL repo in the machine and run the following command:
./gradlew debugEnv -Dai.djl.default_engine=PyTorch

I am using a raw version of Windows Server 2016. This are the pom dependencies I am using:
`
ai.djl
api
0.7.0

ai.djl.pytorch
pytorch-native-cpu
1.6.0
runtime

ai.djl.pytorch
pytorch-engine
0.7.0

ai.djl.pytorch
pytorch-model-zoo
0.7.0

`
The error is the same as before:
`ai.djl.engine.EngineException: No deep learning engine found.
Please refer to https://github.com/awslabs/djl/blob/master/docs/development/troubleshooting.md for more details.
	at ai.djl.engine.Engine.getInstance(Engine.java:98)
	at ai.djl.repository.zoo.DefaultModelZoo.getSupportedEngines(DefaultModelZoo.java:71)
	at ai.djl.repository.zoo.ModelZoo.loadModel(ModelZoo.java:111)`
And the code I am using is:
Criteria<NDList, NDList> criteria = Criteria.builder() 		.setTypes(NDList.class, NDList.class) 		.optModelUrls(url.toString()) // Parent folder of the model 		.optModelName(modelName) // model filename without extension 		.optProgress(new ProgressBar()).build(); ZooModel<NDList, NDList> model = ModelZoo.loadModel(criteria);
In addition to this, if I use the following dependencies:
`
ai.djl
api
0.7.0

ai.djl.pytorch
pytorch-native-cpu
1.5.0
runtime

ai.djl.pytorch
pytorch-engine
0.7.0

ai.djl.pytorch
pytorch-model-zoo
0.7.0
`
I get the same error mentioned above with the same code even after installing the redistributables. I dont know if this is an expected behaviour  but I wanted to mention it just in case.
Regards,
Carlos

@carlosuc3m Would you please clone latest DJL in your windows and run the following command:
./gradlew debugEnv -Dai.djl.default_engine=PyTorch

The above command will print extrad debug log, you should see extra failure before "No deep learning engine found."

I am using maven, should I switch to gradle for this?

@carlosuc3m you don't need to install anything, the command I provided will just work out of the box. We use gradle wrapper, which will download the gradle.jar on the fly.

Sorry for the delay, I am not an expert in these frameworks.
I have not been able to run
./gradlew debugEnv -Dai.djl.default_engine=PyTorch
It returns:
\git\djl\\gradle\wrapper\gradle-wrapper.jar was unexpected at this time.

Do you know what should I do?
I tried to run the program in debug mode (as mentioned above) and this is what I got, just in case it is helpful.
Unsupported format or not found
[Thread-2] DEBUG ai.djl.repository.zoo.DefaultModelZoo - Scanning models in repo: class ai.djl.repository.SimpleRepository, file:/C:/Users/AzureUser/Documents/models/New%20folder/
[Thread-2] DEBUG ai.djl.repository.zoo.ModelZoo - Searching model in specified model zoo: ai.djl.localmodelzoo
[Thread-2] DEBUG ai.djl.util.cuda.CudaUtils - Found cudart: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudart64_100.dll
[Thread-2] WARN ai.djl.pytorch.engine.PtEngine - Failed to load PyTorch native library
java.lang.NoSuchMethodError: com.sun.jna.Native.load(Ljava/lang/String;Ljava/lang/Class;)Lcom/sun/jna/Library;
	at ai.djl.util.cuda.CudaUtils.loadLibrary(CudaUtils.java:169)
	at ai.djl.util.cuda.CudaUtils.<clinit>(CudaUtils.java:29)
	at ai.djl.util.Platform.fromSystem(Platform.java:89)
	at ai.djl.pytorch.jni.LibUtils.findNativeLibrary(LibUtils.java:212)
	at ai.djl.pytorch.jni.LibUtils.loadLibrary(LibUtils.java:71)
	at ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:42)
	at ai.djl.pytorch.engine.PtEngineProvider.getEngine(PtEngineProvider.java:27)
	at ai.djl.engine.Engine.initEngine(Engine.java:50)
	at ai.djl.engine.Engine.<clinit>(Engine.java:44)
	at ai.djl.repository.zoo.DefaultModelZoo.getSupportedEngines(DefaultModelZoo.java:71)
	at ai.djl.repository.zoo.ModelZoo.loadModel(ModelZoo.java:111)
	at deepimagej.stamp.LoadPytorchStamp.run(LoadPytorchStamp.java:212)
	at java.lang.Thread.run(Unknown Source)
[Thread-2] WARN ai.djl.engine.Engine - Failed to load engine from: ai.djl.pytorch.engine.PtEngineProvider
[Thread-2] DEBUG ai.djl.engine.Engine - No engine found from EngineProvider
ai.djl.engine.EngineException: No deep learning engine found.
Please refer to https://github.com/awslabs/djl/blob/master/docs/development/troubleshooting.md for more details.
	at ai.djl.engine.Engine.getInstance(Engine.java:98)
	at ai.djl.repository.zoo.DefaultModelZoo.getSupportedEngines(DefaultModelZoo.java:71)
	at ai.djl.repository.zoo.ModelZoo.loadModel(ModelZoo.java:111)
	at deepimagej.stamp.LoadPytorchStamp.run(LoadPytorchStamp.java:212)
	at java.lang.Thread.run(Unknown Source)

REgards,
Carlos

@carlosuc3m
Sorry, the command for windows should be:
gradlew debugEnv -Dai.djl.default_engine=PyTorch

From the stack trace you provided, it looks like you included different version of JNA in your project.

with that command I get the same output.
\git\djl\\gradle\wrapper\gradle-wrapper.jar was unexpected at this time.

Isn´t the JNA provided by the dependencies?

@carlosuc3m
That's strange, can you remove gradle\wrapper\gradle-wrapper.jar and try it again?
Or can you manually download gradle-wrapper.jar and save it in gradle\wrapper folder: https://raw.githubusercontent.com/gradle/gradle/master/gradle/wrapper/gradle-wrapper.jar
java.lang.NoSuchMethodError indicate that there is another JNA jar file is included in the classpath. I guess it came from another library you included in your project.

Still cannot run it. Should I clone the latest DJL version in a specific directory? Should I change "debugEnv" by something?
Yes my project used another library that required reading native libraries, but with with the insallation of the Visual Studio 2019 redistributables there was no problem running both libraries.
Sorry if I am asking something too obvious but I am not an expert in this field.
Regards,
Carlos

@carlosuc3m
The java.lang.UnsatisfiedLinkError most likely is caused by missing VC2019 redistribute package, see: http://docs.djl.ai/docs/development/troubleshooting.html#13-unsatisfiedlinkerror-issue
A separate issue I noticed in your log is you are using CUDA 10.0, we don't support CUDA 10.0, only 10.1 and 10.2 are supported. The pytorch download on your windows is CPU only version.

Si in the new version we still need to instal VC2019 redistributables? I understood that with the new version they were no longer needed.
Yes I know that I need to change the CUDA version, but I though that for CPU it would not interfere.
Regards,
Carlos

@carlosuc3m
Sorry for miscommunication. We still need VC2019 redistributable. We won't be able to bundle the redistributable in the jar. The old version will failed to fallback to CPU in you have CUDA 10.0 installed. We only fixed that bug.

Hello again,
I tried using DJL in another computer with CUDA 10.0. After installing VS2019 and its redistributables I was still unable to run DJL either using:
<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-native-auto</artifactId>
    <version>1.6.0</version>
</dependency>

or
<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-native-cpu</artifactId>
    <version>1.6.0</version>
</dependency>

I always get the same error
ai.djl.engine.EngineException: No deep learning engine found. Please refer to https://github.com/awslabs/djl/blob/master/docs/development/troubleshooting.md for more details. at ai.djl.engine.Engine.getInstance(Engine.java:98) at ai.djl.repository.zoo.DefaultModelZoo.getSupportedEngines(DefaultModelZoo.java:71) at ai.djl.repository.zoo.ModelZoo.loadModel(ModelZoo.java:111)
I then installed CUDA 10.2 on my computer to see if it worked but again an error:
[Thread-2] DEBUG ai.djl.repository.zoo.DefaultModelZoo - Scanning models in repo: class ai.djl.repository.SimpleRepository, file:/C:/Users/AzureUser/Documents/models/New%20folder/ [Thread-2] DEBUG ai.djl.repository.zoo.ModelZoo - Searching model in specified model zoo: ai.djl.localmodelzoo [Thread-2] DEBUG ai.djl.util.cuda.CudaUtils - Found cudart: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin\cudart64_102.dll [Thread-2] WARN ai.djl.pytorch.engine.PtEngine - Failed to load PyTorch native library java.lang.NoSuchMethodError: com.sun.jna.Native.load(Ljava/lang/String;Ljava/lang/Class;)Lcom/sun/jna/Library; 	at ai.djl.util.cuda.CudaUtils.loadLibrary(CudaUtils.java:169) 	at ai.djl.util.cuda.CudaUtils.<clinit>(CudaUtils.java:29) 	at ai.djl.util.Platform.fromSystem(Platform.java:89) 	at ai.djl.pytorch.jni.LibUtils.findNativeLibrary(LibUtils.java:212) 	at ai.djl.pytorch.jni.LibUtils.loadLibrary(LibUtils.java:71) 	at ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:42) 	at ai.djl.pytorch.engine.PtEngineProvider.getEngine(PtEngineProvider.java:27) 	at ai.djl.engine.Engine.initEngine(Engine.java:50) 	at ai.djl.engine.Engine.<clinit>(Engine.java:44) 	at ai.djl.repository.zoo.DefaultModelZoo.getSupportedEngines(DefaultModelZoo.java:71) 	at ai.djl.repository.zoo.ModelZoo.loadModel(ModelZoo.java:111) 	at deepimagej.stamp.LoadPytorchStamp.run(LoadPytorchStamp.java:212) 	at java.lang.Thread.run(Unknown Source) [Thread-2] WARN ai.djl.engine.Engine - Failed to load engine from: ai.djl.pytorch.engine.PtEngineProvider [Thread-2] DEBUG ai.djl.engine.Engine - No engine found from EngineProvider ai.djl.engine.EngineException: No deep learning engine found. Please refer to https://github.com/awslabs/djl/blob/master/docs/development/troubleshooting.md for more details. 	at ai.djl.engine.Engine.getInstance(Engine.java:98) 	at ai.djl.repository.zoo.DefaultModelZoo.getSupportedEngines(DefaultModelZoo.java:71) 	at ai.djl.repository.zoo.ModelZoo.loadModel(ModelZoo.java:111) 	at deepimagej.stamp.LoadPytorchStamp.run(LoadPytorchStamp.java:212) 	at java.lang.Thread.run(Unknown Source)
I am using a WIndowsServer 2016 with the following pom.xml:

		<dependency>
		    <groupId>ai.djl</groupId>
		    <artifactId>api</artifactId>
		    <version>0.7.0</version>
		</dependency>
		<dependency>
		    <groupId>ai.djl.pytorch</groupId>
		    <artifactId>pytorch-native-auto</artifactId>
		    <version>1.6.0</version>
		</dependency>
	    <dependency>
	      <groupId>ai.djl.pytorch</groupId>
	      <artifactId>pytorch-engine</artifactId>
		    <version>0.7.0</version>
	    </dependency>

       <!-- https://mvnrepository.com/artifact/ai.djl.pytorch/pytorch-model-zoo -->
       <dependency>
           <groupId>ai.djl.pytorch</groupId>
           <artifactId>pytorch-model-zoo</artifactId>
           <version>0.7.0</version>
       </dependency>
	    <dependency>
	      <groupId>ai.djl.pytorch</groupId>
	      <artifactId>pytorch-model-zoo</artifactId>
	      <version>0.7.0</version>
	    </dependency>

Regards and thanks for your time,
Carlos

Using
<dependency>
    <groupId>ai.djl.pytorch</groupId>
    <artifactId>pytorch-native-cu102</artifactId>
    <version>1.6.0</version>
</dependency>

Still produces the same error

However if I uninstall CUDA 10.0 and delete C:\Program Files\NVIDIA GPU Computing Toolkit and C:\Program Files\NVIDIA Corporation it works well either with CUDA10.2 or without any CUDA and just the CPU

I have also realised that removing all the enviroment variables related to CUDA from the PATH allow falling back to CPU. If any of those variables is present and there is an incompatible CUDA version, fallback does not happen in version 1.6.0.
The variables are:

C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\libnvvp
C:\Program Files\NVIDIA Corporation\Nsight Compute 2019.5.0\

HOwever I have not found a way to have two CUDA versions installed and make the problem run well.
Regards,
Carlos

repo: awslabs/djl


text:
[wip] Ltsubnetsecgrp

1. Issue, if available:
https://github.com/awslabs/karpenter/issues/661
2. Description of changes:
Implements provider fields for subnets, securitygroups, launchtemplate, and instanceprofile.
Currently none of these are overridable at the pod level. This is done intentionally to maintain separation of concerns between cluster operators and application developers. This may change if we get feature requests for this in the future.
3. Does this change impact docs?

[ ] Yes, PR includes docs updates
[ ] Yes, issue opened: link to issue
[ ] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

I took a first pass on the WIP, looking good!

repo: awslabs/karpenter


text:
GuardDutyCreatePublishingDestinationCommand creation fails with InvalidInputException

Describe the bug
We manually added a new AWS account to our Organization, added the necessary LZA config, and ran this pipeline. When running the AWSAccelerator-Pipeline, it runs until it reaches the Deploy stage and fails on the Security step with this error:
AWSAccelerator-SecurityStack-*-us-east-1 | 7:07:00 PM | CREATE_FAILED        | Custom::GuardDutyCreatePublishingDestinationCommand | GuardDutyPublishingDestination/Resource/Default (GuardDutyPublishingDestination52AE4412) Received response status [FAILED] from custom resource. Message returned: InvalidInputException: The request failed because you do not have the required permissions for the s3:GetObject or s3:ListBucket actions.
The new account is in the same OU as our other workload accounts and we did not experience this error with those.
To Reproduce
Error occurs every time we run the pipeline since adding the new account.
Expected behavior
Pipeline runs successfully.
Please complete the following information about the solution:

[1.4.3 ] Version: [e.g. v1.1.0]

To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "(SO0021) - Video On Demand workflow with AWS Step Functions, MediaConvert, MediaPackage, S3, CloudFront and DynamoDB. Version v5.0.0". If the description does not contain the version information, you can look at the mappings section of the template:
Mappings:
  SourceCode:
    General:
      S3Bucket: "solutions"
      KeyPrefix: "video-on-demand-on-aws/v5.0.0"

[ ] Region: [e.g. us-east-1] us-east-1
[ ] Was the solution modified from the version published on this repository? No
[ ] If the answer to the previous question was yes, are the changes available on GitHub?
[ ] Have you checked your service quotas for the sevices this solution uses? Yes
[ ] Were there any errors in the CloudWatch Logs? CloudFormation logs were same seen in pipeline

Screenshots
If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).
Additional context
Add any other context about the problem here.

Hi Grant! Thank you for using the Landing Zone Accelerator on AWS solution. Given the error you've provided, it looks like the root cause of this is due to an issue with the lifecycle rule removing the GuardDuty prefix. To confirm this behavior, please run the following command for your Logging account.
aws s3api get-object-acl --bucket aws-accelerator-central-logs-<your-account>-us-east-1 --key "guardduty/"

If you receive a NoSuchKey error, then the object representing the folder most likely got deleted by a lifecycle. If this is the case, we have this bug addressed in our upcoming v1.5.0 release. In the meantime, you should be able to recreate this prefix, then re-run your pipeline.
If you're receiving a response back from the aforementioned API call, then please let me know and I can dig further for you.

repo: awslabs/landing-zone-accelerator-on-aws


text:
Scope down IAM policies in templates/iam.py

From the TODO here: https://github.com/awslabs/pipeformer/pull/13/files#diff-d27d002e8b30320ac43519019059850fR59

TODO: Figure out how to scope this down without breaking IAM. IAM policies break if there is a * in certain fields, so this does not work: arn:PARTITION:*:REGION:ACCOUNT:*

Closing down project: #52

repo: awslabs/pipeformer


text:
Ability to build s2n on Travis without sudo permissions

From: https://docs.travis-ci.com/user/trusty-ci-environment/

The advantage of running with sudo: false is dramatically reduced time between commit push to GitHub and the start of a job on Travis CI, which works especially well when one’s average job duration is under 3 minutes.

This may help with our build speeds if we are able to remove our requirement on using sudo.

Is this still true? After https://blog.travis-ci.com/2017-06-21-trusty-updates-2017-Q2-launch the builds seem quicker.

We'll be leaving travis soon.

authors: alexw91 dougch raycoll