text:
Parsing error handling rule file

General Issue
yes
The Question

I am running the cfn-guard validation command to test the cloudformation template but getting the error with multiple rules. Attaching the screenshot with the Parsing error handling rule file error message. Am I doing it in a wrong way or there is any issue with cfn-guard handling these rules files.
using command: cfn-guard validate -v --data cfn-template --rules ./aws-guard-rules-registry-1.0.2/rules/aws
where i have my template file in yaml format inside cfn-template folder.
CloudFormation Guard Version
2.1.3
OS
Amazon Linux
OS Version
No response
Other information
Running the command while build in AWS CodeBuild. was testing it for IAM policy and all the rules related to IAM policy were PASS. But not sure why this parsing error with other rule files.

Can anyone help on this issue?

Hi, have you downloaded the release rule sets and used the rule set files in there? Check out the releases and download the aggregated rule set files from there.

Take a look at Using Guard Rules Registry docs for examples.

Hi, have you downloaded the release rule sets and used the rule set files in there? Check out the releases and download the aggregated rule set files from there.

Yes, did the same.

Take a look at Using Guard Rules Registry docs for examples.

yes, followed the same.

@grolston, Any other suggestion. I am not sure if I am the only one facing this issue.

What is appears you are doing is using the cfn-guard command not as documented:
You have cfn-guard validate -v --data cfn-template --rules ./aws-guard-rules-registry-1.0.2/rules/aws
Your rules you are using is the Guard rules directory and not a the compiled rules (it was not tested nor intended to be used like that in the raw form). When you download the release rules from here and unzip the folder you will see files you can use in the output directory. For example, in there is a file named NIST800-53Rev5.guard if I use the command:
cfn-guard validate -v --data cfn-template --rules ./NIST800-53Rev5.guard
it will test the template against the rules in the NIST800-53Rev5.guard file.
If you are looking for testing against every rule (not recommended as you should have a plan for what your rule set should include), you an use the guard-rules-registry-all-rules.guard located in the output directory of the release.

authors: grolston nitinitare


text:
Revert "Update Otel Version15.0 dependencies and licenses"

Reverts aws-observability/aws-otel-java-instrumentation#180

Codecov Report

Merging #189 (0d4e2f3) into main (4654c1e) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main     #189   +/-   ##
=========================================
  Coverage     73.91%   73.91%
  Complexity       15       15
=========================================
  Files             3        3
  Lines            46       46
  Branches          5        5
=========================================
  Hits             34       34
  Misses            8        8
  Partials          4        4

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

source: gharchive/pull-request

license_type: permissive

license_source: github-api


text:
Bug: Examples should serialize under parameters when using Path parameters

Expected Behaviour
Original report: https://github.com/aws-powertools/powertools-lambda-python/pull/5575/files#r1846670069
The examples should be under parameters, not under schema.
https://swagger.io/specification/#parameter-object
I was not able to change the output location of the examples. In this PR, it will only enable the definition of examples in the Schema Object.
https://swagger.io/specification/#schema-object
wrong
        "parameters": [
          {
            "required": false,
            "schema": {
              "type": "integer",
              "exclusiveMaximum": 100.0,
              "exclusiveMinimum": 0.0,
              "title": "Count",
              "default": 1,
              "examples": [
                {
                  "summary": "Example 1",
                  "description": null,
                  "value": 10,
                  "externalValue": null
                }
              ]
            },
            "name": "count",
            "in": "query"
          }
        ],

expected
        "parameters": [
          {
            "required": false,
            "schema": {
              "type": "integer",
              "exclusiveMaximum": 100.0,
              "exclusiveMinimum": 0.0,
              "title": "Count",
              "default": 1
            },
            "examples": [
              {
                "summary": "Example 1",
                "description": null,
                "value": 10,
                "externalValue": null
              }
            ],
            "name": "count",
            "in": "query"
          }
        ],

Current Behaviour
This is being serialized under schema and not showing in SwaggerUI.
        "parameters": [
          {
            "required": false,
            "schema": {
              "type": "integer",
              "exclusiveMaximum": 100.0,
              "exclusiveMinimum": 0.0,
              "title": "Count",
              "default": 1,
              "examples": [
                {
                  "summary": "Example 1",
                  "description": null,
                  "value": 10,
                  "externalValue": null
                }
              ]
            },
            "name": "count",
            "in": "query"
          }
        ],

Code snippet
from typing import Annotated, List

import requests
from aws_lambda_powertools.event_handler import APIGatewayRestResolver
from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools.event_handler.openapi.params import Path
from aws_lambda_powertools.event_handler.openapi.models import Example
from pydantic import BaseModel, Field
from dataclasses import dataclass

app = APIGatewayRestResolver(enable_validation=True)
app.enable_swagger(path="/swagger")  # (1)!

@app.get("/todos/<todo_id>")
def get_todo_by_id(todo_id: Annotated[str, Path(examples=Example(summary='a',value="1"))]) -> str:
    todo = requests.get("https://jsonplaceholder.typicode.com/todos")
    todo.raise_for_status()

    return "test"

def lambda_handler(event: dict, context: LambdaContext) -> dict:
    return app.resolve(event, context)

if __name__ == "__main__":
    print(app.get_openapi_json_schema())

Possible Solution
No response
Steps to Reproduce
Run the code above
Powertools for AWS Lambda (Python) version
latest
AWS Lambda function runtime
3.13
Packaging format used
Lambda Layers, PyPi
Debugging logs

In FastAPI, the example definitions for JSON Schema and OpenAPI are handled separately.
I think it would be better to follow this approach in Powertools as well and prepare openapi_examples separately.
openapi_examples: Optional[Dict[str, Example]] = None

https://fastapi.tiangolo.com/tutorial/schema-extra-example/#using-the-openapi_examples-parameter
https://github.com/fastapi/fastapi/blob/0.115.5/fastapi/params.py#L69

source: gharchive/issue

authors: leandrodamascena tonsho

repo: aws-powertools/powertools-lambda-python


text:
Ack addon must support inline policies for EKS

Description of changes:
The EKS ack controller doesn't have a recommended managed policy, only an inline one, and the currently provided managed policy in blueprints doesn't seem to give any eks permissions, nor does any other managed policy seem to be appropriate. The best option is to support inline policies. There may be other ack controllers with similar issues, but EKS is the one I'm aware of.
Also, this file appeared to have CRLF, so I've converted it to LF to match the rest of the project - I suggest viewing the diff without whitespace
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

I don't observe tests for this, how did you validate it?

I validated by running the EKS ACK controller, which does not work without this change as it needs eks permissions which were not present before. You can simply add the following addon to a blueprint to test:
const addOn = new blueprints.addons.AckAddOn({
  serviceName: AckServiceName.EKS,
}),

And then you can try applying a resource such as:
apiVersion: eks.services.k8s.aws/v1alpha1
kind: PodIdentityAssociation
metadata:
  name: test
  namespace: default
spec:
  clusterName: your-cluster-name
  namespace: default
  roleARN: some-role-arn-in-this-account
  serviceAccount: default

and running kubectl describe to see that the ack controller had enough permissions to attempt to create it (most likely it will fail if you didnt set the trust policy on the role appropriately, but that still proves the permissions work)

@Pjv93 that is weird. If you put the following in examples/blueprint-construct/index.ts:
import * as cdk from 'aws-cdk-lib';
import {Construct} from "constructs";
import * as blueprints from '../../lib';

const blueprintID = 'blueprint-construct-dev';

export default class BlueprintConstruct {
  constructor(scope: Construct, props: cdk.StackProps) {
    const addOns: Array<blueprints.ClusterAddOn> = [
      new blueprints.addons.AckAddOn({
        serviceName: blueprints.AckServiceName.EKS
      }),
    ];

    blueprints.EksBlueprint.builder()
      .version("auto")
      .addOns(...addOns)
      .build(scope, blueprintID, props);
  }
}

And run npx cdk synth, you should see the inline policy as an attachment to the role:
  blueprintconstructdevekschartsaRoleB7051B6A:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRoleWithWebIdentity
            Condition:
              StringEquals:
                Fn::GetAtt:
                  - blueprintconstructdevekschartsaConditionJson6E37C7DB
                  - Value
            Effect: Allow
            Principal:
              Federated:
                Ref: blueprintconstructdevOpenIdConnectProvider3733B8DD
        Version: "2012-10-17"
      ManagedPolicyArns:
        - Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :iam::aws:policy/AmazonEKSClusterPolicy
...
  inlinepolicy168DC373:
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Statement:
          - Action:
              - eks:*
              - iam:GetRole
              - iam:PassRole
            Effect: Allow
            Resource: "*"
        Version: "2012-10-17"
      PolicyName: inlinepolicy168DC373
      Roles:
        - Ref: blueprintconstructdevekschartsaRoleB7051B6A
    Metadata:
      aws:cdk:path: blueprint-construct-dev/inline-policy/Resource

And when I run npx cdk deploy --require-approval=any-change, it does show the inline policy:

So I am not sure what could be the problem
As a sidenote, I think I should remove the EksClusterPolicy for EKS as it is unnecessary and confusing

@jackkleeman Thanks for that, the mistake was actually on my end. After running npx cdk synth && npx cdk deploy --require-approval=any-change, i see the inline policy and it looks good! I can also confirm the s3 inline policy pattern is also working. Great work.

@elamaran11 your review is marked as "requested changes". can't merge with that one, please review.

aa923ee

+1. Thankyou for the Contribution

repo: aws-quickstart/cdk-eks-blueprints


text:
Docker packaging improvements

Overview
Fixes package output when using docker in docker (codebuild/dockerized taskcat) and fixes cases where package zips could be owned by root.

/do-e2e-tests

/do-e2e-tests

repo: aws-quickstart/taskcat

license_source: bigquery


text:
Adding Converse API notebooks

Issue #, if available: None.
Description of changes: Adding 2 notebooks for the Converse API in Bedrock, one in the 'introduction-to-bedrock' folder, and another in the 'function-calling' folder.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Intro notebook works
Can we simplify this:

    tools_list_obj = ToolsList()
    function_calling = ''
    tool_args = {}
    for c in output['output']['message']['content']:
        if 'toolUse' in c:
            function_calling = c['toolUse']

    #Check if function calling is triggered:
    if (function_calling):
    ```

    2. Error in the function calling example, full trace

    ```
    ---------------------------------------------------------------------------
ParamValidationError                      Traceback (most recent call last)
Cell In[10], line 7
      1 prompts = [
      2     "What is the weather like in Queens, NY?",
      3     "What is the capital of France?"
      4 ]
      6 for prompt in prompts:
----> 7     converse(
      8         system = "You're provided with a tool that can get the weather information for a specific locations 'get_weather'; \
      9             only use the tool if required. Don't make reference to the tools in your final answer.",
     10         prompt = prompt
     11 )

Cell In[9], line 18, in converse(prompt, system)
     15 print(f"\n{datetime.now().strftime('%H:%M:%S')} - Initial prompt:\n{json.dumps(messages, indent=2)}")
     17 #Invoke the model the first time:
---> 18 output = converse_with_tools(messages, system)
     19 print(f"\n{datetime.now().strftime('%H:%M:%S')} - Output so far:\n{json.dumps(output['output'], indent=2)}")
     21 #Add the intermediate output to the prompt:

Cell In[7], line 3, in converse_with_tools(messages, system, toolConfig)
      2 def converse_with_tools(messages, system='', toolConfig=toolConfig):
----> 3     response = bedrock.converse(
      4         modelId=modelId,
      5         system=system,
      6         messages=messages,
      7         toolConfig=toolConfig
      8     )
      9     return response

File /opt/conda/lib/python3.10/site-packages/botocore/client.py:565, in ClientCreator._create_api_method.<locals>._api_call(self, *args, **kwargs)
    561     raise TypeError(
    562         f"{py_operation_name}() only accepts keyword arguments."
    563     )
    564 # The "self" in this scope is referring to the BaseClient.
--> 565 return self._make_api_call(operation_name, kwargs)

File /opt/conda/lib/python3.10/site-packages/botocore/client.py:974, in BaseClient._make_api_call(self, operation_name, api_params)
    970 if properties:
    971     # Pass arbitrary endpoint info with the Request
    972     # for use during construction.
    973     request_context['endpoint_properties'] = properties
--> 974 request_dict = self._convert_to_request_dict(
    975     api_params=api_params,
    976     operation_model=operation_model,
    977     endpoint_url=endpoint_url,
    978     context=request_context,
    979     headers=additional_headers,
    980 )
    981 resolve_checksum_context(request_dict, operation_model, api_params)
    983 service_id = self._service_model.service_id.hyphenize()

File /opt/conda/lib/python3.10/site-packages/botocore/client.py:1048, in BaseClient._convert_to_request_dict(self, api_params, operation_model, endpoint_url, context, headers, set_user_agent_header)
   1039 def _convert_to_request_dict(
   1040     self,
   1041     api_params,
   (...)
   1046     set_user_agent_header=True,
   1047 ):
-> 1048     request_dict = self._serializer.serialize_to_request(
   1049         api_params, operation_model
   1050     )
   1051     if not self._client_config.inject_host_prefix:
   1052         request_dict.pop('host_prefix', None)

File /opt/conda/lib/python3.10/site-packages/botocore/validate.py:381, in ParamValidationDecorator.serialize_to_request(self, parameters, operation_model)
    377     report = self._param_validator.validate(
    378         parameters, operation_model.input_shape
    379     )
    380     if report.has_errors():
--> 381         raise ParamValidationError(report=report.generate_report())
    382 return self._serializer.serialize_to_request(
    383     parameters, operation_model
    384 )

ParamValidationError: Parameter validation failed:
Invalid type for parameter system, value: You're provided with a tool that can get the weather information for a specific locations 'get_weather';             only use the tool if required. Don't make reference to the tools in your final answer., type: <class 'str'>, valid types: <class 'list'>, <class 'tuple'>

updated boto3 and botocore to latest as of now. @markproy @rodzanto

works without passing in the system message FYI

repo: aws-samples/amazon-bedrock-samples


text:
How to make a rewrite?

The docs says that is possible to use a rewrite, but there isn't any example on how to achieve that
CloudFront Functions Docs
I tried to change the request, in the test tab it works but no in the cloudfront distribution.
Code:
  if (host.startsWith("www.")) {
    host = host.replace("www.", "");
    request.headers.host.value = host;
    return request;

  }

Test in cloudfront Function:

Test against distribution URL:
502 ERROR
The request could not be satisfied.
The CloudFront function tried to add, delete, or change a read-only header. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.

I ended with this code for all my needs:
function handler(event) {
  var shouldRedirect = false;
  var request = event.request;
  var queryString = queryStringObjectToString(request.querystring);
  var uri = request.uri;
  var regex = /[A-Z]/g;
  var headers = request.headers;
  var host = headers.host.value;
  var last_uri_character = uri.slice(-1);

  // If the URI have a dot we assume is a file and we don't redirect or rewrite
  if (uri.includes(".")) {
    return request;
  }

  // If the URI contains a capital letter, rewrite to the lowercase version
  if (regex.test(uri)) {
    uri = request.uri.toLowerCase();
    request.uri = uri;
  }

  // If the host is www we redirect to the naked domain
  if (host.startsWith("www.")) {
    host = host.replace("www.", "");
    delete headers.host;
    shouldRedirect = true;
  }

  // If the URI ends with a slash, redirect to the non-slash version
  if (last_uri_character !== "/") {
    uri = uri + "/";
    shouldRedirect = true;
  }

  // If we should redirect, do it
  if (shouldRedirect) {
    var url = "https://" + host + uri + queryString;
    var response = {
      statusCode: 301,
      statusDescription: "Permanently Moved",
      headers: { location: { value: url } },
      cookies: request.cookies,
    };
    return response;
  }
  return request;
}

/**
 * @description: Converts an event.request.querystring object back to a query string
 * @date 2022-11-28
 * @param {object} queryStringObject: Query string object as provided by event.request
 * @return {string}:                  A normalised query string in the form ?key1=valn&key2=val2...
 */
function queryStringObjectToString(queryStringObject) {
  // Convert the query string object to an array of entries and reduce to a single string
  return (
    Object.entries(queryStringObject)
      .reduce((p, q) => {
        if (!q[1].multiValue) {
          // Process a single key/value property
          return (p += q[0] + "=" + q[1].value + "&");
        } else {
          // Process multiValue properties. E.g arrays
          return (p +=
            q[1].multiValue.map((v) => q[0] + "=" + v.value).join("&") + "&");
        }
      }, "?")
      // Remove the trailing ampersand
      .slice(0, -1)
  );
}

Thanks.

repo: aws-samples/amazon-cloudfront-functions


text:
Codeserver Icon doesn't appear

So I've followed this guide to set up coderserver on SageMaker Studio.
The codeserver icon doesn't appear but I can still access the codeserver by slightly changing the URL -
[domain-id].studio.[aws-region].sagemaker.aws/jupyter/default/codeserver. Screenshot below.

I suspect that icon fails to attach because of the new SageMaker Studio UI. Screenshot below.

I noticed the new UI on 5th December 22 which is when I tried to set up the codeserver on SageMaker Studio.

Hi @nskrjabins, thanks for reaching out and for opening this issue.
We are aware of this behavior, and we will provide an update soon.
In the meantime, the workaround you have identified is what we would suggest doing to continue accessing code-server.
I'll keep this issue open for updates.

@giuseppeporcelli Any ETA on when this will be fixed?

Any ETA?

It appears for me now with the latest version of the studio install script.

This didn't work for me, the code server icon still doesn't appear in the UI

I also don't see the icon, changing the url works (JL3).

After updating the domain and setting the default SageMakerImageArn to "arn:aws:sagemaker:eu-central-1:936697816551:image/jupyter-server-3" the icon shows up...

Hi,
I'm happy to share that this issue is solved by upgrading to the latest version (5.1000.0) of Amazon SageMaker Studio (JupyterServer). Then, you just need to re-install code-server as explained in this repo.
The code-server icon will appear in the launcher as expected, as shown below:

Instructions on how to upgrade Amazon SageMaker Studio are here: https://docs.aws.amazon.com/sagemaker/latest/dg/studio-tasks-update-studio.html
Thanks for your patience.

@giuseppeporcelli I think https://github.com/jupyterlab/jupyterlab/releases/tag/v3.6.1 might have broken the icons in studio again.

authors: Almenon giuseppeporcelli massi-ang nskrjabins orangewise tonyszhang wiliscavalcante

repo: aws-samples/amazon-sagemaker-codeserver


text:
Log downloads NoRegionError.

I am trying to download the logs with cw_utils but I keep getting NoRegionError. Does anyone know how I can I fix this?
stream_name = 'XXXX'
fname = 'logs/deepracer-%s.log' %stream_name
cw_utils.download_log(fname, stream_prefix=stream_name)

---------------------------------------------------------------------------
NoRegionError                             Traceback (most recent call last)
<ipython-input-48-da5f8f73ddad> in <module>
----> 1 cw_utils.download_log(fname, stream_prefix=stream_name)

~/projects/deepracer-models/aws-deepracer-workshops/log-analysis/cw_utils.py in download_log(fname, stream_name, stream_prefix, log_group, start_time, end_time)
     59             end_time=end_time
     60         )
---> 61         for event in logs:
     62             f.write(event['message'].rstrip())
     63             f.write("\n")

~/projects/deepracer-models/aws-deepracer-workshops/log-analysis/cw_utils.py in get_log_events(log_group, stream_name, stream_prefix, start_time, end_time)
     12
     13 def get_log_events(log_group, stream_name=None, stream_prefix=None, start_time=None, end_time=None):
---> 14     client = boto3.client('logs')
     15     if stream_name is None and stream_prefix is None:
     16         print("both stream name and prefix can't be None")

/anaconda3/lib/python3.7/site-packages/boto3/__init__.py in client(*args, **kwargs)
     89     See :py:meth:`boto3.session.Session.client`.
     90     """
---> 91     return _get_default_session().client(*args, **kwargs)
     92
     93

/anaconda3/lib/python3.7/site-packages/boto3/session.py in client(self, service_name, region_name, api_version, use_ssl, verify, endpoint_url, aws_access_key_id, aws_secret_access_key, aws_session_token, config)
    261             aws_access_key_id=aws_access_key_id,
    262             aws_secret_access_key=aws_secret_access_key,
--> 263             aws_session_token=aws_session_token, config=config)
    264
    265     def resource(self, service_name, region_name=None, api_version=None,

/anaconda3/lib/python3.7/site-packages/botocore/session.py in create_client(self, service_name, region_name, api_version, use_ssl, verify, endpoint_url, aws_access_key_id, aws_secret_access_key, aws_session_token, config)
    837             is_secure=use_ssl, endpoint_url=endpoint_url, verify=verify,
    838             credentials=credentials, scoped_config=self.get_scoped_config(),
--> 839             client_config=config, api_version=api_version)
    840         monitor = self._get_internal_component('monitor')
    841         if monitor is not None:

/anaconda3/lib/python3.7/site-packages/botocore/client.py in create_client(self, service_name, region_name, is_secure, endpoint_url, verify, credentials, scoped_config, api_version, client_config)
     84         client_args = self._get_client_args(
     85             service_model, region_name, is_secure, endpoint_url,
---> 86             verify, credentials, scoped_config, client_config, endpoint_bridge)
     87         service_client = cls(**client_args)
     88         self._register_retries(service_client)

/anaconda3/lib/python3.7/site-packages/botocore/client.py in _get_client_args(self, service_model, region_name, is_secure, endpoint_url, verify, credentials, scoped_config, client_config, endpoint_bridge)
    326         return args_creator.get_client_args(
    327             service_model, region_name, is_secure, endpoint_url,
--> 328             verify, credentials, scoped_config, client_config, endpoint_bridge)
    329
    330     def _create_methods(self, service_model):

/anaconda3/lib/python3.7/site-packages/botocore/args.py in get_client_args(self, service_model, region_name, is_secure, endpoint_url, verify, credentials, scoped_config, client_config, endpoint_bridge)
     45         final_args = self.compute_client_args(
     46             service_model, client_config, endpoint_bridge, region_name,
---> 47             endpoint_url, is_secure, scoped_config)
     48
     49         service_name = final_args['service_name']

/anaconda3/lib/python3.7/site-packages/botocore/args.py in compute_client_args(self, service_model, client_config, endpoint_bridge, region_name, endpoint_url, is_secure, scoped_config)
    115
    116         endpoint_config = endpoint_bridge.resolve(
--> 117             service_name, region_name, endpoint_url, is_secure)
    118
    119         # Override the user agent if specified in the client config.

/anaconda3/lib/python3.7/site-packages/botocore/client.py in resolve(self, service_name, region_name, endpoint_url, is_secure)
    400         region_name = self._check_default_region(service_name, region_name)
    401         resolved = self.endpoint_resolver.construct_endpoint(
--> 402             service_name, region_name)
    403         if resolved:
    404             return self._create_endpoint(

/anaconda3/lib/python3.7/site-packages/botocore/regions.py in construct_endpoint(self, service_name, region_name)
    120         for partition in self._endpoint_data['partitions']:
    121             result = self._endpoint_for_partition(
--> 122                 partition, service_name, region_name)
    123             if result:
    124                 return result

/anaconda3/lib/python3.7/site-packages/botocore/regions.py in _endpoint_for_partition(self, partition, service_name, region_name)
    133                 region_name = service_data['partitionEndpoint']
    134             else:
--> 135                 raise NoRegionError()
    136         # Attempt to resolve the exact region for this partition.
    137         if region_name in service_data['endpoints']:

NoRegionError: You must specify a region.

I will close this as the problem got fixed.

repo: aws-samples/aws-deepracer-workshops


text:
Why MUSL?

Lambda Rust with provided.al2 can run on native linux binaries: https://github.com/umccr/s3-rust-noodles-bam/tree/s3-server
See: https://github.com/awslabs/aws-lambda-rust-runtime/discussions/306

For now, Lambda Adapter is used in container images, not custom runtime.
We use musl to build Lambda Adapter as a statically linked binary, so that it works in any linux base images that customers may choose. It should even work in a blank baseimage such as SCRATCH.

authors: bnusunny brainstorm

repo: aws-samples/aws-lambda-adapter


text:
Fix stack cleanup

On some instances TemplateDescription can have '\n' encoded inside it but when we process it through bash echo then it would cause line break resulting in invalid json.
Here is sample output I received from list-stacks that resulted in breaking cleanup.sh (notice \n in json response)
➜  aws-saas-factory-ref-solution-serverless-saas git:(adil/fix_stack_cleanup) ✗ saas aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE ROLLBACK_COMPLETE UPDATE_COMPLETE UPDATE_ROLLBACK_COMPLETE IMPORT_COMPLETE IMPORT_ROLLBACK_COMPLETE
...

        {
            "StackId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas-APIs-17O4B7RH955WQ/da23b660-e53e-11ed-8b82-12cd17ee3641",
            "StackName": "serverless-saas-APIs-17O4B7RH955WQ",
            "TemplateDescription": "Template to setup api gateway, apis, api keys and usage plan as part of bootstrap\n",
            "CreationTime": "2023-04-27T21:02:43.084000+00:00",
            "StackStatus": "CREATE_COMPLETE",
            "ParentId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
            "RootId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
            "DriftInformation": {
                "StackDriftStatus": "NOT_CHECKED"
            }
        },
...

@suhussai

Hi @adilhafeez ,
I'm having trouble replicating the issue.
I copied the output in your comment and ran jq to parse it, and it didn't throw any error:
❯ cat jsonNewLine
{
  "StackId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas-APIs-17O4B7RH955WQ/da23b660-e53e-11ed-8b82-12cd17ee3641",
  "StackName": "serverless-saas-APIs-17O4B7RH955WQ",
  "TemplateDescription": "Template to setup api gateway, apis, api keys and usage plan as part of bootstrap\n",
  "CreationTime": "2023-04-27T21:02:43.084000+00:00",
  "StackStatus": "CREATE_COMPLETE",
  "ParentId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
  "RootId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
  "DriftInformation": {
    "StackDriftStatus": "NOT_CHECKED"
  }
}

❯ cat jsonNewLine | jq
{
  "StackId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas-APIs-17O4B7RH955WQ/da23b660-e53e-11ed-8b82-12cd17ee3641",
  "StackName": "serverless-saas-APIs-17O4B7RH955WQ",
  "TemplateDescription": "Template to setup api gateway, apis, api keys and usage plan as part of bootstrap\n",
  "CreationTime": "2023-04-27T21:02:43.084000+00:00",
  "StackStatus": "CREATE_COMPLETE",
  "ParentId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
  "RootId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
  "DriftInformation": {
    "StackDriftStatus": "NOT_CHECKED"
  }
}

❯ cat jsonNewLine | jq '.TemplateDescription'
"Template to setup api gateway, apis, api keys and usage plan as part of bootstrap\n"

I might be missing something. Can you elaborate on the exact error you are seeing and how I can reproduce it?

here is how to repro

➜  ~ cat > json
[
    {
        "StackId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas-APIs-17O4B7RH955WQ/da23b660-e53e-11ed-8b82-12cd17ee3641",
        "StackName": "serverless-saas-APIs-17O4B7RH955WQ",
        "TemplateDescription": "Template to setup api gateway, apis, api keys and usage plan as part of bootstrap\n",
        "CreationTime": "2023-04-27T21:02:43.084000+00:00",
        "StackStatus": "CREATE_COMPLETE",
        "ParentId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
        "RootId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
        "DriftInformation": {
            "StackDriftStatus": "NOT_CHECKED"
        }
    }
]
➜  ~ X=`cat json`
➜  ~ echo $X | jq .
parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 6, column 1

and the fix
➜  ~ Y=$(cat json | sed -e 's/\\n//g')
➜  ~ echo $Y
[
    {
        "StackId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas-APIs-17O4B7RH955WQ/da23b660-e53e-11ed-8b82-12cd17ee3641",
        "StackName": "serverless-saas-APIs-17O4B7RH955WQ",
        "TemplateDescription": "Template to setup api gateway, apis, api keys and usage plan as part of bootstrap",
        "CreationTime": "2023-04-27T21:02:43.084000+00:00",
        "StackStatus": "CREATE_COMPLETE",
        "ParentId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
        "RootId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
        "DriftInformation": {
            "StackDriftStatus": "NOT_CHECKED"
        }
    }
]
➜  ~ echo $Y | jq .
[
  {
    "StackId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas-APIs-17O4B7RH955WQ/da23b660-e53e-11ed-8b82-12cd17ee3641",
    "StackName": "serverless-saas-APIs-17O4B7RH955WQ",
    "TemplateDescription": "Template to setup api gateway, apis, api keys and usage plan as part of bootstrap",
    "CreationTime": "2023-04-27T21:02:43.084000+00:00",
    "StackStatus": "CREATE_COMPLETE",
    "ParentId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
    "RootId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
    "DriftInformation": {
      "StackDriftStatus": "NOT_CHECKED"
    }
  }
]

This is really interesting. I have been using this cleanup.sh script for a while, and have never seen this error before. I'll have to do some digging to figure out why.

FWIW I am using mac m1 max (13.1 (22C65))

This was a real head scratcher, but I think I got it. It seems to be a shell issue. I created this small script to mimic what the script was doing:
cat <<EOT >> jsonNewLine.txt
[
    {
        "StackId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas-APIs-17O4B7RH955WQ/da23b660-e53e-11ed-8b82-12cd17ee3641",
        "StackName": "serverless-saas-APIs-17O4B7RH955WQ",
        "TemplateDescription": "Template to setup api gateway, apis, api keys and usage plan as part of bootstrap\n",
        "CreationTime": "2023-04-27T21:02:43.084000+00:00",
        "StackStatus": "CREATE_COMPLETE",
        "ParentId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
        "RootId": "arn:aws:cloudformation:us-east-1:264380604816:stack/serverless-saas/ebd2a610-e53d-11ed-a021-12e60651c6c3",
        "DriftInformation": {
            "StackDriftStatus": "NOT_CHECKED"
        }
    }
]
EOT

response=$(cat jsonNewLine.txt)

name=$(echo "$response" | jq -r '.[0].StackName')

echo "$name"

rm jsonNewLine.txt

Now, when I run that script using bash, it works:
❯ bash script.sh
serverless-saas-APIs-17O4B7RH955WQ

But, when I run it using sh, I get the error you posted:
❯ sh script.sh
parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 6, column 1

So, I think the reason I never saw this was because I typically run scripts using bash directly.
Regardless, I don't think we need to do anything else here.
Thanks again for the contribution!

That resolves the mystery - I didn't know bash and sh treated escape characters differently. Thanks for reporting back.

authors: adilhafeez suhussai


text:
Improvements/#427

Issue #, if available: https://github.com/aws-samples/awsome-distributed-training/issues/427
Description of changes: This PR has 3 commits, each addressing separate issues.

Commit 92d1b0c to fix incorrect config param in config.py, previously undocumented issue.

Commit abd677e to modify order which Docker / Enroot / Pyxis is called in lifecycle scripts, to mitigate chance of encountering race condition documented in issue 427

Commit 3c9a655 to further address 427 by adding a while loop that will poll (max 120s) dlami-nvme.service for active and execStart messages. This provides assurance that /opt/dlami/nvme is mounted to node prior to executing enroot configuration which will use /opt/dlami/nvme. This commit also updates the order of if/elif statement to first try /opt/dlami/nvme before /opt/sagemaker.

This PR has been tested successfully on a HyperPod cluster with 1 p5 and 4 c5.4xlarge to verify intended outcome and logs analyzed to confirm while loop functioning properly.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

resolved comment from mhugueaws

repo: aws-samples/awsome-distributed-training


text:
SG update to be able to use the template

Opening up the security group to avoid an error in the UI when you try to open the EMR Studio. The SG permission should be updated to the latest definition in the AWS documentation; this is just a patch for the people trying to use this template.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution under the terms of your choice.

Feel free to reject it :) this is just a "patch work" to be able to use the
current template
On Tue, 27 Apr 2021, 02:39 emrnotebooks, @.***> wrote:

@.**** commented on this pull request.
Thanks for submitting this PR. Actually there are more to be updated. Let
us update the service role policy, which should be present in both full and
min dependency YAML files.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/aws-samples/emr-studio-samples/pull/1#pullrequestreview-644892142,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACODEJMFAAKCNKMQHU5KCULTKWJLXANCNFSM43RZAPUA
.

Yep. We have updated the full and min dependency template files. Please let us know if the latest version is working for you.
Again, thanks for bringing this to our attention.
Best regards, Ray
From: Alberto Cubeddu @.>
Reply-To: aws-samples/emr-studio-samples @.>
Date: Monday, April 26, 2021 at 1:43 PM
To: aws-samples/emr-studio-samples @.>
Cc: emrnotebooks @.>, Comment @.***>
Subject: Re: [aws-samples/emr-studio-samples] SG update to be able to use the template (#1)
Feel free to reject it :) this is just a "patch work" to be able to use the
current template
On Tue, 27 Apr 2021, 02:39 emrnotebooks, @.***> wrote:

@.**** commented on this pull request.
Thanks for submitting this PR. Actually there are more to be updated. Let
us update the service role policy, which should be present in both full and
min dependency YAML files.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/aws-samples/emr-studio-samples/pull/1#pullrequestreview-644892142,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACODEJMFAAKCNKMQHU5KCULTKWJLXANCNFSM43RZAPUA
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/aws-samples/emr-studio-samples/pull/1#issuecomment-827133648, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AN4IHLKKWVOZUVJGYB3NUSLTKXF3RANCNFSM43RZAPUA.

You're welcome and thanks for the update
On Tue, 27 Apr 2021, 06:46 emrnotebooks, @.***> wrote:

Yep. We have updated the full and min dependency template files. Please
let us know if the latest version is working for you.
Again, thanks for bringing this to our attention.
Best regards, Ray
From: Alberto Cubeddu @.>
Reply-To: aws-samples/emr-studio-samples @.>
Date: Monday, April 26, 2021 at 1:43 PM
To: aws-samples/emr-studio-samples @.>
Cc: emrnotebooks @.>, Comment @.***>
Subject: Re: [aws-samples/emr-studio-samples] SG update to be able to use
the template (#1)
Feel free to reject it :) this is just a "patch work" to be able to use
the
current template
On Tue, 27 Apr 2021, 02:39 emrnotebooks, @.***> wrote:

@.**** commented on this pull request.
Thanks for submitting this PR. Actually there are more to be updated.
Let
us update the service role policy, which should be present in both full
and
min dependency YAML files.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<
https://github.com/aws-samples/emr-studio-samples/pull/1#pullrequestreview-644892142>,

or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/ACODEJMFAAKCNKMQHU5KCULTKWJLXANCNFSM43RZAPUA>

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<
https://github.com/aws-samples/emr-studio-samples/pull/1#issuecomment-827133648>,
or unsubscribe<
https://github.com/notifications/unsubscribe-auth/AN4IHLKKWVOZUVJGYB3NUSLTKXF3RANCNFSM43RZAPUA>.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/aws-samples/emr-studio-samples/pull/1#issuecomment-827135724,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACODEJN5B3YJU42G6O7MTADTKXGJJANCNFSM43RZAPUA
.

authors: albertocubeddu emrnotebooks

repo: aws-samples/emr-studio-samples


text:
Event Model

Model Description
The event model should provide a way to receive events to they can be sent to an event backbone.
It should also allow the ability to manage subscriptions to different capabilities can subscribe to events of interest.
Required Actions

Receive Event - a way to create a new event.
Create/Update/Delete Subscriptions - a way to manage subscriptions. Use callbacks/webhooks for delivery of matched events.

@gchagnon check out the feature branch called /feat/event-support to see a candidate model and README I added for event support. With this model we would have a dedicated /events endpoint that would be the place events are created. The server for the events model would have a connector just like any other service. The first connector we'd probably create is to Event Bridge. We can also consider using asyncapi to model this, probably requires more discussion.

repo: aws-samples/industry-reference-models


text:
Error with logging permissions - needs redoing with BucketPolicy

When creating the cloudfront with the yaml file it gives error
Bucket cannot have ACLs set with ObjectOwnership's BucketOwnerEnforced setting (Service: Amazon S3; Status Code: 400; Error Code: InvalidBucketAclWithObjectOwnership;

Access logs are now correctly pushed to the S3 logging bucket

Access logs are now correctly pushed to the S3 logging bucket

authors: jeanbaptisteguillois scdba

repo: aws-samples/react-cors-spa


text:
The source code for lambda functions is not present in simple-ssr/edge-build folder

The lambda code referenced in the cdk part doesn't exist in respective folders

This code is generated by webpack during build phase.
You need to build simple-ssr project before deploying it with AWS CDK.
You should run the next commands
cd ../simple-ssr
npm install
npm run build-all
cd ../cdk
cdk deploy SSRAppStack --parameters mySiteBucketName=

This code is generated by webpack during build phase.
You need to build simple-ssr project before deploying it with AWS CDK.
You should run the next commands
cd ../simple-ssr
npm install
npm run build-all
cd ../cdk
cdk deploy SSRAppStack --parameters mySiteBucketName=

repo: aws-samples/react-ssr-lambda


text:
add lora fine tuning

Issue #, if available:
Description of changes:
support safetensor/hf model path/ckpt format
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

It's good for me !


text:
Vue Warning deleting vocabulary: "customVocabularyName" was assigned to but it has no setter.

Transcribe -> Save Vocabulary -> Delete Vocabulary

From stack overflow:
https://stackoverflow.com/questions/46106037/vuex-computed-property-name-was-assigned-to-but-it-has-no-setter

I'm not sure when this was fixed, but it's not occurring anymore so I'm closing the issue.

authors: aburkleaux-amazon ianwow


text:
How to turn off monitoring ?

Hello,
Looking at the doc, I can't find how to turn off monitoring completely. I'd like to turn off cloudwatch + X-ray.
When I get to the the lambda function settings ("monitoring tools" section) I found:

Thanx for your help.

Hi @smknstd
You can try following the answer in this stackoverflow question, to prevent the function from creating logs.
Let me know if you have any issues,
Simon

Thanx for your answer. Will try the workaround even if it seems brutal though (ideally I'd like to keep logging errors I guess).
In the mean time it seems I managed to switch off some logging set up by default from the api gateway:

it seems those stopped immediately:

authors: simonkrol smknstd

repo: aws-solutions/serverless-image-handler


text:
Fix the number of secondary IP (aliases)

Issue #, if available:
None
Description of changes:
When the number of secondary IP is greater than 1, ${aliases} can only get the first element(same as ${aliases[0]}).
By modifying it to ${aliases[@]}, all elements can be traversed.
Changes:
Pathch rewrite_aliases()
${aliases} -> ${aliases[@]}
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Thanks for your accurate diagnosis and fix.
The test suite recently introduced would also have caught this, but didn't actually test >2 (1 primary, 1 secondary) IPv4 addresses on an interface. I will add an additional commit to cover that scenario.


text:
Updating OTA_README.md

Description

Added comments to OTA_README.md regarding supporting Python requirements and compilation flags.
Checklist:

[ ] I have tested my changes. No regression in existing tests.
[ ] My code is Linted.

No code has been changed.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/bot run checks

/bot run checks

repo: aws/amazon-freertos

license: MIT


text:
No "--force" argument to automate delete

Is your feature request related to a problem? Please describe.
Automating dotnet dotnet-aws delete-deployment does not offer an argument to force delete and avoid the interactive confirmation prompt of "Are you sure you want to delete ...?"
Describe the solution you'd like
Add the argument --force and -f to improve automation reliability.
Describe alternatives you've considered
The workaround echo "y" | dotnet dotnet-aws delete-deployment ... is possible, but fragile since the use of piping "y" isn't explicit.
Additional context
Similar to #403

This is a :rocket: feature request

@ericis Thank you for trying out our tooling and providing feedback.
We added the functionality to delete applications without any user prompts as part of https://github.com/aws/aws-dotnet-deploy/pull/445.
Closing this issue as it has been resolved.

repo: aws/aws-dotnet-deploy


text:
feat: Save last used stack and order existing deployments by MRU stack

Issue #, if available:
Description of changes:
feat: Save last used stack and order existing deployments by MRU stack
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

General question: what is the rational behind storing the timestamp in aws-deployments.json file than using LastUpdatedTime to order all the stacks deployed through deploy tool?
This approach will be more accurate in case stack gets updated out of the lifecycle of deploy tool and requires less maintenance on client side.

Another downside of this approach is, given aws-deployments.json file is source controlled, as soon as multiple developers will commit their changes, this logic will not work.
By keeping stack tracking per system (somewhere in temp location) would solve this problem.

authors: ganeshnj philasmar


text:
Revert "Integ-tests: Temporarily disable DCV test."

This reverts commit cc97913e40f30fcd39bec7253c853e977bc123ca.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Closed as duplicate of: https://github.com/aws/aws-parallelcluster/pull/3757

authors: enrico-usai hanwen-pcluste

repo: aws/aws-parallelcluster


text:
g++-11 compile error (Ubuntu 20.04 x86_64)

Confirm by changing [ ] to [x] below to ensure that it's a bug:

[x] I've gone though Developer Guide and API reference
[x] I've searched for previous similar issues and didn't find any solution

Describe the bug
Seeing the following compiler error when trying to build with g++-11
[ 20%] Building C object crt/aws-crt-cpp/crt/s2n/CMakeFiles/s2n.dir/pq-crypto/bike_r1/converts_portable.c.o
/home/ubuntu/Crypto/build/release/externalprojects/awssdk-src/crt/aws-crt-cpp/crt/aws-c-cal/source/unix/openssl_platform_init.c: In function 'aws_cal_platform_init':
/home/ubuntu/Crypto/build/release/externalprojects/awssdk-src/crt/aws-crt-cpp/crt/aws-c-cal/source/unix/openssl_platform_init.c:386:43: error: the comparison will always evaluate as 'false' for the address of 's_locking_fn' will never be NULL [-Werror=address]
  386 |         if (CRYPTO_get_locking_callback() == s_locking_fn) {
      |                                           ^~
/home/ubuntu/Crypto/build/release/externalprojects/awssdk-src/crt/aws-crt-cpp/crt/aws-c-cal/source/unix/openssl_platform_init.c: In function 'aws_cal_platform_clean_up':
/home/ubuntu/Crypto/build/release/externalprojects/awssdk-src/crt/aws-crt-cpp/crt/aws-c-cal/source/unix/openssl_platform_init.c:402:39: error: the comparison will always evaluate as 'false' for the address of 's_locking_fn' will never be NULL [-Werror=address]
  402 |     if (CRYPTO_get_locking_callback() == s_locking_fn) {
      |                                       ^~
/home/ubuntu/Crypto/build/release/externalprojects/awssdk-src/crt/aws-crt-cpp/crt/aws-c-cal/source/unix/openssl_platform_init.c:411:34: error: the comparison will always evaluate as 'false' for the address of 's_id_fn' will never be NULL [-Werror=address]
  411 |     if (CRYPTO_get_id_callback() == s_id_fn) {
      |                                  ^~
cc1: all warnings being treated as errors
make[5]: *** [crt/aws-crt-cpp/crt/aws-c-cal/CMakeFiles/aws-c-cal.dir/build.make:146: crt/aws-crt-cpp/crt/aws-c-cal/CMakeFiles/aws-c-cal.dir/source/unix/openssl_platform_init.c.o] Error 1
make[5]: *** Waiting for unfinished jobs....

SDK version number
1.8.186, 1.9, and the latest
Platform/OS/Hardware/Device
g++-11 Ubuntu 20.04 x86_64
COLLECT_GCC=g++-11
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 11.1.0-1ubuntu1~20.04' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --disable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-11-2V7zgg/gcc-11-11.1.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-2V7zgg/gcc-11-11.1.0/debian/tmp-gcn/usr --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.1.0 (Ubuntu 11.1.0-1ubuntu1~20.04)

To Reproduce (observed behavior)
Ran the build
Expected behavior
A clear and concise description of what you expected to happen.

Hi @bjkaria ,
Thanks for bringing this up to us, we'll take a look.
In the mean time though, if possible I'd suggest downgrading to gcc-10.

I tried gcc-10 too, i get the same error. gcc-9 does work.

https://github.com/awslabs/aws-c-cal/pull/93 fixes the specific warning for GCC11, and a related commit in aws-c-common has adjusted the posture of -Werror and /WX to off by default and can be enabled with an option for all aws-c-* builds.

authors: DavidOgunsAWS KaibaLopez bjkaria

repo: aws/aws-sdk-cpp


text:
Memory leak if s2n_cleanup is not called

Confirm by changing [ ] to [x] below to ensure that it's a bug:

[x] I've gone though Developer Guide and API reference
[x] I've searched for previous similar issues and didn't find any solution

Describe the bug
When upgrading from 1.8.x version to 1.9.49 we see memory leaks reported by the ASAN tool. See ASAN log below.
SDK version number
1.9.49
Platform/OS/Hardware/Device
Linux Debian
To Reproduce (observed behavior)
We create Aws::S3::S3Client objects on several threads. When program is terminated, several leaks are reported.
When we add a call to s2n_cleanup, https://github.com/aws/s2n-tls/blob/main/docs/USAGE-GUIDE.md#s2n_cleanup, during shutdown for each thread created there are no leaks reported.
Expected behavior
No memory leaks.
Do we really need to call s2n_cleanup explicitly just because the implementation creates a UUID?
Logs/output
1
malloc
/usr/local/bin/engine
2
CRYPTO_zalloc
/usr/local/bin/engine
3
s2n_defend_if_forked
/home/vagrant/jws/qix-pc3-build-ws/build/lib/crt/aws-crt-cpp/crt/s2n/utils/s2n_random.c:144:9
4
s2n_get_private_random_data
/home/vagrant/jws/qix-pc3-build-ws/build/lib/crt/aws-crt-cpp/crt/s2n/utils/s2n_random.c:175:5
5
s2n_openssl_compat_rand
/home/vagrant/jws/qix-pc3-build-ws/build/lib/crt/aws-crt-cpp/crt/s2n/utils/s2n_random.c:295:29
6
…::SecureRandomBytes_OpenSSLImpl::GetBytes
/home/vagrant/jws/qix-pc3-build-ws/build/lib/aws-cpp-sdk-core/source/utils/crypto/openssl/CryptoImpl.cpp:142:31
7
…::UUID::RandomUUID
/home/vagrant/jws/qix-pc3-build-ws/build/lib/aws-cpp-sdk-core/source/utils/UUID.cpp:79:27
8
…::STSAssumeRoleWebIdentityCredentialsProvider::STSAssumeRoleWebIdentityCredentialsProvider
/home/vagrant/jws/qix-pc3-build-ws/build/lib/aws-cpp-sdk-core/source/auth/STSCredentialsProvider.cpp:91:25
9
…::__1::__compressed_pair_elem<…>
/usr/lib/llvm-10/bin/../include/c++/v1/memory:2196:46
10
…::__1::__compressed_pair<…>
/usr/lib/llvm-10/bin/../include/c++/v1/memory:2289:42
11
…::__1::__shared_ptr_emplace<…>
/usr/lib/llvm-10/bin/../include/c++/v1/memory:3562:12
12
…::__1::shared_ptr<…>
/usr/lib/llvm-10/bin/../include/c++/v1/memory:4195:9
13
…::__1::enable_if<…>
/usr/lib/llvm-10/bin/../include/c++/v1/memory:4415:12
14
…::__1::shared_ptr<…>
/home/vagrant/jws/qix-pc3-build-ws/build/lib/aws-cpp-sdk-core/include/aws/core/utils/memory/stl/AWSAllocator.h:51:16
15
…::DefaultAWSCredentialsProviderChain::DefaultAWSCredentialsProviderChain
/home/vagrant/jws/qix-pc3-build-ws/build/lib/aws-cpp-sdk-core/source/auth/AWSCredentialsProviderChain.cpp:41:17
16
…::__1::__compressed_pair_elem<…>
/usr/lib/llvm-10/bin/../include/c++/v1/memory:2196:46
17
…::__1::__compressed_pair<…>
/usr/lib/llvm-10/bin/../include/c++/v1/memory:2289:42
18
…::__1::__shared_ptr_emplace<…>
/usr/lib/llvm-10/bin/../include/c++/v1/memory:3562:12
19
…::__1::shared_ptr<…>
/usr/lib/llvm-10/bin/../include/c++/v1/memory:4195:9
20
…::__1::enable_if<…>
/usr/lib/llvm-10/bin/../include/c++/v1/memory:4415:12
21
…::__1::shared_ptr<…>
/home/vagrant/jws/engine-common-ws/build/HEAD-release/../../Packages/aws-sdk-cpp/include/aws/core/utils/memory/stl/AWSAllocator.h:51:16
22
GetCredentialProvider
/home/vagrant/jws/engine-common-ws/build/HEAD-release/../../src/qaws/src/S3Client.cpp:115:12
23
…::S3Client::S3Client
/home/vagrant/jws/engine-common-ws/build/HEAD-release/../../src/qaws/src/S3Client.cpp:131:9
24
…::__1::__compressed_pair_elem<…>
/usr/lib/llvm-10/bin/../include/c++/v1/memory:2196:46
25
…::__1::__compressed_pair<…>
/usr/lib/llvm-10/bin/../include/c++/v1/memory:2289:42
26
…::__1::__shared_ptr_emplace<…>
/usr/lib/llvm-10/bin/../include/c++/v1/memory:3562:12
27
…::__1::enable_if<…>
/usr/lib/llvm-10/bin/../include/c++/v1/memory:4400:26
28
…::S3ObjectStream::S3ObjectStream
The first frame is our own:
    Client = Aws::MakeShared<Aws::S3::S3Client>(
        "S3Client",
        Aws::MakeShared<Aws::Auth::DefaultAWSCredentialsProviderChain>("DefaultAWSCredentialsProviderChain"),
        Config,
        Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::RequestDependent,
        false
        );

Additional context
Memory leaks were not reported on several 1.8.x versions we have used.

Could you try this workaround (not the fix) to suppress a specific leak in S2N by:
export LSAN_OPTIONS=suppressions=scripts/suppressions.txt

exporting the above (or should I have the file present with some content?) doesn't change a bit.
==261254== 6,336 bytes in 24 blocks are indirectly lost in loss record 6 of 6
==261254==    at 0x4842839: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==261254==    by 0x4A726DD: CRYPTO_zalloc (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1)
==261254==    by 0x4A63754: EVP_CipherInit_ex (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1)
==261254==    by 0x50B9251: s2n_drbg_instantiate (aws-sdk-cpp/crt/aws-crt-cpp/crt/s2n/crypto/s2n_drbg.c:162)
==261254==    by 0x50AE15D: s2n_defend_if_forked (aws-sdk-cpp/crt/aws-crt-cpp/crt/s2n/utils/s2n_random.c:143)
==261254==    by 0x50AE210: s2n_get_private_random_data (aws-sdk-cpp/crt/aws-crt-cpp/crt/s2n/utils/s2n_random.c:175)
==261254==    by 0x50AE544: s2n_openssl_compat_rand (aws-sdk-cpp/crt/aws-crt-cpp/crt/s2n/utils/s2n_random.c:295)
==261254==    by 0x4F9DF69: Aws::Utils::Crypto::SecureRandomBytes_OpenSSLImpl::GetBytes(unsigned char*, unsigned long) (aws-sdk-cpp/aws-cpp-sdk-core/source/utils/crypto/openssl/CryptoImpl.cpp:142)
==261254==    by 0x4F88337: Aws::Utils::UUID::RandomUUID() (aws-sdk-cpp/aws-cpp-sdk-core/source/utils/UUID.cpp:79)

and
==261254== 1,560 bytes in 20 blocks are still reachable in loss record 2 of 6
==261254==    at 0x4842839: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==261254==    by 0x50AD85C: s2n_mem_malloc_no_mlock_impl (aws-sdk-cpp/crt/aws-crt-cpp/crt/s2n/utils/s2n_mem.c:126)
==261254==    by 0x50ACC25: s2n_realloc (aws-sdk-cpp/crt/aws-crt-cpp/crt/s2n/utils/s2n_mem.c:195)
==261254==    by 0x50AC9B2: s2n_alloc (aws-sdk-cpp/crt/aws-crt-cpp/crt/s2n/utils/s2n_mem.c:157)
==261254==    by 0x50AD3D5: s2n_dup (aws-sdk-cpp/crt/aws-crt-cpp/crt/s2n/utils/s2n_mem.c:240)
==261254==    by 0x5104172: s2n_cipher_suites_init (aws-sdk-cpp/crt/aws-crt-cpp/crt/s2n/tls/s2n_cipher_suites.c:1029)
==261254==    by 0x50AB632: s2n_init (aws-sdk-cpp/crt/aws-crt-cpp/crt/s2n/utils/s2n_init.c:44)
==261254==    by 0x505C657: aws_tls_init_static_state (aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-io/source/s2n/s2n_tls_channel_handler.c:132)
==261254==    by 0x5050CF2: aws_io_library_init (aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-io/source/io.c:213)
==261254==    by 0x4FDAF58: aws_mqtt_library_init (aws-sdk-cpp/crt/aws-crt-cpp/crt/aws-c-mqtt/source/mqtt.c:186)
==261254==    by 0x4FBB422: Aws::Crt::s_initApi(aws_allocator*) (aws-sdk-cpp/crt/aws-crt-cpp/source/Api.cpp:39)
==261254==    by 0x4FBB3E1: Aws::Crt::ApiHandle::ApiHandle(aws_allocator*) (aws-sdk-cpp/crt/aws-crt-cpp/source/Api.cpp:52)

Kind Reminder of this issue.
Aren't there any other people out there experience this issue?

Kind monthly reminder of this issue :)

We are facing the same issue. We make a lot of s3-crt GetObjectAsync calls, each one of which spins up a new thread (as we are using the DefaultExecutor) and adds to the memory usage.
@theShmoo how exactly did you get the export LSAN_OPTIONS hack working? It isn't helping me.

Hoi! Nice to hear that we are not alone!
We use LSAN as part of ASAN and if you use a suppression list you need to configure ASAN to recover from failures:
-fsanitize-recover=address -fsanitize-address-use-after-scope
these are the flags we use.

Certainly you are not alone. Suppressing the warnings does not feel so right :| We end up using thread_local to call s2n_cleanup(); whenever a thread is terminated. Now valgrind does not complain any more. We are happy again.
File: S2nCleanup.h
#include <s2n.h>

class S2nCleanup {
public:
    ~S2nCleanup() {
        s2n_cleanup();
    }
};

Any where we use s3Client.
thread_local S2nCleanup s2nCleanup{};

...

Aws::S3::Model::GetObjectOutcome outcome = s3Client->GetObject(request);

...

Nice! Thanks for your tipp with the thread_local variable. This also works for my use case.
Maybe it will help others as well:
I use s3 in combination with https://github.com/yhirose/cpp-httplib
to get your fix working I need to create a custom thread pool and add the thread_local s2n clean up:
proof of concept:
struct thread_cleanup
{
  ~thread_cleanup()
  {
    s2n_cleanup();
  }
};

class task_queue : public httplib::TaskQueue
{
 public:
  explicit task_queue(size_t n) : m_pool(n) {}

  void enqueue(std::function<void()> fn) override
  {
    m_pool.enqueue(
        [f = std::move(fn)]
        {
          f();
          thread_local thread_cleanup cleanup{};
        });
  }

  void shutdown() override
  {
    m_pool.shutdown();
  }

 private:
  httplib::ThreadPool m_pool;
};

// when configuring the server:

m_server->new_task_queue = []
{
  // NOLINTNEXTLINE
  return new task_queue(12);
};

This PR fixed a memory leak. Please let me know if you are still seeing any more leaks with the current version of this sdk.

authors: databucketio dhananjays jmklix kwach magnushakansson theShmoo wps132230


text:
Experiencing memory leaks when using RDS Token assembly

When the application generates an RDS authentication token for use with connecting to an RDS database to generate the password, we are experiencing memory leaks in our application. We are using the Assembly AWSSDK.RDS, Version=3.3.0.0 for now and not the latest version due to package dependencies on other projects. To prove that, So we tried switching the GenerateAuthToken off and used the static password and deployed the change, i could see there is a slow decline but much slower than it was earlier which is in measurable scale. So i believe that the RDP token code is the vast majority of the problem it seems. It is just a toggle option which we switch off it works fine and we don't experience any memory issues. Attaching the metrics from the graph to look at the difference when the change for static password was made and deployed. Please let me know in case you need any more information from my side to look into this issue. Looking forward for the support.
This is the toggle option code -
 if (applicationDbContextPostgreSqlSettings.GenerateAuthToken.HasValue && applicationDbContextPostgreSqlSettings.GenerateAuthToken.Value)
            {
                builder.Password = RDSAuthTokenGenerator.GenerateAuthToken(
                   applicationDbContextPostgreSqlSettings.RegionEndpoint,
                    applicationDbContextPostgreSqlSettings.Host,
                    applicationDbContextPostgreSqlSettings.Port.Value,
                    applicationDbContextPostgreSqlSettings.Username);
            }
            else
            {
                SecureString password = applicationDbContextPostgreSqlSettings.Password;

                password.Decrypt(plainText =>
                {
                    builder.Password = plainText;
                });
            }

Hi @archanasharma3,
Good afternoon.
Could you please share the sample application so that we could reproduce the issue? Above code snippet doesn't provide enough data points that support memory leak because of call to RDSAuthTokenGenerator.GenerateAuthToken(). To troubleshoot further, we need to investigate application code and see if it's trying to generate tokens continuously.
Thanks,
Ashish

Hello, as requested attaching the startup, and dbcontext context related files. Please let me know if any other files are needed for investigation.
ApplicationDbContext.txt
Startup.txt
ApplicationDbContextPostgreSql.txt
.

Hi @archanasharma3,
Good morning.
Unfortunately, I do not see any logic in RDSAuthTokenGenerator that might be causing the memory leaks. I examined your Startup.txt and here are my observations:

You are using services.AddScoped<IApplicationDbContext, ApplicationDbContextPostgreSql>();. Per Microsoft's official documentation Dependency injection in ASP.NET Core, the AddScoped method registers the service with a scoped lifetime, the lifetime of a single request. This means for every request from any user would have a new ApplicationDbContextPostgreSql instance, which could end up calling Amazon.RDS.Util.RDSAuthTokenGenerator.GenerateAuthToken() every time. One optimization that could be done here is to cache generated token (however, you would need to add logic to refresh the expired tokens).
I do notice that many services are registered at application startup. You might want to revisit the services to see if there could be any memory leaks there.

Could you try developing sample application with minimal logic which invokes Amazon.RDS.Util.RDSAuthTokenGenerator.GenerateAuthToken() first with AddScoped() and then with AddSingleton()? Also try upgrading to latest AWSDK version, any identified fixes would be pushed on top of latest version, if any.
Thanks,
Ashish

repo: aws/aws-sdk-net


text:
V4 Development: General strategy on how to fill in missing APIs

Describe the feature
I'm currently fiddling around with some performance improvements and for some cases I need to somehow polyfill certain missing APIs or attributes. For example in order to implement overrenting when doing stack allocations it might be beneficial to add SkipsLocalsInit. Another example are missing encoding extension methods that accept ReadOnlySpan.
Several of these cases can be poly filled by adding corresponding attributes or methods into the utils and that might be the teams preferred approach. On the other hand there are libraries like Polyfil or Polysharp that to some extend implement those workarounds already.
In order to proceed forward it would be great if the strategy for the v4 development branch would be documented somewhere. Additionally the team could potentially also set the LangVersion to a reasonable language version that the team wants to allow so that contributors have clear guidelines on what or what not to use.
Use Case
Contributing to v4 development but I guess these questions are anyway something the team on the v4 branch has to have answers for independent of whether someone contributes or not.
Proposed Solution
No response
Other Information
No response
Acknowledgements

[ ] I may be able to implement this feature request
[ ] This feature might incur a breaking change

AWS .NET SDK and/or Package version used
v4
Targeted .NET Platform
all
Operating System and version
all

To add an example. Here is a new URL encode method I quickly spiked https://github.com/danielmarbach/aws-sdk-net/commit/81d85869f3f096d522966b7c93a1cf862c74cb27. Already there I'm missing proper GetBytes and GetString overloads for Netstandard that I would have to shim. I don't have SkipLocalsInit and I have discrepancy between TryGetValue and GetValueOrDefault.

@danielmarbach Agreed we need to get this documented. We haven't really talked too much about V4 yet other then the blog post announcing the drop in support for .NET Framework 3.5. Not trying to keep it a secret but we are taking care of some internal infrastructure changes right now to support both a V3 and V4. After that I was hoping to be more proactively public about V4. That being said don't let that stop you from contributing to V4. The main thing about V4 is it will be evolutionary not revolutionary so only breaking changes when necessary to make a better product.
To your immediate question about poly fills the SDK is still going to take a hard rule of avoiding taking on external dependencies. The SDK is too low level in most people's applications and dependencies we take can cause collection with versions customers use or we could be forced to make breaking changes due to the dependencies we don't control.
Using pragmas to target newer versions and leaving older versions using stock implementations is perfectly fine in the SDK. I'm not very interested in having pragmas with special implementations in older targets. I don't want to build up code we have to maintain for targets people are moving away from.

@normj

To your immediate question about poly fills the SDK is still going to take a hard rule of avoiding taking on external dependencies. The SDK is too low level in most people's applications and dependencies we take can cause collection with versions customers use or we could be forced to make breaking changes due to the dependencies we don't control.

Both Polyfil and Polyshard are source-only dependencies, meaning that customers wouldn't even know that SDK uses them.

Absolutely what @Dreamescaper said. Essentially those are development time dependencies and will not be dependencies on the shipped product at all. So the risk here is probably more of a supply chain risk than anything else (which obviously requires considerations I get that)

I didn't realize the libraries were source-only dependencies. As you can call I haven't ever tried them before. Let me play around with them to get more familiar with them and then I will come back.

authors: Dreamescaper danielmarbach normj


text:
feat: Add support for encoded data to Value

Description of changes:
Add getEncodedBytes() to Value and all its child classes so values can be recreated after encryption.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Commits are showing up wrong, closing PR.


text:
fake change. DO NOT MERGE

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/hold

repo: aws/eks-anywhere-build-tooling


text:
Fix prometheus/prometheus version in the README

Description of changes:
#3683 updated the prometheus/prometheus version in the README which caused the validate-generated-presubmit prow job to fail on the open PRs like #3694 and #3706 with the following error:
Error: Generated files, UPSTREAM_PROJECTS.yaml README.md release/staging-build.yml release/checksums-build.yml batch-build.yml, do not match expected. Please run 'make generate' to update
diff --git a/projects/prometheus/prometheus/README.md b/projects/prometheus/prometheus/README.md
index cf45a32e..dadbcd2d 100644
--- a/projects/prometheus/prometheus/README.md
+++ b/projects/prometheus/prometheus/README.md
@@ -1,4 +1,4 @@
-![Version](https://img.shields.io/badge/version-v2.54.0-blue)
+![Version](https://img.shields.io/badge/version-v2.53.1-blue)
 ![Build Status](https://codebuild.us-west-2.amazonaws.com/badges?uuid=eyJlbmNyeXB0ZWREYXRhIjoiTldscmdZWkd6NzlhUHJBbFJDRzlMc3NmaGxBOFJlYWE1a3BsVG9KcXhldDRCK05PL0lxNmVVUi9odlMzdXZCYXFxWTBCOUZDbS91R21KL1c5VkdQQ004PSIsIml2UGFyYW1ldGVyU3BlYyI6Im94dGM3UFc0MGRDN0pyREIiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D&branch=main)

 ## **Prometheus**
make: *** [validate-generated] Error 1

This PR reverts that change to match the expected version in the GIT_TAG file.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve

/lgtm


text:
Add validation for management cluster referenced in workload cluster spec to be valid

Issue #, if available:
Description of changes:
Added additional validations to ensure that we are able to retrieve the EKS-A cluster object from the management cluster and also confirm that the cluster we have referenced is indeed a management cluster by checking if the management cluster name for that cluster references itself.
Testing (if applicable):
unit tests and testing these scenarios functionally
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve

repo: aws/eks-anywhere


text:
Release EKS Go: v1.20.14-15

Increment release file to publish new EKS Go artifacts for v1.20.14-15

/lgtm
/approve

repo: aws/eks-distro-build-tooling


text:
feat(go) - research using generics for improved UX

:rocket: Feature Request
Affected Languages

[ ] TypeScript or Javascript
[ ] Python
[ ] Java
[ ] .NET (C#, F#, ...)
[x] Go

General Information

JSII Version: 1.49.0<!-- Output of jsii --version -->

Platform: *

[x] This feature might incur a breaking change

Description
Go 1.18 is scheduled to release as stable in Feb 2022 with support for generic types. This new feature allows us to potentially take an alternative approach to supporting optional values, enabling type casting, and various runtime improvements.
Proposed Solution
Optional Types
Definition of some wrapper type like:
type Optional[T any] struct{
	val T
	defined bool
}

Would allow code generating Optional[Type] instead of *Type everywhere. This largely would have similar semantics to the user as all types, regardless of whether they are optional or required in typescript, would have to be this wrapper type in order to uphold backwards compatibility guarantees as detailed in the RFC. It may however emerge as the more idiomatic solution once generics are commonly used in Go, and they can be defined inline instead of using a utility function like jsii.String or needing to allocate a variable previously.
Additionally this could simplify reflection code within the Go runtime where currently some relatively complicated logic exists around inspecting and dereferencing pointers during reflection.
Type Casting
We may be able to define a function, returning a generic type representing a parent or child class (in typescript) to support downcasting as needed for escape hatches.
Runtime Improvements
Changing JSII runtime functionality to use generics:
// from
func (c *Client) Get(props GetProps) (response GetResponse, err error)
// to
func (c *Client) Get[A any](props GetProps) (response GetResponse, err error, val A)

Could potentially give stronger compile time guarantees for the runtime library and generated code and reduce the usage of unsafe pointer manipulation.

I extensively prototyped in the https://github.com/aws/jsii/tree/rmuller/explore/go1.18-generics branch.
Here are some of the high-level findings (pending a more comprehensive writeup):

Methods cannot introduce new type arguments (they may of course use the type parameters of the type they are a member of)
We can design a "marker" interface that can be used in any place where a value is optional (nil-able)type Option[T any] interface {
  // This is merely a marker function to facilitate
  // getting the underlying value type through reflection
  Unwrap__() T
}

All generated types simply implement this interface (T is the implementing type)

This means a signature that accepts any T can be updated to accept Optional[T] without breaking
Unfortunately any given type can only implement the marker ONCE (as it'd otherwise require multiple identically-named methods to be implemented, which is impossible)

Structs (in the jsii sense) and enums are passed by-value when required, so they simply cannot be nil.
We still need to use wrappers for primitive types (string, float64, etc...), but instead of being pointers to these types, they can be "real" aliasestype String string
// String is a valid value for Option[String]
func (s String) Unwrap__() String { return s }

These cannot be nil when required.
They implement Option[T]
These types have perhaps a more "go native" feel to them.

The marker allows runtime validation that structs have values for all required (and nil-able) fields they have

We can streamline the runtime APIs that currently accept reflect.Type so they are instead generic functions// Before
RegisterStruct(reflect.TypeOf((*StructType)(nil).Elem())

// After:
RegisterStruct[StructType]()

We can implement a decent UnsafeCast feature using generics, where the experience could look like so:original := SomeFunction() // A union of structs
downCasted := jsii.UnsafeCast[SpecificStruct](original)

These definitely look like some significant improvements that at first glance appear to make the Go experience easier to get started with and understand. We should work towards providing a working experience for users to try and provide feedback on though if possible without rewriting too much of our current implementation, if only to provide further validation without committing to an entirely new approach.

Additional notes in https://github.com/aws/aws-cdk-rfcs/pull/397

authors: MrArnoldPalmer RomainMuller

repo: aws/jsii


text:
FileNotFoundException: Could not load file or assembly ...

Hi,
I get the error "System.IO.FileNotFoundException: Could not load file or assembly" during the assessment, but Visual Studio's build is successful and the files exist in related folders. I check and see related files when starting a build by Visual Studio 2022, but there are no files in the folder when starting an assessment by Porting Assistant.
This solution file contains the Silverlight and Asp.Net Web Application projects. Is Porting Assistant for .Net support?
Log;
[2023-03-14 15:41:07 ERR] (1.9.2) PortingAssistant.Client.Analysis.PortingAssistantAnalysisHandler: Error while resolving reference Microsoft.CodeAnalysis.MetadataImageReference System.IO.FileNotFoundException: Could not load file or assembly '...\bin\Debug\XYZ.dll'. The system cannot find the file specified. File name: '...\bin\Debug\XYZ.dll'
at System.Reflection.AssemblyName.nGetFileInformation(String s)
at System.Reflection.AssemblyName.GetAssemblyName(String assemblyFile)
at Codelyzer.Analysis.Build.ExternalReferenceLoader.LoadFromCompilation(HashSet`1 projectReferenceNames)

Thank you for opening this issue! We've opened a ticket to track and investigate this issue internally.

authors: mrkdeng orcunhanay

repo: aws/porting-assistant-dotnet-ui


text:
Adds proof harness for aws_nospec_mask function

Signed-off-by: Felipe R. Monteiro felisous@amazon.com
Description of changes:

Updates implementation of aws_nospec_mask function with pre- and post-conditions;
Adds a proof harness for aws_nospec_mask function;

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@karkhaz @danielsn @Baha @dlohar @mmuesly any comments?

repo: awslabs/aws-c-common


text:
"cdk diff" returns error code 1

Calling cdk difffrom NPM fails because the command does not return 0.
package.json
  "scripts": {
    "diff": "cdk diff"
  }

Running npm run diff fails.
cdk diff; echo $?shows the error code returned is 1.

A workaround to get rid of the `npm ERR! ...:
package.json
  "scripts": {
    "diff": "cdk diff || true"
  }

Is this enough of an issue to document? Or are we going to try to make it return 0?

It returns 0 if there are no differences as documented here:
https://github.com/awslabs/aws-cdk/blob/7001f7723cde611f0d8cd9c182f92120b09579c3/packages/aws-cdk/bin/cdk.ts#L69
and here https://docs.aws.amazon.com/CDK/latest/userguide/tools.html

As much as this is a nice a pure way to express that "there is a diff". I doesn't seem like that's what people expect. This is the 3rd or 4th time someone fell into this little pit. I'd argue that we can add a flag --fail for those brave souls who are interested to use cdk diff to check if there is a diff and change the default behavior to what people most expect.

For scripts which rely on the error code from cdk diff, there is no way to know if this is due to diffs or some other error (e.g. invalid python code). The flag seems quite appropriate. This is how git does it as well.
git diff --name-only
echo $? # returns 0

git diff --name-only --exit-code
echo $? # returns 1 if there are diffs

Any updates?

Any updates?

Will open a PR for this.

When running through a pipeline (i.e. Jenkins), I'd actually expect a different result. If there's a diff that's cool and I want to move on to the next stage of the pipeline and deploy it, so an exit code of 1 is actually not good here. If there's an ERROR, like someone committed bad code to the repo which causes cdk diff to actually throw an error, like invalid syntax or anything that would cause it to NOT deploy... That seems like an error code of 1 or 2 or something else, no?
Expected results would be:
exit 0 -- No diff and the expected pre-compilation or whatever didn't find any errors in synthesizing.
exit 1 -- You have an error, I could not synth due to any number of reasons, invalid syntax, etc etc.
exit 2 -- Synth worked, code looks clean, and there is a diff that I would apply if you were to deploy.
With those, I can create really functional pipeline steps. With exit 0, I can skip the deploy stage entirely because there's no need. With exit 1, I can shoot an email to the Dev who last committed to the repo and broke the pipeline with bad syntax or something.  With exit 2, I know something changed and I move on to apply it.
With either a blanket return of 0, or forcing me now to use "|| true" ... I actually get nothing. I end up having to create an Approval stage where the pipeline sits until someone looks at the output of the Diff to see if it was a real change or if the CDK is now broken and wouldn't deploy if we wanted it to.
Suggestions / Thoughts?

When running through a pipeline (i.e. Jenkins), I'd actually expect a different result. If there's a diff that's cool and I want to move on to the next stage of the pipeline and deploy it, so an exit code of 1 is actually not good here. If there's an ERROR, like someone committed bad code to the repo which causes cdk diff to actually throw an error, like invalid syntax or anything that would cause it to NOT deploy... That seems like an error code of 1 or 2 or something else, no?
Expected results would be:
exit 0 -- No diff and the expected pre-compilation or whatever didn't find any errors in synthesizing.
exit 1 -- You have an error, I could not synth due to any number of reasons, invalid syntax, etc etc.
exit 2 -- Synth worked, code looks clean, and there is a diff that I would apply if you were to deploy.
With those, I can create really functional pipeline steps. With exit 0, I can skip the deploy stage entirely because there's no need. With exit 1, I can shoot an email to the Dev who last committed to the repo and broke the pipeline with bad syntax or something.  With exit 2, I know something changed and I move on to apply it.
With either a blanket return of 0, or forcing me now to use "|| true" ... I actually get nothing. I end up having to create an Approval stage where the pipeline sits until someone looks at the output of the Diff to see if it was a real change or if the CDK is now broken and wouldn't deploy if we wanted it to.
Suggestions / Thoughts?

@timwelch what if there was no CloudFormation diff, but you updated the CloudFormation tags? I think that's the only scenario I see that isn't covered in what you mentioned.

@timwelch what if there was no CloudFormation diff, but you updated the CloudFormation tags? I think that's the only scenario I see that isn't covered in what you mentioned.

I didn't know the tags was an issue, but I also don't know the full inner workings of cdk and/or cloudformation. It actually sounds like another instance CDK / CFN not showing any changes are needed.
A key example I have of that is in an ASG, when you modify the update_policy....
         self.asg = autoscaling.AutoScalingGroup(             self,             rekor.id_str(self.app_name+'-ASG'),             role=self.role,             vpc=self.vpc,             instance_type=ec2.InstanceType(self.env_data['instance_type']),             machine_image=self.ami,             vpc_subnets=self.private_subnets,             # desired_capacity=0,             max_capacity=2,             # min_capacity=0,             user_data=self.userdata,             key_name=self.keypair.name,             associate_public_ip_address=False,             security_group=self.sg,             cooldown=core.Duration.seconds(180),             ignore_unmodified_size_properties=True,             notifications=None if not self.notifications_sns_topic else [                 autoscaling.NotificationConfiguration(                     topic=self.notifications_sns_topic,                     scaling_events=autoscaling.ScalingEvents.ALL)],             update_policy=autoscaling.UpdatePolicy.rolling_update(                 max_batch_size=1,                 min_instances_in_service=1,                 suspend_processes=[                     autoscaling.ScalingProcess.HEALTH_CHECK,                     autoscaling.ScalingProcess.REPLACE_UNHEALTHY,                     autoscaling.ScalingProcess.AZ_REBALANCE,                     autoscaling.ScalingProcess.ALARM_NOTIFICATION,                     autoscaling.ScalingProcess.SCHEDULED_ACTIONS                 ]             )         )
You can modify the update policy there ALL day long and the CDK will show NO diff. It's only when you add a useless Output ...
       core.CfnOutput(self, "force_asg_update_policy", value="true")
That the CDK diff will show a change (the output) and then you can deploy the CDK and it will actually update the stack. IF you DO NOT create/update an Output, or any other resource, the CDK DEPLOY will just say there's nothing to deploy and NOT deploy anything, even though the update_policy was modified...
So, my point is... if Tags work that way, then it's not a one-off case, and just getting some kind of return codes like I mentioned could get us further in a direction of happiness for pipelines, while working to come up with solutions for these other random cases.

I didn't know the tags was an issue, but I also don't know the full inner workings of cdk and/or cloudformation. It actually sounds like another instance CDK / CFN not showing any changes are needed.
A key example I have of that is in an ASG, when you modify the update_policy....
         self.asg = autoscaling.AutoScalingGroup(             self,             rekor.id_str(self.app_name+'-ASG'),             role=self.role,             vpc=self.vpc,             instance_type=ec2.InstanceType(self.env_data['instance_type']),             machine_image=self.ami,             vpc_subnets=self.private_subnets,             # desired_capacity=0,             max_capacity=2,             # min_capacity=0,             user_data=self.userdata,             key_name=self.keypair.name,             associate_public_ip_address=False,             security_group=self.sg,             cooldown=core.Duration.seconds(180),             ignore_unmodified_size_properties=True,             notifications=None if not self.notifications_sns_topic else [                 autoscaling.NotificationConfiguration(                     topic=self.notifications_sns_topic,                     scaling_events=autoscaling.ScalingEvents.ALL)],             update_policy=autoscaling.UpdatePolicy.rolling_update(                 max_batch_size=1,                 min_instances_in_service=1,                 suspend_processes=[                     autoscaling.ScalingProcess.HEALTH_CHECK,                     autoscaling.ScalingProcess.REPLACE_UNHEALTHY,                     autoscaling.ScalingProcess.AZ_REBALANCE,                     autoscaling.ScalingProcess.ALARM_NOTIFICATION,                     autoscaling.ScalingProcess.SCHEDULED_ACTIONS                 ]             )         )
You can modify the update policy there ALL day long and the CDK will show NO diff. It's only when you add a useless Output ...
       core.CfnOutput(self, "force_asg_update_policy", value="true")
That the CDK diff will show a change (the output) and then you can deploy the CDK and it will actually update the stack. IF you DO NOT create/update an Output, or any other resource, the CDK DEPLOY will just say there's nothing to deploy and NOT deploy anything, even though the update_policy was modified...
So, my point is... if Tags work that way, then it's not a one-off case, and just getting some kind of return codes like I mentioned could get us further in a direction of happiness for pipelines, while working to come up with solutions for these other random cases.

I would be curious if there could be a use-case for pulling down the current CFN Template and diff it with the synthesized CFN? ... I can clearly see the update_policy / rolling_update changes in the cdk.out folder where it synthesizes what would be pushed. So there could be a comparison and even an error or warning that ... hey, though CDK diff says nothing changed... there's actually a discrepancy between the current / live CFN and what we would be pushing in place.

I would be curious if there could be a use-case for pulling down the current CFN Template and diff it with the synthesized CFN? ... I can clearly see the update_policy / rolling_update changes in the cdk.out folder where it synthesizes what would be pushed. So there could be a comparison and even an error or warning that ... hey, though CDK diff says nothing changed... there's actually a discrepancy between the current / live CFN and what we would be pushing in place.

When running through a pipeline (i.e. Jenkins), I'd actually expect a different result. If there's a diff that's cool and I want to move on to the next stage of the pipeline and deploy it, so an exit code of 1 is actually not good here. If there's an ERROR, like someone committed bad code to the repo which causes cdk diff to actually throw an error, like invalid syntax or anything that would cause it to NOT deploy... That seems like an error code of 1 or 2 or something else, no?
Expected results would be:
exit 0 -- No diff and the expected pre-compilation or whatever didn't find any errors in synthesizing.
exit 1 -- You have an error, I could not synth due to any number of reasons, invalid syntax, etc etc.
exit 2 -- Synth worked, code looks clean, and there is a diff that I would apply if you were to deploy.
With those, I can create really functional pipeline steps. With exit 0, I can skip the deploy stage entirely because there's no need. With exit 1, I can shoot an email to the Dev who last committed to the repo and broke the pipeline with bad syntax or something. With exit 2, I know something changed and I move on to apply it.
With either a blanket return of 0, or forcing me now to use "|| true" ... I actually get nothing. I end up having to create an Approval stage where the pipeline sits until someone looks at the output of the Diff to see if it was a real change or if the CDK is now broken and wouldn't deploy if we wanted it to.
Suggestions / Thoughts?

Running into a similar scenario with building our pipelines. I really like your suggestions.

When running through a pipeline (i.e. Jenkins), I'd actually expect a different result. If there's a diff that's cool and I want to move on to the next stage of the pipeline and deploy it, so an exit code of 1 is actually not good here. If there's an ERROR, like someone committed bad code to the repo which causes cdk diff to actually throw an error, like invalid syntax or anything that would cause it to NOT deploy... That seems like an error code of 1 or 2 or something else, no?
Expected results would be:
exit 0 -- No diff and the expected pre-compilation or whatever didn't find any errors in synthesizing.
exit 1 -- You have an error, I could not synth due to any number of reasons, invalid syntax, etc etc.
exit 2 -- Synth worked, code looks clean, and there is a diff that I would apply if you were to deploy.
With those, I can create really functional pipeline steps. With exit 0, I can skip the deploy stage entirely because there's no need. With exit 1, I can shoot an email to the Dev who last committed to the repo and broke the pipeline with bad syntax or something. With exit 2, I know something changed and I move on to apply it.
With either a blanket return of 0, or forcing me now to use "|| true" ... I actually get nothing. I end up having to create an Approval stage where the pipeline sits until someone looks at the output of the Diff to see if it was a real change or if the CDK is now broken and wouldn't deploy if we wanted it to.
Suggestions / Thoughts?

Running into a similar scenario with building our pipelines. I really like your suggestions.

authors: Doug-AWS abelmokadem cal5barton eladb isubuz jogold timwelch xapou

repo: awslabs/aws-cdk


text:
feat(VpcNetwork): support subnet groups in subnetConfiguration

**Closes #2087
Adds functionality to add subnet groups which reserve a block of IP space within a VPC.
Pull Request Checklist

[x] Testing

Unit test added (prefer not to modify an existing test, otherwise, it's probably a breaking change)
__CLI change?: coordinate update of integration tests with team
__cdk-init template change?: coordinated update of integration tests with team

[x] Docs

jsdocs: All public APIs documented
README: README and/or documentation topic updated

[x] Title and Description

Change type: title prefixed with fix, feat will appear in changelog
Title: use lower-case and doesn't end with a period
Breaking?: last paragraph: "BREAKING CHANGE: <describe what changed + link for details>"
Issues: Indicate issues fixed via: "Fixes #xxx" or "Closes #xxx"

[x] Sensitive Modules (requires 2 PR approvers)

IAM Policy Document (in @aws-cdk/aws-iam)
EC2 Security Groups and ACLs (in @aws-cdk/aws-ec2)
Grant APIs (only if not based on official documentation with a reference)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license.

I think I'm in favor of a simpler solution, see the ticket for further discussion.


text:
New rule on SAGEMAKER_TRAINING_JOB_NETWORK_ISOLATION_ENABLED

New rule to verify that the Amazon SageMaker training jobs have Network Isolation enabled.
Please provide comments on the gherkin if any.

Description:
  Check whether Network Isolation is enabled for all Amazon SageMaker Training Jobs.

Trigger:
  Periodic

Reports on:
  AWS::::Account

Scenarios:
  Scenario: 1
     Given: No Amazon SageMaker training jobs exist
      Then: Return NOT_APPLICABLE
  Scenario: 2
     Given: At least one Amazon SageMaker training job exists
       And: EnableNetworkIsolation is set to False for atleast one Amazon SageMaker training job
      Then: Return NON_COMPLIANT with annotation "Network isolation is not enabled for the Amazon SageMaker Training Job(s): <Training job names>."
  Scenario: 3
     Given: At least one Amazon SageMaker training job exists
       And: EnableNetworkIsolation is set to True for all Amazon SageMaker training jobs
      Then: Return COMPLIANT

Just added a space of "At least". Good to go!

authors: jongogogo kritijha

repo: awslabs/aws-config-rules