text:
Multi account setup with only one IAM role in k8s account

What would you like to be added:
The current situation is:

shared K8S cluster in one AWS account
Team A's pods need to access only team A's AWS account
Team B's pods need to access only team B's AWS account

The possibility now:

create IAM role in K8S account for each team account
create IAM role in each team account trusting the k8s account role

Add feature to whitelist which role should be assumed in the namespace level.
Why is this needed:
What I want to avoid is to create multiple IAM roles in our K8S account, instead to have one role which should be trusted by all team accounts and then restrict which role to be assumed in the namespace level (That's the current concept in KIAM)

@mo-saeed you don't have to first assume role in the shared account, you can have Team A's pods directly assume a role in Team A's account (same for Team B).
Both Team A and Team B can create an OIDC Identity Provider in their respective accounts referencing the URL from the K8s cluster. Then each team just needs to annotate their own service accounts with the IAM role ARN in their own account.
Ex:

Cluster is in Account Z
Cluster OIDC url is https://oidc.eks.us-west-2.amazonaws.com/id/ZZZZ
Team A owns account ID AAAA
Team B owns account ID BBBB
Team A goes into IAM in their account and creates an OIDC IDP, getting the ARN arn:aws:iam::AAAA:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/ZZZZ.
Team A creates an IAM role arn:aws:iam::AAAA:role/App1 trusting the IDP arn:aws:iam::AAAA:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/ZZZZ.
Team A annotates their service account with the role ARN arn:aws:iam::AAAA:role/App1
Team b goes into IAM in their account and creates an OIDC IDP, getting the ARN arn:aws:iam::BBBB:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/ZZZZ.
Team B creates an IAM role arn:aws:iam::BBBB:role/App2 trusting the IDP arn:aws:iam::BBBB:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/ZZZZ.
Team B annotates their service account with the role ARN arn:aws:iam::BBBB:role/App2

Does this accomplish what you're trying to do?
See the first example in the EKS docs

Hello @micahhausler I followed this for one account but I got this error when I tried to list s3 buckets:
An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: Incorrect token audience

when I create the OIDC provider what is the audience? and is it related to this issue that I set it wrongly, if yes what should be the value then?

You can use the same audience for all clusters since they have different signing keys. The webhook automatically injects the audience sts.amazonaws.com into the pods, so if you're using EKS, that would need to be the audience you set on all OIDC providers in IAM.

Anything else here? If not I'll close due to no reply

@micahhausler I'm curious on the 2nd option "Use chained AssumeRole operations". Do you have any sample/example anywhere on how to use that with IRSA. I'm trying to test that using Pod which has aws-cli and that doesn't go as expected.
Here is the sample:
mtvl15367e28a:iam-manager nmogulla$ k exec -it aws-cli /bin/sh
/ # aws sts get-caller-identity
{
    "Account": "898883873262",
    "UserId": "AROA5CSNHNHXDESGZQ73G:botocore-session-1591036429",
    "Arn": "arn:aws:sts::898883873262:assumed-role/k8s-iam-manager-iamrole-irsa/botocore-session-1591036429"
}
/ #
/ # aws --profile ResourceAccountRole sts get-caller-identity

The config profile (ResourceAccountRole) could not be found
/ # cat ~/.aws/config
cat: can't open '/root/.aws/config': No such file or directory
/ # exit

From the setup stand point, I made sure Account B role has a trust policy for this role and also Account A role has permission policy with sts:AssumeRole. Any pointers would be appreciated!!

The [EKS docs](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html#cross-account-access have an example on configuring your AWS config for the chained setup

Hi, I am getting the same error while deploying EKS v1.18 with Fargate profile. The global endpoint like "sts.amazonaws.com" is usually blocked in corporate network so I had to switch to regional sts endpoint like sts.eu-central-1.amazonaws.com in the OIDC audience settings. The ALB deploys correctly but when deploying nginx ingress I get this error . Is there a way to resolve this issue ?
{"level":"info","ts":1608012040.3850577,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding"}
{"level":"info","ts":1608012040.3851316,"logger":"controller","msg":"Starting workers","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","worker count":3}
{"level":"error","ts":1608012218.831581,"logger":"controller","msg":"Reconciler error","controller":"ingress","name":"ex-nginx-ingress","namespace":"kube-system","error":"ingress: kube-system/ex-nginx-ingress: WebIdentityErr: failed to retrieve credentials\ncaused by: InvalidIdentityToken: Incorrect token audience\n\tstatus code: 400, request id: 322b1319-371a-40f3-9009-3bc6165f9e78"}
{"level":"error","ts":1608012265.3753707,"logger":"controller","msg":"Reconciler error","controller":"ingress","name":"ex-nginx-ingress","namespace":"kube-system","error":"ingress: kube-system/ex-nginx-ingress: WebIdentityErr: failed to retrieve credentials\ncaused by: InvalidIdentityToken: Incorrect token audience\n\tstatus code: 400, request id: b30f68fd-00fb-4f24-9fe4-2b2cb2491ee9"}
{"level":"error","ts":1608012304.746419,"logger":"controller","msg":"Reconciler error","controller":"ingress","name":"ex-nginx-ingress","namespace":"kube-system","error":"ingress: kube-system/ex-nginx-ingress: WebIdentityErr: failed to retrieve credentials\ncaused by: InvalidIdentityToken: Incorrect token audience\n\tstatus code: 400, request id: f181ebe8-3e4c-471d-9d97-7ed4f98eb9a2"}

So far the issue looks like OIDC provider works only with global sts endpoint

authors: micahhausler mnkg561 mo-saeed toytown

repo: aws/amazon-eks-pod-identity-webhook


text:
(3.0.0-3.1.4) ParallelCluster API Stack Upgrade Fails for ECR resources

Bug description
Upgrading the CloudFormation stack for the ParallelCluster API from one version to another (within the versions in reported range) is going to fail with the following error messages for these reported resource IDs:

EcrImagesRemover:

UPDATE_FAILED:
Received response status [FAILED] from custom resource.
Message returned: See the details in CloudWatch Log Stream: ...

EcrImage:

UPDATE_FAILED:  Resource creation cancelled

This is preventing customers from upgrading their ParallelCluster API stacks from one version to another.
Root Cause
The EcrImageDeletionLambda IAM Role also has a policy that only matches an ImagePipeline resource found in the current API Stack. This makes it unable to use the ImagePipeline resource of the new API stack.
Mitigation:
For more details and the mitigation strategy, use this wiki: (3.0.0 3.1.4) ParallelCluster API Stack Upgrade Fails for ECR resources

Issue fixed in ParallelCluster 3.2.0

source: gharchive/issue

authors: EddyMM gmarciani

repo: aws/aws-parallelcluster

license_type: permissive

license_source: github-api


text:
c5d instances intermittently fail to get added to SGE

@tanantharaman reported in https://github.com/aws/aws-parallelcluster-cookbook/pull/141#issuecomment-480632129
Looks like the problem is not fully fixed for m5d instance types : instead the problem has become intermittent : I am using ParallelCluster 2.3.1 and find that around 4 out of 5 times the m5d.24xlarge (or m5d.metal) node is added without problem but around 1 out of 5 times it fails to add the node to SGE. I am able to ssh to the node from the master (using the public IP) and the /var/log/cfn-init.log (attached) shows a problem with the ephemeral drives.
Relates to issue #415
cfn-init.log
cfn-init-cmd.log
cfn-wire.log
dmesg.log
It would help if the node recovers without mounting the ephemeral drives when there is a problem instead of hitting an unhandled exception and leaving the node running but not added to SGE : In my case I had only one job in the SGE queue and jobwatcher on the master node keeps saying no more scaleout needed (desired= 1), but my job never runs.

This was fixed in #299 and will be included in the next release


text:
Add validation for supported OSes when using FSx

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #1681 into develop will increase coverage by 0.1%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           develop    #1681     +/-   ##
==========================================
+ Coverage     52.5%   52.61%   +0.1%
==========================================
  Files           24       24
  Lines         3950     3959      +9
==========================================
+ Hits          2074     2083      +9
  Misses        1876     1876

Impacted Files
Coverage Δ

cli/pcluster/config/mappings.py
100% <ø> (ø)
:arrow_up:

cli/pcluster/config/validators.py
81.77% <100%> (+0.38%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 306e0be...0e9427c. Read the comment docs.

source: gharchive/pull-request

authors: codecov-io demartinofra


text:
[3.x][FileCache] Add Fsx FileCache in DFSM and its own Integration Tests

Description of changes

Describe what you're changing and why you're doing these changes.
Adding Fsx FileCache integration tests to test the mounting of the external Fsx FC, reading already existed files and write files in FC.
Adding Fsx FC in DFSM integration tests.
Adding unmanaged Fsx storage  in CW Dashboard unit tests.

Tests

Tested the changes Locally

Checklist

[x] Make sure you are pointing to the right branch and add a label in the PR title (i.e. 2.x vs 3.x)
[x] Check all commits' messages are clear, describing what and why vs how.
[x] Make sure to have added unit tests or integration tests to cover the new/modified code.
[ ] Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Looking at failing tox checks


text:
[Docs] Fix changelog entries for 3.9.1.

Description of changes
Cherry-picked from https://github.com/aws/aws-parallelcluster/pull/6202
References

Link to impacted open issues.
Link to related PRs in other packages (i.e. cookbook, node).
Link to documentation useful to understand the changes.

Checklist

Make sure you are pointing to the right branch.
If you're creating a patch for a branch other than develop add the branch name as prefix in the PR title (e.g. [release-3.6]).
Check all commits' messages are clear, describing what and why vs how.
Make sure to have added unit tests or integration tests to cover the new/modified code.
Check if documentation is impacted by this change.

Please review the guidelines for contributing and Pull Request Instructions.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Closed in favour of https://github.com/aws/aws-parallelcluster/pull/6210


text:
S3-CRT Client signature error on s3 object keys with special characters

Describe the bug
When using the S3-CRT PUT command, this error is encountered if the object key contains a special character. In our case, the '@' character.
The regular S3 PUT command works as intended.
Expected Behavior
s3-crt client would work seamlessly against the same object keys that the regular s3 client supports.
Current Behavior
S3-crt client fails on object keys with special characters with the following message:
The request signature we calculated does not match the signature you provided. Check your key and signing method.

Reproduction Steps
Example code.
Change the bucket name. Make sure the object name has a '@' in the name.
#include <aws/core/Aws.h>
#include <aws/core/client/ClientConfiguration.h>

#include <aws/s3-crt/model/PutObjectRequest.h>
#include <aws/s3-crt/S3CrtClient.h>

#include <aws/s3/S3Client.h>
#include <aws/s3/model/PutObjectRequest.h>

int main(int argc, char ** argv) {

    // Init AWS API
    Aws::SDKOptions awsSdkOptions;
    awsSdkOptions.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
    awsSdkOptions.httpOptions.installSigPipeHandler = true;
    Aws::InitAPI(awsSdkOptions);

    std::string bucket("your-bucket");
    std::string objectKey("scratch/objectwithspecialchar@");
    std::string region("us-east-1");
    std::string message("Hello, World!");

    // s3 client code: --> works
    {
        Aws::Client::ClientConfiguration config;
        config.region = region;

        Aws::S3::S3Client s3Client(config);

        Aws::S3::Model::PutObjectRequest objectRequest;
        objectRequest.WithBucket(bucket).WithKey(objectKey);

        const std::shared_ptr<Aws::IOStream> inputData = Aws::MakeShared<Aws::StringStream>("");
        *inputData << message.c_str();

        objectRequest.SetBody(inputData);

        auto putObjectOutcome = s3Client.PutObject(objectRequest);

        if (!putObjectOutcome.IsSuccess()) {
            std::cerr << "failed s3 client put: " << putObjectOutcome.GetError().GetMessage() << std::endl;
        } else {
            std::cerr << "s3 client success!" << std::endl;
        }
    }

    // s3-crt client code --> fails on signature verification
    {
        Aws::S3Crt::ClientConfiguration config;
        config.region = region;

        Aws::S3Crt::S3CrtClient s3Client(config);

        Aws::S3Crt::Model::PutObjectRequest objectRequest;
        objectRequest.WithBucket(bucket).WithKey(objectKey);

        const std::shared_ptr<Aws::IOStream> inputData = Aws::MakeShared<Aws::StringStream>("");
        *inputData << message.c_str();

        objectRequest.SetBody(inputData);

        auto putObjectOutcome = s3Client.PutObject(objectRequest);

        if (!putObjectOutcome.IsSuccess()) {
            std::cerr << "failed s3 CRT client put: " << putObjectOutcome.GetError().GetMessage() << std::endl;
        } else {
            std::cerr << "s3 CRT client success!" << std::endl;
        }
    }

    Aws::ShutdownAPI(awsSdkOptions);
}

Possible Solution
Possible resolution from https://github.com/aws/aws-sdk-cpp/issues/2898  ??
Additional Information/Context
No response
AWS CPP SDK version used
1.11.258
Compiler and Version used
gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15)
Operating System and version
Rocky Linux 8.7 (Green Obsidian) x86_64

should have been fixed with https://github.com/aws/aws-sdk-cpp/issues/2898. can you try a more recent version?

Sure. I'll try a newer release.  I will update this task with the findings.

The latest "main" works fine.  Thanks for the fix.  This task can be closed.

authors: DmitriyMusatkin epasveer

repo: aws/aws-sdk-cpp


text:
AWS Permissions required to use awswrangler.athena.to_iceberg function

I am using awswrangler.athena.to_iceberg to write data from a pandas dataframe to iceberg table. And this is being run from a lambda function. My question is what permissions would be required to perform this operation,so that I can add those to the lambda execution role. I beleive we should have glue and target S3 bucket access. Can anyone suggest the exact glue action and s3 action required to be added in  aws policy statement. Do we have to add any athena related policies/permissions?
Code Snippet:
import awswrangler as wr
wr.athena.to_iceberg(df=df,database=GLUE_ICEBERG_DB,table=glue_table_name,table_location=OUTPUT_FILE_PATH,temp_path=TEMP_PATH,keep_files=False,index=False,schema_evolution=True,fill_missing_columns_in_df=True,partition_cols=['dt','ts'],additional_table_properties={'write_target_data_file_size_bytes':'536870912','write_compression':'SNAPPY'}
)

Don't have a specific IAM policy to share, but my suggestion would be to design it based on the actions that are carried out in the API call. You can consult the awswrangler/athena/_write_iceberg.py module to identify them and scope them to your resources. For instance, you can see that an Athena StartQueryExecution action also needed from the code.
Alternatively, you can check the AWS CloudTrail logs once you run the lambda to identify all the API calls that were made and build your policy based on that information.

repo: aws/aws-sdk-pandas


text:
Consider not recommending AWS Distro for OpenTelemetry (ADOT)?

I tried out migrating, and the Lambda layer appears to add a significant amount to cold start time. Of course, it probably only matters to people that use Lambda, but that's a lot of people.
I don't see any docs on the potential downsides of migrating. Maybe it's a bit too early to be recommending it?
There was no documentation on migrating, but here's the commit where I migrated from AWS X-Ray SDK: https://github.com/a-h/awsapigatewayv2handler/commit/c45b98eb1b9ec8ad6618b7b38b0347ef4bec963c
Here's the issue in AWS Otel Lambda, with my comments about the Go behaviour: https://github.com/aws-observability/aws-otel-lambda/issues/228#issuecomment-1193215390
Here's the test data, for reference here.

Hi @a-h,
Thanks for raising this. We are tracking the cold-start issues and considering modifying the wording on our documentation, stay tuned!
Are all of the durations in your screenshot cold-start durations (both before and after OTel)? Do you have any warm invocations for comparison between before & after OTel?

Hi @willarmiros, yes, the screenshot above is just showing cold starts.
I've included warm invocations below.
I filtered out invocations of > 10 seconds because they were just tests I was doing of running the Otel collector without the Lambda layer. The docs don't make it clear whether the Lambda layer is required - but I found that it is - without the Lambda layer, the function hangs.
The average billed duration increases from around 2ms to 13ms from the 23rd July, which is when I added the Otel, but I don't believe that to be caused by the Otel because I added some functionality to the test function a check to validate that AWS SDK usage and outbound HTTP calls were instrumented.
fields @timestamp, @billedDuration
| filter @message like /Billed Duration/
| filter @message not like /Init Duration/
| filter @billedDuration < 10000
| stats avg(@billedDuration) by bin(1d) as time
| sort by time

The drop of billed duration on the 17th corresponds with updating the Go build flags to set -tags lambda.norpc on the build flags.

Hi @a-h - we have updated the README to have a more neutral tone & be aligned with our other documentation, and have surfaced this latency issue in our developer guide as well. We are actively investigating the latency issue, and will post updates as we progress on it here: https://github.com/aws-observability/aws-otel-lambda/issues/228

authors: a-h willarmiros

repo: aws/aws-xray-sdk-go


text:
transparently use docker targets when building

Issue #, if available:
Description of changes:
We have traditionally relied on building our components on the host, which requires a fair amount of setup and dependencies.   A while back we did introduce the run-in-docker target(s) to make it easier to run parts of the build locally and match checksums/attribution files by using a docker container running the builder-base image which is what we run in prow/coudebuild.  Recently we started using EKS-D golang builds which are only produced for linux so building on mac and matching checksums (which was possible before) is no longer possible.  Using the builder-base docker approach has worked pretty well.
This PR extends our use of these docker container by converting certain targets, go build/go licenses/attribution and some project specific targets, to use docker automatically if the required toolset is missing.  In the case of golang, we just assume its missing since the likelihood of someone having install EKS-D go is not very high.  This can be overridden if the developer wants to build on the host instead of the container.
The approach used in Common.mk to run the docker targets transparently is a bit hacky....  Its basically the same approach documented and taken in #2546.  The good new is that for project Makefiles where we want to use docker if, say npm is missing in the case of the prometheus build, its a pretty simple change.
I did go through and update the building-locally doc, in which I created and recreated AL2 and AL23 instances to confirm I had as much documented as I could find.
I did add a build-all target to the root Makefile, which I do not intend to be used anywhere but it is a nice helper when trying to validate that an instance has all the dependencies needed to build pretty much the entire stack.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/hold

/lgtm
/approve

/test all

/unhold

authors: abhay-krishna jaxesn

repo: aws/eks-anywhere-build-tooling


text:
Use proper artifact location to compute BR OVA checksums

Tuftool downloads Bottlerocket OVA to a location named after the target, but our convention is to save it as just bottlerocket.ova in the S3 bucket.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: abhay-krishna, TerryHowe
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [TerryHowe,abhay-krishna]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


text:
Emmisary presubmit

Signed-off-by: jonahjon jonahjones094@gmail.com
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

emmisary -> emissary across the job definition and filename.

emissary

o yea good catch

/approve

authors: abhay-krishna jonahjon

repo: aws/eks-anywhere-prow-jobs


text:
Changed DEST_DISK to /dev/sda2

Description of changes:
In the new ubuntu image we have enabled uefi which requires dev/sda1 partition. So for all other actions we have changed DEST_DISK to sda2.
Testing (if applicable):
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/lgtm

repo: aws/eks-anywhere


text:
Install packages on management cluster

Description of changes:

Install packages on management cluster

Testing (if applicable):
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/approve

/cherry-pick release-0.12


text:
Correct golang release automation prowjob name

Issue #, if available:
Description of changes:
The golang pr name expected the wrong Prowjob name for automation.
From prowjobs automation
...
  job: check-upstream-golang-release
  namespace: default
  pod_spec:
...

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/lgtm
/approve

authors: rcrozean vineeth-bandi

repo: aws/eks-distro-build-tooling


text:
Created and updated docs for 1.27

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/lgtm

/approve

repo: aws/eks-distro


text:
Search query & Data Explorer

Description

Updated keyword search query to new style
Use consistent query for both search and Data Explorer
Reduce searches by ignoring changes to search attributes and exact match when no search term is provided
Made limit explicit in search request
Refactor Data Explorer component, breaking it down in to smaller pieces composed together

Validation

Smoke test to make sure nothing breaks

Related Issues

#340

Check List

[x] I confirm that my contribution is made under the terms of the Apache 2.0
license.
[x] I have run pnpm checks to ensure code compiles and meets standards.
[x] I have run pnpm test to check if all tests are passing.
[ ] I have covered new added functionality with unit tests if necessary.
[x] I have added an entry in the Changelog.md.

I forgot to make a couple of changes.

[ ] Add error handling to search modal
[ ] Add error handling to Data Explorer

On second thought, I'll implement those as a separate PR. So this is ready for review.

repo: aws/graph-explorer


text:
[BUG] ML notebook examples not loading data

Describe the bug
When trying to run ML notebooks (specifically Neptune-ML-00-Getting-Started-with-Neptune-ML-Gremlin.ipynb)  the process never loads any data.
response = neptune_ml.prepare_movielens_data(s3_bucket_uri)
runs and shows "Processing X" (where X is Movies or Ratingss) but it doesn't load any data, it can be cheked by inspecting the S3 bucket indicated by variable s3_bucket_uri but also print(response) returns None.
To Reproduce
Steps to reproduce the behavior:

Go to a Neptune sagemaker notebook instance
Open the notebook 04-Machine-Learning/Neptune-ML-00-Getting-Started-with-Neptune-ML-Gremlin.ipynb
Add a valid S3 location on s3_bucket_uri="s3://<INSERT S3 BUCKET OR PATH>"
Run all cells

Expected behavior
Sample data should be loaded to the  S3 location but it is not.

Hi @llealgt, thank you for submitting a bug report!
Given the issue signature, your IAM user/role may not have write permissions to the S3 bucket being used. I was not able to reproduce on my normal setup; however, using another S3 bucket with no write access, I did get stuck at the same place:

You can adjust the Python logging level to ERROR to see if you are getting a similar exception from the prepare_movielens_data command.

Hi @michaelnchin thanks for your reply, indeed it is a "access denied" issue, however I never considered that option because I was able to run the other(non ML) example notebooks using the same S3 bucket. Is there any difference between how this one downloads and copies the data compared to the other notebooks?

Hi @michaelnchin thanks for your help, I found write permission was revoked and the role had read-only, that's why it worked before but not now. We can close this issue

Hi @michaelnchin I know I closed this 1 hour ago but I found another issue, when running:
endpoints=neptune_ml.setup_pretrained_endpoints(s3_bucket_uri, setup_node_classification, setup_node_regression, setup_link_prediction, setup_edge_classification, setup_edge_regression)
the result is None, and I see the following in the logs:
ERROR:root:Unable to determine the Neptune ML IAM Role.
Any suggestions on how to fix it, or how to debug it?

You need to include the Neptune ML IAM role (created and attached to your Neptune cluster as documented here) in your local .bashrc file.
Use the following command to add it:
echo "export NEPTUNE_ML_ROLE_ARN=[YOUR_NEPTUNE_ML_IAM_ROLE_ARN]" >> ~/.bashrc

For other ML notebooks, some neptune_ml commands also look in .bashrc for your Neptune Export service endpoint, so you should also include this using something similar to:
echo "export NEPTUNE_EXPORT_API_URI=https://3ui13o134.execute-api.us-east-1.amazonaws.com/v1/neptune-export" >> ~/.bashrc

Hi @michaelnchin I tested adding the export you suggested to the .bashrc file, it seems like it solved the previous error, now I get a new one:
ERROR:root:An error occurred (AccessDeniedException) when calling the CreateModel operation: User: arn:aws:sts::732279101103:assumed-role/AWSNeptuneNotebookRole-neptune-test2-lleal-role/SageMaker is not authorized to perform: sagemaker:CreateModel on resource: arn:aws:sagemaker:us-east-1:732279101103:model/classifi-2023-02-13-23-16-27 because no identity-based policy allows the sagemaker:CreateModel action
Where AWSNeptuneNotebookRole-neptune-test2-lleal-role is a role created during notebook configuration by choosing "create an iam role", so I guess this role needs some permissions to be added, is there any list of permissions suggested for this role?

Yes, your Sagemaker notebook also requires additional permissions. Attach a policy containing the following to your notebook's IAM role:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "cloudwatch:PutMetricData"
            ],
            "Resource": "arn:aws:cloudwatch:[AWS_REGION]:[AWS_ACCOUNT_ID]:*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents",
                "logs:GetLogEvents"
            ],
            "Resource": "arn:aws:logs:[AWS_REGION]:[AWS_ACCOUNT_ID]:*",
            "Effect": "Allow"
        },
        {
            "Action": "neptune-db:*",
            "Resource": "arn:aws:neptune-db:[AWS_REGION]:[AWS_ACCOUNT_ID]:[CLUSTER_RESOURCE_ID]/*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "s3:Put*",
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": "arn:aws:s3:::*",
            "Effect": "Allow"
        },
        {
            "Action": "execute-api:Invoke",
            "Resource": "arn:aws:execute-api:[AWS_REGION]:[AWS_ACCOUNT_ID]:*/*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "sagemaker:CreateModel",
                "sagemaker:CreateEndpointConfig",
                "sagemaker:CreateEndpoint",
                "sagemaker:DescribeModel",
                "sagemaker:DescribeEndpointConfig",
                "sagemaker:DescribeEndpoint",
                "sagemaker:DeleteModel",
                "sagemaker:DeleteEndpointConfig",
                "sagemaker:DeleteEndpoint"
            ],
            "Resource": "arn:aws:sagemaker:[AWS_REGION]:[AWS_ACCOUNT_ID]:*/*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "iam:PassRole"
            ],
            "Resource": "[YOUR_NEPTUNE_ML_IAM_ROLE_ARN]",
            "Effect": "Allow"
        }
    ]
}

Thanks! It works, now I'm trying to run notebook
Neptune-ML-01-Introduction-to-Node-Classification-Gremlin
It fails at
%%neptune_ml export start --export-url {neptune_ml.get_export_service_host()} --export-iam --wait --store-to export_results
${export_params}

But returns the following error(it seems as the notebook cannot reach the export service)
Max retries exceeded with url: /v1/neptune-export (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fe5b46f8a10>: Failed to establish a new connection: [Errno 110] Connection timed out'))"))}
I had performed the guide twice(https://docs.aws.amazon.com/neptune/latest/userguide/export-service.html) with no luck. Is there anything else needed in order to run the notebook? Like configuring some additional permission for the notebook?

Please verify that your Neptune cluster, Export service, and Sagemaker notebook instance are all in the same VPC and have the subnets/security groups configured correctly.
One other specific thing to check, please also ensure that the NeptuneExportSecurityGroup created by the Export stack is attached to the Sagemaker instance:
Enable access to the Neptune-Export endpoint from a VPC-based EC2 instance

If you make your Neptune-Export endpoint VPC-only, you can only access it from within the VPC in which the Neptune-Export service is installed. To allow connectivity from an Amazon EC2 instance in the VPC from which you can make Neptune-Export API calls, attach the NeptuneExportSecurityGroup created by the AWS CloudFormation stack to that Amazon EC2 instance.

It should return True if your cluster is IAM enabled. The result of neptune_ml.get_iam() is based on the notebook config file in your home directory; this should be /home/ec2-user/graph_notebook_config.json on Sagemaker. Could you check that the contents of this config file match your cluster details (especially for auth_mode field)?

Thanks for your help, it seems as the cluster  VPC security group was not configured correctly to allow access from the export security group, now  the export service is up and running and the cell now runs successfully
and sorry to do so many questions that can be obvious(I'm more of a data science/ML person than an cloud or ops person)
now the cell
%neptune_ml dataprocessing start --wait --store-to processing_results {processing_params}
returns
{
  "detailedMessage": "Unable to connect to vpc endpoint. Please check your vpc configuration.",
  "requestId": "7a6f5966-4e53-41bb-b568-6f8ab2dc620c",
  "code": "BadRequestException"
}

Any hints on that one?

The error suggests that there is a problem not being able to connect to the Sagemaker VPC endpoints, can you confirm that you have created endpoints for both sagemaker.runtime and sagemaker.api , as documented here:
https://docs.aws.amazon.com/neptune/latest/userguide/machine-learning-manual-setup.html#ml-manual-setup-endpoints

Yep, they were created some days ago and they were working correctly when running the first notebook example, but it is fixed now, I found that when the notebook instance was restarted some minutes ago, as expected the variables defined in the .bashrc file(which you suggested some days ago) were lost, so not sure if it had something to do with the role specified there, but when creating the variables again it worked(I will add those exports to the notebook lyfecicle file).
Now I have the following error, but I guess that should be easily fixed by changed the job configurations to a different EC2 instance size:
"detailedMessage": "AmazonSageMakerException: The account-level service limit 'ml.r5.large for processing job usage' is 0 Instances

Thank you for confirming the fix, @llealgt.
For the Sagemaker account service limit exception, you will need to contact AWS support to request an increase:
https://aws.amazon.com/premiumsupport/knowledge-center/sagemaker-resource-limit-exceeded-error/

Thanks! I thought maybe using a different EC2 size could work because I thought the limits where instance size dependent but I will check the link you just shared.
One last question, now the notebook magic
%neptune_ml dataprocessing
works(assuming i fix the service limit) so I assume neptune ML and sagemaker are correctly configured, but assuming I want to curl the datapreprocessing entpoints via curl from the notebook terminal (something like this https://docs.aws.amazon.com/neptune/latest/userguide/machine-learning-on-graphs-processing.html)
do I need to set another configuration? currently it doesn't work, it time-out so I guess the notebook cannot reach neptune endpoints.

Can you share the awscurl command you are using?

awscurl \
  -X POST https://{my neptune endpoint copied from the neptune console}/ml/dataprocessing \
  -H 'Content-Type: application/json' \
  -d '{
        "inputDataS3Location" : "s3://{my bucket}/neptune-export/20230217_215839",
        "id" : "test",
        "processedDataS3Location" : "s3://{my bucket}/preprocessed",
        "configFileName" : "training-job-configuration.json"
      }'

You may need to specify additional options, ex. for --service as Neptune instead of the default execute-api.
Try the following to check against the status endpoint:
awscurl https://<neptune_endpoint>:<neptune_port>/status --service neptune-db --region <region>

I see my issue, I was missing the port, I think that solves it, thanks for all of your help!

authors: llealgt michaelnchin

repo: aws/graph-notebook


text:
track volume mounts per node

1. Issue, if available:
Fixes #919 and #1888
2. Description of changes:
track volume mounts per node
Assume nodes can support infinite volumes until they launch. Once the CSI driver is
reporting the current number of mountable volumes, use that value which may require
launching more nodes.
3. How was this change tested?
Unit testing & deployed to EKS with my EBS CSI driver configured with --volume-attach-limit=3. When the driver created the CSINode Karpenter launched another node for the remaining volumes.
4. Does this change impact docs?

[ ] Yes, PR includes docs updates
[ ] Yes, issue opened: link to issue
[X] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

This might be a silly question, but do we no longer need the PersistentVolumeClaim controller given we're no longer pre-binding pods?
Discussed with @tzneal offline and the short answer is "yes".

authors: dewjam tzneal

repo: aws/karpenter


text:
How can I set mounted secret permissions?

Guys,
Need a little help here: How can I set mounted secret permission?
With EKS secret it's set via
   secret:
     defaultMode: 0600

with result:
-rw-------    1 root     root          1864 May  2 22:01 .mylogin.cnf
With AWS secret it's:
-rw-r--r--    1 root     root          1864 May  2 22:18 .mylogin.cnf
What parameter controls it?
How do I set it to be -rw-------
The pod yaml is:
apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:
  serviceAccountName: tester
  containers:
  - name: busybox
    image: registry.k8s.io/busybox:latest
    command: ["sh"]
    args:
    - -c
    - ls -la /root/
    volumeMounts:
    - name: mylogin
      mountPath: /root/.mylogin.cnf
      subPath: .mylogin.cnf
      readOnly: true
  restartPolicy: Never
  volumes:
  - name: mylogin
    csi:
      driver: secrets-store.csi.k8s.io
      defaultMode: 0600
      readOnly: true
      volumeAttributes:
        secretProviderClass: "aws.mylogin.cnf"
  # volumes:
  # - name: mylogin
  #   secret:
  #     defaultMode: 0600
  #     secretName: mysql
  #     items:
  #     - key: .mylogin.cnf
  #       path: .mylogin.cnf

This value is coming from the upstream CSI Driver. We will reach out to the team to see if there are any recommendations.

Upstream permissions gets un-marshaled here and has test coverage.

authors: jbct ok512 simonmarty


text:
Implement Responsible View on Page

Demo
you can check on https://dev.kendra.fun
(social sign-in is not available. this only available on admin.kendra.fun)
Actions

Import google fonts(Orbitron, Pacifico)

Add 'Kendra Button' Title in account related pages & Rearranged signing screen Layout

Update guide of Embed

Modify the responsible style of Sidebar Button and Content

Search component

Add error validation
refine codes

수정 완료했습니다~!

수고하셨습니다~!
frontend/kendra-button-front/src/components/Confirmation.tsx 에서
77번 line
         <p className={`text-secondary small`}>

사소한 것이지만 mt-4 class를 추가해주시면 더 좋을 것 같습니다.
만약 위의 사항을 추가 하신다면,
83번 line
className="alert alert-dismissible alert-success signUpSuccess"

className에 text-center px-0 를 추가해주시면 더 깔끔할 것 같습니다!

repo: awskrug/kendra-button


text:
WARNING: erroneous pipeline: no element "kvssink" [NVIDIA Jetson Tx2]

Hi,
I am trying to use KVS on Jetson TX2 with JetPack 4.2. I compiled KVS producer library as instructed and copied files to /usr/lib/aarch64-linux-gnu/.
Here is my command.
sudo gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=30/1' ! nvtee ! nvvidconv flip-method=0  ! 'video/x-raw, format=(string)NV12, width=640, height=480, framerate=30/1' ! omxh264enc  ! video/x-h264, stream-format=avc, alignment=au ! kvssink stream-name=*********** storage-size=512 access-key=************ secret-key=*********************

It worked right after compilation finished. Then I rebooted TX2 and it does not work anymore.
The only line of error message is
WARNING: erroneous pipeline: no element "kvssink"

But libgstkvssink.so exists under /usr/lib/aarch64-linux-gnu/gstreamer-1.0.
Here is the gst-inspect-1.0 output.
$ gst-inspect-1.0 kvssink

(gst-plugin-scanner:7820): GStreamer-WARNING **: 16:41:50.069: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstkvssink.so': /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstkvssink.so: undefined symbol: _ZN3com9amazonaws7kinesis5video18CredentialProvider12getCallbacksEP17__ClientCallbacks
No such element or plugin 'kvssink'

How can i fix this error?
Thank you!

Hi @ac4289 ,
You need to add directory of libgstkvssink.so to environment variable GST_PLUGIN_PATH or move it to the directory where gstreamer plugins are in.

It's complaining about not be able to find symbol in libproducer.so. the plugin is dynamically linked to the libproducer.so. Can you do export LD_LIBRARY_PATH=/path/to/dir for the directory that has libproducer.so?

It's complaining about not be able to find symbol in libproducer.so. the plugin is dynamically linked to the libproducer.so. Can you do export LD_LIBRARY_PATH=/path/to/dir for the directory that has libproducer.so?

Thank you for helping!
I tried this:
export LD_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu/gstreamer-1.0

Still does not work.

is that where your libproducer.so is?

is that where your libproducer.so is?

Yes

what about libcproducer.so?

libcproducer.so is under /usr/lib/aarch64-linux-gnu/

can you move it to /usr/lib/aarch64-linux-gnu/gstreamer-1.0 too? and do export LD_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu/gstreamer-1.0

LD_LIBRARY_PATH=/usr/lib/aarch64-linux-gnu/gstreamer-1.0

Does not work. T_T

just wanna confirm libcproducer.so and libproducer.so are all in /usr/lib/aarch64-linux-gnu/gstreamer-1.0?

just wanna confirm libcproducer.so and libproducer.so are all in /usr/lib/aarch64-linux-gnu/gstreamer-1.0?

Yes

Hi @ac4289  ,

It worked right after compilation finished.

Given that it worked first time, could you please run the source set_kvs_sdk_env.sh and see if that helps.

Hi @ac4289 ,

It worked right after compilation finished.

Given that it worked first time, could you please run the source set_kvs_sdk_env.sh and see if that helps.

No I already tried that. I just reflashed TX2 with JetPack 4.2. I'll see how it goes.

Ok, reflashed TX2 works with KVS. The difference is that the old OS had TVM installed which requires a change of Python path in .bashrc.
Thank you again for your help!

I'm getting the same error: no element "kvssink"
plugin scanner says: Failed to load libgstx264.so, undefined symbol x264_deblock_h_chroma_422_intra_neon
Did you have this issue? Do you know how to fix it? Thanks.

Once you've successfully built the plugin, you should set the paths per: https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp/blob/master/install-instructions-macos.md#setting-the-environment-variables-for-library-path

authors: MushMal ac4289 aimroboticsbying chehefen unicornss zhiyua-git


text:
Signaling client error: TypeError: Cannot read property 'digest' of undefined

I am trying to create my own viewer following the example given in the repo.
In my html file, I link to the two js libraries:
https://sdk.amazonaws.com/js/aws-sdk-2.585.0.min.js
https://unpkg.com/amazon-kinesis-video-streams-webrtc/dist/kvs-webrtc.min.js

When I trigger the startViewer(), I get the following error:
app.js?modified=1599143599:20 [VIEWER] Signaling client error:  TypeError: Cannot read property 'digest' of undefined
    at Function.<anonymous> (kvs-webrtc.min.js:8)
    at kvs-webrtc.min.js:8
    at Object.next (kvs-webrtc.min.js:8)
    at kvs-webrtc.min.js:8
    at new Promise (<anonymous>)
    at Module.l (kvs-webrtc.min.js:8)
    at Function.e.sha256 (kvs-webrtc.min.js:8)
    at e.<anonymous> (kvs-webrtc.min.js:8)
    at kvs-webrtc.min.js:8
    at Object.next (kvs-webrtc.min.js:8)

The offending segment of the library seems to be this, where i.default is undefined
case 0:return[4,i.default.subtle.digest({name:"SHA-256"},this.toUint8Array(e))];

Any help in tracking down my error will be much appreciated.

@Morras Would you mind to share a snippet where it crashes?

I found that the problem is that while I am working on this on my development machine, I am serving the traffic locally over http and not https.  However WebCrypto is only available from a "secure origin" which is essentially either https, or local host.
When I tested this with local host it worked fine.
You can read more about that here: https://chromium.googlesource.com/chromium/src/+/master/docs/security/faq.md#why-are-some-web-platform-features-only-available-in-https-page_loads

authors: Morras lherman-cs


text:
Add TCP Keepalives for column encoding tool

This is a pull request for the column encoding tool.  This would help with the issue in  https://github.com/awslabs/amazon-redshift-utils/issues/224
In the ideal case pg8000 will support TCP keepalives however since for this tool it's always beneficial to have keepalives on we can just manually do it by operating on the socket.
[ec2-user@ip-172-31-7-182 ~]$ netstat -top | grep "`ps -ef | grep analyze | grep -v grep | awk '{ print $2}'`"
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 ip-XXX-XX-XX-XXX.eu-we:43566 ip-XXX-XX-XX-XXX.eu-we:5439 ESTABLISHED 3021/python         keepalive (5.75/0/0)

Awesome - never managed to spot the usock property. Thanks!

authors: IanMeyers pvbouwel

repo: awslabs/amazon-redshift-utils


text:
[RL] [Ray] Update #581 errors

Running the RL model evaluation yields a strange error as of #581, as for some reason, wait_for_s3_object somehow is unable to find the model.tar.gz file, which exists, and can be seen manually in S3's bucket.
In detail, running this code section from the updated Jupyter file:
if local_mode:
    model_tar_key = "{}/model.tar.gz".format(job_name)
else:
    model_tar_key = "{}/output/model.tar.gz".format(job_name)

local_checkpoint_dir = "{}/model".format(tmp_dir)

wait_for_s3_object(s3_bucket, model_tar_key, tmp_dir, training_job_name=job_name)

Yields:
Waiting for s3://sagemaker-us-east-2-109563362291/rl-test-env-2019-01-24-16-55-30-915/output/model.tar.gz...
 Downloading rl-test-env-2019-01-24-16-55-30-915/output/model.tar.gz
 ---------------------------------------------------------------------------
 FileNotFoundError                         Traceback (most recent call last)
 <ipython-input-19-bf5b7d9b7b7b> in <module>()
       6 local_checkpoint_dir = "{}/model".format(tmp_dir)
       7
 ----> 8 wait_for_s3_object(s3_bucket, model_tar_key, tmp_dir, training_job_name=job_name)
       9
      10 if not os.path.isfile("{}/model.tar.gz".format(tmp_dir)):

 ~/SageMaker/rl_thesis_ray/common/misc.py in wait_for_s3_object(s3_bucket, key, local_dir, local_prefix, aws_account, aws_region, timeout, limit, fetch_only, training_job_name)
      91         print("Downloading %s" % obj.key)
      92         local_path = os.path.join(local_dir, local_prefix, obj.key.split('/')[-1])
 ---> 93         obj.Object().download_file(local_path)
      94         fetched_files.append(local_path)
      95

 ~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/boto3/s3/inject.py in object_download_file(self, Filename, ExtraArgs, Callback, Config)
     312     return self.meta.client.download_file(
     313         Bucket=self.bucket_name, Key=self.key, Filename=Filename,
 --> 314         ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
     315
     316

 ~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/boto3/s3/inject.py in download_file(self, Bucket, Key, Filename, ExtraArgs, Callback, Config)
     170         return transfer.download_file(
     171             bucket=Bucket, key=Key, filename=Filename,
 --> 172             extra_args=ExtraArgs, callback=Callback)
     173
     174

 ~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/boto3/s3/transfer.py in download_file(self, bucket, key, filename, extra_args, callback)
     305             bucket, key, filename, extra_args, subscribers)
     306         try:
 --> 307             future.result()
     308         # This is for backwards compatibility where when retries are
     309         # exceeded we need to throw the same error from boto3 instead of

 ~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/s3transfer/futures.py in result(self)
      71             # however if a KeyboardInterrupt is raised we want want to exit
      72             # out of this and propogate the exception.
 ---> 73             return self._coordinator.result()
      74         except KeyboardInterrupt as e:
      75             self.cancel()

 ~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/s3transfer/futures.py in result(self)
     231         # final result.
     232         if self._exception:
 --> 233             raise self._exception
     234         return self._result
     235

 ~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/s3transfer/tasks.py in __call__(self)
     124             # main() method.
     125             if not self._transfer_coordinator.done():
 --> 126                 return self._execute_main(kwargs)
     127         except Exception as e:
     128             self._log_and_set_exception(e)

 ~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/s3transfer/tasks.py in _execute_main(self, kwargs)
     148         )
     149
 --> 150         return_value = self._main(**kwargs)
     151         # If the task is the final task, then set the TransferFuture's
     152         # value to the return value from main().

 ~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/s3transfer/download.py in _main(self, fileobj, data, offset)
     580         :param offset: The offset to write the data to.
     581         """
 --> 582         fileobj.seek(offset)
     583         fileobj.write(data)
     584

 ~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/s3transfer/utils.py in seek(self, where)
     333
     334     def seek(self, where):
 --> 335         self._open_if_needed()
     336         self._fileobj.seek(where)
     337

 ~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/s3transfer/utils.py in _open_if_needed(self)
     316     def _open_if_needed(self):
     317         if self._fileobj is None:
 --> 318             self._fileobj = self._open_function(self._filename, self._mode)
     319             if self._start_byte != 0:
     320                 self._fileobj.seek(self._start_byte)

 ~/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/s3transfer/utils.py in open(self, filename, mode)
     242
     243     def open(self, filename, mode):
 --> 244         return open(filename, mode)
     245
     246     def remove_file(self, filename):

 FileNotFoundError: [Errno 2] No such file or directory: '/tmp/rl-test-env-2019-01-24-16-55-30-915/model.tar.gz.EcF21e87'

Hi @Guilherme-B,
Can you elaborate which notebook generate the error? And are you running in local mode?
I went through the rl_cartpole_coach_gymEnv notebook and didn't get into this error.
The wait_for_s3_object  works fine in this notebook to find the model artifact.

Closing this issue due to lack of activity. Feel free to re-open for further related errors.

authors: Guilherme-B annaluo676 yijiezz

repo: awslabs/amazon-sagemaker-examples


text:
Unable to train Object detection with other base network than resnet-50 and vgg-16

Hello, I know that right now we can only train an object-detection model with resnet-50 or vgg-16 as base network.
But I would like to know why are they the only one to be allowed as hyperparameters.
In my case I have a small dataset (~300 pictures) and after doing some researchI found that resnet-18 is better than resnet-50 for small datasets.
So isn't it possible to train this algorithm with other base network than resnet-50 and vgg-16 because they are the only compatible (by design) ? Or is it just a limitation to avoid weird behaviors and results ?
Thank you in advance for your guidance

Hi,
Unfortunately, it is by design. SageMaker Object Detection currently only supports resnet-50 and vgg-16. We'll consider your use case and log this issue as a feature request for more base networks.

authors: Bannaro wiltonwu


text:
TensorBoard example notebook

Description of changes:
A new notebook demonstrating how to use TensorBoard with SageMaker.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

TensorBoard magic in notebooks is blocked by this PR https://github.com/tensorflow/tensorboard/issues/3632
Profiler plugin for TensorBoard running inside of SageMaker Notebook Instance is blocked by this PR https://github.com/tensorflow/profiler/issues/45


text:
Event loop public api

Move unnecessarily public types in event_loop.h to a private header
Refactor event loop group creation to a single constructor function

Let's see what breaks here.  Downstream is obviously hosed.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report
Attention: Patch coverage is 92.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 80.42%. Comparing base (e247ef8) to head (381f3dd).

Files with missing lines
Patch %
Lines

source/event_loop.c
91.30%
2 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #691      +/-   ##
==========================================
+ Coverage   80.15%   80.42%   +0.27%
==========================================
  Files          28       28
  Lines        5965     5968       +3
==========================================
+ Hits         4781     4800      +19
+ Misses       1184     1168      -16

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: bretambrose codecov-commenter

repo: awslabs/aws-c-io


text:
alternative approach to suspending scaling processes that supports concurrent runs

The docs note:

WARNING: If you are using [processes suspend] functionality you should only use CodeDepoyDefault.OneAtATime deployment configuration to ensure a serial execution of the scripts. Concurrent runs are not supported.

To support concurrent runs, we use an alternative approach: suspend/resume around the entire deploy via a deployment group trigger. Here's the code that our lambda function uses; feel free to link it in the docs: https://gist.github.com/simon-weber/285cadb567c11268a15db1b97fe2cbc5.

Hi!
I'll pass the example along to the team. Thanks for the suggestion!

authors: Jmcfar simon-weber

repo: awslabs/aws-codedeploy-samples


text:
What permissions are needed for Athena queries?

In the documentation for making read_sql_query requests, it says that if you choose to use ctas_approach=True it requires create/delete table permissions on Glue. If I also set ctas_database_name, why do I still need permissions on the database?
For example:
df = wr.athena.read_sql_query(
            SELECT * FROM my_table LIMIT 10;",
            database="temp1",
            ctas_database_name="temp2",
            boto3_session=session,
        )

is erroring unless I give my role permission to CreateTable and DeleteTable on 'temp1'. Can you provide a list of iam and LakeFormation permissions or more verbose examples of how to use "ctas_database_name"?
I want wrangler to have no access to create/delete tables in my main db but full access to do whatever it wants in the ctas_database_name database for any temporary tables

Thank you for raising this @andy-han, I believe it's a bug on our end. Specifically this line where the temporary table is deleted once the submitted query has completed. At the moment it always attempts to delete a table in the original database, regardless of whether a ctas_database_name parameter was provided or not. Modifying that line to:
catalog.delete_table_if_exists(database=ctas_database_name or database, table=name, boto3_session=boto3_session)

should solve the issue. I will raise a PR to mitigate this.

This has now been merged to main and can be tested:
pip uninstall awswrangler -y
pip install git+https://github.com/awslabs/aws-data-wrangler.git@main

until it's available in the next release.

Thanks!

authors: andy-han jaidisido

repo: awslabs/aws-data-wrangler


text:
Docker containers run as root

We have these Dockerfiles on the project:
/frontend/docker/prod/Dockerfile
/documentation/userguide/docker/prod/Dockerfile
/backend/docker/dev/Dockerfile
/compose/postgres/Dockerfile
/compose/elasticsearch/Dockerfile
/backend/docker/prod/ecs/Dockerfile
/backend/docker/prod/lambda/Dockerfile
/frontend/docker/dev/Dockerfile
All of these do not use a USER directive and run as root. Even though it's not always relevant it does trigger security scanners which Scan dockerfiles.
A simple solution is described here: https://www.geeksforgeeks.org/docker-user-instruction/

Hi @zsaltys, actually this issue was described by @manjulaK here: https://github.com/awslabs/aws-dataall/issues/445 We will re-pick it up and see if it needs to be prioritized.

@dlpzx we definitely want to prioritize this

PR - https://github.com/awslabs/aws-dataall/pull/895

authors: dlpzx mourya-33 zsaltys

repo: awslabs/aws-dataall


text:
Fix text in images #242

Issue #, if available:
Text detection in images is not working. This fixes it.
#242

This issue relates to the front-end, but the front-end has been moved to a new repo, so I've moved this PR over there:
https://github.com/awslabs/aws-media-insights/pull/2

repo: awslabs/aws-media-insights-engine


text:
Test PR from cloned repository

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Test complete.


text:
SignatureDoesNotMatch Error

I'm trying to use the TransferUtility following the instructions in "Running S3TransferUtility Sample", but whenever the upload begins, I get this error:
03-23 10:18:16.156 9907-12448/com.my.app E/UploadTask: Error initiating multipart upload: 11 due to The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: F508D9023E6F0432)
03-23 10:18:16.199 9907-9907/com.my.app W/System.err: com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: F50xx8D9023E6F0432), S3 Extended Request ID: PRNOVhc5NazeKHzkjE2SI83eBmXvG78ViNhIb6WtLHOlp5wkcGBxQYhGICZOMgHopBLa2erkAYE=
03-23 10:18:16.199 9907-9907/com.my.app W/System.err:     at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:720)
...

Before that exception is thrown, this gets logged:
03-23 10:18:13.868 9907-12209/com.my.app D/CognitoCachingCredentialsProvider: Loading credentials from SharedPreferences
03-23 10:18:13.947 9907-9907/com.my.app D/TransferService: Starting Transfer Service
03-23 10:18:13.949 9907-12209/com.my.app D/CreateOrderService: Stopping service
03-23 10:18:13.954 9907-9907/com.my.app D/TransferService: Network connected: true
03-23 10:18:13.963 9907-12446/com.my.app D/TransferService: Loading transfers from database
03-23 10:18:13.965 9907-12446/com.my.app D/TransferService: 0 transfers are loaded from database
03-23 10:18:13.974 9907-9907/com.my.app D/CreateOrderService: IN_PROGRESS

This is what my code looks like:
CognitoCachingCredentialsProvider credentialsProvider = new CognitoCachingCredentialsProvider(
  getApplicationContext(),
  "321",// your AWS Account id
  BuildConfig.AWS_IDENTITY_POOL_ID, // Identity Pool ID
  "arn:aws:iam::123:role/Cognito_my_appAuth_Role", // an authenticated role ARN
  "arn:aws:iam::123:role/Cognito_my_appUnauth_Role", // an unauthenticated role AR
  Regions.US_WEST_2 // Region
);
AmazonS3 s3 = new AmazonS3Client(credentialsProvider);
s3.setS3ClientOptions(S3ClientOptions.builder().setAccelerateModeEnabled(true).build());
TransferUtility transferUtility = new TransferUtility(s3, getApplicationContext());
TransferObserver observer = transferUtility.upload(
   BuildConfig.AWS_BUCKET_NAME,
   filekey + ".png",
   f
);

observer.setTransferListener(new TransferListener() {
   @Override
   public void onStateChanged(int id, TransferState state) {
      Log.d(TAG, state.toString());
   }
   // ... other event handlers
});

In my gradle file:
    compile 'com.amazonaws:aws-android-sdk-core:2.4.0'
    compile 'com.amazonaws:aws-android-sdk-s3:2.4.0'
    compile 'com.amazonaws:aws-android-sdk-cognito:2.4.0'

Any idea how I can debug this or what might be wrong? I've checked my Cognito and IAM settings like 5x.

The flow you're using for Cognito is the legacy flow.  Can you try using the new flow where the constructor does not take the accountID and the ARNs?
Also, make sure the roles are attached to the identity pool you're using.

That is what I tried initially and it wasn't working either.

Have you tried the S3TransferUtilitySample as is?
What is the attached policy for auth/unauth role?

I have, and for some reason it always works fine in the sample.
I'm closing this ticket because I haven't experienced this error recently.

authors: mlshon teekirol

repo: awslabs/aws-sdk-android-samples

license_source: bigquery


text:
Fix source directive in podfile (S3TransferManager-Sample Swift)

The specs repo at ssh://git.amazon.com:2222/pkg/AWSCocoaPodsSpecs is not publicly accessible.
This appears to be the only Podfile that was affected.

Thank you for the pull request!

authors: keatongreve yosuke-matsuda

repo: awslabs/aws-sdk-ios-samples


text:
chore(merge-back): 1.139.0

See CHANGELOG

AWS CodeBuild CI Report

CodeBuild project: githubautobuild-for-cdk-v1
Commit ID: e06299a06ef7621b97ad0c10bfa6e028b4bcab30
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: githubautobuild-for-cdk-v2
Commit ID: e06299a06ef7621b97ad0c10bfa6e028b4bcab30
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

repo: awslabs/aws-solutions-constructs


text:
Jsii runtime not turning "byref" callback arguments to native

When a ByRef object is passed out to a callback, the JSII runtimes for Java and .NET (at least) won't replace the byref tokens (nor date and enum references), leading to type conversion problems.

Closed by #395

repo: awslabs/jsii


text:
Sidetrail dockerfile

Please note that while we are transitioning from travis-ci to AWS CodeBuld, some tests are run on each platform. Non-AWS contributors will temporarily be unable to see CodeBuild results. We apologize for the inconvenience.
Issue # (if available):
#1492
Description of changes:
Updates to DSN's scripts based on PR comments.
This is a test to kickoff codebuild.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

:exclamation: No coverage uploaded for pull request head (sidetrail-dockerfile@6e7d150). Click here to learn what that means.
The diff coverage is n/a.

authors: codecov-io rday


text:
signature_algorithms_cert extension functionality

Resolved issues:
resolves #2247, resolves #1538
Description of changes:
This pr adds functionality for receiving the signature_algorithms_cert extension which allows clients to choose which signature algorithm they accept for certificate signing.
Call-outs:
This pr cannot be merged yet because it is going in the integrationv2 build, which is still being worked on. As soon as #2306 is merged I can add an edited buildspec to this pr that will set up and run the new jdk integ test. As of now, the codebuild tests will not run the new jdk test.
Testing:
I included both unit tests, functional tests, and a new integration test with a simple jdk client, as that is the only implementation I've found that sends this extension.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #2311 into main will increase coverage by 45.52%.
The diff coverage is 75.00%.

@@             Coverage Diff             @@
##             main    #2311       +/-   ##
===========================================
+ Coverage   39.00%   84.53%   +45.52%
===========================================
  Files         165      168        +3
  Lines       12944    13892      +948
===========================================
+ Hits         5049    11743     +6694
+ Misses       7895     2149     -5746

Closing this as I am splitting this pr into a pr for the extension and a pr for the integration test.

authors: codecov-commenter maddeleine


text:
Amplify failed to build

{"Status": "FAILED", "Reason": "See the details in CloudWatch Log Stream: 2023/12/29/[$LATEST]95208e471041435aaa734b13fa4795d5", "PhysicalResourceId": "2023/12/29/[$LATEST]95208e471041435aaa734b13fa4795d5", "StackId": "arn:aws:cloudformation:us-east-1:409547238403:stack/Sandbox-Accounts-for-Events/3eb14090-a612-11ee-83fa-1232a2361c67", "RequestId": "f0a34649-b710-46d0-abf1-fe2a240f305a", "LogicalResourceId": "AppUiAmplifyDeployment", "NoEcho": "FAILED", "Data": {}}
--
1703831849932
AppUiAmplifyDeployment
FAILED
2023/12/29/[$LATEST]95208e471041435aaa734b13fa4795d5
See the details in CloudWatch Log Stream: 2023/12/29/[$LATEST]95208e471041435aaa734b13fa4795d5
f0a34649-b710-46d0-abf1-fe2a240f305a
arn:aws:cloudformation:us-east-1:409547238403:stack/Sandbox-Accounts-for-Events/3eb14090-a612-11ee-83fa-1232a2361c67
FAILED

I have just re-deployed the latest version (today's commit) without issues. Please undeploy and deploy the latest version again and let me know if that worked.

Thank you working fine, can we generate accesskeys and username, password with api call instead of registering user from frontend

No, we want to enforce AWS Security Best Practices, therefore we are using IAM roles (assumed via the frontend application) instead of IAM users. As soon as a user has logged in, they can of course create IAM users with API access keys if needed.

Yeah got it, in case if we want to validate the activity done by user or if we need snapshot then..?
On 02-Jan-2024 3:00 pm, moellr @.***> wrote:
No, we want to enforce AWS Security Best Practices, therefore we are using IAM roles (assumed via the frontend application) instead of IAM users. As soon as a user has logged in, they can of course create IAM users with API access keys if needed.
—
Reply to this email directly, view it on GitHubhttps://github.com/awslabs/sandbox-accounts-for-events/issues/49#issuecomment-1873785870, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BBBAREIWQKIAD4ALP3IOJ3TYMPHUJAVCNFSM6AAAAABBGJJHESVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZTG44DKOBXGA.
You are receiving this because you authored the thread.Message ID: @.***>

If you need to validate anything in their account, you as operator can always into any account from the lease list (login icon in the right table column).
If you need to grab an EBS snapshot, you could share it with your target account (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-modifying-snapshot-permissions.html) and create a copy of it(https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-copy-snapshot.html), so is not destroyed when the source account is nuked.

Thank you, my idea is to prepare something like https://www.whizlabs.com/aws-sandbox/

authors: BharathRedsecops moellr python7878

repo: awslabs/sandbox-accounts-for-events


text:
[Feature Request] Ability to support multiple CIDR ranges for template

Is your feature request related to a problem? Please describe.
By default you can only add one CIDR by default.  We need multiple CIDR ranges for example 172.17.0.016,192.168.0.1/16 for the AccessFromCIDR.
Describe the solution you'd like
The ability to specify multiple CIDR ranges, for example 172.17.0.016,192.168.0.1/16.
Describe alternatives you've considered
Editing the cloud formation templates directly to use multiple ranges.

Hi @thedavidwhiteside,
Thanks for reaching out!
The CIDR range can be updated to include multiple ranges after the environment is provisioned. I'm not sure if that would unblock you on this issue? Or if you're specifically looking for a way to set multiple ranges when provisioning an environment?
Thanks,
Yanyu

We have multiple CIDR ranges that are allowed.  Users are not really aware of what CIDR ranges to add, so as a short term solution we have provided a list of CIDR ranges to add at launch/provision, but long term we need a way for multiple CIDR ranges by default.

Got it! I'll create a ticket for this request in our backlog. Thanks for your input!

authors: Bingjiling thedavidwhiteside

repo: awslabs/service-workbench-on-aws


text:
Smithy-Tags not present in jar consistently

For any project that uses the Smithy gradle plugin, e.g., https://github.com/awslabs/smithy-gradle-plugin/tree/main/examples/adds-tags, on running gradle clean jar, the jar has the Smithy-Tags. Running gradle jar again changes the jar and it no longer has the Smithy-Tags.

After gradle jar, running the 2nd gradle jar -info shows:
> Task :smithyBuildJar UP-TO-DATE
Caching disabled for task ':smithyBuildJar' because:
  Build cache is disabled
Skipping task ':smithyBuildJar' as it is up-to-date.
:smithyBuildJar (Thread[Execution worker for ':',5,main]) completed. Took 0.011 secs.
:processResources (Thread[Execution worker for ':',5,main]) started.

> Task :processResources UP-TO-DATE
file or directory '/Volumes/workplace/github/smithy-gradle-plugin/examples/adds-tags/src/main/resources', not found
Caching disabled for task ':processResources' because:
  Build cache is disabled
Skipping task ':processResources' as it is up-to-date.
:processResources (Thread[Execution worker for ':',5,main]) completed. Took 0.002 secs.
:classes (Thread[Execution worker for ':',5,main]) started.

> Task :classes UP-TO-DATE
Skipping task ':classes' as it has no actions.
:classes (Thread[Execution worker for ':',5,main]) completed. Took 0.0 secs.
:jar (Thread[Execution worker for ':',5,main]) started.

> Task :jar
file or directory '/Volumes/workplace/github/smithy-gradle-plugin/examples/adds-tags/build/classes/java/main', not found
Caching disabled for task ':jar' because:
  Build cache is disabled
Task ':jar' is not up-to-date because:
  Input property 'rootSpec$2$1' file /Volumes/workplace/github/smithy-gradle-plugin/examples/adds-tags/build/tmp/jar/MANIFEST.MF has changed.
file or directory '/Volumes/workplace/github/smithy-gradle-plugin/examples/adds-tags/build/classes/java/main', not found
:jar (Thread[Execution worker for ':',5,main]) completed. Took 0.007 secs.

BUILD SUCCESSFUL in 573ms
3 actionable tasks: 1 executed, 2 up-to-date

which shows that smithyBuildJar task didn't run again (UP-TO-DATE), and when the jar task runs and says .../build/tmp/jar/MANIFEST.MF has changed and so builds a new jar.

This logic in SmithyBuildJar is finding the Jar task and adding Smithy-Tags to the manifest. So when smithyBuildJar is not re-run, when the jar task runs, the MANIFEST.MF doesn't have the Smithy-Tags.

Fixed in https://github.com/awslabs/smithy-gradle-plugin/pull/47

repo: awslabs/smithy-gradle-plugin


text:
Missing --softmax-temperature T

I would like to sample from my decoder.
The docs say I can:

But I get an error:
usage: translate.py [-h] [--config CONFIG] [--input INPUT] [--input-factors INPUT_FACTORS [INPUT_FACTORS ...]] [--json-input] [--output OUTPUT] --models MODELS [MODELS ...]
                    [--checkpoints CHECKPOINTS [CHECKPOINTS ...]] [--nbest-size NBEST_SIZE] [--beam-size BEAM_SIZE] [--greedy] [--beam-search-stop {all,first}]
                    [--batch-size BATCH_SIZE] [--chunk-size CHUNK_SIZE] [--sample [SAMPLE]] [--seed SEED] [--ensemble-mode {linear,log_linear}] [--bucket-width BUCKET_WIDTH]
                    [--max-input-length MAX_INPUT_LENGTH] [--max-output-length-num-stds MAX_OUTPUT_LENGTH_NUM_STDS] [--max-output-length MAX_OUTPUT_LENGTH]
                    [--restrict-lexicon RESTRICT_LEXICON [RESTRICT_LEXICON ...]] [--restrict-lexicon-topk RESTRICT_LEXICON_TOPK] [--skip-nvs] [--nvs-thresh NVS_THRESH]
                    [--strip-unknown-words] [--prevent-unk] [--output-type {translation,score,translation_with_score,translation_with_factors,benchmark,json}]
                    [--length-penalty-alpha LENGTH_PENALTY_ALPHA] [--length-penalty-beta LENGTH_PENALTY_BETA] [--brevity-penalty-type {none,learned,constant}]
                    [--brevity-penalty-weight BREVITY_PENALTY_WEIGHT] [--brevity-penalty-constant-length-ratio BREVITY_PENALTY_CONSTANT_LENGTH_RATIO]
                    [--dtype {None,float32,float16,bfloat16,int8}] [--clamp-to-dtype] [--device-id DEVICE_ID] [--use-cpu] [--env ENV] [--tf32 TF32] [--quiet]
                    [--quiet-secondary-workers] [--no-logfile] [--loglevel {INFO,DEBUG,ERROR}] [--loglevel-secondary-workers {INFO,DEBUG,ERROR}] [--knn-index KNN_INDEX]
                    [--knn-lambda KNN_LAMBDA]
translate.py: error: unrecognized arguments: --softmax-temperature 2.0

Is this supported otherwise?

Hi Amit,
This was implemented in the old MXNet version of Sockeye. When we updated Sockeye to PyTorch (3.1.0), --softmax-temperature was one of the features we chose not to migrate. If you're interested in taking a look at the MXNet code for this feature, the pull request for that implementation is #869.
Best,
Michael

authors: AmitMY mjdenkowski

repo: awslabs/sockeye


text:
[GIT PULL] Support using a ring exclusively via registered index

This provides the userspace corresponding to the kernel patch at
https://lore.kernel.org/io-uring/3cbedc531b633af4fe8632f7276aa843b5a54875.1664123680.git.josh@joshtriplett.org/T/
.
With this change, a caller of liburing can handle a ring internally without
leaving a file descriptor open that might disrupt its caller, which makes it
safer to retrofit existing libraries with strict compatibility requirements to
use io_uring.
Libraries with even more stringent requirements (e.g. never even transiently
having a file descriptor open) will need two additional pieces:

Adding a flag to io_uring_setup to set up the ring directly as a registered
file descriptor, without ever putting it in the file descriptor table.
Supporting the initial mmap via a registered file descriptor, such as via
an io_uring_register call.

git request-pull output:
The following changes since commit 65791341a5d7e79aa601c96be9df0e7ed4603e05:

  Merge branch 'test_print_counts' of https://github.com/ddiss/liburing (2022-09-23 18:58:59 -0600)

are available in the Git repository at:

  https://github.com/joshtriplett/liburing registered-ring-close

for you to fetch changes up to 5ba0c3c73f979d73e385e1d5a8acdac4c57a9ea8:

  Add example of closing a ring fd and using it via registered index (2022-09-25 21:54:47 +0100)

----------------------------------------------------------------
Josh Triplett (5):
      src/register.c: Call __sys_io_uring_register via a helper taking the ring
      Error on duplicate ring fd registration
      Support using IORING_REGISTER_USE_REGISTERED_RING
      Support closing the ring fd and using it exclusively via registered index
      Add example of closing a ring fd and using it via registered index

 examples/Makefile               |   1 +
 examples/io_uring-close-test.c  | 123 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 man/io_uring_close_ring_fd.3    |  43 +++++++++++++++++++++++++++++++++++++++++++
 man/io_uring_register.2         |   8 ++++++++
 man/io_uring_setup.2            |   7 +++++++
 src/include/liburing.h          |   1 +
 src/include/liburing/io_uring.h |   6 +++++-
 src/liburing.map                |   1 +
 src/register.c                  | 125 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------------------------------------
 src/setup.c                     |   3 ++-
 10 files changed, 250 insertions(+), 68 deletions(-)
 create mode 100644 examples/io_uring-close-test.c
 create mode 100644 man/io_uring_close_ring_fd.3

Click to show/hide pull request guidelines
Pull Request Guidelines

To make everyone easily filter pull request from the email
notification, use [GIT PULL] as a prefix in your PR title.

[GIT PULL] Your Pull Request Title

Follow the commit message format rules below.
Follow the Linux kernel coding style (see: https://github.com/torvalds/linux/blob/master/Documentation/process/coding-style.rst).

Commit message format rules:

The first line is title (don't be more than 72 chars if possible).
Then an empty line.
Then a description (may be omitted for truly trivial changes).
Then an empty line again (if it has a description).
Then a Signed-off-by tag with your real name and email. For example:

Signed-off-by: Foo Bar <foo.bar@gmail.com>

The description should be word-wrapped at 72 chars. Some things should
not be word-wrapped. They may be some kind of quoted text - long
compiler error messages, oops reports, Link, etc. (things that have a
certain specific format).
Note that all of this goes in the commit message, not in the pull
request text. The pull request text should introduce what this pull
request does, and each commit message should explain the rationale for
why that particular change was made. The git tree is canonical source
of truth, not github.
Each patch should do one thing, and one thing only. If you find yourself
writing an explanation for why a patch is fixing multiple issues, that's
a good indication that the change should be split into separate patches.
If the commit is a fix for an issue, add a Fixes tag with the issue
URL.
Don't use GitHub anonymous email like this as the commit author:
123456789+username@users.noreply.github.com

Use a real email address!
Commit message example:
src/queue: don't flush SQ ring for new wait interface

If we have IORING_FEAT_EXT_ARG, then timeouts are done through the
syscall instead of by posting an internal timeout. This was done
to be both more efficient, but also to enable multi-threaded use
the wait side. If we touch the SQ state by flushing it, that isn't
safe without synchronization.

Fixes: https://github.com/axboe/liburing/issues/402
Signed-off-by: Jens Axboe <axboe@kernel.dk>

By submitting this pull request, I acknowledge that:

I have followed the above pull request guidelines.
I have the rights to submit this work under the same license.
I agree to a Developer Certificate of Origin (see https://developercertificate.org for more information).

Josh, this hasn't been forgotten. Let's pick this back up when 2.3 is released shortly, ditto on the kernel page as we're now outside of the merge window and can focus on 6.2 changes.

Josh, this hasn't been forgotten. Let's pick this back up when 2.3 is
released shortly, ditto on the kernel page as we're now outside of
the merge window and can focus on 6.2 changes.

Just a reminder. The 6.3 merge window will come soon. Any update on this?

Rebasing the kernel patch now.

@joshtriplett Since the kernel patch is now staged for 6.3, can you update the liburing side too?

Forgot to mention, it should also include a test verifying basic functionality and misuse.

Updated.
Looking into tests.

Adding a new exported function should also be reflected in liburing-ffi.map.

Done.

@axboe Added a test. Let me know if the test covers what you'd like covered.

Forgot to mention about the CHANGELOG file.
Add a note about this new API in the CHANGELOG file. The change can be
folded into the commit that adds the new function.

@ammarfaizi2 Done.

authors: ammarfaizi2 axboe joshtriplett

repo: axboe/liburing

license: mit


text:
Feat/add KIP token

Adding KIP token to Squid token list.
https://kip.pro/

@benjamin852 Please help review this change. Thank you!

Thank you for your review. We currently do not have a CoinGecko ID. If we obtain one, we will promptly submit a new PR to update it. Thank you! @ecdsafu

I've resolved the code conflicts. Could you help me merge it?  @ecdsafu

repo: axelarnetwork/axelar-configs


text:
Pica画质模糊

Version : 0.0.4
Step: 查看pica源的漫画
Result: 画质模糊

https://github.com/axlecho/manhuaapi/commit/263ce208e50c6b29280d98726997fef783584f34
2ffa7f09e8388c34e96e8c1ccf915603d4971002
done

repo: axlecho/MHViewer


text:
[Feature request] - Multi-tenancy on single instance of Mailpit

Hello,
I'm now maintaining project for organization that is educating girls in IT. We have 1 instance of testing application for each team (now 12 teams + 2 lecturers app).
So that means I have to spin up also 14 instances of Mailpit etc. It would be nice to spin up just 1 instance. And just by configuring it I would set up 14 accounts. Meaning 14 different SMTP credentials and 14 different UI accounts. When after login, each account would see only the own emails.
So I'm wondering if you would be OK with implementing (or even accepting PR with) multi-tenancy feature.
My idea was to reuse tenancy you support now. Meaning if you configure tenantID you cannot setup multi-tenancy and vice-versa.
Then if you setup multi-tenancy (new seciton in config), it would contain section for users and SMTP credentials. Then the rest of code would require minimal changes. As I would be reusing existing Authentication and existing tenant usage.
However, I need to handle that by this Saturday. So I might not be able to prepare PR now and I will end up spinning 14 instances anyway. But for future courses I might be preparing that.

Hi @arxeiss. This feature request has come up a couple of times in the past, but I do not believe it is "that simple". Recently I added tenancy database support which would assist in this feature, however the complication would arise on both the frontend (web UI) as well as the SMTP as to routing & access. Mailpit was never designed to be multi-user, and I suspect that in order to make it so, a huge portion would have to be rewritten, as well as potentially break backwards-compatibility (specifically SMTP and API).
That last part is a deal-breaker - if adding multi-user support meant that changes are required in the API, then this is an absolute no-go.
I understand your point - setting up multiple instances for your use-case, but you will likely have a much easier time scripting a multi-deployment script than trying to change Mailpit. I don't want to discourage you though, as I think there are some cool features that could be gained by adding multi-user, I'm just preparing you to expect complications. I would definitely consider a PR to add this functionality, provided of course it is backwards compatible.

authors: arxeiss axllent

repo: axllent/mailpit

license: MIT


text:
Support generating DotSlash manifests

Facebook open sourced an executable runner that looks pretty interesting and they happen to already have an integration for GitHub Releases[1].
1: https://dotslash-cli.com/docs/github/

whoa this is very interesting. thank you for putting it on our radar!!!

authors: ashleygwilliams msfjarvis

repo: axodotdev/cargo-dist


text:
Get started on a book

TODO:

[ ] config (Cargo.toml)
[ ] config (CLI flags)
[ ] supported workflows

...maybe others...?

fixes #128


text:
neovim-nightly cannot convert given lua table for lsp

Version confirmation

[X] Confirm

Following prerequisites

[X] Confirm

Neovim version
v0.10.0-dev-709+gce56ad2ba
Operating system/version
ubuntu 22.04
Terminal name/version
wezterm 20230420-215610-ff153a92
$TERM environment variable
xterm-256color
Branch info
main (Default/Latest)
Fetch Preferences
HTTPS (use_ssh = false)
How to reproduce the issue

update nvim-nightly
update all plugins
open any lua file

Expected behavior
fix the bug
Actual behavior
Error executing vim.schedule lua callback: /home/charles/tools/nvim/share/nvim/runtime/lua/vim/lsp.lua:1579: Cannot convert given lua table
stack traceback:
        [C]: in function 'nvim_exec_autocmds'
        /home/charles/tools/nvim/share/nvim/runtime/lua/vim/lsp.lua:1579: in function </home/charles/tool/nvim/share/nvim/runtime/lua/vim/lsp.lua:1578>

Additional information
I think it's just about some of our lsp's config not fitting with nvim-nightly, like lua_ls.
I scaned the neovim repo's issue real quick and didn't find any issue similar.
I haven't got the time to trace the issue, just put it here as a reminder.

I think it's just about some of our lsp's config not fitting with nvim-nightly, like lua_ls.

I suspect this is an upstream issue, as stack backtrace does not involve any user-related information.

Looks like it's been triggered by a new event: LspNotify in nvim-nightly.

https://github.com/neovim/neovim/pull/24411
https://github.com/neovim/neovim/pull/24469

@Jint-lzxy do u also got this error?

@CharlesChiuGit I'm usually on the latest master but haven't encountered this in the past few days... So I never knew what's going on here 😢 I have:
NVIM v0.10.0-dev-735+g8fe9f41f7-Homebrew
Build type: Release
LuaJIT 2.1.0-beta3

        系統 vimrc 設定檔: "$VIM/sysinit.vim"
              $VIM 預設值: "/usr/local/Cellar/neovim/HEAD-8fe9f41/share/nvim"
                                                            ^ this commit
Run :checkhealth for more info

Seems like it's related to my other personal config, didn't find any error on 0.10 branch of this repo.
close it for now.

@CharlesChiuGit Try debugging your lspsaga.winbar configs? I'm pretty sure this comes from the autocmd registered by Lspsaga:
執行 LspAttach Autocommands: "*"
autocommand <Lua 1150: ~/.local/share/nvim/site/lazy/lspsaga.nvim/lua/lspsaga/symbol/head.lua:165>

Error executing vim.schedule lua callback: ...r/neovim/HEAD-8fe9f41/share/nvim/runtime/lua/vim/lsp.lua:1579: Cannot convert given lua table
stack traceback:
	[C]: in function 'nvim_exec_autocmds'
	...r/neovim/HEAD-8fe9f41/share/nvim/runtime/lua/vim/lsp.lua:1579: in function <...r/neovim/HEAD-8fe9f41/share/nvim/runtime/lua/vim/lsp.lua:1578>

Reopening b/c I hope we could eschew this issue here as well.

seems like it's related to a NEW version of lspsaga config. trying to locate the issue...

Hmm is this issue still present after disabling lspsaga?

Hmm is this issue still present after disabling lspsaga?

indeed

Try replacing lsp.lua with one that is not problematic and gradually resetting those hunks to see where the problem is?

ok I found the problem, it's about my lua_ls config.
My old one: https://github.com/CharlesChiuGit/nvimdots.lua/blob/df6576052e3d72f8a76e59457c0101203c634d42/lua/modules/configs/completion/servers/lua_ls.lua
The updated version:
-- https://github.com/neovim/nvim-lspconfig/blob/master/lua/lspconfig/server_configurations/lua_ls.lua

return {
	settings = {
		Lua = {
			runtime = {
				version = "LuaJIT",
			},
			diagnostics = {
				enable = true,
				globals = { "vim" },
				disable = { "different-requires" },
			},
			workspace = {
				library = {
					[vim.fn.expand("$VIMRUNTIME/lua")] = true,
					[vim.fn.expand("$VIMRUNTIME/lua/vim/lsp")] = true,
				},
				maxPreload = 2000,
				preloadFileSize = 1000,
			},
			hint = { enable = true, setType = true },
			format = { enable = false },
			telemetry = { enable = false },
			-- Do not override treesitter lua highlighting with lua_ls's highlighting
			semantic = { enable = false },
		},
	},
}

One possible reason the old config break might b/c I used to specify Lua, luaJIT, Luarocks in my shell envㄝbut now I use asdf-vm to manage all lang's version.
Manually specify the lua_runtime might confuse nvim being confuse which lua should it listen to (I guess? I'm too tired to test now lol).
Now I remove the manually installed lua and remove specifying runtime path in lua_ls.lua. Things looks fine now.

P.S. lspsaga got nothing to do with this error, glepnir is doing a great job lol

So lspsaga is also confused lol
He's just doing what he should do - propagate the error

So lspsaga is also confused lol
He's just doing what he should do - propagate the error

lol indeed, i wasted so much time debugging lspsaga config.

lol indeed, i wasted so much time debugging lspsaga config.

Yeah autocmds are always very difficult to debug - stack backtraces (even logs) rarely yield useful information :(

authors: CharlesChiuGit Jint-lzxy

repo: ayamir/nvimdots


text:
是否考虑增加 YouCompleteMe 插件支持呢？

目前感觉代码补全还不是那么的完美，比如 golang 自动导入依赖包，函数完整提示还没有。
是否考虑集成 YouCompleteMe 插件来做代码补全呢？

以nvim-lspconfig为核心的代码补全应该是原生且未来的趋势，暂没有时间新增ycm的配置

目前感觉代码补全还不是那么的完美，比如 golang 自动导入依赖包，函数完整提示还没有。 是否考虑集成 YouCompleteMe 插件来做代码补全呢？

使用vim-go + gopls开发go应该已经能达到vscode的水平了，自动导入和函数提示也都正常，应该是你的配置或者是使用有问题

LspInstall c后面选择安装clangd，请问如何goto denfiniton ?

感谢回答。
那我打开一个工程如何自动intellisence，实现整个工程内各个模块跳转呢，比如打开webrtc？

感谢回答。 那我打开一个工程如何自动intellisence，实现整个工程内各个模块跳转呢，比如打开webrtc？

这个要视不同lsp的支持能力

authors: Shipley1105 ayamir lushenle ttbug


text:
[bug] ram usage increased fast!!!

Describe the bug
when I call it, the ram usage  increased fast! most 3Ghz
Reproduction
No response
Expected behavior
No response
Application and system information
{
  "appName": "EcoPaste",
  "appVersion": "0.0.6",
  "tauriVersion": "1.7.1",
  "platform": "Windows_NT",
  "platformArch": "x86_64",
  "platformVersion": "10.0.19045"
}

Additional context
No response

Can you explain exactly why it was created? Was a large amount of text copied? Like tens of thousands of texts?

I don't know what caused it either. I was just using it normally, with only a few paragraphs of text and screenshots. When I used the shortcut keys to call up the program, suddenly the program froze. I opened the task manager and saw that the RAM memory was occupying a lot, and there were several webview2 processes. I don't know if it was caused by Edge. This problem occasionally occurred today, but it quickly recovered. I don't know exactly what caused it

it crashed again when i restarted the program,and the ram almost 7G . Please look look.

Can you explain exactly why it was created? Was a large amount of text copied? Like tens of thousands of texts?

hi, please take attention of my issue

Can you explain exactly why it was created? Was a large amount of text copied? Like tens of thousands of texts?

hi, please take attention of my issue

when I exit and restart the program, the shortcut keys seem to be offline. but aften several minutes, it opened but frozed(can not do anything)

Download the latest beta version and try it: https://github.com/ayangweb/EcoPaste/actions/runs/10623582585
Keep the feedback coming!

No response

Okay, this will have to come down to a slow troubleshooting!

Maybe problem is the database is  too large, when i delete the EcoPaste.db, all will be right.

OK, I think it's also caused by too much data, I'll optimize the loading of data down the road, currently it's a full batch request, a bit too much of a performance loss!

authors: Kenoncloud ayangweb

repo: ayangweb/EcoPaste


text:
いいところ診断の結果に優しさを追加したい

以下の結果を追加したい
'{userName}のいいところは優しさです。あなたの優しい雰囲気や立ち振る舞いに多くの人が癒やされています。'

いいね！

1d101519abec82c451cb4a084c6266d2618ae846　で対応しました。

repo: ayaringo/assessment


text:
🛑 Bahçıvan Peynir is down

In 686613d, Bahçıvan Peynir (https://www.bahcivanpeynir.com/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Bahçıvan Peynir is back up in 3e54c9c after 1 hour, 26 minutes.

repo: aydgn/upptime


text:
SSH disconnection: broken pipe

I always get a random broken pipe from my rock64. Below is the message:
getaddrinfo: Servname not supported for ai_socktype
packet_write_wait: Connection to UNKNOWN port 0: Broken pipe
sshd_config (Server side):
ClientAliveInterval 20
ClientAliveCountMax 5
TCPKeepAlive no
ssh_config (Client side):
ServerAliveInterval 120

Been using quite a few of the different images and ssh has been solid for me with the 118 release versions.
Prob need to state which image.

Oh, that's right. I forgot to mention that. Following is the result after I run "uname -a":
Linux rock64 4.4.77-rockchip-ayufan-118 #1 SMP Thu Sep 14 21:59:24 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux

I'm not sure if it helps, but I had the same issue with an image from the Rock64 main page.
(Stock Build Debian Jessie LXDE Linux Kernel ver 4.4 [microSD Boot] [20170809] Release version)
Later I downloaded and install your xenial-mate-rock64-0.5.10-118-arm64_sd2emmc.img without this issue.

Did you customize your ssh config or sshd config?

No, I did not customize the configuration of ssh or sshd. I tried ssh under Windows with ConEmu and Cygwin_64 ssh and with Debian Stretch ssh.

authors: StuartIanNaylor lherman-cs mitag


text:
States with processed data

States with processed/cleaned data:

Alaksa
Arizona
California
Florida
Georgia
Idaho
Illinois
Indiana
Kentucky
Maryland
North Carolina
New Mexico
Ohio
Oregon
South Carolina
Tennessee
Texas
Utah
Vermont
Washington
West Virginia
Wyoming

All processed files should point to this processed directory

We have new data from Illinois (as of last month) that looks pretty clean to me, and crucially, includes the reason for separation. Can we utilize this new IL file instead of the 2023 one? https://www.dropbox.com/scl/fi/y4rcehqxii8mmfcnjr6k9/EmploymentHistory.csv?rlkey=ieuv3x89ag5fh2pkin16fo2g2&dl=0

IL has been cleaned and added above. Can you add any new data to the unprocessed issue @stecklow? They'll likely always need to be processed, even if minimally.

I went through the "national-post-db" Dropbox and have some questions and notes about some of the states.
Alaska - hold for launch?
Data is probably too incomplete for launch, not sure how many officers actually have history, too many other questions about immediate usefulness, and the state is now also being difficult with my update request so not even sure we can immediately promise anything
Florida - question
“Separation reason” column is more of an “employment status/change” column - remove “actively employed,” etc., or change name of column?
Georgia - question
“Separation reason” column is more of an “employment status/change” column - remove “actively employed,” etc., or change name of column?
Idaho - hold for launch?
Seems like columns reflect employment changes, and certifications, within an agency - hold for launch?
Illinois - no notes, but seems like a good model for the "reason for separation" column
Indiana - question
“Separation reason” column is more of an “employment status/change” column - remove “active,” or change name of column?
Iowa - question
I didn’t see Iowa in the launch data folder - which is fine, but it’s otherwise on my list, so just not sure what to list it as
Maryland - hold for launch?
The rows appear to show employment status changes, like promotions, often within agencies, rather than between - hold for launch?
North Carolina - hold for launch?
I don’t see any employment history in the index file
New Mexico - hold for launch?
Rows in data appear to show employment status or certification changes within individual departments - honestly really not sure what the index file shows
Oregon - hold for launch?
The rows appear to show employment status changes, like promotions, often within agencies, rather than between - hold for launch?
South Carolina - question
Some of the rows appear to show more employment status changes within departments, though other repeated departments appear to be different stints, which would track with our rule about contiguous service being collapsed
Tennessee - question
No notes, except to ask if things can not be all-caps (this is obviously least urgent)
Utah - question
No notes, except to ask if things can not be all-caps (this is obviously least urgent)
Vermont - hold for launch?
The rows appear to show employment status changes, like promotions, often within agencies, rather than between - hold for launch?
Washington - question
“Separation reason” column is more of an “employment status/change” column - remove “certified,” or change name of column? Also if things could not be all-caps
West Virginia - question
No notes, except to ask if things can not be all-caps (this is obviously least urgent)
Wyoming - question
Not sure “separation reason” is actually showing that - should we change the name of the column?

This all makes sense to me. I named them all "separation reason" because of the list in this issue here #14 . Can you suggest a different name for each of the columns?
All of the states that we're going to launch with still need to be normalized for casing, column names, date formatting, etc. This list is just comprised of states that I had to post-process from the BLN index files. Normalizing things according to our schema conventions is still on the list of things to do before launch.
For states like Oregon where the rows aren't collapsed, Tarak is still working on his open issue.
AK and IN were not going to be on the launch list. I can remove the others you listed as well.

@stecklow to keep this issue focused just on what data has been processed so far, I just created a new issue here where I've rehashed your list above about what should be included in the launch #19 .

This all makes sense to me. I named them all "separation reason" because of the list in this issue here "Reason for Separation" Breakdown #14. Can you suggest a different name for each of the columns?
All of the states that we're going to launch with still need to be normalized for casing, column names, date formatting, etc. This list is just comprised of states that I had to post-process from the BLN index files. Normalizing things according to our schema conventions is still on the list of things to do before launch.
For states like Oregon where the rows aren't collapsed, Tarak is still working on his open issue.
The Iowa data referenced in the States with unprocessed data  #12 is raw data from Ben that hasn't been cleaned, right?
AK and IN were not going to be on the launch list because they only have start_date values. I can remove the others you listed as well.

Thanks Ayyub. I don't know that I see @tarakc02's open issue about row collapsing, so I'm actually not sure if the issues with Idaho, Maryland, New Mexico, and Vermont are in the same boat as Oregon, and maybe should still be here. I made a longer response in #19 with some broader thoughts.

hi all, i just re-opened #6 based on this discussion, but that code is and has been ready to go!

Just added notes to #6 . Thank you!

authors: ayyubibrahimi stecklow tarakc02

repo: ayyubibrahimi/us-post-data


text:
syntax

(dammit)

Spotted during wiring up molecule tests on a different branch...

repo: azavea/ansible-buildkite-agent