text:
Repeated s3.to_parquet writes with index=True raise InvalidSchemaConvergence on s3.read_parquet with validate_schema=True

Describe the bug
When writing using wr.s3.to_parquet(..., index=True, ...) multiple times and reading the data back using wr.s3.read_parquet(..., validate_schema=True, ...), an InvalidSchemaConversion is raised.
Find below the full environment.

# packages in environment at /home/rschmidtke/miniforge3/envs/wrangler35:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
aws-c-auth                0.7.11               h0b4cabd_1    conda-forge
aws-c-cal                 0.6.9                h14ec70c_3    conda-forge
aws-c-common              0.9.12               hd590300_0    conda-forge
aws-c-compression         0.2.17               h572eabf_8    conda-forge
aws-c-event-stream        0.4.1                h97bb272_2    conda-forge
aws-c-http                0.8.0                h9129f04_2    conda-forge
aws-c-io                  0.14.0               hf8f278a_1    conda-forge
aws-c-mqtt                0.10.1               h2b97f5f_0    conda-forge
aws-c-s3                  0.4.9                hca09fc5_0    conda-forge
aws-c-sdkutils            0.1.13               h572eabf_1    conda-forge
aws-checksums             0.1.17               h572eabf_7    conda-forge
aws-crt-cpp               0.26.0               h04327c0_8    conda-forge
aws-sdk-cpp               1.11.210            hba3e011_10    conda-forge
awswrangler               3.5.2              pyhd8ed1ab_0    conda-forge
black                     24.1.1          py311h38be061_0    conda-forge
boto3                     1.34.31            pyhd8ed1ab_0    conda-forge
botocore                  1.34.31            pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py311hb755f60_1    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.26.0               hd590300_0    conda-forge
ca-certificates           2023.11.17           hbcca054_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
glog                      0.6.0                h6f12383_0    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
isort                     5.13.2             pyhd8ed1ab_0    conda-forge
jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libabseil                 20230802.1      cxx17_h59595ed_0    conda-forge
libarrow                  14.0.2           h84dd17c_4_cpu    conda-forge
libarrow-acero            14.0.2           h59595ed_4_cpu    conda-forge
libarrow-dataset          14.0.2           h59595ed_4_cpu    conda-forge
libarrow-flight           14.0.2           hdc44a87_4_cpu    conda-forge
libarrow-flight-sql       14.0.2           hfbc7f12_4_cpu    conda-forge
libarrow-gandiva          14.0.2           hacb8726_4_cpu    conda-forge
libarrow-substrait        14.0.2           hfbc7f12_4_cpu    conda-forge
libblas                   3.9.0           21_linux64_openblas    conda-forge
libbrotlicommon           1.1.0                hd590300_1    conda-forge
libbrotlidec              1.1.0                hd590300_1    conda-forge
libbrotlienc              1.1.0                hd590300_1    conda-forge
libcblas                  3.9.0           21_linux64_openblas    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcurl                   8.5.0                hca28451_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_4    conda-forge
libgfortran-ng            13.2.0               h69a702a_4    conda-forge
libgfortran5              13.2.0               ha4646dd_4    conda-forge
libgomp                   13.2.0               h807b86a_4    conda-forge
libgoogle-cloud           2.12.0               hef10d8f_5    conda-forge
libgrpc                   1.60.0               h74775cd_1    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
liblapack                 3.9.0           21_linux64_openblas    conda-forge
libllvm15                 15.0.7               hb3ce162_4    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnl                     3.9.0                hd590300_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libnuma                   2.0.16               h0b41bf4_1    conda-forge
libopenblas               0.3.26          pthreads_h413a1c8_0    conda-forge
libparquet                14.0.2           h352af49_4_cpu    conda-forge
libprotobuf               4.25.1               hf27288f_0    conda-forge
libre2-11                 2023.06.02           h7a70373_0    conda-forge
libsqlite                 3.44.2               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_4    conda-forge
libthrift                 0.19.0               hb90f79a_1    conda-forge
libutf8proc               2.8.0                h166bdaf_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.12.4               h232c23b_1    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
mypy_extensions           1.0.0              pyha770c72_0    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
numpy                     1.26.3          py311h64a7726_0    conda-forge
openssl                   3.2.1                hd590300_0    conda-forge
orc                       1.9.2                h7829240_1    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pandas                    2.2.0           py311h320fe9a_0    conda-forge
pathspec                  0.12.1             pyhd8ed1ab_0    conda-forge
pip                       23.3.2             pyhd8ed1ab_0    conda-forge
platformdirs              4.2.0              pyhd8ed1ab_0    conda-forge
pyarrow                   14.0.2          py311h39c9aba_4_cpu    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.7          hab00c5b_1_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-tzdata             2023.4             pyhd8ed1ab_0    conda-forge
python_abi                3.11                    4_cp311    conda-forge
pytz                      2023.4             pyhd8ed1ab_0    conda-forge
rdma-core                 50.0                 hd3aeb46_0    conda-forge
re2                       2023.06.02           h2873b5e_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
s2n                       1.4.1                h06160fa_0    conda-forge
s3transfer                0.10.0             pyhd8ed1ab_0    conda-forge
setuptools                69.0.3             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
typing-extensions         4.9.0                hd8ed1ab_0    conda-forge
typing_extensions         4.9.0              pyha770c72_0    conda-forge
tzdata                    2023d                h0c530f3_0    conda-forge
ucx                       1.15.0               h75e419f_3    conda-forge
urllib3                   1.26.18            pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

How to Reproduce
import awswrangler as wr
import boto3
import pandas as pd

PATH = "..."
DATABASE = "..."
TABLE = "..."

def main() -> None:
    boto3_session = boto3.Session(region_name="eu-central-1")
    df = pd.DataFrame({"idx": [1, 2, 3], "val": [1.0, 2.0, 3.0]})
    df = df.set_index("idx")

    for iteration in range(2):
        print("Iteration", iteration)

        wr.s3.to_parquet(
            df,
            path=PATH,
            index=True,
            boto3_session=boto3_session,
            dataset=True,
            database=DATABASE,
            table=TABLE,
        )

        wr.s3.read_parquet(path=PATH, boto3_session=boto3_session, validate_schema=True)

if __name__ == "__main__":
    main()

# Iteration 0
/home/rschmidtke/miniforge3/envs/wrangler35/lib/python3.11/site-packages/awswrangler/_distributed.py:104: FutureWarning: promote has been superseded by mode='default'.
  return cls.dispatch_func(func)(*args, **kw)
# Iteration 1
Traceback (most recent call last):
  File "/home/rschmidtke/workspace/sd/wranglertest/wranglertest.py", line 30, in <module>
    main()
  File "/home/rschmidtke/workspace/sd/wranglertest/wranglertest.py", line 26, in main
    wr.s3.read_parquet(path=path, boto3_session=boto3_session, validate_schema=True)
  File "/home/rschmidtke/miniforge3/envs/wrangler35/lib/python3.11/site-packages/awswrangler/_utils.py", line 178, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/rschmidtke/miniforge3/envs/wrangler35/lib/python3.11/site-packages/awswrangler/_config.py", line 735, in wrapper
    return function(**args)
           ^^^^^^^^^^^^^^^^
  File "/home/rschmidtke/miniforge3/envs/wrangler35/lib/python3.11/site-packages/awswrangler/s3/_read_parquet.py", line 499, in read_parquet
    schema = metadata_reader.validate_schemas(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rschmidtke/miniforge3/envs/wrangler35/lib/python3.11/site-packages/awswrangler/s3/_read.py", line 218, in validate_schemas
    schema = self._validate_schemas_from_files(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rschmidtke/miniforge3/envs/wrangler35/lib/python3.11/site-packages/awswrangler/s3/_read.py", line 204, in _validate_schemas_from_files
    return _validate_schemas(schemas, validate_schema)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rschmidtke/miniforge3/envs/wrangler35/lib/python3.11/site-packages/awswrangler/s3/_read.py", line 311, in _validate_schemas
    raise exceptions.InvalidSchemaConvergence(
awswrangler.exceptions.InvalidSchemaConvergence: At least 2 different schemas were detected:
    1 - val: double
-- schema metadata --
pandas: '{"index_columns": [], "column_indexes": [{"name": null, "field_n' + 301
    2 - val: double
idx: int64
-- schema metadata --
pandas: '{"index_columns": ["idx"], "column_indexes": [{"name": null, "fi' + 409.

Expected behavior
The reads to succeed, as is the case with awswrangler=3.4.2, see the below the output of above script and environment it is run in.
# Iteration 0
/home/rschmidtke/miniforge3/envs/wrangler34/lib/python3.11/site-packages/awswrangler/_distributed.py:105: FutureWarning: promote has been superseded by mode='default'.
  return cls.dispatch_func(func)(*args, **kw)
# Iteration 1
/home/rschmidtke/miniforge3/envs/wrangler34/lib/python3.11/site-packages/awswrangler/_distributed.py:105: FutureWarning: promote has been superseded by mode='default'.
  return cls.dispatch_func(func)(*args, **kw)

# packages in environment at /home/rschmidtke/miniforge3/envs/wrangler34:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
aws-c-auth                0.7.11               h0b4cabd_1    conda-forge
aws-c-cal                 0.6.9                h14ec70c_3    conda-forge
aws-c-common              0.9.12               hd590300_0    conda-forge
aws-c-compression         0.2.17               h572eabf_8    conda-forge
aws-c-event-stream        0.4.1                h97bb272_2    conda-forge
aws-c-http                0.8.0                h9129f04_2    conda-forge
aws-c-io                  0.14.0               hf8f278a_1    conda-forge
aws-c-mqtt                0.10.1               h2b97f5f_0    conda-forge
aws-c-s3                  0.4.9                hca09fc5_0    conda-forge
aws-c-sdkutils            0.1.13               h572eabf_1    conda-forge
aws-checksums             0.1.17               h572eabf_7    conda-forge
aws-crt-cpp               0.26.0               h04327c0_8    conda-forge
aws-sdk-cpp               1.11.210            hba3e011_10    conda-forge
awswrangler               3.4.2              pyhd8ed1ab_0    conda-forge
boto3                     1.34.31            pyhd8ed1ab_0    conda-forge
botocore                  1.34.31            pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py311hb755f60_1    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.26.0               hd590300_0    conda-forge
ca-certificates           2023.11.17           hbcca054_0    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
glog                      0.6.0                h6f12383_0    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libabseil                 20230802.1      cxx17_h59595ed_0    conda-forge
libarrow                  14.0.2           h84dd17c_4_cpu    conda-forge
libarrow-acero            14.0.2           h59595ed_4_cpu    conda-forge
libarrow-dataset          14.0.2           h59595ed_4_cpu    conda-forge
libarrow-flight           14.0.2           hdc44a87_4_cpu    conda-forge
libarrow-flight-sql       14.0.2           hfbc7f12_4_cpu    conda-forge
libarrow-gandiva          14.0.2           hacb8726_4_cpu    conda-forge
libarrow-substrait        14.0.2           hfbc7f12_4_cpu    conda-forge
libblas                   3.9.0           21_linux64_openblas    conda-forge
libbrotlicommon           1.1.0                hd590300_1    conda-forge
libbrotlidec              1.1.0                hd590300_1    conda-forge
libbrotlienc              1.1.0                hd590300_1    conda-forge
libcblas                  3.9.0           21_linux64_openblas    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcurl                   8.5.0                hca28451_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_4    conda-forge
libgfortran-ng            13.2.0               h69a702a_4    conda-forge
libgfortran5              13.2.0               ha4646dd_4    conda-forge
libgomp                   13.2.0               h807b86a_4    conda-forge
libgoogle-cloud           2.12.0               hef10d8f_5    conda-forge
libgrpc                   1.60.0               h74775cd_1    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
liblapack                 3.9.0           21_linux64_openblas    conda-forge
libllvm15                 15.0.7               hb3ce162_4    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnl                     3.9.0                hd590300_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libnuma                   2.0.16               h0b41bf4_1    conda-forge
libopenblas               0.3.26          pthreads_h413a1c8_0    conda-forge
libparquet                14.0.2           h352af49_4_cpu    conda-forge
libprotobuf               4.25.1               hf27288f_0    conda-forge
libre2-11                 2023.06.02           h7a70373_0    conda-forge
libsqlite                 3.44.2               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_4    conda-forge
libthrift                 0.19.0               hb90f79a_1    conda-forge
libutf8proc               2.8.0                h166bdaf_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.12.4               h232c23b_1    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
numpy                     1.26.3          py311h64a7726_0    conda-forge
openssl                   3.2.1                hd590300_0    conda-forge
orc                       1.9.2                h7829240_1    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pandas                    2.2.0           py311h320fe9a_0    conda-forge
pip                       23.3.2             pyhd8ed1ab_0    conda-forge
pyarrow                   14.0.2          py311h39c9aba_4_cpu    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.7          hab00c5b_1_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-tzdata             2023.4             pyhd8ed1ab_0    conda-forge
python_abi                3.11                    4_cp311    conda-forge
pytz                      2023.4             pyhd8ed1ab_0    conda-forge
rdma-core                 50.0                 hd3aeb46_0    conda-forge
re2                       2023.06.02           h2873b5e_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
s2n                       1.4.1                h06160fa_0    conda-forge
s3transfer                0.10.0             pyhd8ed1ab_0    conda-forge
setuptools                69.0.3             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
typing-extensions         4.9.0                hd8ed1ab_0    conda-forge
typing_extensions         4.9.0              pyha770c72_0    conda-forge
tzdata                    2023d                h0c530f3_0    conda-forge
ucx                       1.15.0               h75e419f_3    conda-forge
urllib3                   1.26.18            pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

Your project
No response
Screenshots
No response
OS
Ubuntu 20.04.6 LTS (WSL v2)
Python version
Python 3.11.7
AWS SDK for pandas version
awswrangler 3.5.2 pyhd8ed1ab_0 conda-forge
Additional context
No response

Ran a bisect and it seems this PR is the culprit: https://github.com/aws/aws-sdk-pandas/pull/2528
commit 67efc064a359518b77288c7e52057d8ab1d1dc9c
Author: kukushking <kukushkin.anton@gmail.com>
Date:   Wed Nov 29 23:33:03 2023 +0000

    feat: Support partitioning by index cols (#2528)

I'll take a look, but maybe @kukushking already has an idea?

My downstream tests are now succeeding, so I have put this PR back into ready for review.

repo: aws/aws-sdk-pandas


text:
Use of internal package not allowed

When I try to test or run my app, I get the following error:
../../../github.com/awslabs/aws-xray-sdk-go/xray/segment.go:19:2: use of internal package not allowed
This is after simply getting the package, then adding it to a single endpoint:
http.Handle("/status/check", xray.Handler(xray.NewFixedSegmentNamer("IYFSlackBot"), http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprint(w, "<p style="color:green;">Connection to Integrations API Healthy")
}))))

@mikkergimenez, thank you for reaching out to us about your question. Could you tell me more how you instrument X-Ray Go SDK to your application? It seems like your provided example is similar to what we wrote in README and I was unable to reproduce your issue.

Hello luluzhao,
I believe this is related to the following line
https://github.com/aws/aws-xray-sdk-go/blob/master/xray/segment.go#L19
As go 1.4 seems to have added a rule about importing packages with the name internal:
https://stackoverflow.com/questions/41571946/internal-packages-in-go
https://golang.org/doc/go1.4#internalpackages
I'm not sure why others aren't experiencing this.  My addition of the xray Handler was pretty basic, I just included into an app that was using http as a status check endpoint.

@mikkergimenez , could you tell me which go version are you currently using?

Might want to check this issue out: https://github.com/aws/aws-sdk-go/issues/340

source: gharchive/issue

authors: luluzhao mikkergimenez tskinn

repo: aws/aws-xray-sdk-go

license_type: permissive

license_source: github-api


text:
add builds for tink-actions

Issue #, if available:
This adds builds for some of the actions under https://github.com/tinkerbell/hub/tree/main/actions.
Description of changes:
Couple oddities in here, but nothing too out of the norm:

Looking at upstream builds for these actions, they use cgo, however I am pretty sure this is not necessary they are just doing this to make sure the binary is static, which we can do without cgo just like we build everything else.  This should be tested to validate. https://github.com/tinkerbell/hub/blob/main/actions/cexec/v1/Dockerfile
Since there are a number of binaries we are building here, I used eval and a function to simplify adding the probably target overrides for go-mod/licenses/attributions.  Its not the most clean looking, however, feels better in the long run to avoid duplicating these blocks for each binary. Open to thoughts tho
Went with creating a new attribution for each binary since they each have their own go.mod.  I dont love this, but the licenses and attribution code is not setup to support merging multiple go.mods.

Some remaining todos:

[x] prow jobs
[x] cdk changes
[x] fill out readme

/hold
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/override pbnj-tooling-presubmit

/override pbnj-tooling-presubmit

/unhold
/lgtm

source: gharchive/pull-request

authors: jaxesn pokearu

repo: aws/eks-anywhere-build-tooling


text:
Adding new test for detecting workload cluster rollingupgrade

Issue #, if available:
Resolves https://github.com/aws/eks-anywhere/issues/3543
Description of changes:
The test performs a management cluster upgrade to a newer EKS-A version and checks if the workload cluster also observed a rolling upgrade.
Testing (if applicable):
Tested end to end and observed expected rollingupgrade detected. I was able to work around the current nutanix bundles issue by manually applying eks-a components to the mgmt cluster during the mgmt cluster's upgrade.
2022-10-06T12:27:13.094-0400	V2	Executing command	{"cmd": "/usr/local/bin/docker exec -i eksa_1665072583805575000 kind delete cluster --name eksa-test-047cdee-eks-a-cluster"}
2022-10-06T12:27:24.380-0400	V0	🎉 Cluster upgraded!
2022-10-06T12:27:24.380-0400	V4	Task finished	{"task_name": "delete-kind-cluster", "duration": "13.6399998s"}
2022-10-06T12:27:24.381-0400	V4	----------------------------------
2022-10-06T12:27:24.381-0400	V4	Tasks completed	{"duration": "17m39.245482653s"}
2022-10-06T12:27:24.400-0400	V3	Cleaning up long running container	{"name": "eksa_1665072583805575000"}
2022-10-06T12:27:24.400-0400	V2	Executing command	{"cmd": "/usr/local/bin/docker rm -f -v eksa_1665072583805575000"}
2022-10-06T12:28:25.165-0400	V2	e2e	Executing command	{"cmd": "/usr/local/bin/docker exec -i eksa_1665071770892974000 kubectl get machines -o json --kubeconfig eksa-test-047cdee/eksa-test-047cdee-eks-a-cluster.kubeconfig --selector=cluster.x-k8s.io/cluster-name=eksa-test-047cdee-w-0 --namespace eksa-system"}
    workload_clusters_test.go:59: Found CAPI machines of workload cluster were changed after upgrading management cluster
    cluster.go:641: Skipping VM cleanup
2022-10-06T12:28:29.797-0400	V3	e2e	Cleaning up long running container	{"name": "eksa_1665071773164462000"}
2022-10-06T12:28:29.798-0400	V2	e2e	Executing command	{"cmd": "/usr/local/bin/docker rm -f -v eksa_1665071773164462000"}
    cluster.go:641: Skipping VM cleanup
2022-10-06T12:28:30.828-0400	V3	e2e	Cleaning up long running container	{"name": "eksa_1665071770892974000"}
2022-10-06T12:28:30.830-0400	V2	e2e	Executing command	{"cmd": "/usr/local/bin/docker rm -f -v eksa_1665071770892974000"}
--- FAIL: TestCloudStackKubernetes121ManagementClusterUpgradeFromLatest (1942.06s)
FAIL

The rolling upgrade is expected but undesirable due to https://github.com/aws/eks-anywhere/pull/3402
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/cherrypick release-0.12

/cherrypick release-0.11

repo: aws/eks-anywhere


text:
Revert "Update README.md conformance notification"

Reverts https://github.com/aws/eks-anywhere/pull/8145 because k8s v1.28 logos are missing
Related issue: https://github.com/cncf/artwork/issues/457

/approve


text:
removes golang 1.16 and changes default version to 1.20

Issue #, if available:
Description of changes:
We are not building with 1.16 across eks-distro and anywhere anymore.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/lgtm

/approve

repo: aws/eks-distro-build-tooling


text:
adds upx compression to our biggest bins in builder-base

Issue #, if available:
Description of changes:
@abhay-krishna suggested we see what happens if we use upx to compress some of our larger bins, esp the golang statically compiled ones.  I gave it a shot and actually it does add pretty decent savings in terms of space, esp on the decompressed final image, it ends up going from ~3.8GB to ~2.4GB.  Some of this is from the python and golang cleanup i am also doing here, but a good chunk is the upx compression.  The download size savings is probably 300-400MB, just per the ECR ui in my dev account.
I have tested this by building all of the eks-a stack. I feel pretty good that that compressed binaries do not cause any issues...
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/lgtm

/approve

/lgtm

authors: abhay-krishna jaxesn rcrozean


text:
update prow jobs for 1.25 and 1.26 deprecation

Issue #, if available:
Description of changes:
updates and/or removes prow jobs with deprecation of 1.25 and 1.26
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/lgtm

/lgtm

/approve

authors: bhavi-koduru rcrozean vineeth-bandi

repo: aws/eks-distro-prow-jobs


text:
Downgrade from v0.32.6 to v0.31.2 is stuck

Description
Observed Behavior:
This issue is related to https://github.com/aws/karpenter-provider-aws/issues/5578 where I'm finding it difficult to reproduce the upgrade bug because the downgrade to v0.31.2 gets stuck when trying to terminate the NodeClaims with this error in the logs.
karpenter-7d6fdd79c8-w5j65 controller {"level":"INFO","time":"2024-02-06T19:18:50.231Z","logger":"controller","message":"k8s.io/client-go@v0.26.6/tools/cache/reflector.go:169: failed to list *v1beta1.NodePool: nodepools.karpenter.sh is forbidden: User \"system:serviceaccount:karpenter:karpenter\" cannot list resource \"nodepools\" in API group \"karpenter.sh\" at the cluster scope\n","commit":"709030d"}
karpenter-7d6fdd79c8-w5j65 controller {"level":"ERROR","time":"2024-02-06T19:18:50.231Z","logger":"controller","message":"k8s.io/client-go@v0.26.6/tools/cache/reflector.go:169: Failed to watch *v1beta1.NodePool: failed to list *v1beta1.NodePool: nodepools.karpenter.sh is forbidden: User \"system:serviceaccount:karpenter:karpenter\" cannot list resource \"nodepools\" in API group \"karpenter.sh\" at the cluster scope\n","commit":"709030d"}

This suggests that the permissions are insufficient. We're using ArgoCD to sync changes so likely what is happening is that the ClusterRoles are changing to the previous version before the nodes are done draining.
Output of kubectl get nodes
ip-10-16-17-64.us-west-2.compute.internal    Ready,SchedulingDisabled   <none>   27m    v1.24.17-eks-5e0fdde

Here are the Events for that node
Events:
  Type     Reason                   Age                From                     Message
  ----     ------                   ----               ----                     -------
  Normal   Starting                 28m                kube-proxy
  Warning  InvalidDiskCapacity      28m                kubelet                  invalid capacity 0 on image filesystem
  Normal   NodeHasSufficientMemory  28m (x2 over 28m)  kubelet                  Node ip-10-16-17-64.us-west-2.compute.internal status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    28m (x2 over 28m)  kubelet                  Node ip-10-16-17-64.us-west-2.compute.internal status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     28m (x2 over 28m)  kubelet                  Node ip-10-16-17-64.us-west-2.compute.internal status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced  28m                kubelet                  Updated Node Allocatable limit across pods
  Normal   Starting                 28m                kubelet                  Starting kubelet.
  Normal   ControllerVersionNotice  28m                vpc-resource-controller  The node is managed by VPC resource controller version v1.4.4
  Normal   NodeReady                28m                kubelet                  Node ip-10-16-17-64.us-west-2.compute.internal status is now: NodeReady
  Normal   NodeTrunkInitiated       28m                vpc-resource-controller  The node has trunk interface initialized successfully
  Normal   NodeNotSchedulable       23m                kubelet                  Node ip-10-16-17-64.us-west-2.compute.internal status is now: NodeNotSchedulable

Expected Behavior:
The upgrade docs state that only v0.31.2 or v0.31.3 are the only versions that can be rolled back from v0.32.0
https://karpenter.sh/preview/upgrading/upgrade-guide/#upgrading-to-v0320
Reproduction Steps (Please include YAML):
Downgrade the helm chart from v0.32.6 to v0.31.2.
Please find the YAML for both versions here: https://gist.github.com/apeduru-patreon/bdc4f399cdabdffa59f7322bc9c4d790
Versions:

Chart Version: v0.32.6 -> v0.31.2
Kubernetes Version (kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:49:13Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.17-eks-5e0fdde", GitCommit:"221f136737816ce068aa975f082cd0bf81c7ad5f", GitTreeState:"clean", BuildDate:"2024-01-02T20:35:57Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

I've again tested this on v0.32.0 and I'm seeing the same error message regarding permission errors show up in the logs

I've again tested this on v0.32.0 and I'm seeing the same error message regarding permission errors show up in the logs

Is this the full log output from downgrading the nodes (you get that log message at startup) or you get that message after Karpenter runs for a bit on v0.31.x?

I see that log message after nodes have initiated a drain whilst syncing a Chart. Presumably Karpenter is running on v0.31.2 at this point.

It definitely looks like there may be a bug with Karpenter attempting to grab a NodePool in v0.31.2 when there was a previously-provisioned NodeClaim under a NodePool. As a workaround for now, if you add permission to the ClusterRole for Karpenter to be able to get NodePools, the rollback should succeed.

I'll note also that I'm seeing this for the EC2NodeClass API as well. I think this might be error logging for all v1beta1 APIs

Looking at where you are seeing this error with the informers, it looks like this may be occurring with the newer version of Karpenter (v0.32.0). Are you sure that your permissions were maintained when going between these two versions? From the looks of it, it seems like you may have upgraded back to v0.32.0 and the permission for the ClusterRole on NodePool and EC2NodeClass were dropped.

Hey @apeduru-patreon, we have an updated version of the chart that fixes the issue with the rollback. Are you willing to try KARPENTER_VERSION=v0-bea8ff219b209db8f9a9344533dd00bb72771bbf and confirm that it fixes your issue during rollback?
You can find the instructions for installing this version here: https://karpenter.sh/docs/upgrading/compatibility/#snapshot-releases

Closing as v0.31.4 is released with the fix.

authors: apeduru-patreon engedaam jonathan-innis

repo: aws/karpenter-provider-aws


text:
chore: Bump staging/v1 with main

Fixes #N/A
Description

Bump staging/v1 with main

How was this change tested?

N/A

Does this change impact docs?

[ ] Yes, PR includes docs updates
[ ] Yes, issue opened: #
[x] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Pull Request Test Coverage Report for Build 9572616601
Details

13 of 26 (50.0%)  changed or added relevant lines in 5 files are covered.
4 unchanged lines in 4 files lost coverage.
Overall coverage decreased (-0.04%) to 82.437%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pkg/controllers/nodeclaim/garbagecollection/controller.go
3
6
50.0%

pkg/controllers/providers/instancetype/controller.go
3
6
50.0%

pkg/controllers/providers/pricing/controller.go
3
6
50.0%

pkg/controllers/interruption/controller.go
3
7
42.86%

Files with Coverage Reduction
New Missed Lines
%

pkg/controllers/providers/pricing/controller.go
1
77.78%

pkg/controllers/interruption/controller.go
1
72.83%

pkg/controllers/providers/instancetype/controller.go
1
60.71%

pkg/controllers/nodeclaim/garbagecollection/controller.go
1
71.88%

Totals

Change from base Build 9568229212:
-0.04%

Covered Lines:
5534

Relevant Lines:
6713

💛  - Coveralls

authors: coveralls engedaam


text:
Karpenter does not consolidate nodes with multiple coredns pods.

Tell us about your request
Karpenter support Pod(Anti)Affinity setting and coredns deployment have PodAntiAffinity setting by default.
podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: k8s-app
                  operator: In
                  values:
                  - kube-dns
              topologyKey: kubernetes.io/hostname
            weight: 100

If multiple coredns pods are scheduled on the same node, Karpenter will respect PodAntiAffinity when determining whether a given node can be consolidated, i.e. it will assume that multiple nodes need to be started.
Karpenter will now notify me as an event why a node cannot be consolidated. For example, a node with 9 coredns pods will  show up like this:
Events:
  Type    Reason            Age                     From       Message
  ----    ------            ----                    ----       -------
  Normal  Unconsolidatable  6m36s (x16 over 5h53m)  karpenter  can't remove without creating 9 nodes

I think it's rare that multiple coredns with PodAntiAffinity are scheduled on a single node, but I use cluster-proportional-autoscaler so that coredns scales out with the number of nodes. I want coredns to spread out  the nodes as much as possible, but I don't want to start as many nodes as there are coredns pods. I think the same situation can happen without using cluster-proportional-autoscaler.
Is there a solution to consolidate the excess nodes with only coredns pods? Or could we consider adding the annotation to ignore certain pods from the consolidate evaluation?
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
I want the nodes that have a particular pods with PodAntiAffinity (coredns in this case) scheduled to be consolidated when they have excess resources.
Are you currently working around this issue?

Prevent coredns from being scheduled on Provisioner nodes that scale out significantly.
coredns is scheduled on other nodes, but this doesn't solve the root of the problem.

Set the total number of coredns pods to be less than the number of nodes.
Karpenter consolidate nodes without coredns pods, and cluster-proportional-autoscaler reduces the number of coredns as the number of nodes decreases, so there is likely a chain of nodes merging. But this doesn't solve the problem either.

Remove PodAntiAffinity from coredns.
This also doesn't solve the problem and causes availability issues.

Additional Context
Related issue: #1532
This release fixes an issue that prevents karpenter from deprovisioning nodes with a coredns pod.
Attachments
No response
Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

I want coredns to spread out the nodes as much as possible, but I don't want to start as many nodes as there are coredns pods.

This is a fundamental limitation of Kubernetes scheduling. Your best bet may be topology spread instead of antiaffinity.

I think it's rare that multiple coredns with PodAntiAffinity are scheduled on a single node,

This can only happen if there's a single node with room, where preferences are ignored.

I think there's a chance that we should consider consolidating for preferred antiaffinity, but only if the pods can reschedule else where.

Thanks for your response.
As you say, the behavior I want coredns scheduling should be topology spread, not anti affinity.
I tried replacing anti affinity with topology spread in coredns and then Karpenter consolidated the target  node.
I will close this issue, thanks.

Is the suggestion to use topology spread constraints instead of pod anti-affinity still correct here? From what I understand, both block consolidation and https://github.com/kubernetes-sigs/karpenter/issues/666, which mentions both, supports my understanding.

authors: ellistarn kenny-monster zensai3805

repo: aws/karpenter


text:
Add Support for Nodeaffinity Key "failure-domain.beta.kubernetes.io/zone" in karpenter schedule logic.

Description
What problem are you trying to solve?
We have 300 clusters and in each cluster we have around 500 PV's which has nodeaffinity Key as "failure-domain.beta.kubernetes.io/zone".
    {
        "apiVersion": "v1",
        "kind": "PersistentVolume",
        "metadata": {
            "annotations": {
                "kubernetes.io/createdby": "aws-ebs-dynamic-provisioner",
                "pv.kubernetes.io/bound-by-controller": "yes",
                "pv.kubernetes.io/migrated-to": "ebs.csi.aws.com",
                "pv.kubernetes.io/provisioned-by": "kubernetes.io/aws-ebs"
            },
            "creationTimestamp": "2021-10-20T11:16:35Z",
            "finalizers": [
                "kubernetes.io/pv-protection"
            ],
            "labels": {
                "failure-domain.beta.kubernetes.io/region": "us-east-1",
                "failure-domain.beta.kubernetes.io/zone": "us-east-1a"
            },
            "name": "pvc-0ae3077c-2abf-4178-bc49-xxxxxxxxx",
            "resourceVersion": "1023838636",
            "uid": "1905b103-8208-4bee-ad89-4351be15d6d3"
        },
        "spec": {
            "accessModes": [
                "ReadWriteOnce"
            ],
            "awsElasticBlockStore": {
                "fsType": "ext4",
                "volumeID": "aws://us-east-1a/vol-XXXXXXXXXXX"
            },
            "capacity": {
                "storage": "1Gi"
            },
            "claimRef": {
                "apiVersion": "v1",
                "kind": "PersistentVolumeClaim",
                "name": "XX-service-cert-pvc",
                "namespace": "infrastructure-qXX-XXX-service",
                "resourceVersion": "539457602",
                "uid": "0ae3077c-2abf-4178-bc49-dbc136632e6a"
            },
            "mountOptions": [
                "debug"
            ],
            "nodeAffinity": {
                "required": {
                    "nodeSelectorTerms": [
                        {
                            "matchExpressions": [
                                {
                                    "key": "failure-domain.beta.kubernetes.io/zone",
                                    "operator": "In",
                                    "values": [
                                        "us-east-1a"
                                    ]
                                },
                                {
                                    "key": "failure-domain.beta.kubernetes.io/region",
                                    "operator": "In",
                                    "values": [
                                        "us-east-1"
                                    ]
                                }
                            ]
                        }
                    ]
                }
            },
            "persistentVolumeReclaimPolicy": "Delete",
            "storageClassName": "mailwrapper",
            "volumeMode": "Filesystem"
        },
        "status": {
            "phase": "Bound"
        }
    },

As per docs https://karpenter.sh/docs/concepts/scheduling/#persistent-volume-topology supported nodeaffinity key are "topology.ebs.csi.aws.com/zone" and "topology.kubernetes.io/zone"
This nodeaffinity section of PV's is Immutable so we cant change the key now. only way is to recreate the PV's which will cause Data loss for us.  Can you please add this nodeaffinity key "failure-domain.beta.kubernetes.io/zone" in your code so that karpenter can launch the nodes in the correct AZ as PV volume.
How important is this feature to you?
Without this we can not deploy karpenter in our Orgazination.

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Karpenter does support failure-domain.beta.kubernetes.io/zone, this is a docs miss. It translates it internally to topology.kubernetes.io/zone. I can cut a PR to update the docs.

authors: infa-msheik jmdeal


text:
Use lowercased AMG/FFG Item number as the unique ID for expansions.json

xws-ifying full expansion names as they are listed like "Star Wars X-Wing 2nd Ed: ARC-170 Starfighter" is maybe not the best way to refer to expansions in a "collection.json" document that is independent of the builders.
This would allow listing 1.0 ships with less ambiguity as well.
[
   {
      "item_no": "swz106",
      "name": "Rebel Alliance Squadron Starter Pack",
      "contents": [
          ...existing content list...
      ]

Started in #13, but the skus aren't filled in for all expansions yet.

This is done.

license: MIT


text:
Log4cplus and curl are not built in release mode

Describe the bug
Log4cplus and curl's  release dlls are not generated when SDK is built with -DCMAKE_BUILD_TYPE=Release flag.
To Reproduce
Steps to reproduce the behavior:

Run cmake ..  -DCMAKE_BUILD_TYPE=Release  -DBUILD_GSTREAMER_PLUGIN=ON  -DBUILD_TEST=TRUE
Check kvs\open-source\local\bin folder for log4cplus and curl dlls.
You can notice that release dlls are not generated. Check for -D postfix notation.

Expected behavior
Should generate the dlls based on the release type.
SDK version number
Latest Master
Possible Fix
Release build is issue is fixed by making to Cmake\Utilities.cmake file. I have corrected the code in fetch_repo and build_dependency functions by adding build type flag.
Before:
execute_process(
COMMAND ${CMAKE_COMMAND} --build .
RESULT_VARIABLE result
WORKING_DIRECTORY ${DEPENDENCY_DOWNLOAD_PATH}/lib${lib_name})
After:
execute_process(
COMMAND ${CMAKE_COMMAND} --build . --config ${CMAKE_BUILD_TYPE}
RESULT_VARIABLE result
WORKING_DIRECTORY ${DEPENDENCY_DOWNLOAD_PATH}/lib${lib_name})

Hi, thank you for reporting it. We do welcome contributions, feel free to open a PR with the changes and the team will review it.

Closing due to inactivity, please open a PR for this contribution. Feel free to raise any further issues/questions.

I tried to create a branch before creating a pull request. I am not able to create a branch. I am receiving 403 error


text:
Unable to stream in AWS kinesis video from Ubuntu install on Virtual Box having Camera

Using cmd:
gst-launch-1.0 v4l2src do-timestamp=TRUE device=/dev/video0 ! videoconvert ! video/x-raw,format=I420,width=640,height=480,framerate=30/1 ! x264enc  bframes=0 key-int-max=45 bitrate=500 ! video/x-h264,stream-format=avc,alignment=au,profile=baseline ! kvssink stream-name="TestVideoStream2" storage-size=512 access-key="" secret-key="" aws-region="us-east-1"
Error:
[DEBUG] [17-09-2022 12:11:02:262.457 GMT] streamReadyHandler invoked
Stream is ready
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
../libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.000589469
Setting pipeline to NULL ...
[INFO ] [17-09-2022 12:11:02:263.764 GMT] stopKinesisVideoStreamSync(): Synchronously stopping Kinesis Video Stream 000055db47763600.
[DEBUG] [17-09-2022 12:11:02:263.852 GMT] streamClosedHandler invoked for upload handle: 18446744073709551615
[DEBUG] [17-09-2022 12:11:02:263.883 GMT] Reported streamClosed callback for stream handle 94400285128192. Upload handle 18446744073709551615
[INFO ] [17-09-2022 12:11:02:263.897 GMT] Sending eos
Freeing pipeline ...

Hi @EHVKumar, looks like an issue with the system's camera. Can you please ensure that GStreamer has access to the device's camera before running the pipeline?

I ran gst-device-monitor-1.0 in terminal and got this output.
device_details.txt

@EHVKumar ,
This does not seem to be a KVS specific issue. I would suggest to reach out to GStreamer forums to understand how you need to fix your source pipeline. A few pointers:

Start with fakesink. Ensure your source pipeline is connected appropriately.
Try other sinks - autovideosink for example.
See if one of the sample pipelines here works as well: https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp/blob/master/docs/linux.md. See if removing do-timestamps helps.

I am closing this issue, but do feel free to reopen if after investigation on your end, it does not work with kvssink.


text:
update forecast CI

Need this separate PR to update Jenkinsfile (this PR just uses a dummy forecasting submodule), as this file can only be updated in PRs solely authored by certain authors.

Job PR-1195-3 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-1195/3/index.html

authors: jwmueller szha

repo: awslabs/autogluon


text:
Parquet with decimal columns

I have a job that used pyarrow to write some parquet files to S3, then i switched to awswwrangler, but I'm getting a HIVE_PARTITION_SCHEMA_MISMATCH error in a Decimal column:
The column 'x' in table 'y' is declared as type 'decimal(5,2)', but partition 'z' declared column 'x' as type 'decimal(4,2)'
Is there any way to force the column to a decimal(5,2)?

I discoverd the dtype parameter that solved my problem

I discoverd the dtype parameter that solved my problem

repo: awslabs/aws-data-wrangler


text:
Limit PivotRole trust relationships [Permissions Clean up]

Describe the bug
Pivot Role Trust Relationships:
Can you kindly consider locking down only the required services to be able to assume the data.all pivot role from the central to the linked accounts.  This would greatly help in ensuring Principles of Least Privilege are met.
From the cloud trail logs I see only the following roles from central assuming this role:**

arn:aws:sts::***********:assumed-role/dataall-sandbox-graphql-role/
arn:aws:sts::***********:assumed-role/dataall-sandbox-awsworker-role/
arn:aws:sts::***********:assumed-role/dataall-sandbox-ecs-tasks-role/

We can start with and test further?

Thanks @manjulaK, we will fix it for v1.6

This item is duplicated #491

authors: dlpzx manjulaK

repo: awslabs/aws-dataall


text:
Feature: Add sns notification support to Parser utility

Validation utility support sns notification schema, while the parser does not.
This issue will add the requires schema, envelope and tests for sns notification events.

Resolving as it's now merged in Develop 🎉

authors: heitorlessa risenberg-cyberark

repo: awslabs/aws-lambda-powertools-python


text:
Add Support for API Gateway Lambda authorizer

Is your feature request related to a problem? Please describe.

The API Gateway Resolver does not (appear) to support the ability to dispatch to a function as a Lambda authorizer.
Describe the solution you'd like

Add support to be able to dispatch to a function that is a Lambda authorizer.
Describe alternatives you've considered

Additional context

@gwlester  - this is getting a little closer to being done. The next PR is focusing on the HTTP api lambda authorizer and then finally Rest api lambda authorizer will be done.

https://github.com/awslabs/aws-lambda-powertools-python/pull/620

Now avaiable in 1.20.0 both for v1 and v2: https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/data_classes/#api-gateway-authorizer

authors: gwlester heitorlessa michaelbrewer

repo: awslabs/aws-lambda-powertools-roadmap


text:
fix document for installing entire SDK

When running go get github.com/aws/labs/aws-sdk-go, then got
can't load package: package github.com/awslabs/aws-sdk-go: no buildable Go source files in /Users/ke-suzuki/src/github.com/awslabs/aws-sdk-go

so using
$ go get github.com/awslabs/aws-sdk-go/...

for installation.

Hi @suzuken Thanks for creating a PR. We are actually going to go with the pattern of adding a stub go file to the aws-sdk-go's root to prevent the error.

I just pushed a change which will address this issue with a stub.go file in the root of the project.  Thanks again for bringing this issue up!.

@jasdel I got it! Thanks!

authors: jasdel suzuken

repo: awslabs/aws-sdk-go


text:
Instance metadata credentials support

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue, please leave a comment

I'd love to work on this. I'll mostly be scratching my own itch since I often run code on an EC2 without creating a different set of user credentials.
I've take a look at the code base and I think this will need one more type of credential provider, something like InstanceMetadataCredentialsProvider that implements the ProviderCredentials trait.
This will require two inputs -

Role name attached to the instance
Default region to set

The provider logic will have to query instance metadata url http://169.254.169.254/latest/meta-data/iam/security-credentials/<role-name>/, retrieve the values and create the credentials object from it.
Please let me know if you have any guidance or concerns related to this.

There's also this middle ground where the user might want to use Instance metadata to retrieve credentials. But set this up as an environment credentials through a config file as given here.
In this case the logic behind retrieving the credentials (either from environment variables or from instance metadata) will have to inferred at runtime by checking if credential_source = Ec2InstanceMetadata is set.

You can get the region from the instance metadata via  placement/region; that seems like a good default.

Also, it should be possible to determine the role name automatically as well.

IMDS credentials support has landed and will go out in the next release. Thanks for the upvotes and the patience!

IMDS credentials landed with v0.0.19-alpha.

authors: dialtone jdisanti joshtriplett rcoh twitu

repo: awslabs/aws-sdk-rust


text:
Release v2

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

AWS CodeBuild CI Report

CodeBuild project: githubautobuild-for-cdk-v2
Commit ID: dcbca8899e872d096ffc56535b5e0efc93a8d541
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

authors: aws-solutions-constructs-team biffgaut

repo: awslabs/aws-solutions-constructs


text:
Getting error setting up MWAA

Getting error installing from https://github.com/awslabs/data-on-eks/tree/main/schedulers/terraform/managed-airflow-mwaa
EKS add-ons failing
│ Warning: Helm release "metrics-server" was created but has a failed status. Use the helm command to investigate the error, correct it, then run Terraform again.
│
│ with module.eks_blueprints_kubernetes_addons.module.metrics_server[0].module.helm_addon.helm_release.addon[0],
│ on .terraform/modules/eks_blueprints_kubernetes_addons/modules/kubernetes-addons/helm-addon/main.tf line 1, in resource "helm_release" "addon":
│ 1: resource "helm_release" "addon" {
│
╵
╷
│ Warning: Helm release "cluster-proportional-autoscaler" was created but has a failed status. Use the helm command to investigate the error, correct it, then run Terraform again.
│
│ with module.eks_blueprints_kubernetes_addons.module.aws_coredns[0].module.cluster_proportional_autoscaler[0].module.helm_addon.helm_release.addon[0],
│ on .terraform/modules/eks_blueprints_kubernetes_addons/modules/kubernetes-addons/helm-addon/main.tf line 1, in resource "helm_release" "addon":
│ 1: resource "helm_release" "addon" {
│
╵

Destroying and Recreating fixed the issue.

repo: awslabs/data-on-eks


text:
Deployment with terraform

It would be very convenient if there would be a  module somewhere in this repository to rollout this with Terraform

this project can convert Cloudformation templates to Terraform HCL.
https://github.com/DontShaveTheYak/cf2tf
I've ran the DTH-cognito through it and here's the output. Looks coherent, but I haven't tested it.
DataTransferHub-cognito.tf.txt

authors: dseeker strowk

repo: awslabs/data-transfer-hub


text:
Distribution prediction for Seq2Seq

Issue #, if available:
Description of changes:
add distribution forecast to seq2seq model
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@YuntianYeAWS could you please fix the merge conflicts?

@YuntianYeAWS could you please fix the merge conflicts?

Hi Aaron. So I added some new code to the original files. Could you tell me how can I resolve these conflicts?

@YuntianYeAWS could you please fix the merge conflicts?
there is no conflict shown in this file and I could not resolve the conflict.
src/gluonts/model/seq2seq/_mq_dnn_estimator.py
Could you have a look?

Some questions/observations:

[ ] You are missing something like a ForkingSeq2SeqSamplingNetwork, however, the output of which should have the same shape as ForkingSeq2SeqPredictionNetwork. Btw, there might be a mistake in the documentation regarding the output shape, I believe it should be (batch_size, num_quantiles, prediction_length)). ForkingSeq2SeqSamplingNetwork would have output shape (batch_size, num_samples, prediction_length). And since num_quantiles and num_samples is just a matter of interpretation their respective output shapes would be equivalent.
[ ] The ForkingSeq2SeqDistributionNetwork might be missing something like we have in the ForkingSeq2SeqPredictionNetwork:

      # We only care about the output of the decoder for the last time step
     fcst_output = F.slice_axis(dec_output, axis=1, begin=-1, end=None)
     fcst_output = F.squeeze(fcst_output, axis=1)

[ ] you could have  self.get_decoder_network_output also return the target scale so that it can be utilised by self.distr_output.distribution` instead of scaling twice
[ ] there seem to be a lot of redundant imports in _forking_network.py
[ ] ForkingSeq2SeqSamplingNetwork and ForkingSeq2SeqPredictionNetwork can probably be merged
[ ] ForkingSeq2SeqTrainingNetwork and ForkingSeq2SeqDistributionTrainingNetwork can probably be merged
[ ] In general it might be useful to think of Quantile output just another type of configurable distribution output, i.e. it might not be warranted to have separate Networks for the support of other distributions. I believe we could fix this by just passing additional arguments to the ForkingSeq2SeqNetworkBase and checking in the Network whether a QuantileOutput or other Distribution is set and then executing the argument projection and loss calculation conditionally on that.

[ ] A seperate ForkingSeq2SeqDistributionNetwork  might be warranted though, since it has a different return type all together (although I'm not aware of anyone using the network equivalent for other models, since we are always interested in samples usually, so we could think about leaving that out for now, which would simplify things because we wouldn't have the sampling option anymore since it would always be True)

What do you think about these concerns?
Have I misunderstood/overlooked something?

@YuntianYeAWS Any updates?

@YuntianYeAWS Any updates?

Hi, Aaron. Sorry for the late reply. Last week, I have worked on another project.
I have made few changes according to your previous comments. So I think we can directly draw samples from the distribution output, so maybe it is not necessary to write another SamplingPrediction. The sampling parameter can be changed to quantile?
Also could you explain how to merge two training network together? The choice of network seems only available in the estimator. I am not sure how to do within the network.
Thank you for your reviews and suggestions.

@YuntianYeAWS Any updates?

Hi Aaron. As suggested by Lorenzo, the sampling should be substituted by quantile. If it is possible, could you have a look if there are other issues?

authors: AaronSpieler YuntianYeAWS

repo: awslabs/gluon-ts


text:
replace isort and black with ruff

Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

AWS CodeBuild CI Report

CodeBuild project: IDFModuleTestsInfrastructur-59JoFoVH64CI
Commit ID: d5f7d6bc2200b03371505a96ddaebbf0e2093e2b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: IDFModuleTestsInfrastructur-59JoFoVH64CI
Commit ID: 73710f8fbbe44139c5b2790f5ecf8cbfea7a8e31
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: IDFModuleTestsInfrastructur-59JoFoVH64CI
Commit ID: 1931aa29edaa1c529033a73b523c538786609446
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: IDFModuleTestsInfrastructur-59JoFoVH64CI
Commit ID: 880ec4231ca9a9d4d035fc0cf77580142459c48b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

authors: LeonLuttenberger malachi-constant

repo: awslabs/idf-modules


text:
namespace label for metrics collector injection

Describe the bug
[500] admission webhook "validator.experiment.katib.kubeflow.org" denied the request:
Cannot create the Experiment "random-experiment" in namespace "kubeflow-person":
the namespace lacks label "katib.kubeflow.org/metrics-collector-injection: enabled"

Steps To Reproduce
I had 1.3 installed. I deleted it and reinstalled 1.4
Expected behavior
I was expecting Katib to look for katib-metricscollector-injection: enabled
instead of katib.kubeflow.org/metrics-collector-injection: enabled
Environment

Kubernetes version 1.20
Using EKS (yes/no), if so version? Yes
Kubeflow version 1.4
AWS build number
AWS service targeted (S3, RDS, etc.) S3 & RDS

Additional context
setting the the new label allows me to submit an experiment via Katib but the runs fail. It looks like the collector is configured correctly.

@dewnull Can you additionally share the failed Katib run logs?

I get this from metrics-logger-and-collector
flag provided but not defined: -format
Usage of ./file-metricscollector:
  -f string
    	Metric filters
  -m string
    	Metric names
  -o-type string
    	Objective type
  -p duration
    	Poll interval between running processes check (default 1s)
  -path string
    	Metrics File Path
  -s-db string
    	Katib DB Manager service endpoint
  -s-earlystop string
    	Katib Early Stopping service endpoint
  -stop-rule value
    	The list of early stopping stop rules
  -t string
    	Trial Name
  -timeout duration
    	Timeout before invoke error during running processes check
  -w string
    	Whether wait for all other main process of container exiting (default "true")

the args to themetrics-logger-and-collector:
    Args:
      -t
      random-test-kf-nldx545x
      -m
      Validation-accuracy;Train-accuracy
      -o-type
      maximize
      -s-db
      katib-db-manager.kubeflow:6789
      -path
      /var/log/katib/metrics.log
      -format
      TEXT

so I suspect the issue is with -format. Katib release 0.12 doesn't have -format but master does.
I wonder if I didn't checkout the correct version of upstream
@goswamig and @ryansteakley let me know if you still what the webhook and experiment information

after I manually change the image for the katib-controller deployment fromkubeflowkatib/katib-controller kubeflowkatib/katib-controller:v0.12.0 it works fine. I am not sure why Kustomize is not setting the correct tag for the Katie controller image.

are you able to run the katib jobs properly now?

yes.

authors: dewnull ryansteakley

repo: awslabs/kubeflow-manifests


text:
undo all steps in the fabfile

Can you please add a function to fabfile.py that undoes all the steps that were done in the setup function? Something like fab setup /  fab unsetup would be helpful.

I have been unable to use this script without uncommenting the "docker login" line. Am I doing something wrong or is the script wrong?

Hi koppi,
thanks for your comment and feature request. Unfortunately, I'm very busy right now, so implementing this may take some time, but it’s definitely on my list now.
Thanks,
Constantin

Hi chumanfu,
thanks for your comment.
The original version of this script used Dockerhub to host the images. In this case, the (now commented) docker line is the right mechanism to use.
Then, a feature request was accepted that added support for Amazon EC2 Container Repository (ECR), which uses a different authentication mechanism (the aws ecr get-login line).
If you want to use Dockerhub, then, yes, you should uncomment the docker login line and comment out the three (3) login_str lines, including the aws ecr get-login line and you should be good to go.
If you want to use Amazon ECR for storing your Docker images, then the script should work as is and you would need to set up your Amazon ECR repository first, before using this script.
Hope this helps,
Constantin

authors: chumanfu glez-aws koppi

repo: awslabs/lambda-ecs-worker-pattern

license_source: bigquery


text:
How to describe a lookup either by a Foo, or a Bar

Effectively, I want to have unions as Operation Inputs if we can have such a thing. This is what I tried:
$version: "2.0"

namespace smithy4s.hello

@http(method: "POST", uri: "/lookup", code: 200)
operation LookupCustomer {
    input:= {
        @required
        id: LookupId
    }
    output: Customer,
}

string UserId
string CustomerId

union LookupId {
    @httpQuery("userId")
    customerId: UserId,

    @httpQuery("customerId")
    userId: CustomerId
}

@output
structure Customer {
    @required
    message: String
}

but I am getting:

'httpQuery' may only be applied to shapes that match the following selector: structure > member :test(> :test(string, number, boolean, timestamp), > list > member > :test(string, number, boolean, timestamp))

ie, it doesn't like me using unions.
Looking at https://github.com/awslabs/smithy/issues/1112#issuecomment-1054445719 I am getting the impression it's not possible? quite disappointing really.
The above demonstrated an example of using query parameters to pass union input. Alternative use case is /lookup/{lookupId} where lookupId might be either say a 5 digits long number, or a 10 digits long number, ie legacy vs new Ids. So there would be an order in which you'd try to parse them and use whatever works.
Guessing that can't be expressed in Smithy either?

The httpQuery and httpLabel traits are only effective when applied to members of the structures that are inputs to an operation at the top level. Inputs and outputs for operations must be structures to allow for safe, forwards compatible API evolution.
The id field you're trying to model as a union would be valid if left in the body, by removing the httpQuery trait. Otherwise, this operation seems like it could either overload the input id member as a label or be broken up in to one old, deprecated operation with the legacy id and one with the new id.

Thanks @kstich

Otherwise, this operation seems like it could either overload the input id member as a label

Can you do that? Could you please give a minimal example of this?

It would just mean accepting both types of values for a singular customerId input string member in the model, and handling the difference in service behavior.
@http(method: "POST", uri: "/lookup/{customerId}", code: 200)
operation LookupCustomer {
    input:= {
        @required
        @httpLabel
        customerId: CustomerId // Service behavior allows this to be either type of Id
    }
    output: Customer
}

string CustomerId

Thanks @kstich so to sum up, we can use union types in the input position, but smithy's opinionated approach clashes with http requirements for a GET endpoint. not allowing me to do use it in a http operation since GET http has undefined behaviour for GET request with a payload and request payload seems to be the only way of passing a unon as input.
if so, happy for you to close this issue.

Only top-level members of an operation's input, output, and error structures are considered when serializing HTTP messages.

HTTP Bindings
For a GET, you can either use the overload behavior described above or two separate httpQuery bound parameters instead of using a union or having two separate operations.

authors: ShahOdin kstich

repo: awslabs/smithy


text:
Source factors

Added source factors, as described in.
Linguistic Input Features Improve Neural Machine Translation.
Rico Sennrich & Barry Haddow
In Proceedings of the First Conference on Machine Translation. Berlin, Germany, pp. 83-91.

Source factors are enabled by passing --source-factors file1 [file2 ...] (-sf), where file1, etc. are token-parallel to the source (-s).
This option can be passed both to sockeye.train or in the data preparation step, if data sharding is used.
An analogous parameter, --validation-source-factors, is used to pass factors for validation data.
The flag --source-factors-num-embed D1 [D2 ...] denotes the embedding dimensions.
These are concatenated with the source word dimension (--num-embed), which can continue to be tied to the target (--weight-tying --weight-tying-type=src_trg).
At test time, the input sentence and its factors can be passed by multiple parallel files (--input and --input-factors) or through stdin with token-level annotations, separated by |. Another way is to send a string-serialized JSON object to the CLI through stdin which needs to have a top-level key called 'text' and optionally a key 'factors' of type List[str].
Pull Request Checklist

[x] Changes are complete (if posting work-in-progress code, prefix your pull request title with '[WIP]'
until you can check this box.
[x] Unit tests pass (pytest)
[x] System tests pass (pytest test/system)
[x] Passed code style checking (./pre-commit.sh or manual run of pylint & mypy)
[x] You have considered writing a test
[x] Updated major/minor version in sockeye/__init__.py. Major version bump if this is a backwards incompatible change.
[x] Updated CHANGELOG.md

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Okay, Felix and I just chatted at the whiteboard, and here is the proposal, which I hope pleases everyone:
Training:

--source and --target remain as they are
We will add --source-factors, which takes any number of source factors in files that are token-parallel to --source
The source factor embedding sizes are given as --source-factor-num-embed. These are added to the source dimensions given in --num-embed. The purpose for doing it that way is so that source and target embeddings can still be tied.

Inference:

Presented on STDIN, the factors need to be Moses style, |-delimted. However, we will use a variable to record the delimiter so we could easily change it later or allow it to be parameterized.
We will split using rsplit(..., maxsplit=num_factors), which splits from the right, assigning everything left over to the left-most chunk. This way, we don't need to remove the delimiter from the test data.
We can easily make sure that each word has enough factors by throwing an error if not enough are found for any token.
Presented via --input, the argument to --input is a regular (BPE-split) file as normal, and --input-factors takes a list of paths to token-parallel source factor files.

This looks like a good design that captures the requirements we think are most important.  Thanks Matt for working on this and everyone for the good discussion!

from my side code is now in a pretty good stage

Lets decide whether we want this to be backwards compatible or breaking. If it is the latter we can also change the name of the source vocabs from vocab.src.json to vocab.src.i.json where i corresponds to the factors. This would make the loading and saving logic for vocabs cleaner.

I'm fine with making it backwards-incompatible. It's a fairly large change. If we do so then we could think about pulling some things out of DataConfig that are there only for documentation purposes (unlike the length ratios). We could have a DataInfo object that is also stored in the model folder, but not used at inference time.

I created a PR to separate these two objects in #293

As far as I can tell, I adressed all comments. @davvil @tdomhan I would merge once both of you approve.

authors: fhieber mjdenkowski mjpost tdomhan

repo: awslabs/sockeye


text:
[WIP] Adding pointer networks for summarization

[]   Changes are complete (if posting work-in-progress code, prefix your pull request title with '[WIP]'  until you can check this box.
[x]   Unit tests pass (pytest)
[]   Were system tests modified? If so did you run these at least 5 times to account for the variation across runs?
[]   System tests pass (pytest test/system)
[]  Passed code style checking (./style-check.sh)
[]  You have considered writing a test
[]   Updated major/minor version in sockeye/__init__.py. Major version bump if this is a backwards incompatible change.
[]  Updated CHANGELOG.md

By submitting this pull request, I confirm that my contribution is made as an Amazon employee.

Closing for now.

authors: Zarana-Parekh mjpost


text:
Consider using mime magic to detect file type, instead of only excluding by file extension

SSIA

So we actually only look at file extensions to exclude certain files. It's up to each backend to decide how it wants to scan things. At the moment we exclude certain files like .bmp. So if someone hides GPL source code in such a file, we will miss it. I think that's a fair trade off for now, but maybe an improvement would be to just skip certain binary files in general. (Ones that we don't decompress for example).
This is a good issue to track the discussion and feature, thanks!

Let's explore our options. I would like to understand how the mechanism works.

FYI: This issue will depend on https://github.com/awslabs/yesiscan/issues/46 getting fixed first.

long run for MVP, moving to backlog

authors: amznpurple jamin-aws-ospo ovalba

repo: awslabs/yesiscan


text:
Not able to override AWS endpoint during Integration test

Type: Bug
Component: Secrets Manager

Describe the bug
I am using the latest 3.0.0-M2 version for spring cloud to pull secrets from the secrets manager.
spring:
  cloud:
    aws:
       endpoint: http://localhost:4566
      credentials:
        profile:
          name: localstack
  config:
    import:
    - aws-secretsmanager:/secret/spring-boot-app;/secret/db-credential

I run a local docker image of localstack , start the application and it all works fines. It pulls the secret and no issues.
Now I am trying to write an integration test using localstack and I am overriding the endpoint properties using @DynamicPropertySource mechanism.
@Testcontainers
@SpringBootTest
class MainApplicationTests {

    @Autowired
    Environment environment;
    @Container
    static LocalStackContainer localStackContainer = new LocalStackContainer(DockerImageName.parse("localstack/localstack"))
            .withCopyFileToContainer(MountableFile.forClasspathResource("script.sh"),
                    "/docker-entrypoint-initaws.d/")
            .withServices(LocalStackContainer.Service.SECRETSMANAGER)
            .withExposedPorts(4566)
            .waitingFor(Wait.forLogMessage(".*localstack.request.aws.*", 2));

    @DynamicPropertySource
    static void properties(DynamicPropertyRegistry registry) {

        registry.add("spring.cloud.aws.secretsmanager.endpoint", () -> localStackContainer.getEndpointOverride(LocalStackContainer.Service.SECRETSMANAGER));
        registry.add("spring.cloud.aws.endpoint", () -> "http://localhost:"+localStackContainer.getMappedPort(4566));
        registry.add("spring.datasource.url", () -> database.getJdbcUrl());
    }

when I run the test, the client is still using the endpoint defined in the properties file ( not the test properties file) and fails with an error that it could not connect to the endpoint http://localhost:4566 ( which is hte url from the main properties file)
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:4566 [localhost/127.0.0.1] failed: Connection refused
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
	at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.connect(ClientConnectionManagerFactory.java:86)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)

I debugged the properties fetch for the database properties override with test containers, it is all correct.
I am not sure why the lambda given in the DynamicPropertySource to override the properties does not even get invoked and hence it uses the properties from the main properties file and fails.
I am not sure if I am doing something wrong or missing something.

This is a limitation of @DynamicPropertySource - it does not work with config import (https://github.com/spring-projects/spring-boot/issues/26148). I see two ways to workaround it:

Use SpringApplication testing approach instead of @SpringBootTest as suggested in the linked issue
Set fixed port number to Localstack running through Testcontainers like this:

@SpringBootTest(properties = "spring.cloud.aws.secretsmanager.endpoint=http://localhost:4566")
class AppTests {

	@Autowired
	Environment environment;

	static LocalStackContainer localStackContainer;

	@BeforeAll
	static void beforeAll() {
		MyLocalStackContainer myLocalStackContainer = new MyLocalStackContainer(DockerImageName.parse("localstack/localstack"));
		myLocalStackContainer.addFixedExposedPort(4566, 4566);
		myLocalStackContainer.withServices(LocalStackContainer.Service.SECRETSMANAGER);
		myLocalStackContainer.start();
		localStackContainer = myLocalStackContainer;
	}

	@AfterAll
	static void afterAll() {
		localStackContainer.stop();
	}

	@Test
	void foo() {
		System.out.println("Foo");
	}

	static class MyLocalStackContainer extends LocalStackContainer {
		public MyLocalStackContainer(DockerImageName dockerImageName) {
			super(dockerImageName);
		}

		public void addFixedExposedPort(int hostPort, int containerPort) {
			super.addFixedExposedPort(hostPort, containerPort);
		}
	}
}

Let me know if any of these fits your use case or perhaps you have some other ideas.

Hey @maciejwalkowiak ,
Maybe I didn't understand properly.
The link that you shared shows it does not work for spring.config.import but I am trying to override the spring.cloud.aws.secretsmanager.endpoint .
Also what exactly do you mean by Use SpringApplication testing approach instead of @SpringBootTest? I always wrote integration tests with @SpringBootTest.  Is there another way of testing? do you mean by @ExtendWith(SpringExtension.class)
Sorry for my noob question 😅

Yes you're overriding the endpoint property but for the purpose of changing beans created in the bootstrap phase for Secrets Manager spring.config.import implementation.
There is another way of testing but it's not as nice as @SpringBootTest (see https://github.com/spring-projects/spring-boot/blob/6254ad634ecb0dc73034038f82fcde75fb99117a/spring-boot-project/spring-boot/src/test/java/org/springframework/boot/context/config/ConfigFileApplicationListenerLegacyReproTests.java).

Sorry for my noob question 😅

No worries this is not a noob question and even if it would be - this is a safe space to ask noob questions :)

System.setProperty can be used and still use the random port from testcontainers. See the following example https://github.com/eddumelendez/testcontainers-localstack/blob/main/secretsmanager/src/test/java/com/example/secretsmanager/SecretsmanagerApplicationTests.java
The same applies for parameterstore

So I preferred using System.setProperty as @eddumelendez said. And it works and it is cleaner I felt.
But it was nice to know about you can run tests the way that @maciejwalkowiak showed in the example.
I will be writing an article about the AWS Secrets Manager integration using Spring Cloud Secrets Manager soon.
Also nice work with the new Spring Cloud AWS 3.0 version. :+1:
Also thank you once again @maciejwalkowiak and @eddumelendez.

@amrutprabhu , thanks a lot for your article, it helped me to find the solution with System.setProperty, as I was also banging my head against the @DynamicPropertySource wall.
It's a pity that https://github.com/spring-projects/spring-boot/issues/26148 was just closed, I think at least making the documentation for @DynamicPropertySource a bit more clear would have helped people not to waste time on it in these cases.

authors: amrutprabhu bigunyak eddumelendez maciejwalkowiak

repo: awspring/spring-cloud-aws


text:
Issue in LogarithmicToneMapper

Please the the attached Xcode build error when using the master branch (April 8th 2024).
Changing from log10f_c to log10f works on my machine.
Versions:

Machine: Macbook Pro 2019 - 2.6 GHz 6-Core Intel Core i7
Swift version: 5.10
Xcode Version: 15.3

My mistake, it function has deleted. Please, try update to the latest version from master

authors: BenShutt awxkee


text:
Dev

update InitXXXFunc signature
support mutiple db instance
mv response,retcode to top dir
add cn readme, fix script for macOS
delete auto migrate option, delete gorm custom logger
use zap replace logrus
merge middleware in LogRequestInfo
use new sentry sdk

update InitXXXFunc signature
support mutiple db instance
mv response,retcode  to top dir
add cn readme, fix script for macOS
delete auto migrate option, delete gorm custom logger
use zap replace logrus
merge middleware in LogRequestInfo
use new sentry sdk

Codecov Report

Merging #28 into master will increase coverage by 1.43%.
The diff coverage is 60.46%.

@@            Coverage Diff             @@
##           master      #28      +/-   ##
==========================================
+ Coverage   59.44%   60.87%   +1.43%
==========================================
  Files          25       25
  Lines         646      800     +154
==========================================
+ Hits          384      487     +103
- Misses        221      266      +45
- Partials       41       47       +6

Impacted Files
Coverage Δ

app/apis/init.go
100% <ø> (ø)
:arrow_up:

app/apis/routes.go
100% <ø> (+17.64%)
:arrow_up:

app/retcode/retcode.go
100% <ø> (ø)

app/apis/demo/labeling.go
40% <ø> (ø)
:arrow_up:

app/response/response.go
100% <ø> (ø)

app/apis/demo/object.go
60.86% <ø> (ø)
:arrow_up:

app/apis/demo/label.go
60.86% <ø> (ø)
:arrow_up:

app/utils/testing.go
9.09% <0%> (ø)
:arrow_up:

app/utils/endless.go
0% <0%> (ø)
:arrow_up:

app/services/demo/object.go
88% <100%> (ø)
:arrow_up:

... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e48f437...ed6f4c8. Read the comment docs.

authors: axiaoxin codecov-io

repo: axiaoxin/pink-lady


text:
fix(tags): scroll highlighted tag into view

Fixed the following issue
https://github.com/ayoung19/clipboard-history/issues/24

This is awesome!!! I was pretty stumped by it because 1) I didn't know block: "nearest" was an option and 2) I was trying to solve this with refs but the good ol' querySelectorAll is just way easier here.
I believe there's technically a chance the side effect fires before the tag options finish rendering though. However, in practice this never comes up though because if matchedTags changes we call setFocusedTagIndex(0) anyway and there's always at least 1 tag option for the query selector to find.
Thanks a bunch! I love the fix.


text:
Typos chapter 3

This concerns the PDF version of the book
p57

...check that it’s not empty, and call getDada(), which will make the XHR request...

@madHamish42 i did find any typos but I re-wrote this place

repo: azat-co/fullstack-javascript

license: mit


text:
Color each of the 18 districts differently

Overview
Each district should be a different color. The outlines should be darker.
Checklist

[x] PR has a name that won't get you publicly shamed for vagueness
[ ] Files changed in the PR have been yapf-ed for style violations

Demo

Testing Instructions

Open a not blank map and see the color. Interact with the map and see them change as you edit districts.
Open a blank map and start creating districts. Watch them get colored.

Thanks for the map comment. I often forget the JS array methods! I also changed the colors. You should see the colors on the screenshot now.

repo: azavea/district-builder-dtl-pa


text:
fix: loop in fs polyfill when readdirSync and path /

Fixing the fs polyfill when readdirsync and path == /, in this case the worker goes into a loop and crashes the process.

Context
A new library (@nodelib/fs.scandir) was introduced that created a recursion in the function execution. It seems to be related to the FS polyfill.
Error in cells: [CELLS: RUNTIME ERROR Uncaught Error: execution terminated]
Error in Vulcan local dev: infinite loop + zombie process
https://github.com/aziontech/vulcan/assets/12740219/84b9c8d7-934d-4c51-b1b6-9688b69e8ef0

:tada: This PR is included in version 2.2.0-stage.3 :tada:
The release is available on:

npm package (@stage dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 2.2.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

authors: jcbsfilho jotanarciso tiagokrebs

repo: aziontech/vulcan


text:
Update sbt-sonatype to 3.9.20

Updates org.xerial.sbt:sbt-sonatype from 3.9.17 to 3.9.20.
GitHub Release Notes - Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.xerial.sbt", artifactId = "sbt-sonatype" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "org.xerial.sbt", artifactId = "sbt-sonatype" }
}]

labels: sbt-plugin-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #69.

repo: azolotko/sbt-gitlab


text:
Having atomic public on CellRef and CellRefMut make API unsound.

Caller may reset atomic to 0 and borrow mutably again and again.

oh thanks, shall fix this in 0.2.0 :bow:


text:
Support for empty patterns

e.g. tracktitle: empty pattern

on second thought, the only use case for this is to run an action on all tracks. which we can support if a use case appears; however, this is a shitty way of encoding such logic. better to instead have rose rules run-all 'action'.

repo: azuline/rose


text:
Added support for New Zealand

work in progress

Thanks for the support of adding New Zealand! Looking forward to the final PR 👍

Thanks @stelgenhof I saw your post on /r/php and liked the library, so I wanted to contribute to improve the coverage.
My plan is to finish off the national holidays for NZ, then I'll create a new PR for provincial holidays. If you want, and I have time I'll start another PR for Australia support later.
I'd appreciate if you could review what I've done so far and make sure I've not deviated from the goals of the project, I've added Mondayisation to all the applicable holidays

Thanks @badams! Let me have a look later this week. I will then merge it with the base branch. I have set this as a milestone for the 1.3.0 version release :)

Sounds good to me @stelgenhof :+1:

@stelgenhof I've adjusted all the tests as requested, let me know if anything else :)

authors: badams stelgenhof

repo: azuyalabs/yasumi


text:
pretty-quick uses a different version of prettier than the installed in the repo

I found a mismatch between the fixes pretty-quick was doing on commit (the husky setup) and when running prettier directly.
I couldn't figure out what was causing the mismatch. Turns out the prettier version I have installed is quite old and is not the same version pretty-quick is using.

First format the file with prettier, it generates some changes. Then run pretty-quick, changes aren't there anymore (since the commit was done with pretty-quick so the fix basically returns repo to initial state).
So then I check the versions of prettier and turns out they are different.
I looked a bit at the code of pretty-quick and cannot figure out why it is using a different version of prettier than the installed one (thought it could be a global version, but I don't have one).

Interesting. pretty-quick only has a peerDependency on prettier and your version 1.10.2 matches it. pretty-quick does have a devDependency of 1.18.2 on prettier here which does match what you're seeing.
Could you change the console log you inserted to show require.resolve('prettier') instead, to confirm the path that prettier is being loaded from?

It gets resolved to this: node_modules/prettier/index.js, which is indeed version 1.18.2.
$ cat node_modules/prettier/index.js | grep version | head -n 1
var version$1 = "1.18.2";

And then I checked what version the scripts are using:
$ cat $(yarn bin)/prettier | grep version | head -n 1
var version$1 = "1.10.2";

It seems more of yarn/npm problem, in the way dependencies are resolved.
The only solution that occurs to me is to use execa to run prettier on pretty-quick side . Instead of using it as a library. But personally I don't like the idea, the library is handier to use.
~/p/cw-buyer master*
$ cat with-execa.js
const execa = require('execa')

console.log(execa.sync('prettier', ['--version']).stdout)

~/p/cw-buyer master*
$ node with-execa.js
1.10.2

Where does ls -l $(yarn bin)/prettier link to then?
Don't want to switch to sub-processes in the interest of being quick 😉, so I will close this issue, but still interested to see where your prettier is actually being resolved from!

Makes sense.
It didn't occur to me to check were prettier was linked to.
We have it installed as a dependency of a toolbox ("cw-sdk-js") so it comes from there.
~/projects/cw-buyer/node_modules/.bin/prettier -> ../cw-sdk-js/node_modules/prettier/bin-prettier.js

authors: azz gillchristian

repo: azz/pretty-quick


text:
Casting starts then instantly stops

So I have looked at old issues, gone through all the steps multiple times, but for some reason it just does not work for me.
Using a Google Hub
Here is a video of what happens, pretty straight forward
Here is my device info:

I have double checked the ha-catt-fix:

Here is my continuously_casting_dashboards config:
continuously_casting_dashboards:
  logging_level: debug
  cast_delay: 45
  start_time: "07:00"
  end_time: "01:00"
  devices:
    "Kitchen display":
      dashboard_url: "https://10.0.0.247/lovelace/homepage?kiosk"

One thing to note, I have changed my port to be 443 instead of 8123
Here is my http config:
http:
  server_port: 443
  ssl_certificate: /ssl/fullchain.pem
  ssl_key: /ssl/privkey.pem
  use_x_forwarded_for: true
  trusted_proxies:
    - 10.0.0.0/24
    - 10.253.0.0/24
    - 10.10.10.0/24
  ip_ban_enabled: true
  login_attempts_threshold: 5

Here is my homeassistant config:
homeassistant:
  country: ZA
  packages: !include_dir_named packages
  name: Home
  customize: !include customize.yaml
  auth_providers:
    - type: trusted_networks
      trusted_networks:
        - 10.0.0.0/24
        - 10.253.0.0/24
        - 10.10.10.0/24
      trusted_users:
        10.0.0.91: !secret my_user_id
      allow_bypass_login: true
    - type: homeassistant
  external_url: !secret external_url

Then lastly, here are my logs:

2023-10-18 18:41:07.837 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Current local time: 18:41:07.837409
2023-10-18 18:41:07.837 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Local time is inside the allowed casting time for Kitchen display. Start time: 07:00:00 - End time: 01:00:00
2023-10-18 18:41:08.349 DEBUG (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Status output for Kitchen display when checking for dashboard state 'Dummy': Volume: 5
2023-10-18 18:41:08.896 DEBUG (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Status output for Kitchen display when checking for dashboard state 'Dummy': Volume: 5
2023-10-18 18:41:08.896 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] HA Dashboard (or media) is NOT playing on Kitchen display!
2023-10-18 18:41:08.896 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Casting dashboard to Kitchen display
2023-10-18 18:41:08.897 DEBUG (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Executing stop command...
2023-10-18 18:41:09.919 DEBUG (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Setting volume to 0...
2023-10-18 18:41:10.407 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Executing the dashboard cast command...
2023-10-18 18:41:11.521 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Setting volume to 5...
2023-10-18 18:41:56.992 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Current local time: 18:41:56.992410
2023-10-18 18:41:56.992 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Local time is inside the allowed casting time for Kitchen display. Start time: 07:00:00 - End time: 01:00:00
2023-10-18 18:41:57.483 DEBUG (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Status output for Kitchen display when checking for dashboard state 'Dummy': Volume: 5
2023-10-18 18:41:58.043 DEBUG (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Status output for Kitchen display when checking for dashboard state 'Dummy': Volume: 5
2023-10-18 18:41:58.043 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] HA Dashboard (or media) is NOT playing on Kitchen display!
2023-10-18 18:41:58.043 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Casting dashboard to Kitchen display
2023-10-18 18:41:58.044 DEBUG (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Executing stop command...
2023-10-18 18:41:59.005 DEBUG (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Setting volume to 0...
2023-10-18 18:41:59.519 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Executing the dashboard cast command...
2023-10-18 18:42:00.935 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Setting volume to 5...
2023-10-18 18:42:46.482 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Current local time: 18:42:46.482088
2023-10-18 18:42:46.482 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Local time is inside the allowed casting time for Kitchen display. Start time: 07:00:00 - End time: 01:00:00
2023-10-18 18:42:47.021 DEBUG (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Status output for Kitchen display when checking for dashboard state 'Dummy': Volume: 5
2023-10-18 18:42:47.527 DEBUG (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Status output for Kitchen display when checking for dashboard state 'Dummy': Volume: 5
2023-10-18 18:42:47.527 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] HA Dashboard (or media) is NOT playing on Kitchen display!
2023-10-18 18:42:47.527 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Casting dashboard to Kitchen display
2023-10-18 18:42:47.529 DEBUG (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Executing stop command...
2023-10-18 18:42:48.550 DEBUG (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Setting volume to 0...
2023-10-18 18:42:49.051 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Executing the dashboard cast command...
2023-10-18 18:42:50.634 INFO (MainThread) [custom_components.continuously_casting_dashboards.dashboard_caster] Setting volume to 5...

Any ideas on what could be causing this and how I could get this to work on my environment?
Thank you so much

Same here. I think it has something todo with local https usage since the certificate is not installed on the cast device.

Hey @louispires apologies about the late reply to this. Are you still experiencing the same issue?
I think @panhans is onto something, Instead of using 'https' for your URL in your configuration can you try 'http' instead?

I have to disable https first. HTTP is blocked in my setup.
Can't access the google admin console for some reason. As I read there I would be able to install my self signed certificates for managed devices like nest hub. This would be my favorite solution.

I also have HTTP blocked from my side, thus I am unable to change to HTTP.

authors: b0mbays louispires panhans