text:
hostNetwork true

Version

Karpenter: 0.13.2

Kubernetes: v1.22
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.10-eks-84b4fe6", GitCommit:"cc6a1b4915a99f49f5510ef0667f94b9ca832a8a", GitTreeState:"clean", BuildDate:"2022-06-09T18:24:04Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
Expected Behavior
enable hostNetwork
Actual Behavior
Can't seem  to see it turn on or add additional ENIs
Steps to Reproduce the Problem
I'm trying to setup karpenter nodes to use secondary subnets from a range 100.64.0.0/16
I've created 5 ENIConfig in us-east-1 and attached 3 Security Groups to them.

I've enabled AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG and ENI_CONFIG_LABEL_DEF
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: aws-node
  namespace: kube-system
  uid: 5fce226d-b11d-4053-9d26-6ed41d06e224
  resourceVersion: "874226"
  generation: 28
  creationTimestamp: "2022-07-19T12:18:40Z"
  labels:
    k8s-app: aws-node
  annotations:
    deprecated.daemonset.template.generation: "28"
spec:
  selector:
    matchLabels:
      k8s-app: aws-node
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/name: aws-node
        k8s-app: aws-node
    spec:
      volumes:
      - name: cni-bin-dir
        hostPath:
          path: /opt/cni/bin
          type: ""
      - name: cni-net-dir
        hostPath:
          path: /etc/cni/net.d
          type: ""
      - name: dockershim
        hostPath:
          path: /var/run/dockershim.sock
          type: ""
      - name: log-dir
        hostPath:
          path: /var/log/aws-routed-eni
          type: DirectoryOrCreate
      - name: run-dir
        hostPath:
          path: /var/run/aws-node
          type: DirectoryOrCreate
      - name: xtables-lock
        hostPath:
          path: /run/xtables.lock
          type: ""
      initContainers:
      - name: aws-vpc-cni-init
        image: 602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon-k8s-cni-init:v1.11.2-eksbuild.1
        env:
        - name: DISABLE_TCP_EARLY_DEMUX
          value: "false"
        - name: ENABLE_IPv6
          value: "false"
        - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
          value: "true"
        - name: ENI_CONFIG_LABEL_DEF
          value: failure-domain.beta.kubernetes.io/zone
        - name: ENABLE_PREFIX_DELEGATION
          value: "false"
        - name: WARM_PREFIX_TARGET
          value: "1"
        resources: {}
        volumeMounts:
        - name: cni-bin-dir
          mountPath: /host/opt/cni/bin
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true
      containers:
      - name: aws-node
        image: 602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon-k8s-cni:v1.11.2-eksbuild.1
        ports:
        - name: metrics
          hostPort: 61678
          containerPort: 61678
          protocol: TCP
        env:
        - name: ADDITIONAL_ENI_TAGS
          value: "{}"
        - name: AWS_VPC_CNI_NODE_PORT_SUPPORT
          value: "true"
        - name: AWS_VPC_ENI_MTU
          value: "9001"
        - name: AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER
          value: "false"
        - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
          value: "true"
        - name: AWS_VPC_K8S_CNI_EXTERNALSNAT
          value: "false"
        - name: AWS_VPC_K8S_CNI_LOGLEVEL
          value: DEBUG
        - name: AWS_VPC_K8S_CNI_LOG_FILE
          value: /host/var/log/aws-routed-eni/ipamd.log
        - name: AWS_VPC_K8S_CNI_RANDOMIZESNAT
          value: prng
        - name: AWS_VPC_K8S_CNI_VETHPREFIX
          value: eni
        - name: AWS_VPC_K8S_PLUGIN_LOG_FILE
          value: /var/log/aws-routed-eni/plugin.log
        - name: AWS_VPC_K8S_PLUGIN_LOG_LEVEL
          value: DEBUG
        - name: DISABLE_INTROSPECTION
          value: "false"
        - name: DISABLE_METRICS
          value: "false"
        - name: DISABLE_NETWORK_RESOURCE_PROVISIONING
          value: "false"
        - name: ENABLE_IPv4
          value: "true"
        - name: ENABLE_IPv6
          value: "false"
        - name: ENABLE_POD_ENI
          value: "false"
        - name: ENABLE_PREFIX_DELEGATION
          value: "false"
        - name: MY_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: WARM_ENI_TARGET
          value: "1"
        - name: WARM_PREFIX_TARGET
          value: "1"
        - name: ENI_CONFIG_LABEL_DEF
          value: failure-domain.beta.kubernetes.io/zone
        resources:
          requests:
            cpu: 25m
        volumeMounts:
        - name: cni-bin-dir
          mountPath: /host/opt/cni/bin
        - name: cni-net-dir
          mountPath: /host/etc/cni/net.d
        - name: log-dir
          mountPath: /host/var/log/aws-routed-eni
        - name: dockershim
          mountPath: /var/run/dockershim.sock
        - name: run-dir
          mountPath: /var/run/aws-node
        - name: xtables-lock
          mountPath: /run/xtables.lock
        livenessProbe:
          exec:
            command:
            - /app/grpc-health-probe
            - -addr=:50051
            - -connect-timeout=5s
            - -rpc-timeout=5s
          initialDelaySeconds: 60
          timeoutSeconds: 10
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          exec:
            command:
            - /app/grpc-health-probe
            - -addr=:50051
            - -connect-timeout=5s
            - -rpc-timeout=5s
          initialDelaySeconds: 1
          timeoutSeconds: 10
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
      restartPolicy: Always
      terminationGracePeriodSeconds: 10
      dnsPolicy: ClusterFirst
      serviceAccountName: aws-node
      serviceAccount: aws-node
      hostNetwork: true
      securityContext: {}
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
              - key: kubernetes.io/arch
                operator: In
                values:
                - amd64
                - arm64
              - key: eks.amazonaws.com/compute-type
                operator: NotIn
                values:
                - fargate
      schedulerName: default-scheduler
      tolerations:
      - operator: Exists
      priorityClassName: system-node-critical
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 10%
      maxSurge: 0
  revisionHistoryLimit: 10
status:
  currentNumberScheduled: 2
  numberMisscheduled: 0
  desiredNumberScheduled: 2
  numberReady: 2
  observedGeneration: 28
  updatedNumberScheduled: 2
  numberAvailable: 2

As per karpenter docs , hostNetwork needs to be true to use custom CNI
I've modified the Chart to set as such
module "eks_blueprints_kubernetes_addons_karpenter" {
  source = "git::ssh://git@github.com/aws-ia/terraform-aws-eks-blueprints.git//modules/kubernetes-addons/karpenter?ref=v4.5.0"
  # source = "git::ssh://git@github.com/aws-ia/terraform-aws-eks-blueprints.git//modules/kubernetes-addons/karpenter?ref=main"

  # enable_aws_node_termination_handler = true
  helm_config = {
    cleanup_on_fail = true
    force_update    = true
    replace         = true
    version         = var.karpenter_helm_config_version
    set = [
      # {
      #   name  = "logLevel"
      #   value = "debug"
      # },
      {
        name  = "hostNetwork"
        value = "true"
      },
      {
        name  = "controller.env[0].name"
        value = "AWS_ENI_LIMITED_POD_DENSITY"
      },
      {
        name  = "controller.env[0].value"
        value = "\"false\""
        type  = "string"
      },
    ]
  }
  addon_context = var.addon_context
}

But I don't see that enabled in the Deployment
kind: Deployment
apiVersion: apps/v1
metadata:
  name: karpenter
  namespace: karpenter
  uid: 72cd4750-e3f5-4c6f-81df-305cec749ec3
  resourceVersion: "881946"
  generation: 13
  creationTimestamp: "2022-07-19T12:56:12Z"
  labels:
    app.kubernetes.io/instance: karpenter
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: karpenter
    app.kubernetes.io/version: 0.13.2
    helm.sh/chart: karpenter-0.13.2
  annotations:
    deployment.kubernetes.io/revision: "9"
    meta.helm.sh/release-name: karpenter
    meta.helm.sh/release-namespace: karpenter
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: karpenter
      app.kubernetes.io/name: karpenter
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: karpenter
        app.kubernetes.io/name: karpenter
    spec:
      containers:
      - name: controller
        image: public.ecr.aws/karpenter/controller:v0.13.2@sha256:af463b2ab0a9b7b1fdf0991ee733dd8bcf5eabf80907f69ceddda28556aead31
        ports:
        - name: http-metrics
          containerPort: 8080
          protocol: TCP
        - name: http
          containerPort: 8081
          protocol: TCP
        env:
        - name: CLUSTER_NAME
          value: fernando-rare-anchovy
        - name: CLUSTER_ENDPOINT
          value: https://8F0612BE376069E8F02961CDADE488AF.gr7.us-east-1.eks.amazonaws.com
        - name: KARPENTER_SERVICE
          value: karpenter
        - name: SYSTEM_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: AWS_ENI_LIMITED_POD_DENSITY
          value: '"false"'
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "1"
            memory: 1Gi
        livenessProbe:
          httpGet:
            path: /healthz
            port: http
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 1
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /readyz
            port: http
            scheme: HTTP
          timeoutSeconds: 1
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      - name: webhook
        image: public.ecr.aws/karpenter/webhook:v0.13.2@sha256:e10488262a58173911d2b17d6ef1385979e33334807efd8783e040aa241dd239
        args:
        - -port=8443
        ports:
        - name: https-webhook
          containerPort: 8443
          protocol: TCP
        env:
        - name: CLUSTER_NAME
          value: fernando-rare-anchovy
        - name: KUBERNETES_MIN_VERSION
          value: 1.19.0-0
        - name: CLUSTER_ENDPOINT
          value: https://8F0612BE376069E8F02961CDADE488AF.gr7.us-east-1.eks.amazonaws.com
        - name: KARPENTER_SERVICE
          value: karpenter
        - name: SYSTEM_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
          requests:
            cpu: 100m
            memory: 50Mi
        livenessProbe:
          httpGet:
            path: /
            port: https-webhook
            scheme: HTTPS
          initialDelaySeconds: 30
          timeoutSeconds: 1
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /
            port: https-webhook
            scheme: HTTPS
          timeoutSeconds: 1
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: karpenter
      serviceAccount: karpenter
      securityContext:
        fsGroup: 1000
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: karpenter.sh/provisioner-name
                operator: DoesNotExist
      schedulerName: default-scheduler
      priorityClassName: system-cluster-critical
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app.kubernetes.io/instance: karpenter
            app.kubernetes.io/name: karpenter
  strategy:
    type: Recreate
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
status:
  observedGeneration: 13
  replicas: 1
  updatedReplicas: 1
  readyReplicas: 1
  availableReplicas: 1
  conditions:
  - type: Progressing
    status: "True"
    lastUpdateTime: "2022-07-21T10:08:57Z"
    lastTransitionTime: "2022-07-19T13:42:04Z"
    reason: NewReplicaSetAvailable
    message: ReplicaSet "karpenter-55cd5bcfcd" has successfully progressed.
  - type: Available
    status: "True"
    lastUpdateTime: "2022-07-21T10:36:52Z"
    lastTransitionTime: "2022-07-21T10:36:52Z"
    reason: MinimumReplicasAvailable
    message: Deployment has minimum availability.

Can this be summarized in that the eks_blueprints_kubernetes_addons_karpenter terraform module being used here does not seem to produce the expected K8s deployment?
Are you by any chance able to use this example instead to have Karpenter running with Terraform?
https://karpenter.sh/v0.13.2/getting-started/getting-started-with-terraform/

Can this be summarized in that the eks_blueprints_kubernetes_addons_karpenter terraform module being used here does not seem to produce the expected K8s deployment?
Are you by any chance able to use this example instead to have Karpenter running with Terraform? https://karpenter.sh/v0.13.2/getting-started/getting-started-with-terraform/

I'll try to give it a go... about to go on Annual Leave, so short on time

repo: aws/karpenter


text:
403 error when trying to download helm chart

Version

Karpenter Version: v0.25.0

Kubernetes Version: v1.21.0
Expected Behavior
Able to download helm chart
Actual Behavior
Receive a 403 error when trying to download helm chart (same for older versions too)
Error: pulling from host public.ecr.aws failed with status code [manifests v0.25.0]: 403 Forbidden
Steps to Reproduce the Problem
helm template karpenter oci://public.ecr.aws/karpenter/karpenter --version v0.25.0 --namespace karpenter \     --set settings.aws.defaultInstanceProfile=KarpenterInstanceProfile \     --set settings.aws.clusterEndpoint="" \     --set settings.aws.clusterName=game-server-cluster \     --set serviceAccount.annotations." \     --version v0.25.0 > karpenter.yaml
Resource Specs and Logs
n/a
Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

check out our helm troubleshooting section, particularly this entry: https://karpenter.sh/v0.25.0/troubleshooting/#helm-error-when-pulling-the-chart:~:text=If you are getting a 403 forbidden error%2C you can try docker logout public.ecr.aws as explained here

If you are getting a 403 forbidden error, you can try docker logout public.ecr.aws as explained here

Hi @bwagner5 ,
Thanks for getting back to me, apologies I missed the troubleshooting page, logging out of ecr has sorted
THanks

Meet the same issue and troubleshooting guide not work. Here are my cluster info

EKS 1.24
Helm 3.10.1
karpenter v0.25.0

and detailed debug info are listed here:
[ec2-user@ip-10-1-1-239 ~]$ docker logout public.ecr.aws
Not logged in to public.ecr.aws
[ec2-user@ip-10-1-1-239 ~]$ helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace karpenter --create-namespace   --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN}   --set settings.aws.clusterName=${CLUSTER_NAME}   --set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME}   --set settings.aws.interruptionQueueName=${CLUSTER_NAME}   --wait --debug
history.go:56: [debug] getting history for release karpenter
Release "karpenter" does not exist. Installing it now.
install.go:192: [debug] Original chart version: "v0.25.0"
DEBU[0000] resolving                                     host=public.ecr.aws
DEBU[0000] do request                                    host=public.ecr.aws request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=Helm/3.10.1 request.method=HEAD url="https://public.ecr.aws/v2/karpenter/karpenter/manifests/v0.25.0"
DEBU[0000] fetch response received                       host=public.ecr.aws response.header.content-length=58 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 28 Feb 2023 12:16:09 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.www-authenticate="Bearer realm=\"https://public.ecr.aws/token/\",service=\"public.ecr.aws\",scope=\"aws\"" response.status="401 Unauthorized" url="https://public.ecr.aws/v2/karpenter/karpenter/manifests/v0.25.0"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://public.ecr.aws/token/\",service=\"public.ecr.aws\",scope=\"aws\"" host=public.ecr.aws
DEBU[0000] do request                                    host=public.ecr.aws request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=Helm/3.10.1 request.method=HEAD url="https://public.ecr.aws/v2/karpenter/karpenter/manifests/v0.25.0"
DEBU[0000] fetch response received                       host=public.ecr.aws response.header.content-length=111 response.header.content-type="application/json; charset=utf-8" response.header.date="Tue, 28 Feb 2023 12:16:09 GMT" response.header.docker-distribution-api-version=registry/2.0 response.status="403 Forbidden" url="https://public.ecr.aws/v2/karpenter/karpenter/manifests/v0.25.0"
Error: pulling from host public.ecr.aws failed with status code [manifests v0.25.0]: 403 Forbidden
helm.go:84: [debug] pulling from host public.ecr.aws failed with status code [manifests v0.25.0]: 403 Forbidden

Ditto on later versions.
loaderd@lithium • $docker logout public.ecr.aws
Not logged in to public.ecr.aws
loaderd@lithium • $ helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace karpenter --create-namespace   --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN}   --set settings.aws.clusterName=${CLUSTER_NAME}   --set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME}   --set settings.aws.interruptionQueueName=${CLUSTER_NAME}   --wait --debug
history.go:56: [debug] getting history for release karpenter
Release "karpenter" does not exist. Installing it now.
install.go:178: [debug] Original chart version: "v0.27.0"
Error: pulling from host public.ecr.aws failed with status code [manifests v0.27.0]: 403 Forbidden
helm.go:84: [debug] pulling from host public.ecr.aws failed with status code [manifests v0.27.0]: 403 Forbidden

hello everyone here from 2024!
The issue with pulling the karpenter image still persists for me here.
➜  wednesday git:(wednesday) helm pull oci://public.ecr.aws/karpenter/karpenter --version v0.36.1 --debug
DEBU[0000] resolving                                     host=public.ecr.aws
DEBU[0000] do request                                    host=public.ecr.aws request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=Helm/3.13.1 request.method=HEAD url="https://public.ecr.aws/v2/karpenter/karpenter/manifests/v0.36.1"
DEBU[0000] fetch response received                       host=public.ecr.aws response.header.content-length=80 response.header.content-type="application/json; charset=utf-8" response.header.date="Sun, 12 May 2024 20:32:28 GMT" response.header.docker-distribution-api-version=registry/2.0 response.status="400 Bad Request" url="https://public.ecr.aws/v2/karpenter/karpenter/manifests/v0.36.1"
Error: unexpected status from HEAD request to https://public.ecr.aws/v2/karpenter/karpenter/manifests/v0.36.1: 400 Bad Request
helm.go:84: [debug] unexpected status from HEAD request to https://public.ecr.aws/v2/karpenter/karpenter/manifests/v0.36.1: 400 Bad Request

➜  wednesday git:(wednesday) docker logout
Removing login credentials for https://index.docker.io/v1/
➜  wednesday git:(wednesday) helm registry logout public.ecr.aws
Removing login credentials for public.ecr.aws
➜  wednesday git:(wednesday) helm pull oci://public.ecr.aws/karpenter/karpenter --version v0.36.1 --debug
DEBU[0000] resolving                                     host=public.ecr.aws
DEBU[0000] do request                                    host=public.ecr.aws request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=Helm/3.13.1 request.method=HEAD url="https://public.ecr.aws/v2/karpenter/karpenter/manifests/v0.36.1"
DEBU[0000] fetch response received                       host=public.ecr.aws response.header.content-length=58 response.header.content-type="application/json; charset=utf-8" response.header.date="Sun, 12 May 2024 20:39:55 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.www-authenticate="Bearer realm=\"https://public.ecr.aws/token/\",service=\"public.ecr.aws\",scope=\"aws\"" response.status="401 Unauthorized" url="https://public.ecr.aws/v2/karpenter/karpenter/manifests/v0.36.1"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://public.ecr.aws/token/\",service=\"public.ecr.aws\",scope=\"aws\"" host=public.ecr.aws
DEBU[0000] do request                                    host=public.ecr.aws request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=Helm/3.13.1 request.method=HEAD url="https://public.ecr.aws/v2/karpenter/karpenter/manifests/v0.36.1"
DEBU[0000] fetch response received                       host=public.ecr.aws response.header.content-length=79 response.header.content-type="application/json; charset=utf-8" response.header.date="Sun, 12 May 2024 20:39:55 GMT" response.header.docker-distribution-api-version=registry/2.0 response.status="404 Not Found" url="https://public.ecr.aws/v2/karpenter/karpenter/manifests/v0.36.1"
INFO[0000] trying next host - response was http.StatusNotFound  host=public.ecr.aws
Error: public.ecr.aws/karpenter/karpenter:v0.36.1: not found
helm.go:84: [debug] public.ecr.aws/karpenter/karpenter:v0.36.1: not found

The most interesting moment is that the image successfully pulled at the moment of initial install. But when later I'm trying to call terraform plan -out=tfplan the it fails to fetch the image.
The only solution I've found so far - temporarily disable this chunk of terraform code, plan, apply, reenable, plan, apply
resource "helm_release" "karpenter" {
  namespace           = "kube-system"
  name                = "karpenter"
  repository          = "oci://public.ecr.aws/karpenter"
  repository_username = data.aws_ecrpublic_authorization_token.token.user_name
  repository_password = data.aws_ecrpublic_authorization_token.token.password
  chart               = "karpenter"
  version             = "0.36.1"
  wait                = false

  values = [
    <<-EOT
    nodeSelector:
      karpenter.sh/controller: 'true'
    tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - key: karpenter.sh/controller
        operator: Exists
        effect: NoSchedule
    settings:
      clusterName: ${module.eks.cluster_name}
      clusterEndpoint: ${module.eks.cluster_endpoint}
      interruptionQueue: ${module.karpenter.queue_name}
    EOT
  ]

  lifecycle {
    ignore_changes = [
      repository_password
    ]
  }
}

Any suggestions por favor?

docker logout public.ecr.aws
This works
Reference : https://gallery.ecr.aws/karpenter/karpenter

helm registry logout public.ecr.aws fixes it.
Which means that somebody (helm?) caches the credentials for no reason.

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
Maven GitHub Action

closes #3

Branch protection should also be enabled, requiring this check to pass before merging

Branch protection should also be enabled, requiring this check to pass before merging

@embano1 I believe it can be done with add rule on branch in the repository settings, I don't have enough permissions to add new rule

@embano1 I believe it can be done with add rule on branch in the repository settings, I don't have enough permissions to add new rule

Yeah, you need permissions for that - or reach out to the team.

source: gharchive/pull-request

repo: aws/personalize-kafka-connector


text:
greengrass-bin: fix directory link, version upgrade 2.12.5 -> 2.12.6

${GG_ROOT}/alts/init/distro must be a symbolic link to /${GG_BASENAME}/packages/artifacts-unarchived/aws.greengrass.Nucleus/${PV}/aws.greengrass.nucleus
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

might fix:
https://github.com/aws4embeddedlinux/meta-aws/issues/8492
https://github.com/aws4embeddedlinux/meta-aws/issues/7926

license: MIT


text:
EKS k8s 1.19 - AMI 1.19-v20210414 - (combined from similar events): System OOM encountered, victim process:

What happened:
System OOM happens in some of nodes after upgrade to ks8 1.19 (using AMI v20210414). It seems this happens more for bigger nodes, like r5.4x and r5.8x.
  Warning  SystemOOM  18m                   kubelet, ip-10-10-10-10.eu-central-1.compute.internal  System OOM encountered, victim process: iptables, pid: 13778
  Warning  SystemOOM  18m                   kubelet, ip-10-10-10-10.eu-central-1.compute.internal  System OOM encountered, victim process: iptables, pid: 13782
  Warning  SystemOOM  18m                   kubelet, ip-10-10-10-10.eu-central-1.compute.internal  System OOM encountered, victim process: iptables, pid: 13836
  Warning  SystemOOM  18m                   kubelet, ip-10-10-10-10.eu-central-1.compute.internal  System OOM encountered, victim process: iptables, pid: 13853
  Warning  SystemOOM  17m                   kubelet, ip-10-10-10-10.eu-central-1.compute.internal  System OOM encountered, victim process: iptables, pid: 18796
  Warning  SystemOOM  17m                   kubelet, ip-10-10-10-10.eu-central-1.compute.internal  System OOM encountered, victim process: iptables, pid: 18808
  Warning  SystemOOM  17m                   kubelet, ip-10-10-10-10.eu-central-1.compute.internal  System OOM encountered, victim process: iptables, pid: 18819
  Warning  SystemOOM  17m (x538 over 2d6h)  kubelet, ip-10-10-10-10.eu-central-1.compute.internal  (combined from similar events): System OOM encountered, victim process: iptables, pid: 18883
  Warning  SystemOOM  17m                   kubelet, ip-10-10-10-10.eu-central-1.compute.internal  System OOM encountered, victim process: iptables, pid: 18854
  Warning  SystemOOM  17m                   kubelet, ip-10-10-10-10.eu-central-1.compute.internal  System OOM encountered, victim process: iptables, pid: 18880

What you expected to happen:
We did not have System OOM issue in EKS k8s 1.18
How to reproduce it (as minimally and precisely as possible):
EKS k8s 1.19
Nodes AMI: amazon-eks-node-1.19-v20210414
Region: Frankfurt (eu-central-1)
Anything else we need to know?:
Environment:

AWS Region: Frankfurt
Instance Type(s): r5.4xlarge, r5.8xlarge
EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.4
Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.19
AMI Version: amazon-eks-node-1.19-v20210414
Kernel (e.g. uname -a): Linux ip-10-10-10-10.eu-central-1.compute.internal 5.4.105-48.177.amzn2.x86_64 #1 SMP Tue Mar 16 04:56:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Release information (run cat /etc/eks/release on a node):

BASE_AMI_ID="ami-0849ada759754b5f5"
BUILD_TIME="Wed Apr 14 20:10:54 UTC 2021"
BUILD_KERNEL="5.4.105-48.177.amzn2.x86_64"
ARCH="x86_64"

Please note that when at first we upgraded to k8s 1.19, we used the AMI version v20210329, and we had a lot of node NotReady issues. Based on this comment (https://github.com/awslabs/amazon-eks-ami/issues/195#issuecomment-821195669), we moved to AMI v20210414. The node NotReady does not happen any more. However, we now have System OOM in some of the nodes.

We just completed a 1.16 -> 1.20 upgrade process and we're seeing similar things starting from when we hit 1.19.
We run managed node groups with the EKS-optimized AMI on c4.xl, c5.xl and m5.xl in eu-west-1 and as soon we hit 1.19.6-20210526 we started seeing an increase in pods memory usage, containers being OOM Killed, and System OOM. Given 1.19 for us was just an intermediate step towards 1.20, we continued with the upgrade but the issue is exactly the same on 1.20.4-20210526.
We'll try an AMI downgrade to 1.20.4-20210519 since it runs a different kernel version hoping this is just something affecting the 20210526 release but I'm not really sure, at this point we're running out of ideas.

Hi @roccozanni , any luck with v20210519? Thanks

@imriss we're not there yet. We discovered that in order to downgrade to that AMI (or in general, to start using custom ami instead of the default "latest" one) we need to rebuild the nodegroups from scratch, so it's taking a bit more time than initially planned

We tried to downgrade, the issue is still there, also with the very latest AMI version. We also noticed we were still running the 1.7 branch of the CNI plug-in, so we upgraded to 1.8. The issue is still there. We're completely out of ideas at this stage and we don't understand why this cluster is plagued by memory issues all over the place.

Hello, looks like this missed updating this. We're working on root causing the problem as well, one difference is that the 1.19+ AMIs use the Linux 5.4 Kernel. Our current hypothesis is that there was a change in memory management and the OOM Killer in the 5.4 Kernel - https://bugzilla.kernel.org/show_bug.cgi?id=207273#c2
We are investigating this and will provide an update once we have some more details.

@abeer91 thanks for acknowledging this. Do you have any suggestion for any temporary workaround that can be put in place to reduce the likelihood of this to happen while a long term solution is being figured out?

Sorry for the lack of traction here. The AmazonLinux team identified some patches that they expect to resolve this issue, which is likely going to be released sometime in October. We will update here when those are available.
Alternatively, customers have two options to work around this issue until then:

Downgrade the kernel to 4.14, which is already present in the AMI.
Upgrade to the 5.10 kernel. We know of one customer who believes this resolved the issue for them. Running amazon-linux-extras install kernel-5.10 and rebooting the instance should do it.

That being said, neither of these options have been validated through our release testing, so while both kernels are officially supported by AL2, I'd recommend doing so cautiously and ideally testing first.

AmazonLinux has released a new 5.4 kernel that includes some patches that should help with this issue. The latest AMIs, v20211004 or later, will have the patched kernel.
Please let us know if they fixes (or doesn't fix) this issue!

I'm still seeing this issue on some of our larger nodes.

Soem more details on this - We are also seeing this effect larger instances more than smaller ones.
One of our clusters is on things in the size range around m5.2xl and r5.2xl, and is fine. The other is on m5.8xl and r5.12xl sized machines and that has far more errors.
It also appears (without doing stats), that the more recent kernels may make it wrose. Our stablest cluster had kernels based on 5.4.149-73.259.amzn2.x86_64 and other less stable one used 5.4.156-83.273.amzn2.x86_64. Downgrading back to that 5.4.149 showed immediate improvement.

This is still ongoing, and after some more research, my previous comments were mostly, if not entirely incorrect. The kernel version doesn't seem to matter too much, and the other cluster I mentioned as not being affected is now affected. If I find any more information,  or a solution - I'll reply to this thread

I've recently discovered the --eviction-hard flag for kubelet, and once I set that up, the errors appear to have gone away. I only recently applied this change, so it's too soon to be sure it entirely fixed the problem, but I hope this helps someone else.

We have started the roll out to 1.19-v20211206 base on the suggestion from @mmerkes

It is looking like in our case the "System OOM" events are a false positive. Not sure if this will be the case for others coming to this GH issue.

The K8s OOM watcher relies on cAdvisor
K8s version 1.19 uses cAdvisor 0.37.5 (or earlier, depending on K8s build)
cAdvisor had a bug where it incorrectly parsed OOM events resulting in all OOM events being reported by K8s as "System OOM"
That bug was included in cAdvisor release 0.39.0, which is included included in K8s version v1.21

I've recently discovered the --eviction-hard flag for kubelet, and once I set that up, the errors appear to have gone away. I only recently applied this change, so it's too soon to be sure it entirely fixed the problem, but I hope this helps someone else.
EDIT: Update - this is only a partial fix, I combined it with some adjustments to my pod specs to avoid large burstable regions of memory, which has been helpful, but it's still not perfect.

@AlexMichaelJonesNC Did this help in your case in the long term?

Does anyone know if that issue is fixed in any of the 1.19 AMIs?

This should be resolved by the cAdvisor update in k/k; please let us know if you're still seeing this on the latest binaries in s3://amazon-eks.

authors: AlexMichaelJonesNC Satheesh-Balachandran abeer91 cartermckinnon imriss jaroslaw-meller maxenglander mmerkes roccozanni

repo: awslabs/amazon-eks-ami


text:
feat(assets): enable local tooling scenarios such as lambda debugging

Adds CloudFormation resource metadata which enables tools such as SAM
CLI to find local assets used by resources in the template.
See design document under design/code-asset-metadata.md
Fixes #1432

@jfuss please review this and let me know if this looks good

Looking now.

authors: eladb jfuss

repo: awslabs/aws-cdk


text:
chore: Add version suffixes to Java & .NET artifacts

Label all Java artifacts with .DEVPREVIEW, and all .NET artifacts
with -devpreview.
Fixes: awslabs/cdk-ops#373

Please read the contribution guidelines and follow the pull-request checklist.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@RomainMuller this is ready for merge

authors: RomainMuller eladb


text:
Proposal for final CodePipeline Actions scheme: naming and code organization

This PR contains what will (hopefully) be the final iteration of the way we structure our CodePipeline Actions.
The scheme is the following:

All Actions are in their dedicated modules, named after the service they represent integrations with (so, a CodeCommit Action lives in aws-codecommit-codepipeline, the 2 CodeBuild Actions live in aws-codebuild-codepipeline, etc.).
The names of the Actions do not include the word 'Pipeline'. The reason for that is that we will expect (and promote it in our docs, examples and code) that customer will import them with the word 'pipeline' included in the namespace prefix they use.

Examples:
import codebuildPipeline = require('@aws-cdk/aws-codebuild-codepipeline');
import codecommitPipeline = require('@aws-cdk/aws-codecommit-codepipeline');

new codecommitPipeline.SourceAction(sourceStage, 'source', {
    // ...
});
new codebuildPipeline.BuildAction(buildStage, 'build', {
    // ...
});

There is a new module, aws-codepipeline-aws-actions, that has all of the CodePipeline integrations packages (aws-codecommit-codepipeline, aws-codebuild-codepipeline, etc.) as dependencies. It re-exports all of the Action under the namespace of the integration service. This is in oder to aid in discoverability, and make it easier for customers to just get all of the integrations with one import.

Examples:
import codepipelineAwsActions = require('@aws-cdk/aws-codepipeline-aws-actions');

new codepipelineAwsActions.codecommit.SourceAction(sourceStage, 'Source', {
    // ...
});

new codepipelineAwsActions.codebuild.BuildAction(buildStage, 'Build', {
    // ...
});

Commits:

[BREAKING] Rename Lambda's PipelineInvokeAction to just InvokeAction.
[BREAKING] Rename CodeCommit's PipelineSource to SourceAction.
[BREAKING] Rename CodeBuild's PipelineBuildAction to simply BuildAction.
Introduce the 'aws-codepipeline-aws-actions' package.

By submitting this pull request, I confirm that my contribution is made under
the terms of the beta license.

Moved to #459 after fork problems when making the repo public.


text:
Removed .git from .dockerignore

Make files which get executed inside the docker needs .git to be there, hence this should not be ignored.
*Issue #, if available: Running docker build fails!
# docker build . -t img2lambda

.
.
find: '.git/': No such file or directory
.
.

*Description of changes:
removed .git from the .dockerignore file so the copy command inside the docker file copy the .git into the containers building.
Tested it and it succeed after the change.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Thanks!

authors: arashilmg clareliguori


text:
feat:(parser): Add KinesisFirehoseEvent

Similar to what https://github.com/awslabs/aws-lambda-powertools-python/issues/1059 mentions
Add support for a KinesisFirehoseEvent parser model and envelope for request and response.

This appears to be a duplicate of feat(data-classes): Add KinesisFirehoseEvent #1059.  Since the other issue has details, I'd suggest this issue be closed/rejected.

@gwlester Parser and data classes are two completely different utilities and implementations.

@ran-isenberg -- sorry you are correct.  Too late at night!  I did not see the what inside of feat!!!
Should I delete this and the other comment??

no worries, dont delete

yep, we made things nice and confusing :)

released

authors: gwlester heitorlessa michaelbrewer ran-isenberg

repo: awslabs/aws-lambda-powertools-python


text:
Correct cost information no longer being displayed

Description
Following an update to the S3 console the Cost bucket replication steps in the [Set up the cost feature] section of the implementation guide are no longer valid.  This can lead to $0 values being displayed in Perspective for cost generating resources.
How do I know if I am affected?

You have deployed AWS Perspective after 01 Nov 2020 and have configured the Cost feature with replication
After 24 hours you are seeing $0 values when viewing cost generating resources in Perspective
You are seeing [Replication status : FAILED] when viewing a Cost and Usage report in its S3 bucket

Remediation
Set up bucket replication
Set up replication to the S3 bucket created during deployment. The S3 bucket is named with the following format: aws-perspective-<*deployment-accountID*>-<*deployment-region*>-cost-bucket.  This allows AWS Perspective to process the cost and usage data and map it to the resources it has already discovered.  Please perform the following actions to complete replication configuration.
The account where Cost and Usage reports are configured
1. Sign in to the Amazon S3 console - https://console.aws.amazon.com/s3/
2. Select the S3 bucket created when configuring your AWS Cost and Usage Report
3. Select the [Management] tab, and then choose [Replication rules]
4. Choose  [Create replication rule]
5. Within the [Replication rule] configuration section enter a descriptive rule ID for the [Replication rule name]
6. In the [Source bucket] section select [This rule applies to all objects in the bucket] to configure the rule scope
7. In the [Destination] section configure the following
    1. Select [Specify a bucket in another account]
    2. Enter the account ID
    3. Enter the value for the [Bucket name] with logical-id [PerspectiveCostBucket]
    4. Select the checkbox next to [Change object ownership to destination bucket owner]
8. In the [IAM] section
    1. Under [IAM role], choose [Create a new role]
    2. If a replication role already exists select it and ensure it includes the [Replication Role Policy Actions] below
9. Log into the AWS console where AWS Perspective is installed, navigate to the [S3] service page and select the bucket from [step 3] above
10. From the destination buckets [Management] tab scroll down to the [Replication rules] section and select [Receive replicated objects] from the [Actions] drop-down menu
11. Within the [Source bucket account settings] section
    1. Add a value for the [Source bucket account ID]
    2. Select [Generate policies]
    3. From the newly created [Policies] section expand the [Bucket policy]
    4. Select the option to [Include permission to change object ownership to destination bucket owner]
    5. Select [Copy] to grab the generated policy

The account where AWS Perspective is deployed
1. Navigate to the [S3] service page and find the bucket with the logical ID of PerspectiveCostBucket
2. Select the [Permissions] tab, scroll down to the [Bucket policy] section and click on [edit]
3. Paste the previously copied policy and click on [Save changes]

When the reports are in the AWS Perspective account you will start to see cost data appearing on the bounding boxes and individual resources
Replication Role Policy Actions

IAM Policy Action
Description

s3:ReplicateObject
Grants permission to replicate objects and object tags to the destination bucket

s3:ReplicateDelete
Grants permission to replicate delete markers to the destination bucket

s3:ReplicateTags
Grants permission to replicate object tags to the destination bucket

s3:ObjectOwnerOverrideToBucketOwner
Grants permission to change replica ownership

s3:ListBucket
Grants permission to list some or all of the objects in an Amazon S3 bucket

s3:GetReplicationConfiguration
Grants permission to get the replication configuration information set on an Amazon S3 bucket

s3:GetObjectVersionForReplication
Grants permission to replicate both unencrypted objects and objects encrypted with SSE-S3 or SSE-KMS

s3:GetObjectVersionAcl
Grants permission to return the access control list (ACL) of a specific object version

s3:GetObjectVersionTagging
Grants permission to return the tag set for a specific version of the object

s3:GetObjectRetention
Grants permission to retrieve the retention settings for an object

s3:GetObjectLegalHold
Grants permission to get an object's current Legal Hold status

Yes, I am also seeing the same cost issue. It's not showing correct cost.

repo: awslabs/aws-perspective


text:
chore: explore analysis performance improvement (#1410)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license
Summary
cherry pick #1410
Implementation highlights
(describe how the merge request does for feature changes, share the RFC link if it has)
Test checklist

[ ] add new test cases
[ ] all code changes are covered by unit tests
[ ] end-to-end tests

[ ] deploy web console with CloudFront + S3 + API gateway
[ ] deploy web console within VPC
[ ] deploy ingestion server

[ ] with MSK sink
[ ] with KDS sink
[ ] with S3 sink

[ ] deploy data processing
[ ] deploy data modeling

[ ] new Redshift Serverless
[ ] provisioned Redshift
[ ] Athena

[ ] deploy with reporting
[ ] streaming ingestion

[ ] with Redshift Serverless
[ ] with provisioned Redshift

Is it a breaking change

[ ] add parameters without default value in stack
[ ] introduce new service permission in stack
[ ] introduce new top level stack module

Miscellaneous

[ ] introduce new symbol link source file(s) to be shared among infra code, web console frontend, and web console backend

SonarQube Quality Gate Result

Result: :exclamation: Error
Triggered by @llmin on pull_request

Metric
Status
Value
Error Threshold

Reliability rating
:white_check_mark: OK
1
> 1

Security rating
:white_check_mark: OK
1
> 1

Sqale rating
:white_check_mark: OK
1
> 1

Coverage
:exclamation: Error
73.50
< 80

Duplicated lines density
:white_check_mark: OK
5.20
> 30

Blocker violations
:white_check_mark: OK
0
> 0

Bugs
:white_check_mark: OK
0
> 0

Code smells
:white_check_mark: OK
8
> 40

Critical violations
:white_check_mark: OK
0
> 0

Major violations
:white_check_mark: OK
0
> 0

Vulnerabilities
:white_check_mark: OK
0
> 0

View on SonarQube
updated: 6/4/2024, 23:24:24 (UTC+0)

SonarQube Code Analytics
Quality Gate failed

Additional information
The following metrics might not affect the Quality Gate status but improving them will improve your project code quality.
Issues
 0 Bugs
 0 Vulnerabilities
 0 Code Smells
Coverage and Duplications
No data Coverage
No data Duplication

SonarQube Quality Gate Result

Result: :exclamation: Error
Triggered by @llmin on pull_request

Metric
Status
Value
Error Threshold

Reliability rating
:white_check_mark: OK
1
> 1

Security rating
:white_check_mark: OK
1
> 1

Sqale rating
:white_check_mark: OK
1
> 1

Coverage
:exclamation: Error
73.60
< 80

Duplicated lines density
:white_check_mark: OK
5.20
> 30

Blocker violations
:white_check_mark: OK
0
> 0

Bugs
:white_check_mark: OK
0
> 0

Code smells
:white_check_mark: OK
8
> 40

Critical violations
:white_check_mark: OK
0
> 0

Major violations
:white_check_mark: OK
0
> 0

Vulnerabilities
:white_check_mark: OK
0
> 0

View on SonarQube
updated: 6/5/2024, 02:28:42 (UTC+0)

authors: llmin zxkane

repo: awslabs/clickstream-analytics-on-aws


text:
Recipe synthetic data genration small refactoring

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Job PR-327/1 is complete.
Docs are uploaded to http://gluon-ts-staging.s3-accelerate.dualstack.amazonaws.com/PR-327/1/index.html

Job PR-327/2 is complete.
Docs are uploaded to http://gluon-ts-staging.s3-accelerate.dualstack.amazonaws.com/PR-327/2/index.html

Looks like mypy is complaining now. Check in requirements/requirements-setup.txt which versions of black and mypy should be used.
black==19.3b0
mypy==0.630

Job PR-327/3 is complete.
Docs are uploaded to http://gluon-ts-staging.s3-accelerate.dualstack.amazonaws.com/PR-327/3/index.html

Codecov Report

Merging #327 into master will increase coverage by 39.53%.
The diff coverage is 75.49%.

@@            Coverage Diff             @@
##           master    #327       +/-   ##
==========================================
+ Coverage   40.66%   80.2%   +39.53%
==========================================
  Files         142     142
  Lines        8150    8177       +27
==========================================
+ Hits         3314    6558     +3244
+ Misses       4836    1619     -3217

Impacted Files
Coverage Δ

src/gluonts/dataset/artificial/_base.py
36.67% <0%> (+1.43%)
:arrow_up:

src/gluonts/dataset/artificial/recipe.py
74.61% <76.23%> (+9.76%)
:arrow_up:

src/gluonts/core/exception.py
62.5% <0%> (+2.08%)
:arrow_up:

src/gluonts/model/trivial/constant.py
76.92% <0%> (+11.53%)
:arrow_up:

src/gluonts/kernels/_kernel_output.py
87.09% <0%> (+12.9%)
:arrow_up:

src/gluonts/model/seasonal_naive/_estimator.py
100% <0%> (+14.28%)
:arrow_up:

src/gluonts/model/npts/_estimator.py
100% <0%> (+14.28%)
:arrow_up:

src/gluonts/model/deepar/_estimator.py
100% <0%> (+15.09%)
:arrow_up:

... and 94 more

Job PR-327/5 is complete.
Docs are uploaded to http://gluon-ts-staging.s3-accelerate.dualstack.amazonaws.com/PR-327/5/index.html

authors: codecov-io jgasthaus lostella szha

repo: awslabs/gluon-ts


text:
Frequent blank transcripts

I'm testing a modified version of this solution, with the stock SuperJAR provided here.  I'm seeing a decent amount of blank text coming back from Transcribe, though -- probably 5-10% of requests.
Logging out the response object shows that the Java Lambda is returning properly (and the execution times are in line with successful ones), it's just blank data:
{'words': [], 'transcript': ''}
Is this just quirkiness of Transcribe and to be expected, or am I running into a problem with limits, etc?  I do see the rate of blanks rise when I run multiple streams simultaneously.  I'm using 6-second segments, so it's being asked to process 10 segments/min per stream.

Hi Joshua.
Possibly Transcribe didn't see spoken text during that 6 second segment. To rule out any throttling you can try to deploy in a different region that doesn't have any Transcribe use, since the throttling is region based. The limits are pretty low at 10 concurrent requests (Streams), those can be increased with a ticket through the console.

Thanks, Eddie.
I can confirm that the submitted segments do contain audio; the loop I'm playing back for testing is a pretty clean studio interview, and a chunk that may be missed on one loop will come out with text on the next loop several minutes later.
Does each request to Transcribe (ie, each 6-second segment) count as a Stream, or is it some lower number based on how many segments are being worked on at the same time? (ie, if processing of two segments overlap, it would count as 2, but if each segment finishes prior to the next one beginning, it's seen as just 1 Stream)?
Not knowing how the internals of the Java portion work, I'm also curious how the short segments lengths interact with the Transcribe billing apparatus -- the minimum charge is noted as 15 seconds.  Will each of my 6-second segments be billed for 15 seconds, or are they somehow combined and I will be billed based on the overall runtime of the live broadcast?

We have an update coming soon that changes to use long running Transcribe. Since the billing changed to 15 seconds, each 6 second segment is billed as 15 seconds. This new implementation has a long running stream to Transcribe using ECS.

authors: EddieGoynesAmazon joshua-wyatt


text:
Support for Web Identity Token authentication

Tell us more about this new feature.
We're interested in using OIDC tokens as the identity medium for mountpoint. In Python, this flow looks like:
def get_s3_client(role_arn):
    sts_client = boto3.client("sts")

    # Assume role with Web Identity
    credential_response = sts_client.assume_role_with_web_identity(
        RoleArn=role_arn, RoleSessionName="OIDCSession", WebIdentityToken=IDENTITY_TOKEN_HERE
    )

    # Extract credentials
    credentials = credential_response["Credentials"]
    return boto3.client(
        "s3",
        aws_access_key_id=credentials["AccessKeyId"],
        aws_secret_access_key=credentials["SecretAccessKey"],
        aws_session_token=credentials["SessionToken"],
    )

We can hack this functionality on by making the sts flow ourselves, but we're wondering if this is something the mountpoint team has on the roadmap - it would be a big unlock for us.

Hi @pawalt, thanks for creating this issue.
Mountpoint already supports using web identity tokens using the AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN environment variables.
See the AWS docs for more information.
I'll be closing this, but if you find any issues running Mountpoint with OIDC, feel free to add additional comments or create a new issue.

@muddyfish Oh great I didn't realize this. Thanks!

authors: muddyfish pawalt


text:
Feature request: Send Traces to Observability Frameworks

Use case
For production use-cases, we need some way to send traces of the orchestrator and agents to an Observability Framework like Langfuse, so they can be visualized there nicely and combined with user feedback.
Right now, it is unclear to me how this is possible. Only getting text from logs is not sufficient for debugging when a lot of users utilize the system.
Solution/User Experience
Need some kind of mechanism to write traces to external SDKs like the one from Langfuse (https://langfuse.com/docs/integrations/amazon-bedrock).
Alternative solutions
No response

@brnaba-aws  Thanks for your help. I will try to implement this to send the input, output and reasoning steps to LangFuse like in this example from AWS (https://aws.amazon.com/de/blogs/machine-learning/accelerate-analysis-and-discovery-of-cancer-biomarkers-with-amazon-bedrock-agents/):

Hi @maxritter
Please let us know how it goes, we might provide this as an example in our repo for others.

We have started some experimentation with langfuse and minimal modification in our framework.
So far so good. We need to finalize a bit the design

@brnaba-aws Thanks for your efforts! I do not have time in the next couple of days to tackle this, but the way I thought about implementing it would be using https://langfuse.com/docs/sdk/python/low-level-sdk to include:

Traces (Tracks agent execution from input query to final output including token usage and all spans / generations)
Spans (Log sub-events like Knowledge Base call including input / output, execution time)
Generations (Metadata about model calls including input / output messages, cost, execution time)

Here is a good example of how the different type of events can be captured after calling the invoke_agent function from Boto3: https://github.com/aws-samples/amazon-bedrock-agents-cancer-biomarker-discovery/blob/c86dd6c07a445d8c3134d2eb11db5667ebf49945/streamlitapp/app.py#L230
It's not a complete list of all things that can be captured, there is even more in https://docs.aws.amazon.com/bedrock/latest/userguide/trace-events.html.

Thanks for sharing this. I think, as a best option for us, we would have a callback for each steps (before/after calling LLM, tools, Retriever, Agents...)

authors: brnaba-aws maxritter

repo: awslabs/multi-agent-orchestrator


text:
Keep getting "Linear Solver not converging" and "PCG Solver not converging"

Hi,
I am attempting to run simulation using Palace for the first time. However, I keep getting "Linear Solver not converging" and "PCG Solver not converging". I have been trying different possible ways to solve it, for example lowering convergence tolerance or increasing Maximum number of iterations, etc. But still got the same error.
Any suggestions on this issue?

gmsh_palace_example_2wb.pdf
gmsh_palace_example_2wb.json
These are the msh file and config file we are using. (Since .msh file can not be attached here, I converted into a pdf file)

Hi @Jenny-Huang-629, thanks for sharing your mesh. I think there is a problem with the definition of your mesh and geometry. I am seeing that you have an inconsistent overlapping mesh at the outer boundary of your substrate and the outer box of your computational domain. Here is what I see when I open your mesh in Gmsh:

You probably need to fix your mesh generation code to avoid this and ensure your mesh is well formed. My suspicion is that is why Palace is giving warning messages.

Hi, thanks for replying!
Do you know if there is a way to remove the overlap in the mesh through Gmsh?

This needs to be done at the geometry level before generating the mesh, since it would appear that the 3D geometry model doesn't know that these two surfaces are overlapping. How are you constructing your 3D geometry? Is it through the Gmsh API?

I am constructing the geometry using qiskit metal and exporting just the wireframe of the design (with no mesh) as a .msh file using the native GmshRenderer. Then, I add the mesh in using Gmsh and export it from Gmsh.

I have not used the Qiskit Metal Gmsh renderer myself so am not sure how to fix the issue there unfortunately. There might also be a way to resolve the issue in Gmsh after you import it, but again I'm not sure. The problem is clear, though. Somehow your outer "vacuum_box" domain is being meshed without conformity to the inner "dielectric_(layer 3)" domain. In fact it looks like the "vacuum_box" domain might be misformed altogether, as visualizing just that mesh domain looks incorrect even outside of the substrate:

Hi @Jenny-Huang-629, did you ever get a chance to revisit this an investigate your geometry and mesh? I am inclined to close the issue for now as mesh related and not related to Palace, but let me know if there is reason to keep it open.

repo: awslabs/palace


text:
Cleanup in geodata.hpp/cpp

Minor cleanup of geodata helper functions and declarations, in preparation for palace::Mesh class. Includes some const correctness fixes for mfem::Mesh enabled by https://github.com/awslabs/palace/pull/151.

https://github.com/awslabs/palace/pull/164 minor mfem patch for the segfault issue with "SaveAdaptMesh"

authors: hughcars sebastiangrimberg


text:
Fix reference of uninitialized values

This commit zeroes out structures that valgrind complained were
referenced before initialization.
s2n_free looks at the value of 'mlocked' which isn't initialized
in a number of locations.
s2n_rsa_keys_match also complained about an invalid memcmp.
This may not be the optimal way to solve these problems, but zeroing
out as indicated allows our application to run without valgrind errors.

It looks like the development guide still shows the s2n_blob structure as containing only data and size.  s2n_blob_init doesn't initialize those values, so it may be better to update it in addition to these changes.


text:
Do not split string shapes in HTTP header

Issue #, if available: https://github.com/awslabs/aws-sdk-rust/issues/122
Description of changes:
This PR skips splitting header values by , character if the target shape is string. It should fix deserialization of string header values containing , characters like x-amz-expiration header.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_Example_4
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

resolved in #525 thanks for debugging @eagletmt !!

authors: eagletmt rcoh

repo: awslabs/smithy-rs


text:
repr for the search domain objects

Dear syne-tune team, thank you for a great library!
A useful feature for the config space domains: E.g: randint: https://github.com/awslabs/syne-tune/blob/main/syne_tune/config_space.py#L1032 would be to define the__repr__ function for the domain object that would, for example, display the lower and the upper bounds. This would be useful for logging of a particular search job on the user side to know how the config space was set for a given experiment.

Hey! Is this something I can work on?

This is already done. randint, uniform, choice are just functions. The classes of objects they create, all have __repr__ implemented.

I'll close this one, but if I missed something, please reopen

Dear @mseeger and @geoalgo  thank you for looking at this. I see what you mean but for some return objects it does not seem to be the case, please see:
>>> print(syne_tune.__version__)
0.3.0
>>> from syne_tune.config_space import uniform, loguniform, choice
>>> print(uniform(0.001, 0.01))
<syne_tune.config_space.Float object at 0x1034462e0>
>>> print(loguniform(0.001, 0.01))
<syne_tune.config_space.Float object at 0x103446c10>
>>> print(choice([0.001, 0.01])) # For example choice seems to give a nice print
choice([0.001, 0.01])

Thanks a lot @martinferianc for providing a simple example.
@martinferianc do you want to work on this or do you prefer to let @Vinayak-Kannan do it?
Btw, given what Martin provided, it would be great to add unit-tests that checks that repr provides the expected string.
For sending the PR, there is a script to run to add license automatically and format the code, you can take a look at our contributing guideline.

Please have a look at the current main branch:
https://github.com/awslabs/syne-tune/blob/main/syne_tune/config_space.py#L392
We have not released this yet. In general, if you like to know whether something has been done, please check the main branch.
@martinferianc, @Vinayak-Kannan : I'll shortly put up some issues we'd love to work one of you on. Thanks a lot for your interest!

Indeed, thanks for the catch Matthias it is already done in mainline.
That being said, there would still be lots of value in adding an unit-tests for the different cases which can also be a good first issue :-)

authors: Vinayak-Kannan geoalgo martinferianc mseeger

repo: awslabs/syne-tune


text:
Auto-configure credentials and region with AWS SDK v2.

Fixes #54

I have created a discussion related to spring cloud aws 3 and moving to aws sdk v2.

Thanks @arunkpatra for a review! I forgot about spring.factories, good catch!

authors: eddumelendez maciejwalkowiak

repo: awspring/spring-cloud-aws


text:
XYB Jpegli

Add an option for converting to XYB too. I tested it yesterday (with cjpegli --xyb) and it shaved off another 4% without any quality loss (other than a little bit you wont ever notice without pixel peeping)
Related:
https://github.com/T8RIN/ImageToolbox/issues/920

I'm a little sceptical about XYB feature in android. It works on behalf of ICC profiles and the most of android devs even have no idea what is it, even android itself just started support ICC profiles only from API 34+ (even not all famous software properly handle ICC profiles all the times). I think it is not really needed for real end user especially with really high chance that part of software will not support ICC profile and when it will that 4% compressed will cost twice / or triple decoding time due to quite slow implementation of rendering ICC.
So between a choice of generation potentially slow jpegs or perhaps invalid rendering I'd like to avoid any features that have a bounds to ICC.

So the question is: Is it really 4% worth 100-200%+ rendering speed decrease, and highly likely invalid rendering on some software? As to me it is not sounds as a really good trade-off

Didnt know it just used ICC, so yeah you can ignore this if you wish to not deal with that and android doesnt like it

Yeah, the whole idea of XYB if to use ICC. The image info saved in XYB colorspace instead of more traditional YUV, CMYK or RGB and use ICC as transfer function between the XYB and RGB. That means ICC not should be handled but must be handled.
That is not only android. ICC in full support properly done very rear.

So if you unsure that software that will use images with ICC will handle it better avoid this one.
And I'm not sure.

authors: awxkee kremzli

repo: awxkee/aire


text:
axtboot  init  를 다운로드 받았는데 컴파일 에러가 나서 확인 부탁드려요

Qxxxxxx   로 시작하는 클래스를  소스트리에서 찾을 수가 없습니다.
설치가 잘못된 것인가요?
$mvn install
$npm install
로 패키지 설치는 했습니다.
확인 부탁드립니다.
package zoe.youngplussoft.antimalware.domain;
import zoe.youngplussoft.antimalware.domain.code.QCommonCode;
import zoe.youngplussoft.antimalware.domain.file.QCommonFile;
import zoe.youngplussoft.antimalware.domain.program.QProgram;
import zoe.youngplussoft.antimalware.domain.program.menu.QMenu;
import zoe.youngplussoft.antimalware.domain.user.QUser;
import zoe.youngplussoft.antimalware.domain.user.auth.QUserAuth;
import zoe.youngplussoft.antimalware.domain.user.auth.menu.QAuthGroupMenu;
import zoe.youngplussoft.antimalware.domain.user.role.QUserRole;
import com.chequer.axboot.core.domain.base.AXBootBaseService;
import com.chequer.axboot.core.domain.base.AXBootJPAQueryDSLRepository;
import java.io.Serializable;
public class BaseService<T, ID extends Serializable> extends AXBootBaseService<T, ID> {
protected QUserRole qUserRole = QUserRole.userRole;
protected QAuthGroupMenu qAuthGroupMenu = QAuthGroupMenu.authGroupMenu;
protected QCommonCode qCommonCode = QCommonCode.commonCode;
protected QUser qUser = QUser.user;
protected QProgram qProgram = QProgram.program;
protected QUserAuth qUserAuth = QUserAuth.userAuth;
protected QMenu qMenu = QMenu.menu;
protected QCommonFile qCommonFile = QCommonFile.commonFile;

protected AXBootJPAQueryDSLRepository<T, ID> repository;

public BaseService() {
    super();
}

public BaseService(AXBootJPAQueryDSLRepository<T, ID> repository) {
    super(repository);
    this.repository = repository;
}

}

http://stackoverflow.com/questions/40580093/axboot-baseservice-java-compile-error
Stackoverflow에 답변해드렸습니다 :)

감사합니다.
--------- 원본 메일 ---------
    보낸사람: Brant <notifications@github.com>
    받는사람: axboot/ax-boot-framework <ax-boot-framework@noreply.github.com>

        참조: peterkhang <mgkaki@daum.net>, Author <author@noreply.github.com>

    날짜: 2016년 11월 13일 일요일, 21시 06분 02초 +0900
    제목: Re: [axboot/ax-boot-framework] axtboot init 를 다운로드 받았는데 컴파일 에러가 나서 확인 부탁드려요 (#25)

    http://stackoverflow.com/questions/40580093/axboot-baseservice-java-compile-error

Stackoverflow에 답변해드렸습니다 :)
—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or mute the thread.

authors: brant-hwang peterkhang

repo: axboot/ax-boot-framework

license: mit

license_source: bigquery


text:
chore(its): move test.rs to tests folder

AXE-6575

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.74%. Comparing base (2d5e74c) to head (c278b6b).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #76      +/-   ##
==========================================
- Coverage   99.24%   94.74%   -4.51%
==========================================
  Files          32       31       -1
  Lines        1998     1845     -153
==========================================
- Hits         1983     1748     -235
- Misses         15       97      +82

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

authors: AttissNgo codecov-commenter

repo: axelarnetwork/axelar-cgp-soroban


text:
feat: tool for field declaration completeness checks

This introduces a tool that can be integrated into IDE linting checks or run separately to find structs that are declared via object initializer but do not assign values to all fields.

moving to utils repo

repo: axelarnetwork/axelar-core


text:
chore: axelar api dependencies

when running @axelar/api from a script, it complained that axelar/utils was not found. so making it an explicit dependency of the package

hmm, these failing tests related to frozen lockfile are weird. probably silent error but i'll check it out

hmm, these failing tests related to frozen lockfile are weird. probably silent error but i'll check it out
nvm, i just had to rebuild the lockfile. should be good now. the failing test is unrelated to this PR

repo: axelarnetwork/axelarjs


text:
Connection resiliency

Fix for the ECONNRESET errors.
This PR:

improves error logging, makes clear where the error occured; in the grunt-saucelabs library or in the onTestComplete callback
handles transient network- and HTTP errors in the grunt-saucelabs library

Would you say this fixes the issue for good?
Any idea where the issue arrises? You think it's just that sometimes connections drop? We were thinking that maybe it has to do with running sauce tunnel, maybe the tunnel closes before the last test finishes.

I'm not 100% sure.
I suspect that these node.js commits may relate to the error:
joyent/node@14a4245
joyent/node@88686aa
The first one is in node since v0.9.10, the second one since v0.10.3. I'll run tests against versions where none, one and both of the commits are applied.
Until then, could you, @LaurentGoderre or @cvrebert monkey patch your grunt-saucelabs package and run your tests locally? To check if the situation is improved?
The monkey patching consists of

copy the files from this PR to the node_modules/grunt-saucelabs/src and node_modules/grunt-saucelabs/tasks subdirectories of your projects
installing the requestreply package (npm install requestreply) in the node_modules/grunt-saucelabs subdirectory of your projects

I will be able to do that on Thursday, is that ok?

I wonder if we can split this into two PR, the first two commits are good even if the rest doesn't fix the problem

Sure, I will put the logging improvements into a separate PR if the fix does not work.
Speaking of which, I have run my test suite (98 tests) a couple of times under different node.js versions with both the request and the requestretry package. Under node.js v0.8.25 each test run finished successfully, with both the request and the requestretry packages. Under node.js v0.10.2 there were weird ECONNRESET errors with both the request and the requestretry packages, which even requestretry could not handle. Under node.js v0.10.3 with the request package each test run failed with ECONNRESET. With the requestretry package each test run succeeded.

node.js
request
requestretry

0.8.25
:white_check_mark:
:white_check_mark:

0.10.2
:no_entry_sign:
:no_entry_sign:

0.10.3
:no_entry_sign:
:white_check_mark:

0.10.35
:no_entry_sign:
:white_check_mark:

It seems that these ECONNRESET errors are indeed transient network errors, which were swallowed in node.js before v0.9.10. After v0.9.10 these errors surfaced, but before v0.10.3 there was some bug in node.js which prevented requestretry to handle these.
In short, this PR fixes the ECONNRESET errors under node v0.10.3+.

I think it's fair to require to ask user to upgrade to node 10.3. My only concern is with Travis and other CI, are they already up to date?

Travis is up to date!

@gvas I find it simpler to clone the repo, checkout the PR as a branch than use npm link to test ;)

Wow, awesome work. Going to merge this and publish a new version. Will add your handy graph to the readme.

@LaurentGoderre: that's a better method, indeed. I didn't know npm link. What were the results of your tests?
@Jonahss: Thanks. Are you sure about inserting the table to the readme? The request column will be irrelevant to the users, because the library will use requestretry after merging this PR. Maybe a warning that node  0.10.0, 0.10.1, 0.10.2 may cause problems would be sufficient.

The error was intermittent so I ran 20 or so tests and it didn't fail with this specific error. It looks good to me

@gvas Are you able to run the tests? I'm having trouble with running "sauce_tunnel:server"

These fixes were able to give WinJS its first clean test run since December 16.

@Jonahss : My apologies, I have missed the email for your comment. I fixed the tests, could you give them another try?

Awesome.

Thank you from noflo/noflo-ui!!! :bow:
Before this, I'd resigned to only running Sauce on Saturdays :grimacing:

authors: Jonahss LaurentGoderre forresto gvas xirzec

repo: axemclion/grunt-saucelabs


text:
Fix CI Action

https://axds.atlassian.net/browse/ATN-717

fix flake8 dependency issue by updating it's version in pre-commit config
fix tests failing because of attempted int(math.inf) in bufr.py

not sure if this is actually a good behavior: I cap the number to sys.maxsize before conversion to int
inf is introduced when calculating drift and distance > 0 and time delta = 0 (on encoding/wildlife_computers.py:76)

maybe replacing inf with nan right at the source would be better? I don't know the usage context so I'm not sure.

Turns out there is a deeper cause to the inf problem.
At some point, numpy (or pandas?) changed how datetime64 is parsed/stored.
The time column in the examples/profile.parquet used to load as datetime64[ns] so for conversion to seconds it was divided (wholly //) by 1e9.
In new versions of these packages it gets loaded as datetime64[ms] so after dividing it by extra 1e6 the resulting numbers were often 0. That was the source of divide by zero which produced inf.
New approach to conversion that I just force-pushed uses the library API so it should be slightly more bulletproof. (.astype('timedelta64[s]') should convert the units for us regardless of what they were originally)

repo: axiom-data-science/bufrtools


text:
🛑 Axiom of choice web page is down

In f9dd4b6, Axiom of choice web page (https://axiom-of-choice.herokuapp.com/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Axiom of choice web page is back up in 8385e85.

Bug?

repo: axiom-of-choice/uptime


text:
[1.6.6] [db] SQL logic error: no such column: Sort (1)

Just noticed this when running the docker container today
mysites-mailhog       | ERRO[2023/05/04 11:25:05] [db] SQL logic error: no such column: Sort (1)
mysites-mailhog       | ERRO[2023/05/04 11:26:05] [db] SQL logic error: no such column: Sort (1)
mysites-mailhog       | ERRO[2023/05/04 11:27:05] [db] SQL logic error: no such column: Sort (1)
mysites-mailhog       | ERRO[2023/05/04 11:28:05] [db] SQL logic error: no such column: Sort (1)

Hmmm, is it running with an existing database, or a temporary one? If existing, is it possible that more than one Mailpit instance is running using the same database, and that this error message is coming from an older version of Mailpit? The Sort column had been replaced and I'm fairly sure there should be no references to Sort in the code.

sorry should have said this is running as a docker container - so totally new 1.6.6 container instantiated by docker-compose
  mailhog.mysites.guru:
    container_name: 'mysites-mailhog'
    image: axllent/mailpit
    ports:
      - "25:1025"
      - "1080:8025"

Same happening to me using an existing database running locally (no docker).
I'm sure that I'm running one instance.

I have completely removed the docker container, and started again from scratch using the documented standard way of running the container
docker run  -p 8025:8025 -p 1025:1025 axllent/mailpit

and I get output
INFO[2023/05/04 16:30:39] [smtpd] starting on 0.0.0.0:1025
INFO[2023/05/04 16:30:39] [http] starting server on http://0.0.0.0:8025/

and then after a short while I get the error
ERRO[2023/05/04 16:31:39] [db] SQL logic error: no such column: Sort (1)

I know nothing about the go programming language, but a quick search of your repo has a suspect Sort here:
https://github.com/axllent/mailpit/blob/f2bce03e9e57a32b7aa83c815e540fe2d112075a/storage/utils.go#L95

You're absolutely correct. Fortunately this didn't break anything, it just did not auto-delete any messages  (if greater than the current maximum setting) due to the error. I have fixed this now in v1.6.7. Thanks for the report!

authors: PhilETaylor amaurycid axllent

repo: axllent/mailpit


text:
Missing extension options in CMakeOptions.md

The following extension options are missing in CMakeOptions.md

AX_ENABLE_EXT_SDFGEN
AX_ENABLE_EXT_JSONDEFAULT
AX_ENABLE_EXT_LUA

Unrelated discussion: I think that all extesnsions should be turned off by default, besides

Imgui
Inspector
GUI

At the moment almost all extensions are turned on by default, which is bad, it forces everyone that doesn't need them to turn all of them off manually which is annoying. It is preferable that a user turns extensions on when they need them. Feedback?

Maybe differnt buikd flags be better?

Default
All
None

repo: axmolengine/axmol


text:
'AdamW' object has no attribute 'optim_bits'

Please check that this issue hasn't been reported before.

[X] I searched previous Bug Reports didn't find any similar reports.

Expected Behavior
Full-parameter chatml finetuning of Llama 3.1 should work on the main:latest docker image on runpod on 6x A40s with deepspeed
Current behaviour
Training never gets a chance to start:
Stacktrace:
[2024-12-08 22:54:15,191] [INFO] [axolotl.load_model:1115] [PID:13086] [RANK:2] Converting modules to torch.bfloat16
[2024-12-08 22:54:15,296] [INFO] [axolotl.load_model:1082] [PID:13084] [RANK:0] cuda memory usage after model load: 14.958GB (+0.126GB cache, +1.099GB misc)
[2024-12-08 22:54:15,306] [INFO] [axolotl.load_model:1115] [PID:13084] [RANK:0] Converting modules to torch.bfloat16
[rank3]: Traceback (most recent call last):
[rank3]:   File "<frozen runpy>", line 198, in _run_module_as_main
[rank3]:   File "<frozen runpy>", line 88, in _run_code
[rank3]:   File "/workspace/axolotl/src/axolotl/cli/train.py", line 58, in <module>
[rank3]:     fire.Fire(do_cli)
[rank3]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
[rank3]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank3]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
[rank3]:     component, remaining_args = _CallAndUpdateTrace(
[rank3]:                                 ^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
[rank3]:     component = fn(*varargs, **kwargs)
[rank3]:                 ^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/workspace/axolotl/src/axolotl/cli/train.py", line 34, in do_cli
[rank3]:     return do_train(parsed_cfg, parsed_cli_args)
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/workspace/axolotl/src/axolotl/cli/train.py", line 47, in do_train
[rank3]:     model, tokenizer = train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
[rank3]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/workspace/axolotl/src/axolotl/train.py", line 192, in train
[rank3]:     trainer.train(resume_from_checkpoint=resume_from_checkpoint)
[rank3]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/trainer.py", line 2123, in train
[rank3]:     return inner_training_loop(
[rank3]:            ^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/trainer.py", line 2275, in _inner_training_loop
[rank3]:     model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
[rank3]:                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/accelerator.py", line 1333, in prepare
[rank3]:     result = self._prepare_deepspeed(*args)
[rank3]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/accelerator.py", line 1843, in _prepare_deepspeed
[rank3]:     optimizer = map_pytorch_optim_to_deepspeed(optimizer)
[rank3]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/utils/deepspeed.py", line 53, in map_pytorch_optim_to_deepspeed
[rank3]:     is_adaw = isinstance(optimizer, (bnb_opt.AdamW, bnb_opt.AdamW32bit)) and optimizer.optim_bits == 32
[rank3]:                                                                              ^^^^^^^^^^^^^^^^^^^^
[rank3]: AttributeError: 'AdamW' object has no attribute 'optim_bits'

This issue has been around for about a week now? I first reported it on the Discord.
Steps to reproduce
Attempt to full finetune llama 3 using the settings provided (need to add some generic chatml dataset as I had to redact my data files)
Config yaml
base_model: Heralax/private-llama3.1-model-whose-name-is-censored
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: false
strict: false

datasets: # data files have to be redacted sorry

dataset_prepared_path: last_run_prepared-ft-lowerbatchsize
output_dir: ./out

sequence_len: 4096
sample_packing: false
pad_to_sequence_len: true
shuffle_merged_datasets: true

wandb_project: llama_3.1_8b
wandb_entity:
wandb_watch:
wandb_run_id:
wandb_log_model:

gradient_accumulation_steps: 7 # meant for use on 6 GPUs to achieve same effective batch size as earlier. Swapped # GPUs and Grad accumulation steps.
micro_batch_size: 2
eval_batch_size: 1
num_epochs: 4
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 0.000012
weight_decay: 0
# Gradient clipping max norm``
max_grad_norm: 1.0
noisy_embedding_alpha: 5
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false

gradient_checkpointing: unsloth
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true

chat_template: chatml

warmup_ratio: 0.5
auto_resume_from_checkpoints: false
#warmup_ratio: 0.5
eval_steps: 10
saves_per_epoch: 1
eval_sample_packing: false
save_total_limit: 2
debug:
deepspeed: deepspeed_configs/zero2.json
special_tokens:
  pad_token: "<|end_of_text|>"

### Possible solution

Rolling back to axolotlai/axolotl-cloud:main-20241129-py3.11-cu124-2.4.1 lets me train again. Unfortunately, the pinned nightly version I was relying on (winglian/axolotl-cloud:main-20241124) no longer lets me connect. by that I mean, direct SSH connection does not appear as an option and when I try to go through the proxy it hangs and then tells me the container is not running. This has happened for all winglian/axolotl-cloud images I have tried to run sometime after the date 12/8/24, but that is a separate issue.

### Which Operating Systems are you using?

- [X] Linux
- [ ] macOS
- [ ] Windows

### Python Version

whatever version the main latest comes with.

### axolotl branch-commit

main/whatever the most recent docker image update comes with

### Acknowledgements

- [X] My issue title is concise, descriptive, and in title casing.
- [X] I have searched the existing issues to make sure this bug has not been reported yet.
- [X] I am using the latest version of axolotl.
- [X] I have provided enough information for the maintainers to reproduce and diagnose the issue.

Could you try with the regular adamw_8bit optimizer please?

OK I will try with that and get back to you

@e-p-armstrong @winglian Looks like the issue is with accelerate. I find that downgrading accelerate to version 1.0.1 bypass this error for now. Will follow up on accelerate upstream

This issue seems to only affect zero2. Zero3 works fine.

@winglian Reproduced with a different optimizer and it happened even with DPO tuning.
pytorch_model.bin.index.json: 100%|_______________________________________________| 23.9k/23.9k [00:00<00:00, 80.9MB/s]
pytorch_model-00001-of-00002.bin: 100%|____________________________________________| 16.1G/16.1G [00:34<00:00, 465MB/s]
pytorch_model-00002-of-00002.bin: 100%|_____________________________________________| 542k/542k [00:00<00:00, 77.7MB/s]
Downloading shards: 100%|________________________________________________________________| 2/2 [00:35<00:00, 17.53s/it]
Downloading shards: 100%|________________________________________________________________| 2/2 [00:34<00:00, 17.48s/it]
Downloading shards: 100%|________________________________________________________________| 2/2 [00:35<00:00, 17.54s/it]
Loading checkpoint shards: 100%|_________________________________________________________| 2/2 [00:04<00:00,  2.03s/it]
generation_config.json: 100%|_________________________________________________________| 180/180 [00:00<00:00, 1.10MB/s]
[2024-12-18 21:31:27,225] [INFO] [axolotl.load_model:1077] [PID:1521] [RANK:0] cuda memory usage after model load: 14.958GB (+0.126GB cache, +1.447GB misc)
[2024-12-18 21:31:27,229] [INFO] [axolotl.load_model:1110] [PID:1521] [RANK:0] Converting modules to torch.bfloat16
Loading checkpoint shards: 100%|_________________________________________________________| 2/2 [00:05<00:00,  2.56s/it]
Loading checkpoint shards: 100%|_________________________________________________________| 2/2 [00:05<00:00,  2.56s/it]
[2024-12-18 21:31:28,223] [INFO] [axolotl.load_model:1077] [PID:1523] [RANK:2] cuda memory usage after model load: 14.958GB (+0.126GB cache, +1.447GB misc)
[2024-12-18 21:31:28,227] [INFO] [axolotl.load_model:1110] [PID:1523] [RANK:2] Converting modules to torch.bfloat16
/workspace/axolotl/src/axolotl/core/trainer_builder.py:446: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `AxolotlTrainer.__init__`. Use `processing_class` instead.
  super().__init__(*_args, **kwargs)
[2024-12-18 21:31:28,828] [INFO] [axolotl.train.train:174] [PID:1521] [RANK:0] Starting trainer...
[2024-12-18 21:31:28,892] [INFO] [axolotl.load_model:1077] [PID:1522] [RANK:1] cuda memory usage after model load: 14.958GB (+0.126GB cache, +1.447GB misc)
[2024-12-18 21:31:28,896] [INFO] [axolotl.load_model:1110] [PID:1522] [RANK:1] Converting modules to torch.bfloat16
/workspace/axolotl/src/axolotl/core/trainer_builder.py:446: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `AxolotlTrainer.__init__`. Use `processing_class` instead.
  super().__init__(*_args, **kwargs)
/workspace/axolotl/src/axolotl/core/trainer_builder.py:446: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `AxolotlTrainer.__init__`. Use `processing_class` instead.
  super().__init__(*_args, **kwargs)
[2024-12-18 21:31:30,383] [INFO] [axolotl.utils.samplers.multipack.calc_min_len:197] [PID:1521] [RANK:0] gather_len_batches: [580, 580, 580]
[rank1]: Traceback (most recent call last):
[rank1]:   File "<frozen runpy>", line 198, in _run_module_as_main
[rank1]:   File "<frozen runpy>", line 88, in _run_code
[rank1]:   File "/workspace/axolotl/src/axolotl/cli/train.py", line 58, in <module>
[rank1]:     fire.Fire(do_cli)
[rank1]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
[rank1]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank1]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
[rank1]:     component, remaining_args = _CallAndUpdateTrace(
[rank1]:                                 ^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
[rank1]:     component = fn(*varargs, **kwargs)
[rank1]:                 ^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/workspace/axolotl/src/axolotl/cli/train.py", line 34, in do_cli
[rank1]:     return do_train(parsed_cfg, parsed_cli_args)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/workspace/axolotl/src/axolotl/cli/train.py", line 47, in do_train
[rank1]:     model, tokenizer = train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
[rank1]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/workspace/axolotl/src/axolotl/train.py", line 188, in train
[rank1]:     trainer.train(resume_from_checkpoint=resume_from_checkpoint)
[rank1]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/trainer.py", line 2164, in train
[rank1]:     return inner_training_loop(
[rank1]:            ^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/trainer.py", line 2323, in _inner_training_loop
[rank1]:     model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
[rank1]:                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/accelerator.py", line 1333, in prepare
[rank1]:     result = self._prepare_deepspeed(*args)
[rank1]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/accelerator.py", line 1843, in _prepare_deepspeed
[rank1]:     optimizer = map_pytorch_optim_to_deepspeed(optimizer)
[rank1]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/utils/deepspeed.py", line 53, in map_pytorch_optim_to_deepspeed
[rank1]:     is_adaw = isinstance(optimizer, (bnb_opt.AdamW, bnb_opt.AdamW32bit)) and optimizer.optim_bits == 32
[rank1]:                                                                              ^^^^^^^^^^^^^^^^^^^^
[rank1]: AttributeError: 'AdamW' object has no attribute 'optim_bits'
[rank2]: Traceback (most recent call last):
[rank2]:   File "<frozen runpy>", line 198, in _run_module_as_main
[rank2]:   File "<frozen runpy>", line 88, in _run_code
[rank2]:   File "/workspace/axolotl/src/axolotl/cli/train.py", line 58, in <module>
[rank2]:     fire.Fire(do_cli)
[rank2]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
[rank2]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank2]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
[rank2]:     component, remaining_args = _CallAndUpdateTrace(
[rank2]:                                 ^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
[rank2]:     component = fn(*varargs, **kwargs)
[rank2]:                 ^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/workspace/axolotl/src/axolotl/cli/train.py", line 34, in do_cli
[rank2]:     return do_train(parsed_cfg, parsed_cli_args)
[rank2]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/workspace/axolotl/src/axolotl/cli/train.py", line 47, in do_train
[rank2]:     model, tokenizer = train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
[rank2]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/workspace/axolotl/src/axolotl/train.py", line 188, in train
[rank2]:     trainer.train(resume_from_checkpoint=resume_from_checkpoint)
[rank2]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/trainer.py", line 2164, in train
[rank2]:     return inner_training_loop(
[rank2]:            ^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/trainer.py", line 2323, in _inner_training_loop
[rank2]:     model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
[rank2]:                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/accelerator.py", line 1333, in prepare
[rank2]:     result = self._prepare_deepspeed(*args)
[rank2]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/accelerator.py", line 1843, in _prepare_deepspeed
[rank2]:     optimizer = map_pytorch_optim_to_deepspeed(optimizer)
[rank2]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/utils/deepspeed.py", line 53, in map_pytorch_optim_to_deepspeed
[rank2]:     is_adaw = isinstance(optimizer, (bnb_opt.AdamW, bnb_opt.AdamW32bit)) and optimizer.optim_bits == 32
[rank2]:                                                                              ^^^^^^^^^^^^^^^^^^^^
[rank2]: AttributeError: 'AdamW' object has no attribute 'optim_bits'
[rank0]: Traceback (most recent call last):
[rank0]:   File "<frozen runpy>", line 198, in _run_module_as_main
[rank0]:   File "<frozen runpy>", line 88, in _run_code
[rank0]:   File "/workspace/axolotl/src/axolotl/cli/train.py", line 58, in <module>
[rank0]:     fire.Fire(do_cli)
[rank0]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
[rank0]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank0]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
[rank0]:     component, remaining_args = _CallAndUpdateTrace(
[rank0]:                                 ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
[rank0]:     component = fn(*varargs, **kwargs)
[rank0]:                 ^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/workspace/axolotl/src/axolotl/cli/train.py", line 34, in do_cli
[rank0]:     return do_train(parsed_cfg, parsed_cli_args)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/workspace/axolotl/src/axolotl/cli/train.py", line 47, in do_train
[rank0]:     model, tokenizer = train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
[rank0]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/workspace/axolotl/src/axolotl/train.py", line 188, in train
[rank0]:     trainer.train(resume_from_checkpoint=resume_from_checkpoint)
[rank0]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/trainer.py", line 2164, in train
[rank0]:     return inner_training_loop(
[rank0]:            ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/transformers/trainer.py", line 2323, in _inner_training_loop
[rank0]:     model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
[rank0]:                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/accelerator.py", line 1333, in prepare
[rank0]:     result = self._prepare_deepspeed(*args)
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/accelerator.py", line 1843, in _prepare_deepspeed
[rank0]:     optimizer = map_pytorch_optim_to_deepspeed(optimizer)
[rank0]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/utils/deepspeed.py", line 53, in map_pytorch_optim_to_deepspeed
[rank0]:     is_adaw = isinstance(optimizer, (bnb_opt.AdamW, bnb_opt.AdamW32bit)) and optimizer.optim_bits == 32
[rank0]:                                                                              ^^^^^^^^^^^^^^^^^^^^
[rank0]: AttributeError: 'AdamW' object has no attribute 'optim_bits'
W1218 21:31:33.244000 140323527751488 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1521 closing signal SIGTERM
W1218 21:31:33.245000 140323527751488 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1523 closing signal SIGTERM
E1218 21:31:33.390000 140323527751488 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 1 (pid: 1522) of binary: /root/miniconda3/envs/py3.11/bin/python3
Traceback (most recent call last):
  File "/root/miniconda3/envs/py3.11/bin/accelerate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1153, in launch_command
    deepspeed_launcher(args)
  File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/accelerate/commands/launch.py", line 846, in deepspeed_launcher
    distrib_run.run(args)
  File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
axolotl.cli.train FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-12-18_21:31:33
  host      : b6131755c915
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 1522)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

@bursteratom Thanks for searching to find the problem!

@e-p-armstrong In the meantime I recommend using zero3 instead of zero2

@e-p-armstrong @winglian started an upstream PR on accelerate to fix this: https://github.com/huggingface/accelerate/pull/3305

https://github.com/huggingface/accelerate/pull/3311

authors: bursteratom e-p-armstrong winglian

repo: axolotl-ai-cloud/axolotl


text:
为什么去除了 lspsaga 插件？

请问下为什么要去除 lspsaga 插件？
虽然最近 nvim0.6 有些不兼容，但还是可以使用 0.5的分支，还是说有什么其他原因吗？
感谢分享的这份配置，非常好用 ：）

主要是因为lspsaga已经好久没有更新了

我切了  tami5/lspsaga.nvim 的分支现在还在维护，最近没发现什么问题，考虑试下吗？vim.lsp.buf.hover 的框实在看着有些怪和背景色区分度太低了。

使用的是低于 0.6 版本的 neovim 可能需要以下的配置：
completion['tami5/lspsaga.nvim'] = {
    opt = true,
    branch = 'nvim51',
    after = 'nvim-lspconfig',
}

lspsaga:
NEOVIM NIGHTLY (+v0.6.0-dev+1865-g3beea1fe1) or use nvim51 branch use { 'tami5/lspsaga.nvim', branch = 'nvim51' }

Readme中已经要求使用neovim-git了

恩  我只是在这个 issue 里提一下，服务器里一般不会很新。赞！

authors: HATTER-LONG ayamir

repo: ayamir/nvimdots


text:
🛑 Bahçıvan Peynir is down

In e7303a0, Bahçıvan Peynir (https://www.bahcivanpeynir.com/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Bahçıvan Peynir is back up in bf8c9af after 20 minutes.

repo: aydgn/upptime