text:
Executing commands on bind mounts doesn't always work

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

Description
When using podman to launch a container, with a bind mount, attachment to the container doesn't always succeed.
Steps to reproduce the issue:
1.podman pull registry.access.redhat.com/rhel7
2.podman run -v /usr/sbin:/usr/sbin --rm -it rhel7 /usr/sbin/ip a

or -

podman run -v /usr/bin:/usr/bin --rm -it rhel7 /usr/bin/file /etc/hosts

Describe the results you received:
When it doesn't work, you get:
error attaching to container cda641fe922585a52ef37de83533910b684c27a3bbd75cd988a0f7708199ac05: failed to
 connect to container's attach socket: %!v(MISSING): dial unixpacket /var/run/libpod/socket/cda

Describe the results you expected:
When it works, you get what you expect:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: eth0@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether fa:78:6c:23:90:dd brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.88.0.11/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f878:6cff:fe23:90dd/64 scope link tentative
       valid_lft forever preferred_lft forever

or-

/etc/hosts: ASCII text

Additional information you deem important (e.g. issue happens only occasionally):
The issue seems to happen about 50% of the time.
Output of podman version:
# podman version
Version:       0.4.1
Go Version:    go1.9.2
OS/Arch:       linux/amd64

Output of podman info:
# podman info
host:
  MemFree: 1547030528
  MemTotal: 1926369280
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 1
  hostname: ip-10-0-2-203.ec2.internal
  kernel: 3.10.0-862.3.2.el7.x86_64
  os: linux
  uptime: 6m 31.74s
insecure registries:
  registries: []
registries:
  registries:
  - registry.access.redhat.com
store:
  ContainerStore:
    number: 14
  GraphDriverName: overlay
  GraphOptions:
  - overlay.override_kernel_check=true
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
  ImageStore:
    number: 1
  RunRoot: /var/run/containers/storage

Additional environment details (AWS, VirtualBox, physical, etc.):
The environment is running on RHEL 7.5, on AWS EC2.

I wonder if we're not seeing a race between conmon and the attach part of podman run here. Setting up the attach socket in conmon happens fairly late in the process of creating the container, after we inform libpod that it's safe to proceed, but conmon is generally so lightweight that the timing condition is never hit.
We could potentially set up the attach socket in conmon before we fork off runc - it'll potentially leave a socket and a symlink lying around in cases where runc fails, though.

Brent can you look into this?

@rhatdan I think this has to be a fix in conmon

@giuseppe Mind taking a look at this one? We'd like conmon to make the attach socket earlier in its lifetime, before it starts runc, to try and avoid a timing issue.

hm.. I see there is no race in conmon when the attach socket is created, as podman blocks on receiving the container PID which is sent by conmon after the attach socket is created.  I was not able to reproduce this issue.  I wonder if it depends on a change we did in conmon that correctly reports the exit status of a container now (before it was reporting 0 when the container was signaled) and podman still trying to attach to the container.
@ajacocks is it still reproducible for you?  If yes, could you attach the output of podman --log-level=debug ... when it fails?

@giuseppe Yes, indeed, I can still reproduce it on RHEL7.5-current:
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)
$  uname -r
3.10.0-862.3.3.el7.x86_64

And here are the logs, as requested:
# podman --log-level=debug run -v /usr/sbin:/usr/sbin --rm -it rhel7 /usr/sbin/ip a
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: override_kernelcheck=true
DEBU[0000] overlay test mount with multiple lowers succeeded
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true
INFO[0000] CNI network podman (type=bridge) is used from /etc/cni/net.d/87-podman-bridge.conflist
INFO[0000] Initial CNI setting succeeded
DEBU[0000] parsed reference to refname into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]docker.io/library/rhel7:latest"
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]docker.io/library/rhel7:latest" does not resolve to an image ID
DEBU[0000] parsed reference to refname into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]docker.io/library/rhel7:latest"
DEBU[0000] reference "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]docker.io/library/rhel7:latest" does not resolve to an image ID
DEBU[0000] parsed reference to id into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]@7a840db7f020be49bb60fb1cc4f1669e83221d61c1af23ff2cac2f870f9deee8"
DEBU[0000] exporting opaque data as blob "sha256:7a840db7f020be49bb60fb1cc4f1669e83221d61c1af23ff2cac2f870f9deee8"
DEBU[0000] parsed reference to id into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]@7a840db7f020be49bb60fb1cc4f1669e83221d61c1af23ff2cac2f870f9deee8"
DEBU[0000] exporting opaque data as blob "sha256:7a840db7f020be49bb60fb1cc4f1669e83221d61c1af23ff2cac2f870f9deee8"
DEBU[0000] parsed reference to id into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]@7a840db7f020be49bb60fb1cc4f1669e83221d61c1af23ff2cac2f870f9deee8"
DEBU[0000] Using bridge netmode
DEBU[0000] parsed reference to id into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]@7a840db7f020be49bb60fb1cc4f1669e83221d61c1af23ff2cac2f870f9deee8"
DEBU[0000] created container "61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b"
DEBU[0000] container "61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b" has work directory "/var/lib/containers/storage/overlay-containers/61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b/userdata"
DEBU[0000] container "61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b" has run directory "/var/run/containers/storage/overlay-containers/61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b/userdata"
DEBU[0000] New container created "61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b"
DEBU[0000] container "61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b" has CgroupParent "/libpod_parent/libpod-conmon-61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b/61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b"
DEBU[0000] mounted container "61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b" at "/var/lib/containers/storage/overlay/3b19d27499861f080befd6dac91aeabbb857095bc0d8d43083cb0c85d9e4360b/merged"
DEBU[0000] Created root filesystem for container 61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b at /var/lib/containers/storage/overlay/3b19d27499861f080befd6dac91aeabbb857095bc0d8d43083cb0c85d9e4360b/merged
DEBU[0000] Made network namespace at /var/run/netns/cni-d65d8258-7130-ea13-6aee-3becd5e8bc1e for container 61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b
INFO[0000] Got pod network {Name:nervous_mayer Namespace:nervous_mayer ID:61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b NetNS:/var/run/netns/cni-d65d8258-7130-ea13-6aee-3becd5e8bc1e PortMappings:[]}
INFO[0000] About to add CNI network cni-loopback (type=loopback)
INFO[0000] Got pod network {Name:nervous_mayer Namespace:nervous_mayer ID:61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b NetNS:/var/run/netns/cni-d65d8258-7130-ea13-6aee-3becd5e8bc1e PortMappings:[]}
INFO[0000] About to add CNI network podman (type=bridge)
DEBU[0000] Response from CNI plugins: Interfaces:[{Name:cni0 Mac:1a:7e:d7:a1:bf:7e Sandbox:} {Name:veth3e515f33 Mac:32:c4:1a:96:32:f7 Sandbox:} {Name:eth0 Mac: Sandbox:/var/run/netns/cni-d65d8258-7130-ea13-6aee-3becd5e8bc1e}], IP:[{Version:4 Interface:0xc420338660 Address:{IP:10.88.0.28 Mask:ffff0000} Gateway:10.88.0.1}], Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} GW:<nil>}], DNS:{Nameservers:[] Domain: Search:[] Options:[]}
DEBU[0000] Running iptables command: -t filter -I FORWARD -s 10.88.0.28 ! -o 10.88.0.28 -j ACCEPT
WARN[0000] file "/etc/containers/mounts.conf" not found, skipping...
DEBU[0000] Creating dest directory: /var/run/containers/storage/overlay-containers/61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b/userdata/run/secrets
DEBU[0000] Calling TarUntar(/usr/share/rhel/secrets, /var/run/containers/storage/overlay-containers/61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b/userdata/run/secrets)
DEBU[0000] TarUntar(/usr/share/rhel/secrets /var/run/containers/storage/overlay-containers/61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b/userdata/run/secrets)
WARN[0000] hooks path: "/usr/share/containers/oci/hooks.d" does not exist
WARN[0000] hooks path: "/etc/containers/oci/hooks.d" does not exist
DEBU[0000] parsed reference to id into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]@7a840db7f020be49bb60fb1cc4f1669e83221d61c1af23ff2cac2f870f9deee8"
DEBU[0000] parsed reference to id into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]@7a840db7f020be49bb60fb1cc4f1669e83221d61c1af23ff2cac2f870f9deee8"
DEBU[0000] exporting opaque data as blob "sha256:7a840db7f020be49bb60fb1cc4f1669e83221d61c1af23ff2cac2f870f9deee8"
DEBU[0000] parsed reference to id into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]@7a840db7f020be49bb60fb1cc4f1669e83221d61c1af23ff2cac2f870f9deee8"
DEBU[0000] exporting opaque data as blob "sha256:7a840db7f020be49bb60fb1cc4f1669e83221d61c1af23ff2cac2f870f9deee8"
DEBU[0000] parsed reference to id into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]@7a840db7f020be49bb60fb1cc4f1669e83221d61c1af23ff2cac2f870f9deee8"
DEBU[0000] Created OCI spec for container 61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b at /var/lib/containers/storage/overlay-containers/61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b/userdata/config.json
DEBU[0000] running conmon: /usr/libexec/podman/conmon    args=[-c 61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b -u 61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b/userdata -p /var/run/containers/storage/overlay-containers/61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b/userdata/pidfile -l /var/lib/containers/storage/overlay-containers/61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket -t]
DEBU[0000] Received container pid: 11642
DEBU[0000] Created container 61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b in OCI runtime
DEBU[0000] Attaching to container 61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b
DEBU[0000] connecting to socket /var/run/libpod/socket/61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b/attach
DEBU[0000] Started container 61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b
DEBU[0000] Enabling signal proxying
ERRO[0000] error attaching to container 61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b: failed to connect to container's attach socket: %!!(MISSING)v(MISSING): dial unixpacket /var/run/libpod/socket/61def006a50630502069ed457e43f75d6910b4496b8129198afe3c3cd451e79b/attach: connect: no such file or directory

Hopefully this helps.

I've tried on RHEL 7.5 with podman version 0.4.1 and I am still unable to reproduce this issue.
Could you please attach any output from conmon that you see in the journal?

@giuseppe Your wish is my command, sir!
Jun 18 21:33:01 ip-10-0-2-174.ec2.internal conmon[22450]: conmon <ninfo>: addr{sun_family=AF_UNIX, sun_path=/tmp/conmon-term.JM3RKZ}
Jun 18 21:33:01 ip-10-0-2-174.ec2.internal conmon[22450]: conmon <ninfo>: about to waitpid: 22451
Jun 18 21:33:01 ip-10-0-2-174.ec2.internal conmon[22450]: conmon <ninfo>: about to accept from console_socket_fd: 9
Jun 18 21:33:01 ip-10-0-2-174.ec2.internal conmon[22450]: conmon <ninfo>: about to recvfd from connfd: 15
Jun 18 21:33:02 ip-10-0-2-174.ec2.internal kernel: SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
Jun 18 21:33:02 ip-10-0-2-174.ec2.internal conmon[22450]: conmon <ninfo>: console = {.name = '/dev/ptmx8 21:33:01 conmon: conmon <ninfo>: about to recvfd from connfd: 15
                                                          '; .fd = 9}
Jun 18 21:33:02 ip-10-0-2-174.ec2.internal conmon[22450]: conmon <ninfo>: container PID: 22458
Jun 18 21:33:02 ip-10-0-2-174.ec2.internal conmon[22450]: conmon <ninfo>: attach sock path: /var/run/libpod/socket/4e30365196879c256cc6f0bf5bcea63e39acb8929374ae6508ef74f7f4293305/attach
Jun 18 21:33:02 ip-10-0-2-174.ec2.internal conmon[22450]: conmon <ninfo>: addr{sun_family=AF_UNIX, sun_path=/var/run/libpod/socket/4e30365196879c256cc6f0bf5bcea63e39acb8929374ae6508ef74f7f4293305/attach}
Jun 18 21:33:02 ip-10-0-2-174.ec2.internal conmon[22450]: conmon <ninfo>: ctl fifo path: /var/lib/containers/storage/overlay-containers/4e30365196879c256cc6f0bf5bcea63e39acb8929374ae6508ef74f7f4293305/userdata/ctl
Jun 18 21:33:02 ip-10-0-2-174.ec2.internal conmon[22450]: conmon <ninfo>: terminal_ctrl_fd: 16
Jun 18 21:33:02 ip-10-0-2-174.ec2.internal conmon[22450]: conmon <ninfo>: container 22458 exited with status 0
Jun 18 21:33:02 ip-10-0-2-174.ec2.internal conmon[22450]: conmon <nwarn>: stdio_input read failed Input/output error

@ajacocks thanks a lot!  That helped to understand.  It seems like the container terminates so quickly that conmon deletes the attach socket during the cleanup and podman doesn't find it.  We should handle the ENOENT differently from podman as it means the container was already terminated

hm.. that should not happen though, as the container is started after the console was attached.  I need to investigate it further

@rhatdan, @mheon I think the issue is here:
https://github.com/projectatomic/libpod/blob/master/libpod/container_api.go#L178-L192
We do the attach from a goroutine, but the c.start() might happen before the attach is done.  We'll need to change how  c.attach() works and be able to plug the start immediately after the net.DialUnix is done.

Please create a patch.

And we'd just removed the syncronization calls from attach()... Damn it.

@mheon is it something we can easily revert?

@giuseppe Don't think so, I did a large refactor to implement --attach and the sync primitives were one of the casualties. Thought they were unnecessary if we just ordered the calls in StartWithAttach()

Ok thanks.  I will work on a patch as tomorrow I am back from PTO

PR here: https://github.com/projectatomic/libpod/pull/970

Thanks so much, everyone!

authors: ajacocks giuseppe mheon rhatdan

repo: projectatomic/libpod


text:
Calico with libnetwork not support Profile rules?

I cannot find any guide to use profile rules with libnetwork

And in this doc, https://github.com/projectcalico/calico-docker/blob/master/docs/calicoctl/profile.md , it says:

NOTE: The calicoctl profile commands should NOT be used when running Calico with the Docker libnetwork driver. The libnetwork driver manages the security policy for containers.

Hi @frostynova - this one is on our radar, but still needs a little thought about how to handle.  It may simply end up being a testing/documentation thing - but we'll keep you posted.

@frostynova
We've just release v0.12.0 which adds Calico IPAM support.  One upshot of this is that provided you are using Calico IPAM then it should be possible to edit the profile rules rather than using the default behavior.
We do still need to do a better job of documenting this - or possibly thinking about new commands, or new docker network options to improve this.  I'll close this issue and raise a new one for that side of things.
In the meantime, we do now touch on the manipulation of profiles and endpoints in the libnetwork demonstration.

source: gharchive/issue

authors: frostynova robbrockbank

repo: projectcalico/calico-docker

license_type: permissive

license_source: bigquery


text:
Still seeing "error adding host side routes for interface: xxx, error: failed to add route file exists" in Calico 2.6.1

Still seeing error adding host side routes for interface: xxx, error: failed to add route file exists which is keeping containers from starting (stuck in "ContainerCreating").  This is detailed in https://github.com/projectcalico/cni-plugin/issues/352 (and also more detail about orphan WEPs which are a part of this problem can be found here https://github.com/projectcalico/calico/issues/1115).  It was thought that fix(es) in Calico 2.6.1 would solve this problem, but we are still seeing it.
Your Environment

Calico version: 2.6.1
Orchestrator version: Kubernetes 1.8.1
Operating System and version: Ubuntu 16.04.3

I will attach the logs, there are quite a few containers stuck in this ContainerCreating state, one is master-b4fe00b59ff948088731b4985367b705-6b987df84d-bz9zv in the kubx-masters namespace

calico-node.log
kubelet.zip
calico-node.log

Above are the logs I could attach.  I also have syslog, but it is too large to attach, let me know if you need it and I'll get it to you somehow.

@bradbehle is this on a fresh cluster, or an upgraded cluster from a previous version? Could you do a quick double-check of the version of the CNI plugin in use?
This should do it:
/opt/cni/bin/calico -v

@caseydavenport
CNI plugin is at 1.11.0:
behle@dev-mex01-carrier80-worker-01:~$ /opt/cni/bin/calico -v
v1.11.0
behle@dev-mex01-carrier80-worker-01:~$

And it was an upgraded cluster that was originally at kube 1.5 and Calico 2.1.5

I've seen similar behaviour:
# /opt/cni/bin/calico -v
v1.11.0

k8s: 1.8.1
calico: 2.6.1
OS: ubuntu 14.04
kubelet logs:
https://gist.github.com/r0bj/a656c75a7b08e1b79be4671206b70779
calico-node logs:
https://gist.github.com/r0bj/1279b3fa321ea5d47d66f5e4c945eadb

@bradbehle @r0bj I have 2 PRs up for the fix, one for CNI v2.0 and one backported to CNI v1.11.x, I've made a CNI image with the fix backported to CNI v1.11.0, so you can try it out, I haven't been able to reproduce it but I've added a test to replicate the issue as best as I can. You can try the debug image with the fix at gunjan5/cni:route2 it should also have the CNI binary

@bradbehle @r0bj just checking to see if you've had a chance to try out the debug image, if it works then we can get the PRs merged and get the fix included in the next patch releases

@gunjan5 for me it's just difficult to reproduce it. I have encountered it twice on production cluster and after node restart everything worked as expected again. If I have better way of reproducing I'll test your debug image for sure.

HI @gunjan5
I have exact same problem as @r0bj
I took your patch and applied in my environment.
And now i see below error :
Nov  8 00:53:33 node1 kubelet: E1108 00:53:33.577354   15218 pod_workers.go:182] Error syncing pod fb3638af-c443-11e7-9f0f-0894ef42f61e ("test1-test1-1193689166-zqh8b_namespace1(fb3638af-c443-11e7-9f0f-0894ef42f61e)"), skipping: failed to "SetupNetwork" for "aio-stage3-serviceavailability-mobilezip-1193689166-zqh8b_staging3" with SetupNetworkError: "NetworkPlugin cni failed to set up pod "aio-stage3-serviceavailability-mobilezip-1193689166-zqh8b_staging3" network: error adding host side routes for interface: cali38f5e43d7eb, error: route (Ifindex: 8374, Dst: 172.40.107.158/32, Scope: %!!(MISSING)s(netlink.Scope=253)) already exists for an interface other than 'cali38f5e43d7eb'"
Error is little differnent this time.
It says route already exists for an interface other than 'calixxxxx'
What needs to be done to resolve this ?
Any WA solution you can give for now ?

@msavlani can you post the CNI debug logs?

@gunjan5 I used your debug image and I was able to reproduce it.
kubelet error message:
2017-11-10 16:34:00.247 [WARNING][27292] calico-ipam.go 236: Asked to release address but it doesn't exist. Ignoring Workload="ops.nginx-stateful-test-0" workloadId="ops.nginx-stateful-test-0"
E1110 16:34:00.256432    1467 cni.go:301] Error adding network: error adding host side routes for interface: cali05ad595351c, error: route (Ifindex: 93, Dst: 10.203.131.144/32, Scope: %!!(MISSING)s(netlink.Scope=253)) already exists for an interface other than 'cali05ad595351c'
E1110 16:34:00.256462    1467 cni.go:250] Error while adding to cni network: error adding host side routes for interface: cali05ad595351c, error: route (Ifindex: 93, Dst: 10.203.131.144/32, Scope: %!!(MISSING)s(netlink.Scope=253)) already exists for an interface other than 'cali05ad595351c'
I1110 16:34:00.306370    1467 kubelet.go:1871] SyncLoop (PLEG): "nginx-stateful-test-0_ops(90302ede-c634-11e7-883a-5254009ef6db)", event: &pleg.PodLifecycleEvent{ID:"90302ede-c634-11e7-883a-5254009ef6db", Type:"ContainerStarted", Data:"85e04cf0fcb9830d0824805bb9ca4849b5c53ee93124332a746d22a8956de823"}
E1110 16:34:00.457030    1467 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "nginx-stateful-test-0_ops" network: error adding host side routes for interface: cali05ad595351c, error: route (Ifindex: 93, Dst: 10.203.131.144/32, Scope: %!!(MISSING)s(netlink.Scope=253)) already exists for an interface other than 'cali05ad595351c'

calico error:
2017-11-10 16:33:16.862 [WARNING][156] route_table.go 477: Failed to add route error=file exists ifaceName="cali05ad595351c" ipVersion=0x4 targetCIDR=10.203.131.144/32
2017-11-10 16:33:16.862 [INFO][156] route_table.go 200: Trying to connect to netlink
2017-11-10 16:33:16.863 [WARNING][156] route_table.go 577: Failed to access interface but it now appears to be up error=netlink update operation failed ifaceName="cali05ad595351c" ipVersion=0x4 link=&netlink.Veth{LinkAttrs:netlink.LinkAttrs{Index:78, MTU:1500, TxQLen:0, Name:"cali05ad595351c", HardwareAddr:net.HardwareAddr{0x6e, 0x40, 0x72, 0xce, 0xb0, 0xb3}, Flags:0x13, RawFlags:0x11043, ParentIndex:4, MasterIndex:0, Namespace:interface {}(nil), Alias:"", Statistics:(*netlink.LinkStatistics)(0xc420ad7140), Promisc:0, Xdp:(*netlink.LinkXdp)(nil), EncapType:"ether", Protinfo:(*netlink.Protinfo)(nil), OperState:0x6}, PeerName:""}

full kubelet logs:
https://gist.github.com/r0bj/9a96a0041733e9cb9a2e126d4ef224b2
full calico logs:
https://gist.github.com/r0bj/ab637b6ab3deeaa4a88646d5f22dad4d

Another example:
kubelet logs:
https://gist.github.com/r0bj/9ee5a7b1d15fde85390f5ceb99a76c15
calico logs:
https://gist.github.com/r0bj/a31672f4835ed4d12c41a788ede495ac

@r0bj @msavlani Thanks for providing the logs, both the kubelet and calico/node logs make it look like we've assigned the IP to one endpoint, then tried to assign it to another but it's not clear from the logs how that happened because the logs start around the time that we're trying to assign the IP to the second endpoint.
If you can get a node in the bad state again, it'd help to have:

any logs from before the problem, if you could look back to see if the IP of the problematic route occurs earlier in the log, that'd be good
the output of ip route on the affected node, that'll confirm that the bad route belongs to Calico and that it's still in-place
the output of calicoctl get workloadendpoint --node=<nodename> -o yaml for the affected node; that should dump all the workloads that Calico thinks it has on the node.

@gunjan5 From code reading, I spot one issue; after a failed networking attempt, we always clean up the IPAM allocation.  However, if this is an attempt to re-network a pod then we've already written the IP into the workload endpoint when we first networked the pod so, I think, we end up deleting the IPAM reservation without removing the IP from the workload endpoint.  Later, we'll then try to re-use that IP for another pod and hit this failure mode.
If we're in the existing endpoint case, I think we just need to leave the IPAM allocation as is and fail the CNI request so that it gets retried.  I guess we could delete the workload endpoint.

@fasaxc data you requested:

logs from the the time when node started:
kubelet logs:
https://gist.github.com/r0bj/f795bf18dec385ebd5e0035544649d17
calico-node logs:
https://gist.github.com/r0bj/02819862d4e30c19501b06ecd9dc67c3

ip route on the node:
https://gist.github.com/r0bj/58c931dec5e795d8fa068be21cf5c747

output of calicoctl get workloadendpoint --node=dev-k8s-worker-p4 -o yaml:
https://gist.github.com/r0bj/4eda1531de42bb3d18df439604dd4db6

Awesome, thanks @r0bj, it looks like you may have hit the case I described above.  You have two workload endpoints with the same IP assigned.  In addition, the IPAM error in the log indicates that the IPAM allocation was incorrectly lost or cleaned up.
Assuming this is the only instance of the problem on your node, a temporary workaround would be to delete and recreate these two pods: ops.nsqlookupd2-7574c7fbd5-rnntr and ops.nginx-stateful-test-0.
I found those by searching for the IP address of the failing route in the workload endpoint dump.

@fasaxc , i am encountering exact same problem.
Lately i am facing this with every pod running on that one node and had to manually delete pod with same endpoint (Going in cycles)
Can you provide your fix patch so that i can try in my environment to see if that resolves the problem ?

@fasaxc thanks for the fix, I upgraded to v1.x-series/v2.x-series docker tags for cni and node, it solved the issue for me.

@msavlani The fix is in This release of the CNI plugin.  We're about to release a Calico v2.6.3 patch release to include it. https://github.com/projectcalico/cni-plugin/releases/tag/v1.11.1
Note: after taking the fix, you'll need to remove and recreate all the pods on affected nodes.

Fixed by #425 #418 #408 #406, to be released in Calico v2.6.3 (CNI plugin v1.11.1) and and Calico v3.0.
Please open a new issue if you see this again after upgrading (the Calico release should be out in the next few days).  Note: as mentioned above, one issue was that, after a failure, we were cleaning up the IPAM entry int he datastore even though it was in use.  After that has occurred the datastore is inconsistent.  To resolve:

upgrade to release of Calico with the fix
list all the pods on nodes with the problem
delete/recreate any pods that share an IP address.

I am still facing this issue and raised another issue #1406 as per above suggestion.

authors: MikaelCluseau bradbehle caseydavenport fasaxc gunjan5 msavlani r0bj

repo: projectcalico/calico

license_source: github-api


text:
vxlan interface always down , cause calico-node failed to add route for pods on other node

calico version: 3.11.2
k8s version: 1.17
I have a k8s cluster with 3 nodes  , which uses calico CNI .
2 calico  ippool is set up , both of them use vxvlan mode
# calicoctl get ippool -o wide
NAME                      CIDR            NAT    IPIPMODE   VXLANMODE   DISABLED   SELECTOR
default-ipv4-ippool       172.29.0.0/16   true   Never      Always      false      all()
kube-system-ipv4-ippool   172.28.0.0/16   true   Never      Always      false      all()

the strange thing happened ,  the vxlan.calico interface on a node always stay down , and I tried to set it up , but after about 3seconds , this tunnel interface goes down again . That make calico-node failed to add route for other 2 node  .
After I delete the daemonset calico-node, I can set the vxlan.calico interface up  . However, I apply the daemonset calico-node  again , the vxlan.calico interface on the same node begin to stay down again , and I could not set it up agian . So I thinkg it is should that calico-node found something wrong and always make the  vxlan.calico interface down
However other 2 node2 work fine
BTW, my other Cluster with same calico configuration , has no this issue
Finally , I found a work-around way :  delete the tunnel interface , which trigger the calico-node recreate the tunnel interface,  then everything go to work fine
this cluster environment may scale up nodes or reinstall calico operation , Maybe the tunnel interface is created by previously with wrong VXLAN configuration for right now . So deleting it can trigger calico to regenerate right one
the calico-node log of the bad node is bellow
2020-02-21 13:51:39.739 [INFO][50] int_dataplane.go 849: Received interface update msg=&intdataplane.ifaceUpdate{Name:"vxlan.calico", State:"up"}
2020-02-21 13:51:39.740 [INFO][50] int_dataplane.go 962: Applying dataplane updates
2020-02-21 13:51:39.740 [INFO][50] int_dataplane.go 597: Linux interface state changed. ifaceName="vxlan.calico" state="down"
2020-02-21 13:51:39.745 [INFO][50] route_table.go 577: Syncing routes: adding new route. ifaceName="vxlan.calico" ipVersion=0x4 targetCIDR=172.29.122.0/26
2020-02-21 13:51:39.745 [WARNING][50] route_table.go 604: Failed to add route error=network is down ifaceName="vxlan.calico" ipVersion=0x4 targetCIDR=172.29.122.0/26
2020-02-21 13:51:39.745 [INFO][50] route_table.go 577: Syncing routes: adding new route. ifaceName="vxlan.calico" ipVersion=0x4 targetCIDR=172.28.122.0/26
2020-02-21 13:51:39.745 [WARNING][50] route_table.go 604: Failed to add route error=network is down ifaceName="vxlan.calico" ipVersion=0x4 targetCIDR=172.28.122.0/26
2020-02-21 13:51:39.745 [INFO][50] route_table.go 577: Syncing routes: adding new route. ifaceName="vxlan.calico" ipVersion=0x4 targetCIDR=172.29.193.64/26
2020-02-21 13:51:39.745 [WARNING][50] route_table.go 604: Failed to add route error=network is down ifaceName="vxlan.calico" ipVersion=0x4 targetCIDR=172.29.193.64/26
2020-02-21 13:51:39.745 [INFO][50] route_table.go 247: Trying to connect to netlink

the route table of the bad node is bellow , it only have local pod route , and miss subnet route for pods on other node
# ip r
default via 10.6.0.1 dev dce-mng proto static metric 100
default via 10.7.0.1 dev ens224 proto static metric 101
10.6.0.0/16 dev dce-mng proto kernel scope link src 10.6.150.57 metric 100
10.7.0.0/16 dev ens224 proto kernel scope link src 10.7.117.177 metric 100
169.254.0.0/16 dev parcel-vip proto kernel scope link src 169.254.232.109
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.28.70.128 dev calia1e9256f877 scope link
172.28.70.129 dev cali23d0198f573 scope link
172.28.70.130 dev cali63579a15e5c scope link
172.28.70.131 dev calie07fbc9b479 scope link

networkmanager has no configuration for this tunnel
# nmcli con show
NAME                 UUID                                  TYPE            DEVICE
Wired connection 1  a7eba6a4-1e06-3217-a476-8fb0326c9a38  802-3-ethernet  ens224
dce-mng             7adf786c-6968-4d2c-aba0-8f0e89606097  802-3-ethernet  dce-mng
docker0             d0d7ca77-316e-4260-8b52-8bf6a811ea3f  bridge          docker0

Expected Behavior

tunnel interface should stay up , and route should be added for pod on other nodes
Current Behavior

tunnel interface stay down , and pods on the bad node can not access pods on other nodes
Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

Calico version
Orchestrator version (e.g. kubernetes, mesos, rkt):
Operating System and version:
Link to your project (optional):

@weizhouBlue
Try to add the following to /etc/NetworkManager/conf.d/calico.conf
unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico

After that restart the NetworkManager and check if the vxlan.calico interface comes up


text:
Request to build multiarch docker images

When attempting to use Calico with operator install and enable the apiserver the api pod fails to deploy as no arm64 image is available
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  110s                default-scheduler  Successfully assigned calico-apiserver/calico-apiserver-6ccdbf984d-jz568 to k8s-controller
  Normal   BackOff    21s (x5 over 104s)  kubelet            Back-off pulling image "docker.io/calico/apiserver:master"
  Warning  Failed     21s (x5 over 104s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    6s (x4 over 107s)   kubelet            Pulling image "docker.io/calico/apiserver:master"
  Warning  Failed     4s (x4 over 105s)   kubelet            Failed to pull image "docker.io/calico/apiserver:master": rpc error: code = Unknown desc = no matching manifest for linux/arm64/v8 in the manifest list entries
  Warning  Failed     4s (x4 over 105s)   kubelet            Error: ErrImagePull

Merged https://github.com/projectcalico/calico/pull/5386

apiserver is still missing some other arch, but this issue can be closed for now.

That's great thanks very much... however, I see the following currently:
`Events:
Type     Reason     Age                    From               Message

Normal   Scheduled  3m49s                  default-scheduler  Successfully assigned calico-apiserver/calico-apiserver-6fdddbdc9c-gr5xg to k8s-worker03
Normal   Pulling    2m11s (x4 over 3m46s)  kubelet            Pulling image "docker.io/calico/apiserver:v3.21.4"
Warning  Failed     2m9s (x4 over 3m45s)   kubelet            Failed to pull image "docker.io/calico/apiserver:v3.21.4": rpc error: code = Unknown desc = no matching manifest for linux/arm64/v8 in the manifest list entries
Warning  Failed     2m9s (x4 over 3m45s)   kubelet            Error: ErrImagePull
Warning  Failed     117s (x6 over 3m44s)   kubelet            Error: ImagePullBackOff
Normal   BackOff    105s (x7 over 3m44s)   kubelet            Back-off pulling image "docker.io/calico/apiserver:v3.21.4"
`

authors: caseydavenport frozenprocess jonstevecg


text:
Calico on EKS - disable source/destination check

Hey, I am sorry if this is a bit silly but I am trying to get Calico configured on my EKS cluster and I don't quite get what I am supposed to do about the AWS src/dest check setting
I am following the Tigera Operator approach and the doc has no reference to it. However, if I check the "manifests" approach, I see a reference to the below:

kubectl -n kube-system set env daemonset/calico-node FELIX_AWSSRCDSTCHECK=Disable

What should I do?

For anyone who finds this in the future: I found the answer and I am happy to leave it in here.

disabling src/dst checks is a prerequisite for using VXLAN-CrossSubnet but not for VXLAN

This means that using a Calico Installation manifest like the below (which is the one from the Calico docs), does not need changes on the AWS side:
kind: Installation
apiVersion: operator.tigera.io/v1
metadata:
  name: default
spec:
  kubernetesProvider: EKS
  cni:
    type: Calico
    ipam:
      type: Calico
  calicoNetwork:
    bgp: Disabled
    hostPorts: Enabled
    linuxDataplane: Iptables
    ipPools:
      - blockSize: 26
        cidr: 172.16.0.0/16
        encapsulation: VXLAN # <---------
        natOutgoing: Enabled
        nodeSelector: all()

This video helped me a lot: https://www.tigera.io/blog/everything-you-need-to-know-about-kubernetes-pod-networking-on-aws/


text:
Calico-apiserver pods are all restarting on change to extension-apiserver-authentication Configmap causing webhooks to fail

Expected Behavior
calico-apiservers should always be available to service webhooks.
Current Behavior
Currently when calico-apiserver pods detect a change to the extension-apiserver-authentication ConfigMap in the kube-system namespace they all terminate with exit code 0 causing the kubelet to restart them.  During this time they are not available to handle webhooks and so can cause issues with deployments.
The following logs lines are show.
time="2024-02-06T18:40:58Z" level=info msg="Main client watcher loop"
I0206 18:41:24.185839       1 run_server.go:64] Detected change in extension-apiserver-authentication ConfigMap, exiting so apiserver can be restarted

Possible Solution
calico-apiserver should either not restart on changes to the extension-apiserver-authentication Configmap and load the changes it needs
OR
Trigger a rolling restart of the deployment to ensure only one pod is down at a time.
Steps to Reproduce (for bugs)
Trigger a change to the extension-apiserver-authentication Configmap in the kube-system namespace.
Context
Our CI is failing periodically because the webhooks are not available.
We are getting alerts for pods restarting.
Your Environment

apiserver:v3.27.0
AWS EKS
Amazon linux

@caseydavenport

Just giving my 2 cents, but keep in mind I don't know much about calico-apiserver and I'm coming from having some experience doing KTHW and setting this stuff up for the datadog-cluster-agent's implementation of the custom metrics API, so I may say some stuff that's a taller order than I think it is.

The only caveat here is that I believe there are some possible changes to the extension-apiserver-authentication configmap that would invalidate all running apiservers, so a rolling update would potentially be slower than desired. That configmap contains the certs needed to authenticate with the main Kubernetes API server, so if those certificates change it's quite possible your Calico API servers won't be functional anyway.

As long as the signing CA root for the certs in the extension-apiserver-authentication ConfigMap doesn't change, then I don't think it's going to make all the API servers non-functional. You can have requestheader-client-ca-file in that ConfigMap change its value without things breaking, since that file can be a cert bundle containing an intermediate CA (it is on my cluster, at least.) I am 90% certain I recall rotating the intermediate CA without things breaking.
Regardless, since it is a matter of the client cert you are presenting being rejected or not, I think that's kind of moot; you would just reinitialize your K8s client if you get an SSL error saying your cert was rejected. You already are proactive when you see changes to the ConfigMap, so I think you just need to make the action you take to be to reinitialize the K8s client. I assume that whatever concurrency you have going on will make that easier said than done, but hopefully it is not too hard to finagle.
AFAICT, we never have this problem with the custom metrics API backed by the datadog-cluster-agent, so I do believe this is possible to accomplish without the pods restarting themselves.

Add a config option to disable that restart in the Calico API server, so it doesn't restart itself on change.
Instead, make sure we use an annotation of the hashed configmap contents, applied to the pods from the tigera/operator.

I believe that an approach like this will be kind of annoying to end-users; my team right now is just rewriting our monitors to be less sensitive to calico-apiserver restarting because that is easier than having to finagle stuff with the tigera operator. But, I am ignorant about how calico-apiserver's internals work, so maybe this is the only feasible solution for some reason, so I don't want to discredit it. I'm mostly just saying that as an end-user, I wouldn't want to have to set some config option that by all reasonable accounts should just be the default behavior, and I wouldn't want to have to interact with the tigera operator in any capacity to facilitate this behavior that I would want to be the default. As an end-user, I would want things to just transparently work without interruption if possible. If that's not done by refreshing a K8s client when a ConfigMap change is detected, then that's fine, I would just want it to be hands-off, if possible.
Hopefully my comment gives you some good ideas or a good lead toward a solution, and you forgive me for speaking in generalities since I don't know calico-apiserver's internals. Thank you very much!

The rolling update approach was my first though just because it ensures that, speaking generally, changes that result in a "bad" configuration can be caught by a hung upgrade. But I'm not super attached to that.
If we can manage to simply recreate the client using the new configuration, I am good with that!


text:
Added dummy routetable for network policy only mode

Description
This PR adds a config flag for felix named RouteSyncDisabled:

The default value is false.
Setting this to true will prevent felix from doing any updates on the route table.
This is applied to all instances of RouteTable (IPv4, IPv6, VXLAN and Wireguard).

In order to make sure no operations are done against the linux routing table, a new implementation of RouteTable called DummyTable was added. While using the DummyTable all operations are skipped, enabling the use of felix as a network policy controller only.
Related issues/PRs
More information on the context for this change can be found in #5247.

Release Note
None required?

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks @juanfresia for the PR. There are multiple places need to be updated in order to add a Felix configuration.

./api directory. https://github.com/projectcalico/calico/blob/master/api/pkg/apis/projectcalico/v3/felixconfig.go and auto generated files
./libcalico directory.  CI and auto generated files

Thanks you both for the comments, I'll send a commit to address them shortly

Hi @song-jiang @fasaxc sorry for the delay. I've just updated the PR with the requested changes.

I added the same change for `IPv6, VXLAN and Wireguard, which are all the uses of RouteTable I could find.

I think IPv6 and IPv4 are quite similar; but I'm not super familiar with the other two. If you know of any side effects of this change for those, you can point me in the right direction to fix them; or maybe it'd be an alternative to keep this flag from affecting VXLAN and Wireguard.

I also changed the semantics to be RouteSyncDisabled with a omitempty config and a false default.
api and libcalico-go sub-projects have been updated (hopefully, in the right way).

Also updated the PR's original description.
Let me know if there are any more needed changes/or ideas to be added here. Since this is basically adding a Dummy implementation of an interface, I'm not quite sure which tests would be valuable here; let me know if you have any ideas here as well.

@juanfresia I've reviewed your changes and added couple of comments. Thanks!

@song-jiang Just fixed the log level as per your comment. I had left you some question regarding your other comments, what do you think would be the best alternative?

@juanfresia I'm sorry I somehow missed this thread and have been on vacation lately. Thanks for all your work.
I'd like to get input from @mikestephen on changes related to wireguard, but the rest code changes LGTM.
By the way, could you please rebase on master branch again?

Thanks @song-jiang , I'll rebase the changes

@juanfresia looks like this still needs a rebase.

/sem-approve

Hi @mikestephen! I believe @song-jiang wanted your input on this change, do you have any thoughs on this?

@juanfresia I had a chat with @mikestephen and @caseydavenport , we think at this point, the PR is good to get merged. I'll raise a follow up PR to address two improvements:

Fix Calico doc for the new

@juanfresia I had a chat with @mikestephen and @caseydavenport , we think at this point, the PR is good to get merged. I'll raise a follow up PR to address two improvements:

Fix Calico doc for the new

/sem-approve

/sem-approve

@juanfresia looks like this needs another rebase and then it's ready to go with the passing tests.

/sem-approve

Thanks @mgleung, I'll sync it again against master shortly

/sem-approve

Hi @song-jiang, you mentioned some documentation PRs missing; is there an example I can use as an example? Do you think we can merge this PR or should we keep it open until documentation is ready?
Other than that, is there anything we are missing to merge this change?

@juanfresia Sorry I missed your comment. I think we can just merge this PR. We need to do the following:

Fix Calico doc for the new felix parameter.
Make wireguard start using the routeTable interface.

I can take care both point but please let me know if you are happy to do the first one. Thanks.

source: gharchive/pull-request

authors: CLAassistant caseydavenport juanfresia mgleung song-jiang


text:
Documented CALICO_NAT_OUTGOING Parameter

Added documentation for CALICO_NAT_OUTGOING  added by pull request #1640.
Fixes #516

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

This looks good but will need some updates after https://github.com/projectcalico/calicoctl/pull/1640 is finalized.

@VincentS Could you please rebase this and get rid of that merge commit from me?  I hit the appealing "Update branch" button and it created that merge commit, sorry I'd never used it before and need to remember to never use it again.
After the rebase then I think this is good.

Done!

LGTM
@robbrockbank What do you think?

LGTM

authors: CLAassistant VincentS robbrockbank tmjd


text:
Clean up Makefile

Description
A few sentences describing the overall goals of the pull request's commits.
Please include

the type of fix - (e.g. bug fix, new feature, documentation)
some details on why this PR should be merged
the details of the testing you've done on it (both manual and automated)
which components are affected by this PR
links to issues that this PR addresses

Todos

[ ] Tests
[ ] Documentation
[ ] Release note

Release Note

None required

CC @svInfra17

@bcreane thanks, I've updated according to your comments.


text:
Moving scripts to vm_manager_binaries project

Tracked-On: OAM-105759
Signed-off-by: tprabhu vignesh.t.prabhu@intel.com

LGTM

authors: YadongQi iViggyPrabhu


text:
Adds Contour and Envoy Service Support

Adds support for managing the Contour and Envoy Service.
Fixes https://github.com/projectcontour/contour-operator/issues/51
Requires https://github.com/projectcontour/contour-operator/pull/65 for port variables.
/assign @jpeach @stevesloka
/cc @Miciah
Signed-off-by: Daneyon Hansen daneyonhansen@gmail.com

@stevesloka @jpeach what are your thoughts on https://github.com/projectcontour/contour-operator/pull/52#discussion_r510429040. As we've discussed, I would like v1alpha of contour-operator to be as consitent as possible with the upstream manifests. Do you have any reservations for Envoy's Service to use AWS NLB instead of ELB?

@Miciah commit 953224f addresses all your comments accept https://github.com/projectcontour/contour-operator/pull/52#discussion_r509854161, PTAL.

@stevesloka @jpeach what are your thoughts on #52 (comment). As we've discussed, I would like v1alpha of contour-operator to be as consitent as possible with the upstream manifests. Do you have any reservations for Envoy's Service to use AWS NLB instead of ELB?

I don't know whether anyone actually uses the upstream default in production configurations. I have no objection to changing it, but I'd prefer that change to be informed by AWS deployment experience. Note that, IIRC, proxy protocol is not enabled by default.

/lgtm

Note that, IIRC, proxy protocol is not enabled by default.

@jpeach understood and why I created https://github.com/projectcontour/contour-operator/issues/49.

authors: Miciah danehans jpeach

repo: projectcontour/contour-operator


text:
Use utils helper libraries

Use utils.
This PR closes #68.

Thanks @edoardottt !

repo: projectdiscovery/fastdialer

license: MIT


text:
Inability to List Both Normal URLs and Extensions in a Single Operation

Katana Version:
v1.0.5
Current Behavior:
Currently, Katana does not provide a way to list both normal URLs and specific file extensions in one operation. Users can either obtain a list of URLs using the default settings or apply the extension match option (-em) to filter for specific file extensions. However, when using -em, URLs without an extension are omitted from the final output.
Desired Behavior:
Modify Katana's behavior to list all URLs and file extensions by default. Introduce functionality for the -em option to serve as a filter that includes only the specified file extensions in the output. This change would allow users to see the complete set of resources initially and have the option to narrow down the results based on specific extension criteria, enhancing usability and flexibility.
Steps To Reproduce (Current Behavior):

Run Katana with the command: katana -u https://chaos.projectdiscovery.io -headless -depth 2
Observe that only URLs are listed, and file extensions are not included.
Execute Katana with extension filtering: katana -u https://chaos.projectdiscovery.io -headless -depth 2 -em css,js,ico,jpg,png,html
Notice the inclusion of specified extensions in the results, but URLs without an extension are missing.

Results
katana -u https://chaos.projectdiscovery.io -headless -depth 2

https://chaos.projectdiscovery.io
https://chaos.projectdiscovery.io/app.bundle.css
https://chaos.projectdiscovery.io/app.js

katana -u https://chaos.projectdiscovery.io -headless -depth 2 -em css,js,ico,jpg,png,html

https://chaos.projectdiscovery.io/fevicon.png
https://chaos.projectdiscovery.io/app.bundle.css
https://chaos.projectdiscovery.io/app.js
https://chaos.projectdiscovery.io/361bc8b680f5b7c8f0bd7fb587ea7666.png
https://chaos.projectdiscovery.io/326b684b7243f6148e7ec7dcd3ba1d5b.png
https://chaos.projectdiscovery.io/e9b61c5e5a0c43cdcd96fcc568af8e36.png

Proposed Fix:
Implement changes to the crawling and listing mechanism to display all accessible URLs and assets by default. Adjust the -em flag functionality to act as a post-crawl filter that refines the output to include only the assets with the specified extensions. This approach ensures a comprehensive view of the site's resources is available by default, with the flexibility to focus on specific types of files as needed.
Benefits:

Provides a complete overview of all site resources without the need to run multiple commands.
Enhances user efficiency by simplifying the process of targeting specific file types.
Improves Katana's flexibility and adaptability to different use cases.

Thanks for this issue @swdbo - I do think this is intended behavior but your ideas could be great enhancements.

cc @Mzack9999

authors: olearycrew swdbo

repo: projectdiscovery/katana


text:
Notify config file isn't loading

Hello guys,
I am using a fully updated kali Linux on a VPS.
Notify Version: 0.0.2
As i installed notify and runt it as root, i got this message:
Running as root, skipping config file write to avoid permissions issues: /root/.config/notify/notify.conf
Could not read configuration file /root/.config/notify/notify.conf - ignoring error: open /root/.config/notify/noti
fy.conf: no such file or directory

Then proceeded to run it as a normal user
and i got this message:
Found existing config file: /home/user/.config/notify/notify.conf
Could not read configuration file /home/user/.config/notify/notify.conf - ignoring error: EOF

I edited the .config file with my webhook and also removed the comment from discord: true
Then i copied the notify.conf from the /home/user/.config/notify directory to the root directory: /root/.config/notify/ (because it was missing from the root directory)
And run it again.
The message i got is this one:
Found existing config file: /root/.config/notify/notify.conf
Could not read configuration file /root/.config/notify/notify.conf - ignoring error: yaml: line 8: did not find expected key

After that i  tried again with the normal user and it WORKED.
But running it as root causes the error i mentioned above.
what is the problem and it cannot read the configuration file when running it as root?

@sh0tal soon we will be pushing new version of notify, we can confirm this is already fixed in the dev version.

Hello @ehsandeep
Okay, thank you for your answer.

repo: projectdiscovery/notify


text:
Geo predicates wiki

Create wiki doc listing geo specific predicates that are used in GeoConcerns. Requested by @tpendragon and the Hydra URI Management Working Group.

Let's take another look at projection predicate.

DCMI Box encoding scheme also accommodates recording projection. Would that be a solution? http://dublincore.org/documents/dcmi-box/

@johnhuck What we want, I think, is just the projection bit. It seems to me that dcmi-box describes the projection of the bounding box not necessarily the projection of the underlying dataset. This is quite common. I suppose what I meant, is, does there exist a predicate/vocabulary that describes projection that we can use out of the box. Or is it like it like geospatial file format, and we'll have to cobble something together ourselves.

@johnhuck  I'm sure you've seen this, but EPSG is a comprehensive, and industry standard list of f coordinate reference systems. Is there a linked data vocab that already uses this list? If not, then perhaps we can build one. This one is easier than file format, because ESPG is exhaustive. Not entirely complete I'm sure, but pretty darn close I'd bet.
http://spatialreference.org/ref/epsg/4326/

Since EPSG assigns URNs to manage different projections, what they are maintaining is already a controlled vocabulary, so there wouldn't be a need to make a different one. As least as far as RDF is concerned, a URN is a URI and can be used anywhere a URI is required.  But if we used bf:projection, whose range is rdfs:Literal, its URI status would be irrelevant, except as an unambiguous identifier, which could perhaps be repurposed later. Several online services like the one you mention seem to provide the value-added service of a URL for the terms. This one supplies the GML metadata for the projection, but I think it's a third-party offering: https://epsg.io/4326.gml

Added a wiki page with a basic table. Needs more information.

We have the basic wiki. I think we can close this issue.

Agreed.

authors: eliotjordan johnhuck


text:
"Integrate tab" - Actions not working

Repro steps

Create a new function via Empy C# template
Go to integrate tab
Add "Azure Storage blob" as output binding
Click save
Click "Go" button in the actions section

Expected result
+New functions page should have opened with "blobTrigger-c#" template selected and settings path and storage account same as Empty-C# output binding.
Actual
+New functions page opens with nothing selected

Fixed with 69d854d41

authors: fashaikh soninaren

repo: projectkudu/AzureFunctionsPortal


text:
Not possible anymore to deploy a sln which also contains a new project.json .net core project ?

I'm not sure if this is the correct entry point to ask a questions, but I noticed that's not possible anymore to deploy a webjob from github if the project contains a project.json .net core project.
It seems that msbuild is not executed ?
See error messages below: (The error message is caused because the KendoGridBinderEx project is not build ?

Adding package 'KendoUIWeb.2014.1.318' to folder 'D:\home\site\repository\packages'
Added package 'KendoUIWeb.2014.1.318' to folder 'D:\home\site\repository\packages'
Restoring packages for D:\home\site\repository\KendoGridBinderEx\project.json...
Committing restore...
Writing lock file to disk. Path: D:\home\site\repository\KendoGridBinderEx\project.lock.json
D:\home\site\repository\KendoGridBinderEx\KendoGridBinderEx.xproj
Restore completed in 968ms.
NuGet Config files used:
C:\DWASFiles\Sites#1stef-kendogridbinderex\AppData\NuGet\NuGet.Config
Feeds used:
https://api.nuget.org/v3/index.json
Installed:
46 package(s) to packages.config projects
D:\Program Files (x86)\MSBuild\14.0\bin\Microsoft.Common.CurrentVersion.targets(1819,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "KendoGridBinderEx". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [D:\home\site\repository\KendoGridBinderEx.Examples.Business\KendoGridBinderEx.Examples.Business.csproj]
QueryContext\QueryContext.cs(43,73): error CS0246: The type or namespace name 'KendoGridBaseRequest' could not be found (are you missing a using directive or an assembly reference?) [D:\home\site\repository\KendoGridBinderEx.Examples.Business\KendoGridBinderEx.Examples.Business.csproj]
QueryContext\QueryContext.cs(43,16): error CS0246: The type or namespace name 'KendoGridEx<TEntity, TViewModel>' could not be found (are you missing a using directive or an assembly reference?) [D:\home\site\repository\KendoGridBinderEx.Examples.Business\KendoGridBinderEx.Examples.Business.csproj]
QueryContext\IQueryContext.cs(18,66): error CS0246: The type or namespace name 'KendoGridBaseRequest' could not be found (are you missing a using directive or an assembly reference?) [D:\home\site\repository\KendoGridBinderEx.Examples.Business\KendoGridBinderEx.Examples.Business.csproj]
QueryContext\IQueryContext.cs(18,9): error CS0246: The type or namespace name 'KendoGridEx<TEntity, TViewModel>' could not be found (are you missing a using directive or an assembly reference?) [D:\home\site\repository\KendoGridBinderEx.Examples.Business\KendoGridBinderEx.Examples.Business.csproj]
Failed exitCode=1, command="D:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe" "D:\home\site\repository\KendoGridBinderEx.Examples.MVC\KendoGridBinderEx.Examples.MVC.csproj" /nologo /verbosity:m /t:Build /t:pipelinePreDeployCopyAllFilesToOneFolder /p:_PackageTempDir="D:\local\Temp\8d4047cf48f3562";AutoParameterizationWebConfigConnectionStrings=false;Configuration=Release;UseSharedCompilation=false /p:SolutionDir="D:\home\site\repository.\"
An error has occurred during web site deployment.
\r\nD:\Program Files (x86)\SiteExtensions\Kudu\59.51102.2512\bin\Scripts\starter.cmd "D:\home\site\deployments\tools\deploy.cmd"

I don't see any msbuild commands, only restore ?

When you say 'not possible anymore', do you mean that the exact same scenario used to work and no longer does? Or that you had not tried it before on solutions that also have a project.json?
Note that project.json is on its way out with the new Core tooling.

Yes sorry for the vague description, but indeed it did work fine for several years/months and all of a sudden, it did not work anymore.
Can you indicate which log files I can review in kudo which show some more details?

I has a problem which seems to cause the same errors. I have opened a bug on nuget.
My workaround was to add a .deployment file with my specific project, because the problematic project.json was only used on a test project not necessary for my website.

@StefH does this issue still persist?
Can you share a github repository that highlights your project structure so that I can try to reproduce the issue at my end, thanks

This one:
https://github.com/StefH/KendoGridBinderEx/tree/master/KendoGridBinderEx.Examples.MVC

I just submitted a pull request, it should work for you
build in Azure is different than build in VS, you probably build your entire solution (all projects) locally whereas Azure only build what's necessary (the web project and projects it depends on)
in your case, KendoGridBinderEx.Examples.MVC.csproj is the one argument consumed by msbuild and inside this csproj file, it references KendoGridBinderEx.Examples.Business.csproj (so msbuild knows it needs to build this reference)

I don't see any msbuild commands, only restore ?

to answer your question, the error message you are seeing here:

D:\Program Files (x86)\MSBuild\14.0\bin\Microsoft.Common.CurrentVersion.targets(1819,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "KendoGridBinderEx". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [D:\home\site\repository\KendoGridBinderEx.Examples.Business\KendoGridBinderEx.Examples.Business.csproj]

is the output of msbuild KendoGridBinderEx.Examples.Business.csproj, it is complaining that you referenced KendoGridBinderEx.dll but this file is missing. The solution it simply tell msbuild that it needs to build this assembly itself

This PR solved my issue.
Thanks for your help.

authors: StefH davidebbo tbolon watashiSHUN

repo: projectkudu/kudu


text:
Fix synctriggers for functions

This commit added code to make sure that synctriggers is always called
after a function app has finished deploying. Additionally, this change
also adds an empty settriggers call that is required by the front end to
notify the zip cache that changes were made.
Fixes https://github.com/projectkudu/kudu/issues/3238

This PR was waiting on successful ANT92 deployment (platform). That's almost completed, and I will now update this PR to get it ready to be completed.

Merged here -- https://github.com/projectkudu/kudu/commit/86918e0c0f1098afefcd5e1a03e1d7281cb850c8


text:
Release 4.0.1

Pull requests into cqm-execution require the following. Submitter and reviewer should :white_check_mark: when done. For items that are not-applicable, note it's not-applicable ("N/A") and :white_check_mark:.
Submitter:

[ ] This pull request describes why these changes were made.
[ ] Internal ticket for this PR:
[ ] Internal ticket links to this PR
[ ] Code diff has been done and been reviewed
[ ] Tests are included and test edge cases
[ ] Tests have been run locally and pass

Reviewer 1:
Name:

[ ] Code is maintainable and reusable, reuses existing code and infrastructure where appropriate, and accomplishes the task’s purpose
[ ] The tests appropriately test the new code, including edge cases
[ ] You have tried to break the code

Reviewer 2:
Name:

[ ] Code is maintainable and reusable, reuses existing code and infrastructure where appropriate, and accomplishes the task’s purpose
[ ] The tests appropriately test the new code, including edge cases
[ ] You have tried to break the code

Do we need to update the readme file? It points to QDM models

Do we need to update the readme file? It points to QDM models

Yeah. Looks like README should be replaced. It has QDM samples and links to QDM models.

authors: adongare serhii-ilin

repo: projecttacoma/cqm-execution


text:
When making a FromArray deriving shouldn't an enum use fromValue instead of fromName?

Code generated is:
public static function fromArray(array $data): Commission
{
    $condition = \BolCom\RetailerApi\Model\Offer\Condition::fromValue($data['condition']);
}

It would make sense in my mind that the value of a payload should use the actual enum value instead of the name of the enum?

Let me try to explain. In my case I'm not using the types to use it in the prooph event store. I'm getting a payload from an external service (calling something over rest) and want to return a type based on the response:
public function __invoke(GetCommission $getCommission): Commission
{
    $response = $this->client->get("/retailer/commission/{$getCommission->ean()->value()}", [
        'query' => [
            'condition' => $getCommission->condition()->value(),
            'price' => $getCommission->price()->value()
        ],
        'headers' => ['Accept' => 'application/vnd.retailer.v4+json']
    ]);
    return Commission::fromArray($response->getBody()->json());
}

I get the actual 'ugly' enum value from the remote system, and want to be able to set the value. Would be nice if I wouldn't have to reformat the response before putting it in a type.

Oh I see what the system does when there isn't a while construct, it just gives all the values an integer number..

Sorry what? Which while construct?
I understand that you have a use case where you want fromArray / toArray to use enum values and not enum names. However I am not willing to change this behavior, as I have the other use case more often.
But... There is a solution for both of us:
Foo deriving (Enum(useValue))
There is an open ticket to allow to pass options to derivings, but that's not an easy task because it requires to update the parser as well.
I have no time currently to do this, but if you are willing to work on this, let me know.
Still struggling to find out what while construct you are referring to.

Sorry what? Which while construct?

I can't type, was a little bit late: I meant the Enum with(FIELD='bla') construct.
As for the changes to the parser logic, might be interesting, I'll have to see if I can figure that out.

Got something working! https://github.com/paales/fpp/pull/2
Will create a PR when i've got the code sniffer thing set up locally.

resolved via https://github.com/prolic/fpp/pull/106

authors: paales prolic

repo: prolic/fpp


text:
namespace attributes and ressources

This is the fix for issue #12

:+1:


text:
JSON Parse Failures count keeps growing

We're running elasticsearch and elasticsearch_exporter on k8s and querying elasticsearch_cluster_health_json_parse_failures with Grafana. Each time our elasticsearch pod restarts, we see increments of this value, which makes sense since the health-check may not receive an expected response. The issue is that this value doesn't seem to drop unless we restart the exporter. Is this by intentional behavior, or should it decrease over time?
The problem we're facing with this is we're not sure what our alert conditions should look like. Intuitively, we assumed and set off an alert whenever json_parse_failures is greater than zero, but doesn't feel right to restart the exporter every time an alert is received.
Further, is there a command to flush this counter that I might've missed?

This seems resolved somehow, as our elasticsearch pods restart a lot from time to time, but haven't got such an issue for months.


text:
Remote write receiver support

It would be awesome to have remote write receiver support in prom-label-proxy, so we are able to push metrics into Prometheus through prom-label-proxy which forces the metrics on all incoming metrics.
The endpoint that should be supported is /api/v1/write, docs can be found here.

I think this would be very nice. We recently implemented exactly this in the Observatorium API, so it's proof that doing so is relatively straightforward

repo: prometheus-community/prom-label-proxy


text:
matcher is not native regexp why?

What did you do?
I found that the matcher is not a native regexp when configuring routes using alertmanager, for example
- receiver: DBA
matchers:
- cluster =~ "^(mysql|redis)"
Can only match 'mysql' or 'redis' itself, but not match start with 'mysql01','mysql02','redis01'...
Use  as is expect
I checkd source code for matcher.go:
if t == MatchRegexp || t == MatchNotRegexp {
re, err := regexp.Compile("^(?:" + v + ")$")
if err != nil {
return nil, err
}
m.re = re
}
return m, nil
why regexp.Compile("^(?:" + v + ")$") or  regexp.Compile(v)?

Alertmanager version:
All versions that support matchers

Alertmanager regexp matchers are always anchored (as it is for Prometheus). To get back to your question, you could do (mysql|redis).* to match any label starting with mysql and redis.

But there is a visual editor in the official documentation that is a native regexp.
https://prometheus.io/docs/alerting/latest/configuration/
That's what makes me wonder

repo: prometheus/alertmanager


text:
Basic state persistence for the alertmanager silences

Extends the silences.json model to have the memoryAlertManager persist its state to an on-disk JSON file. This allows restarts to avoid re-dispatching sent notifications. The persistence file can be disabled by setting it to a blank string "".

Hey, thanks for sending this! Since we're planning on rewriting the Alertmanager completely (which will include features like this), I'm thinking whether to make this ready for merging. On the other hand, it doesn't have to be perfect then if it gets replaced soon anyways.
Besides minor issues (and lack of tests, which is a general AM problem), there are at least some things that make me unsure about the full correctness here: a restart should at least be treated like a configuration reload to make sure aggregates are (re)associated with the correct (potentially new) rules, so alertManager.SetAggregationRules(conf.AggregationRules()) should be called after loading the persisted file. In general, just serializing and deserializing whole datastructures including pointered objects that weren't initially designed for that makes me a bit nervous - I couldn't immediately see a problem with that though. Are you reasonably confident that's all ok and no references will be broken after reload?

In my (limited) testing it all seemed fine but I'll run it through it's paces a bit more extensively in the next few days. Good point about the rule changes - I'll test that to see what the damage is.
I pulled this together to solve an immediate short term problem (namely, when I do a config reload I inevitably have to take down the alertmanager anyway, so saving my alert emits stops me spamming out a whole bunch of new notifications).

Also noticed that alertManager.Run(), which does the loading, is called asynchronously in main.go, so it's possible for the web interface to start up and accept alerts already and those would be removed again when the alertstates are loaded afterwards...

Since it looks like the new alertmanager is ramping up for roll out, and I'm moving over to it myself, I'm going to close this off for the time being since it won't apply to it.

authors: juliusv wrouesnel


text:
silences: avoid deadlock

Calling gossip.GossipBroadcast() will cause a deadlock if
there is a currently executing OnBroadcast* function.
See #982

LGTM


text:
Document that invalid label name chars get replaced by underscores in SD meta label names

This is about SD meta labels like __meta_kubernetes_node_label_<labelname>.
https://prometheus.io/docs/operating/configuration does not mention that fact anywhere.

Can I take this one?

Go ahead.

@infoverload are you still interested in working on this?

authors: brian-brazil hdost infoverload juliusv

repo: prometheus/docs


text:
Carry over pre-allocation sizes across TSDB blocks and targets

We see in #2774 that new TSDB blocks being cut as well as pods being scaled cause significant memory spikes. (Seems we regressed a bit in the former since alpha.0 – not exactly sure why.)
One possible reason is that we allocate maps and slices, which then basically immediately converge to their max size. On the way, we grow these structures, which may make a lot of allocations that go to garbage shortly after.
For slices, this is easy to measure (see below). For maps, which account for most of those structures, I'm not entirely sure whether in how far it behaves the same or just dynamically adds new buckets.
For slices however, the growth is only by power of two until 1KB and slows down after, causing more garbage to be generated. Until 200k elements are created, 860k elements of garbage are created: https://play.golang.org/p/KcvqVdZjR1
We should try to pre-allocate on instantiation based on size info we have from previous blocks and targets and see whether that reduces the spikes.
@Gouthamve

Closing as this no longer happens in 2.0 as we don't cut blocks anymore.

authors: Gouthamve fabxc

repo: prometheus/prometheus


text:
TimeOut in Histogram and Summaries

Proposal:
Sometimes a user start timer, but don't want to stop it. In such cases, the duration is not logged in any bucket. So, there should be a timeout settings in Histogram/Summary, so that user can configure after how much time, the timer should be automatically stopped, and logged in the appropriate bucket.
Use Case:
In my case, I send an HTTP request to an API, which could give a TimeOutException. In this case, I start the timer before sending the request. Here, I catch a TimeOutException in which I log the exception using Counter metric, but don't stop the Histogram.Timer. That is why, the timer starts, but it dont stop and is not logged in any of the bucket. There can be timeout setting in Histogram in which I can configure that after this much time, the duration should be logged in infinity bucket

You should file this against the client library you are using, as this repository is for the Prometheus server itself.
I would say in this case that it's your responsibility to stop the timer.

Thank you @brian-brazil ,
Filing against the appropriate client library.


text:
remote write filesystem data disorder

What did you do?
we use remote write and nodeexporter to forward data to other system。but we find that filesystem  is error.
the value and mountain of one host is others.

/gitlab is not exist in this host， it belong to other host remotewrite by the same Prometheus.
we cant find '/backup/gitlab' in this host's metrics.
Environment
docker  k8s  linux

Prometheus version:
2.28.1

We use GitHub issues to track Prometheus development. Please use our community channels for usage questions which you can find here https://prometheus.io/community/.


text:
Reduce chunk write queue memory usage 1

This avoids wasting memory on the c.chunkRefMap by re-initializing it regularly. When re-initializing it, it gets initialized with a size which is half of the peak usage of the time period since the last re-init event, for this we also track the peak usage and reset it on every re-init event.
Very frequent re-initialization of the map would cause unnecessary allocations, to avoid that there are two factors which limit the frequency of the re-initializations:

There is a minimum interval of 10min between re-init events
In order to re-init the map the recorded peak usage since the last re-init event must be at least 1000 (objects in c.chunkRefMap).

When re-initializing it we initialize it to half of the peak usage since the last re-init event to try to hit the sweet spot in the trade-off between initializing it to a very low size potentially resulting in many allocations to grow it, and initializing it to a large size potentially resulting in unused allocated memory.
With this solution we have the following advantages:

If a tenant's number of active series decreases over time then we want that their queue size also shrinks over time. By always resetting it to half of the previous peak it will shrink together with the usage over time
We don't want to initialize it to a size of 0 because this would cause a lot of allocations to grow it back to the size which it actually needs. By initializing it to half of the previous peak it will rarely have to be grown to more than double of the initialized size.
We don't want to initialize it too frequently because that would also cause avoidable allocations, so there is a minimum interval of 10min between re-init events

This PR comes from https://github.com/grafana/mimir-prometheus/pull/131. We use it in Grafana Mimir to reduce memory usage in Mimir clusters with thousands of open TSDBs in a single process.
Related to https://github.com/prometheus/prometheus/pull/10874.

Before this change, we have seen this cumulative memory usage across pods in single Mimir deployment.

Legend:

zone-a using new chunk mapper without fix from this PR
zone-b and zone-c are using old chunk mapper before https://github.com/prometheus/prometheus/pull/10051

After applying change from this PR into zone-b and enabling new chunk mapper there, we can see that memory usage has improved compared to zone-a:

Legend:

zone-a using new chunk mapper without fix from this PR
zone-b using new chunk mapper with the fix from this PR
zone-c using old chunk mapper before https://github.com/prometheus/prometheus/pull/10051

Note that memory usage in zone-b still isn't the same as with old chunk mapper (zone-c), but second part is addressed by https://github.com/prometheus/prometheus/pull/10874

Could you say which metric is being graphed, please?  (working set?)

Query is:
sum by (zone) (label_replace(go_memstats_heap_inuse_bytes{job=~"cortex-dev-01/ingester.*"}, "zone", "$1", "pod", "(ingester-zone-.)-\\d+")) / 1e9

With pod names like ingester-zone-<a|b|c>-<number>.

authors: bboreham pstibrany


text:
test

sorry


text:
Change is_superuser to support #88557194

We need to only show the admin dashboard to cloud_admin.

Is this a change we can make? Could it be added to the OPENSTACK_KEYSTONE_ADMIN_ROLES env variable?

I'll try that now Jeff, if so I'll close this PR and update the other.

Found a better way to do this thanks to @jeffdeville

authors: jeffdeville mattamizer


text:
Add Mark a todo as done functionality

Fixes issue #1

build failed, please fix php cs

authors: DannyvdSluijs prolic

repo: prooph/proophessor-do


text:
[gh-55] Revert change that was made to postgres schema parsing that caus...

...es regressions and has no tests.

Can you be more specific about the regression, and point the original commit and associated issue?

Click through to the old ticket.
Basically a default of
Default ''::character varying
Stopped being parsed correctly when someone else fixed a corner case.
Sent from iPhone.
On Nov 18, 2011, at 6:44 AM, Francois Zaninottoreply@reply.github.com wrote:

Can you be more specific about the regression, and point the original commit and associated issue?

Reply to this email directly or view it on GitHub:
https://github.com/propelorm/Propel2/pull/56#issuecomment-2788229

Ok I'm closing that one as it should be better to base your work on your Propel 1.6 PR.

Ok, I will port that patch.

authors: apinstein fzaninotto willdurand

license: mit


text:
Installation not working

I'm trying to install propel-laravel through composer in my Laravel 5.2.* framework, but it's not working for me.
First of all I tried to do what the documentation said, editting my composer.json like so:
"config": {
    "preferred-install": "dist"
},
"minimum-stability": "alpha"

But when I do a: composer require propel/propel-laravel it seems that it can't find the package. I have to do this in order to install propel-laravel:
composer require propel/propel-laravel:~2.0.0-alpha5
Now it's able to download the package, but  I'm getting the following error during the installation:
  Problem 1
    - Installation request for propel/propel ~2.0.0-alpha5 -> satisfiable by propel/propel[2.0.0-alpha5].
    - Conclusion: remove symfony/finder v3.0.4
    - Conclusion: don't install symfony/finder v3.0.4
    - propel/propel 2.0.0-alpha5 requires symfony/finder ~2.3 -> satisfiable by
symfony/finder[v2.3.0, v2.3.1, v2.3.10, v2.3.11, v2.3.12, v2.3.13, v2.3.14, v2.3
.15, v2.3.16, v2.3.17, v2.3.18, v2.3.19, v2.3.2, v2.3.20, v2.3.21, v2.3.22, v2.3
.23, v2.3.24, v2.3.25, v2.3.26, v2.3.27, v2.3.28, v2.3.29, v2.3.3, v2.3.30, v2.3
.31, v2.3.32, v2.3.33, v2.3.34, v2.3.35, v2.3.36, v2.3.37, v2.3.38, v2.3.39, v2.
3.4, v2.3.40, v2.3.5, v2.3.6, v2.3.7, v2.3.8, v2.3.9, v2.4.0, v2.4.1, v2.4.10, v
2.4.2, v2.4.3, v2.4.4, v2.4.5, v2.4.6, v2.4.7, v2.4.8, v2.4.9, v2.5.0, v2.5.1, v
2.5.10, v2.5.11, v2.5.12, v2.5.2, v2.5.3, v2.5.4, v2.5.5, v2.5.6, v2.5.7, v2.5.8
, v2.5.9, v2.6.0, v2.6.1, v2.6.10, v2.6.11, v2.6.12, v2.6.13, v2.6.2, v2.6.3, v2
.6.4, v2.6.5, v2.6.6, v2.6.7, v2.6.8, v2.6.9, v2.7.0, v2.7.1, v2.7.10, v2.7.11,
v2.7.12, v2.7.2, v2.7.3, v2.7.4, v2.7.5, v2.7.6, v2.7.7, v2.7.8, v2.7.9, v2.8.0,
 v2.8.1, v2.8.2, v2.8.3, v2.8.4].
    - Can only install one of: symfony/finder[v2.8.0, v3.0.4].
    - Can only install one of: symfony/finder[v2.8.1, v3.0.4].
    - Can only install one of: symfony/finder[v2.8.2, v3.0.4].
    - Can only install one of: symfony/finder[v2.8.3, v3.0.4].
	...
	...
    - Can only install one of: symfony/finder[v2.7.8, v3.0.4].
    - Can only install one of: symfony/finder[v2.7.9, v3.0.4].
    - Installation request for symfony/finder (locked at v3.0.4) -> satisfiable
by symfony/finder[v3.0.4].

Installation failed, reverting ./composer.json to its original content.

Anyone idea how I can  fix this problem?

Another note, I had the package propel/propel installed. I just removed that through Composer.
Now when I try to install propel/propel-laravel I get the error:
  Problem 1
    - The requested package propel/propel-laravel 2.0.0-alpha5 exists as
propel/propel-laravel[dev-Big-Shark-patch-1, dev-Big-Shark-patch-2, dev-develop, dev-ma
ster] but these are rejected by your constraint.

Not sure why this error is now suddenly showing. I still have `"minimum-stability": "alpha"  in my composer.json file.

Please try dev-develop version, 'propel/propel-laravel: dev-develop'

@Big-Shark , @chinookproject Develop branch did updated and introduced 5.2 support and new command. Read docs and it's recommended for new installations and testing purposes for now.

authors: Big-Shark SCIF chinookproject

repo: propelorm/PropelLaravel


text:
Add support for running CFU-Playground natively with notebook in rad-lab env

Updated environment with CFU-Playground packages, repo, and other needed toolchains. Added CFU-Playground notebook after converting to markdown.

FYI @tcal-x

repo: proppy/rad-lab


text:
Only insert 1 line break after frontmatter

Fixes #916 - unless there was some other reason to insert 2 lines? If so, we can probably just trim a leading \n off content.

Jekyll doesn't care if there's a \n after frontmatter, and parsing the leading \n off content is more trouble than it's worth. I'm fine with this, and allowing people to add their own spaces if that's their thing.

authors: dereklieu timwis

repo: prose/prose


text:
🛑 pl-launchpad.io is down

In 7f83ca3, pl-launchpad.io (https://pl-launchpad.io) was down:

HTTP code: 0
Response time: 0 ms

Resolved: pl-launchpad.io is back up in 75e3813 after 8 minutes.

repo: protocol/upptime-pln


text:
🛑 nftschool.dev is down

In e29a6fc, nftschool.dev (https://nftschool.dev) was down:

HTTP code: 0
Response time: 0 ms

Resolved: nftschool.dev is back up in c6f535b after 1 hour, 24 minutes.


text:
🛑 nftschool.dev is down

In 8eed021, nftschool.dev (https://nftschool.dev) was down:

HTTP code: 0
Response time: 0 ms

Resolved: nftschool.dev is back up in 6bc7afe after 46 minutes.


text:
🛑 nftschool.dev is down

In 42e9d06, nftschool.dev (https://nftschool.dev) was down:

HTTP code: 0
Response time: 0 ms

Resolved: nftschool.dev is back up in 4189521 after 19 minutes.


text:
Display value as HEX

Issue submitter TODO list

[X] I've searched for an already existing issues here
[X] I'm running a supported version of the application which is listed here and the feature is not present there

Is your proposal related to a problem?
I'm frustrated when I'm trying to view a binary message posted to kafka.
Describe the feature you're interested in
A button I could press to switch between the native view and a hex view of the value.
Describe alternatives you've considered
Writing my own client.
Version you're running
56fa824
Additional context
This is related to trying to sort out why my protobuf messages aren't coming out correctly.  The messages themselves come from a 3rd party so I need to extract the hex to show exactly what they are sending.

Hello @jcpunk, thank you for suggestion. We will implement this feature as a Hex serde .

authors: iliax jcpunk

repo: provectus/kafka-ui


text:
Roomba Remote Client: Unknown disconnection error: ID=16

I am regularly getting a warning saying "Unexpectedly disconnected from Roomba xxx.xxx.xxx.xxx, code The connection was lost" which I usually ignore.
But today I got this warning asking to create a new issue. So here it is. Thanks
Logger: roombapy.remote_client
Source: /usr/local/lib/python3.12/site-packages/roombapy/remote_client.py:172
First occurred: 01:05:37 (1 occurrences)
Last logged: 01:05:37

Unknown disconnection error: ID=16.Kindly use https://github.com/pschmitt/roombapy/issues/new

Getting the same error but only since a few days.
Logger: roombapy.remote_client
Bron: /usr/local/lib/python3.12/site-packages/roombapy/remote_client.py:172
Eerst voorgekomen: 17:01:02 (6 gebeurtenissen)
Laatst gelogd: 18:18:10
Unknown disconnection error: ID=16.Kindly use https://github.com/pschmitt/roombapy/issues/new

I got this error ID yesterday when my Roomba's battery died after getting stuck

Yeah i took my robot out of it's dock and back in and+reset HA, didn't have the error since.

@Pastaloverzzz What do you mean by "reset HA" ? Do you mean restarting HA? I did that but I am still getting the error.

@snowmangh Yes i meant a reboot. In my case the iRobot app wouldn't connect either so i think he was trying to connect to another router than the one closest to him. Could it be that the docking station isn't in a good wifi spot? If you get this error every day you could put it closer to a router and see if you keep getting the error then..

@Pastaloverzzz Interesting. I do have a mesh network at home and wondering if that could be the reason I get the error. I do have devices that for some reasons do switch between the mesh point and the actual router which is probably enough to trigger a disconnect.

@snowmangh Well, i suspect on the day i got the error that my router closest to the docking station was down. So chances are you found the culprit 🙂

@Pastaloverzzz Looks like it. Maybe the error handling should account for a short disconnect before throwing this warning. 🙂

authors: Pastaloverzzz snowmangh

repo: pschmitt/roombapy


text:
Add psptoolchain-extra dispatch

Right now this is missing. I'm not sure if this would require us to still set some variables in the github runner settings or if they are already set.

I don’t have grants in this repo to setup the keys

That's a bother, me neither 😬

I'll convert this to a draft for now until we have someone who can set this up.

authors: fjtrujy sharkwouter

repo: pspdev/psp-pacman

license: Unlicense


text:
Appstudio update build-suite-test-component-git-source-omst

Pipelines as Code configuration proposal

Pipelines as Code CI/build-suite-test-component-git-source-omst-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
7 seconds
appstudio-init

✅ Succeeded
14 seconds
clone-repository

✅ Succeeded
7 seconds
appstudio-configure-build

✅ Succeeded
30 seconds
build-container

✅ Succeeded
5 seconds
show-summary

repo: psturc-org/devfile-sample-hello-world


text:
WIP: Coveralls combined

Test pr to check Coveralls test coverage if we run one combined test in Travis.

Changes Unknown when pulling 272a11cec44841c2da8f2b368b2cd5dd6451de26 on coveralls-combined into ** on develop**.

@cam156 well, that's interesting.

@awead That is interesting.  I'm guessing coveralls uses the rpsec process to gather information.

@cam156 so is it worth changing the Travis build? I'm not sure we're really slowing things down all the much with a single build as opposed to a split one.

authors: awead cam156 coveralls

repo: psu-stewardship/scholarsphere


text:
Better map appearance

Change the appearance of the map to look more like League Of Legends (not only some grass-tiles repeated, but maybe some lanes, water, etc)

Connected to #23

authors: EliottBICS ptal


text:
Cache is global, can potentially conflict with external modules

Since required modules are cached in Node.js, it means that multiple modules could be using the same cache at the same time. There should be a .createStore() method that creates a new cache instance.

I think this is a matter of personal opinion.  There are other node caches that do this, but then they suffer from not following typical require patterns (requiring in multiple places does not duplicate itself).
Why not instead namespace your cache keys?

@brandonculver that still wouldn't work. Let's say I have a library which exposes a SomeClient class which uses the node-cache module internally.
var SomeClient = require('my-module').SomeClient;

var client1 = new SomeClient();
var client2 = new SomeClient();

Both client1 and client2 will be using the same cache. This makes node-cache fine to use for standalone apps but restricts its use in published libraries.

@olalonde I understand what you are saying.  I mean namspacing your cache keys.  ('myapp-cache1', myapp-cach2)
I would also question why you are using this particular module in other libraries.  Its a pretty bare bones cache solution.  Again, there are other caching solutions that do exactly what you want https://www.npmjs.com/package/node-cache
It just seems like unnecessary overhead in a currently simple module.  Ultimately its not my call though :wink:

Oh thanks I wasn't aware of this module! FWIW, I don't think it's unnecessary overhead. In fact, I just implemented this feature in 4 lines of code :)
https://github.com/ptarjan/node-cache/pull/43

+1。namespace key is a workaround friendly for mistakes, for example, some module also use memory cache and finally call
cache.clear();

then all my module's cache is away.

What happens when running in cluster mode, like pm2 does? Could this mean the cache is shared between threads?

authors: amenadiel brandonculver olalonde springuper

repo: ptarjan/node-cache


text:
CoinRanking Auth

Updated the authentication for CoinRanking from "No" to "apiKey.

[x] My submission is formatted according to the guidelines in the contributing guide
[x] My addition is ordered alphabetically
[x] My submission has a useful description
[x] The description does not end with punctuation
[x] Each table column is padded with one space on either side
[x] I have searched the repository for any relevant issues or pull requests
[x] Any category I am creating has the minimum requirement of 3 items
[x] All changes have been squashed into a single commit

Just a small change, but increasing accuracy

Just a small change, but increasing accuracy

@tannershimanek
Don't forget to squash all the commits into the single one after committing the changes suggested by @marekdano

This PR is inactive, so the API was added in another PR/commit.
More information at: #2002

authors: matheusfelipeog pawelbr tannershimanek

repo: public-apis/public-apis


text:
ZeroSSL 429 Too Many Requests

ZeroSSL implement a 429 Too Many Requests error if you fire too many requests at once,
even tho the code does one thing after another rather than all at once, you can still hit this limit
what should happen is IF you hot a 429 error, it should retry the request 1-2 seconds later,
im hitting this problem at different stages.
sometimes i hit it when requesting 5 domains, at the newNonce stage
and sometimes i hit it at the after the newNonce but when it tries to get the challenges
https://github.com/cert-manager/cert-manager/issues/5867

Should be fixed with https://github.com/publishlab/node-acme-client/commit/e4e8bde250b78fd07d28ebdf035b6f7b0b71aacb - it will now retry requests that receive 429 or 5xx status codes, respecting Retry-After header if included with the response. Landed in v5.4.0.
Have not been able to test this in the wild yet, please let me know if this resolved your issue and feel free to re-open if the problems persist.

repo: publishlab/node-acme-client


text:
CTM-576: add environment variables for email template. add error stat…

…us if match is unsuccessful.

With the new error state, you need to update the code where it was checking for pending or matched and include the newly added state

ctml-json.service run_match needs to include case to update from error state
useGetMatchResult
trialResultService
...

authors: kevinwangcanada mickey-ng

repo: pughlab/ctims


text:
Rename 'Rare Books and Special Collections' to 'Special Collections'

This was changed in Blacklight but stayed with the old name in voyager.
We should change it in Alma.
Update 'Rare Books and Special Collections' to 'Special Collections'.

I changed the Organization Unit Name for now.
What does the location prefix "Rare Books:" relate to? Is it shorthand for "Rare Books and Special Collections"? In that case, we should change it to "Special Collections:".
Or does it mean the umbrella collection "ex" (=rare books)? In that case, some of them need correcting, e.g. "hsvm" (=manuscripts high security vault) : "Rare Books: South East (MSS)"

Thank you for changing the name @regineheberlein.
Your questions make a lot of sense. @kevinreiss @mzelesky who is the right person/group of people to answer these questions?

Per conversation with Alexis:

Change external name to human readable expanded string
Change internal name to code + abbreviated human readable string
remove all prefixes

I'll get started now.

Also pinged Joshua about the East Asian location labels (currently all "East Asian Rare")

@mzelesky @kevinreiss Can we delete the location "UNASSIGNED"?

@christinach What gets displayed in Blacklight, Name or External Name?

We cannot delete the location "UNASSIGNED"; it's used for migration purposes. At least, that's my understanding. If there is nothing in the locations after go-live we can delete. In general, we cannot delete any locations prior to go-live unless those locations are also deleted in Voyager.

@regineheberlein we use 'name', not the 'external_name'

@regineheberlein but we can change it to the 'external_name' if that makes more sense.

@christinach Yes please. There is a difference between what staff need to see in the UI and what we want patrons to see in Blacklight, so change the source to external_name would be great.

Now reaching out to Martin per Joshua

Martin confirms that the labels for EA rare, while all the same, are good as is.

authors: christinach mzelesky regineheberlein

repo: pulibrary/bibdata


text:
Bring coverage up to 90%

https://coveralls.io/builds/54801434

After merging a pulbot pr it went above 90%


text:
Regenerating catalogo item pages removed catalogo info

When I run bin/generate_item_pages, it updates the item pages, but the info from the Catalogo is removed from the top of the pages.

Fixed by 734c5bf61c9d10a8207977fcfec3e6af89dbaa32

repo: pulibrary/cicognara-static


text:
American Jewess MVW is not displaying in DPUL or online Catalog

the MVW is not coming up and only a black screen is visible for https://catalog.princeton.edu/catalog/9138143 and https://dpul.princeton.edu/wa/catalog/d217qt45s. Displaying fine in figgy
If I sign in, still a no go. cannot see anything.  I do not know if others are affected (and why).  Tested on Chrome, Firefox, and Safari. Reported by Steve Ferguson

This is because the first volume is marked private (https://figgy.princeton.edu/catalog/20c031a8-70fc-4d75-a30f-4438034f0e4f)

Changed to open just now

Works! Thank you!

repo: pulibrary/dpul


text:
Add unit testing to the provider

Fixes https://github.com/pulumi/ci-mgmt/issues/661
The reason I'm working on this now is that I'd like to really move upgrade tests out of examples and under provider/ in AWS (and soon GCP). https://github.com/pulumi/pulumi-aws/pull/2855

FYI: If you want to get tests working "right now", you can add the extra test to extraTests and run make ci-mgmt. We already have an existing extra test for pulumi-aws:
https://github.com/pulumi/pulumi-aws/blob/0b33b02036fcf1cae354aae98a9d278e957e0199/.ci-mgmt.yaml#L69-L94

This simpler version seems to work in:
https://github.com/pulumi/pulumi-azuread/actions/runs/6437059888/job/17481565040#step:10:37
By running tests after building as a separate step, this bypasses the need to model the dependency in Make. A bit awkward but the least evil quick option at the moment it seems like.  Locally, you just have to "know" you need to run tfgen before.
@iwahbe another look?

I think in the ideal state it really should not need tfgen. There is the issue with go:embed pointing to non-existing files; this is really a papercut, this can be worked around. Beyond that, why should testing the provider depend on tfgen?

repo: pulumi/ci-mgmt


text:
feat request: preview a staged hugo web stack in PRs

For doc and blog writers, it'd be great if we could see and share with others a preview of a staged hugo web stack when a PR is opened. This would help the reviewers and users see the changes being proposed from the PR itself without having to pull down the PR and run hugo locally.

This would indeed be very nice.
My ideal wishlist here is to have a Netlify-like experience, but using Pulumi:

Upon a new PR being opened

Stand up a fresh, new static website, being served out of an S3 bucket
Post back that URL using our GitHub App, as an output property of the Pulumi stack

Upon a PR being merged

Destroy and remove the stack

If we do this using S3 websites, we won't get great URLs, but it'll be fast.
Not only would this be a terrific flow, it would be a great real-world example of Pulumi in action.

For the immediate term, I was thinking we'd just hook up Netlify so we can have something to unblock people while we work on revamping our documentation.
Having a "Pulumi-powered Review App" would be the ideal end-result though. It should be very doable using GitHub Actions, I think it's just a matter of getting the workflow right. For reference https://github.com/pulumi/actions/issues/9 is the tracking this type of feature in that repo, and is linked to some other PRs that have been sent out around enabling that type of thing.

Christian created a solution for this internally. We'll still want to create a Pulumi stack that can provide this service, and using GitHub Actions would be an ideal compliment. But that effort is out of scope for the current milestone.

authors: chrsmith joeduffy metral

repo: pulumi/docs