text:
When disable Access Control after adding some new accounts, environment management behave strange.

Rancher Version:
v1.0.1
Docker Version:
1.10.3
OS and where are the hosts located? (cloud, bare metal, etc):
centos 7.2
Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)
single node rancher
Environment Type: (Cattle/Kubernetes/Swarm/Mesos)
Cattle
Steps to Reproduce:

Enable Local Authentication,and  add some accounts, also add new accounts to default environment.
Disable Local Authentication.
Edit default environment, change its Description,Click "Save"

Results:
Waring message prints:
"There are no active environments. You will need to create or activate one before you can do anything else."
All top-level menus(e.g. Application, Catalog) disappear.

I was unable to reproduce. Are you sure you didn't accidentally deactivate your environment, which is right next to the "Edit" on the drop down menu?

@deniseschannon

Enable  Local Authentication

Add some accounts

Add these accounts to Default environment

4.Disable Local Authenticatoin

Edit Default environment

Error appear

If I click "Swith to this environment" Icon, everything get right.  But once I edit Default environment, error happens again.

I am still unable to v1.1.0-dev3. I followed your steps exactly, but my environment doesn't change.
Can you try with the latest release?

@wangweihong Please re-open if you still see this issue.

authors: deniseschannon wangweihong


text:
'Timeout getting IP address' error on healthcheck and ipsec containers in v1.2.0

Rancher Version: 1.2.0
Docker Version: 1.12.1
OS and where are the hosts located? (cloud, bare metal, etc): cloud
Setup Details: single node rancher, external DB
Environment Type: Cattle
Upgraded to Rancher 1.2.0 today after seeing it was in general release.  One server has upgraded perfectly well but a second server is continually struggling to startup a 'healthcheck' container as well as an 'ipsec' container.  It also additionally fails to startup a customized 'logrotate' container that we use.  All three failing containers result in the same error:

Timeout getting IP address

Nothing particularly useful coming from the rancher-server logs:

2016-12-01 16:32:44,719 ERROR [7949843f-82d7-486d-a347-97224a8d0415:1411369] [instance:66195->instanceHostMap:29323] [instance.start->(InstanceStart)->instancehostmap.activate] [] [cutorService-29] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [compute.instance.activate.reply;agent=854]: Timeout getting IP address
2016-12-01 16:32:44,719 ERROR [7949843f-82d7-486d-a347-97224a8d0415:1411369] [instance:66195] [instance.start->(InstanceStart)] [] [cutorService-29] [i.c.p.process.instance.InstanceStart] Failed [1/2] to Starting for instance [66195]
2016-12-01 16:32:44,784 ERROR [7949843f-82d7-486d-a347-97224a8d0415:1411369] [instance:66195->instanceHostMap:29323] [instance.start->(InstanceStart)->instancehostmap.activate] [] [cutorService-29] [c.p.e.p.i.DefaultProcessInstanceImpl] Agent error for [compute.instance.activate.reply;agent=854]: Timeout getting IP address
2016-12-01 16:32:44,784 ERROR [7949843f-82d7-486d-a347-97224a8d0415:1411369] [instance:66195] [instance.start->(InstanceStart)] [] [cutorService-29] [i.c.p.process.instance.InstanceStart] Failed [2/2] to Starting for instance [66195]
2016-12-01 16:32:46,323 ERROR [:] [] [] [] [cutorService-15] [o.a.c.m.context.NoExceptionRunnable ] Expected state running but got stopped: Timeout getting IP address

Could you please advise on anything we can do to fix the issue or further diagnose?
Many thanks in advance.
NOTE: FOR THOSE RUNNING BOOT2DOCKER
If you are running network-services stack with plugin-manager:v0.2.12, please upgrade to the newest version of network services, there will be an "Upgrade Available" button to upgrade to network-manager:v0.2.13.
NOTE: v1.2.0 only works with the docker bridge as docker0 https://github.com/rancher/rancher/issues/6896 and docker must be installed in  /var/lib/docker https://github.com/rancher/rancher/issues/6897
We will be making it more flexible in the next release.
NOTE: If neither of the above solutions fix your issue
Can you please run the following commands with the CLI and share the files?
mkdir -p support
rancher hosts -a > support/hosts
rancher logs --tail=-1 ipsec/ipsec > support/ipsec 2>&1
rancher logs --tail=-1 network-services/metadata > support/metadata 2>&1
rancher logs --tail=-1 network-services/network-manager > support/network-manager 2>&1

Please be advised the files will have all your hosts and IP in them.

How many hosts do you have in the environement? Just those 2?

I am experiencing the same issue. I can not get the ipsec or scheduler container to start properly. I keep seeing the Timeout error mentioned above.
I am using boot2docker hosts with docker 1.12.1
Has anyone actually been able to get Rancher 1.2.0 to work correctly? I have re-installed everything from scratch and rebuilt my server with new database, removed all containers from hosts, etc...Basically a clean slate.

Same issue for me. It's happening on multiple servers and it brought down two of my sites. A general release should not bring downtime like this..

Exact same issues here, going to recreate all boot2docker hosts and redeploy containers since i have external NFS for persistence. My entire production platform is down right now, plus since i restarted all the boot2docker hosts it created duplicates for all 8 of my hosts. So now things are dancing all over the place starting/failing/starting/failing repeatedly. What a great "general availability" release here. Will report back with findings.

When I first upgraded a brand-new v1.1.4 single-node 2-host Rancher environment to v1.2.0, everything went mostly smoothly. All infra and user services were green.
Overnight, it all went to hell and started experiencing these issues. The following steps has led to all-green again (for now):

Using the Rancher UI, stop all services (just so Rancher stops trying to start them)
Stop Rancher Server with docker stop rancher-server or equivalent
On all hosts: docker rm -f {every_container_id}, including running and stopped containers
Start Rancher Server
Infra -> Hosts -> Add Host -> Custom -> Copy Command, and run it on all hosts
Wait for infra services to turn green
Slowly start user services

Will report back if things go south over time.

I prepared my migration path in a clean docker machine setup. I created one machine for the rancher-server (1.1.4) and another one for a simple demo stack with nginx:stable and HA Proxy.

rancher-server (Boot2Docker version 1.12.3)
rancher-node-01 (Boot2Docker version 1.12.3)

The infrastructure stacks healthcheck, ipsec and scheduler ends up with Expected state running but got error: Timeout getting IP address and endless Need to restart service reconcile.
Thats really frustrating for a stable release.

+1
Same situation here; all my environments are currently down since the upgrade also redeployed all my loadbalancers and this issue prevented them from ever coming online again!
Healthcheck and scheduler stacks are alternating between 'timeout getting IP address' and 'need to restart service reconcile'.
I'm down to all except one host agent deactivated, removing and re-installing the remaining agent doesn't seem to bring any difference.
Sometimes I can also read a NullPointerException in the rancher master logs; can't tell if that is related or not?
2016-12-01 20:22:40,643 ERROR [4cf7d31a-b187-481e-b056-b05ea2898b67:2190478] [service:410] [service.finishupgrade] [] [cutorService-11] [c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception java.lang.NullPointerException: null 	at io.cattle.platform.servicediscovery.process.ServiceFinishUpgrade$1.run(ServiceFinishUpgrade.java:39) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na] 	at io.cattle.platform.activity.ActivityService.run(ActivityService.java:43) ~[cattle-activity-log-0.5.0-SNAPSHOT.jar:na] 	at io.cattle.platform.servicediscovery.process.ServiceFinishUpgrade.handle(ServiceFinishUpgrade.java:33) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na] 	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:448) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na] 	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:399) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na] 	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:393) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na] 	at io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]

@jslatten No. I made a clean Rancher 1.2.0 install from scratch too, same issues.

I'd also like to report the same thing - I'm using Vagrant with multiple instances. Timeout on the IP address.
Completely clean install with new VMs, no legacy whatsoever, and just nominal packages installed. The agent containers (healthcheck, scheduler, etc.) are bouncing and restarting.

@kallqvist the exception you are seeing, is unrelated to ip address obtain failure

@alena1108 thanks! I removed that part from my comment to avoid creating confusion in here

It appears b2d does not work.  To confirm that this is your issue, run ln -s / /mnt/sda1 in the network-manager container.  After a bit the ipsec container will come up.  We will work on a fix for this.

For anybody else not running b2d, can you please run the following commands with the CLI and share the files.
mkdir -p support
rancher hosts -a > support/hosts
rancher logs --tail=-1 ipsec/ipsec > support/ipsec 2>&1
rancher logs --tail=-1 network-services/metadata > support/metadata 2>&1
rancher logs --tail=-1 network-services/network-manager > support/network-manager 2>&1

Please be advised the files will have all your hosts and IP in them.

@ibuildthecloud ln -s / /mnt/sda1 works! All infrastructure stacks comes online now.

I just recreated the full environment (120 hosts) and rancher is still crapping out on the infrastructure services (ipsec, network-services, scheduler) with a ton of errors like this:
2016-12-01 22:32:32,978 ERROR [902d8b92-a493-4a32-b12c-6ff1de043257:4900] [instance:346] [instance.start->(InstanceStart)] [] [cutorService-18] [i.c.p.process.instance.InstanceStart] Failed to Waiting for deployment unit instances to create for instance [346]
As of now, both my sites have been down for 7 hours, and clients are mad. This is NOT what a stable upgrade should look like. After I finish fixing this mess, I'll look for more stable and serious alternatives

@lucacri can you run the commands I shared.  You can email the files directly to me darren@rancher.com if you don't want to share with the world.

@ibuildthecloud I sent you an email

@ibuildthecloud
ipsec log turned out empty.
metadata is pretty much only logging this over and over again:
network-services-metadata-1     | time="2016-12-01T22:35:12Z" level=info msg="Downloaded in 543.187019ms"
network-services-metadata-1     | time="2016-12-01T22:35:12Z" level=info msg="Loading answers"
network-services-metadata-1     | time="2016-12-01T22:35:12Z" level=info msg="Loaded answers"
network-services-metadata-1     | time="2016-12-01T22:35:12Z" level=info msg="Applied https://ranchermasters.ops.example.com/v1/configcontent/metadata-answers?version=2649-eb849cc452eab2e95f5d30ac7decee95"

network-manager however is logging a bunch of errors
mainly about problems reading resolv.conf and
nsenter: cannot open /proc/6899/ns/ipc: No such file or directory
network-manager.txt

@cjellick yes, just two hosts as this is my test system prior to rolling out to our other systems.
Checking again this morning, we are still seeing the same issue.  One server working, one not.  They are both Azure D1_v2 servers and registered in exactly the same manner.  I've also tried removing the host from Rancher and re-registering as well as rebuilding from scratch but still no dice.

I can reliably repro this on a b2d cluster. Is b2d supported for local testing?

@samattridge can you follow the instructions that @ibuildthecloud for grabbing all the relevant logs and either posting them or emailing them over?

Same issue here, completely new setup: Ubuntu16.04 docker 1.12.1, running a agent on same host and it wont start anything, just puking the "Expected state running but got error: instance is not running : Dependencies readiness error" message

I'm also getting the error "Timeout getting IP address. Using 2 fresh Debian 8 VMs. Sent a mail to @ibuildthecloud with the log files.

Tested it again with Vagrant, VirtualBox and a Ubuntu 16.04 node and Docker 1.12.3.. All infrastructure stacks works fine.

same issue, ipsec log that might help:
12/2/2016 10:23:22 PMtime="2016-12-02T20:23:22Z" level=debug msg="About to start server and listen on port: :8111"
12/2/2016 10:23:22 PMtime="2016-12-02T20:23:22Z" level=debug msg="Reloading ..."
12/2/2016 10:23:22 PMtime="2016-12-02T20:23:22Z" level=info msg="Listening for ARP requests on eth0"
12/2/2016 10:23:22 PMtime="2016-12-02T20:23:22Z" level=info msg="Listening on :8111"
12/2/2016 10:23:22 PMtime="2016-12-02T20:23:22Z" level=error msg="couldn't get self container from metadata: Error 404 accessing /self/container path"
12/2/2016 10:23:22 PMtime="2016-12-02T20:23:22Z" level=error msg="couldn't reload the overlay: Error 404 accessing /self/container path"
12/2/2016 10:23:22 PMtime="2016-12-02T20:23:22Z" level=fatal msg="Error 404 accessing /self/container path"
12/2/2016 10:23:22 PM00[LIB] loading feature SIGNER:HMAC_SHA2_512_256 in plugi

@tbohnen the ln fix only solves the issue on boot2Docker

@tbohnen while Ubuntu 16.04 with all available Updates and Docker 1.12.3 works for me, i'm interessted in your kernel and Docker version.

I ran into this also. In my case, I had duplicate containers for the rancher/dns and rancher/metadata containers. See https://snag.gy/Uxr24e.jpg .
Some containers were named like network-services_metadata_dns_1 (with underscores), which were created by Rancher v1.1.4. Containers named like network-services-metadata-dns_1, with  dashes in their name but not underscores, were created by v1.2.0.
Once I deleted the containers created by Rancher v1.1.4 I was able to get Rancher working again. I also had to run docker ps -a |grep rancher | grep -v rancher/agent | while read CID junk; do docker rm $CID; done on each host, and rancher/agent created all of the Infrastructure containers again, and most of those images worked.

Anyone using boot2docker hosts must be running network services v0.0.2, which is running rancher/network-manager:v0.2.13. This will be the new default for new installs, but if you already have Rancher deployed and there is an "Upgrade Available" next to your "Network Services" stack, please upgrade to get your networking working.
For other people having issues, v1.2.0 only works with your docker bridge as docker0 [https://github.com/rancher/rancher/issues/6896] and docker being installed in var/lib/docker [https://github.com/rancher/rancher/issues/6897]. We are looking to make this more flexible in our next release.

In my 2 server setup I use only docker0 as bridge interface. On the rancher-server I used /data/docker instead of /var/lib/docker. I moved the folder to /var/lib/docker, restartet docker, updated the network-services stack to 0.0.2 and the ipsec stack is running. The healthcheck and one other load-balancer is still initializing.
On the second server the folder was since install of docker (month ago) in /var/lib/docker also with bridge docker0 and the ipsec and healthcheck stack still bring the "Timeout gettings IP address".

I'm running into this issue on a RancherOS set of nodes in AWS, which don't have the "rancher" command, so I'm not sure what I can do to provide the logs for that.

Thank you @deniseschannon .
My docker path was /home/docker-data use an existing partition. Your comment just fix my host connection (ipsec & healthcheck wasnt working). I'll follow #6897

As always, everyone should make sure that the IP of their host is not the docker bridge IP but the actual IP of the host. Whenever you add a host that is also running the Rancher server container, it will pick up the docker bridge IP instead.
http://docs.rancher.com/rancher/v1.2/en/faqs/agents/#how-does-the-host-determine-ip-address-and-how-can-i-change-it-what-do-i-do-if-the-ip-of-my-host-has-changed-due-to-reboot
@marvinwankersteen Can you double check that your IPs are set correctly since you are using one of your 2 servers to run Rancher server and Rancher agent commands?
@stormerider In AWS, have you made sure your security group is set up correctly? The rancher commands come from the Rancher CLI. It is not related to the specific OS that is running on your hosts.

Yeah, the SG is permissive enough ("All traffic" is allowed inside the SG for members of the SG), but I was using the internal IPs for the nodes instead of the public. I stopped the server, wiped all user containers (ie, anything from 'docker ps -a', but not touching anything from system-docker). Relaunched the server container (which uses an external RDS DB), deleted the hosts, and reconnected the hosts by recreating the agent containers, making sure to use the public IP instead. I did this host by host, letting the stacks spin up as well as they could before moving on to the next host. Now the healthchecker containers across all 3 nodes are still stuck in "Initializing" after several hours. The scheduler instance is also stuck similarly. And I see the hosts getting marked as disconnected frequently. I'm attaching the logs now that I installed the rancher cli tool (really unclear as to why that's missing from RancherOS, to be honest, that doesn't make sense to me...). I'm not sure if I'm dealing with the same issue as others at this point and I just got further down the process, or if this is a separate issue with 1.2.0 at this point:
rancher-support-20161204-0609.tar.gz

I got same problem as follows:
time="2016-12-04T15:10:17Z" level=info msg="Timeout waiting for ipsec to finish"
time="2016-12-04T15:10:17Z" level=info msg="Timeout processing stack:1st6"
time="2016-12-04T15:10:28Z" level=info msg="Finish Upgrade Stack Event Received" eventId=812100c0-c63d-421f-9380-03dd93e0db93 resourceId=1st6
time="2016-12-04T15:10:32Z" level=info msg="Timeout waiting for ipsec to finish"
time="2016-12-04T15:10:32Z" level=info msg="Timeout processing stack:1st6"
time="2016-12-04T15:12:46Z" level=info msg="Finish Upgrade Stack Event Received" eventId=61738e29-ab80-4a35-9c25-1fe87d968048 resourceId=1st6
time="2016-12-04T15:12:55Z" level=info msg="Timeout waiting for ipsec to finish"
time="2016-12-04T15:12:55Z" level=info msg="Timeout processing stack:1st6"
time="2016-12-04T15:13:02Z" level=info msg="Finish Upgrade Stack Event Received" eventId=61738e29-ab80-4a35-9c25-1fe87d968048 resourceId=1st6
2016-12-04 15:13:11,115 ERROR [d8d5ce87-6888-46ed-8c31-24146f97f8cf:5748] [instance:195] [instance.start->(InstanceStart)] [] [ecutorService-6] [i.c.p.process.instance.InstanceStart] Failed to Waiting for dependencies for instance [195]
time="2016-12-04T15:13:13Z" level=info msg="Timeout waiting for ipsec to finish"
time="2016-12-04T15:13:13Z" level=info msg="Timeout processing stack:1st6"
And the question is "can I increase timeout ?"

@cjellick I'll try and sort out the logs this morning. Something that may help you diagnose the issue is that I just read issue #6897 regarding changing docker location from the default of /var/lib/docker. On the machine that is seeing the error I have symbolic linked that location so we can run it on another drive without changing too much configuration.  I'm assuming this could be the reason why I'm seeing this issue in particular but may not be the answer for others.
Unfortunately, we don't have much option other than to change the location of docker since we're using Azure and want to use a secondary SSD backed drive for the containers.

@deniseschannon I double checked the ip address. On the rancher-server I added the host with the environment CATTLE_AGENT_IP and the right host ip address. Its not the bridge ip address.
The rancher-server server is running with all the container. The problem is the second server.
Maybe the problem is, that I have an openvpn to this server. So I have one public interface and one tun0 interface on the second server. I added the rancher-agent without the CATTLE_AGENT_IP. The server shows in Infrastructure --> Hosts the ip address of the tun0 interface. Perhaps this is wrong? Should it be the public address?
The second server reaches the rancher-server only over the openvpn.

@deniseschannon

As always, everyone should make sure that the IP of their host is not the docker bridge IP but the actual IP of the host. Whenever you add a host that is also running the Rancher server container, it will pick up the docker bridge IP instead.

I checked multiple times already with raw Debian/Ubuntu installations. Which means the bridge is already called docker0, the IP range doesn't overlap and the docker directory is in /var/lib. I also tried manually passing the IP of the host along when adding the new host, which didn't work either. The correct IP address of the host is also shown in the host overview tab. Also, setting up a host on the same machine will fail too.
Is there any other debug output that actually gives insight in what exactly is failing? "Timeout getting IP address" isn't really telling me anything so I can't check any further.

When I add a host in a new environment it takes an enormous amount of time before all services turn green. It's constantly restarting everything (and I mean everything except the proxy and kubelet services) becuase of "Timeout getting IP address". After like 10 minutes everything seems to be running finally.
After that I tried adding a deployment from the UI, which depployed a pod that kept being destroyed / removed / restarted, without ever starting a container with my actual application.
Before all that, I upgraded from 1.1.4 and my existing Kubernetes environment got stuck at 7 of 9 services, with all etcd services stuck in initialization + a ton of red dots on my screen.
Also very annoying, once my enviroment is up and running it redirects me to the add your first stack page now and then, randomly, which is btw a broken page, the Add from catalog button is broken..
In my opinion, Rancher 1.2 should not have been released as a stable release yet. We are talking about a piece of software that people use to manage their infrastructure. I'm running a very simple environment with only 3 hosts and no more than 20 containers including k8s stack, yet it got completely wrecked by this update.
To anyone that hasn't upgraded yet, I would advice to wait for a more stable release.

@samattridge A workaround I found to solve the issue related to /var/lib/docker is to use a mount point instead of a symlink.
For example:
stop docker service
rm -f /var/lib/docker && mkdir -p /var/lib/docker
echo "</path/src>        /var/lib/docker         auto    rw,bind        0       0" >> /etc/fstab
mout -a
start docker service
Hope that helps !

Thanks @swartel-fa for taking the time to reply.  Much appreciated!  I'll give that a go.

@swartel-fa Thanks for the reply. It worked perfectly in my case

@mkurzeja you welcome

Just tried VXLAN - same issue

Same issue with me. I've tried with RancherOS v0.7.1, RancherOS v0.7.0 and Ubuntu 16.04. Docker versions 1.12.x

I was able to find the problem and solve it. Since Rancher keeps reporting it can't get some IP address I started sniffing all DNS queries on my DNS server and found the following:
Dec 9 19:13:46 	unbound 	51000:0 	info: 192.168.1.xxx rancher-metadata.example.org. A IN
Dec 9 19:13:42 	unbound 	51000:0 	info: 192.168.1.xxx rancher-metadata.example.org. A IN

(where example.org is my internal domain.)
Obviously this isn't supposed to happen, *.example.org will always resolve to my public IP unless it is overwritten by the local DNS server. I never bothered changing this since I didn't think it would cause any issues.
Anyways, *.example.org will not resolve on my internal network anymore unless it is defined internally. This fixed the issue. Rancher will still issue the useless DNS request but won't get a valid response anymore.
I still think this probably should be fixed? I doesn't look like this DNS lookup should reach the network.

@stormerider I finally had a chance to look at your logs. Your issue seems different as it looks as if your go-machine-service is flapping in your server. Could you open a new issue for that?
@marvinwankersteen The IP address of the the server running rancher/agent should use the public IP address unless the rancher/server server can access the tun0 IP.
@hawkwang There is no way to increase the timeout. @DomiStyle There is nothing else distinctive evidence except for providing us logs.
@hawkwang  If you use the Rancher CLI, you can provide these logs. Note: this will share the IP address of your hosts.
mkdir -p support
rancher hosts -a > support/hosts
rancher logs --tail=-1 ipsec/ipsec > support/ipsec 2>&1
rancher logs --tail=-1 network-services/metadata > support/metadata 2>&1
rancher logs --tail=-1 network-services/network-manager > support/network-manager 2>&1

@DomiStyle I'll forward your latest update to our team to see what's going on.
@T-Grave I've seen the k8s issue of the UI before. If you manually refresh the page, it'll start working. I haven't been able to pinpoint what are the exact steps to get in that loop where you can't access the catalog.

are you enabling ICMP between servers? maybe thats it?

@swartel-fa
i dont know
echo "</path/src>        /var/lib/docker         auto    rw,bind        0       0" >> /etc/fstab
what is </path/src>

@IrisDande the path to your Docker working directory, e.g., /data/docker.

@VincenzoFerme
But what is correct path if docker is installed from repo by default?

I that is the case, then I think this workaround does not apply to you, because the directory should already be the default one: /var/lib/docker

@VincenzoFerme  did you have any solution in that case? i can't install old version of rancher (old version did meet this error)

Can you report all the information suggested on the following issue template: https://github.com/rancher/rancher/blob/master/.github/ISSUE_TEMPLATE.md?

@VincenzoFerme  Sorry but my issue  have same with thread ( no different ) so i dont want to add a duplicate issue  :(

I mean, you can report the requested information in this issue, if they are different than the one discussed in the first comment of this issue: https://github.com/rancher/rancher/issues/6858#issue-192896871

Rancher Version: 1.2.0
Docker Version: 1.12.3, build 6b644ec
OS and where are the hosts located? (cloud, bare metal, etc): Debian 8 on VMware ESXI
Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB): single node rancher, internal DB
Environment Type: (Cattle/Kubernetes/Swarm/Mesos): Cattle
Steps to Reproduce:

Install docker on debian 8 (or ubuntu 16.04)
Install Rancher/server on this server
Install Rancher/Agent:v1.1.0 on this server

Results: Expected state running but got error: Timeout getting IP address
Expected:

This issue kept several of our bare-metal Ubuntu 16.04 servers off network for over a week.
A work-around for me was:

Remove any reference to rancher (kill and remove containers), and umount /var/lib/rancher/volumes
rm -rf /var/lib/rancher
apt-get purge docker-engine
rm -rf /var/lib/docker
apt-get install docker-engine
install rancher-agent (however you choose).

.

Same here.  Fresh install ESXi 6.0, Ubuntu 16.04, latest everything else following the Quick Start Guide.
What sort of attention is this getting at Rancher?

@marrotte There are multiple reasons why this error could have occurred as we have outlined in this issue.
If you have boot2docker hosts, you need to make sure you upgrade to the latest network services.
There are known issues with if your docker is installed in a different location or if your docker bridge is not at docker0.
We are working on 1.2.1 that will address the above issues. For all other cases, we've asked for users to use the Rancher CLI to provide us with the following information so we can help debug:
mkdir -p support
rancher hosts -a > support/hosts
rancher logs --tail=-1 ipsec/ipsec > support/ipsec 2>&1
rancher logs --tail=-1 network-services/metadata > support/metadata 2>&1
rancher logs --tail=-1 network-services/network-manager > support/network-manager 2>&1

Note: This would show the IP of your hosts.

After retrying the apt-get purge docker-engine  solution, it is failing again.
Where do we send the logs?

@kkvilekval You can send them to darren@rancher.com and/or me (denise@rancher.com)

We have the same problem here after upgrading to Rancher 1.2.0 with a rancher server on a current rancher os release and several clients running ubuntu 16.04. No boot2docker, no non-standard docker-root. We deleted all docker containers on the agent and switched to 1.2.1-rc2 (using agent version 1.1.1-rc1) but it didn't help so far. We will downgrade to 1.1 now.

Me too, have similar setup to  @IrisDande .
Thanks @deniseschannon - I'll send you and Darren my log output

+1
I am also experiencing same error. 'Timeout getting IP address' error on healthcheck and ipsec  containers and it is keep restarting. But my own stacks running fine and i am not sure about the impacts due to above failed containers. . So I have reverted the upgrade to previous version(1.1.4).
Can we expect the fix quickly for this issue?

I forgot to add: Our apllication stacks were looking like they'd be running fine but then we saw that the containers couldn't connect to each other: Even the ones running within the same stack on the same host. And this was not a DNS issue - we couldn't ping the other containers even using their IP addresses. Unfortunately I couldn't find out whether this was a consequence or the (same) reason of the degraded infrastructure stack.

I'm seeing this as well,
Ubuntu 12.04 with Docker 1.12.3 and on AWS.
All the hosts can talk to each other just fine, just sometimes the it can't get the IP.  I've built in re-tires to our deploy scripts but it would be nice to not have to do that.  Also
@deniseschannon  or @ibuildthecloud  Can you provide me what logs you need and I can not tell from from all the comments.

@jwhitcraft
mkdir -p support
rancher hosts -a > support/hosts
rancher logs --tail=-1 ipsec/ipsec > support/ipsec 2>&1
rancher logs --tail=-1 network-services/metadata > support/metadata 2>&1
rancher logs --tail=-1 network-services/network-manager > support/network-manager 2>&1

Just want to add I have the same issues and that the latest version 1.2.1 version does not solve it for me. I tried it with various servers on Digital Ocean (Ubuntu, CoreOS etc.) and tried the mentioned hints above. (Sent my logs too). Did not work for me and I had no choice but to downgrade to 1.4.1.
1.2 is really not a resilient/stable version for me.

Hi,
healthcheck and ipsec containers came up for me after upgrading it to 1.2.1. But none of my services are functional. Name resolution is working but getting connection time out from one service to other.
For ex: service1 connecting to Mongo service on same stack using service link/alias.  I am able resolve the hostname of mongo but connection is timing out. Mongo is up and i am able to connect directly.
WARNING: Server seen down: DB-MONGO-1/198.105.244.11:27017 - java.io.IOException - message: connect timed out
I had to revert back to 1.1.4 without any choice.

I created a brand new Rancher 1.2 and 1.2.1 setup and am unable to get it the infrastructure services deployed.   I emailed my logs.  Its a shame that 1.2 doesn't seem to work easily out of the box.  Just to detail what I am using
Rancher Version:
v1.2.0-1 and v1.2.1
Docker Version:
1.12.4
OS and where are the hosts located? (cloud, bare metal, etc):
Ubuntu 16.04 AWS
Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)
Single node external db and HA Rancher
Environment Type: (Cattle/Kubernetes/Swarm/Mesos)
Cattle
Steps to Reproduce:
Create brand new HA Rancher cluster.  Add a host to an environment.
Results:
All infrastructure services except metadata and network manager fail to start.

There are an incredible amount of issues with Rancher 1.2.1 we've burned a huge amount of time dealing with this.  It is clearly not ready for production.  You guys need to revert the :latest tag to something stable.  You are screwing more and more people over.

We failed in a rouge upgrade to Rancher 1.2.x (We didnt pin the version of our 'rancher-server' and an EC2 instance rebooted and force upgraded us).
Our topology is
[Rancher server (HA set with RDS DB)]  <--->   [N number of instances that add specific labels that register with Rancher Server]

We are hosted in AWS and we have tight security groups between our instances and the rancher server. We block ICMP by default, but enable HTTP/ws to Rancher Server and UDP ports for health checks in Rancher 1.1
I have a feeling that the new upgrade and new network containers that are launched as a result need ICMP. I wasnt able to confirm as we had to abandon trying to force upgrade.

@tnh that's interesting as it matches our setup exactly, where it failed as well: rancher server on aws, strict firewalling with no icmp, external DB on rds. And we use an ELB as our rancher endpoint, forwarding the TCP traffic to the rancher servers using an ELB proxy policy to make websockets work correctly. Hope this helps others or the rancher dev team to find parallels causing the issue. Or is the reason known already at the rancher team?

Hi Team,
Any update? Is 1.2.2 has fix for this issue?

Hi, I found a solution to this problem that works on ubuntu server, and desktop as well.
The cause is that ubuntu ships by default using dns server as local address 127.0.0.1 and is by design. Problem is docker containers can't lookup in 127.0.0.1
To diagnose you can do the follow procedure:
See your /etc/resolv.conf file
Perform a test:
docker run --it ubuntu bash
apt update
apt install dnsutils
# This will not respond
dig @127.0.0.1 your.hostname.com

Note: ping WILL work fine, and could trick you in thinking that name resolution is working. Dig is a proper way of doing that.
You can also use rancher cli to get a hint of the problem:
mkdir -p support
rancher hosts -a > support/hosts
rancher logs --tail=-1 ipsec/ipsec > support/ipsec 2>&1
rancher logs --tail=-1 network-services/metadata > support/metadata 2>&1
rancher logs --tail=-1 network-services/network-manager > support/network-manager 2>&1

The error will be like network-services-metadata-1 | time="2016-12-26T23:58:21Z" level=fatal msg="Failed to subscribeGet http://xx.xxx.xxx:xxx/v2-beta: dial tcp: lookup xxx.xxx.xxx.xxx on 127.0.0.1:53: read udp 127.0.0.1:44285->127.0.0.1:53: read: connection refused"
There are two solutions:
1 - Configure ubuntu to use other nameserver like google public dns (8.8.8.8, 8.8.4.4). I try this one, and is by far too complicated for a simple change, as a said, ubuntu use that by design.
2 - Change docker dns server. This worked fine for me. You will edit or create the file /etc/docker/daemon.json and put the line:
{
  "dns": ["8.8.8.8", "8.8.4.4"]
}

Stop the containers and Restart the daemon:
docker stop $(docker ps -q)
docker stop $(docker ps -q) # yes twice :-) rancher will try do restart your dying containers
systemctl restart docker

Oh boy, after days in this problem, I will drink a could beer :-)
Thank you @deniseschannon, your procedure of using rancher cli to generate logs was what give me the hint necessary to find the cause.
Question: Is there some problem in using a external DNS, with rancher ?
And finally, for the ones who want the solution 1, I let some links to get started (none of the solutions worked for me :-(
http://askubuntu.com/questions/143819/how-do-i-configure-my-static-dns-in-interfaces
http://askubuntu.com/questions/327532/why-would-127-0-0-1-in-resolv-conf-cause-problems-in-dns-resolution
http://askubuntu.com/questions/627899/nameserver-127-0-1-1-in-resolv-conf-wont-go-away

@giovannicandido
Thanks for the solution but it doesn't fix the problem i'm seeing.
After bringing a new host online, it starts the network-services-network-metadata but this is what i get in the logs
time="2016-12-27T03:01:48Z" level=info msg="Waiting for metadata"
Creating metadata client: Get http://rancher-metadata/2016-07-29/version: dial tcp: lookup rancher-metadata on 169.254.169.250:53: read udp 10.42.0.1:35338->169.254.169.250:53: i/o timeout
time="2016-12-27T03:04:01Z" level=info msg="Waiting for metadata"
Creating metadata client: Get http://rancher-metadata/2016-07-29/version: dial tcp: lookup rancher-metadata on 169.254.169.250:53: read udp 10.42.0.1:34519->169.254.169.250:53: i/o timeout
time="2016-12-27T03:06:13Z" level=info msg="Waiting for metadata"
Creating metadata client: Get http://rancher-metadata/2016-07-29/version: dial tcp: lookup rancher-metadata on 169.254.169.250:53: read udp 10.42.0.1:56142->169.254.169.250:53: i/o timeout

And even on my second host i bring up, i still get the the ip timeout errors.  I'm about to scrap this whole 1.2.2. environment and go back to 1.1.4 till they can get this figured out.

@jwhitcraft Can you just double check the output of ip route on the host and additionally, are you running with the default docker0 bridge?  I'm looking through the last set of info you sent me now.

@ibuildthecloud, sadly, I've already deleted that environment.  I'm in the process of testing out an environment with a vxlan instead of ipsec and so far it's looking good.  I've added 4 hosts now and they've all come up the first time with no issues 🤞 that it keeps up.
If i run into it again i'll let you know.

@jwhitcraft There's no fundamental difference between IPsec and vxlan in how it is managed.  So if you repeatably had issue with IPsec, I would not expect VXLAN to be better.
You had previously sent a zip with two servers.  Do you know if the resolv.conf in the zip was from the  metadata-dns container?  The container running the rancher-dns process?

@ibuildthecloud,
Yes they came from the sidekick containers which are the metadata-dns servers.

@jwhitcraft Looking at your answers.json I can see that rancher-dns is in fact recursing to itself "169.254.169.250".  That is basically what is causing your issue.  Now why it's configured as such, no clue.  Let me come up with some theories.

@ibuildthecloud, the hosts are on Ubuntu 16.04 and 14.04 fully patched with docker 1.12.3 or newer.  Tonight i upgraded them all to 1.12.5 before trying the vxlan tests.

@jwhitcraft You don't by chance have either 169.254.169.250 set in the host /etc/resolv.conf or have --dns 169.254.169.250 set on the Docker daemon?

My resolv.conf points to my companies internal dns servers and the docker daemon doesn't anything extra set on it, just a stock apt-get install docker-engine was run.

@jwhitcraft Do you have any broken hosts left?  We run rancher-dns as --net container:metadata and then metadata runs --net bridge.  For whatever reason the resolv.conf in metadata is getting nameserver 169.254.169.250 set.  If you happen to have a broken host if you do a docker inspect on the r-network-services-metadata-XXXXX you should see the HostConfig DNS set as follows:
"Dns": null,
"DnsOptions": null,
"DnsSearch": [
    "network-services.rancher.internal",
    "metadata.network-services.rancher.internal"
],

So basically we tell Docker to just use the default DNS server, which would be what is in /etc/resolv.conf on the host.
So I only have two theories right now.

Docker was configured as 169.254.169.250 as the DNS:  I'm guess this is no because you said it was a default setup
Some how Rancher managed to edit the hosts /etc/resolv.conf.  The only way I could imagine that is if maybe --net host -l io.rancher.container.dns=true has a bad behavior in docker 1.12 and we end up actually changing the host DNS.  I'm testing this now.

@jwhitcraft It's got to be option two.  Do you ever run a container with -v /etc/resolv.conf:/etc/resolv.conf That would cause it.

Naw, i'll email you the compose files that we use, only the image name and variables change in it.  We don't mount anything to the hosts.

@jwhitcraft Thanks, I also realize that while -v /etc/resolv.conf:/etc/resolv.conf has a really bad behavior in Docker 1.12, it won't exactly cause the issue you are seeing.

I can confirm that adding external DNS as @giovannicandido suggested, fixed the issue in my case. Thanks a lot, man!

2 - Change docker dns server. This worked fine for me. You will edit or create the file /etc/docker/daemon.json and put the line:
{
"dns": ["8.8.8.8", "8.8.4.4"]
}

Update
A lot of people are experiencing issues with Rancher w/ a setup where a caching DNS resolver is used on the host. @giovannicandido pointed out is the case with Ubuntu.  The root cause of this is basically what I believe is a bug in Docker in which Docker does not set the DNS nameserver properly when only the DnsSearch option is used.  For futher reference I've created this bug in Docker https://github.com/docker/docker/issues/29815
In Rancher 1.3 (due out this week) we are putting in work around to deal with this behavior in Docker.  For Rancher 1.2.x the best approach is to set --dns on the docker daemon command or set the below in /etc/docker/daemon.json:
{
  "dns": ["8.8.8.8", "8.8.4.4"]
}

Sorry for the troubles here.  We do test quite extensively on Ubuntu but not with a configuration that has a local dnsmasq resolver and as such we didn't catch this one.

Thx again @ibuildthecloud

I have did some debugging for the dns-issue.
On the last host I have problems I get this error-message:

Creating metadata client: Get http://rancher-metadata/2016-07-29/version: dial tcp: lookup rancher-metadata on 169.254.169.250:53: read udp 172.17.0.1:42662->169.254.169.250:53: i/o timeout

If I try to do a ping rancher-metadata.rancher.internal in the container network-services-network-manager-4 the dns-name is not resolved.
So I edited the /etc/hosts and added 169.254.169.250 rancher-metadata.rancher.internal.
Now I get the ip IN the container. Then I tried to do the curl http://rancher-metadata/2016-07-29/version.
But this is not possible. It cant reach the IP 169.254.169.250 with HTTP Port. I have no idea what exactly is the problem.
I have some custom iptables rules. But I set for every chain the policy to ACCEPT and also inputed an -j ACCEPT on position 1 in INPUT, FORWARD and OUTPUT.

In my case, my network-services/metadata got the correct resolv.conf settings. But network-manager got the "incorrect" resolv.conf. It doesn't make sense.
I tried the --dns and daemon.json but they don't work for me.

Turns out that I added proxy settings when adding the host. Removing the proxy settings resolved my issue.

@marvinwankersteen on your agent hosts do you have a dns caching service running, like nscd?

Several problems here too.
__Only with Ubuntu 16.04 __. Ubuntu 14.04 and Debian 8.5 works correctly.
Fixed one ubuntu 16 with these steps:
-removing host and stacks
-create new environment
-service docker stop
-delete dns=dnsmasq from /etc/NetworkManager/NetworkManager.conf
-service network-manager restart
-service docker start
-adding host to the new environment
But... I can do this on 20 machines. I'm restoring 1.1.4 atm.
btw... same issue upgrading with 1.2.1, 1.2.2 and 1.3.0

@aemneina No, nothing like that.
But I found the problem on my server. The problem was the iptables-rules. The rules were to strict. I just allowed traffic to specific ports in the FORWARD-chain. But not between the containers.
The node is reachable from external and I want to set some iptables-rules. But it where to strict. So I edited my rules and now it works. Which rule exactly was the problem I have no idea.
@Rancher-Team: It would be nice if you can tell us which rules MUST exist that the containers can communicate among each other. Or add them automatic.

After upgrading to 1.3.0 I'm still seeing the Timeout Getting IP Address on one of the two Rancher environments I had set up previously. Interestingly, one of the environments works, but the other one won't let me start any stacks/services because of that error. Even the Rancher Infrastructure stacks (like scheduler, healthcheck, etc) aren't able to be started or upgraded.
When I docker inspect the network-services-metadata-### containers in both environments the DNS records all look the same:
"dnsSearch": [
  "network-services.rancher.internal",
  "metadata.network-services.rancher.internal"
]

The "broken" environment consists of two hosts, one running RancherOS 0.6.1 (docker version 1.12.1) and the other runs Ubuntu 14.04 (docker version 1.10.3). No changes to the system resolv.conf or DNS settings. The timeout error happens on both hosts. Here's a likely relevant line from the network-manager-X logs from a host when trying to bring up a stack.
network-services-network-manager-1 | time="2017-01-08T18:13:44Z" level=error msg="Failed to evaluate network state for 6f43c8282c1e8934b5d78e88703e591cf5c5a2c4d48a9c4f02db003eddc82137: Bringing up networking: No IP address found"
network-services-network-manager-1 | time="2017-01-08T18:13:44Z" level=error msg="Error processing event &docker.APIEvents{Action:\"start\", Type:\"container\", Actor:docker.APIActor{ID:\"6f43c8282c1e8934b5d78e88703e591cf5c5a2c4d48a9c4f02db003eddc82137\", Attributes:map[string]string{\"image\":\"rancher/scheduler:v0.5.0\", \"io.rancher.container.agent_id\":\"34223\", \"io.rancher.container.name\":\"scheduler-scheduler-1\", \"io.rancher.service.hash\":\"fac0a074c6689097fbbfb12e410bab30c3d07941\", \"io.rancher.cni.network\":\"ipsec\", \"io.rancher.project.name\":\"scheduler\", \"io.rancher.container.ip\":\"10.42.99.87/16\", \"io.rancher.service.deployment.unit\":\"713893f6-c035-4f45-8c81-21d62299cfb4\", \"io.rancher.service.launch.config\":\"io.rancher.service.primary.launch.config\", \"io.rancher.stack_service.name\":\"scheduler/scheduler\", \"io.rancher.cni.wait\":\"true\", \"io.rancher.container.agent_service.scheduling\":\"true\", \"io.rancher.container.create_agent\":\"true\", \"io.rancher.container.uuid\":\"20449261-40dc-4acb-9fb8-13785771bf1f\", \"io.rancher.project_service.name\":\"scheduler/scheduler\", \"io.rancher.stack.name\":\"scheduler\", \"name\":\"r-scheduler-scheduler-1-20449261\"}}, Status:\"start\", ID:\"6f43c8282c1e8934b5d78e88703e591cf5c5a2c4d48a9c4f02db003eddc82137\", From:\"rancher/scheduler:v0.5.0\", Time:1483899103, TimeNano:1483899103900821540}. Error: Bringing up networking: No IP address found"
network-services-network-manager-1 | time="2017-01-08T18:13:46Z" level=info msg="Evaluating state from retry" cid=6f43c8282c1e8934b5d78e88703e591cf5c5a2c4d48a9c4f02db003eddc82137

This environment worked with Rancher 1.1.4. Anything I can test? I figure I may be in a unique and potentially interesting situation by having one environment that works and one that does not.

I tested with clean db (because #7245) and no errors happens

@JaviSabalete That is what I had to do, my old rancher install was so unreliable that I just started over on 1.3.0 and it's been humming along ever since.  I send my database dump to them to see if they could find anything.

@mattdodge try stopping the scheduler and see if your network-services reconcile before starting the scheduler again. This may be related to
https://github.com/rancher/rancher/issues/7375

There have been many releases of our infrastructure services to help address the issues relating to timeouts. I will be closing this issue for now so that it's easier to track any new issues with the same message.
As we've tried to state before, there have been many reasons why the stacks might get into this state and we've tried to isolate the issues as they've come in and push out updates as quickly as possible.
Please try again with the latest version v1.3.0 and the default infra stacks for that version. If you're upgrading, please read the Release Notes regarding upgrading.
https://github.com/rancher/rancher/releases/tag/v1.3.0

Stopping the scheduler didn't resolve the issue for me. I even upgraded network-services while it was stopped and then restarted the scheduler. At that point the scheduler wouldn't even start, it got stuck in "Initializing" state. There are plenty of other issues relating to this so no need to go into it here.
It looks as though I should just start over from scratch, something in my set up is clearly hosed. I'd been meaning to clean up our AWS and Rancher environments anyways, I guess this is as good an excuse as any to make it happen.

After still having the 'Timeout getting IP address' issue on some hosts on Rancher 1.3.0 I found another workaround I'd like to share. The /etc/resolv.conf inside docker not only inherits the host system's dns settings but also its search host(s) which as a result can lead to the same problem.
I was able to fix that by overwriting the dns-search hosts adding a new line to the suggested entry in /etc/docker/daemon.json (replace example.org with a proper search domain in your network)
{   "dns": ["8.8.8.8", "8.8.4.4"],   "dns-search": ["example.org"] }
Please note that dns-search may neither be empty (the hosts' search domains would still be inherited) nor contain an empty string (your docker service wouldn't start anymore). Hope this helps some of you guys out there.

@DomiStyle I've been having the exact same problem with rancher 1.3.0 and 1.3.3, I finally tried your approach and started looking at DNS traffic with tcp dump and my failing host did indeed look up rancher-metadata in all the search domains of the host, until it got to the public company domain which has a * record that resolves to an external website.
My solution was to remove the internet-only domain from /etc/resolv.conf and that caused ipsec to start, now I just have healthcheck stuck in initializing.
I really wish the internal names looked up by services such as ipsec will be changed to use the fqdn ending in .internal. so we can avoid both spamming the external DNS with useless requests and this problem in particular.

I've been seeing the "Timeout getting IP address" issue off and on, some containers schedule, some don't. After looking at the network-services/metadata logs, I saw that about half of them had the following entries near the end (IPs/hostnames changed):
network-services-metadata-9     | time="2017-01-27T14:08:56Z" level=info msg="Loading answers"
network-services-metadata-9     | time="2017-01-27T14:08:56Z" level=info msg="Loaded answers"
network-services-metadata-9     | time="2017-01-27T14:08:56Z" level=info msg="Applied https://rancher.acme.com/v1/configcontent/metadata-answers?version=15755-8b84aebc80dd7a28cb0d0e082d379dd7"
network-services-metadata-9     | time="2017-01-27T14:08:56Z" level=info msg="Downloaded in 230.423534ms"
network-services-metadata-9     | time="2017-01-27T14:08:56Z" level=info msg="Loading answers"
network-services-metadata-9     | time="2017-01-27T14:08:56Z" level=info msg="Loaded answers"
network-services-metadata-9     | time="2017-01-27T14:08:56Z" level=info msg="Applied https://rancher.acme.com/v1/configcontent/metadata-answers?version=15755-8b84aebc80dd7a28cb0d0e082d379dd7"
network-services-metadata-9     | time="2017-01-27T15:33:03Z" level=info msg="Hit websocket pong timeout. Last websocket ping received at 2017-01-27 15:32:50.015981928 +0000 UTC. Closing connection."
network-services-metadata-9     | time="2017-01-27T15:33:03Z" level=warning msg="websocket closed: websocket: close sent"
network-services-metadata-9     | time="2017-01-27T15:33:04Z" level=warning msg="websocket closed: websocket: close sent"
network-services-metadata-9     | time="2017-01-27T15:33:04Z" level=info msg="Initializing event router" workerCount=3
network-services-metadata-9     | time="2017-01-27T15:33:05Z" level=warning msg="websocket closed: websocket: close sent"
network-services-metadata-9     | time="2017-01-27T15:33:34Z" level=error msg="Failed to download and reload metadata: Get https://rancher.acme.com/v1/configcontent/metadata-answers: dial tcp 10.x.x.x:443: i/o timeout"

After the websocket closes, it doesn't appear to reopen on it's own (it's less alarming to me why the websocket closed). I restarted all the metadata containers that had closed the connection and the containers I saw receiving the "Timeout getting IP address" error finally started.

what's disturbing me is the download time:
ime="2017-01-31T18:38:06Z" level=info msg="Applied https://xxx/v1/configcontent/metadata-answers?version=127595-8b84aebc80dd7a28cb0d0e082d379dd7"
time="2017-01-31T18:38:25Z" level=info msg="Downloaded in 18.454818263s"
it takes up to 24 secs to download, I have 27 servers running the rancher DB currently gets up to 3000 req/s and has a 50% load (RDS aurora r3.large - that's crazy)

Theory:
it takes around one minute download and apply the metadata. So in some cases it can take over 2 minutes until an IP is visible. This runs into the 120 sec timeout of the agent.
Some logs:
31.1.2017 20:18:40time="2017-01-31T19:18:40Z" level=info msg="Applied https://xxx/v1/configcontent/metadata-answers?version=44878-8b84aebc80dd7a28cb0d0e082d379dd7" 31.1.2017 20:19:00time="2017-01-31T19:19:00Z" level=info msg="Downloaded in 19.787896964s" 31.1.2017 20:19:20time="2017-01-31T19:19:20Z" level=info msg="Loading answers" 31.1.2017 20:19:21time="2017-01-31T19:19:21Z" level=info msg="Loaded answers" 31.1.2017 20:19:21time="2017-01-31T19:19:21Z" level=info msg="Applied https://xxx/v1/configcontent/metadata-answers?version=44922-8b84aebc80dd7a28cb0d0e082d379dd7" 31.1.2017 20:19:45time="2017-01-31T19:19:45Z" level=info msg="Downloaded in 23.568940433s" 31.1.2017 20:20:02time="2017-01-31T19:20:02Z" level=info msg="Loading answers" 31.1.2017 20:20:03time="2017-01-31T19:20:03Z" level=info msg="Loaded answers" 31.1.2017 20:20:03time="2017-01-31T19:20:03Z" level=info msg="Applied https://xxx/v1/configcontent/metadata-answers?version=44958-8b84aebc80dd7a28cb0d0e082d379dd7" 31.1.2017 20:20:22time="2017-01-31T19:20:22Z" level=info msg="Downloaded in 18.835192201s" 31.1.2017 20:20:42time="2017-01-31T19:20:42Z" level=info msg="Loading answers"

This has happened to me once in two days for last couple of weeks. Restart of the node helps, but i found out that restart of network-manager and dns helps also

@pschrammel @miv I think we're going off topic here but just in case it helps you: We had similrar issues with high load/traffic and found out that there were stacks containing flapping services that were stopping/starting all the time. Although that were only a few containers that produced a majority of the load and traffic we were experiencing. After cleaning that up the load on our rancher server is minimal.

We're still seeing this issue with 1.3.4.

Theory:
it takes around one minute download and apply the metadata. So in some cases it can take over 2 minutes until an IP is visible. This runs into the 120 sec timeout of the agent.
Some logs:
31.1.2017 20:18:40time="2017-01-31T19:18:40Z" level=info msg="Applied https://xxx/v1/configcontent/metadata-answers?version=44878-8b84aebc80dd7a28cb0d0e082d379dd7" 31.1.2017 20:19:00time="2017-01-31T19:19:00Z" level=info msg="Downloaded in 19.787896964s" 31.1.2017 20:19:20time="2017-01-31T19:19:20Z" level=info msg="Loading answers" 31.1.2017 20:19:21time="2017-01-31T19:19:21Z" level=info msg="Loaded answers" 31.1.2017 20:19:21time="2017-01-31T19:19:21Z" level=info msg="Applied https://xxx/v1/configcontent/metadata-answers?version=44922-8b84aebc80dd7a28cb0d0e082d379dd7" 31.1.2017 20:19:45time="2017-01-31T19:19:45Z" level=info msg="Downloaded in 23.568940433s" 31.1.2017 20:20:02time="2017-01-31T19:20:02Z" level=info msg="Loading answers" 31.1.2017 20:20:03time="2017-01-31T19:20:03Z" level=info msg="Loaded answers" 31.1.2017 20:20:03time="2017-01-31T19:20:03Z" level=info msg="Applied https://xxx/v1/configcontent/metadata-answers?version=44958-8b84aebc80dd7a28cb0d0e082d379dd7" 31.1.2017 20:20:22time="2017-01-31T19:20:22Z" level=info msg="Downloaded in 18.835192201s" 31.1.2017 20:20:42time="2017-01-31T19:20:42Z" level=info msg="Loading answers"

One of our sys-admins tracked this down to wildcard DNS configurations on our servers. Maybe that helps someone here.

Still happens with rancher/server: 1.5.2 and rancher/agent: 1.2.1.
It is something related to the environment (as in cattle environment) creating a new environment and adding hosts to it works just fine, and there's more, once the host has been added, it can be deleted from the fresh environment and it works just fine in the 'broken' environment.
I reproduced quite a few times, getting snapshots of the database in the process

What the f...g problem. It still happens every time after system upgrade or rancher upgrade. Sometimes it happens after reboot host. I use rancher from beta version... it was 0.6 and it is best open-source UI GUI for docker, kubernetes and other services for DevOps. But why you, rancher team, can not to resolve this problem with your Cattle Stack. I don't want to remove and recreating on all hosts new environment every time, after my home-server rebooting. I will have good sleeping and don't think about possible situations or pray to Digital God for success starting services and containers after electricity interruptions.
Can you research this problem and describe all possible variants of guides how to issue this global problem! I hope that you'l decide to do it right.

Please check if you hit this problem:
https://github.com/rancher/rancher/issues/9367#issuecomment-315893457
I had the same problem and could solved it.

Also check the https://github.com/rancher/rancher/issues/8383#issuecomment-322617101, one of our hosts had docker bridge on IP different from  '172.17.0.1', and after forcing it to  '172.17.0.1'
/etc/docker/daemon.json: {   "bip": "172.17.0.1/16" }
all our stack works like a charm, i can bounce any hosts and it goes back to normal. Though whole night spent on it.

source: gharchive/issue

authors: Antauri DomiStyle Gmentsik IrisDande JaviSabalete T-Grave VincenzoFerme adamgoose aemneina alach1 alena1108 balzaczyy cjellick ckreutz dax0 ddelnano deniseschannon dren-dk firehist galindro garjunan gcristofol giovannicandido hawkwang ibuildthecloud iskrid jslatten jtcressy justsml jwhitcraft kallqvist kkvilekval lucacri marrotte marvinwankersteen mattdodge miv mkurzeja nextrevision proligde pschirch pschrammel raphink samattridge stefanlasiewski stormerider swartel-fa tbohnen tnh tpagden willseward yevheniidehtiar

repo: rancher/rancher

license_type: permissive

license_source: github-api


text:
If config.yml is missing, the template doesn't show up in catalog

Rancher Versions: master 2/16
Steps to Reproduce:

Create a catalog folder and leave out the config.yml
Add the catalog
Go to Catalog drop-down menu and select your catalog

Results: The template with the missing config.yml is not showing up.

Version - master 2/17
Verified fixed


text:
webhook-service enhancement: serviceUpgrade: support docker registry notifications

From @mnzaki:
The serviceUpgrade driver currently supports Docker Hub webhooks. It would be immensely useful to also support notification events from the docker registry, for people hosting their own private registries.
Example payloads: https://docs.docker.com/registry/notifications/#envelope

any updates on this?
I just ran into the issue when creating a continuous deployment pipeline with gitlab and a private registry. My workaround for now is doing the http call at the end of the script (after the image has been pushed). But it would be much cleaner if rancher webhooks were compatible with private registries.

authors: deniseschannon kobellem


text:
Account API key can not access Environments

Rancher versions:
rancher/server: 1.5.10
rancher/agent: 1.2.2
Infrastructure Stack versions:
healthcheck: 0.2.3
ipsec: n/a
network-services: 0.1.2
scheduler: 0.5.1
kubernetes (if applicable):
Docker version: (docker version,docker info preferred)
1.10.3
Operating system and kernel: (cat /etc/os-release, uname -r preferred)
Ubuntu 14.04
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
AWS
Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)
single node rancher
Environment Template: (Cattle/Kubernetes/Swarm/Mesos)
Cattle
Steps to Reproduce:
Trying to create an API key that can create new Environments, and can be used to access multiple Environments via /v2-beta/projects/.
Results:
When running a simple GET request using API account user I created, I get the following returned to me.  Note the "data" section is not returning any data related to the projects.
However, if I run a GET to a path underneath this, such as http://hostname/v2-beta/projects/1a7/hosts, data will return.
{
"type": "collection",
"resourceType": "project",
"links": {
"self": "https://hostname/v2-beta/projects/"
},
"createTypes": {},
"actions": {},
"data": [],
"sortLinks": {},
"pagination": null,
"sort": null,
"filters": {},
"createDefaults": {}
}

@dmilltech you have added the user to the environment? If so what role does that user have (admin, restricted, user, read only)?

So this is an account API key, and by default these should have admin-like access to all environments.  Turns out that the bug I mention only occurs in weird circumstances wherein the access to the "data" array happens if you create a account API key with pre-defined user/password, with one identity provider (local, LDAP, etc.) and then switch it.
Long story short, this problem does not occur if you let Rancher assign the user/password values automatically and use those.  Not sure if this is by design or not, but it satisfies my issue and I am not sure this is worth digging more into.

Setting API key values on create is a generally bad idea and is only allowed for admins  as a more or less unadvertised testing mechanism.  data is the raw column from the db and should not be used directly (and also only visible to admin).

Thanks for the clarification.

authors: aemneina dmilltech vincent99


text:
Capability override

Problem:
Users want to use psp feature on imported cluster's that have it enabled. Before providing UI for this, we need a way of know whether an imported cluster has it enabled.
Solution:
Provide a way of overriding capabilities via annotations.
Depends on:
https://github.com/rancher/types/pull/998
Issue:
https://github.com/rancher/rancher/issues/21039

@rmweir can you add examples of what this annotation looks like? This will also need a docs issue.

To override some of the current capabilities:
"capabilities.cattle.io/pspEnabled": "true"
"capabilities.cattle.io/ingressCapabilities": "[{"customDefaultBackend":true,"ingressProvider":"asdf"}]"

This looks fine, also need to add setting the psp capability on cluster create/update in the store as this is what the UI is going to trigger off of.

Test instructions:

Import a cluster provsiioned with rke with psps enabled.
Use annotation described in earlier comment to set it to true.
Use annotation to set it back to false.
Check that override capabilities value is being changed.

Override other capability fields using annotation. Confirm that values are being changed.

lgtm

source: gharchive/pull-request

authors: dramich rmweir


text:
Redirect Logging to ELK or FileSystem etc.,

Hi @StrongMonkey,
How to redirect the logs of the application to ElasticSearch/Filesystem or any other option that fluentd supports.How to customize it ?
Thanks & regards,
Hari

@mkharibalaji At this point Rio doesn't really manage logging for you. So you might have to look at documentation on how to setup ElasticSearch/Fluentd in Kubernetes.

Thanks a lot @StrongMonkey

authors: StrongMonkey mkharibalaji

repo: rancher/rio


text:
Delete webhook from Github when deleting service from Rio

If I delete a service, the webhook remains. If I create it again, a second webhook is added. Can the original webhook be deleted when the service is deleted?
This runs into an issue if we're deleting the webhook secret after using it (feature request), since we lose the ability to communicate with the API. Perhaps record that a webhook was created and advise that it needs to be manually deleted, or ask for the secret again to check the repo and delete the webhook automatically?

This is related to re-using webhooks: https://github.com/rancher/rio/issues/676
When multiple services start re-using a single webhook, deleting it will be more complicated. But we should be able to check if any exist and if not then delete it. Will need to add all this to gitwatcher library.
Also if we are re-using webhooks I doubt we would want to delete webhook secret.

authors: cbron oskapt


text:
In main Project list page, service is first, but in details my balancer is first

v0.21.3
Steps:

Create a project
Add a Service named Service1
Add a Balancer named lb1

Results:
Notice in the Main list view page, the Service is first and the Balancer is second.  But if you go into the details page for that project, the Balancer is first and the Service is second.
Expected:
The order should be same no matter what page I'm in

They were just generally unsorted on the list page.

v0.22.0-rc1
failed - the balancer is still sorting after Service on main page and first on details page.

I'm unable to reproduce on v0.49.0-rc1, but will have @tfiduccia check as well.

Both pages have changed quite a bit so this bug is no longer valid.

authors: deniseschannon tfiduccia vincent99

repo: rancherio/rancher

license_source: bigquery


text:
The ip and ports of stopped instance still display on RancherVM Web UI in abnormal scene

I had mentioned this issue in https://github.com/rancherio/vm/issues/42
but now i find how to reproduce it.
step1, create and start a instance
step2, kill docker process or restart host os
step3, start docker and ranchervm
The instance which is up status in step1, its ip and ports still display on web ui, but it is down status, such as following pic:

I read the code of this file: vm/mgmt/ranchervm, docker object contain the value of ip and ports, so maybe there was same cahe or temp file which record the ip and ports  not been deleted while docker service killed?
if not, i think we can workaroud this issue.
if status.startswith("Up"):
.......
else:
ip = ""
ports = ""
PS: my machine env is,
zhaoguang@zgvm3:/tmp$ uname -a
Linux zgvm3 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
zhaoguang@zgvm3:/tmp$ sudo docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64
zhaoguang@zgvm3:/tmp$

Yes it is possible to work around this issue. But let me give it some more thought to make sure there're no situations where the IP and port are valid when the container is not up.

I'm inclined to not do anything about this issue as there's no harm to display the meta data we get from Docker inspect. Any reason why this has to be fixed?

Yes, this issue is low severity, but confuse users such as following:

I pushed a fix and updated the ranchervm image. Please give it a try.

Yes, it's OK now:)

authors: cnzg sheng-liang

repo: rancherio/vm


text:
Operation cancelled

Hi All,
We have a user who is experiencing this error, Operation cancelled which seems to stem from signTransaction.
Unfortunately we can't reproduce and are having a hard time getting much from the user, but I thought I'd reach out to see if anyone is seeing something similar, or has an troubleshooting tips.
Basically all we are doing is asking our users to connect their wallet (popup 1), then sign a 0 amount transaction (popup 2). Each with a user action to avoid popup warnings.
We have at least 10 people regularly and successfully doing this (multiple times a day) on our web app.
This user however, seems to always get the error Operation cancelled and we have tracked the message back to the call to signTransaction.
Again, not much to go off, we are asking the client to try a different browser, new account, etc... but any tips would be welcome.
Cheers.

Update: this appears to only happen when connecting from Firefox browser.

Hell @shourovb  @hutch120
Do you know if the signTransaction popup was opened?
It was opened and closed immediately?

Popup appeared, TX appeared correctly and user was able to go through with entering password and press "Sign". Popup closed and the above error was thrown.

We are almost certainly seeing this issue: https://github.com/randlabs/myalgo-connect/issues/35#issuecomment-852994208

@hutch120 I'm not sure, because this issue blocks the MyAlgo Connect popup immediately:
https://connect.myalgo.com/docs/getting-started/important-considerations/#firefox-and-safari-issues
@M-Picco will be following this issue

repo: randlabs/myalgo-connect


text:
Move lucene's mockfs into junit or randomizedtesting?

Lucene mockfs (https://github.com/apache/lucene-solr/tree/master/lucene/test-framework/src/java/org/apache/lucene/mockfile) basically tries to wrap the default nio.2 filesystem with a provider that changes things up:

ShuffleFS: Directory listings in an unpredictable but deterministic order regardless of underlying filesystem order (e.g. OS X returns in sorted order, Linux does not)
LeakFS: Fails tests if they leave open filehandles. (Only files with nio.2 api, not sockets, but useful).
DisableFsyncFS: Makes actual fsync() calls a no-op. If you are e.g. a database or similar and using the filesystem seriously, this speeds up tests with no downside...
VerboseFS: Prints destructive filesystem operations to infostream (Logging writes/deletes/renames/etc). For debugging.
ExtrasFS: Adds 'bonus' files to directories. Kind of annoying and lucene-specific. But you could tame it by only adding 'realistic bonus files' like '.DS_Store'.
WindowsFS: Acts like windows (imperfect, just prevents deleting open files at the moment)

The idea of the thing is that you can build a chain, of several of these, wrapping the default filesystem. None of the stuff is really dependent on lucene though, and some of these filesystem issues are real traps (e.g. system-specific stuff like order of results from listing directories)?
However, there are some problems/downsides/questions as it stands out:

We wrap at test runner level (we pass to getTempDir() and all lucene tests use that). Ideally it would be at a higher level and pass system properties to the jvm to wrap the default filesystem so that it can't be backdoored by calls to e.g. FileSystem.getDefault()/Paths.get()/.... I have hacked it to work this way for debugging production code (VerboseFS) with https://github.com/rmuir/verbosefs .
NIO.2 api is incompletely wrapped for some stuff. We don't yet wrap the FileLock API (because lucene has a bad history with fs locking code, but we should do it now), some implementations are incomplete because Lucene only uses a subset of the filesystem operations.
There are unit tests, but they could be improved...
There are some dragons in the current code, while it works for lucene's case, if you add to an app more extensively using the java NIO apis, you will find: https://github.com/rmuir/verbosefs/blob/master/src/rmuir/FilterPath.java#L57 These are basically  VFS layer violation bugs with things being wrapped twice or something like that.
Is NIO.2 really being adopted? Not many projects are using it yet. But if you fully cut over, you can take advantage of stuff like this.

Anyway, I'd be interested in helping out if its a better home or fixes to the issues or whatever. Or may be we could only take some of ideas and try to rewrite it to work better, etc.

Hi Robert! I like these fake filesystems very much. I wonder if it makes sense to make them tightly coupled to the runner though. Somebody willing to randomize the fs could still include the dependency and set it up at the runner's invocation level...
Anyway, I don't see any problem in moving that work over to randomizedtesting/* namespace, either as an independnet subproject or as a built-in feature of the runner, both sound like great additions.

authors: dweiss rmuir

repo: randomizedtesting/randomizedtesting


text:
forward actions without meta.

Solves issue mentioned in #45

It seems like redux-segment is broken unless this gets merged.

I concur

when will this issue be fixed?

But like seriously. When will this get merged?

+1

This looks great. Thanks @coopermaruyama.

Sorry this took so long gents. 😓

@bertrandk thanks mate. its really helpful, I am no longer need to import all your source code and patch it myself now.

authors: bertrandk chollier coopermaruyama gsheppard htxiong kevinhos samcorcos

repo: rangle/redux-segment

license: mit


text:
Implement edge merging post-Relay

This issue stems from a discussion of “How do we fix subclass of?” during the 8/19 Aragorn Team Meeting
https://github.com/ranking-agent/aragorn/issues/115

We do edge merging now.

authors: capasfield maximusunc

repo: ranking-agent/strider

license: MIT


text:
Network Receiver/Transmitters not working with rftools dim

Issue description:
Network Receiver not working across rftools dimensions
What happens:
When using transmitter/receivers across rftools dimensions, even while loaded they don't work most of the time, occasionally it will but leaving and returning to the dimension brings this issue back. This only occurs when the receiver is in the rftools dim, not the other way around
Version

Minecraft: 1.10.2
Refined Storage: 1.2.23

Does this issue occur on a server? [yes/no]
yes

Are both sides chunkloaded?

yes

Duplicate of #370

repo: raoulvdberge/refinedstorage


text:
external_storage dupe bug(1.12)

1.build an rs system with 2 external storage bus connect to 2 different chest
2.put 32 item into each chest
3.try to get 64 item by shift clicking in terminal or by export bus
4.you get 64 items and 32 remain  in rs sys
Version (make sure you are on the latest version before reporting):

Minecraft: 1.12
Forge:2420
Refined Storage:1513

Does this issue occur on a server? [yes/no]
yes

Are you using Colossal chests? If so, are you connecting External Storage to Chest core?

Nope ,i tried vanilla chest , extrautil tiny chest and acatully addition wooden crate,all of them can reproduce this dupe

I can't reproduce this.

I have yet to test it on a server, though.

need 1 chest from mod and have higher prority
tested extrautils small chest and actually addons crate they both can reproduce this

@raoulvdberge confirmed.
This bug can be reproduced. @CatAndHorse just forgot to mention about priority.

But, as far as i can see it can be reproduced only with one-slot inventories. It just doesn't happen with regular chests only nor quark chests. Only Extra Utils mini chest can trigger that bug.

In other words: this only happens with ExU mini chest? In that case, it might be an ExU dupe bug issue. Will still investigate.

This appears to be a ExU issue.
I tested this with 2 external storages both connected to its own Mini-Chest.
The problem is as follows:

When shift clicking, RS asks 64 of that item
Then, ExU gives back a stack of 64 items even though it only has 32

Proof:

requested is 64, because we want 64 items for shift click
We got back a stack of 64 even though the Mini-Chest only has 32 of that item (look at took)

My testing setup:

Both Mini-Chests have 32 items. But ExU just gives back 64 items for the first chest, even though it only has 32. So RS stops and doesn't extract from the second one.
In conclusion: this is a ExU issue. They appear to trust the "amount" parameter without checking if it doesn't exceed their own stack size.

And the problem is - @rwtema never reads github issues... ExtraUtils 2 issue tracker just drowned in issues

I just pinged him on IRC, he might fix it.

Actually additions' storage crates can reproduce this too @raoulvdberge
okay i will open issue there too

I just looked into again some more and I was wrong: this is not an ExU or ActAdd issue. It's in fact an RS issue.
What's going on: I assumed IItemHandler#extract is guaranteed to make copies of the stack, and I could just safely modify the stack returned by it.
Apparantly it does copy the stack, but not when requesting more than the stack size.
    @Nonnull
    public ItemStack extractItem(int slot, int amount, boolean simulate) {
        if (amount == 0) {
            return ItemStack.EMPTY;
        } else {
            this.validateSlotIndex(slot);
            ItemStack existing = (ItemStack)this.stacks.get(slot);
            if (existing.isEmpty()) {
                return ItemStack.EMPTY;
            } else {
                int toExtract = Math.min(amount, existing.getMaxStackSize());
                if (existing.getCount() <= toExtract) {
                    if (!simulate) {
                        this.stacks.set(slot, ItemStack.EMPTY);
                        this.onContentsChanged(slot);
                    }

                    return existing;
                } else {
                    if (!simulate) {
                        this.stacks.set(slot, ItemHandlerHelper.copyStackWithSize(existing, existing.getCount() - toExtract));
                        this.onContentsChanged(slot);
                    }

                    return ItemHandlerHelper.copyStackWithSize(existing, toExtract);
                }
            }
        }
    }

Look at the return existing;.
Thanks to that, RS just modifies the stack when extracting and false assumptions are

authors: CatAndHorse raoulvdberge roboderpy


text:
Библиотеки libwitnesscalc_

Видно что в проекте есть статические библиотеки libwitnesscalc, но не видно чтобы они использовались из кода.
Для чего они тогда в таком случае нужны? Удалив их из бандла, проект успешно собирается.

Hi, static libraries that start with libwitnesscalc are used to calculate [witness] (https://nmohnblatt.github.io/zk-jargon-decoder/definitions/witness.html). Those libraries are compiled with the witnesscalc. Their source files are artifacts generated by the Circom compiler. Circuits that are used for compilers you can find here and here.

repo: rarimo/FreedomTool


text:
Adding support for 'ip' network utility instead of 'ifconfig' if available

Also moved all package dependencies other than git to a single script that can be sourced.

@dragon788: Great work. I'm going to accept it, as it works for me on Ubuntu. I haven't had time to test it in other environments.

repo: rasa/vmware-tools-patches


text:
Is it possible to not use pointer offset?

Currently, the actions will be placed around the touch point (offset):

Is it possible to place the actions around the center of the child regardless of the pointer offset?

not right now but good idea! I'll work on this when I get the chance.

I have done a quick implementation, please take a look at this if you have a time.
https://github.com/sadespresso/flutter-pie-menu/commit/69c45eeab83458ef5e7e0ea0f23e90e34fd2b667
Not sure about what to test for now, but I'll make a PR and improve if you approve :)
Thank you, awesome package!!!

sure, pull requests are welcome

authors: rasitayaz sadespresso

repo: rasitayaz/flutter-pie-menu


text:
request.getmetadata no longer returns FocusFOM

Describe the bug
My app has a live camera preview window that displays some metadata alongside. After the last Picamera2 update it crashes with KeyError: "FocusFOM"
To Reproduce
It looks like the FocusFOM value is no longer being returned by request.get_metadata(), causing the error.
Hardware :
Raspberry Pi 4, High Quality Camera, current Raspberry Pi OS
Additional context
The camera has a macro lens attached and is manually focused onto a 16mm or 8mm film frame. This was a useful metric to display, as it helps with getting critical focus - adjust the focus until the maximum value is reached. Removing the reference makes my code work, but without this useful feature.
Here's an excerpt from my previously working code:
    def preview_data_update_callback(self, request):
        m = request.get_metadata()
        text = (f'Exposure: {m["ExposureTime"]}\n' +
                f'Gain: {m["AnalogueGain"]:.2f}\n' +
                f'Focus: {m["FocusFoM"]//100}\n')

Hi, thanks for reporting this. Indeed something does seem to have got lost there. I'll look into it, but somewhat suspect we're going to have to make up a new libcamera release.

Thanks! I'm glad it wasn't an intentional feature change. :)

It's only a small fix and I've committed it to the repo. Just need to wait while the packaging/release wheels turn. Hopefully not long.

I think
sudo apt update
sudo apt install libcamera0

should fix it for you now.

Verified working. That did the trick. Thanks for your work on this!

FWIW, I encountered a similar issue ("FocusFoM" keyword missing from metadata dictionary) recently, even after a sudo apt update && sudo apt install libcamera0. However, the problem occurs for me only in the first frame captured after a .start() call. Subsequent frames' metadata contains all the expected keywords, so it is possible to "fix" the issue by just throwing away the first frame, but I figured that this may also be indicative of a larger issue and/or something that is worth investigating. Note also that despite some of the other values being wonky in the first frame(s)—e.g. #824—FocusFoM seems to be the only thing that is ever missing entirely.

Hi, thanks for reporting this. As you know, there are some difficulties particularly with the very first frame, though there are probably some other things that take a bit longer to settle as well. I'm not quite sure how it comes about that some things report default values early on, and some might be missing. It could be a bug, but we probably won't do much until we've done all Pi 5 work. Pi 5 works quite differently in some crucial respects and so we'll need to find solutions that work on both our platforms. So please bear with us... thanks!

Probably worked for me as the camera runs a short while before my code gets round to reading "FocusFoM".
I've a Pi 5 on order and am looking forward to migrating my cine film scanning project to it. The processing and storage speed improvements should really help as these are the main bottlenecks at the moment.
Looks like there might be some fun and games with picamera2/libcamera? My Python (+Cython) application is processing full resolution raw bayer data (from stills captures) from the HQ camera, with the live preview for critical focus and to obtain base exposure and colour balance data. I'm willing to help out and test as and when the Pi 5 and camera cable lands.

Mostly the Pi 5 should behave exactly the same for you. The one important difference which you may encounter is that when you ask for unpacked raw data (presumably this is what you do?) , the pixels will be delivered with one pixel per 2 bytes (as on a Pi 4), but the pixel values will be left-shifted so that all the zero bits are at the bottom and not at the top (as is the case on Pi 4).

Thanks for the heads-up. Looks simple enough - just a change to the lookup table that expands the 12bit data to the full 16bit range used in the rest of the processing.

I now have the Pi 5 and working on modifying my code for Pi5/Bookworm. The extra speed is nice.
It looks like FocusFoM takes a while to appear in the metadata after the camera starts as described by cgobat above causing my application to crash as well, and it's returning a much larger number than previously. As a workaround, I modified my metadata callback to check if it's available first:
    def preview_data_update_callback(self, request):
        m = request.get_metadata()
        text = (f'Exposure: {m["ExposureTime"]}\n' +
                f'Gain: {m["AnalogueGain"]:.2f}\n')
        if 'FocusFoM' in m:
                text += f'Focus: {m["FocusFoM"]//1000000}\n'
...
        self.controls['status_label'].setText(text)

It's not really "meant" to return a larger number, though I recall that during development it kept doing that. So we might want to fix that. It is, of course, entirely arbitrary, so can only meaningful be used in comparisons against other FoMs!

I noticed that it reduces by about a factor of 4 when going from 4056x3040 to 2028x1520 resolution. Based on the number of pixels?  Although arbitary, it's a useful metric to see when manually focussing. Though I think it'd be better if it would return a value normalized to some range independent of image size, or 0 if the FocusFoM value isn't present in the underlying metadata at all. :)

Yes, the FoM is just a summation over all the pixels (in the region or regions being counted). Normalising to the number of pixels sounds like it might be worthwhile, and it would be amusing to see which sensors seem to give best "per pixel sharpness". One for the list of things to look at!

Yes, we finally got round to posting the updates that I've been alluding to in the forums for a while now! So we did fix that oversized FoM thing, then, I was always complaining about it...
I put a post on the forum about the updated software. Mostly things should work the same, but manipulating pixels within image buffers should be faster.

Thanks - I'll check the forum. :)

authors: cgobat davidplowman jlprojects


text:
CMake cmake_minimum_required Deprecation Warning

Fixes: #1545
Recently, when building pioasm and/or elf2uf2, CMake warns about cmake_minimum_required:
[ 96%] Performing configure step for 'PioasmBuild'
loading initial cache file /home/gmt/Projects/pico/build/pico-sdk/src/rp2_common/pico_cyw43_driver/pioasm/tmp/PioasmBuild-cache-Release.cmake
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

#1477 reported the issue, but submitted patch is not correct.

https://github.com/raspberrypi/pico-sdk/pull/1477 reported the issue, but submitted patch is not correct.

Why not?

#1477 reported the issue, but submitted patch is not correct.

Why not?

Simply by applying it, I see the warning again! I have cmake 3.27.8.
But I will check it again.

repo: raspberrypi/pico-sdk


text:
Unable to boot fresh installation. RF Kill Switch Status

Hi,
I downloaded the v.1.3 image and used etcher to write to my SD card.
I tested it on both my new PI4 and my old PI3B+ and both are unable to boot.
Please note that I use standard Raspis, without LCD
The startup log hangs on:
Started Load/Save RF Kill Switch Status.
Any idea what might cause this? (I also tried it last week with v.1.3-RC5 on my raspi 4 with the same error)

It's normal. Without an LCD you won't see what's going on unless you SSH into your node.
Have you tried?

No, I haven't tried that yet. I will try it when I am home again.
I don't have a built in LCD, but I connected an external monitor through hdmi, shouldn't that monitor display the same as the LCD?

The RaspberryPi can either output the video on the LCD or the HDMI port - not both. I will try to add a config script for the next release to make the switching between both more easy #767

I have the EXACT same issue, I have the built in LCD screen. That screen shot helped me figure out what was going on.

I’m having this issue.. and when I try to ssh with the password it say sorry try again ?

@slc1975 whats the exact command you try to login with ssh? You need to have the user "admin@" before the IP and then password "raspiblitz" should work. Without adding the admin part it tries to login with your local laptop username which is not configured on the RaspiBlitz.

ok .. all up and running now.. many thanks .. I would like to get RTL up and running .. but the guild is not very detailed .. you don't fancy doing a howto.. or could you point me in the right direction ? Ta

to clarify.. is RTL running on the node or my PC ?

Installing RTL in the main menu go under SERVICES and then set the star under RTL with SPACE and then install with ENTER.

I'm having the same issue.
After some poking around, it seems as though when running a Pi4, paticularly with 7" touch screen, the screen never gets past "Load/Save RF Kill Switch Status."
Despite the screen hanging here, the Pi continues to function as normal. Ie. You can SSH to it and do a complete install and run it without a LCD.
Anyone else confirm this issue, and/or isolate what could be causing the problem?

was this ever resolved? i'm having the exact same issue and i've search every forum and telegram i could find. here are my specs i would love some help.
Enokay R-Pi 4B Power Supply 5V 3A USB C Charger Adapter
SanDisk Ultra 32GB microSDHC
Crucial MX500 1TB 3D NAND SATA 2.5 Inch Internal SSD
ORICO External Hard Drive Enclosure SATA III
Raspberry Pi 4 Model B 2019 (4GB)
Miuzei Raspberry Pi 4 Touch Screen with Case & Fan

Did you ever get it working properly?

I never did, I just returned the screen and bought the one on the list and
it works fine. Do you think the new version would support different sized
screens?
On Fri, Feb 14, 2020 at 12:57 AM SteakDestroyer notifications@github.com
wrote:

was this ever resolved? i'm having the exact same issue and i've search
every forum and telegram i could find. here are my specs i would love some
help.
Enokay R-Pi 4B Power Supply 5V 3A USB C Charger Adapter
SanDisk Ultra 32GB microSDHC
Crucial MX500 1TB 3D NAND SATA 2.5 Inch Internal SSD
ORICO External Hard Drive Enclosure SATA III
Raspberry Pi 4 Model B 2019 (4GB)
Miuzei Raspberry Pi 4 Touch Screen with Case & Fan
Did you ever get it working properly?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rootzoll/raspiblitz/issues/753?email_source=notifications&email_token=AOL6LQQBRKGCLJVUXOBDWDLRCYXGNA5CNFSM4IUGONJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELXVOHY#issuecomment-586110751,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOL6LQUI7RDJYU2BWQU3BNTRCYXGNANCNFSM4IUGONJQ
.

I needed the LCD display attached.

i have the exact same issue. i have no LCD laying around here so it's only a Pi4..

2023 same issue. Not using raspblitz, but official os.

authors: Aki113s SteakDestroyer dirtysteveo fluidvoice john-clark michaelWuensch mikeblakeuk mroxso rootzoll slc1975

repo: raspiblitz/raspiblitz


text:
Feature request: Behaviour if no icon is found

What do you think about showing an additional icon if the search term does not match?
e.g.

also I added the following css paramters for e better optical consistency and a small shadow for a better visualization of the floating window in my setup.
div.afp-dropdown-container div.afp-dropdown div.afp-dropdown--searchbox {
    background-color: #fff;
    box-shadow: 0 8px 10px -4px rgba(0,0,0, .15);
}

But this kann be done by custom stylesheets of course :-)

no longer relevant


text:
Its possible conflicts happen ?

it is possible to occur conflicts with $conn->resourceID?
I realized that it uses something like a count starting from the moment the server starts ..

Not likely as the resourceID comes from the stream socket id.

authors: cboden thebestclass

repo: ratchetphp/Ratchet


text:
Websocket dies after 2 days

Hi everyone,
Ive had the websocket running for last 7 months with little problems.
I have just changed servers and passed everything over to the new server and for some reason my websocket dies after two days.
Have tried to find an error log or something to give me a reason why this is happening but i have not found anything.
Is there any where that logs the exeception/errors? Or a possible reason to why this is happening?
Thanks in advance.

If it was a major php fault, you should have something in the site's error log. I would also check uptimes of the server and apache to see if they crashed.
Error/exceptions to connections and messaging is not automatically logged, afaik.

authors: Fyod david-marweblous


text:
Can't receive access_token

Does it work correctly fro Twitter/Facebook or Google+ ?
Can't manage any of them.
Can anyone provide an example with those networks?
e.g. I have following url for twitter:
"https://api.twitter.com/oauth/authorize/?client_id=\(clientId)&scope=\(scopes.joined(separator: "+"))&redirect_uri=(redirectURI)&response_type=code"
accessTokenURL = https://api.twitter.com/oauth/access_token"
redirectURI = "https://www.google.com"
Then, I put my ClientID and Secret, but nothing works. I can login and do not receive my token after switching to redirectURL.

Hi,
The redirect URL needs to be a callback to your app scheme, check the README instructions on how to setup your project https://github.com/raulriera/AuthenticationViewController

Still do not have an answer. Could anyone provide a workable example for Twitter, for example?
Which url should I use for redirectURI and for requestToken there?

I'm the creator of this library, I just said in the previous message what redirectURI to use. 🤔

Yep, thank you. Now I can authenticate, but after that it opens redirect_uri and code from AppDelegate: (authenticationViewController.authenticateWithCode(code)) doesn't retrieve. What am I doing wrong? Doesn;t work even with Instagram...

Can you provide some code example?

Here you can find it:
https://github.com/AndreyMomot/AuthenticationViewController
I just can't get what to use for URL Scheme in my case.

// Before doing this, you should check the url is your redirect-uri before doing anything. Be safe :)
if let components = URLComponents(url: url, resolvingAgainstBaseURL: false), let queryItems = components.queryItems, let code = queryItems.first?.value {

Have you checked that the URL you are getting here is the scheme you specified? Does it match the one you used in Instagram?

Make you are you used OAuthExample://oauth-callback in your Instagram app and also that you are getting the same one in the AppDelegate (previous code snippet)

authors: AndreyMomot raulriera

repo: raulriera/AuthenticationViewController


text:
[Feature][Docs][Discussion] Provider consistent guidance on resource Request and Limits

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
Documentation on resource request and limits should be made clearer and more prominent, perhaps repeated in several places in the docs.
Best practices are laid out here:
https://discuss.ray.io/t/questions-for-configurations-using-helm-chart/7762?u=dmitri
https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/config.html#resources
Subtleties:
We currently make the following recommendations:

KubeRay takes CPU/memory resource info from limits and advertises that info to Ray. For that reason, we recommend setting requests=limits for KubeRay configs. (Otherwise, the Ray pod may be scheduled with less resource availability than is advertised to Ray.)
Because a Ray cluster does not benefit from running multiple pods per node, we recommend sizing pods to be as large as possible. When possible within a user's constraints it is best to run one Ray pod per Kubernetes node.
Because Ray round up the CPU limit to the nearest whole CPU, we recommend setting an integer CPU limit.

Some issues with these recommendations.

The documentation is not fully consistent with this recommendation.
If you take points 1. and 2. literally, it requires determining the exact allocatable resources of a node and then subtracting some safety amount to make sure your Ray pods can get scheduled. Another valid approach is to set limits equal to the node's total capacity and requests to something just under the allocatable capacity.
Taking points 1. and 3. literally requires setting CPU requests and limits both to at least 1 in all configs. This is difficult for example configs and CI test configs meant to run in CPU-constrained Minikube/Kind environments.

Use case
Less user confusion when figuring out resources for Ray on K8s.
Related issues
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Because Ray does not benefit from running multiple pods per node, we recommend sizing pods to take up entire nodes.

I think we can not assume we only run one Ray cluster in K8s cluster. So different size clusters will fill the k8s clusters.

I've amended that bullet slightly. It's of course subject to the constraints of the user's Kubernetes environment, but generally speaking, it does not make sense to pack many tiny Ray pods into a single K8s node.

A common setup (and one we've used at Anyscale in the past) is to set up cluster autoscaling such that you get a new K8s node of the appropriate type each time you request a Ray pod. The Ray pod and supporting machinery then fill up the entire K8s node.

Hey @DmitriGekhtman for number 2:
Because a Ray cluster does not benefit from running multiple pods per node, we recommend sizing pods to be as large as possible. When possible within a user's constraints it is best to run one Ray pod per Kubernetes node

Is there a way to configure the autoscaler to use STRICT_SPREAD? https://docs.ray.io/en/latest/ray-core/scheduling/placement-group.html?highlight=placement
Seems that is is only possible from the SDK and not when configuring the cluster? https://github.com/ray-project/kuberay/blob/3aebd8c9f5ae5d9d9d12489d5636d3cf1b97548e/ray-operator/config/samples/ray-cluster.autoscaler.large.yaml

STRICT_SPREAD helps to spread Ray tasks across different Ray pods.
To spread Ray pods across different Kubernetes nodes, I think the thing to look into would be pod anti-affinity.

authors: DmitriGekhtman Jeffwan peterghaddad

repo: ray-project/kuberay


text:
Miss two symbols in README.md "All configure options"

tagfile = 'tags' →
tagfile = 'tags',

options = '-R --exclude=.git --exclude=node_modules --exclude=test --exclude=vendor --excmd=number'
} →
options = '-R --exclude=.git --exclude=node_modules --exclude=test --exclude=vendor --excmd=number'
},

should be fixed in the latest. thanks!

authors: marcuslannister ray-x