text:
Follower units of mysql-router-k8s never go active

Bug Description
When deploying the mysql-router-k8s charmed operator with multiple units, only the leader unit (as identified by Juju) manages to achieve the active status with a running mysql-router service.
To Reproduce
juju deploy --trust -n 3 --channel edge mysql-router-k8s
juju deploy --trust -n 3 --channel edge mysql-k8s
juju deploy --trust --channel yoga/edge keystone-k8s
juju relate mysql-k8s mysql-router-k8s
juju relate keystone-k8s mysql-router-k8s
Environment
microk9s (1.25-strict/stable channel)
Relevant log output
unit-keystone-mysql-router-1: 11:36:49 INFO juju.cmd running containerAgent [3.0.2 8bf53dc35b25145ef39051fe4136135a3dd53d5d gc go1.19.3]
unit-keystone-mysql-router-1: 11:36:49 INFO juju.cmd.containeragent.unit start "unit"
unit-keystone-mysql-router-1: 11:36:49 INFO juju.worker.upgradesteps upgrade steps for 3.0.2 have already been run.
unit-keystone-mysql-router-1: 11:36:49 INFO juju.worker.probehttpserver starting http server on 127.0.0.1:65301
unit-keystone-mysql-router-1: 11:36:49 INFO juju.api cannot resolve "controller-service.controller-microk8s-localhost.svc.cluster.local": lookup controller-service.controller-microk8s-localhost.svc.cluster.local: operation was canceled
unit-keystone-mysql-router-1: 11:36:49 INFO juju.api connection established to "wss://10.152.183.64:17070/model/b34505d0-73a8-4f01-8e5c-d7194904701e/api"
unit-keystone-mysql-router-1: 11:36:49 INFO juju.worker.apicaller [b34505] "unit-keystone-mysql-router-1" successfully connected to "10.152.183.64:17070"
unit-keystone-mysql-router-2: 11:36:49 INFO juju.cmd running containerAgent [3.0.2 8bf53dc35b25145ef39051fe4136135a3dd53d5d gc go1.19.3]
unit-keystone-mysql-router-2: 11:36:49 INFO juju.cmd.containeragent.unit start "unit"
unit-keystone-mysql-router-2: 11:36:49 INFO juju.worker.upgradesteps upgrade steps for 3.0.2 have already been run.
unit-keystone-mysql-router-2: 11:36:49 INFO juju.worker.probehttpserver starting http server on 127.0.0.1:65301
unit-keystone-mysql-router-1: 11:36:49 INFO juju.api cannot resolve "controller-service.controller-microk8s-localhost.svc.cluster.local": lookup controller-service.controller-microk8s-localhost.svc.cluster.local: operation was canceled
unit-keystone-mysql-router-1: 11:36:49 INFO juju.api connection established to "wss://10.152.183.64:17070/model/b34505d0-73a8-4f01-8e5c-d7194904701e/api"
unit-keystone-mysql-router-1: 11:36:49 INFO juju.worker.apicaller [b34505] "unit-keystone-mysql-router-1" successfully connected to "10.152.183.64:17070"
unit-keystone-mysql-router-2: 11:36:49 INFO juju.api cannot resolve "controller-service.controller-microk8s-localhost.svc.cluster.local": lookup controller-service.controller-microk8s-localhost.svc.cluster.local: operation was canceled
unit-keystone-mysql-router-2: 11:36:49 INFO juju.api connection established to "wss://10.152.183.64:17070/model/b34505d0-73a8-4f01-8e5c-d7194904701e/api"
unit-keystone-mysql-router-2: 11:36:49 INFO juju.worker.apicaller [b34505] "unit-keystone-mysql-router-2" successfully connected to "10.152.183.64:17070"
unit-keystone-mysql-router-1: 11:36:49 INFO juju.worker.migrationminion migration phase is now: NONE
unit-keystone-mysql-router-1: 11:36:49 INFO juju.worker.logger logger worker started
unit-keystone-mysql-router-1: 11:36:49 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-keystone-mysql-router-1: 11:36:49 INFO juju.worker.leadership keystone-mysql-router/1 promoted to leadership of keystone-mysql-router
unit-keystone-mysql-router-1: 11:36:49 INFO juju.worker.caasupgrader unblocking abort check
unit-keystone-mysql-router-1: 11:36:49 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-keystone-mysql-router-2: 11:36:49 INFO juju.worker.migrationminion migration phase is now: NONE
unit-keystone-mysql-router-2: 11:36:49 INFO juju.worker.logger logger worker started
unit-keystone-mysql-router-2: 11:36:49 INFO juju.worker.leadership keystone-mysql-router leadership for keystone-mysql-router/2 denied
unit-keystone-mysql-router-2: 11:36:49 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-keystone-mysql-router-2: 11:36:49 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-keystone-mysql-router-2: 11:36:49 INFO juju.worker.caasupgrader unblocking abort check
unit-keystone-mysql-router-1: 11:36:49 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-keystone-mysql-router-1
unit-keystone-mysql-router-1: 11:36:49 INFO juju.worker.uniter unit "keystone-mysql-router/1" started
unit-keystone-mysql-router-2: 11:36:49 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-keystone-mysql-router-2
unit-keystone-mysql-router-1: 11:36:49 INFO juju.worker.uniter resuming charm install
unit-keystone-mysql-router-1: 11:36:49 INFO juju.worker.uniter.charm downloading ch:amd64/focal/mysql-router-k8s-5 from API server
unit-keystone-mysql-router-1: 11:36:49 INFO juju.downloader downloading from ch:amd64/focal/mysql-router-k8s-5
unit-keystone-mysql-router-2: 11:36:49 INFO juju.worker.uniter unit "keystone-mysql-router/2" started
unit-keystone-mysql-router-2: 11:36:49 INFO juju.worker.uniter resuming charm install
unit-keystone-mysql-router-2: 11:36:49 INFO juju.worker.uniter.charm downloading ch:amd64/focal/mysql-router-k8s-5 from API server
unit-keystone-mysql-router-2: 11:36:49 INFO juju.downloader downloading from ch:amd64/focal/mysql-router-k8s-5
unit-keystone-mysql-router-1: 11:36:49 INFO juju.downloader download complete ("ch:amd64/focal/mysql-router-k8s-5")
unit-keystone-mysql-router-2: 11:36:50 INFO juju.downloader download complete ("ch:amd64/focal/mysql-router-k8s-5")
unit-keystone-mysql-router-1: 11:36:50 INFO juju.downloader download verified ("ch:amd64/focal/mysql-router-k8s-5")
unit-keystone-mysql-router-2: 11:36:50 INFO juju.downloader download verified ("ch:amd64/focal/mysql-router-k8s-5")
unit-keystone-mysql-router-0: 11:36:53 INFO juju.cmd running containerAgent [3.0.2 8bf53dc35b25145ef39051fe4136135a3dd53d5d gc go1.19.3]
unit-keystone-mysql-router-0: 11:36:53 INFO juju.cmd.containeragent.unit start "unit"
unit-keystone-mysql-router-0: 11:36:53 INFO juju.worker.upgradesteps upgrade steps for 3.0.2 have already been run.
unit-keystone-mysql-router-0: 11:36:53 INFO juju.worker.probehttpserver starting http server on 127.0.0.1:65301
unit-keystone-mysql-router-0: 11:36:53 INFO juju.api cannot resolve "controller-service.controller-microk8s-localhost.svc.cluster.local": lookup controller-service.controller-microk8s-localhost.svc.cluster.local: operation was canceled
unit-keystone-mysql-router-0: 11:36:53 INFO juju.api connection established to "wss://10.152.183.64:17070/model/b34505d0-73a8-4f01-8e5c-d7194904701e/api"
unit-keystone-mysql-router-0: 11:36:53 INFO juju.worker.apicaller [b34505] "unit-keystone-mysql-router-0" successfully connected to "10.152.183.64:17070"
unit-keystone-mysql-router-0: 11:36:53 INFO juju.worker.migrationminion migration phase is now: NONE
unit-keystone-mysql-router-0: 11:36:53 INFO juju.worker.logger logger worker started
unit-keystone-mysql-router-0: 11:36:53 INFO juju.worker.leadership keystone-mysql-router leadership for keystone-mysql-router/0 denied
unit-keystone-mysql-router-0: 11:36:53 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-keystone-mysql-router-0: 11:36:53 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-keystone-mysql-router-0: 11:36:53 INFO juju.worker.caasupgrader unblocking abort check
unit-keystone-mysql-router-0: 11:36:53 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-keystone-mysql-router-0
unit-keystone-mysql-router-0: 11:36:53 INFO juju.worker.uniter unit "keystone-mysql-router/0" started
unit-keystone-mysql-router-0: 11:36:53 INFO juju.worker.uniter resuming charm install
unit-keystone-mysql-router-0: 11:36:53 INFO juju.worker.uniter.charm downloading ch:amd64/focal/mysql-router-k8s-5 from API server
unit-keystone-mysql-router-0: 11:36:53 INFO juju.downloader downloading from ch:amd64/focal/mysql-router-k8s-5
unit-keystone-mysql-router-0: 11:36:53 INFO juju.downloader download complete ("ch:amd64/focal/mysql-router-k8s-5")
unit-keystone-mysql-router-0: 11:36:53 INFO juju.downloader download verified ("ch:amd64/focal/mysql-router-k8s-5")
unit-keystone-mysql-router-2: 11:36:55 INFO juju.worker.uniter hooks are retried true
unit-keystone-mysql-router-2: 11:36:56 INFO juju.worker.uniter found queued "install" hook
unit-keystone-mysql-router-1: 11:36:56 INFO juju.worker.uniter hooks are retried true
unit-keystone-mysql-router-1: 11:36:56 INFO juju.worker.uniter found queued "install" hook
unit-keystone-mysql-router-0: 11:36:56 INFO juju.worker.uniter hooks are retried true
unit-keystone-mysql-router-0: 11:36:56 INFO juju.worker.uniter found queued "install" hook
unit-keystone-mysql-router-2: 11:36:57 INFO unit.keystone-mysql-router/2.juju-log Running legacy hooks/install.
unit-keystone-mysql-router-1: 11:36:57 INFO unit.keystone-mysql-router/1.juju-log Running legacy hooks/install.
unit-keystone-mysql-router-0: 11:36:57 INFO unit.keystone-mysql-router/0.juju-log Running legacy hooks/install.
unit-keystone-mysql-router-2: 11:36:58 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:36:58 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:36:58 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:36:59 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:36:59 INFO juju.worker.uniter.operation ran "database-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:00 INFO juju.worker.uniter.operation ran "database-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:00 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:00 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:00 INFO juju.worker.uniter.operation ran "backend-database-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:00 INFO juju.worker.uniter.operation ran "backend-database-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:01 INFO juju.worker.uniter.operation ran "backend-database-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:01 INFO juju.worker.uniter found queued "leader-elected" hook
unit-keystone-mysql-router-2: 11:37:01 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:01 INFO juju.worker.uniter.operation ran "database-relation-created" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:02 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:02 INFO juju.worker.uniter.operation ran "leader-settings-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:02 INFO juju.worker.uniter.operation ran "mysql-router-pebble-ready" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:03 INFO juju.worker.uniter.operation ran "mysql-router-pebble-ready" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:03 INFO juju.worker.uniter.operation ran "mysql-router-pebble-ready" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:03 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:03 INFO juju.worker.uniter found queued "start" hook
unit-keystone-mysql-router-2: 11:37:04 INFO unit.keystone-mysql-router/2.juju-log Running legacy hooks/start.
unit-keystone-mysql-router-1: 11:37:04 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:04 INFO juju.worker.uniter found queued "start" hook
unit-keystone-mysql-router-0: 11:37:04 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:04 INFO juju.worker.uniter found queued "start" hook
unit-keystone-mysql-router-1: 11:37:04 INFO unit.keystone-mysql-router/1.juju-log Running legacy hooks/start.
unit-keystone-mysql-router-0: 11:37:04 INFO unit.keystone-mysql-router/0.juju-log Running legacy hooks/start.
unit-keystone-mysql-router-2: 11:37:04 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:05 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:05 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:05 INFO juju.worker.uniter.operation ran "backend-database-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:06 INFO juju.worker.uniter.operation ran "database-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:06 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:06 INFO juju.worker.uniter.operation ran "database-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:07 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:07 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:07 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:08 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:08 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:08 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:08 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:09 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:09 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:09 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:10 INFO juju.worker.uniter.operation ran "backend-database-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:10 INFO juju.worker.uniter.operation ran "backend-database-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:10 INFO juju.worker.uniter.operation ran "backend-database-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:11 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:11 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:11 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:11 INFO juju.worker.uniter.operation ran "database-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:12 INFO juju.worker.uniter.operation ran "backend-database-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:12 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:12 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:13 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:13 INFO juju.worker.uniter.operation ran "backend-database-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:13 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:14 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:14 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:14 INFO juju.worker.uniter.operation ran "backend-database-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:15 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:15 INFO juju.worker.uniter.operation ran "backend-database-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:15 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:16 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:16 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:16 INFO juju.worker.uniter.operation ran "backend-database-relation-joined" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:16 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:17 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:23 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:23 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:23 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:27 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:27 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:27 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:28 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:28 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:28 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:28 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:29 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:29 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:37 INFO unit.keystone-mysql-router/0.juju-log backend-database:11: database created at 2022-11-30 11:37:37.884263
unit-keystone-mysql-router-2: 11:37:37 INFO unit.keystone-mysql-router/2.juju-log backend-database:11: database created at 2022-11-30 11:37:37.897890
unit-keystone-mysql-router-1: 11:37:37 INFO unit.keystone-mysql-router/1.juju-log backend-database:11: database created at 2022-11-30 11:37:37.907434
unit-keystone-mysql-router-0: 11:37:38 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:38 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:38 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:37:39 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:37:39 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:37:39 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:38:02 INFO unit.keystone-mysql-router/2.juju-log backend-database:11: read-only-endpoints changed on 2022-11-30 11:38:02.468297
unit-keystone-mysql-router-1: 11:38:02 INFO unit.keystone-mysql-router/1.juju-log backend-database:11: read-only-endpoints changed on 2022-11-30 11:38:02.477394
unit-keystone-mysql-router-0: 11:38:02 INFO unit.keystone-mysql-router/0.juju-log backend-database:11: read-only-endpoints changed on 2022-11-30 11:38:02.553478
unit-keystone-mysql-router-2: 11:38:02 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:38:02 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:38:02 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:38:26 INFO unit.keystone-mysql-router/1.juju-log backend-database:11: read-only-endpoints changed on 2022-11-30 11:38:26.156025
unit-keystone-mysql-router-2: 11:38:26 INFO unit.keystone-mysql-router/2.juju-log backend-database:11: read-only-endpoints changed on 2022-11-30 11:38:26.157238
unit-keystone-mysql-router-0: 11:38:26 INFO unit.keystone-mysql-router/0.juju-log backend-database:11: read-only-endpoints changed on 2022-11-30 11:38:26.218326
unit-keystone-mysql-router-1: 11:38:26 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:38:26 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:38:26 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:41:09 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:41:27 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:41:28 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:41:29 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:41:29 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:41:29 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:41:30 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:41:30 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:41:30 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:41:30 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:41:30 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-0: 11:43:22 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-2: 11:43:22 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)
unit-keystone-mysql-router-1: 11:43:22 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch)

Additional context
No response

JFYI, reported internally as https://warthogs.atlassian.net/browse/DPE-1114

Dear @javacruft , the issue reported here is resolved in PR#25.
Please check it from your side. Feel free to reopen the issue if necessary!

authors: javacruft taurus-forever


text:
Implement the legacy shared-db relation interface with the ability to scale

Issue
We do not have a functioning mysqlrouter charm.
Solution
Implement a subordinate mysqlrouter charm with the legacy shared-db interface and the ability to scale. The flow of the charm is described below:

components: consumer application, mysqlrouter, mysql database
The consumer application requests a database, username and hostname
mysqlrouter uses the mysql_client relation interface (new relation interface) to request a user with mysqlrouter extra_roles. this user is used to bootstrap mysqlrouter and create the requested consumer application user.

MySQLRouter is registered and run with systemd for resiliency.
Context
Below is a dump of a deployment with the legacy mysqlrouter charm
(venv) ubuntu@mysql1:~/jhack$ jhack utils show-relation mysql/1:db-router keystone-mysql-router/0:db-router
                                                                                relation data v0.3
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ relation (id: 4) ┃ keystone-mysql-router                                                           ┃ mysql                                                                          ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ relation name    │ db-router                                                                       │ db-router                                                                      │
│ interface        │ mysql-router                                                                    │ mysql-router                                                                   │
│ leader unit      │ 0                                                                               │ 1                                                                              │
├──────────────────┼─────────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────┤
│ application data │ ╭─────────────────────────────────────────────────────────────────────────────╮ │ ╭────────────────────────────────────────────────────────────────────────────╮ │
│                  │ │ <empty>                                                                     │ │ │ <empty>                                                                    │ │
│                  │ ╰─────────────────────────────────────────────────────────────────────────────╯ │ ╰────────────────────────────────────────────────────────────────────────────╯ │
│ unit data        │ ╭─ keystone-mysql-router/0* ──────────────╮                                     │ ╭─ mysql/1* ──────────────────────────────────────────────────────╮            │
│                  │ │                                         │                                     │ │                                                                 │            │
│                  │ │  MRUP_database         keystone         │                                     │ │  MRUP_allowed_units         "keystone-mysql-router/0"           │            │
│                  │ │  MRUP_hostname         10.154.46.110    │                                     │ │  MRUP_password              "S5zLSGcjYYhrfLpcdszW9nWHVZqHtBWh"  │            │
│                  │ │  MRUP_username         keystone         │                                     │ │  db_host                    "10.154.46.231"                     │            │
│                  │ │  mysqlrouter_hostname  10.154.46.110    │                                     │ │  mysqlrouter_allowed_units  "keystone-mysql-router/0"           │            │
│                  │ │  mysqlrouter_username  mysqlrouteruser  │                                     │ │  mysqlrouter_password       "TTsPxHZhf8HKL6rsTTWm4stMj7bVTdGs"  │            │
│                  │ ╰─────────────────────────────────────────╯                                     │ │  wait_timeout               3600                                │            │
│                  │                                                                                │ ╰─────────────────────────────────────────────────────────────────╯            │
└──────────────────┴─────────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────┘
(venv) ubuntu@mysql1:~/jhack$ jhack utils show-relation keystone-mysql-router/0:shared-db keystone/0:shared-db
                                                                                relation data v0.3
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ relation (id: 3) ┃ keystone                                                                        ┃ keystone-mysql-router                                                          ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ relation name    │ shared-db                                                                       │ shared-db                                                                      │
│ interface        │ mysql-shared                                                                    │ mysql-shared                                                                   │
│ leader unit      │ 0                                                                               │ 0                                                                              │
├──────────────────┼─────────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────┤
│ application data │ ╭─────────────────────────────────────────────────────────────────────────────╮ │ ╭────────────────────────────────────────────────────────────────────────────╮ │
│                  │ │ <empty>                                                                     │ │ │ <empty>                                                                    │ │
│                  │ ╰─────────────────────────────────────────────────────────────────────────────╯ │ ╰────────────────────────────────────────────────────────────────────────────╯ │
│ unit data        │ ╭─ keystone/0* ─────────────╮                                                   │ ╭─ keystone-mysql-router/0* ────────────────────────╮                          │
│                  │ │                           │                                                   │ │                                                   │                          │
│                  │ │  database  keystone       │                                                   │ │  allowed_units  keystone/0                        │                          │
│                  │ │  hostname  10.154.46.110  │                                                   │ │  db_host        127.0.0.1                         │                          │
│                  │ │  username  keystone       │                                                   │ │  db_port        3306                              │                          │
│                  │ ╰───────────────────────────╯                                                   │ │  password       S5zLSGcjYYhrfLpcdszW9nWHVZqHtBWh  │                          │
│                  │                                                                                │ │  wait_timeout   3600                              │                          │
│                  │                                                                                │ ╰───────────────────────────────────────────────────╯                          │
└──────────────────┴─────────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────┘

Release Notes
Implement the legacy shared-db relation interface with the ability to scale
Testing
Integration and unit tests to follow shortly

Thanks for the reviews! @paulomach Will probably look at better documentation once the new relation has been implemented

source: gharchive/pull-request

repo: canonical/mysql-router-operator

license_type: permissive

license_source: github-api


text:
test-branch/execute-job/comment-from-wrong-user-on-pr/eb85d703-5c42-4cab-9779-50000d6ac1b4

PR for testing aac2f275a135e988c6b9f2f781a506cbe02fd739

/canonical/self-hosted-runners/run-workflows aac2f275a135e988c6b9f2f781a506cbe02fd739

repo: canonical/repo-policy-compliance-tests


text:
test-branch/execute-job/fork-branch/430bf302-2427-4359-8e03-6e2da72c283e

PR for testing 632472de26ece2d49816211f958b8734f6ebf310

padding /canonical/self-hosted-runners/run-workflows 632472de26ece2d49816211f958b8734f6ebf310 padding

repo: canonical/repo-policy-compliance


text:
execute-job/wrong-commit-sha-on-pr/9ec8d6df-8f03-4b8f-a9cd-1486553dd658

PR for testing df29c1c4cb485620557393afb188445f1ed9dc97

/canonical/self-hosted-runners/run-workflows 92a3af83b63af8d423a2f2c104925ba7ffd08d0f


text:
🛑 Ubuntu.com is down

In f88bde1, Ubuntu.com (https://ubuntu.com) was down:

HTTP code: 503
Response time: 514 ms

Resolved: Ubuntu.com is back up in 2b40767.

source: gharchive/issue

repo: canonical/upptime

license: MIT


text:
🛑 Ubuntu Security API - CVE page is down

In 94df6a2, Ubuntu Security API - CVE page (https://ubuntu.com/security/cves/CVE-2020-15180.json) was down:

HTTP code: 429
Response time: 112 ms

Resolved: Ubuntu Security API - CVE page is back up in 703874f.


text:
How to pass through "title" through TriggerAgent to EmailAgent

My input to the TriggerAgent contains parameters such as "title". However the TriggerAgent seems to only output a "message" and nothing else parametrized is passed through. Am I missing something how to get this out of a TriggerAgent event?
Thanks

Set keep_event to true
On Mon, Aug 8, 2016, 8:11 AM bobbysteel notifications@github.com wrote:

My input to the TriggerAgent contains parameters such as "title". However
the TriggerAgent seems to only output a "message" and nothing else
parametrized is passed through. Am I missing something how to get this out
of a TriggerAgent event?
Thanks
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/cantino/huginn/issues/1625, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABzHPkk1XW7dls99b-o5lJtQ6gEUWSuTks5qdxvxgaJpZM4Je-in
.

In the TriggerAgent? That's set already yes.

In my EmailAgent it seems to be getting the full event data but is not using it
"subject": "My alert: {{title}}",
"headline": "Thread: {{description}}",
"expected_receive_period_in_days": "30",
"recipients": "myemail"
The output is blank where title and description would go.

Ack nevermind. Realized the digest agent wouldn't handle this properly and I should use a plain EmailAgent instead. Sorry!

authors: bobbysteel brianpetro

repo: cantino/huginn

license: mit

license_source: bigquery


text:
New agent: Firebase

This agent would know how to edit data at a location in a Firebase tree using set(), update(), and push(). You could provide it some kind of authentication key if your Firebase required it.
One use-case I'm thinking about is having a Firebase agent listen to a Dropbox agent. Whenever a file is added or changed in a watched Dropbox folder, its corresponding entry (including its public url) could be updated in Firebase.
Then a website could generate its contents by pulling in resource urls from Firebase. Dropbox would act like the file store, and Firebase like the index / query engine. Changing the contents of the site would just be a matter of adding/removing files to/from the dropbox folder.
It would also be a really useful agent if you want some website dashboard to update in real-time to data collected by other agents.

Thats a very cool idea!  I think it wouldn't be too hard to write: on a
schedule, or when an event comes in, update the data in Firebase.
On Friday, February 6, 2015, Benny Lichtner notifications@github.com
wrote:

This agent would know how to edit data at a location in a Firebase
https://www.firebase.com/how-it-works.html tree using set(), update(),
and push(). You could provide it some kind of authentication key if your
Firebase required it.
One use-case I'm thinking about is having a Firebase agent listen to a
Dropbox agent. Whenever a file is added or changed in a watched Dropbox
folder, its corresponding entry (including its public url) could be updated
in Firebase.
Then a website could generate its contents by pulling in resource urls
from Firebase. Dropbox would act like the file store, and Firebase like the
index / query engine. Changing the contents of the site would just be a
matter of adding/removing files to/from the dropbox folder.
It would also be a really useful agent if you want some website dashboard
to update in real-time to data collected by other agents.
—
Reply to this email directly or view it on GitHub
https://github.com/cantino/huginn/issues/700.

Hoping for the other direction too: watch a firebase value, and fire an event when it changes

Please Firebase...
https://zapier.com/zapbook/firebase/

authors: JosefJezek bennlich cantino gabrielgrant


text:
dj: memory leak

delay_job is using more and more memory over time. it starts at ~150M and rises forever, slowly tough. As i have 16 workers, im affected much faster.
Any way to debug this? Im currently just restarting the workers every X hours.
Im using Web,Mail,MailDigest,XMPP,Trigger-Agents. With Web+Trigger being the most used.
Can foreman handle memory usage of dj? ie, restart a dj if mem is higher than X and no jobs currently running? restarting a worker after X jobs might also be an option.

Hey @K1773R, that's frustrating!  I haven't noticed that.  It's possible that upstart is killing and restarting them for me when ram runs low, but I haven't observed that.  Are you running any unusual Agents?  I suspect it's some Agent that I don't use often.

Are the ones i listed unusual agents?

XMPP seems the most likely.  If you disable it, does memory stabilize?

it leaks even when there are no xmpp agents...

That's very odd.  @dsander, have you seen memory issues with DJ?

@cantino I did not really monitor the memory usage on my server, but I doubt the issue is DJ iteself.
@K1773R debugging memory leaks in ruby is a bit annoying, if you notice the leak within a few minutes the quickest way would probably be to disable all but one agent type, restart your worker and check if the leak still exists.
If that is not an option and you are using 2.1+ I will try to add some sort of memory profiling option tomorrow.

im using ruby 2.2

I may have just run into this (where 'just' == sometime early last evening).  Two virtual CPUs, four job runners, and both jobs and the web user interface died with a signal 137 after staying online for something like four days running.  I had to kill the delayed_job processes and restart from scratch to get Huginn running again.  As an experiment I've cut my delayed job fleet to three to see how long it takes to run out of RAM.
Ruby v2.1.5p273.

Did this issue start recently?

The hang-up happened some time last night (maybe 1900 hours PDT).  I last updated my running install on 19 August (commit c3e9545b6ab99ecaf29dcfca1c23b1ca8ef8f734), after which memory utilization started creeping up until something broke.  Going through log/development.log, there isn't anything in there because the error message was emitted by the shell Huginn was started from.  It didn't say which process received the out-of-memory signal, though.  I can fire up a few more DJ's to artifically use up RAM and see what part falls over if it would help.

What commit were you running before this, that seemed more stable?  And are
you running in RAILS_ENV=production or RAILS_ENV=development?
On Sat, Aug 22, 2015 at 7:01 PM, The Doctor notifications@github.com
wrote:

The hang-up happened some time last night (maybe 1900 hours PDT). I last
updated my running install on 19 August (commit c3e9545
https://github.com/cantino/huginn/commit/c3e9545b6ab99ecaf29dcfca1c23b1ca8ef8f734),
after which memory utilization started creeping up until something broke.
Going through log/development.log, there isn't anything in there because
the error message was emitted by the shell Huginn was started from. It
didn't say which process received the out-of-memory signal, though. I can
fire up a few more DJ's to artifically use up RAM and see what part falls
over if it would help.
—
Reply to this email directly or view it on GitHub
https://github.com/cantino/huginn/issues/978#issuecomment-133770561.

I was running commit 9e8f9bf7f9a4e720b0fdb9ed1817f14a14809034 in development mode.  Still in development mode.

I have a stack trace from two days ago, from approximately the time of the crash.  I don't know if it'll help debug this issue (I'm working to reproduce it independently, which is why I jumped into this ticket's discussion thread) but I can throw it on pastebin if it'll help.

Sure, let's see it.

Well, we shouldn't be running Huginn in development mode for production use, but there was really very little code change between your two commits.
The only things we did that I could imagine affecting RAM were to remove spring and to add letter_opener_web.
You should ensure spring isn't being used with spring stop and you could try commenting out gem 'letter_opener_web' in the Gemfile, config.action_mailer.delivery_method = :letter_opener_web in development.rb, and mount LetterOpenerWeb::Engine, at: "/letter_opener" in routes.rb, and see if that helps.

Here it is: http://pastebin.com/TcBR22tq
Removing Spring has done a world of good - things are much more stable now!  I've been keeping an eye on it, and it appears to be gone for good.

That looks like the box running out of RAM or disk.  Was this correlated with ballooning RAM?  If so, do you know how high it got?

That was my analysis also - the amount of free RAM hit zero and the OOM killer took out a process.  I don't know which one because this is all the kernel message buffer has in it:
[3813894.197850] Killed process 11626 (ruby) total-vm:1596164kB, anon-rss:610824kB, file-rss:0kB

So, some aspect of Huginn got killed but it's not clear which one it was.  I'm trying to recreate this problem on my instance to help debug it, so I just captured the ruby processes running right now, and if one mysteriously dies I can compare the lists and see which one was the victim.

@K1773R I updated my memory profile branch. If you import this scenario the MemoryProfileAgent will count the allocated objects every 5 minutes and send the data to a PeakDetectionAgent, when a peak is detected a memory dump will be written to the configured directory. It works with multiple background workers but the results will probably look a bit strange because the job will run on a random worker process. To get an general overview (see how many objects are leaked over time) it would probably the best to run just one worker for a few hours.
To analyze a memory dump a small script is added to the bin directory:
Get an overview how many objects have been allocated (and still are) in which garbage collection generation run: ruby bin/analyze.rb <pathto.json>. To show which file and line allocated the object add the generation as second parameter: ruby bin/analyze.rb <pathto.json> <generation number>
I ran the branch over night on my production server and also see a small leak of objects but was not able to find the source, if you are running more agents the results might be more conclusive.

@dsander switching to a single worker is currently not really an option.
how can i migrate from cantino/huginn master to your branch and afterwards back w/o breaking things?

My branch is based on the current master branch and just adds the memory profile agent. As long as you delete that agent before you switch back there should not be any problems.

Data!
06:28:11 dj1.1      | script/delayed_job:5:in `<main>'
06:28:52 schedule.1 | Queuing event propagation
06:28:52 schedule.1 | Queuing schedule for every_1m
06:29:51 schedule.1 | Queuing schedule for every_2m
06:29:54 schedule.1 | Queuing event propagation
06:29:54 schedule.1 | Queuing schedule for every_1m
06:31:17 schedule.1 | Queuing event propagation
06:31:44 twitter.1  | Man in stable condition after East Oakland shooting http://t.co/owRD0aeFBa
06:31:49 twitter.1  |  -> Lifeline: Monitor Twitter for keywords
06:34:13 dj1.1      | exited with code 137
06:34:13 dj1.1      | Killed
06:34:13 system     | sending SIGTERM to all processes
06:34:13 web.1      | terminated by SIGTERM
06:34:13 schedule.1 | terminated by SIGTERM
06:34:13 twitter.1  | terminated by SIGTERM
06:34:13 dj0.1      | terminated by SIGTERM
06:34:13 dj2.1      | terminated by SIGTERM

dj.1 was killed with a signal 137 (out of memory).  The 'system` process detected this and sent SIGTERMs to the web server, scheduler, Twitter subsystem, and dj0.1 and dj2.1.  However, not everything is responding to SIGTERM correctly.  From my currently running process list:
14106 pts/5    Sl    13:20 /home/drwho/.rvm/rubies/ruby-2.1.5/bin/ruby bin/rails server -b127.0.0.1
14114 pts/5    Sl   132:16 ruby script/delayed_job -i 0 run
14120 pts/5    Rl   136:16 ruby script/delayed_job -i 2 run

@virtadpt can you try @dsander's MemoryProfilingAgent?

How up to date is @dsander's fork of the codebase?

I'm trying to recreate the OOM problem by force by manually starting DJs.  I'm capturing stdout to a logfile.  When I have something I'll comment on this issue.

When jobs aren't running, memory utilization doesn't change (e.g., 485 MB free and holding).
I'm up to 9 delayed job runners now with my usual agent load of approximately 750 agents.

Results!  I'll post what I got in separate Screen terminals in successive comments.
22:25:22 schedule.1 |   69896720     Mysql2::Error: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111): BEGIN
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:299:in `query'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:299:in `block in execute'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:473:in `block in log'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:467:in `log'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:299:in `execute'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in `execute'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:315:in `begin_db_transaction'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract/transaction.rb:144:in `initialize'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract/transaction.rb:170:in `new'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract/transaction.rb:170:in `begin_transaction'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract/transaction.rb:187:in `within_new_transaction'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/transactions.rb:220:in `transaction'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/transactions.rb:344:in `with_transaction_returning_status'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/transactions.rb:286:in `block in save'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/transactions.rb:301:in `rollback_active_record_state!'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/transactions.rb:285:in `save'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/backend/base.rb:35:in `block (2 levels) in enqueue'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `call'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `block in initialize'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `call'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `execute'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:40:in `run_callbacks'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/backend/base.rb:33:in `block in enqueue'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/backend/base.rb:32:in `tap'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/backend/base.rb:32:in `enqueue'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/message_sending.rb:12:in `method_missing'
22:25:22 schedule.1 |   69896720       /home/drwho/huginn/lib/huginn_scheduler.rb:153:in `block in run_schedule'
22:25:22 schedule.1 |   69896720       /home/drwho/huginn/lib/huginn_scheduler.rb:192:in `block (2 levels) in with_mutex'
22:25:22 schedule.1 |   69896720       /home/drwho/huginn/lib/huginn_scheduler.rb:191:in `synchronize'
22:25:22 schedule.1 |   69896720       /home/drwho/huginn/lib/huginn_scheduler.rb:191:in `block in with_mutex'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'
22:25:22 schedule.1 |   69896720       /home/drwho/huginn/lib/huginn_scheduler.rb:190:in `with_mutex'
22:25:22 schedule.1 |   69896720       /home/drwho/huginn/lib/huginn_scheduler.rb:151:in `run_schedule'
22:25:22 schedule.1 |   69896720       /home/drwho/huginn/lib/huginn_scheduler.rb:129:in `block (2 levels) in run!'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:234:in `call'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:234:in `do_call'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:258:in `do_trigger'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:300:in `block (3 levels) in start_work_thread'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:303:in `call'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:303:in `block (2 levels) in start_work_thread'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:289:in `loop'
22:25:22 schedule.1 |   69896720       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:289:in `block in start_work_thread'
22:25:22 schedule.1 |   69896720   tz:
22:25:22 schedule.1 |   69896720     ENV['TZ']:
22:25:22 schedule.1 |   69896720     Time.now: 2015-09-04 22:25:21 +0000
22:25:22 schedule.1 |   69896720   scheduler:
22:25:22 schedule.1 |   69896720     object_id: 87855320
22:25:22 schedule.1 |   69896720     opts:
22:25:22 schedule.1 |   69896720       {:frequency=>"1.0"}
22:25:22 schedule.1 |   69896720       frequency: 1.0
22:25:22 schedule.1 |   69896720       scheduler_lock: #<Rufus::Scheduler::NullLock:0x0000000a7a7100>
22:25:22 schedule.1 |   69896720       trigger_lock: #<Rufus::Scheduler::NullLock:0x0000000a7a70d8>
22:25:22 schedule.1 |   69896720     uptime: 21669.938076734543 (6h1m9s938)
22:25:22 schedule.1 |   69896720     down?: false
22:25:22 schedule.1 |   69896720     threads: 5
22:25:22 schedule.1 |   69896720       thread: #<Thread:0x0000000a7a6ed0>
22:25:22 schedule.1 |   69896720       thread_key: rufus_scheduler_87855320
22:25:22 schedule.1 |   69896720       work_threads: 4
22:25:22 schedule.1 |   69896720         active: 3
22:25:22 schedule.1 |   69896720         vacant: 1
22:25:22 schedule.1 |   69896720         max_work_threads: 28
22:25:22 schedule.1 |   69896720       mutexes: {}
22:25:22 schedule.1 |   69896720     jobs: 44
22:25:22 schedule.1 |   69896720       at_jobs: 0
22:25:22 schedule.1 |   69896720       in_jobs: 0
22:25:22 schedule.1 |   69896720       every_jobs: 16
22:25:22 schedule.1 |   69896720       interval_jobs: 0
22:25:22 schedule.1 |   69896720       cron_jobs: 28
22:25:22 schedule.1 |   69896720     running_jobs: 3
22:25:22 schedule.1 |   69896720     work_queue: 0
22:25:22 schedule.1 | } 69896720 .
22:25:22 schedule.1 | { 69995240 rufus-scheduler intercepted an error:
22:25:22 schedule.1 |   69995240   job:
22:25:22 schedule.1 |   69995240     Rufus::Scheduler::EveryJob "2m" {}
22:25:22 schedule.1 |   69995240   error:
22:25:22 schedule.1 |   69995240     69995240
22:25:22 schedule.1 |   69995240     ActiveRecord::StatementInvalid
22:25:22 schedule.1 |   69995240     Mysql2::Error: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111): BEGIN
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:299:in `query'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:299:in `block in execute'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:473:in `block in log'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:467:in `log'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:299:in `execute'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in `execute'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:315:in `begin_db_transaction'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract/transaction.rb:144:in `initialize'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract/transaction.rb:170:in `new'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract/transaction.rb:170:in `begin_transaction'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract/transaction.rb:187:in `within_new_transaction'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/transactions.rb:220:in `transaction'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/transactions.rb:344:in `with_transaction_returning_status'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/transactions.rb:286:in `block in save'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/transactions.rb:301:in `rollback_active_record_state!'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/transactions.rb:285:in `save'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/backend/base.rb:35:in `block (2 levels) in enqueue'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `call'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `block in initialize'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `call'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `execute'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:40:in `run_callbacks'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/backend/base.rb:33:in `block in enqueue'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/backend/base.rb:32:in `tap'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/backend/base.rb:32:in `enqueue'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/message_sending.rb:12:in `method_missing'
22:25:22 schedule.1 |   69995240       /home/drwho/huginn/lib/huginn_scheduler.rb:153:in `block in run_schedule'
22:25:22 schedule.1 |   69995240       /home/drwho/huginn/lib/huginn_scheduler.rb:192:in `block (2 levels) in with_mutex'
22:25:22 schedule.1 |   69995240       /home/drwho/huginn/lib/huginn_scheduler.rb:191:in `synchronize'
22:25:22 schedule.1 |   69995240       /home/drwho/huginn/lib/huginn_scheduler.rb:191:in `block in with_mutex'
22:25:22 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'
22:25:23 schedule.1 |   69995240       /home/drwho/huginn/lib/huginn_scheduler.rb:190:in `with_mutex'
22:25:23 schedule.1 |   69995240       /home/drwho/huginn/lib/huginn_scheduler.rb:151:in `run_schedule'
22:25:23 schedule.1 |   69995240       /home/drwho/huginn/lib/huginn_scheduler.rb:129:in `block (2 levels) in run!'
22:25:23 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:234:in `call'
22:25:23 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:234:in `do_call'
22:25:23 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:258:in `do_trigger'
22:25:23 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:300:in `block (3 levels) in start_work_thread'
22:25:23 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:303:in `call'
22:25:23 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:303:in `block (2 levels) in start_work_thread'
22:25:23 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:289:in `loop'
22:25:23 schedule.1 |   69995240       /home/drwho/.rvm/gems/ruby-2.1.5/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:289:in `block in start_work_thread'

Nothing emitted on stderr.  Nothing in ~/stderr.log.

In one of the extra workers I'd spawned:
Agent#563: Exception during check. Mysql2::Error: Lost connection to MySQL server during query: SELECT  agent_logs.id FROM `agent_logs` WHERE `agent_logs`.`agent_id` = 563  ORDER BY agent_logs.id desc LIMIT 1 OFFSET 249: /home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:299:in `query'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:299:in `block in execute'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:473:in `block in log'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_adapter.rb:467:in `log'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:299:in `execute'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/mysql2_adapter.rb:231:in `execute'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/mysql2_adapter.rb:235:in `exec_query'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:347:in `select'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activerecord-4.2.2/lib/active_record/relation/calculations.rb:180:in `pluck'
/home/drwho/huginn/app/models/agent_log.rb:22:in `log_for_agent'
/home/drwho/huginn/app/models/agent.rb:212:in `log'
/home/drwho/huginn/app/concerns/dry_runnable.rb:61:in `log'
/home/drwho/huginn/app/models/agents/imap_folder_agent.rb:358:in `each_unread_mail'
/home/drwho/huginn/app/models/agents/imap_folder_agent.rb:196:in `check'
/home/drwho/huginn/app/jobs/agent_check_job.rb:7:in `perform'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activejob-4.2.2/lib/active_job/execution.rb:32:in `block in perform_now'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:117:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:117:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:505:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:505:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:343:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:343:in `block (2 levels) in simple'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activejob-4.2.2/lib/active_job/logging.rb:23:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activejob-4.2.2/lib/active_job/logging.rb:23:in `block (4 levels) in <module:Logging>'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/notifications.rb:164:in `block in instrument'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/notifications.rb:164:in `instrument'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activejob-4.2.2/lib/active_job/logging.rb:22:in `block (3 levels) in <module:Logging>'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activejob-4.2.2/lib/active_job/logging.rb:43:in `block in tag_logger'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/tagged_logging.rb:68:in `block in tagged'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/tagged_logging.rb:26:in `tagged'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/tagged_logging.rb:68:in `tagged'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activejob-4.2.2/lib/active_job/logging.rb:43:in `tag_logger'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activejob-4.2.2/lib/active_job/logging.rb:19:in `block (2 levels) in <module:Logging>'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:441:in `instance_exec'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:441:in `block in make_lambda'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:342:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:342:in `block in simple'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:497:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:497:in `block in around'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:505:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:505:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:92:in `_run_callbacks'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:776:in `_run_perform_callbacks'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activesupport-4.2.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activejob-4.2.2/lib/active_job/execution.rb:31:in `perform_now'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activejob-4.2.2/lib/active_job/execution.rb:21:in `execute'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/activejob-4.2.2/lib/active_job/queue_adapters/delayed_job_adapter.rb:34:in `perform'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/backend/base.rb:94:in `block in invoke_job'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `block in initialize'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `execute'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/backend/base.rb:91:in `invoke_job'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/worker.rb:204:in `block (2 levels) in run'
/home/drwho/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/timeout.rb:91:in `block in timeout'
/home/drwho/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/timeout.rb:101:in `call'
/home/drwho/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/timeout.rb:101:in `timeout'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/worker.rb:204:in `block in run'
/home/drwho/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/worker.rb:203:in `run'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/worker.rb:280:in `block in reserve_and_run_one_job'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `block in initialize'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `execute'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/worker.rb:280:in `reserve_and_run_one_job'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/worker.rb:187:in `block in work_off'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/worker.rb:186:in `times'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/worker.rb:186:in `work_off'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/worker.rb:150:in `block (4 levels) in start'
/home/drwho/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/worker.rb:149:in `block (3 levels) in start'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `execute'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/worker.rb:148:in `block (2 levels) in start'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/worker.rb:147:in `loop'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/worker.rb:147:in `block in start'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/plugins/clear_locks.rb:7:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:79:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:61:in `block in initialize'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:79:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:79:in `block in add'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:66:in `execute'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/worker.rb:146:in `start'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/command.rb:124:in `run'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/command.rb:112:in `block in run_process'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/daemons-1.1.9/lib/daemons/application.rb:255:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/daemons-1.1.9/lib/daemons/application.rb:255:in `block in start_proc'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/daemons-1.1.9/lib/daemons/application.rb:264:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/daemons-1.1.9/lib/daemons/application.rb:264:in `start_proc'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/daemons-1.1.9/lib/daemons/application.rb:296:in `start'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/daemons-1.1.9/lib/daemons/controller.rb:73:in `run'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/daemons-1.1.9/lib/daemons.rb:197:in `block in run_proc'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/daemons-1.1.9/lib/daemons/cmdline.rb:109:in `call'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/daemons-1.1.9/lib/daemons/cmdline.rb:109:in `catch_exceptions'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/daemons-1.1.9/lib/daemons.rb:196:in `run_proc'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/command.rb:110:in `run_process'
/home/drwho/.rvm/gems/ruby-2.1.5/gems/delayed_job-4.0.6/lib/delayed/command.rb:86:in `daemonize'
script/delayed_job:5:in `<main>'
Mysql2::Error: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111): UPDATE `delayed_jobs` SET `delayed_jobs`.`locked_by` = NULL, `delayed_jobs`.`locked_at` = NULL WHERE `delayed_jobs`.`locked_by` = 'delayed_job.6 host:exocortex pid:25535'

The same stack trace happened in all of the extra DJs I spawned to force this to happen.  It killed almost all of the spares I spawned to test this; a few survived because they were running at the time the others died, but now that the scheduler's died it's sitting there waiting for things to do.
Looking at the process list:
...
8881 pts/3    Sl     6:52 /home/drwho/.rvm/rubies/ruby-2.1.5/bin/ruby bin/rails server -b127.0.0.1
8884 pts/3    Sl     1:20 bin/schedule.rb
...

From the kernel message buffer:
[714345.093138] Out of memory: Kill process 1025 (mysqld) score 146 or sacrifice child
[714345.093202] Killed process 1025 (mysqld) total-vm:1794572kB, anon-rss:593576kB, file-rss:0kB
[714345.795771] init: mysql main process (1025) killed by KILL signal
[714345.795802] init: mysql main process ended, respawning
[714345.887978] audit_printk_skb: 18 callbacks suppressed
[714345.887985] type=1400 audit(1441405520.696:17): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/mysqld" pid=27440 comm="apparmor_parser"

I'm tweaking MySQL a little bit to have a larger thread cache.  Looking at the MySQL log file, mysqld was killed and auto-respawned during the scheduler blowing up.

I don't know if any of this helps, but I think that recreates this bug under more controlled conditions.

@virtadpt I just updated the branch to include the latest changes from huginn/master
The MemoryProfile agent beautifully show that we are in fact slowly leaking objects even on my instace with just a few dozen agents. However I was not able to pinpoint the source of the leak using the memory dumps.
Not sure what you are trying to do by spawning more DJ workers your server can fit into RAM. Processes killed by the out of memory killer do not show a useful stackstrace. The oom killer just sends SIGTERM or SIGKILL to a random process to ensure some memory is available for the system itself.

Got it.  I'm going to cut over to it this afternoon.  Hypothetically speaking, would there be any harm in cutting between your branch and @cantino's branch while using the same database in the future?
As for your question, I was trying to recreate the problem in a more controlled fashion and see what information, if any, I could glean from it to help debug it.  If the information isn't helpful, I'm sorry, I was just trying to help even though I'm not a Ruby coder.

@dsander Got a crash on your memory profiling branch.  Rather than paste lots of stuff in comments, here's a pastebin link to everything Huginn printed when it died: http://pastebin.com/KRrSbSFj
The Rails server and DJs 0 and 1 are still running, it's only dj.2 that died.  Trying to log in, I get a blank page.  Curiously, schedule.rb is still up.

Got it. I'm going to cut over to it this afternoon. Hypothetically speaking, would there be any harm in cutting between your branch and @cantino's branch while using the same database in the future?

If you remove the MemoryProfileAgent before you switch the branch it should be fine.
The SEGFAULT looks odd, it might be caused by the MemoryProfileAgent trying to create an object dump. Can you remove the PeakDetectionAgent from the sources of the MemoryProfileAgent?

Will do.  Just disabling that agent didn't help, it caused a bunch of dead jobs to pile up in the queue.
Control link deleted.
I've got a bunch of object dumps for you in /tmp if you think they'll be of help debugging this problem.  I will be AFK for most of today so here's a link to  them:
https://www.dropbox.com/sh/ry6a63dg6b8imk7/AAC_m9dOPpYrVAvck1hrQY4La?dl=0

Factoid: Six days ago I bumped my VPS to 8 GB/4CPUs and kept Huginn configured for three Delayed Job workers and 846 running agents, and I haven't seen any problems or gotten any OOM crashes in Huginn (knock on wood).

At 1859 hours GMT on 20 September 2015, Huginn finally crashed with an OOM condition.

It seems like we must have a memory leak.  Was the MemoryProfileAgent helpful?

@cantino It was not.  I re-imported the MemoryProfileAgent and Huginn died within eleven minutes (using the new VPS' stats) with much the same stack trace as before.
I'm not sure what to do.  I've been able to replicate the situation in @K1773R's ticket pretty reliably, but I'm no closer to solving it than I was before.  I'm considering upgrading my ruby install to the latest and greatest in the v2.2.x line to see what happens, but I don't know if it'll be worth doing so.

It's odd that I haven't been seeing memory issues, but it's possible that upstart is periodically restarting my Huginn process..
@dsander, any thoughts?

If Upstart is periodically restarting Huginn you probably won't know it, but you will see it in the Upstart logfiles.

@cantino I don't think upstart is automatically restarting the workers. About the leak I am pretty much out of ideas, we are leaking objects (i saw a jump of the allocated object count from 600k to over 1m a few times), but do not know how we can locate the it.
My memory allocation dumps did not help, they just showed some obvious allocations related to ActiveRecord and Postgres.
@virtadpt I think the easiest fix would be to restart your DJ workers when they hit a memory limit (monit is great at motinoring processes, a more modern option would be inspeqtor)

im currently just resarting all my workers once a day via cron. not a great option as i waste RAM, but it works.

I'll see if I can figure out the leakage.  Over what kind of timespan are
we seeing it?  I wonder if NewRelic would help find it?
On Monday, September 21, 2015, Dominik Sander notifications@github.com
wrote:

@cantino https://github.com/cantino I don't think upstart is
automatically restarting the workers. About the leak I am pretty much out
of ideas, we are leaking objects (i saw a jump of the allocated object
count from 600k to over 1m a few times), but do not know how we can locate
the it.
My memory allocation dumps did not help, they just showed some obvious
allocations related to ActiveRecord and Postgres.
@virtadpt https://github.com/virtadpt I think the easiest fix would be
to restart your DJ workers when they hit a memory limit (monit
https://mmonit.com/ is great at motinoring processes, a more modern
option would be inspeqtor http://contribsys.com/inspeqtor/)
—
Reply to this email directly or view it on GitHub
https://github.com/cantino/huginn/issues/978#issuecomment-142112175.

It might also be helpful to have huginn/bin/schedule.rb and huginn/twitter_stream.rb write PID files to huginn/tmp/pids/, if only for the purposes of monitoring.

I'll try to look into the RAM issue this weekend.  Over what timespan are we seeing it climb, and by how much? What types of Agents are you running?  Does it climb if all Agents are disabled?

i only saw it rise when the worker were running agents, in idle it dosnt increase. most of my agents are WebSiteAgent with multiple extraction fields. ~5% regex, 95% xpath.

Memory utilization climbs steadily over six or seven days, resulting in a crash or lockup between days six and seven.  I'm running almost 900 agents covering about half of the possible agent types.  I don't know if it climbs with agents disabled because it's not really feasible for me to disable all of them.

@dsander, does your profiler do heap dumping too? http://samsaffron.com/archive/2015/03/31/debugging-memory-leaks-in-ruby

@cantino Yes, I used that blog post as a base for my branch and the "analyzer".

Would it be helpful if I ran it on my instance? Do you have any hints or guesses on what's happening?

I did not find anything useful in the dumps I analyzed, there was too much AR related noise in them, maybe you have more luck.
Just running the scenario I linked might give some hints when you can correlate an object count spike to the runtime of an agent, but that also did not work for me.

Okay, I have your Scenario running now.

You need to watch the disk usage though, per default the PeakDetectorAgent in the scenario will trigger a dump for every detected peak. I ran out of space of my root disk once :smile:

Thanks! Just added a monitor to keep an eye on that.

I'm seeing the memory leak too after a few days running.  I suspect it may be the cause of some of my crashes.  According to #1114, this will happen with a completely empty agent too, which makes me suspect DJ or the scheduler?

@cantino My test instance crashed for the same reason today, I recently added some more debugging information and found that multiple instances of TwitterStreamAgent::Worker were still in memory. Applied a fix and I am now waiting if that fixes the crashes.
This does not explain #1114 though.

My instance (now running with the multithreaded scheduler) is doing the same thing.  I've had two lockups in as many days.  Until I purged the pending jobs this morning, I had a backlog of 7900 jobs waiting to run.
I'm interested to see if commit 07e2d9c4402a2c739aa5e1063febe3efa6930845 fixes it.

I'm reading in heroku discussion forum that some people was experimenting memory leaks running the latests version of Ruby. Have anybody tried running huggin with Ruby 2.0.0?
More info:
https://discussion.heroku.com/t/memory-leaks-with-ruby-2-1-2/644/23

The leak's happening in v2.1.5, too.

Are you still seeing a steady climb of memory?

Yes.  However, JabberAgent's been locking up Huginn way more than OOMs have been killing Huginn lately (two weeks or therabouts).

@virtadpt thanks for telling, was just about to update huginn and a lockup with jabberagent would be horrible for me. i'll try once this issue is fixed.

Pretty sure it is not an issue with the jabber agent in general, as far as I can remember we did not change it for a least a few months and I am using it as a receiver (JabberAgent::Worker) without any issues.

Insofar as the memory leak, I think the article that @jn11 linked to has a lot to do with it.
I'm using JabberAgent as a sender, which might be the problem.  But that's a separate issue (literally).

Thought of this issue when I read this How Ruby 2.2 can cause an out-of-memory server crash

@brianpetro  versions below 2.2 are affected too

I think this would only happen if recursion was interrupted by a signal, so the GC got disabled?  Then we wouldn't see a slow climb in memory, but probably a fairly quick one?  But I could totally be wrong.  If you're having memory issues, would you mind trying the newer version of Ruby and seeing if it helps?  It'd be awesome if this fixed our issues.

Jumping in here to help troubleshoot. As I was fiddling around with Huginn today I noticed that Celluloid 0.16.0 gets installed during setup. There is a known memory leak with that version of Celluloid, see https://github.com/celluloid/celluloid/issues/455 You might look at requiring Celluloid version 0.15.2 or 0.17.2, see https://gist.github.com/gazay/3b518f72266b5a7e88ff
I also bookmarked a list of "leaky gems" and delayed_job and therubyracer, both of which are in the Huginn gemfile, are listed. See https://github.com/ASoftCo/leaky-gems
When I set up Huginn today I noticed that delayed_job version 4.0.6 was installed -- that version is known to have memory leaks: https://github.com/collectiveidea/delayed_job/issues/776
therubyracer was installed with gem version 0.12.2 which fixes the memory leak that was previously reported, so that one should be fine. https://github.com/cowboyd/therubyracer/pull/336
Hope that helps!

Thanks @wasafiri, this is very helpful!  I'll try updating the Gemfile.

Does the memory leak persist @virtadpt @cantino ?

I haven't had issues recently, but it wasn't affecting me much. @k1773r?
On Wed, Feb 3, 2016 at 5:14 AM David Bahar notifications@github.com wrote:

Does the memory leak persist @virtadpt https://github.com/virtadpt
@cantino https://github.com/cantino ?
—
Reply to this email directly or view it on GitHub
https://github.com/cantino/huginn/issues/978#issuecomment-179222574.

I think so.  I'm running the threaded job runner (have been since I got Huginn into production mode) and it's been remarkably stable since I disabled all of my JabberAgents in favor of BeeperAgent instances.

@cantino I should have time for an upgrade at the weekend. i'll report back.

I've been having memory leak issues for the past couple of months. I'm using the latest codebase on an ubuntu xenial box with 1 GB memory.
Huginn runs fine for a day - and then the server get's very sluggish - the threaded.rb process starts at 5% and then steadily increases - after a day it's around 16%....
The logs don't seem to show any errors.

That's not good. I've seen my RAM usage climb once in the last month I think. Seems like there's still some subtle issue.

I did some monitoring of my installation and it looks like the memory usage stabilized after a few days. The amount of RAM used for the twitter stream process still surprises me a bit.
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                      NAMES
8c4a99ed0f4c        cantino/huginn-single-process    "/scripts/init bin/tw"   4 minutes ago       Up 4 minutes        3000/tcp                   huginndocker_huginn_tw_1
6a9a28389370        cantino/huginn-single-process    "/scripts/init bin/sc"   4 minutes ago       Up 4 minutes        3000/tcp                   huginndocker_huginn_scheduler_1
2065a5122238        cantino/huginn-single-process    "/scripts/init"          4 minutes ago       Up 4 minutes        127.0.0.1:3000->3000/tcp   huginndocker_huginn_web_1
a859b884a9a4        cantino/huginn-single-process    "/scripts/init script"   4 minutes ago       Up 4 minutes        3000/tcp                   huginndocker_huginn_dj_1
cdd35eae5260        postgres:9.5                     "/docker-entrypoint.s"   4 minutes ago       Up 4 minutes        5432/tcp                   huginndocker_postgres_1
Mon Jan 16 17:47:37 CET 2017
CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O               BLOCK I/O             PIDS
8c4a99ed0f4c        0.12%               190.6 MiB / 31.32 GiB   0.59%               65.05 MB / 2.116 MB   0 B / 1.769 MB        0
6a9a28389370        0.07%               150.9 MiB / 31.32 GiB   0.47%               1.17 MB / 253.7 kB    0 B / 1.769 MB        0
2065a5122238        0.00%               262.1 MiB / 31.32 GiB   0.82%               1.513 MB / 602.9 kB   376.8 kB / 1.769 MB   0
a859b884a9a4        0.27%               162.6 MiB / 31.32 GiB   0.51%               1.355 MB / 450.1 kB   2.335 MB / 1.786 MB   0
cdd35eae5260        0.11%               69.21 MiB / 31.32 GiB   0.22%               2.09 MB / 61.83 MB    2.31 MB / 12.53 MB    0
Mon Jan 16 21:32:11 CET 2017
CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O               BLOCK I/O             PIDS
8c4a99ed0f4c        3.84%               225.7 MiB / 31.32 GiB   0.70%               862 MB / 148.5 MB     0 B / 1.769 MB        0
6a9a28389370        0.08%               190 MiB / 31.32 GiB     0.59%               1.717 MB / 981.1 kB   0 B / 1.769 MB        0
2065a5122238        0.00%               287.9 MiB / 31.32 GiB   0.90%               1.665 MB / 728.1 kB   376.8 kB / 1.769 MB   0
a859b884a9a4        0.12%               187.9 MiB / 31.32 GiB   0.59%               37.75 MB / 22.31 MB   2.335 MB / 1.786 MB   0
cdd35eae5260        1.38%               89 MiB / 31.32 GiB      0.28%               161.5 MB / 719.2 MB   2.31 MB / 1.092 GB    0
Tue Jan 17 22:15:49 CET 2017
CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O               BLOCK I/O             PIDS
8c4a99ed0f4c        2.10%               313.1 MiB / 31.32 GiB   0.98%               3.673 GB / 986.3 MB   0 B / 1.769 MB        0
6a9a28389370        0.08%               206.1 MiB / 31.32 GiB   0.64%               4.839 MB / 5.647 MB   0 B / 1.769 MB        0
2065a5122238        0.00%               448.4 MiB / 31.32 GiB   1.40%               16.38 MB / 12.57 MB   557.1 kB / 1.769 MB   0
a859b884a9a4        0.12%               189.3 MiB / 31.32 GiB   0.59%               274.9 MB / 165.4 MB   2.335 MB / 1.786 MB   0
cdd35eae5260        0.84%               113.2 MiB / 31.32 GiB   0.35%               1.05 GB / 1.598 GB    3.015 MB / 7.56 GB    0
Fri Jan 20 13:27:54 CET 2017
CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O               BLOCK I/O             PIDS
8c4a99ed0f4c        0.14%               307.8 MiB / 31.32 GiB   0.96%               1.023 GB / 2.93 GB    0 B / 1.769 MB        0
6a9a28389370        0.09%               209.5 MiB / 31.32 GiB   0.65%               12.73 MB / 17.55 MB   0 B / 1.769 MB        0
2065a5122238        0.00%               455.7 MiB / 31.32 GiB   1.42%               18.2 MB / 14.03 MB    557.1 kB / 1.769 MB   0
a859b884a9a4        0.12%               188.7 MiB / 31.32 GiB   0.59%               884.1 MB / 531.6 MB   2.335 MB / 1.786 MB   0
cdd35eae5260        0.06%               122.7 MiB / 31.32 GiB   0.38%               3.147 GB / 3.795 GB   3.727 MB / 23.39 GB   0

How high does your RAM usage get @HyShai? I know that Ruby's generational garbage collector can take a while before it reaches a steady state, which is always higher than where it starts.

authors: HyShai K1773R brianpetro cantino dsander jn11 virtadpt wasafiri


text:
Unable to reuse the pipeline definition

Hi ,
I didn't find anywhere in the doc on how to re-use the pipeline.
For eg:
var h=_([data])
var stage1=  _.pipeline(h.flatMap(h.wrapCallback(function1),
h.flatMap(h.wrapCallback(function2))
var stage2=  _.pipeline(h.flatMap(h.wrapCallback(function3),
h.flatMap(h.wrapCallback(function4))
     h.pipe(stage1).
        pipe(stage2).
        pipe(stage1).  // If stage1 is re-used again, it says you can't use an already consumed
                             //stream, you may need to fork/observe
        stopOnError().each();

As a workaround , I have defined stage3 whose pipeline definition is same as stage1.
Please let me know if there is a better way.
Thanks

You can't reuse pipelines. The pipeline method returns Highland through stream that, like all other Highland streams is one-time use.
If you want a custom transformation pipeline, use `seq

You can't reuse pipelines. The pipeline method returns Highland through stream that, like all other Highland streams is one-time use.
If you want a custom transformation pipeline, use seq and through.
var h=_([data])

var stage1 = _.seq(
    _.flatMap(_.wrapCallback(function1)),
    _.flatMap(_.wrapCallback(function2)))

var stage2 = _.seq(
    _.flatMap(_.wrapCallback(function3)),
    _.flatMap(_.wrapCallback(function4)))

h.through(stage1)
    .through(stage2)
    .through(stage1)
    ...

Note: Previous incomplete response was accidentally submitted.

Thanks Victor, that works...!!

authors: GolfRider vqvu

repo: caolan/highland