text:
Implement Secure Channel Protocol

Problem
Secure Channel Protocol defines the control plane for many facets of secure channel communication and security, including: message counter synchronization, reliable messaging acknowledgements.
https://github.com/CHIP-Specifications/connectedhomeip-spec/blob/master/src/secure_channel/Secure_Channel_Protocol.adoc
Proposed Solution
Implement the Secure Channel Protocol as a CHIP protocol, please note, the session establish protocol(Sigma, Spake2+) is not within the scope of this implementation.
@woody-apple @pan-apple @andy31415 @robszewczyk

I don't know which one would be the right forum to ask for this but, would it be possible to have access to the specifications repository?

I don't know which one would be the right forum to ask for this but, would it be possible to have access to the specifications repository?

Maybe contact Chris LaPré first for access

Thanks @yufengwangca I'll contact him then, thanks :D

We're tracking high level tasks like this in projects: https://github.com/project-chip/connectedhomeip/projects/36
Please file issues there as needed!

authors: glerchundi woody-apple yufengwangca


text:
[BUG] TC_EEVSE_2_3 does not run on EST timezone

Reproduction steps
Running something of the form:
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env out/test_env.yaml --script src/python_testing/TC_EEVSE_2_3.py'

fails on my machine with:
│      [2024-09-17 09:55:12.771290][TEST][STDOUT]    return fut.result()                                                                                │
│      [2024-09-17 09:55:12.771296][TEST][STDOUT]           ^^^^^^^^^^^^                                                                                │
│      [2024-09-17 09:55:12.771301][TEST][STDOUT]  File "/usr/local/google/home/andreilitvin/devel/connectedhomeip/src/python_testing/TC_EEVSE_2_3.py", line 270, in test_TC_EEVSE_2_3                                                               │
│      [2024-09-17 09:55:12.771307][TEST][STDOUT]    asserts.assert_less(next_start_time_epoch_s,                                                                                │
│      [2024-09-17 09:55:12.771313][TEST][STDOUT]  File "/usr/local/google/home/andreilitvin/devel/connectedhomeip/out/venv/lib/python3.11/site-packages/mobly/asserts.py", line 191, in assert_less                                                 │
│      [2024-09-17 09:55:12.771319][TEST][STDOUT]    _call_unittest_assertion(                                                                                │
│      [2024-09-17 09:55:12.771325][TEST][STDOUT]  File "/usr/local/google/home/andreilitvin/devel/connectedhomeip/out/venv/lib/python3.11/site-packages/mobly/asserts.py", line 50, in _call_unittest_assertion                                     │
│      [2024-09-17 09:55:12.771331][TEST][STDOUT]    raise signals.TestFailure(my_msg, extras=extras)                                                                                │
│      [2024-09-17 09:55:12.771337][TEST][STDOUT]mobly.signals.TestFailure: Details=779934015 not less than 779929140, Extras=None

The comparison of times seem to be timezone dependent.  This prevents me from being able to "run all CI tests" on my machine.
Bug prevalence
Always
GitHub hash of the SDK that was being used
514e810a7cfd3a8c99f612cbfe51a4e403f2ffd5
Platform
linux-x64

Specifically setting TZ=UTC makes the test pass however setting TZ=EST causes the faiulure for me.

@jamesharrow could you please check this?

source: gharchive/issue

repo: project-chip/connectedhomeip

license_type: permissive

license_source: github-api


text:
board name changed

if do not update here, build in vscode + docker fails

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

source: gharchive/pull-request


text:
[Silabs] Update matter support submodule pointer

Description
PR updates the silabs/matter_support submodule pointer to pull in updated files.
PR also modifies efr32_sdk.gni paths to point to the right ones.
Tests
CI & Manual tests

@mkardous-silabs should this be draft or is it ready for review?


text:
[telink] Update build instructions in Readme

Problem
What is being fixed?  Examples:
Current build instructions for telink example is out of date, need to be updated with latest changes.
Change overview
Update build instructions in Readme
Testing
How was this tested? (at least one bullet point required)

N/A, documentation change only.

@yufengwangca Conflicts?

Already done in #8661

authors: bzbarsky-apple yufengwangca


text:
LowParse.Low.Bytes: rlimit bump

Trying to fix everest CI (untested)

Can we get this one to master @tahina-pro ? https://github.com/project-everest/everparse/commit/f54dfab9a6aad32780ae910dc62a5a6b645642f3

repo: project-everest/everparse


text:
"Guard Mode"

Jacob on Slack had a cool idea, and I'm just gonna screencap it here:

So, basically a "rerun failed tests" button? Except wouldn't that have to recompile the tests and (without hot-swapping working) mean destroying the state about what failed? Might work better server-side...

Yeah I imagine it'd just keep running, watching for changes, and rerun as
appropriate when things changed. Definitely seems doable!
On Fri, Jun 17, 2016, 9:41 PM Max Goldstein notifications@github.com
wrote:

So, basically a "rerun failed tests" button? Except wouldn't that have to
recompile the tests and (without hot-swapping working) mean destroying the
state about what failed? Might work better server-side...
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/project-fuzzball/test/issues/7#issuecomment-226921634,
or mute the thread
https://github.com/notifications/unsubscribe/ABCxwLNxqBL6B6MDaszusOpLHe2sMqepks5qM3cJgaJpZM4I4trJ
.

authors: mgold rtfeldman

repo: project-fuzzball/test

license_source: bigquery


text:
POC Overview usage bar charts

Replace "Compute instance types" and "storage services" charts with bar charts. This would apply to Azure, Aws, and Ocp. Confirm with @nlcwong that the 30-60 data points looks ok.
Maybe make it into a two card width layout rather than three to accommodate the bar charts?
What happens with historical data with 6 data series?
Replace line charts with bar charts

FYI, @nlcwong @chargio
I captured some screen shots from a POC that utilizes bar charts. Had to make the bars thin, so it looks ok with 2 sets of data. It's too crowded with 4 sets of data, even with 2 cards per row.
OCP on AWS tab, 3 cards per row

OCP on AWS tab, 2 cards per row

This is what 2 cards per row look like when the window is resized smaller. This uses a multi-color theme, so the bars are more visible.
Note that the bars overlap because there simply is not enough space.

This is what 2 cards per row look like when the window is resized smaller. This uses a multi-color theme, so the bars are more visible.
Note that the bars overlap because there simply is not enough space.

This is an example of the OCP on AWS details historical charts. These charts show 6 sets of data.

Same historical charts, but without limits data

I agree that in this case it could be beneficial to show the two months one after the other

Meaning: not overlapping

@chargio doesn't matter if we show it "overlapping" or have it nov 1 - dec 31, if you look at it either way, having that many bars w/in the viewport that we have it is too much. the width is just not big enough for the amount of data points we want to show on the graph.  So, my question to you is if there is a real need to show usage vs. requests?  Can we just have a graph to just show usage and historical usage and another graph (not on the same graph) show requests and historical requests?

authors: chargio dlabrecq nlcwong

repo: project-koku/koku-ui


text:
Show cost instead of infrastructure_cost, which omits markup cost

For Ocp cloud infrastructure, we should display cost instead of infrastructure_cost, which omits markup cost.
Fixes https://github.com/project-koku/koku-ui/issues/1206

Codecov Report

Merging #1208 into master will not change coverage.
The diff coverage is 50%.

@@          Coverage Diff           @@
##           master   #1208   +/-   ##
======================================
  Coverage    75.4%   75.4%
======================================
  Files         227     227
  Lines        4448    4448
  Branches      867     867
======================================
  Hits         3354    3354
  Misses        968     968
  Partials      126     126

Impacted Files
Coverage Δ

...ages/ocpCloudDashboard/ocpCloudDashboardWidget.tsx
71.42% <50%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e7ef654...f82eb77. Read the comment docs.

authors: codecov-io dlabrecq


text:
Release 1.7.0

Adding UI Designs for Users and Groups

Following should be fixed -

Proper alignment in html and ts files.
data in html should come from ts files using angular data binding. It would be easier later to integrate with actual data.

Not valid

authors: AmiableAnil souvikmondal subranil

repo: project-sunbird/sunbird-mobile

license: MIT


text:
Profile & Cards Issues fix

Before - Header avatar Misaligned

After - Header avatar aligned

Before - Cards empty space in image

After - Cards empty space in image fixed

After - Last card fully visible and no white space

@bvinayakumar - Pls merge this PR.

repo: project-sunbird/sunbird-portal


text:
feat: adding model name column in service page

What does this PR do?
As recommended by @lstocchi adding a model name column to the Services list page
Screenshot / video of UI

What issues does this PR fix or reference?
Fixes https://github.com/projectatomic/ai-studio/issues/540
How to test this PR?

[x] unit tests has been provided

@lstocchi I went to have a look at how it's done throughout the app atm and there seems to be inconsistencies with it.

imo the left aligned looks better and that seems to be what is trying to be done throughout

@lstocchi I went to have a look at how it's done throughout the app atm and there seems to be inconsistencies with it.

imo the left aligned looks better and that seems to be what is trying to be done throughout

Thanks, i'll open an issue then 👍

authors: axel7083 ekidneyrh lstocchi

repo: projectatomic/ai-studio


text:
Commit branch: fix user param key

Commit branch was taken from 'git_commit' instead of 'git_branch' user
param.

OSBS-6599

Signed-off-by: Martin Bašti mbasti@redhat.com

No RN, fixing unreleased feature

repo: projectatomic/atomic-reactor


text:
Update podman_tutorial.md

Fix the familiarizing-yourself-with-podman link.

LGTM

LGTM

@mheon
The previous commition was directly operated in github page, and I added the sign & re-push.
Please check it, thanks!

@warmchang Signoff looks good, thanks

bot, add author to whitelist

LGTM, good for merge once tests are green

@rh-atomic-bot r+

authors: TomSweeneyRedHat mheon rhatdan warmchang

repo: projectatomic/libpod


text:
AWS Kubernetes Integration Doc

Complete guide for configuring an existing AWS-Kube cluster to use Calico
Modify existing Kube docs for clarity, accuracy

Just have to run through the commands on the minion and confirm it is working.

Done with this round of comments.  I'll take one more quick look when you're done addressing.

Back to you

One more comment you still need to address.

Ok - waiting on the calico-docker release now.

LGTM

authors: TrimBiggs alexhersh caseydavenport

repo: projectcalico/calico-docker


text:
Calico node not starting

I have installed this version of the calico for local cluster
I used those steps to make it work
https://projectcalico.docs.tigera.io/getting-started/kubernetes/self-managed-onprem/onpremises#install-calico-with-kubernetes-api-datastore-more-than-50-nodes
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.4/manifests/calico.yaml
I have manually changed the DNS on the machine to use 1.1.1.1
Here is the log I can see on the pods
2022-11-08 03:56:12.500 [ERROR][86] felix/health.go 296: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 1.1.1.1:53: no such host
2022-11-08 03:56:13.520 [ERROR][86] felix/health.go 296: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 1.1.1.1:53: no such host
2022-11-08 03:56:14.582 [ERROR][86] felix/health.go 296: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 1.1.1.1:53: no such host
2022-11-08 03:56:15.624 [ERROR][86] felix/health.go 296: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 1.1.1.1:53: no such host
2022-11-08 03:56:16.649 [ERROR][86] felix/health.go 296: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 1.1.1.1:53: no such host

Expected Behavior
Should be running
Current Behavior
Does not seems to start
Possible Solution
I was able to fix the issue if I change the host file and add localhost in it
nano /etc/hosts
# add this line
127.0.0.1       localhost

Steps to Reproduce (for bugs)
Context
Your Environment
onpremises version of Kubernetes
Kubernetes v1.25.3
Calico v3.24.4

@miaooss your fix seems reasonable to me. Generally you want localhost to be resolve without a DNS lookup and that's done via /etc/hosts.
An alternative would be to configure FELIX_HEALTHHOST=127.0.0.1 to configure Felix to bind to a specific IP address rather than localhost.

Sounds suspiciously similar to the issues in #6710. Was there no /etc/hosts at all here?

I found it weird that I have to change the /etc/hosts of the docker container to add localhost

Do we need to include the localhost change as part of the image container?
If not we can close this issue.

I found it weird that I have to change the /etc/hosts of the docker container to add localhost

You shouldn't need to modify localhost within the docker container. The localhost value should come from the node's /etc/hosts file, which is copied into the running host network container.
If your node's /etc/hosts file doesn't have a localhost entry defined, then neither will calico/node.

authors: caseydavenport miaooss wdoekes

repo: projectcalico/calico


text:
Fixed calicoctl not being able to list networkpolicies

Description
added permission to get and list networkpolicies.networking.k8s.io
Related issues/PRs
fixes #2749
Todos

[ ] Tests
[ ] Documentation
[ ] Release note

Release Note

None required

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Deploy preview for calico ready!
Built with commit c5d92ba9b0175eab9e0c08a08c745df0f939823a
https://deploy-preview-2750--calico.netlify.com

authors: CLAassistant caseydavenport fpicot


text:
Logic refactor

Description
This PR contains the following changes (some breaking-change in functions signature syntax):

removed internal gologger fatal and errors propagated upstream => Closes #47
Added DefaultClient => Unblocks https://github.com/projectdiscovery/mapcidr/issues/138
Fixed unwanted data removal on reused client => Closes #48 and closes https://github.com/projectdiscovery/interactsh/pull/427 (indirectly)
Single threaded as anyway multi-thread was ineffective due to unwanted data removal

Note: this PR might require a major release version due to breaking changes in the syntax

repo: projectdiscovery/asnmap


text:
add mit license for public-bugbounty-programs

Hi,
It is stated to be missing for the community so I wanted to add it.
References : https://github.com/projectdiscovery/public-bugbounty-programs/community

Regards,
@ismailtasdelen

Thank you for adding this.

authors: bauthard ismailtasdelen


text:
Scenario selection is lost when changing the language

New function
Set Scenario to Data Processing
Set language to C#

Result: scenario resets to Core, even though C# has Data Processing templates.

dd435c940ed8e2e73f6b3d66d70bdd245932ca48

authors: alrod davidebbo

repo: projectkudu/AzureFunctionsPortal


text:
Develop tab: remove label "Code" and just use filename

Label should just be the filename being edited. See screenshot below.

This has already been fixed.

authors: lindydonna mattchenderson


text:
Show which option is currently selected

e.g. if i'm switching environments and i'm currently on Staging, i'd like to see this highlighted in some way

also, if the current option is reselected, the callback shouldn't be called

Can we prioritize this? We're working on Checkout now and I want to make sure testers are aware if they are going to submit a real order.

This has been resolved with #31

authors: MichaelCampbell cassiewallace hkellaway

repo: prolificinteractive/Yoshi

license: mit


text:
[Enhancement]Value Placeholder'decimal point

Even integers will have decimal points (100.0), so I can't use [/scoreboard players set] through skillapi. I hope I can fix it

Which placeholder are you referring to exactly?

https://promcteam.com/wiki/index.php/Proskillapi:Placeholders
Is it one from this list?

With%sapi_ default_ Currentlevel% as an example
Skill 2.zip
/papi parse me %sapi_default_currentlevel%
This command will get an integer, but in this skill, you get a number with one decimal point

authors: Travja YYYYYYase

repo: promcteam/proskillapi


text:
Add ability to generate png graph on the alert for embedding with receivers

Both Slack and PagerDuty allows including one or more images with the generated alert. At least with PagerDuty the image needs to be accessible via https.
Alertmanager should have a way to generate a publicly accessible rendered image on the alert query, so that the image can be attached to the alert. This way the person receiving the alert could easily see a visual explanation on the alert context.
The generated image could at least show the alert expression, separately for the left and right side of the expression.
After thinking this a bit there's one way how this could be implemented:

Extend the Alert model to include the expression, perhaps broken down to left and right parts of the comparison.
Create a template language which generates a cryptographically signed url for a prometheus query, which would then be passed as a parameter from the Alert model mentioned above.
Create a http endpoint to Alertmanager which would render a png if provided with a correctly signed query. Signing is required because the alertmanager endpoint would need to be exposed to public internet for PagerDuty and Slack to be able to connect to it.

These steps would enable to construct appropriate alertmanager receiver route rules, which would be able to generate required image urls with the mentioned templating functionality.

We have that feature in a custom we hook reveiver but not with the original alert; we do add custom promql expressions as annotations

I will break this down into two feature requests:

Rendering images in Alertmanager: Alertmanager is notified by Prometheus that a given alert is firing. It does not send along the corresponding timeseries nor an image of the graph. Thereby Alertmanager by itself can not render the image.
One could expose an endpoint on Prometheus that would return an image given a Prometheus Query, which could then be combined with the Alert from Alertmanager. While this is technically possible I think this would include several problems:

Separation of concerns: Prometheus is a monitoring tool and time series database. All graphic-features have been kept as minimal as possible so far.
Rendering images of graphs is difficult to achieve in a generic way as there are so many options (file format, size, ...)

Hence I think using a custom webhook as @roidelapluie suggested is a great option. It would receive the alert from Alertmanager, retrieve the time series from Prometheus, render the graphic and send everything to your notification-system.

HTTP endpoint accepting signed queries: The Prometheus project does not provide authn authz or encryption features as of today. This might change but is unlikely to happen any time soon.

@garo what do you think?

The main reason for not accepting this feature request is that the alerting rule is not relevant to understand the context.
Some alerts have >, on() joins...  and when it is resolved you can not know if it was because of missing mectric or threshold... the alerting query is meaningless.

I understand @roidelapluie your reasoning that there are many cases where the alerting rule is not easily usable for this case.
Could the alerting rule have a field (label / annotation) where the user could describe a relevant query, which could then be rendered out and sent as an attachment? This approach would require the ability to specify multiple queries which would then be embedded in the same image.
Remember that the goal is not replicate a full featured visualisation backend such as Grafana, but just to provide a first look for the on-call engineer to get a context.
@mxinden How about Alertmanager would be the component doing the actual image rendering and not Prometheus? It is true that adding image rendering components (I'm not sure how the current prometheus UI handles image rendering? Is it done fully client side with javascript?) isn't trivial.
Another option would be to handle the image rendering to an external service (such as grafana, which does have this kind of API) and then just let Alertmanager to combine all this together. In this case Alertmanager would know to call external rendering service as an user configurable URL (use templating language to construct appropriate url to be sent to the rendering service), download the created image as an attachment and then somehow expose that to public internet, for example as a random generated static url.

https://github.com/qvl/promplot

Could the alerting rule have a field (label / annotation) where the user could describe a relevant query, which could then be rendered out and sent as an attachment? This approach would require the ability to specify multiple queries which would then be embedded in the same image.

This is something you could create yourself using a webhook. The notification goes to the webhook, which creates an image+url to reference the image.

Another option would be to handle the image rendering to an external service (such as grafana, which does have this kind of API) and then just let Alertmanager to combine all this together. In this case Alertmanager would know to call external rendering service as an user configurable URL (use templating language to construct appropriate url to be sent to the rendering service), download the created image as an attachment and then somehow expose that to public internet, for example as a random generated static url.

If your webhook creates a url that you can "know beforehand", you could set it as an annotation on the notification.
In general, I'm not in favor of adding this specific behavior natively to alertmanager.

For the likes of Slack, I would suggest a chatbot that grabbed such predefined dashboards. Say you're monitoring your website and you get a lot of 502 responses, I could imagine a chatbot that could respond with images. To illustrate very loosely...
bot: dude, there's lots of 502/503 responses from the website, and its freaking me out
me: spiders?
bot: here's a graph showing breakdown of web crawlers and top user-agents
/me doesn't see anything obvious there...
me: ips?
bot: here's a graph showing the top ips over time
/me sees unusually high activity from a certain IP
me: ban ip x.x.x.x
bot: intiating Ansible playbook to ban IP x.x.x.x for the default of 6 hours and recording that @me requested this action at <timestamp>
/me goes back to watch movie in peace

That does also show that you could then roll in data from logs as well as metrics, and also intiate routine responses.
If that doesn't satisfy your needs (it is a bit of a stretch-goal), you could consider moving the alerting functionality to Grafana, which could have the advantage of one alerting system that could cover Prometheus, Elasticsearch, and others..... (I haven't investigated that, but I was pretty interested by a screenshot of Slack showing a Grafana graph.)

@garo @cameronkerrnz we built that chatbot (well, something similar anyway) and it can show:

a graph for the currently firing alert
an export of any grafana dashboard
the output of any cli command

...and at least 50 other builtin actions. It's all open source and I would love to hear some ideas for how we can improve it.

authors: aantn cameronkerrnz garo mxinden roidelapluie stuartnelson3

repo: prometheus/alertmanager


text:
"repeat_interval=1h",but i still going to get a repeat alarm in a short time

my alertmanager.yml
global:
  resolve_timeout: 5m
route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: 'web.hook'
receivers:
- name: 'web.hook'
  webhook_configs:
  - url: 'http://*****:3000'
    send_resolved: false

The repeat alarm is:
{ '{"receiver":"web\\.hook","status":"resolved","alerts":[{"status":"resolved","labels":{"alertname":"TooMuchGoroutines","app":"prometheus","component":"core","instance":"10.233.118.160:9090","job":"kubernetes-endpoints","kubernetes_name":"prometheus","kubernetes_namespace":"monitoring","severity":"warning"},"annotations":{"description":"testing","summary":"too much goroutines of job prometheus."},"startsAt":"2018-12-27T06:53:30.090023826Z","endsAt":"2018-12-28T01:30:30.090023826Z","generatorURL":"http://prometheus-core-6b5dc8dd4d-z4dvl:9090/graph?g0.expr': 'go_goroutines > 100\\u0026g0.tab=1"}],"groupLabels":{"alertname":"TooMuchGoroutines"},"commonLabels":{"alertname":"TooMuchGoroutines","app":"prometheus","component":"core","instance":"10.233.118.160:9090","job":"kubernetes-endpoints","kubernetes_name":"prometheus","kubernetes_namespace":"monitoring","severity":"warning"},"commonAnnotations":{"description":"testing","summary":"too much goroutines of job prometheus."},"externalURL":"http://alertmanager-56797bdb65-x9sj5:9093","version":"4","groupKey":"{}:{alertname=\\"TooMuchGoroutines\\"}"}\n' }
{ '{"receiver":"web\\.hook","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"TooMuchGoroutines","app":"prometheus","component":"core","instance":"10.233.118.160:9090","job":"kubernetes-endpoints","kubernetes_name":"prometheus","kubernetes_namespace":"monitoring","severity":"warning"},"annotations":{"description":"testing","summary":"too much goroutines of job prometheus."},"startsAt":"2018-12-27T06:53:30.090023826Z","endsAt":"2018-12-28T01:36:30.090023826Z","generatorURL":"http://prometheus-core-6b5dc8dd4d-z4dvl:9090/graph?g0.expr': 'go_goroutines > 100\\u0026g0.tab=1"}],"groupLabels":{"alertname":"TooMuchGoroutines"},"commonLabels":{"alertname":"TooMuchGoroutines","app":"prometheus","component":"core","instance":"10.233.118.160:9090","job":"kubernetes-endpoints","kubernetes_name":"prometheus","kubernetes_namespace":"monitoring","severity":"warning"},"commonAnnotations":{"description":"testing","summary":"too much goroutines of job prometheus."},"externalURL":"http://alertmanager-56797bdb65-x9sj5:9093","version":"4","groupKey":"{}:{alertname=\\"TooMuchGoroutines\\"}"}\n' }
{ '{"receiver":"web\\.hook","status":"resolved","alerts":[{"status":"resolved","labels":{"alertname":"TooMuchGoroutines","app":"prometheus","component":"core","instance":"10.233.118.160:9090","job":"kubernetes-endpoints","kubernetes_name":"prometheus","kubernetes_namespace":"monitoring","severity":"warning"},"annotations":{"description":"testing","summary":"too much goroutines of job prometheus."},"startsAt":"2018-12-27T06:53:30.090023826Z","endsAt":"2018-12-28T01:36:30.090023826Z","generatorURL":"http://prometheus-core-6b5dc8dd4d-z4dvl:9090/graph?g0.expr': 'go_goroutines > 100\\u0026g0.tab=1"}],"groupLabels":{"alertname":"TooMuchGoroutines"},"commonLabels":{"alertname":"TooMuchGoroutines","app":"prometheus","component":"core","instance":"10.233.118.160:9090","job":"kubernetes-endpoints","kubernetes_name":"prometheus","kubernetes_namespace":"monitoring","severity":"warning"},"commonAnnotations":{"description":"testing","summary":"too much goroutines of job prometheus."},"externalURL":"http://alertmanager-56797bdb65-x9sj5:9093","version":"4","groupKey":"{}:{alertname=\\"TooMuchGoroutines\\"}"}\n' }

The alarm always turn firing to resolved ,and turn resolved to firing.But the metrics hasn't changed.

same problem, how to solve?

same problem, how to solve?

what is your version?

same problem, how to solve?

what is your version?

it was AlertManger: 0.15.2,
Prometheus: 2.3.2
But current environment also has same problem,
AlertManger: 0.16.1,
Prometheus: 2.6.0

same problem, how to solve?

what is your version?

it was AlertManger: 0.15.2,
Prometheus: 2.3.2
But current environment also has same problem,
AlertManger: 0.16.1,
Prometheus: 2.6.0

Please show me your alertmanager.yaml :)

@aixeshunter
this is alertmanager.yml
global:
  resolve_timeout: 5m
inhibit_rules:
- equal:
  - alertname
  - job
  source_match:
    severity: critical
  target_match:
    severity: warning
receivers:
- name: smartcmp-default-webhook
  webhook_configs:
  - url: http://hpuliyang.natapp1.cc/alarm-activities/trigger
route:
  continue: false
  group_by:
  - alertname
  - cluster
  - service
  group_interval: 1s
  group_wait: 1s
  receiver: smartcmp-default-webhook
  repeat_interval: 4h
  routes: []
templates:
- template/*.tmpl

I changed group_wait & group_interval to 15s & 1m, but it didn't work...

Are u sure this webhook,
  webhook_configs:
  - url: http://hpuliyang.natapp1.cc/alarm-activities/trigger

it handles the alerts and return status code 200 to alertmanager?

Hi @beijiaxu, are u sure this webhook,
  webhook_configs:
  - url: http://hpuliyang.natapp1.cc/alarm-activities/trigger

it handles the alerts and sends status code 200 back to alertmanager?

yes, in fact this url is localhost. This service receive many requests in a few minutes...
And this problem rarely happen until I use alert for k8s containers, prometheus fires alerts for more than 10 targets at the same time...

Same issue in alertmanager 0.21.0 - any update?

authors: Gaozizhong aixeshunter beijiaxu dengceltics


text:
Alertmanager continues to send messages after resolving

Alertmanager
branch  unknown
buildDate   20160209-08:25:44
buildUser   @30db80bfe262
goVersion   1.5.2
revision    unknown
version 0.1.0-beta2
Prometheus
branch  stable
buildDate   20160202-22:31:06
buildUser   @9f8f0f8d724a
goVersion   1.5.3
revision    287d9b2
version 0.16.2
Alert manager config
route:
  receiver: sms_team1
  group_by: ['alertname', 'cluster']
  group_wait: 0s
  group_interval: 5s
  repeat_interval: 5s

receivers:
  - name: sms_team1
    webhook_configs:
      - url: http://sms-gate:8900/15551111/15552222/
        send_resolved: true
    slack_configs:
      - api_url: https://hooks.slack.com/services/X000/X000/X000
        channel: '#alerts'

Alert rule
ALERT TestStatsDAlert
  IF irate(dispatcher_events_counter{processor="ff"}[15s]) > 1.5
  SUMMARY "Test alerting"
  DESCRIPTION "TEST!!!! {{$labels.instance}} processor={{$labels.processor}} dispatcher_events_counter (current value: {{$value}})"

Alerts working, but after resolving alertmanager continues send alerts. In prometheus alert tab all alerts is green but in alertmanager tab still present alert. It send alert messages with last alert value
Prometheus ui
http://take.ms/Ug1gP
Alertmanager UI
http://take.ms/MHxLr
Webhook receiver logs
http://take.ms/ZwfVU
This behavior was observed in webhook and slack.
This environment work in docker compose

How long did it continue to send the notifications for? 5 minutes by any chance?

Please wait, i'll check it

Yes, 5 minutes. How turn off this?

This is the intended behavior.
Int he global configuration block you can set the resolve_timeout to somehting lower than 5 minute. But it should always be at least 3x of the longest rule evaluation interval of your Prometheus servers.
Prometheus 0.17.0 will send explicit resolved messages to Alertmanager, so you will no longer encounter this.

Got it! Thank you. And i have another question. In webhook i not receive resolved message, but in config i set field send_resolved: true

Sorry about the late response – could you figure this out eventually. It should work as expected.

Hello, i observed very similar behavior, but in my case resolve_timeout is 5m and max evaluation time is 1m. I dont think that this is normal to receive more than 1 'resolved' message.
In my env i have 2 alertmanagers in cluster+2 prometheus which scrape the same data.
Alertmanager version: 0.5.1
Prometheus version: 1.4.1
cat prometheus.yml
global:
scrape_interval:     15s
scrape_timeout:      15s
evaluation_interval: 1m
rule_files:

"alerts.yml"
[...]

cat alerts.yml
ALERT NovaContainerDown
IF time() - container_last_seen{container_name=~"nova-scheduler"} > 60
LABELS {
severity = "warning",
service = "nova",
affected_hosts = "{{ $labels.kubernetes_io_hostname }}"
}
ANNOTATIONS {
summary = "Nova Service {{ $labels.container_name }} down",
description = "{{ $labels.container_name }} is down on {{ $labels.kubernetes_io_hostname }}"
}
cat alertmanager.yml
global:
resolve_timeout: 5m
route:
group_by: ['alertname', 'cluster', 'service']
group_wait: 60s
group_interval: 5m
repeat_interval: 3h
receiver: HTTP-notification
inhibit_rules:
Mute critical when down is fired for given service

source_match:
severity: 'down'
target_match:
severity: 'critical'
equal: ['cluster', 'service']

Mute warning when down is fired for given service

source_match:
severity: 'down'
target_match:
severity: 'warning'
equal: ['cluster', 'service']

Mute warning when critical is fired for given alert

source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'cluster', 'service']

receivers:

name: 'HTTP-notification'
webhook_configs:

url: http://localhost:8901/notify
send_resolved: true

cat alertmanager.log
time="2017-01-11T11:15:04Z" level=info msg="Starting alertmanager (version=0.5.1, branch=master, revision=0ea1cac51e6a620ec09d053f0484b97932b5c902)" source="main.go:101"
time="2017-01-11T11:15:04Z" level=info msg="Build context (go=go1.7.3, user=root@fb407787b8bf, date=20161125-08:14:40)" source="main.go:102"
time="2017-01-11T11:15:04Z" level=info msg="Loading configuration file" file="/opt/alertmanager/alertmanager.yml" source="main.go:195"
time="2017-01-11T11:15:04Z" level=info msg="Listening on :9093" source="main.go:250"
time="2017-01-11T11:21:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:21:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:22:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:22:37Z" level=debug msg="flushing [NovaContainerDown[18c0872][active] NovaContainerDown[dfa977e][active]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:22:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:22:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][active] NovaContainerDown[dfa977e][active]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:23:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:23:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:24:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:24:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][active] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:25:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:25:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:26:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:26:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:27:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:27:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:27:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved] NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:28:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:28:37Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:28:38Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:28:42Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:29:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:29:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:29:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:29:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:30:04Z" level=info msg="running maintenance" component=silences source="silence.go:233"
time="2017-01-11T11:30:04Z" level=info msg="running maintenance" component=nflog source="nflog.go:288"
time="2017-01-11T11:30:04Z" level=info msg="maintenance done" component=silences duration=165.34µs source="silence.go:250"
time="2017-01-11T11:30:04Z" level=info msg="maintenance done" component=nflog duration=2.196039ms source="nflog.go:305"
time="2017-01-11T11:30:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:30:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:30:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:30:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:31:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:31:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:31:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:31:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:32:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:32:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:32:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:32:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:33:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:33:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:33:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:33:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:34:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:34:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:34:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:34:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:35:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:35:37Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:35:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:35:42Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:36:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:36:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:36:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:36:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:37:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:37:37Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:37:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:37:42Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:38:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:38:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:38:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:38:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:39:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:39:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:39:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:39:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:40:37Z" level=debug msg="Received alert" alert=NovaContainerDown[dfa977e][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:40:37Z" level=debug msg="flushing [NovaContainerDown[dfa977e][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
time="2017-01-11T11:40:37Z" level=debug msg="Received alert" alert=NovaContainerDown[18c0872][resolved] component=dispatcher source="dispatch.go:168"
time="2017-01-11T11:40:42Z" level=debug msg="flushing [NovaContainerDown[18c0872][resolved]]" aggrGroup=6a7aaf109d02a350 source="dispatch.go:405"
grep POST /tmp/simple_notify.log |wc -l
26
cat /tmp/simple_notify.log
2017-01-11 11:22:37,869 - INFO - Got POST notify request with data '{"status": "firing", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "firing", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "0001-01-01T00:00:00Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}, {"status": "firing", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "0001-01-01T00:00:00Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-a0j3l:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:27:37,904 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}, {"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-a0j3l:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:28:42,965 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:28:47,969 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:29:47,843 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:30:42,858 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:30:47,869 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:31:42,828 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:31:47,834 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:32:37,929 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}, {"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-a0j3l:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:32:42,818 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:32:47,825 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:33:37,852 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}, {"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-a0j3l:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:33:42,838 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:33:47,865 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:34:37,819 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-a0j3l:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:34:47,818 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:35:47,837 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:36:47,823 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.818Z", "generatorURL": "http://prometheus-3903331295-nvda3:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.818Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus-3903331295-nvda3", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:37:47,873 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}, "endsAt": "2017-01-11T11:25:37.808Z", "generatorURL": "http://prometheus-3903331295-ulaz1:9090/graph?g0.expr=time()+-+container_last_seen{container_name%3D~"nova-(compute|scheduler|conductor|api|novncproxy|virtlogd|libvirt|consoleauth)"}+>+60&g0.tab=0", "startsAt": "2017-01-11T11:21:37.809Z", "annotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}}], "version": "3", "receiver": "HTTP-notification", "externalURL": "http://alertmanager-1790141429-yxvox:9093", "commonLabels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.71.44", "prometheus_pod": "prometheus-3903331295-ulaz1", "region": "region1", "beta_kubernetes_io_arch": "amd64", "id": "/docker/3ba3e0952336505c0feedd46d36b4b383d3b2d977d81fb159310557693e9a148", "kubernetes_io_hostname": "node2"}}'
2017-01-11 11:38:47,821 - INFO - Got POST notify request with data '{"status": "resolved", "groupLabels": {"service": "nova", "alertname": "NovaContainerDown"}, "groupKey": 7672637401067594576, "commonAnnotations": {"description": "nova-scheduler is down on node2", "summary": "Nova Service nova-scheduler down"}, "alerts": [{"status": "resolved", "labels": {"beta_kubernetes_io_os": "linux", "name": "k8s_nova-scheduler.1fedd457_nova-scheduler-1037894809-c3sba_ccp_7d20c96a-d7e5-11e6-bd0a-6445bd19421f_9b7d5b0f", "affected_hosts": "node2", "severity": "warning", "image": "127.0.0.1:31500/ccp/nova-scheduler:latest", "namespace": "ccp", "instance": "node2", "job": "kubernetes-nodes", "service": "nova", "pod_name": "nova-scheduler-1037894809-c3sba", "alertname": "NovaContainerDown", "container_name": "nova-scheduler", "prometheus_ip": "10.233.84.37", "prometheus_pod": "prometheus

It was caused by 2 instances of prometheus with different external_labels. After alert relabeling everything back to normal.
alerting:
  alert_relabel_configs:
    - source_labels: [prometheus_pod]
      action: replace
      replacement: prometheus
      target_label: prometheus_pod
    - source_labels: [prometheus_ip]
      action: replace
      replacement: prometheus
      target_label: prometheus_ip

It is still a problem for me. Can we reopen this issue @fabxc ?

@bkupidura I created a new ticket for our problem. Here you can find it: https://github.com/prometheus/alertmanager/issues/611
Would you be so nice and leave your comment there too? Thanks :)