text:
Alerts can inhibit themselves

I have a bunch of alerts in the form of Prefix_Some_Name. I've set up the following inhibition rule:
- source_match_re:
    alertname: Prefix_.+
  target_match:
    alertname: Prefix_Failsafe_Alert
  equal:
    - label1
    - label2

As far as I can tell, this alert will inhibit itself, and never be delivered. Depending on how you look at it this is a bug in the alertmanager, or an important gotcha that should be documented.
Happy to contribute a patch either way.
Edit: I should add that I empirically verified this against an older alertmanager. I looked through the current master branch, and AFAICT the behaviour is still the same.

As far as I can tell, this alert will inhibit itself

Hm yes, that is the behavior I would expect. I guess you expected that alerts generally cannot inhibit themselves?
I think adding a clarfiying sentence to https://github.com/prometheus/docs/blob/master/content/docs/alerting/configuration.md#inhibit_rule would be generally welcome if you think this is something that is confusing.

I also discovered this behavior the hard way. I agree that it's strictly reasonable within the context of how inhibitions are currently structured, but is this behavior considered desirable?
If alerts never inhibited themselves, certain types of alerting noise could be reduced in a way that's currently infeasible. The only use case I can imagine for self-inhibiting alerts is ensuring they never notify anybody, which would surely be better handled by configuring a matching route with no receivers. But maybe I'm just not thinking big enough?

Yeah, I can't come up with a reason why an alert should be able to inhibit itself either. The question is, is it worth introducing an explicit exception for this case? That's ultimately up to the Alertmanager maintainers (@stuartnelson3 @fabxc).

I think the general expectation here is that there's some other label you can work off to distinguish these, such as a "dependency" label or similar.
The "Prefix_" being the same on both is also a bit of a smell, that should probably be a label.

@brian-brazil that might be correct, but doesn't change the underlying issue.

This should probably a documented caveat. I think it doesn't really happen for typical inhibition usage as there should be some clearly distinguishing labels. Probably an edge case not worth complicating implementation for.

@lmb Would you mind updating the documentation as suggested by @juliusv in his comment?

Yup can do, feel free to assign to me.

@lmb Thanks! Just tag me on the PR. Let us know if you need any help.

The problem here is that the most obvious fix requires negative matches, which the inhibit rule syntax does not support. E.g. you have a NodeDown page that should inhibit all other pages that share the same node label (because you know the node is down so you don't need to be notified that your HAProxy on that node is down and your MySQL and your Prometheus etc.).
Naive rule syntax that inhibits the original NodeDown alert, too:
inhibit_rules:
- source_match:
    alertname: NodeDown
    severity: critical
  target_match:
    severity: critical
  equal:
  - node

This works:
inhibit_rules:
- source_match:
    alertname: NodeDown
    severity: critical
  target_match:
    severity: critical
  target_match_re:
    alertname: '.{0,7}|.{9,}|([^N]|.([^o]|.([^d]|.([^e]|.([^D]|.([^o]|.([^w]|.[^n]))))))).*'
  equal:
  - node

Much nicer would be:
inhibit_rules:
- source_match:
    alertname: NodeDown
    severity: critical
  target_match:
    severity: critical
  target_not_match:
    alertname: NodeDown
  equal:
  - node

Well, since I seem to have wasted a couple of hours implementing a basic "alert shouldn't inhibit itself" filter that's not wanted, does anyone but @beorn7 think that having something along the lines of his suggested target_not_match is a good idea?

I don't think it's a good idea. I think the issues presented here are merely the first level issues with this approach, and I believe that going down that route would lead to very fragile and tricky to maintain configs as the second and third level issues come along (think a multi-level set of nested inhibitions).
Using alertnames in the alertmanager config is a smell, severity etc. labels are a better and more maintainable approach.

I can believe that a working "an alert cannot inhibit itself" implementation might turn out tricky.
I think we should either add negative matchers as suggested above, or we could adopt the syntax as already used in the Alertmanager UI (=, !=, =~, !~), which would also allow us to remove the separate ..._re config entries. If we named that differently, e.g. target_matchers and source_matchers, we could have both system in parallel for a while so that we don't break existing configs.
Same would be applicable to the routing tree.
I vaguely remember @fabxc suggested something like that in IRC.

FYI, the documentation change in prometheus/docs#812 is already live on prometheus.io, so this issue should probably be closed.

Indeed, discussion of potential changes to routing&inhibition syntax can go in a new issue.

I have filed #1023 for extended/alternative matcher syntax.
About the original problem of this issue: So far, there have been zero reported use-cases for an alert actually required to inhibit itself, while there are multiple where it is awkward that an alert inhibits itself and requires work-arounds of varying weirdness, including a number where the issue was discovered the hard way (even if the semantics was well known in principle, i.e. it is not just a documentation issue, it's also a problem of counter-intuition). Thus, if there actually is a sane implementation of preventing alerts from inhibiting themselves, I'd prefer if we used it. It would be great if those maintaining and knowing the Alertmanager codebase could have a look at #1017 and make a call if this is a possible way at all.

FYI, #1017 has just been merged, solving the basic case of a single inhibition rule possibly inhibiting the source alert(s). Now, if an alert matches both the source and target matchers of a inhibition rule it will not be inhibited by that rule.
Multiple inhibition rules (e.g. {A inhibits B} and {B inhibits A}) will still cause unwanted effects, but that is outside of the scope of this issue.
BTW, the example in the OP will now be a no-op because all possible target matches will also match the source.

So yeah, in the case that @lmb presented, we are now encountering a different but still surprising effect, as the whole rule doesn't apply anymore. That's happening if a rule is matching a lot of source alerts (while the usual inhibition rule is probably quite specific on the source side and wider on the target side).
I guess the intuitively desired behavior is that the alert would not inhibit itself, but another alert, that matches the source but not the target would inhibit an alert that matches both source and target.
I think #1017 improves things because surprisingly getting alerted is still better than surprisingly not getting alerted, but we should decide if we want to refine the semantics before we do the next release.

The original report is not resolved, but it's the only one I've seen trying to do that. All the other cases I've seen so far should be handled by #1017.

I guess the intuitively desired behavior is that the alert would not inhibit itself, but another alert, that matches the source but not the target would inhibit an alert that matches both source and target.

I can't quite get my head around that. Do you have an example?
I suspect such logic is too intricate to be understandable in practice. #1017 is understandable, if a tad more complicated than the previous logic.

I guess the intuitively desired behavior is that the alert would not inhibit itself, but another alert, that matches the source but not the target would inhibit an alert that matches both source and target.

I can't quite get my head around that. Do you have an example?

Yeah, it's kind of weird. I think the example is exactly the one @lmb gave when opening this issue. I can speculate about the exact use case around it, but perhaps @lmb can explain himself. (My tl;dr take: If Prefix_Failsafe_Alert fires on its own, it should notify. But if any other Prefix_... alert is firing too, Prefix_Failsafe_Alert should be inhibited, presumably because the Prefix_Failsafe_Alert isn't helpful anymore if you also get one of the other Prefix_... alerts.)

That's a good summary, and your suggestion sounds at first glance as a logical step if we were to go further down the road of making this more sophisticated. For now I'd suggest giving the new logic a month or two to bake in, and seeing if our current state is good enough.

Sounds good. I might also find the time to look at implementation to see how hard or easy it really is.

Bar 1-2 users using inhibits for things they shouldn't have been, this seems to have worked out fine and confusion in this area has gone away. I think we can close this.

This is still on my list to try out. The asymmetry in the current implementation worries me. If the implementation of my final suggestion above turns out to be clean and straight-forward, I'd suggest to go for it.
I have assigned this to myself and would like to keep it open.

As a bit of an aside, the AM config docs (https://prometheus.io/docs/alerting/configuration/#<inhibit_rule>) still say "Alerts can inhibit themselves. Avoid writing inhibition rules where an alert matches both source and target.", which I believe is incorrect given #1017 has been merged.  I'm presuming a doc-fix PR be accepted?

Yes, that's out of date.

That should be solved for good now. https://github.com/prometheus/docs/pull/1297 to be merged once released.

authors: alin-amana beorn7 brian-brazil fabxc juliusv lmb mpalmer mxinden threemachines


text:
Support that send wechat textcard message to user or group

Use Environment Variable First if APISecret/CorpID not set
Support new message type "textcard"
Support than send message to Group

https://work.weixin.qq.com/api/doc#90000/90135/90243

@pulord you'd need to run make assets and commit the changes to asset/assets_vfsdata.go.

@pulord you'd need to run make assets and commit the changes to asset/assets_vfsdata.go.

make assets error
`cd /Users/putaohong/Documents/work/git-code/go/src/github.com/prometheus/alertmanager/ui/app && /Library/Developer/CommandLineTools/usr/bin/make script.js

building elm-env docker image
The command '/bin/sh -c npm install -g   elm@0.19.0   elm-format@0.8.0   elm-test@0.19.0-beta5   uglify-js@3.4.7' returned a non-zero code: 1
make[1]: *** [elm-env] Error 1
make: *** [ui/app/script.js] Error 2`

Hmm, it works for me but I'm not on MacOS... Nevermind I'll push the make assets change on top of your PR once we're ok with it.

Hmm, it works for me but I'm not on MacOS... Nevermind I'll push the make assets change on top of your PR once we're ok with it.

I have fixed ! pls help fix the CI Error, thanks

@simonpasquier check again, thanks for you help

@simonpasquier check again, thanks for your help

@simonpasquier @brian-brazil Hi, pls check this PR, thanks

You would need to rebase your PR because of https://github.com/prometheus/alertmanager/pull/1929. Changes should go to notify/wechat/ now.

source: gharchive/pull-request

authors: pulord simonpasquier

repo: prometheus/alertmanager

license_type: permissive

license_source: github-api


text:
Add mongodb-query-exporter

Add mongodb-query-exporter to the list of exporters.
The mongodb-query-exporter creates prometheus metrics out of MongoDB aggregations.

Thanks for sharing,  it looks like you should be using ConstMetrics. For example https://github.com/raffis/mongodb-query-exporter/blob/c871a06191355d06180dd88ee2ceba8ab68707fb/collector/main.go#L262 is trying to workaround incorrectly using direct instrumentation in an exporter (and more generally I question whether this sort of exporter can actually produce a Counter - everything should be a Gauge). This will be simpler, and ensure series disappear correctly.
You should also be pulling the metrics synchronously at scrape time, not based on a background timer - see https://prometheus.io/docs/instrumenting/writing_exporters/#scheduling

Thanks for sharing, it looks like you should be using ConstMetrics. For example https://github.com/raffis/mongodb-query-exporter/blob/c871a06191355d06180dd88ee2ceba8ab68707fb/collector/main.go#L262 is trying to workaround incorrectly using direct instrumentation in an exporter (and more generally I question whether this sort of exporter can actually produce a Counter - everything should be a Gauge). This will be simpler, and ensure series disappear correctly.
You should also be pulling the metrics synchronously at scrape time, not based on a background timer - see https://prometheus.io/docs/instrumenting/writing_exporters/#scheduling

Appreciate the review 👍 , will think about the counters again but yes I think I have to agree on this that there will probably never be an actual use case for this.

Thanks for sharing, it looks like you should be using ConstMetrics. For example https://github.com/raffis/mongodb-query-exporter/blob/c871a06191355d06180dd88ee2ceba8ab68707fb/collector/main.go#L262 is trying to workaround incorrectly using direct instrumentation in an exporter (and more generally I question whether this sort of exporter can actually produce a Counter - everything should be a Gauge). This will be simpler, and ensure series disappear correctly.
You should also be pulling the metrics synchronously at scrape time, not based on a background timer - see https://prometheus.io/docs/instrumenting/writing_exporters/#scheduling

Your suggestions are now implemented:

Dropped support for counter
Using ConstMetrics now
Pulling synchronously at scrape time
(Added support for cache instead)

Your cache is broken, as you're concurrently reading and modifying c.cache from different goroutines. Concurrent map modification and access must be protected by a mutex.
I also think you don't need a custom registry here, the default one should suffice.

Your cache is broken, as you're concurrently reading and modifying c.cache from different goroutines. Concurrent map modification and access must be protected by a mutex.

Indeed, good catch.

I also think you don't need a custom registry here, the default one should suffice.

I used a custom one because of testing, with the one at package level you would have to unregister depening on the case. Or did I miss here something?

I used a custom one because of testing, with the one at package level you would have to unregister depening on the case. Or did I miss here something?

I'm looking at your main function, which seems to have an unneeded registry. I didn't look at the tests, but custom registries aren't unusual there.

Your cache is broken, as you're concurrently reading and modifying c.cache from different goroutines. Concurrent map modification and access must be protected by a mutex.
I also think you don't need a custom registry here, the default one should suffice.

It's fixed now in master.

Thanks!
I'd suggest using a defer for the unlock in getCached, as that's going to be more robust in the face of future code changes.

Thanks for the suggestions, appreciate it!

authors: brian-brazil raffis

repo: prometheus/docs

license_source: bigquery


text:
Unable to send alert from Prometheus after Upgrade

What did you do?
I have 2 Prometheus deployment and an alertmanager running in my cluster. One is 2.27.1 Prometheus version and other one is 2.45.0 version. Both are connected to the same alertmanager with version 0.25.0.
2.27.1 Prometheus  version is deployed using our  own chart and 2.45.0 Prometheus  version is deployed using latest kube-prometheus-stack chart
What did you expect to see?
I expect to see both the prometheus sending alerts to Alertmanager,
What did you see instead? Under which circumstances?
I am able to see alerts from 2.27.1 Prometheus version but from 2.45.0 Prometheus  version i am receiving below error while sending alerts.
ts=2023-08-22T10:03:11.134Z caller=notifier.go:529 level=error component=notifier alertmanager=http://10.xx.xx.xx:9093/api/v2/alerts count=5 msg="Error sending alert" err="Post \"http://10.xx.xx.xx:9093/api/v2/alerts\": read tcp 10.xx.xx.xx:39924->10.xx.xx.xx:9093: read: connection reset by peer"
I am able to telnet from 2.45.0  prometheus to alertmanager.
/prometheus $ telnet 10.xx.xx.xx:9093  Connected to 10.xx.xx.xx:9093
But wget is not
/prometheus $ wget http://10.xx.xx.xx:9093/  Connecting to 10.xx.xx.xx:9093 (10.xx.xx.xx:9093) wget: error getting response: Connection reset by peer
From 2.27.1 Prometheus version
/etc/prometheus/config_out $  wget http://10.xx.xx.xx:9093/ Connecting to 10.xx.xx.xx:9093 (10.xx.xx.xx:9093) wget: can't open 'index.html': Read-only file system
System information
Linux 5.4.249-163.359.amzn2.x86_64 x86_64
Prometheus version
prometheus, version 2.45.0 (branch: HEAD, revision: 8ef767e396bf8445f009f945b0162fd71827f445)
  build user:       root@920118f645b7
  build date:       20230623-15:09:49
  go version:       go1.20.5
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

Prometheus configuration file
global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s
  external_labels:
    prometheus: prometheus/prometheus-kube-prometheus-prometheus
    prometheus_replica: prometheus-prometheus-kube-prometheus-prometheus-0
alerting:
  alert_relabel_configs:
  - separator: ;
    regex: prometheus_replica
    replacement: $1
    action: labeldrop
  alertmanagers:
  - follow_redirects: true
    enable_http2: false
    scheme: http
    path_prefix: /
    timeout: 10s
    api_version: v2
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: infra-monitoring-alertmanager
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: web
      replacement: $1
      action: keep
    kubernetes_sd_configs:
    - role: endpoints
      kubeconfig_file: ""
      follow_redirects: true
      enable_http2: true
      namespaces:
        own_namespace: false
        names:
        - staging
rule_files:
- /etc/prometheus/rules/prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0/*.yaml
scrape_configs:
- job_name: serviceMonitor/prometheus/dcgm-dcgmexporter/0
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_component, __meta_kubernetes_service_labelpresent_app_kubernetes_io_component]
    separator: ;
    regex: (dcgm-exporter);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
    separator: ;
    regex: (dcgm);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
    separator: ;
    regex: (dcgmexporter);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: metrics
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - prometheus
- job_name: serviceMonitor/prometheus/prometheus-kube-prometheus-apiserver/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    server_name: kubernetes
    insecure_skip_verify: false
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_component, __meta_kubernetes_service_labelpresent_component]
    separator: ;
    regex: (apiserver);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_provider, __meta_kubernetes_service_labelpresent_provider]
    separator: ;
    regex: (kubernetes);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_component]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  metric_relabel_configs:
  - source_labels: [__name__, le]
    separator: ;
    regex: apiserver_request_duration_seconds_bucket;(0.15|0.2|0.3|0.35|0.4|0.45|0.6|0.7|0.8|0.9|1.25|1.5|1.75|2|3|3.5|4|4.5|6|7|8|9|15|25|40|50)
    replacement: $1
    action: drop
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - default
- job_name: serviceMonitor/prometheus/prometheus-kube-prometheus-coredns/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (kube-prometheus-stack-coredns);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_release, __meta_kubernetes_service_labelpresent_release]
    separator: ;
    regex: (prometheus);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_jobLabel]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - kube-system
- job_name: serviceMonitor/prometheus/prometheus-kube-prometheus-kube-controller-manager/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (kube-prometheus-stack-kube-controller-manager);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_release, __meta_kubernetes_service_labelpresent_release]
    separator: ;
    regex: (prometheus);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_jobLabel]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - kube-system
- job_name: serviceMonitor/prometheus/prometheus-kube-prometheus-kube-dns/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (kube-prometheus-stack-kube-dns);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_release, __meta_kubernetes_service_labelpresent_release]
    separator: ;
    regex: (prometheus);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics-dnsmasq
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_jobLabel]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics-dnsmasq
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - kube-system
- job_name: serviceMonitor/prometheus/prometheus-kube-prometheus-kube-dns/1
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (kube-prometheus-stack-kube-dns);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_release, __meta_kubernetes_service_labelpresent_release]
    separator: ;
    regex: (prometheus);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics-skydns
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_jobLabel]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics-skydns
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - kube-system
- job_name: serviceMonitor/prometheus/prometheus-kube-prometheus-kube-etcd/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (kube-prometheus-stack-kube-etcd);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_release, __meta_kubernetes_service_labelpresent_release]
    separator: ;
    regex: (prometheus);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_jobLabel]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - kube-system
- job_name: serviceMonitor/prometheus/prometheus-kube-prometheus-kube-proxy/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (kube-prometheus-stack-kube-proxy);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_release, __meta_kubernetes_service_labelpresent_release]
    separator: ;
    regex: (prometheus);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_jobLabel]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - kube-system
- job_name: serviceMonitor/prometheus/prometheus-kube-prometheus-kube-scheduler/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (kube-prometheus-stack-kube-scheduler);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_release, __meta_kubernetes_service_labelpresent_release]
    separator: ;
    regex: (prometheus);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_jobLabel]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - kube-system
- job_name: serviceMonitor/prometheus/prometheus-kube-prometheus-kubelet/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
    separator: ;
    regex: (kubelet);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_k8s_app, __meta_kubernetes_service_labelpresent_k8s_app]
    separator: ;
    regex: (kubelet);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https-metrics
    action: replace
  - source_labels: [__metrics_path__]
    separator: ;
    regex: (.*)
    target_label: metrics_path
    replacement: $1
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - kube-system
- job_name: serviceMonitor/prometheus/prometheus-kube-prometheus-kubelet/1
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics/cadvisor
  scheme: https
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
    separator: ;
    regex: (kubelet);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_k8s_app, __meta_kubernetes_service_labelpresent_k8s_app]
    separator: ;
    regex: (kubelet);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https-metrics
    action: replace
  - source_labels: [__metrics_path__]
    separator: ;
    regex: (.*)
    target_label: metrics_path
    replacement: $1
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  metric_relabel_configs:
  - source_labels: [__name__]
    separator: ;
    regex: container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)
    replacement: $1
    action: drop
  - source_labels: [__name__]
    separator: ;
    regex: container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)
    replacement: $1
    action: drop
  - source_labels: [__name__]
    separator: ;
    regex: container_memory_(mapped_file|swap)
    replacement: $1
    action: drop
  - source_labels: [__name__]
    separator: ;
    regex: container_(file_descriptors|tasks_state|threads_max)
    replacement: $1
    action: drop
  - source_labels: [__name__]
    separator: ;
    regex: container_spec.*
    replacement: $1
    action: drop
  - source_labels: [id, pod]
    separator: ;
    regex: .+;
    replacement: $1
    action: drop
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - kube-system
- job_name: serviceMonitor/prometheus/prometheus-kube-prometheus-kubelet/2
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics/probes
  scheme: https
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
    separator: ;
    regex: (kubelet);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_k8s_app, __meta_kubernetes_service_labelpresent_k8s_app]
    separator: ;
    regex: (kubelet);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_k8s_app]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https-metrics
    action: replace
  - source_labels: [__metrics_path__]
    separator: ;
    regex: (.*)
    target_label: metrics_path
    replacement: $1
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - kube-system
- job_name: serviceMonitor/prometheus/prometheus-kube-prometheus-operator/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  tls_config:
    ca_file: /etc/prometheus/certs/secret_prometheus_prometheus-kube-prometheus-admission_ca
    server_name: prometheus-kube-prometheus-operator
    insecure_skip_verify: false
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (kube-prometheus-stack-operator);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_release, __meta_kubernetes_service_labelpresent_release]
    separator: ;
    regex: (prometheus);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: https
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: https
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - prometheus
- job_name: serviceMonitor/prometheus/prometheus-kube-prometheus-prometheus/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (kube-prometheus-stack-prometheus);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_release, __meta_kubernetes_service_labelpresent_release]
    separator: ;
    regex: (prometheus);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_self_monitor, __meta_kubernetes_service_labelpresent_self_monitor]
    separator: ;
    regex: (true);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-web
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-web
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - prometheus
- job_name: serviceMonitor/prometheus/prometheus-kube-prometheus-prometheus/1
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (kube-prometheus-stack-prometheus);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_release, __meta_kubernetes_service_labelpresent_release]
    separator: ;
    regex: (prometheus);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_self_monitor, __meta_kubernetes_service_labelpresent_self_monitor]
    separator: ;
    regex: (true);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: reloader-web
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: reloader-web
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - prometheus
- job_name: serviceMonitor/prometheus/prometheus-kube-state-metrics/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
    separator: ;
    regex: (prometheus);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
    separator: ;
    regex: (kube-state-metrics);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - prometheus
- job_name: serviceMonitor/prometheus/prometheus-prometheus-node-exporter/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance]
    separator: ;
    regex: (prometheus);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name]
    separator: ;
    regex: (prometheus-node-exporter);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_jobLabel]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http-metrics
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - prometheus
storage:
  tsdb:
    outofordertimewindow: 0

Alertmanager version
alertmanager, version 0.25.0 (branch: HEAD, revision: 258fab7cdd551f2cf251ed0348f0ad7289aee789)
  build user:       root@abe866dd5717
  build date:       20221222-14:51:36
  go version:       go1.19.4
  platform:         linux/amd64

Alertmanager configuration file
global:
  resolve_timeout: 5m
  http_config:
    follow_redirects: true
    enable_http2: true
  smtp_hello: localhost
  smtp_require_tls: true
  pagerduty_url: https://events.pagerduty.com/v2/enqueue
  opsgenie_api_url: https://api.opsgenie.com/
  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
  telegram_api_url: https://api.telegram.org
  webex_api_url: https://webexapis.com/v1/messages
route:
  receiver: staging/infra-monitoring/OpsGenieWarning
  group_by:
  - alertname
  continue: false
  routes:
  - receiver: staging/infra-monitoring/OpsGenieCritical
    continue: false
  - receiver: staging/infra-monitoring/OpsGenieWarning
    continue: false
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
receivers:
- name: staging/infra-monitoring/OpsGenieCritical
  opsgenie_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
      enable_http2: true
    api_key: <secret>
    api_url: https://api.opsgenie.com/
    message: '{{ template "opsgenie.default.message" . }}'
    description: '{{ template "opsgenie.default.description" . }}'
    source: '{{ template "opsgenie.default.source" . }}'
    responders:
    - name: Enterprise Cognition
      type: team
    priority: P3
- name: staging/infra-monitoring/OpsGenieWarning
  opsgenie_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
      enable_http2: true
    api_key: <secret>
    api_url: https://api.opsgenie.com/
    message: '{{ template "opsgenie.default.message" . }}'
    description: '{{ template "opsgenie.default.description" . }}'
    source: '{{ template "opsgenie.default.source" . }}'
    responders:
    - name: Enterprise Cognition
      type: team
    priority: P4
templates: []

Logs
`ts=2023-08-22T10:03:11.134Z caller=notifier.go:529 level=error component=notifier alertmanager=http://10.xx.xx.xx:9093/api/v2/alerts count=5 msg="Error sending alert" err="Post \"http://10.xx.xx.xx:9093/api/v2/alerts\": read tcp 10.xx.xx.xx:39924->10.xx.xx.xx:9093: read: connection reset by peer"`

I met the same issue. I deployed prometheus using lastest kube -prometheus-stack. Do you resolved this issue later? Thank you!

source: gharchive/issue

repo: prometheus/prometheus


text:
Add trace ID headers to scrape requests

Proposal
In order to generate end-to-end trace information of Prometheus scrapes, it would be useful to add the initial trace ID headers at the Prometheus scrape.
This would allow for full end-to-end traces for situations like this:
Prometheus scrape -> blackbox_exporter http_2xx probe -> service

THX,how to configure in prometheus.yml?

The proposal is very beneficial. I think this is a great idea.

authors: SupeRoy100 SuperQ freshaier


text:
Issue with count_values()?

Hi,
I've been using Prometheus with Cadvisor and node_exporter to monitor some dockers, currently i'm attempting to write a query to detect when a docker container goes down.
So far I've found this query that almost does the job to my liking:
count(container_last_seen{image!=""})
This detects the amount of running containers nicely, but if prometheus is running and a docker goes down after prometheus is already running, the metric isn't removed.
This is cadvisor's part and maybe it is designed this way, because what i do notice is that the value of metric changes if the docker goes down!
Then I found the count_values() operator instead of the count() in the documentation of your website and was thinking, that's it!
Unfortunately, when I replaced count() with count_values() I received the following error:
count_values(container_last_seen{image!=""})

Error executing query: parse error at char 44: unexpected ")" in aggregation, expected ","

Is this working as intended? If so, could you please clarify this a bit in your documentation and maybe put it in another category as count() or something? Because at this time, it seems like they can both be used in the same way  (syntax wise).
If not, a bug is hereby reported :smile:

Check the documentation of the count_values(), there is a required parameter for specifying the label name which will hold the unique value. In you case, something like count_values("timestamp", container_last_seen{image!=""}).

Thank you! That did the job!
Seeing the result of the query, it also actually makes sense that you need to add a label parameter.
It still isn't very clear to me though where exactly this is documented, but that probably is a fault on my part. I was looking at this page and following the logic here, to me it seemed that it could be used the same way as count()
But again, this is probably a fault on my part.
Thank you again for your quick response and making this awesome tool!

authors: Hetkoekje grobie


text:
Only allow single K8S API server in SD config

The Kubernetes upstream client library does not support multiple API server URLs but instead expects DNS load balancing or using a proxy:
https://godoc.org/k8s.io/client-go/1.4/rest#Config
We should consider making the K8S SD config argument just a single API server URL. If we want to use the official client library there's no way to use multiple ones. With cluster federation and all, there's a fair bit of proxying involved behind the scenes anyway.
@brian-brazil @jimmidyson

If that's the way the upstream API works, then that's the way it works.

OK let's go with that. Needs to be noted that as we move to static configuration for API servers to scrape metrics from (not k8s SD) that users will have to configure all the API servers, not just the RR DNS or proxy address, but that's a separate issue.

The new k8 SD does this.

authors: brian-brazil fabxc jimmidyson


text:
Fix HTTP error handling in remote.Client.Store()

Regression introduced in
https://github.com/prometheus/prometheus/commit/e5d7bbfc3c7e38b4ee257dd1c6bc9b3e28b23d57

@fabxc @tomwilkie

👍 from me, couple of minor nits on the test.
Travis failure is due to license check... which is weird.

You need to add the legal stuff to the new test file.

Fixed all the things. Just doing a DeepReflect on err makes the code much simpler, although it now requires an exact replication of the error strings...

Just doing a DeepReflect on err makes the code much simpler, although it now requires an exact replication of the error strings...

You could extract the error generation out into a function if you liked.
👍

You could extract the error generation out into a function if you liked.

Meh, overkill IMO...

authors: brian-brazil juliusv tomwilkie


text:
Update doc

This PR fixes a couple of typos in the format documentation. It also references this documentation from the main README file.

LGTM! Thanks!

authors: gouthamve simonpasquier

repo: prometheus/tsdb


text:
Implement Video-Source services to set/get the frame-rate of the output buffers - scale up/down from source element

New services
/**
 * @brief Returns the scaled frame-rate as a fraction for the named Video Source.
 * The default values of 0 for fps_n and fps_d indicate no scaling.
 * @param[in] name unique name of the source to query
 * @param[out] fps_n frames per second numerator
 * @param[out] fps_d frames per second denominator
 * @return DSL_RESULT_SUCCESS on success, DSL_RESULT_SOURCE_RESULT otherwise.
 */
DslReturnType dsl_source_video_buffer_out_frame_rate_get(const wchar_t* name,
    uint* fps_n, uint* fps_d);

/**
 * @brief Sets the scaled frame-rate as a fraction for the named Video Source.
 * Set fps_n and fps_d to 0 indicate no scaling.
 * @param[in] name unique name of the source to query
 * @param[in] fps_n frames per second numerator
 * @param[in] fps_d frames per second denominator
 * @return DSL_RESULT_SUCCESS on success, DSL_RESULT_SOURCE_RESULT otherwise.
 */
DslReturnType dsl_source_video_buffer_out_frame_rate_set(const wchar_t* name,
    uint fps_n, uint fps_d);

Work has been merged into the v0.27.alpha branch

license: MIT


text:
Custom values for O, OU and C

We want to send more information on the certificate than just CN to indicate heritage of the key, mainly what helm-chart they belong to, but also to read up "roles" for authorization.
(edit: I hijacked this issue because I realised my previous task was already addressing the original subject)

For context, I've been approaching this from the perspective that only the SANs matter.
How would the approval flow know how to validate these fields?
You'd be encoding roles in the O/OU?

Currently I am, yes. I think the approval flow could validate O/OU based on labels on the containers, perhaps? So you can say that allowed values for O/OU are values that needs to exist as labels.
The use case here are systems that use parts of the principal to establish role/groupings of users. So you can have several pods in a deployment that have unique CNs, but still have them grouped into the same permissions based on other components of the principal.

I'll mention that this is an inadvisable misuse of the subject field. Such things should be in separate extensions, presuming you don't want to go full attribute certificate. The web PKI has suffered from its misuse of the CN field for domain names and browsers are making moves towards finally ignoring any domain in the CN.
Allowing additional flags for specifying these seems like it would be sufficiently localized to be accepted.

Are there any X.509 extensions that could be used to fill a similar function? I could suggest it to upstream rather than the one outlined here.
I do however think that as you conclude, that this is an 'ok' feature to have with the imposed limits, no matter whether we can convince people to stop abusing the subject. :)

That is a bit outside of my knowledge of X.509.

authors: johngmyers mikn

repo: proofpoint/certificate-init-container


text:
PostgreSQL DB on reverse command generates defaultValue instead of defaultExpr for timestamp column

PostgreSQL DB on reverse command generates defaultValue instead of defaultExpr for timestamp column
<column name="a_datetime" phpName="ADatetime" type="TIMESTAMP" defaultValue="(now6) with time zone"/>

After so long ... :smile:
Which table composition in PostgreSQL?

I suggest close this issue, because in Postgresql 9.2, 9.3, 9.4 and 9.5 works perfectly.

license: mit


text:
[Automatic validation] Missing License

How to fix
Add a license for this repository. See help for guidance. Private repositories don't need a license.
About validation
These issues are created, closed and reopened by repository validator when commits are pushed to repository.
If there are problems, please add an issue to repository validator
DO NOT change the name of this issue. Names are used to identify the issues created by automation.

[Automatic validation]: Issue fixed. Closing issue.


text:
🛑 recursive.ipfs.io is down

In 7ed613d, recursive.ipfs.io (https://recursive.ipfs.io) was down:

HTTP code: 0
Response time: 0 ms

Resolved: recursive.ipfs.io is back up in 109a45d after 2 hours, 54 minutes.

repo: protocol/upptime-pln


text:
🛑 filecoingreen.io is down

In b7621fb, filecoingreen.io (https://filecoingreen.io) was down:

HTTP code: 0
Response time: 0 ms

Resolved: filecoingreen.io is back up in bfd3c94 after 9 minutes.


text:
🛑 randomplanetfacts.xyz is down

In 0bbe28e, randomplanetfacts.xyz (https://randomplanetfacts.xyz) was down:

HTTP code: 0
Response time: 0 ms

Resolved: randomplanetfacts.xyz is back up in 6db1c8f after 6 minutes.


text:
How to create a bot using tourmaline that reads any messages from the chat and responds to them?

Actually the question I asked. I can't figure out how to create such a bot

Pretty simple. If you want to make a method that responds to any incoming message you just need to use the On annotation. For example:
@[On(:message)]
def on_message(update)
  if message = update.message
    # do stuff here
  end
end

For now the method decorated with the On annotation always has to accept an Update object, which has an optional message property. So to make crystal happy you'd either have to do message = update.message.not_nil! (since we know that in this case the message will not be nil) or do a simple guard clause like I did in the example above.

thank you for your answer

authors: VanyaNeytrino watzon

repo: protoncr/tourmaline


text:
Update README.md

The development of a machine learning project for accurate wind power curve estimation aligns perfectly with Open Sustainable Technology's mission, as it contributes to the efficient harnessing of renewable energy sources. By optimizing wind energy production through advanced data analysis, this project aids in reducing reliance on non-renewable, polluting energy sources, thus helping to preserve natural ecosystems. Furthermore, as an open-source initiative, it democratizes access to cutting-edge technology, allowing a broader community to participate in and benefit from sustainable energy innovations.
Insert URLs to the project here:
[ https://github.com/TAMU-AML/DSWE-Package ]

[YES ] The projects is active, documented, open source licensed, shows usage from external parties and is directly targeting environmental sustainability. Find more details in the Contribution Guide.

All issues labeled as 'Good First Issue' of the project listed on OpenSustain.tech will be visible on ClimateTriage.com. This is a great way to welcome new community members to your project.

Thank you @AhmadChokhachian for contributing this project to Open Sustainable Technology. It fits really well into the list. To give you a bit of feedback:
When I first reviewed the project I was thinking it is not open source because the License just showes a copyright note. But since you made it a CRAN package it must be open source. Theirfor I recommend you to add the MIT LICENSE mentioned in the DESCRIPTION file. Most people searching and browsing GitHub might think this is not a open source project because no open source license can be found.

repo: protontypes/open-sustainable-technology


text:
Custom options

I found that current parser implementation ignore custom options like this:
message MyMessage {
  option (foo.my_option) = {
	x: 1
	y: 2
	r: {
		r: 1
	}
  };
}

message MyMessage1 {
  option (foo.my_option).x = 1;
  option (foo.my_option).y = 2;
  option (foo.my_option).r.r = 1;
}

// foo.proto
syntax = "proto3";
import "google/protobuf/descriptor.proto";
package foo;

message Pair {
  int32 x = 1;
  int32 y = 2;
  R r = 3;
}

message R {
  int32 r = 1;
}

extend google.protobuf.MessageOptions {
  Pair my_option = 51234;
}

I am not sure that we should support such complicated scenarios in 1.x branch. Few days ago I started work on proto3 parser based on ANTLR4, so most probably it will be supported in 2.x release.

Yep.  I had no need for complex static options like in protobuf.
Protostuff is very close to flatbuffer's dsl in terms of simplicity where the options/attributes are simple types and are dynamic (hash).
For my hybrid c++/java apps, I have a flatbuffers+protostuff dsl/generator which uses ST4 that works out of the box because of the similarity.
I know that those looking for proto3 compat would certainly welcome the new parser.

I really like ANTLR4, initially I was thinking it will be complex, but at the end I came to simple set of rules:
option
    : optionName ASSIGN optionValue
    ;
optionName
    : NAME
    | customOptionName
    ;
customOptionName
    : '(' declarationRef ')' ('.' declarationRef)?
    ;
declarationRef
    : NAME ('.' NAME)*
    ;
optionValue
    : INTEGER_VALUE
    | BOOLEAN_VALUE
    | STRING_VALUE
    | textFormat
    ;
textFormat
    : '{' textFormatEntry* '}'
    ;
textFormatEntry
    : NAME ':' optionValue
    ;

By simple I meant options mapping to simple values (basically a hash with string key and any dynamic value, not a static user-defined option).

I understand :)
I'm using map with dynamic values too - it is the most efficient data structure in this case.
The difference is that value for complex option is also Map<String,Object>, where values might be primitives or another maps (basically, it is a tree). For example, output of my Enum.toString() (one of unit tests) :
Enum{name=E, values=[], standardOptions={}, customOptions={core.foo={a=1, b=2, c={d=3}}}}

source:
enum E {
  option (core.foo).a = 1;
  option (core.foo).b = 2;
  option (core.foo).c.d = 3;
}

But that's not related to current protostuff implementation.

UPD: Maybe it makes sense to add parser rules for such options, but just to suppress errors. One of options from my project:
option (google.protobuf.objectivec_file_options).class_prefix = "LV2";

This line produces errors. We do not use it for code generation, but other team use (for objective c).

Forgive my late reply (Had a short vacation, whoa that was awesome!).  I'm not opposed to your proposal.  I definitely see the use-case where you want proto3 semantics.

UPD: I finished ANTLR4 parser, compatible with proto 2 specification: https://github.com/kshchepanovskyi/proto-compiler/blob/master/src/main/antlr4/io/protostuff/compiler/parser/ProtoParser.g4
I used protobuf's unit tests to ensure compatibility: https://github.com/kshchepanovskyi/proto-compiler/blob/master/src/test/resources/protobuf_unittest/unittest_custom_options.proto
All forms of custom options are supported. Currently I am working on validation part.

Nice work!

implemented in https://github.com/protostuff/protostuff-compiler

authors: dyu kshchepanovskyi

repo: protostuff/protostuff


text:
feat: All in landing page

To solve the 'ghost town' problem, I suggest gathering all the content in the landing page. It can easily be split apart later when it gets more stuffy, in about, community, events, media, publications, etc

This sounds like a great idea! I'll review it over the weekend and add intro content based on #8. Thanks!

authors: agentofuser orpheuslummis

repo: provablysafeai/website