Alertmanager

Checks Prometheus AlertManager for any firing alerts.

The following example transforms the list of alerts so that each alert becomes a single check result. Without the transform the health check fails if any alerts are firing.

alert-manager-check.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: alertmanager
spec:
  schedule: "@every 5m"
  alertmanager:
    - url: https://alertmanager.demo.aws.flanksource.com
      name: alertmanager-check
      alerts:
        - .*
      ignore:
        - KubeScheduler.*
      exclude_filters:
        namespace: elastic-system
      transform:
        expr: |
          results.alerts.map(r, {
            'name': r.name + r.fingerprint,
            'namespace': 'namespace' in r.labels ? r.labels.namespace : '',
            'labels': r.labels,
            'icon': 'alert',
            'message': r.message,
            'description': r.message,
          }).toJSON()
      relationships:
        components:
          - name:
              label: pod
            namespace:
              label: namespace
            type:
              value: KubernetesPod
        configs:
          - name:
              label: pod
            namespace:
              label: namespace
            type:
              value: Kubernetes::Pod

Field	Description	Scheme
name*	Name of the check, must be unique within the canary	string
url*	The URL of the Prometheus Alertmanager instance	string
alerts	A list of alert prefix names to include	[]string
filters	A map of label to value prefixes to find alerts on	map[string]string
ignore	A list of alert prefix names to exclude	[]string
description	Description for the check	string
display	Expression to change the formatting of the display	Expression
icon	Icon for overwriting default icon on the dashboard	Icon
labels	Labels for check	map[string]string
metrics	Metrics to export from	[]Metrics
test	Evaluate whether a check is healthy	Expression
transform	Transform data from a check into multiple individual checks	Expression

Relationships

Field	Description	Scheme
components	Select all the components to link to this check	Lookup
config	Select all the configs to link to this check	Lookup

Lookup

Field	Description	Scheme
name	Initial delay before the checks are run	Lookup Selector
namespace	Timeout for the check	Lookup Selector
type	Timeout for the check	Lookup Selector

Lookup selector

Lookup specifies the type of lookup to perform.

Field	Description	Scheme
expr	Expr is a cel-expression	string
value	Value is the static value to use	string
value	Label specifies the key to lookup on the label	string

Inserting checks into different namespaces

You can specify different namespaces for checks using the namespace field. This is helpful when checks are dynamically generated via transformation and need to be assigned their respective namespace.

alert-manager-check.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: alertmanager
spec:
  schedule: "@every 5m"
  alertmanager:
    - url: https://alertmanager.demo.aws.flanksource.com
      name: alertmanager-check
      alerts:
        - .*
      ignore:
        - KubeScheduler.*
      exclude_filters:
        namespace: elastic-system
      transform:
        expr: |
          results.alerts.map(r, {
            'name': r.name + r.fingerprint,
            'namespace': 'namespace' in r.labels ? r.labels.namespace : '',
            'labels': r.labels,
            'icon': 'alert',
            'message': r.message,
            'description': r.message,
          }).toJSON()
      relationships:
        components:
          - name:
              label: pod
            namespace:
              label: namespace
            type:
              value: KubernetesPod
        configs:
          - name:
              label: pod
            namespace:
              label: namespace
            type:
              value: Kubernetes::Pod

The above alertmanager canary can reside on any namespace and the check it generates are assigned the correct namespace derived from the alerts.