Notifications

Mission Control provides a flexible, event-based notification system that enables filtering and templating of notifications via email, push, slack, teams, etc.

Notifications are triggered by events. An event is a specific occurrence or change in state within the system. For example, notifications can be granular such as a health check failing or a pod crashlooping - or high level based on the health of the system as a whole.

check-failed.yaml
apiVersion: mission-control.flanksource.com/v1
kind: Notification
metadata:
  name: http-check
  namespace: default
spec:
  events:
    - check.failed
  to:
    email: alerts@acme.com

This notification will send an email to alerts@acme.com whenever a health check fails.

Field	Description	Scheme	Template Env
events*	A list of events that should trigger this notification	[]Event
body	Channel dependent e.g. email body for email	Go Template	CheckEvents ConfigEvents ComponentEvents
filter	Filter narrows down the event trigger. Example: check.type == 'http' filter on check.failed event will only send notification for failing http checks.	CEL	CheckEvents ConfigEvents ComponentEvents
repeatGroup	RepeatGroup allows notifications to be grouped by certain set of keys and only send one per group within the specified repeat interval. Valid group keys: resource_id & source_event.	[]string
repeatInterval	The waiting time to resend a notification after it has been succefully sent.	Duration
title	Channel dependent e.g. subject for email	Go Template	CheckEvents ConfigEvents ComponentEvents
to.connection	Connection to external service	Connection
to.email	Send an email to any email address	email
to.person	Send an email to the person	UUID or email of a person
to.properties	Properties are channel dependent special directives to modify the notification message. Example: for email, FromAddress=admin@flanksource.com modifies the sender of the email. Read more	map[string]string
to.team	Send an email to every person in the team	UUID or name of team
to.url	Custom notification URL	Channel
waitFor	The duration to delay sending a health-based notification. After this period, the health status is reassessed to confirm it hasn't changed, helping prevent false alarms from transient issues.	Duration

Single Recipient

Only one recipient can be specified

Templating

You can use Go Templates to customize the content of notification titles and bodies. This allows you to include dynamic information from the event that triggered the notification.

For example, the following notification definition uses a Go Template to include the name and health status of the check in the notification title:

notification-template
apiVersion: mission-control.flanksource.com/v1
kind: Notification
spec:
  events:
    - check.failed
    - check.passed
  title: Check {{.check.name}} is {{.check.health}}

In this example {{.check.name}} and {{.check.health}} are placeholders that will be replaced with the actual values of the name and health fields from the check object associated with the event.

Markdown

All notification templates in Mission Control are defined in markdown, and depending on the support of the notification channel it will either be converted to HTML or plain text.

Filtering Events

Mission Control allows you to fine-tune your notification delivery by filtering events using CEL. This enables you to specify precise conditions that dictate when a notification should be triggered. For instance, you can configure notifications to be sent only for specific event types, specific resources, or when certain conditions are met.

The CEL filters you define have access to the same set of variables as the templates, providing flexibility in defining your filtering criteria. These variables, representing various attributes and properties of the events, empower you to create highly targeted notifications.

To illustrate, consider a scenario where you want to receive notifications exclusively for failed HTTP checks. The following example demonstrates how to achieve this using a CEL filter:

filter-http.yaml
apiVersion: mission-control.flanksource.com/v1
kind: Notification
metadata:
  name: http-alerts
spec:
  events:
    - check.failed
  # Filter for events where the check type is 'http'
  filter: check.type == 'http'
  to:
    email: alerts@acme.com

In this example:

• The filter field specifies the CEL expression check.type == 'http'.
• This expression evaluates to true only when the type attribute of the check object is equal to 'http'.
• Consequently, notifications will be sent only for events that satisfy this condition, ensuring that you are alerted specifically for failed HTTP checks.