Folder

Checks the contents of a folder for size, age, and count. Folder based checks are useful in a number of scenarios:

Verifying that backups have been uploaded and are the appropriate size
Checking that logs or other temporary files are being cleaned up
For batch processes:
- Checking if files are being processed (and/or produced)
- Checking the size of queue processing backlog
- Checking if any error (.err or .log) files have been produced.

folder-check.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: folder-check
spec:
  schedule: "@every 5m"
  folder:
    - path: /etc/
      name: folder-check-min
      description: Checks if there are at least 10 files in the folder
      minCount: 10

Field	Description	Scheme
`name*`	Name of the check, must be unique within the canary	`string`
`path*`	A local folder path or a remote folder `smb://`, `sftp://`, `s3://` or `gcs://`
`awsConnection`	AWS Access credentials	AWS
`filter`	Filter objects out before checking if they are valid	FolderFilter
`gcpConnection`	Connection details for GCP	GCP
`maxAge`	The oldest age a file can be, often used to check for unprocessed files or files that have not been cleaned up	Duration
`maxCount`	The maximum number of files inside the `path`, can be used in conjunction with `filter.regex` to detect error files	`integer`
`maxSize`	The maximum file size	Size
`minAge`	The youngest age a file can be	Duration
`minCount`	The minimum number of files inside the `path`	`integer`
`minSize`	The minimum file size, can be used to detect backups that did not upload successfully	Size
`sftpConnection`	Connection details	SFTP
`smbConnection`	SMB connection details	SMB
`description`	Description for the check	`string`
`display`	Expression to change the formatting of the display	`Expression`
`icon`	Icon for overwriting default icon on the dashboard	Icon
`labels`	Labels for check	map[string]string
`metrics`	Metrics to export from	`[]Metrics`
`test`	Evaluate whether a check is healthy	`Expression`
`transform`	Transform data from a check into multiple individual checks	`Expression`

FolderFilter

Field	Description	Scheme
`maxAge`	MaxAge the latest object should be younger than defined age	Duration
`maxSize`	MaxSize of the files inside the searchPath	Size
`minAge`	MinAge the latest object should be older than defined age	Duration
`minSize`	MinSize of the files inside the searchPath	Size
`regex`	Filter files based on regular expression	regex

e.g. to verify that database backups are being performed

postgres-backup-check.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: folder-check
spec:
  schedule: "@every 5m"
  folder:
    - name: pg-backup checks
      path: /data/backups
      filter:
        regex: "pg-backups-.*.zip"
      maxAge: 1d # require a daily backup
      minSize: 10mb # the backup should be at least 10mb

Result Variables

The following fields are available in test, display and transform expressions

Field	Scheme
`Oldest`	os.FileInfo
`Newest`	os.FileInfo
`MinSize`	os.FileInfo
`MaxSize`	os.FileInfo
`SupportsTotalSize` (Only true for SMB folders)	bool
`SupportsAvailableSize` (Only true for SMB folders)	bool
`TotalSize`	int64
`AvailableSize`	int64
`Files`	[]os.FileInfo

FolderFilter

Field	Description	Scheme	Required
`minAge`		`Duration`
`maxAge`		`Duration`
`minSize`		`Size`
`maxSize`		`Size`
`regex`		`string`

Connection Types

SFTP

Field	Description	Scheme
`connection`	The connection url to use, mutually exclusive with `host` and `port`	Connection
`host`	SFTP host	`string`
`port`	SFTP port	`integer`
`username`		EnvVar
`password`		EnvVar

SFTP Check

exec_artifact.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: sftp-pass
spec:
  schedule: "@every 5m"
  folder:
    - path: /tmp
      name: sample sftp check
      sftpConnection:
        host: 192.168.1.5
        username:
          value: <username>
        password:
          value: <password>
      maxCount: 10

S3

Field	Description	Scheme
`connection`	The connection url to use, mutually exclusive with `accessKey` and `secretKey`	Connection
`accessKey`		EnvVar
`secretKey`		EnvVar
`region`	The AWS region	`string`
`endpoint`	Custom AWS Endpoint to use
`skipTLSVerify`	Skip TLS verify when connecting to AWS	`boolean`

S3 Bucket

exec_artifact.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: s3-bucket-pass
  namespace: canaries
spec:
  schedule: "@every 5m"
  folder:
    # Check for any backup not older than 7 days and min size 25 bytes
    - path: s3://tests-e2e-1
      name: mysql backup check
      awsConnection:
        accessKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_ACCESS_KEY_ID
        secretKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_SECRET_ACCESS_KEY
        region: "minio"
        endpoint: "http://minio.minio:9000"
        usePathStyle: true
        skipTLSVerify: true
      filter:
        regex: "(.*)backup.zip$"
      maxAge: 7d
      minSize: 25b
    # Check for any mysql backup not older than 7 days and min size 25 bytes
    - path: s3://tests-e2e-1
      name: mysql backup check (7 days and 25 bytes size)
      awsConnection:
        accessKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_ACCESS_KEY_ID
        secretKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_SECRET_ACCESS_KEY
        region: "minio"
        endpoint: "http://minio.minio:9000"
        usePathStyle: true
        skipTLSVerify: true
      filter:
        regex: "mysql\\/backups\\/(.*)\\/mysql.zip$"
      maxAge: 7d
      minSize: 25b
    # Check for any pg backup not older than 7 days and min size 50 bytes
    - path: s3://tests-e2e-1
      name: pg backup check (7 days and 50 bytes size)
      awsConnection:
        accessKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_ACCESS_KEY_ID
        secretKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_SECRET_ACCESS_KEY
        region: "minio"
        endpoint: "http://minio.minio:9000"
        usePathStyle: true
        skipTLSVerify: true
      filter:
        regex: "pg\\/backups\\/(.*)\\/backup.zip$"
      maxAge: 7d
      minSize: 25b
    - name: recursive folders
      namespace: default
      path: s3://recursive-test/developers/**/*.txt
      minCount: 4
      display:
        expr: results.?files.orValue([]).map(i, i.name).join(", ")
      awsConnection:
        region: "minio"
        endpoint: "http://minio.minio:9000"
        usePathStyle: true
        skipTLSVerify: true
        accessKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_ACCESS_KEY_ID
        secretKey:
          valueFrom:
            secretKeyRef:
              name: aws-credentials
              key: AWS_SECRET_ACCESS_KEY

SMB

Field	Description	Scheme
`connection`	The connection url to use, mutually exclusive with `host`, `share`, and `credentials`	Connection
`host`	The SMB host	`string`
`port`		`integer`
`domain`	Windows Domain
`username`		EnvVar
`password`		EnvVar

SMB / CIFS

smb.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: smb-pass
spec:
  schedule: "@every 5m"
  folder:
    # Check for any backup not older than 7 days and min size 25 bytes
    - path: \\windows-server\sharename\folder
      smbConnection:
        username:
          valueFrom:
            secretKeyRef:
              name: smb-credentials
              key: USERNAME
        password:
          valueFrom:
            secretKeyRef:
              name: ssmb-credentials
              key: PASSWORD
      filter:
        regex: "(.*)backup.zip$"
      maxAge: 7d
      minSize: 25b

GCS

Field	Description	Scheme
`connection`	The connection url to use, mutually exclusive with `credentials`	Connection
`credentials`	The credentials to use for authentication	EnvVar

GCS Check

gcs.yaml
---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: recursive-folder-check
spec:
  schedule: "@every 5m"
  folder:
    - path: gcs://folder-check-test/recursive-test
      name: recursive folders
      namespace: default
      minCount: 3
      recursive: true
      display:
        expr: results.?files.orValue([]).map(i, i.name).join(", ")
      gcpConnection:
        connection: connection://gcs/flanksource-prod

Folder

FolderFilter​

Result Variables​

FolderFilter​

Connection Types​

SFTP​

S3​

SMB​

GCS​

FolderFilter

Result Variables

FolderFilter

Connection Types

SFTP

S3

SMB

GCS