File

The file scraper is used to create config items from files in a local folder (or git). This can be used to track changes in files like /etc/hosts or /etc/passwd, or for service metadata stored in git.

See Kubernetes Files for scraping files inside running kubernetes pods.

file-scraper.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: file-git-scraper
spec:
  file:
    - type: $.kind
      id: $.metadata.name
      url: github.com/flanksource/canary-checker?ref=076cf8b888f2dbaca26a7cc98a4153c154220a22
      paths:
        - fixtures/minimal/http_pass.yaml

Scraper

Field	Description	Scheme	Required
schedule	Specify the interval to scrape in cron format. Defaults to every 60 minutes.	string
full	Set to true to extract changes from scraped configurations. Defaults to false.	bool
retention	Settings for retaining changes, analysis and scraped items	Retention
file	Specifies the list of File configurations to scrape.	[]File

File

Field	Description	Scheme
id*	A deterministic or natural id for the resource	string or JSONPath
paths*	Specify paths to configuration(s) for scraping	[]glob
type*	e.g. File::Host, File::Tomcat, File::Pom	string or JSONPath
url	Specify URL e.g github repository containing the configuration(s)	string
class		string or JSONPath
createFields	Identify the created time for a resource (if different to scrape time). If multiple fields are specified, the first non-empty value will be used	[]string or []JSONPath
deleteFields	Identify when a config item was deleted. If multiple fields are specified, the first non-empty value will be used	[]string or []JSONPath
format	Format of config item e.g. xml, properties. Defaults to JSON	string
items	Extract multiple config items from this array	JSONPath
labels	Labels for each config item.	map[string]string
name	. Defaults to id	string or JSONPath
properties	Custom templatable properties for the scraped config items.	[]ConfigProperty
tags	Tags for each config item. Max allowed: 5	[]ConfigTag
timestampFormat	Format to parse timestamps in createFields and deletedFields. Defaults to RFC3339	Go time format
transform	Transform configs after they've been scraped	Transform