Transform

Transformations allows you to modify scraped config items before they are saved, common use cases include:

• Linking configuration items
• Removing extraneous or overly verbose fields
• Masking sensitive data
• Excluding duplicate changes or changes with a high rate

Field	Description	Scheme
transform.exclude	Remove fields from a scraped config	[]Exclude
transform.mask	Replace sensitive fields with a hash to enable change detection on secrets	[]Mask
transform.changes.exclude	Ignore changes	[]CEL with Change Context
transform.changes.mapping	Categorize changes	Mapping
transform.expr		CEL
transform.relationship	Create relationships between items	Relationships

Config Items

Field Exclusions

Exclusions allow you to remove fields from the config of an item. This is useful when you want to remove sensitive or overly verbose from being recorded.

kubernetes-exclude-superfluous-fields.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: kubernetes-scraper
spec:
  kubernetes:
    - clusterName: local-kind-cluster
      transform:
        exclude:
          - types:
              - Kubernetes::Pod
            jsonpath: '.metadata.generateName'

Field	Description	Scheme	Required
jsonpath	All matching elements will be removed from the config	jsonpath	true
types	Only run exclusion rules for these config types, if empty apply to all	[]string

Masking

Masking replaces sensitive fields with a hash or static string. A hash can be used to determine if a field changed without revealing original values.

file-mask-scraper.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: file-mask-scraper
spec:
  file:
    - type: Config
      id: $.id
      name: $.name
      transform:
        mask:
          - selector: config.name == 'Config1'
            jsonpath: $.password
            value: md5sum # Change detection will pick up that a change has occurred, but not what the change was
          - selector: config.name == 'Config1'
            jsonpath: $.secret
            value: '***' # Replace the secret with a fixed mask, no change detection will be possible
      paths:
        - fixtures/data/single-config.json

Field	Description	Scheme
selector	Filter which config items to apply masks on	CEL with Config Item context
jsonpath	Values to mask	JSONPath
value	The replacement value of matched elements	md5 or any static string e.g. ***

info

Masks are applied in the order they are specified in the configuration file.

Changes

Exclusions

Some configurations can change frequently and may not be relevant. For example, a Kubernetes::Node configuration changes often as pods launched and stopped. From the node's perspective, these image changes are irrelevant.

This is where exclusions become useful. Here's an example that ignores all image changes in a Kubernetes::Node configuration:

kubernetes-scraper.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: kubernetes-scraper
spec:
  kubernetes:
    - clusterName: local-kind-cluster
      transform:
        changes:
          exclude:
            - 'config_type == "Kubernetes::Node" && details.message == "status.images"'

Mapping

When you encounter a diff change, unlike an event-based change, it can sometimes appear unclear. The summary of the change may not immediately indicate its purpose. For example, the change 'status.images' might not be self-explanatory. To clarify this, you can assign types to these diff changes using mapping.

kubernetes-scraper.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: kubernetes-scraper
spec:
  kubernetes:
    - clusterName: local-kind-cluster
      transform:
        changes:
          mapping:
            - filter: >
                change.change_type == 'diff' && change.summary == "status.containerStatuses" &&
                patch != null && has(patch.status) && has(patch.status.containerStatuses) &&
                patch.status.containerStatuses.size() > 0 &&
                has(patch.status.containerStatuses[0].restartCount)
              type: PodCrashLooping
            - filter: >
                change.change_type == 'diff' && change.summary == "status.images" && config.kind == "Node"
              type: ImageUpdated

Field	Description	Scheme
filter	Selects changes to apply the mapping	CEL with Change Context
action	What action to take on the change, if delete then the corresponding config item is marked as deleted	delete or ignore
type	New change type	string
summary	New summary of the change	Go Template with Change Context

Scripting

Scripting modifies the scraped configuration using CEL before saving it to the database. This process is beneficial for data normalization, default value population, and sensitive field masking.

Field	Description	Scheme	Context
expr	Transform a config item	CEL that returns []ScrapeResult	config JSON result Scrape Result

file-scraper.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: file-scraper
spec:
  file:
    - type: Config
      id: $.id
      name: $.name
      transform:
        expr: |
          [(config + {'source': 'scraper', 'password': config.password.size()})].toJSON()
      paths:
        - config.json

Using the following file

  {
    "name": "Config1",
    "id": 1,
    "password": "p1",
    "secret": "secret_1"
  }

The transformation would emit:

  {
    "name": "Config1",
    "id": 1,
    "password": 2,
    "source": "scraper",
    "secret": "secret_1"
  }