Skip to main content

Transform

Transformations allows you to modify scraped config items before they are saved, common use cases include:

  • Linking configuration items
  • Removing extraneous or overly verbose fields
  • Masking sensitive data
  • Excluding duplicate changes or changes with a high rate
FieldDescriptionScheme
transform.excludeRemove fields from a scraped config[]Exclude
transform.maskReplace sensitive fields with a hash to enable change detection on secrets[]Mask
transform.changes.excludeIgnore changes[]CEL with Change Context
transform.changes.mappingCategorize changesMapping
transform.exprCEL
transform.relationshipCreate relationships between itemsRelationships

Config Items

Field Exclusions

Exclusions allow you to remove fields from the config of an item. This is useful when you want to remove sensitive or overly verbose from being recorded.

kubernetes-exclude-superfluous-fields.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
name: kubernetes-scraper
spec:
kubernetes:
- clusterName: local-kind-cluster
transform:
exclude:
- types:
- Kubernetes::Pod
jsonpath: '.metadata.generateName'
FieldDescriptionSchemeRequired
jsonpathAll matching elements will be removed from the configjsonpathtrue
typesOnly run exclusion rules for these config types, if empty apply to all[]string

Masking

Masking replaces sensitive fields with a hash or static string. A hash can be used to determine if a field changed without revealing original values.

file-mask-scraper.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
name: file-mask-scraper
spec:
file:
- type: Config
id: $.id
name: $.name
transform:
mask:
- selector: config.name == 'Config1'
jsonpath: $.password
value: md5sum # Change detection will pick up that a change has occurred, but not what the change was
- selector: config.name == 'Config1'
jsonpath: $.secret
value: '***' # Replace the secret with a fixed mask, no change detection will be possible
paths:
- fixtures/data/single-config.json
FieldDescriptionScheme
selectorFilter which config items to apply masks onCEL with ScrapeResult context
jsonpathValues to maskJSONPath
valueThe replacement value of matched elementsmd5 or any static string e.g. ***
info

Masks are applied in the order they are specified in the configuration file.

Changes

Exclusions

Some configurations can change frequently and may not be relevant. For example, a Kubernetes::Node configuration changes often as pods launched and stopped. From the node's perspective, these image changes are irrelevant.

This is where exclusions become useful. Here's an example that ignores all image changes in a Kubernetes::Node configuration:

kubernetes-scraper.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
name: kubernetes-scraper
spec:
kubernetes:
- clusterName: local-kind-cluster
transform:
changes:
exclude:
- 'config_type == "Kubernetes::Node" && details.message == "status.images"'

Mapping

When you encounter a diff change, unlike an event-based change, it can sometimes appear unclear. The summary of the change may not immediately indicate its purpose. For example, the change 'status.images' might not be self-explanatory. To clarify this, you can assign types to these diff changes using mapping.

kubernetes-scraper.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
name: kubernetes-scraper
spec:
kubernetes:
- clusterName: local-kind-cluster
transform:
changes:
mapping:
- filter: >
change.change_type == 'diff' && change.summary == "status.containerStatuses" &&
patch != null && has(patch.status) && has(patch.status.containerStatuses) &&
patch.status.containerStatuses.size() > 0 &&
has(patch.status.containerStatuses[0].restartCount)
type: PodCrashLooping
- filter: >
change.change_type == 'diff' && change.summary == "status.images" && config.kind == "Node"
type: ImageUpdated
FieldDescriptionScheme
filterSelects changes to apply the mappingCEL with Change Context
actionWhat action to take on the change, if delete then the corresponding config item is marked as deleteddelete or ignore
typeNew change typestring
summaryNew summary of the changeGo Template with Change Context

Scripting

Scripting modifies the scraped configuration using CEL before saving it to the database. This process is beneficial for data normalization, default value population, and sensitive field masking.

FieldDescriptionSchemeContext
exprTransform a config itemCEL that returns []ScrapeResultconfig JSON
result Scrape Result
file-scraper.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
name: file-scraper
spec:
file:
- type: Config
id: $.id
name: $.name
transform:
expr: |
[(config + {'source': 'scraper', 'password': config.password.size()})].toJSON()
paths:
- config.json

Using the following file

{
"name": "Config1",
"id": 1,
"password": "p1",
"secret": "secret_1"
}

The transformation would emit:

{
"name": "Config1",
"id": 1,
"password": 2,
"source": "scraper",
"secret": "secret_1"
}