Transform
Transformations allows you to modify scraped config items before they are saved, common use cases include:
- Linking configuration items
- Removing extraneous or overly verbose fields
- Masking sensitive data
- Excluding duplicate changes or changes with a high rate
Field | Description | Scheme |
---|---|---|
transform.exclude | Remove fields from a scraped config | []Exclude |
transform.mask | Replace sensitive fields with a hash to enable change detection on secrets | []Mask |
transform.changes.exclude | Ignore changes | []CEL with Change Context |
transform.changes.mapping | Categorize changes | Mapping |
transform.expr | CEL | |
transform.relationship | Create relationships between items | Relationships |
Config Items
Field Exclusions
Exclusions allow you to remove fields from the config
of an item. This is useful when you want to remove sensitive or overly verbose from being recorded.
kubernetes-exclude-superfluous-fields.yamlapiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
name: kubernetes-scraper
spec:
kubernetes:
- clusterName: local-kind-cluster
transform:
exclude:
- types:
- Kubernetes::Pod
jsonpath: '.metadata.generateName'
Field | Description | Scheme | Required |
---|---|---|---|
jsonpath | All matching elements will be removed from the config | jsonpath | true |
types | Only run exclusion rules for these config types, if empty apply to all | []string |
Masking
Masking replaces sensitive fields with a hash or static string. A hash can be used to determine if a field changed without revealing original values.
file-mask-scraper.yamlapiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
name: file-mask-scraper
spec:
file:
- type: Config
id: $.id
name: $.name
transform:
mask:
- selector: config.name == 'Config1'
jsonpath: $.password
value: md5sum # Change detection will pick up that a change has occurred, but not what the change was
- selector: config.name == 'Config1'
jsonpath: $.secret
value: '***' # Replace the secret with a fixed mask, no change detection will be possible
paths:
- fixtures/data/single-config.json
Field | Description | Scheme |
---|---|---|
selector | Filter which config items to apply masks on | CEL with ScrapeResult context |
jsonpath | Values to mask | JSONPath |
value | The replacement value of matched elements | md5 or any static string e.g. *** |
Masks are applied in the order they are specified in the configuration file.
Changes
Exclusions
Some configurations can change frequently and may not be relevant. For example, a Kubernetes::Node
configuration changes often as pods launched and stopped. From the node's perspective, these image changes are irrelevant.
This is where exclusions become useful. Here's an example that ignores all image changes in a Kubernetes::Node
configuration:
kubernetes-scraper.yamlapiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
name: kubernetes-scraper
spec:
kubernetes:
- clusterName: local-kind-cluster
transform:
changes:
exclude:
- 'config_type == "Kubernetes::Node" && details.message == "status.images"'
Mapping
When you encounter a diff change, unlike an event-based change, it can sometimes appear unclear. The summary of the change may not immediately indicate its purpose. For example, the change 'status.images' might not be self-explanatory. To clarify this, you can assign types to these diff changes using mapping.
kubernetes-scraper.yamlapiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
name: kubernetes-scraper
spec:
kubernetes:
- clusterName: local-kind-cluster
transform:
changes:
mapping:
- filter: >
change.change_type == 'diff' && change.summary == "status.containerStatuses" &&
patch != null && has(patch.status) && has(patch.status.containerStatuses) &&
patch.status.containerStatuses.size() > 0 &&
has(patch.status.containerStatuses[0].restartCount)
type: PodCrashLooping
- filter: >
change.change_type == 'diff' && change.summary == "status.images" && config.kind == "Node"
type: ImageUpdated
Field | Description | Scheme |
---|---|---|
filter | Selects changes to apply the mapping | CEL with Change Context |
action | What action to take on the change, if delete then the corresponding config item is marked as deleted | delete or ignore |
type | New change type | string |
summary | New summary of the change | Go Template with Change Context |
Scripting
Scripting modifies the scraped configuration using CEL before saving it to the database. This process is beneficial for data normalization, default value population, and sensitive field masking.
Field | Description | Scheme | Context |
---|---|---|---|
expr | Transform a config item | CEL that returns []ScrapeResult | config JSON result Scrape Result |
file-scraper.yamlapiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
name: file-scraper
spec:
file:
- type: Config
id: $.id
name: $.name
transform:
expr: |
[(config + {'source': 'scraper', 'password': config.password.size()})].toJSON()
paths:
- config.json
Using the following file
{
"name": "Config1",
"id": 1,
"password": "p1",
"secret": "secret_1"
}
The transformation would emit:
{
"name": "Config1",
"id": 1,
"password": 2,
"source": "scraper",
"secret": "secret_1"
}