kedro.config.TemplatedConfigLoader

class kedro.config.TemplatedConfigLoader(conf_source, env=None, runtime_params=None, config_patterns=None, *, base_env='base', default_run_env='local', globals_pattern=None, globals_dict=None)[source]

Extension of the ConfigLoader class that allows for template values, wrapped in brackets like: ${…}, to be automatically formatted based on the configs.

The easiest way to use this class is by setting the CONFIG_LOADER_CLASS constant in settings.py.

Example:

# in settings.py
from kedro.config import TemplatedConfigLoader

CONFIG_LOADER_CLASS = TemplatedConfigLoader
CONFIG_LOADER_ARGS = {
    "globals_pattern": "*globals.yml",
}

The contents of the dictionary resulting from the globals_pattern get merged with the globals_dict. In case of conflicts, the keys in globals_dict take precedence. If the formatting key is missing from the dictionary, the default template value is used (the format is “${key|default value}”). If no default is set, a ValueError will be raised.

Global parameters can be namespaced as well. An example could work as follows:

globals.yml

bucket: "my_s3_bucket"

environment: "dev"

datasets:
    csv: "pandas.CSVDataSet"
    spark: "spark.SparkDataSet"

folders:
    raw: "01_raw"
    int: "02_intermediate"
    pri: "03_primary"
    fea: "04_feature"

catalog.yml

raw_boat_data:
    type: "${datasets.spark}"
    filepath: "s3a://${bucket}/${environment}/${folders.raw}/boats.csv"
    file_format: parquet

raw_car_data:
    type: "${datasets.csv}"
    filepath: "s3://${bucket}/data/${environment}/${folders.raw}/cars.csv"

This uses jmespath in the background. For more information see: https://github.com/jmespath/jmespath.py and https://jmespath.org/.

Attributes

conf_paths

Property method to return deduplicated configuration paths.

Methods

clear()

copy()

fromkeys(iterable[, value])

get(*patterns)

Tries to resolve the template variables in the config dictionary provided by the ConfigLoader (super class) get method using the dictionary of replacement values obtained in the __init__ method.

items()

keys()

pop(k[,d])

If key is not found, d is returned if given, otherwise KeyError is raised.

popitem()

as a 2-tuple; but raise KeyError if D is empty.

setdefault(k[,d])

update([E, ]**F)

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

values()

__init__(conf_source, env=None, runtime_params=None, config_patterns=None, *, base_env='base', default_run_env='local', globals_pattern=None, globals_dict=None)[source]

Instantiates a TemplatedConfigLoader.

Parameters:

  • conf_source – Path to use as root directory for loading configuration.

  • env – Environment that will take precedence over base.

  • runtime_params – Extra parameters passed to a Kedro run.

  • config_patterns – Regex patterns that specify the naming convention for configuration files so they can be loaded. Can be customised by supplying config_patterns as in CONFIG_LOADER_ARGS in settings.py.

  • base_env –

  • default_run_env –

  • globals_pattern – Optional keyword-only argument specifying a glob pattern. Files that match the pattern will be loaded as a formatting dictionary.

  • globals_dict – Optional keyword-only argument specifying a formatting dictionary. This dictionary will get merged with the globals dictionary obtained from the globals_pattern. In case of duplicate keys, the globals_dict keys take precedence.

clear() → None.  Remove all items from D.
property conf_paths

Property method to return deduplicated configuration paths.

copy()
classmethod fromkeys(iterable, value=None)
get(*patterns)[source]

Tries to resolve the template variables in the config dictionary provided by the ConfigLoader (super class) get method using the dictionary of replacement values obtained in the __init__ method.

Parameters:

*patterns – Glob patterns to match. Files, which names match any of the specified patterns, will be processed.

Returns:

A Python dictionary with the combined configuration from all configuration files. Note: any keys that start with _ will be ignored. String values wrapped in ${…} will be replaced with the result of the corresponding JMESpath expression evaluated against globals.

Raises:

ValueError – malformed config found.

items() → a set-like object providing a view on D's items
keys() → a set-like object providing a view on D's keys
pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() → (k, v), remove and return some (key, value) pair

as a 2-tuple; but raise KeyError if D is empty.

setdefault(k[, d]) → D.get(k,d), also set D[k]=d if k not in D
update([E, ]**F) → None.  Update D from mapping/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

values() → an object providing a view on D's values