Pipelines mit Config-Dicts parametrisieren

A pipeline with hardcoded file paths, column names, and threshold values breaks whenever the environment changes — a different server, a renamed column, or a changed business rule. Every change requires editing the pipeline code itself, creating risk of introducing bugs. The solution is to externalise all variable values into a configuration dictionary that is loaded at runtime and passed to the pipeline functions.

Replace every hardcoded value with an entry in a configuration dictionary. Group related settings logically: input/output paths together, cleaning thresholds together, column name mappings together. The config dict becomes the single source of truth for all pipeline parameters. Changing one value in the config updates every function that uses it without touching the function bodies.

The extract function reads all its parameters from the config: the input path, the date columns to parse, and any encoding or delimiter settings. This means running the same pipeline against a test dataset or a different month's file requires only a config change — no code change. You can maintain separate configs for development, staging, and production environments.

Each transformation function receives the full config and extracts the values it needs. Functions should use config.get('key', default) with sensible defaults so the pipeline is robust against incomplete configs. A function that requires a threshold of 5000 by default but can be overridden via config is both safe and flexible.

For production pipelines, store the config in a JSON file rather than a Python dictionary hardcoded in the script. Load it with json.load() at pipeline start. This allows operations teams to change thresholds without access to the Python code, and enables config versioning through Git — every config change is a tracked commit with a description of the business reason for the change.

Maintain separate config files for each environment: config_dev.json, config_staging.json, and config_prod.json. Determine which to load based on an environment variable. This pattern prevents accidental use of production file paths during development and keeps environment-specific secrets (like database credentials) out of the shared code repository.

Source data often has column names that differ from your internal naming convention. Rather than hard-coding df.rename(columns={'OrderDate': 'order_date', 'Qty': 'quantity'}) in the pipeline body, store the rename mapping in the config. This makes the pipeline agnostic to the source column names and easy to adapt when the upstream data provider changes their export format.

The aggregation phase often groups by different columns for different use cases. Store the group keys and aggregation specs in the config rather than hardcoding them. This lets analysts produce different summary tables (by region, by category, by month) by changing the config, without touching the aggregation function. The function becomes a general-purpose aggregator driven entirely by configuration.

Validate the config at pipeline start to catch missing or invalid keys before any data is loaded. A pipeline that runs for 10 minutes and then fails because revenue_cap was a string instead of a float wastes time. Validate all required keys exist, values are the correct type, and paths are accessible with a short config-check function that runs before any expensive I/O.

Allow users to provide a partial config that overrides only the values they care about. Merge the user config on top of a default config using {**defaults, **user_config}. This pattern provides sensible defaults while remaining fully configurable. It is the same pattern used by popular Python libraries that accept configuration as a dict or kwargs.

Save the config alongside the output file so anyone inspecting the output can immediately reproduce the pipeline run that created it. Store it as a JSON sidecar file with the same name as the output but a .config.json extension. Include the pipeline run timestamp in the saved config for full traceability of every output file.

Test your understanding of Data Analysis concepts from this lesson.

In this lesson you learned: replacing hardcoded values with a config dictionary loaded from JSON, passing config to extract, transform, and aggregate functions for full parameterisation, and validating configs at startup and saving them alongside output files for reproducibility. Next up we explore testing pipeline steps with row-count checks and assertion guards.