Configuration
pluginlake.config
Root settings for pluginlake.
Provides Pydantic Settings-based configuration with environment variable support and a medallion-architecture storage layer structure:
All settings can be overridden via environment variables. See each class for the applicable prefix.
LogLevel
StorageLayer
Bases: StrEnum
Storage layers following the medallion architecture.
Attributes:
| Name | Type | Description |
|---|---|---|
RAW |
Bronze layer — raw ingested data, stored as-is. |
|
PROCESSED |
Silver layer — cleaned and transformed data. |
|
OUTPUT |
Gold layer — curated, publication-ready datasets. |
Source code in src/pluginlake/config.py
Settings
Bases: BaseSettings
Base settings for pluginlake.
All settings can be overridden via environment variables
prefixed with PLUGINLAKE_.
Source code in src/pluginlake/config.py
effective_log_level
property
Return the effective log level, accounting for verbose/debug flags.
StorageSettings
Bases: BaseSettings
Storage layer configuration.
Defines the base data directory and the medallion-architecture layer structure (raw / processed / output).
Environment variables are prefixed with PLUGINLAKE_STORAGE_.
Examples:
>>> settings = StorageSettings(base_dir="/tmp/data")
>>> settings.raw_dir
PosixPath('/tmp/data/raw')
>>> settings.get_layer_path(StorageLayer.OUTPUT, "patients.parquet")
PosixPath('/tmp/data/output/patients.parquet')
Source code in src/pluginlake/config.py
83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 | |
raw_dir
property
Path to the raw (bronze) storage layer.
processed_dir
property
Path to the processed (silver) storage layer.
output_dir
property
Path to the output (gold) storage layer.
layer_dirs
property
Return a mapping of each storage layer to its directory path.
get_layer_path(layer, *parts)
Return a path within a specific storage layer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
layer
|
StorageLayer
|
The storage layer to build the path in. |
required |
*parts
|
str
|
Additional path segments appended after the layer directory. |
()
|
Returns:
| Type | Description |
|---|---|
Path
|
Absolute path combining base_dir, layer, and any extra segments. |
Source code in src/pluginlake/config.py
ensure_directories()
Create the base directory and all layer subdirectories.
Raises:
| Type | Description |
|---|---|
OSError
|
If directory creation fails due to permissions or other filesystem errors. |
Source code in src/pluginlake/config.py
ServerSettings
Bases: BaseSettings
HTTP server configuration.
Environment variables are prefixed with PLUGINLAKE_SERVER_.
Source code in src/pluginlake/config.py
PostgresSettings
Bases: BaseSettings
PostgreSQL connection configuration.
Environment variables are prefixed with PLUGINLAKE_POSTGRES_.
Source code in src/pluginlake/config.py
LogSettings
Bases: BaseSettings
Operational JSONL log configuration.
Environment variables are prefixed with PLUGINLAKE_LOG_.
Source code in src/pluginlake/config.py
query_log_dir
property
Path to the query/traffic log directory.
ingestion_log_dir
property
Path to the ingestion history log directory.
DagsterSettings
Bases: BaseSettings
Dagster orchestrator configuration.
Environment variables are prefixed with PLUGINLAKE_DAGSTER_.