Environment Variables

Python tasks in Datablast receive date information and dynamic configuration through environment variables.

Available Environment Variables

Variable	Description	Example
BLAST_DATA_INTERVAL_START	Start of data interval	2024-01-15T00:00:00+00:00
BLAST_DATA_INTERVAL_END	End of data interval	2024-01-16T00:00:00+00:00
BLAST_START_DATE	Data interval start date	2024-01-15
BLAST_END_DATE	Data interval end date	2024-01-16
BLAST_START_DATE_NODASH	Start date without dashes	20240115
BLAST_END_DATE_NODASH	End date without dashes	20240116

Usage Examples

Basic Date Access

import os
from datetime import datetime

# Access date variables through environment variables
execution_date = os.getenv('BLAST_START_DATE')
end_date = os.getenv('BLAST_END_DATE')

print(f"Processing data from {execution_date} to {end_date}")

Date Processing

import os
from datetime import datetime, timedelta

# Get execution date
execution_date = os.getenv('BLAST_START_DATE')

# Convert to datetime object
execution_dt = datetime.strptime(execution_date, '%Y-%m-%d')

# Calculate previous day
previous_day = execution_dt - timedelta(days=1)
previous_day_str = previous_day.strftime('%Y-%m-%d')

print(f"Processing data for {execution_date}")
print(f"Previous day: {previous_day_str}")

File Path Generation

import os
from pathlib import Path

# Generate file paths based on execution date
execution_date = os.getenv('BLAST_START_DATE')
execution_date_nodash = os.getenv('BLAST_START_DATE_NODASH')

# Generate various file path patterns
input_file = f'data/raw_{execution_date}.csv'
output_file = f'data/processed_{execution_date}.csv'
log_file = f'logs/processing_{execution_date}.log'

print(f"Input file: {input_file}")
print(f"Output file: {output_file}")

Database Query Parameters

import os
from datetime import datetime

# Get parameters for database queries
start_date = os.getenv('BLAST_START_DATE')
end_date = os.getenv('BLAST_END_DATE')

# Build query with date parameters
query = f"""
SELECT *
FROM events
WHERE event_date >= '{start_date}'
  AND event_date < '{end_date}'
"""

print(f"Query: {query}")

Configuration Examples

YAML Configuration

name: "data.processing"
type: "python"
description: "Process data using environment variables"
run: "process_data.py"

Python Script Example

data.processing

# @blast.type: python
# @blast.description: Process data using environment variables

import os
from datetime import datetime

# Get execution date from environment variables
execution_date = os.getenv('BLAST_START_DATE')

# Your Python logic here
result = process_data(execution_date)

print(f"Successfully processed data for {execution_date}")

def process_data(execution_date):
    """Process data for the given execution date."""
    # Implementation here
    return "Processing completed"

Local Testing

For local testing, set environment variables manually:

export BLAST_START_DATE=2024-01-15
export BLAST_END_DATE=2024-01-16
export BLAST_START_DATE_NODASH=20240115
export BLAST_END_DATE_NODASH=20240116
export BLAST_DATA_INTERVAL_START=2024-01-15T00:00:00+00:00
export BLAST_DATA_INTERVAL_END=2024-01-16T00:00:00+00:00
python your_script.py

Configuration Parameters

Required Fields

• name: Unique task identifier
• type: Must be python
• run: Python file to execute

Optional Fields

• description: Human-readable description
• depends: Task dependencies
• instance: Instance type
• secrets: Secret management

Related Documentation

• Python Development Guide - Best practices and advanced features
• Python Task Overview
• Instance Types
• Dependencies