Pipeline Connections

Pipeline connections in Datablast provide secure access to external systems, databases, and services. Connections are configured at the pipeline level and can be used by individual tasks.

Key Connection Features

Secure Storage: Encrypted connection credentials
Environment Support: Environment-specific connections
Connection Pooling: Efficient connection management
Validation: Built-in connection validation

Connection Types

Database Connections

BigQuery Connection

# BigQuery connection configuration
default_connections:
  gcpConnectionId: my-bigquery-conn

Snowflake Connection

# Snowflake connection configuration
default_connections:
  snowflake: my-snowflake-conn

PostgreSQL Connection

# PostgreSQL connection configuration
default_connections:
  postgres: my-postgres-conn

MySQL Connection

# MySQL connection configuration
default_connections:
  mysql: my-mysql-conn

Athena Connection

# Athena connection configuration
default_connections:
  aws_conn_id: my-athena-conn

Connection Configuration

Basic Connection Setup

# Basic pipeline with connections
id: analytics-pipeline
schedule: "0 6 * * *"
start_date: "2024-01-01"
# Default connections
default_connections:
  gcpConnectionId: analytics-gcp
  snowflake: analytics-snowflake

Multiple Connection Types

# Pipeline with multiple connection types
id: multi-conn-pipeline
schedule: "0 6 * * *"
start_date: "2024-01-01"
# Multiple connection types
default_connections:
  # Database connections
  gcpConnectionId: analytics-gcp
  snowflake: analytics-snowflake
  postgres: analytics-postgres
  # Cloud connections
  aws_conn_id: analytics-aws

Connection Management

Connection Naming Conventions

Descriptive Names

# Good: Descriptive connection names
default_connections:
  gcpConnectionId: marketing-analytics-gcp
  snowflake: customer-data-snowflake
# Avoid: Generic connection names
default_connections:
  gcpConnectionId: gcp1
  snowflake: sf1

Environment Prefixes

# Good: Environment-prefixed names
default_connections:
  gcpConnectionId: prod-analytics-gcp
  snowflake: prod-customer-snowflake
# Avoid: No environment indication
default_connections:
  gcpConnectionId: analytics-gcp
  snowflake: customer-snowflake

Connection Validation

Built-in Validation

Datablast automatically validates connections:

# Valid connection reference
default_connections:
  gcpConnectionId: existing-gcp-conn  # ✅ Valid

# Invalid connection reference
default_connections:
  gcpConnectionId: non-existent-conn  # ❌ Invalid

Manual Validation

You can validate connections manually:

# Validate connection
datablast validate-connection my-gcp-conn
# Test connection
datablast test-connection my-gcp-conn

Connection Security

Credential Management

Secure Storage

Connections use encrypted credential storage:

# Credentials are stored securely
default_connections:
  gcpConnectionId: secure-gcp-conn  # Credentials are encrypted and stored separately

Access Control

Connections have access control:

# Connection access control
default_connections:
  gcpConnectionId: restricted-gcp-conn  # Access is controlled by connection permissions

Best Practices

1. Use Environment Variables

# Good: Use environment variables
default_connections:
  gcpConnectionId: {{ env.GCP_CONNECTION }}
  snowflake: {{ env.SNOWFLAKE_CONNECTION }}
# Avoid: Hard-coded connection names
default_connections:
  gcpConnectionId: prod-gcp-conn
  snowflake: prod-snowflake-conn

2. Implement Connection Pooling

# Good: Connection pooling configuration
default_connections:
  gcpConnectionId: pooled-gcp-conn  # Connection pooling is configured automatically

3. Monitor Connection Usage

# Good: Monitor connection usage
default_connections:
  gcpConnectionId: monitored-gcp-conn  # Connection usage is monitored automatically

Connection Examples

Analytics Pipeline

# Analytics pipeline with multiple connections
id: analytics-pipeline
schedule: "0 6 * * *"
start_date: "2024-01-01"
description: Daily analytics pipeline
# Analytics-specific connections
default_connections:
  gcpConnectionId: analytics-gcp
  snowflake: analytics-snowflake

Data Processing Pipeline

# Data processing pipeline
id: data-processing
schedule: "0 4 * * *"
start_date: "2024-01-01"
description: Data processing pipeline
# Data processing connections
default_connections:
  gcpConnectionId: processing-gcp
  postgres: processing-postgres
  mysql: processing-mysql

Machine Learning Pipeline

# Machine learning pipeline
id: ml-pipeline
schedule: "0 8 * * *"
start_date: "2024-01-01"
description: Machine learning pipeline
# ML-specific connections
default_connections:
  gcpConnectionId: ml-gcp
  snowflake: ml-snowflake

Troubleshooting

Common Connection Issues

Invalid Connection Reference

Problem: Task fails due to invalid connection

Solution: Use valid connection IDs

# Good: Valid connection reference
default_connections:
  gcpConnectionId: existing-gcp-conn
# Avoid: Invalid connection reference
default_connections:
  gcpConnectionId: non-existent-conn```

#### Connection Timeout

**Problem**: Connection timeout errors

**Solution**: Check connection configuration

```yaml
# Good: Proper connection configuration
default_connections:
  gcpConnectionId: configured-gcp-conn
# Avoid: Misconfigured connection
default_connections:
  gcpConnectionId: misconfigured-gcp-conn```

#### Permission Issues

**Problem**: Connection permission errors

**Solution**: Check connection permissions

```yaml
# Good: Proper permissions
default_connections:
  gcpConnectionId: permissioned-gcp-conn
# Avoid: Insufficient permissions
default_connections:
  gcpConnectionId: restricted-gcp-conn```

### Debugging Connections

#### Connection Testing

Test connections before deployment:

```bash
# Test connection
datablast test-connection my-gcp-conn
# Test with specific environment
datablast test-connection my-gcp-conn --env prod

Connection Logging

Enable connection logging for debugging:

# Enable connection logging
id: debug-pipeline
schedule: "0 6 * * *"
start_date: "2024-01-01"
# Connection with logging
default_connections:
  gcpConnectionId: logged-gcp-conn  # Connection logging is enabled