Overview

Comprehensive guides to help you build, deploy, and manage data pipelines on Datablast. These guides take you from beginner to advanced usage with practical examples and best practices.

Getting Started

Quick Start

• Quickstart – Build your first pipeline in under 10 minutes
• Platform Overview – Learn about Datablast capabilities

Development Guides

SQL Development

• BigQuery Development – Develop SQL tasks for Google Cloud BigQuery
  • Core Development – Core BigQuery development and configuration
  • Materialization Strategies – Table creation, partitioning, clustering
  • Data Quality Testing – Built-in tests and custom validation
  • Performance Optimization – Query optimization and cost management
• Snowflake Development – Develop SQL tasks for Snowflake data warehouse
  • Core Development – Core Snowflake development and configuration
  • Performance Optimization – Query optimization, clustering, and warehouse sizing
  • Cost Management – Credit optimization and cost monitoring
• Athena Development – Develop SQL tasks for AWS Athena
  • Core Development – Core Athena development and configuration
  • Performance Optimization – Query optimization and performance tuning
  • Data Formats – Parquet, ORC, JSON, and CSV optimization
  • Cost Optimization – Cost management and credit optimization
• PostgreSQL Development – Develop SQL tasks for PostgreSQL databases
  • Core Development – Core PostgreSQL development and configuration
  • Performance Optimization – Query optimization, indexing, and connection management
  • Advanced Features – JSON support, window functions, and custom functions

Python Development

• Environment Variables – Use environment variables in Python tasks
• Instance Types – Configure Python task compute resources
• Python Dependencies – Manage Python package dependencies

Data Quality

• Column Tests – Implement column-level data validation
• Custom Tests – Create custom business logic tests
• Best Practices – Data quality testing best practices

Platform Features

Core Features

• Data Quality Testing – Built-in data validation framework
• Materialization Strategies – Table and view creation strategies
• Pipeline Scheduling – Schedule and manage pipeline execution
• Pipeline Monitoring – Monitor pipeline performance and health

Integration Guides

• Jinja Templates – Use dynamic date and time references
• Multiple Repositories – Organize multiple pipelines
• Shared Utilities – Create and manage shared utility functions

Project Organization

Structure and Setup

• Project Structure – Organize your code and files
• Pipeline Configuration – Configure schedules, connections, and notifications
• Task Configuration – Define and configure tasks

Advanced Topics

• Sensors – Wait for external conditions
• Notifications – Set up alerts and notifications

Examples and Tutorials

SQL Development

• BigQuery Development – BigQuery development guides and examples
• Athena Development – Athena development guides and examples

Python Development

• Python Development – Python development guides and examples

Best Practices

Development

• Code Organization – Organize your code effectively
• Performance Optimization – Optimize pipeline performance
• Cost Management – Manage and optimize costs

Operations

• Monitoring and Alerting – Set up effective monitoring
• Error Handling – Implement robust error handling
• Security – Secure your pipelines and data

Need Help?

• Search: Use ⌘/Ctrl + K to search across all documentation
• Navigation: Use the sidebar to browse topics
• Examples: Check the examples section for practical implementations
• Support: Contact our support team for assistance

---

Ready to build your first pipeline? Start with the Quickstart Guide!