Skip to content
Datablast Docs

Platform Overview

Datablast is a cloud-based data platform that enables you to automate your daily data operations through SQL and Python assets. The platform provides a unified interface for data processing, analytics, machine learning, and monitoring across multiple cloud providers and data sources.

What is Datablast?

Section titled “What is Datablast?”

Datablast simplifies data pipeline development and management by providing:

  • Unified Interface: Single platform for all your data operations
  • Multi-Cloud Support: Work with BigQuery, Snowflake, Athena, and PostgreSQL
  • Automated Execution: Reliable scheduling and dependency management
  • Built-in Quality: Data validation and testing framework
  • Cost Optimization: Intelligent resource management and cost tracking

Key Features

Section titled “Key Features”

Multi-Cloud Support

Section titled “Multi-Cloud Support”
  • BigQuery: Google Cloud’s data warehouse with automatic materialization
  • Snowflake: Cloud data platform with advanced analytics capabilities
  • Athena: AWS serverless query service for S3 data
  • PostgreSQL: Open-source relational database support

Automated Data Pipelines

Section titled “Automated Data Pipelines”
  • YAML Configuration: Simple, declarative pipeline definitions
  • Dependency Management: Automatic task ordering and execution
  • Sensor Integration: Wait for external data availability
  • Error Handling: Retry logic and failure management

Data Quality Framework

Section titled “Data Quality Framework”
  • Column Tests: Validate data types, nulls, uniqueness, and ranges
  • Custom Tests: Complex business logic validation
  • Blocking vs Non-blocking: Control pipeline behavior on test failures
  • Integrated Reporting: View test results in the platform UI

Machine Learning Integration

Section titled “Machine Learning Integration”
  • Python Tasks: Execute ML models and data science workflows
  • Instance Types: Choose appropriate compute resources
  • Dependency Management: Integrate ML workflows with data pipelines
  • Model Deployment: Deploy and manage ML models

Monitoring and Alerting

Section titled “Monitoring and Alerting”
  • Real-time Tracking: Monitor pipeline and task execution
  • Performance Metrics: Track execution times and resource usage
  • Cost Monitoring: Monitor and optimize spending
  • Notification Integration: Slack and Discord alerts

Materialization Strategies

Section titled “Materialization Strategies”
  • Table Creation: Physical storage with partitioning and clustering
  • View Creation: Virtual tables for real-time access
  • Incremental Updates: Efficient data processing patterns
  • Cost Optimization: Intelligent storage and query optimization

What You Can Build

Section titled “What You Can Build”

Data Warehouses

Section titled “Data Warehouses”

Transform and organize your data into structured, queryable formats:

  • ETL Pipelines: Extract, transform, and load data workflows
  • Data Modeling: Create dimensional models and fact tables
  • Data Quality: Ensure data integrity and consistency
  • Performance Optimization: Optimize queries and storage

Analytics Dashboards

Section titled “Analytics Dashboards”

Create business intelligence solutions:

  • KPI Dashboards: Track key business metrics
  • Operational Reports: Monitor business operations
  • Trend Analysis: Identify patterns and trends
  • Real-time Analytics: Get insights as data changes

Machine Learning Models

Section titled “Machine Learning Models”

Train and deploy ML models:

  • Feature Engineering: Prepare data for ML models
  • Model Training: Train models on your data
  • Prediction Pipelines: Generate predictions automatically
  • Model Monitoring: Track model performance and drift

Data Quality Pipelines

Section titled “Data Quality Pipelines”

Ensure data integrity and consistency:

  • Validation Rules: Implement business logic validation
  • Data Profiling: Understand data characteristics
  • Anomaly Detection: Identify unusual patterns
  • Compliance Monitoring: Ensure regulatory compliance

Batch Processing

Section titled “Batch Processing”

Batch data processing:

  • Batch Processing: Process large datasets efficiently
  • Data Integration: Combine data from multiple sources
  • API Integration: Connect with external services

Platform Architecture

Section titled “Platform Architecture”

Core Components

Section titled “Core Components”

Pipeline Engine

Section titled “Pipeline Engine”
  • Scheduler: Manages pipeline execution and timing
  • Executor: Runs tasks and manages dependencies
  • Monitor: Tracks execution and performance
  • Notifier: Sends alerts and notifications

Data Processing

Section titled “Data Processing”
  • SQL Engine: Executes SQL queries across multiple databases
  • Python Runtime: Executes Python code with various instance types
  • Sensor Framework: Monitors external conditions
  • Quality Framework: Validates data and business rules

Storage and Compute

Section titled “Storage and Compute”
  • Cloud Integration: Connects to various cloud providers
  • Resource Management: Optimizes compute and storage usage
  • Cost Tracking: Monitors and optimizes spending
  • Security: Ensures data security and compliance

Data Flow

Section titled “Data Flow”
  1. Configuration: Define pipelines using YAML and annotations
  2. Scheduling: Platform schedules pipeline execution
  3. Execution: Tasks run in dependency order
  4. Processing: Data is transformed and validated
  5. Storage: Results are materialized to tables or views
  6. Monitoring: Platform tracks execution and performance
  7. Alerting: Notifications are sent for failures or issues

Getting Started

Section titled “Getting Started”

Prerequisites

Section titled “Prerequisites”
  • Git repository access for your project
  • Cloud provider credentials (GCP, AWS, Snowflake)
  • Basic knowledge of SQL and Python
  • Understanding of data pipeline concepts

Quick Start

Section titled “Quick Start”
  1. Create Repository: Set up a Git repository for your project
  2. Configure Pipeline: Define your first pipeline in pipeline.yml
  3. Create Tasks: Add SQL or Python tasks to your pipeline
  4. Deploy: Push your code to trigger pipeline execution
  5. Monitor: Track execution and results in the platform UI

Next Steps

Section titled “Next Steps”

Benefits

Section titled “Benefits”

For Data Engineers

Section titled “For Data Engineers”
  • Simplified Development: Focus on business logic, not infrastructure
  • Reliable Execution: Built-in error handling and retry logic
  • Cost Optimization: Intelligent resource management
  • Quality Assurance: Built-in data validation framework

For Data Scientists

Section titled “For Data Scientists”
  • ML Integration: Seamless integration with data pipelines
  • Resource Flexibility: Choose appropriate compute resources
  • Model Deployment: Deploy models as part of data workflows
  • Experiment Tracking: Monitor model performance and drift

For Business Users

Section titled “For Business Users”
  • Reliable Data: Consistent, high-quality data delivery
  • Real-time Insights: Get data as soon as it’s available
  • Cost Transparency: Understand and control data costs
  • Operational Excellence: Reliable, monitored data operations

Support and Resources

Section titled “Support and Resources”

Documentation

Section titled “Documentation”
  • Guides: Step-by-step tutorials and best practices
  • Reference: Complete API and configuration reference
  • Examples: Real-world examples and use cases
  • Troubleshooting: Common issues and solutions

Community

Section titled “Community”
  • Support Team: Expert assistance and guidance
  • Best Practices: Learn from community experiences
  • Updates: Stay informed about new features and improvements
  • Feedback: Contribute to platform development
Section titled “Related Documentation”