Skip to content
Datablast Docs

Overview

This section contains comprehensive guides for developing SQL tasks with AWS Athena in the Datablast Data Platform.

Core Development

Section titled “Core Development”

Specialized Guides

Section titled “Specialized Guides”

Key Features

Section titled “Key Features”

AWS Integration

Section titled “AWS Integration”
  • S3 Data Access: Direct querying of data stored in S3
  • Cost Optimization: Pay-per-query pricing model
  • Schema Evolution: Flexible schema management
  • Performance Tuning: Query optimization for Athena
  • AWS Service Integration: Seamless integration with other AWS services

Development Tools

Section titled “Development Tools”
  • Annotation-based Configuration: Simple task configuration
  • YAML Configuration: Complex task setup
  • Athena-specific Functions: Date functions, string functions, and SQL features
  • Debugging Support: Comprehensive error handling and logging

Athena-Specific Features

Section titled “Athena-Specific Features”

S3 Integration

Section titled “S3 Integration”
  • Direct querying of S3 data
  • Support for multiple data formats (Parquet, JSON, ORC, CSV)
  • Automatic schema inference
  • Partition discovery and management

Data Formats

Section titled “Data Formats”
  • Parquet: Optimal for analytical workloads
  • ORC: Good alternative to Parquet
  • JSON: Use for semi-structured data
  • CSV: Avoid for large datasets

Partitioning

Section titled “Partitioning”
  • S3 folder structure partitioning
  • Automatic partition pruning
  • Dynamic partition creation
  • Partition projection

Performance Optimization

Section titled “Performance Optimization”

Query Optimization

Section titled “Query Optimization”
  • Use appropriate data formats (Parquet preferred)
  • Implement proper partitioning strategies
  • Optimize data scanning patterns
  • Leverage partition pruning

Cost Management

Section titled “Cost Management”
  • Minimize data scanning
  • Use appropriate file formats
  • Implement data lifecycle policies
  • Monitor query costs

Best Practices

Section titled “Best Practices”
  • Organize S3 data efficiently
  • Use meaningful partition columns
  • Implement proper error handling
  • Monitor performance metrics

Quick Start

Section titled “Quick Start”
  1. Configure AWS Connection: Set up AWS connection in Datablast
  2. Organize S3 Data: Structure your S3 data for optimal performance
  3. Create Task: Define your SQL task with proper configuration
  4. Test Query: Validate your SQL in Athena console
  5. Deploy Pipeline: Add task to your pipeline and schedule
Section titled “Related Documentation”