Data Automation Blueprint

# Role You are a Data Engineer specializing in automation architecture, ETL pipelines, and resilient data workflows. # Task Design a complete data automation blueprint for [DATA_WORKFLOW_DESCRIPTION]. **Input Data:** [DESCRIBE_SOURCE_DATA] **Desired Output:** [DESCRIBE_FINAL_OUTPUT] **Frequency:** [ONE_TIME / SCHEDULED / REAL_TIME] **Tools Available:** [LIST_TOOLS_OR_SAY_RECOMMEND] # Instructions ## Phase 1: Requirements Analysis 1. **Data Profiling**: - Expected schema and data types - Volume estimates (rows, size, frequency) - Quality issues to anticipate - Source system constraints 2. **Transformation Mapping**: - Input to output field mapping - Derived fields and calculations - Business rules and logic - Validation requirements ## Phase 2: Architecture Design 1. **Pipeline Stages**: - Extract: How and when to pull data - Transform: Processing steps in order - Load: Destination and format - Orchestration: Scheduling and dependencies 2. **Error Handling**: - Validation checkpoints - Failure modes and responses - Retry logic and backoff - Alerting and escalation 3. **Monitoring**: - Success/failure metrics - Data quality checks - Performance benchmarks - Audit logging ## Phase 3: Implementation Specification 1. Pseudo-code or script outline 2. Configuration requirements 3. Testing strategy 4. Deployment steps