# Clone the repository
git clone https://github.com/allen13/flake-runner.git
cd flake-runner

# Build the binary
go build -o flake-runner ./cmd

# Or install directly
go install github.com/allen13/flake-runner/cmd@latest

# Validate configuration and AWS resources
$ flake-runner init -config production.json

✅ Flake Runner initialized successfully!
📊 Job ID: job-a1b2c3d4
🗂️  Configured with 3 prefix mappings:
   • customers/ → CUSTOMERS
   • orders/ → ORDERS
   • products/ → PRODUCTS
🚀 Ready to process files!

# Process a single file
$ flake-runner process -file s3://my-bucket/customers/customer_data.csv

Processing file: s3://my-bucket/customers/customer_data.csv
✅ File processing initiated successfully!
📊 Job ID: job-a1b2c3d4

# Process with control data validation
$ flake-runner process -file s3://my-bucket/customers/data.csv \
  -control-data '{"file_name":"data.csv","file_size":1024,"file_hash":"abc123","record_count":100}'

Processing file: s3://my-bucket/customers/data.csv
Using provided control data for validation
✅ File processing initiated successfully!
📊 Job ID: job-b2c3d4e5

# Process and wait for completion
$ flake-runner process -file s3://my-bucket/orders/orders_2024.csv -wait

Processing file: s3://my-bucket/orders/orders_2024.csv
✅ File processing initiated successfully!
📊 Job ID: job-c3d4e5f6
⏳ Waiting for job completion (timeout: 120 minutes, poll interval: 30 seconds)...
📈 Starting job polling at 14:23:45
[0s] State: PROCESSING - EMR Serverless Processing
   📊 EMR Job: RUNNING
[30s] State: PROCESSING - EMR Serverless Processing
   📊 EMR Job: SUCCESS
   ✅ EMR job completed successfully, updating orchestration state...
   📝 Orchestration state updated to PROCESSED
[31s] State: PROCESSED - Processing Completed Successfully
🎉 Job processing completed successfully in 31s!

📋 Job Summary:
   File: orders_2024.csv
   Target Table: ORDERS
   Batch ID: batch_20240107_001
   Started: 2024-01-07 14:23:45
   Completed: 2024-01-07 14:24:16
   Duration: 31s

🔍 Validation Results:
   Expected Records: 50000
   Actual Records: 50000
   Record Count Match: true
   File Size Match: true
   Checksum Match: true

⚡ EMR Job Details:
   Job Run ID: 00f1qa2lmnmnhj1n
   Status: SUCCESS
   Processed Records: 50000

# Get status of a specific file
$ flake-runner status -file s3://my-bucket/customers/data.csv

File: s3://my-bucket/customers/data.csv
State: COMPLETED
Target: CUSTOMERS
Started: 2024-01-07 10:15:23
Completed: 2024-01-07 10:18:45
Duration: 3m22s

# Query files by state
$ flake-runner status -state FAILED

Found 2 files in FAILED state:
1. s3://my-bucket/orders/bad_file.csv
   Failed at: 2024-01-07 09:45:12
   Error: Record count validation failed: expected 1000, got 950
   
2. s3://my-bucket/products/corrupt.parquet
   Failed at: 2024-01-07 11:23:45
   Error: Parquet file corrupted: invalid magic bytes

# View job logs
$ flake-runner emr -action logs -file s3://my-bucket/customers/data.csv

Fetching EMR job logs for: job-abc12345
Job Run ID: 00f1qa2lmnmnhj1n

[DRIVER] 2024-01-07 10:15:45 INFO  Processing started for s3://my-bucket/customers/data.csv
[DRIVER] 2024-01-07 10:15:46 INFO  Found 50000 records in CSV file
[DRIVER] 2024-01-07 10:15:47 INFO  Applying transformations...
[DRIVER] 2024-01-07 10:16:23 INFO  Writing output to s3://my-output/processed/CUSTOMERS/
[DRIVER] 2024-01-07 10:16:45 INFO  Processing completed successfully

# Cancel a running job
$ flake-runner emr -action cancel -file s3://my-bucket/customers/large_file.csv

✅ Job cancellation requested successfully

# List all AWS resources
$ flake-runner aws --action list

🔍 Listing AWS Resources...
Region: us-east-1

📦 S3 Buckets:
   ✅ Input Bucket: my-data-input (accessible)
   ✅ Output Bucket: my-data-output (accessible)
   ✅ Staging Bucket: my-data-staging (accessible)

🗄️  DynamoDB Tables:
   ✅ Control Table: file-orchestrations-prod (accessible)

⚡ EMR Serverless Applications:
   ✅ Application: 00abc123def456gh (accessible)

✅ Resource listing complete

# Create missing AWS resources
$ flake-runner aws --action create --create --force

🏗️  Creating AWS Resources...

📦 S3 Buckets:
   ✅ Input Bucket: my-data-input (already exists)
   ✅ Created Output Bucket: my-data-output
   ✅ Created Staging Bucket: my-data-staging

🗄️  DynamoDB Tables:
   ✅ Created Control Table: file-orchestrations-prod

⚡ EMR Serverless Applications:
   🏗️  Creating EMR Application with IAM Role: flake-runner-production...
   ✅ Created EMR Application: flake-runner-production
   📋 Application ID: 00xyz789ghi012jk
   🔐 IAM Role ARN: arn:aws:iam::123456789012:role/EMRServerlessExecutionRole-flake-runner-production
   💡 Update your configuration file with these values:
      "emr_application_id": "00xyz789ghi012jk",
      "emr_execution_role_arn": "arn:aws:iam::123456789012:role/EMRServerlessExecutionRole-flake-runner-production"

✅ Resource creation complete

# Upload files to S3
$ flake-runner aws --action upload --local customer_data.csv --file customers/upload_20240107.csv

📤 Uploading file: customer_data.csv → customers/upload_20240107.csv
✅ Successfully uploaded to: s3://my-data-input/customers/upload_20240107.csv
📏 File size: 2483901 bytes
🎯 Target table: CUSTOMERS
💡 Process with: flake-runner process --file s3://my-data-input/customers/upload_20240107.csv

# Run the CSV to Parquet conversion demo
./demo_csv_to_parquet_full.sh

FlakeRunner Full End-to-End CSV to Parquet Demo
================================================
This demo showcases the complete FlakeRunner framework workflow:
• AWS resource creation (S3, DynamoDB, EMR Serverless)
• CSV file upload and validation
• Actual EMR job submission and processing
• Job monitoring and log retrieval
• Processed file download and verification

# Run tests
go test ./...

# Run linting
./scripts/go-lint.sh

# Build for different platforms
GOOS=linux GOARCH=amd64 go build -o flake-runner-linux ./cmd
GOOS=darwin GOARCH=amd64 go build -o flake-runner-darwin ./cmd
GOOS=windows GOARCH=amd64 go build -o flake-runner.exe ./cmd