Sling v1.4.17 (2025-08-19T21:47:49Z)

Database Connections & Support

New Database Support

• Added Exasol connector: New database type with comprehensive bulk import functionality, CSV handling, and timestamp formatting

Connection Improvements

• BigQuery: Improved error handling for dataset access failures (warnings instead of errors)
• Athena: Enhanced workgroup error messaging with specific workgroup names
• Iceberg: Added S3 property pass-through for REST access and filesystem operations
• DuckDB: Enhanced AWS credential chain support with automatic provider detection

Transform System Overhaul

• Major refactor: Complete rewrite of the transform system architecture
  • Replaced legacy Transform with new TransformLegacy for backward compatibility
  • Introduced staged transforms with ParseStageTransforms function
  • Enhanced expression evaluation with record references
• New transform features:
  • Support for multi-stage transformations
  • Cross-column references using record.field_name syntax
  • Conditional logic and calculated fields
  • Enhanced function support (hash, cast, coalesce, etc.)

Data Type Enhancements

Time Data Types

• PostgreSQL TIME: Proper preservation of TIME, TIME WITH TIME ZONE, and TIME(precision) types
• MSSQL UNIQUEIDENTIFIER: Full support for UUID/GUID data type preservation
• Enhanced time handling: Better formatting and timezone support across all databases

Column Type Improvements

• Position handling: Fixed column position assignment when merging new columns
• String type detection: Refined logic for TIME vs STRING type classification
• Arrow format: Enhanced support for Arrow file format detection and streaming

File Processing & Encoding

Encoding Support

• Target encoding: Full implementation of encoding options for file outputs
• Multiple encodings: Support for Latin1, UTF8, Windows1252 encoding in target files
• Source/target distinction: Separate decode (source) and encode (target) handling

File Format Enhancements

• Arrow format: Improved detection and processing of Arrow files vs streams
• CSV improvements: Better handling of special characters and null values
• Parquet optimization: Enhanced reading performance and DuckDB integration

Performance & Optimization

• Debug logging: Converted g.DebugLow to g.Debug across codebase for better performance
• Memory management: Improved handling of large datasets and file processing
• Concurrent processing: Enhanced parallel processing capabilities

API & Pagination

• Fixed API pagination: Corrected pagination logic in API specifications
  • Fixed length(records) to length(response.records)
  • Improved header parsing with proper bracket notation

Testing & Quality Assurance

• New test suites: Added comprehensive tests for:
  • Staged transforms functionality
  • Encoding target options
  • Time data type preservation
  • UNIQUEIDENTIFIER handling
  • Large file ingestion (100K rows)
• Enhanced test coverage: Improved validation and error checking across test cases

Bug Fixes

• CSV string casting: Fixed method calls from CastToString to CastToStringCSV for proper CSV formatting
• Compression handling: Added compression parameter support for Databricks and Snowflake connections
• Error handling: Improved error messages and warnings across various connectors
• API column handling: Fixed column detection and handling in API responses

Sling v1.4.16 (2025-08-03T00:57:21Z)

New Database Support

• Exasol Database Integration: Added comprehensive support for Exasol database
• Azure Table Storage: Added read support for Azure Table storage

Bug Fixes & Improvements

Replication & Data Handling

• Fixed hook stage validation before execution in replication
• Improved null value handling in query results for integer and decimal columns
• Enhanced StreamLoad to use LoaderStreamConfig for writing operations
• Removed redundant _SLING_LOADED_AT logging during column operations

Data Types & Format Support

• Added support for ClickHouse simpleaggregatefunction and fixedstring types
• Refactored NativeTypeToGeneral to use switch statement for better performance
• Improved column type preservation for SLING_LOADED_AT

Export & File Handling

• Fixed empty file handling for Parquet format in BulkExportFlow
• Added tests for empty dataset exporting with Parquet
• Enhanced exclusion of extra columns in Parquet exports
• Removed ORDER BY clause from incremental_select queries for better performance

Sling v1.4.15 (2025-07-26T00:25:00Z)

• Changed FileSelect from pointer to slice for consistency across codebase to fix critical bug.
• Refactored hook parsing and execution system for improved clarity
• Added context management to replication tasks and hooks
• Fixed status check conditions to reference correct run status

Sling v1.4.14 (2025-07-23T21:43:15Z)

Major Feature Categories

Data Chunking & Replication

Enhanced Chunking Functionality:

• New expression support for chunking configurations
• SQL stream chunking with conditional checks
• Improved chunking logic with new options and configurations
• File handling support in stream task configuration

Replication Improvements:

• Enhanced task preparation logic in replication configuration
• Thread-safe operations with proper locking mechanisms
• Clean state management for chunk loading operations

Database & Storage Enhancements

Database Improvements:

• MongoDB column casing handling fixes
• Redshift S3 unload operations with proper file format extensions
• MySQL to Snowflake decimal to string cast support
• DuckDB ARM64 architecture support

File System Updates:

• Google Drive integration improvements:
  • Enhanced folder info and support for all drives
  • Updated queries to properly handle trashed files
  • Improved client functionality
• File deletion before writing dataflow to ensure clean state

Testing & Quality Assurance

New Test Suites:

• API response extraction tests using YAML configuration
• HTTP call and response extraction comprehensive tests
• Setup and teardown lifecycle tests with mock responses
• Parquet file deletion pipeline tests
• CLI execution timeout handling

Developer Experience

Logging & Environment:

• Color output control with SLING_LOGGING=NO_COLOR support
• Thread child detection with env.IsThreadChild
• Conditional message colorization based on NoColor settings

Build & Development:

• Executable build script permissions updates
• Test timeout handling for CLI execution
• Enhanced error handling across multiple components

Sling v1.4.13 (2025-07-09T15:41:01Z)

Key Features & Enhancements

Database & Connection Improvements

• Iceberg Support: Enhanced Iceberg connection with AWS config and improved database handling
• Glue Integration: Added support for iceberg_glue target and enhanced Glue connection capabilities
• DuckDB Updates: Updated DuckDB version and added adjustCol flag for DuckDB and related types
• Redshift Enhancement: Added Redshift parquet export functionality

API & Spec Management

• API Spec Refactoring: Refactored API specs to use curly braces for variable interpolation
• MCP Tools: Enhanced connection and API spec tools with detailed action support
• Pagination Fixes: Updated stop_condition in pagination to use strict less than comparison

File System & Storage

• S3 Enhancements:
  • Enhanced S3 client with custom HTTP settings and retry logic
  • Streamlined AWS credential handling by removing unnecessary components
• File Management:
  • Fixed local temporary file deletion after writing to prevent duplication
  • Fixed single file copy operation to prevent filename duplication
  • Added support for multiple Azure Storage files replication

Testing & Quality

• Test Suite Expansion:
  • Added new connection types for DuckLake variants
  • Enhanced test cases with output verification
  • Added encoding tests for latin1, windows1252, and utf8_bom
  • Added transform functions test suite
  • Added inspect hook tests for database tables and files
  • Added MSSQL to parquet overwrite test
  • Added timestamp string casting tests

Performance & Reliability

• Streaming Improvements:
  • Replaced error logging with LogError for better error handling
  • Improved reader robustness and concurrency
• Workflow Optimization:
  • Increased timeout for build and release jobs (25 minutes)
  • Enhanced streaming capabilities

Dependencies & Maintenance

• Dependency Updates:
  • Updated flarco/g dependency to v0.1.154
  • Updated mark3labs/mcp-go to v0.33.0
• Snowflake Fix: Fixed temporary file handling with SLING_KEEP_TEMP environment variable

Sling v1.4.12 (2025-06-29T12:16:17Z)

• Critical fix for a bug when creating DuckLake secrets

Sling v1.4.11 (2025-06-27T16:45:21Z)

🚀 Major Features & Enhancements

📊 Databricks Support

• NEW: Complete Databricks database integration
• Added connection, query, and data import/export functionality
• Implemented Databricks-specific SQL templates
• Added support for Databricks COPY INTO commands
• Enhanced AWS S3 integration for Databricks workflows
• Added comprehensive JSON manipulation functions support
• Implemented Databricks volume operations with multiple file format support

🧊 Iceberg & Data Lake Improvements

• Enhanced DuckDB connection handling with Iceberg catalog support
• Improved REST catalog interface for table operations
• Added CatalogSchema support to DuckLake connections
• Enhanced error handling for manifest retrieval
• Updated catalog type references and authentication

🦆 DuckDB Enhancements

• Bumped DuckDB version from 1.3.0 to 1.3.1
• Added R2 file type support in dbio
• Improved URI handling and secret management
• Enhanced schema handling in buildAttachSQL

🔧 Bug Fixes & Improvements

💾 Data Type & Precision Handling

• FIXED: Oracle NUMBER scale preservation issues
• Enhanced decimal column precision and scale handling
• Improved SQLColumns and Apply methods for data types
• Removed unnecessary DbScale checks

🔐 Authentication & Credentials

• Standardized AWS credential handling across Athena, Iceberg, and S3
• Improved error reporting for authentication issues
• Enhanced secret management for various connectors

📁 File System & Formats

• Improved CSV delimiter handling (changed from rune to string)
• Enhanced file format assignment logic
• Streamlined S3 GetReader functionality
• Added String method to FileType for better representation

🔍 SQL & Query Improvements

• Fully qualified table names in DELETE statements for upsert operations
• Enhanced error messaging in GetTableColumns
• Improved SQL template compatibility

🧪 Testing & Quality

✅ Test Enhancements

• Updated CLI suite configuration from TSV to YAML format
• Added comprehensive test cases for:
  • Starrocks with empty_as_null option
  • Pipeline copy operations
  • Delta R2 secret creation
  • Oracle NUMBER scale fixes
  • MySQL BIT type replication
• Improved test logging and error handling
• Enhanced environment variable initialization

Sling v1.4.10 (2025-06-12T10:42:03Z)

• Upgrade to AWS SDK for Go v2: The AWS SDK has been upgraded to v2, which brings in a more modern, modular architecture. This required substantial code updates, including adding context to all AWS API calls for better cancellation and timeout control, and improving error handling.

• New Connector and Format Support:

  • Apache Iceberg: Initial support for the Apache Iceberg table format has been added, using the iceberg-go library. This allows Sling to interact with Iceberg tables for data replication tasks.
  • DuckLake: Support for DuckLake has been introduced as a new database type, extending the capabilities of DuckDB.
  • Google Drive: The tool now supports Google Drive as a file system source.
  • Apache Arrow: Support for the Apache Arrow file format has been implemented, including reader/writer channels and integration into the HTTP import functionality.

• Enhancements to Existing Connectors:

  • Amazon Athena: Support has been significantly improved with better handling of data and staging locations, standardized SQL keywords, and a PURGE option when dropping tables.
  • PostgreSQL: It's now possible to specify a role in the PostgreSQL connection URL for more granular access control.
  • MySQL/MariaDB: Support for BIT column transformations has been added.
  • DuckDB: The embedded DuckDB version has been updated from 1.1.3 to 1.3.0.

• General Improvements and Fixes:

  • Timeout handling via SLING_TIMEOUT has been added to replication and pipeline runs.
  • A new TableExists method has been added to the connection interface to check for table existence before performing operations.
  • Several bug fixes were made, including improvements to error messages, handling of dbt parameters for Athena, and fixing unnecessary semicolons in generated DuckDB SQL.

Sling v1.4.9 (2025-05-25T01:01:37Z)

Bug Fixes & Improvements

• CSV Schema Handling: Fixed potential out-of-bounds errors and addressed schema inconsistencies
• Data Type Corrections:
  • Changed 'real' data type from decimal to float in multiple databases
  • Changed 'real' data type from float to decimal in SQLite and DuckDB
• Node Management: Corrected node addition in List function
• Metadata Processing: Skip metadata type coercion to improve performance
• Logging:
  • Added version information to logs
  • Prevented duplicate log messages in child threads

New Features

• Task Loading: Added support for loading compiled tasks from file
• Connection Properties: Added connect_time property to connection methods
• Scoop Integration: Improved scoop upgrade handling

Dependency Updates

• Bumped github.com/flarco/g from v0.1.143 → v0.1.146
• Updated github.com/spf13/cast from v1.6.0 → v1.7.1

Sling v1.4.8 (2025-05-13T18:38:17Z)

• Fix Redshift Unload
• Fix critical bug with sling agent