Sling v1.4.7 (2025-05-13T10:06:04Z)

• Error Handling & Robustness:
  • Improved error handling for incremental mode (Commit c9dadd0).
  • Handled missing primary keys in API sources (Commit cd31e26).
  • Enhanced error handling in GetSchemata (Commit 4d5c9b5).
  • Handled empty result sets in GetCount (Commit 4666114).
  • Improved handling of field selection in database reads (Commit 1d1d1d9).
• Configuration & Schema:
  • Added a JSON schema for Sling pipeline configuration (Commit 4244c34).
  • Enhanced the JSON schema for pipeline steps by adding an id property to each (Commit 87024b5).
  • Updated pipeline.json schema (Commit 87274d9, 4bcfb66).
• Database & Data Handling:
  • Added support for Clickhouse JSON data type (Commit 2b986d0), which was later reverted (Commit db7b294).
  • Added basic support for AWS Athena (Commit 7f70b19).
  • Improved CSV reader to handle string values (Commit 82fa1a5) and column number mismatches (Commit b201e68).
  • Fixed an issue where Sling was not respecting the private_key property (Commits baff658, 2f4b7a3 - merged from PR #553 by GeoffTearle).
  • Ensured database connections are closed after wildcard processing (Commit 0b004c3).
  • Introduced a TrimSQLComments function (Commit bf0c02b).
  • Added tests for parsing multi-statements and trimming SQL comments (Commit d40d044).
  • Increased nvarchar length factor for temporary tables on SQL Server to provide a buffer, addressing issue #554 (Commit 7f8f3ed).
  • Applied quotes to select fields in incremental select statements (Commit 6d43d8a).
• Pipeline & Execution:
  • Enhanced pipeline execution and logging (Commit 731ff55).
• Development & CI/CD:
  • Updated build workflow runner matrix (Commit 75e8000).
  • Handled missing bcp (Bulk Copy Program) executable in tests (Commit 4369307).

Sling v1.4.6 (2025-04-27T01:17:57Z)

API & Database Improvements

• Improved API specs and documentation
• Added stream read mode to API
• Enhanced API request handling with bloom filter and concurrency
• Added incremental_select query for SQL Server
• Set incremental state check for API

Performance & Functionality

• Improved DuckDB performance
• Enhanced range function to support different data types
• Added database connection check
• Added unit tests for Excel date conversion

Bug Fixes

• Improved database connection handling
• Fixed start/end hooks when a thread child
• Handled nil args in ExecContext and empty target type
• Improved Snowflake COPY INTO Table command
• Ensured file closure in SFTP write operations

Dependency Updates

• Bumped github.com/flarco/g from 0.1.138 to 0.1.140
• Bumped timeplus-io/proton-go-driver/v2
• Bumped sftp from 1.13.7 to 1.13.9
• Updated bloom filter library
• Multiple other dependency updates

Other Changes

• Changed log level from debug to trace in BulkImportStream
• Increased max_column_length to 1024 in StarRocks
• Added comment to stripe replication test
• Improved CI workflow for version bumping

Files Modified

• Database connectors (Snowflake, ClickHouse, D1)
• API specifications and documentation
• File system handlers (SFTP)
• Data processing components
• CI/CD workflows

Sling v1.4.5 (2025-04-17T23:50:11Z)

New Command Line Options

• Added --trace flag to set logging level to TRACE (not for production use)
• Added --endpoints flag for testing specific API connection endpoints

Code Enhancements

• Added trace-level debugging support
• Implemented timeout functionality via SLING_TIMEOUT environment variable
• Improved handling of API streams with no data
• Added proper connection closing with CloseAll()
• Refactored dataflow preparation functions

API Integration & Enhancements

• API Connection Support: Introduced support for API connections (Commit a5b4635).
• Endpoint Discovery and Testing: Implemented capabilities for endpoint testing and discovery (Commits b8d610c, 3b10488).
• Authentication & Dataflow: Improved API authentication mechanisms and dataflow handling (Commit 904fa6a).
• Source Configuration: Improved API source configuration and data handling processes (Commit c29b280).

Bug Fixes

• Handled empty NullIf configuration (Commit 27a8f9a).
• Addressed nil df (DataFrame) in task execution (Commit ae645d6).
• Fixed handling of nil values in the state map for dbio (Commit 9f2e78b).
• Simplified debug flag handling (Commit 6e7d4c4).
• Improved datastream logging and configuration (Commit dc76593).
• Addressed column name mismatch issues (Commit bc05b3a).
• Corrected the UUID transform function (Commit 4e21dcf).
• Fixed handling of precision and scale for string and decimal types (Commit aa62c31).
• Improved decimal type handling in GetNativeType (Commit 1fa88fe).
• Handled empty incremental target columns (Commit 2eeaada).
• Improved SFTP error message clarity (Commit 34f1486).

Sling v1.4.4 (2025-03-15T22:33:13Z)

Database Handling Improvements

Azure Database Support

• Added support for Azure SQL Database and Azure Data Warehouse in multiple code paths
• Extended database type checks to include TypeDbAzure and TypeDbAzureDWH alongside SQL Server

Column Length Handling

• Added proper null checks before setting column length values
• Fixed handling of column length metadata in several functions
• Added Length field to table column retrieval operations

BigQuery Connection Enhancements

• Refactored authentication system to support multiple options
• Added capability to specify additional scopes for BigQuery connections:
  • Now supports adding custom scopes beyond the default https://www.googleapis.com/auth/bigquery
  • Can include additional scopes like Drive and Spreadsheets access

Processing & Error Handling Improvements

• Added explicit error for missing update_key with message: "did not find update_key: %s"
• Improved chunk processing with better diagnostics and error handling
• Added support for custom SQL in chunk processing
• Enhanced debugging information for chunk operations

File Path & Configuration Fixes

• Fixed stream file name handling in GetFormatMap() method
  • Changed from using underscore (_) to dot (.) as separator

Dependencies Updated

• Updated github.com/flarco/g from v0.1.136 to v0.1.138
• Updated github.com/sijms/go-ora/v2 from v2.8.22 to v2.8.24

Sling v1.4.3 (2025-02-14T09:57:45Z)

SQL Server BCP Import Improvements

• Added handling for empty values with new emptyRep placeholder
• Improved string replacement logic for quotes, newlines, and empty values

DuckDB File Writing Enhancements

• Added better support for file splitting by rows
• Improved handling of file paths and directories
• Fixed file size handling when splitting by rows

JSON Processing

• Added better handling of JSON values in string fields
• Added special handling for "null" string values in JSON

Data Type Casting

• Improved integer casting with boolean value support
• Added handling for "true"/"false" string values to convert to 1/0

Configuration and Pipeline Updates

• Changed blank key warning to trace level when in debug mode
• Added environment variable handling in pipeline configuration

Sling v1.4.2 (2025-02-03T21:40:44Z)

• Added support for AWS_SESSION_TOKEN in Redshift connections
• Removed dbio.TypeFileS3 from a switch case in WriteDataflowViaDuckDB

Sling v1.4.1 (2025-02-01T21:46:18Z)

Major Changes

1. Pipeline Implementation

   • Added new Pipeline type for executing sequential steps
   • Introduced pipeline configuration loading from YAML/JSON files
   • Added support for pipeline steps with various types (log, replication, command, etc.)

2. State Management Refactoring

   • Renamed Hooks to State in runtime state management
   • Created RuntimeState interface
   • Added separate ReplicationState and PipelineState implementations

3. Chunking Functionality

   • Added support for chunking data processing based on column ranges
   • Implemented ProcessChunks() method for handling data partitioning
   • Added chunk size configuration options

4. Case Handling Improvements

   • Added support for lower/upper case schema and table names
   • Improved temp table name handling for different database types
   • Added new format variables: stream_schema_lower/upper, stream_table_lower/upper

5. File System Updates

   • Enhanced file pattern matching across different file system implementations
   • Improved file deletion handling for SFTP
   • Better handling of file paths in copy operations

Minor Changes

• Added binary data handling for DuckDB
• Improved UTF-8 validation in string casting
• Added environment variable controls for DuckDB compute
• Various bug fixes and code cleanup

Sling v1.4.0 (2025-01-22T21:31:08Z)

See https://github.com/slingdata-io/sling-cli/ for more details.

Sling v1.3.5 (2025-01-15T10:49:20Z)

See https://github.com/slingdata-io/sling-cli/ for more details.

Sling v1.3.4 (2024-12-27T16:16:34Z)

See https://github.com/slingdata-io/sling-cli/ for more details.