Sling v1.2.17 (2024-08-28T16:39:52Z)

Changelog

• Fix Setting & GetMachineID
• Add home/sling in Dockerfiles
• Update caching and logging

Sling v1.2.16 (2024-08-27T21:25:58Z)

Changelog Summary

Docker and CI/CD

• Updated Dockerfiles and CI workflows
• Improved build processes for dev and production
• Added self-hosted Mac to build-dev workflow

Database Connections and Operations

• Enhanced connection handling for various databases (DuckDB, Snowflake, Oracle)
• Improved timestamp and timezone handling
• Added support for Delta and Iceberg formats
• Enhanced file system operations and caching

Testing and Error Handling

• Added and updated various tests
• Improved error handling and logging
• Added panic recovery in Parquet operations

Features and Improvements

• Added column constraints and improved column ordering
• Enhanced replication and incremental mode functionality
• Improved wildcard matching and folder glob operations
• Added support for Azure service principal keys
• Implemented connection objects and entries

Code Refactoring and Cleanup

• Moved and reorganized various functions and types
• Cleaned up and updated dependencies (go.mod)
• Improved code structure and naming conventions

Security and Configuration

• Updated SSH/SFTP key exchanges
• Improved secrets handling in CI/CD
• Updated gitignore and other configuration files

Sling v1.2.15

Major Enhancements

• Performance Improvements

  • Enhanced CastVal function
  • Improved setGetMetadata for StarRocks
  • Updated fs.GetRefTs to handle timestamp values
  • Improved Replace0x00 function

• New Features

  • Added SSH tunnel support for FTP, S3 (MinIO), and SFTP
  • Implemented KEEP_TEMP_FILES option for BigQuery and StarRocks
  • Added EnvFileConnectionEntries and LoadSlingEnvFileBody
  • Introduced store.Settings
  • Implemented primary key and index table-key logic
  • Added column coercion for BigQuery, Snowflake & Redshift sources

Bug Fixes

• Resolved issues with ClickHouse tests and DDL with primary key
• Fixed blank value logic when using JSON
• Addressed ds.SetFields functionality
• Fixed MotherDuck checksum_boolean issue

Database-Specific Updates

• ClickHouse: Improved DDL handling with primary key
• BigQuery: Allow partitioning with custom SQL
• Snowflake, BigQuery, DuckDB, Redshift: Added dummy select for create_index & drop_index
• Oracle: Updated client handling
• Microsoft SQL Server: Upgraded driver

Configuration and Logging

• Enhanced logging throughout the application
• Improved DefaultStreamConfig to preserve transforms
• Updated SQL logging to retain full text

Dependency Updates

• Multiple updates to github.com/flarco/g
• Upgraded github.com/microsoft/go-mssqldb

Miscellaneous

• Expanded allowed glob pattern
• Improved sling update process
• Added warning for large single parquet file writes
• Updated README and documentation

Sling v1.2.14 (2024-07-25T13:35:14Z)

General Improvements and Fixes

• Enhanced error handling and messages:
  • Improve errors (d89b764)
  • Improve fs.Delete error messages (4299414, 9d65968)
  • Improve connection error logic (4715616)
  • Fix SQLite binary location (d38ed0f)
  • Update processConns return (8deba9b)
  • Add proper os.Exit (86c541f)
  • Use legacy FILEPATH_SLUG (f007fde)
• Improved functions and logic:
  • Improve CastToStringSafeMask (c625f8c)
  • Improve limit logic & templates (c3c8e47)
  • Fix Table.Select for SQL Server offset (745f0b2)
  • Allow URI connection for wildcard (9233fe6)
  • Improve Snowflake GetFile (eb69995)
  • Clean up BulkExportFlow tables var (571a266)
  • Optimize Snowflake BulkExportFlow (5dde593)
  • Add partition functions (5765734)
  • Update BulkExportFlow params (6d82013)
  • Add IsJSONorYAML (8d70514)
  • Update store struct & logic (815628c)
  • Improve FormatTargetObjectName (6002868, ff7d4d9)
  • Improve ParseTableName (c6c3ff1)
  • Use CastToStringSafeMask for processBwRows (4732505)
• Fixed specific issues:
  • Fix schemata_test (0fec0e2)
  • Fix Snowflake StageGET (0af47e2)
  • Fix Table.Select for SQL Server offset (745f0b2)
  • Use SLING_TEMP_CLOUD_FOLDER (d76949b)

Database Enhancements

• Improved PostgreSQL and Snowflake functionalities:
  • Improve PostgreSQL schemata columns listing (materialized views) (8a2198c)
  • Fix Snowflake StageGET (0af47e2)
  • Improve Snowflake GetFile (eb69995)
  • Optimize Snowflake BulkExportFlow (5dde593)
  • Customize the batch size for proton (5cbce10)
  • Update database_proton.go (adaed27)
• Added and refined Snowflake and DuckDB features:
  • Update DuckDB version error (f68d71a)
  • Upgrade DuckDB to 1.0.0 (4da9b34)
  • Add offset option (726dd02)
  • Update processConns return (8deba9b)
  • Use SLING_TEMP_CLOUD_FOLDER (d76949b)
• Enhanced MongoDB connection handling:
  • Close cursor when done (e7a1f39)

Code and Dependency Updates

• Updated dependencies and modules:
  • Update github.com/flarco/g (2946b04, d70c8b5)
  • Upgrade Go modules to 1.22 (c195e79)
  • Update Go mod minor versions (fc70cc0)
  • Upgrade github.com/parquet-go/parquet-go v0.23.0 (b8d6fe2)

New Features and Configurations

• Added new functionalities:
  • Add IsJSONorYAML (8d70514)
  • Ignore test 22 for Oracle (129ceef)
  • Clean up BulkExportFlow tables var (571a266)
  • Add partition functions (5765734)
  • Customize the batch size for proton (5cbce10)
  • Override insert_from_table SQL template (3ec03ca)
  • Update MD5 logic, add lookupReplication (41e5ba0)
  • Use replication.OriginalCfg (a9b9a6e)
  • Add props to parquet writer (fdcd75d)
• Updated configurations and parameters:
  • Update BulkExportFlow params (6d82013)
  • Clean up store (4e43b84)
  • Update store struct & logic (815628c)
  • Improve FormatTargetObjectName (6002868, ff7d4d9)
  • Improve ParseTableName (c6c3ff1)
  • Use CastToStringSafeMask for processBwRows (4732505)

Test and Documentation Updates

• Improved tests and documentation:
  • Fix schemata_test (0fec0e2)
  • Improve tests (72d1488)
  • Update README, mention timeplus proton (ea2592d)

Sling v1.2.13 (2024-06-29T16:06:47Z)

Changelog

• 8bfa6f8 update goreleaser version

Sling v1.2.11 (2024-06-04T18:54:45Z)

Changelog

• 58465e1 Merge pull request #303 from slingdata-io/v1.2.11
• 439c05f improve CopyToGCS for views
• 54e65de add error logging
• 353e726 revert export changes, add header
• 3a9c6a2 improve redshift data types & copy_to_s3
• 027e3e1 handle mongodb+srv
• 41825a5 add test for "single" prop
• d5144b8 fix defaults logic with wildcards, add single prop
• 36f1db1 add target_options.batch_limit
• ed2e167 improve store logging
• 5c52a8a update default repetitions
• 31604f1 cleanup
• cc6d22b add clickhouse batch.Limit
• deaf46d improve NewCsvReader
• e63dd83 clean up
• fe03d9c mssql: don't set defaultPort if instanceName is provided
• 90778dd redshift unload: add header
• 9ea426d update redshift unload
• 7cb8bdc set timestamp_layout to micro-second precision
• 3b37f31 add AllKind
• f5b96fa improve BQ BulkExportFlow with nulls
• daa93dd improve redshift BulkExportFlow
• 4131cb5 fix ignore_existing error on missing files
• 62c7efa improve IncrementalVal logic

Sling v1.2.10 (2024-05-18T11:10:04Z)

Changelog

• b38e6c0 Merge pull request #289 from slingdata-io/v1.2.10
• 7163326 upgrade cloud.google.com/go/bigquery
• 11cb4fa upgrade github.com/apache/arrow/go
• 5708c6b upgrade github.com/snowflakedb/gosnowflake
• d7180b0 upgrade github.com/go-sql-driver/mysql
• 2660eb2 upgrade github.com/lib/pq
• 921861e upgrade https://github.com/microsoft/go-mssqldb
• 533b4ff upgrade github.com/sijms/go-ora/v2
• f50c701 upgrade github.com/trinodb/trino-go-client
• a9292de update panic error logging for processBwRows
• 5d45a81 set config.originalCfg correctly
• cc6b5d1 add ignore_existing feature
• a5cee8a rename fs.WriteDataflow
• ec70e9c fix NewSharedKeyCredential
• 04221e2 add azure default credential, shared key auth
• 89643d5 improve replication Compile logic
• 64e4103 add file incremental test
• 09eb001 improve file incremental logic
• 45f489a improve store
• d765ad9 update SLING_PPROF logging
• 9c341b5 add Body to EnvFile
• 9147cc7 let sp.ParseTime return null for blank string
• 9e470c5 update columns.Clone
• 3ce7446 lowercase CompressorType values
• 04d386e update github.com/ClickHouse/clickhouse-go/v2
• cd0ca77 add AllFileType, AllCompressorType, AllType, AllMode, AllColumnCasing
• b4d6ad1 add table info
• 9ed844b improve Mongo type parsing
• 08aca6e add logic to adapt to different columns from various csvs
• 3fd1c2d update totalBytes
• 48f8c31 allow writing of headers on empty stream
• 5c81874 add delay in AddMissingColumns for BigQuery
• 6c7ea45 add SLING_PROCESS_BW
• 6c9f857 formatting
• f7d825d typo
• 0f150ec allow redshift to export without s3 creds
• 6135f1e improve incremental logic for files
• c422ca3 add better timestampZ when inferring
• 12197b1 add github.com/shirou/gopsutil/v3
• 4db316f update gopsutil
• f8e817b update bigquery CastColumnForSelect
• c6c75ff clean up parquet dbType
• 2e91b5b add logging
• aac7281 update build scripts [skip ci]

Sling v1.2.9 (2024-04-28T17:26:45Z)

Changelog

• 0886007 v1.2.9

Sling v1.2.6 (2024-04-22T11:29:49Z)

Changelog

• 39bf63a Merge pull request #269 from slingdata-io/v1.2.6
• d75e49d add unlockFileContext for duckdb stability
• 0adb388 fix table.Select for SQLServer
• 2bbcfb7 increase timeout
• 9ce5fa4 improve table Select logic
• 98717e4 improve ConsumeCsvReaderChl empty file logic
• 83d01e8 handle sql server inner order by
• b1832a7 update test parallelism
• 60b839f improve OnColumnAdded
• 40542aa add ReaderReady for _sling_stream_url, improve tests
• fb11efe add comment
• f48f5e4 use SLING_ALLOW_EMPTY
• 5cb54a8 update comment
• f114f34 use tVal.UTC() for dates
• 93fad2b add CastRowToStringSafe

Sling v1.2.5 (2024-04-15T14:51:05Z)

Changelog

• 1b96952 Merge pull request #261 from slingdata-io/v1.2.5
• 8da5234 add AutoDecompress to csv.ReadStreamContext
• 59b2ac3 fix ConsumeCsvReaderChl logic
• b4af629 fix bcp with bcp_entra_auth
• e7d37bd move AutoDecompress upstream
• 4687a81 use ReaderReady