Background

Currently, information date format and type are ignored when the metastore persistence format is 'delta'.

For example, here the date format will be ignored:

pramen.metastore.tables = [
    {
      name = "table1"
      description = "Table 1 description"
      format = "delta"
      path = ${base.path}/table1

      information.date.column = "info_date"
      information.date.format = "yyyyMM"
    }
]

while here it will work:

pramen.metastore.tables = [
    {
      name = "table1"
      description = "Table 1 description"
      format = "parquet"
      path = ${base.path}/table1

      information.date.column = "info_date"
      information.date.format = "yyyyMM"
    }
]

(the only difference is the storage format)

Feature

Add the ability to have metastore partitioning column as a string, not a date.

Example

--

Proposed Solution

Pramen writes Parquet partitions directly, it constructs the path. For Delta we rely on the type system. So I guess we need to treat any date format that is not 'yyyy-MM-dd' as having string type.

Code to look: