Skip to content

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3,412 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

infa2td

Tool-agnostic ETL metadata converter and semantic migration planner.

Today, the shipped source frontends include Informatica PowerCenter plus lower-maturity SSIS, DataStage, Talend, dbt, and CDC adapters. Shipped target emitters generate ready-to-review SQL, DDL, workflow orchestration scripts, explain artifacts, runtime packages, and Graphviz data-flow diagrams for Teradata, Snowflake (co-primary), BigQuery, and dbt targets. Snowflake is now certified as a production-ready co-primary target alongside Teradata. The dbt emitter includes automatic inference of not_null and unique tests in schema.yml from canonical data lineage. Tool-agnosticism is the project's primary design invariant — not just an aspiration: the architecture, planner, and IR stay source-tool and target-tool agnostic even where individual adapter coverage is still expanding, and that neutrality is machine-enforced at the hourglass waist by import-linter ring contracts and source-vocabulary leak ratchets (make check-source-adapter-leaks; input-side core coverage is in progress).


Quick start

# Preview one mapping on stdout (database auto-derived from XML DBDNAME)
uv run python main.py --source-tool powerCenter --target teradata --input my_mapping.xml --stdout

# Convert a folder; write SQL files
uv run python main.py --source-tool powerCenter --target teradata --folder ./exports/ --output ./sql/

# Full output: DML + DDL + workflow scripts + graphs
uv run python main.py --source-tool powerCenter --target teradata --folder ./exports/ --output ./sql/ \
               --with-ddl --with-workflows --scheduler bteq --with-graphs

# Override database prefix when target differs from source XML
uv run python main.py --source-tool powerCenter --target teradata --folder ./exports/ --output ./sql/ --database DW_PROD --with-ddl

Documentation


Output

sql/
  _summary.txt                  scan and conversion report
  _manual_actions.txt           deduped manual action items
  dml/
    <PC_FOLDER>/
      <mapping_name>.sql        INSERT/SELECT, MERGE, UPDATE, or DELETE
  ddl/
    <table_name>.sql            CREATE TABLE  (with --with-ddl)
    error_tables/
      <table_name>_ERR.sql      error table DDL for DD_REJECT targets  (auto-generated)
  wf/
    workflow_<name>.sql          BTEQ orchestration script  (with --with-workflows)
    worklet_<name>.sql
  reusable_components/
    <component_name>.sql        reusable component validation SQL  (with --with-reusable-components / --with-mapplets)
  graphs/
    mapping_<name>.dot           data-flow diagram  (with --with-graphs)
    lineage_<name>.dot           column-level lineage per mapping
    mapplet_<name>.dot
    workflow_<name>.dot          / worklet_<name>.dot
    orch_workflow_<name>.dot     per-workflow orchestration artifact graph
    _collection.dot             high-level overview of all objects
    _everything.dot             single graph covering the entire suite
    _dependency_lineage.dot     cross-mapping dependency lineage
    _impact_analysis.dot        column-level impact analysis
    _output_artifacts.dot       output artifact dependency graph  (with --with-output-graphs)
  explain/
    <PC_FOLDER>/
      <mapping_name>.explain.json   per-mapping structured explain payload
  lineage/
    <mapping_name>.lineage.json      structured column lineage  (with --with-lineage-json)
    <mapping_name>.openlineage.json  static OpenLineage JobEvent  (with --with-openlineage-json)
  validation/
    <mapping_name>.sql               reconciliation SQL  (with --with-validation)
  params/
    <PC_FOLDER>/
      <mapping_name>_parameters.txt  unresolved parameter report
  tpt/
    <mapping_name>.tpt               Teradata TPT template  (with --target teradata --with-tpt)
  runtime/
    <mapping_name>/
      manifest.json             runtime package metadata
      *.sql                     runtime state management SQL steps

Each DML file opens with a -- Pipeline: comment block that lists every Informatica object in execution order — source tables, transforms (with key properties), and targets — so the SQL is self-documenting without needing to open the XML.


CLI reference

For detailed command line usage, packaging, examples, and every current option, see Command Line Instructions.

uv run python main.py --help
Flag Description
--folder DIR Scan a folder of PC XML exports recursively
--input FILE Convert a single XML file
--output DIR Root output directory (default: ./output)
--database DB Teradata database prefix for target tables (optional — auto-derived from XML DBDNAME when omitted). Overrides both DBDNAME and $$TARGET_DB parameter.
--with-ddl Also write CREATE TABLE DDL for target tables
--ddl-only DDL only; skip DML
--include-sources Include DDL for source tables
--row-uniqueness auto|allow_duplicates|unique_rows auto (default), allow_duplicates, or unique_rows; Teradata resolves auto to its default (MULTISET)
--ddl-fallback Emit FALLBACK on generated CREATE TABLE
--ddl-before-journal on|off Emit BEFORE JOURNAL / NO BEFORE JOURNAL
--ddl-after-journal on|off Emit AFTER JOURNAL / NO AFTER JOURNAL
--ddl-freespace PCT Emit FREESPACE = PCT PERCENT (0-100)
--ddl-checksum MODE Emit CHECKSUM = MODE (for example DEFAULT)
--ddl-datablocksize BYTES Emit DATABLOCKSIZE = BYTES
--param-file FILE Informatica .par file — resolves $$Param and $$Variable tokens
--source-locking none|auto|row_access|share_mode|exclusive_mode Optional source-read lock modifier. auto (recommended) emits LOCKING ROW FOR ACCESS when all sources are Teradata; row_access always emits it; share_mode/exclusive_mode emit ACTION REQUIRED review markers.
--source-ambiguity-policy pick|fail Source anchor ambiguity behavior. pick (default) auto-selects best match and emits a note; fail emits ACTION REQUIRED and skips executable DML for ambiguous/unresolved cases. Alias: --sq-ambiguity-policy.
--readiness-gate CI gate (folder mode). Returns non-zero when blockers remain: generation errors, unresolved cross-reference gaps, ambiguous SQ matches, or ACTION REQUIRED/TODO markers.
--readiness-min-confidence FLOAT Optional readiness threshold. Fails gate if average explain-trace confidence is below this value.
--readiness-max-fallback-rate FLOAT Optional readiness threshold. Fails gate if explain-trace fallback ratio exceeds this value.
--strict-profile Production-safe profile: sets --source-ambiguity-policy fail, enables --readiness-gate, sets --window-order-policy fail, and defaults thresholds to min confidence 0.90 / max fallback 0.05 (unless explicitly provided).
--no-recurse Do not descend into sub-directories (folder mode)
--stdout Print to stdout instead of writing files
--with-workflows / --no-with-workflows Emit orchestration scripts per workflow / worklet when the source has workflows (on by default; --no-with-workflows suppresses)
--target teradata|snowflake|sqlite|bigquery Required target SQL dialect. snowflake emits Snowflake-dialect SQL; sqlite emits SQLite-compatible SQL (INSERT only); bigquery emits BigQuery-dialect SQL.
--qualify-source-schema off|owner|auto Schema-qualify source table names in FROM clauses (default: auto)
--connector-coercion safe|strict Connector coercion behavior for datatype mismatches (default: safe)
--dynamic-lookup-mode advisory|volatile|replay Dynamic Lookup lowering strategy (default: advisory)
--window-order-policy warn|fail Behavior when window function has no upstream Sorter (default: warn; --strict-profile sets fail)
--null-safe-equality off|selective|strict NULL-safe equality mode (default: off; selective uses IS NOT DISTINCT FROM in JOINs)
--dml-temporal-qualifier none|current|nonsequenced Temporal qualifier for INSERT/MERGE on temporal tables (default: none)
--ddl-temp-table-mode none|volatile|gtt Temporary-table DDL contract (default: none)
--ddl-temp-on-commit auto|preserve|delete ON COMMIT behavior for temporary tables (default: auto)
--ddl-temporal-mode none|basic|advanced Temporal table clause emission: basic for single-clause, advanced for bitemporal (default: none)
--infer-secondary-indexes on|off Infer USI/NUSI secondary indexes from Lookup/Joiner conditions (default: off)
--with-graphs Generate Graphviz .dot diagrams
--with-output-graphs Generate output-artifact dependency graph under graphs/
--render-graphs FMT Render .dot files to svg, png, pdf, etc. (requires dot on PATH)
--with-reusable-components / --with-mapplets Generate reusable component validation SQL under reusable_components/ (PowerCenter)
--scheduler airflow|bteq|ctrlm|kestra Target scheduler format for emitted workflows (defaults to airflow)
--worklet-max-depth N Maximum worklet nesting depth for inline expansion (default: 8)
--post-load-stats Append COLLECT STATISTICS steps after session DML in workflow scripts
--with-tpt Emit Teradata Parallel Transporter templates and TPT load hints for partition directives (Teradata target only)
--bteq-maxerror N Emit .MAXERROR N at top of BTEQ scripts (default: 8; 0 to suppress)
--bteq-error-handling none|standard|strict Emit .IF ERRORCODE / .IF ACTIVITYCOUNT guards in BTEQ scripts. standard adds error-code guards; strict also checks zero-row detection (default: none)
--bteq-recovery none|checkpoint|full BTEQ recovery checkpoint mode (default: none)
--readiness-max-notes INT Fail readiness gate when total NOTE advisory marker count exceeds this value
--scorer-config PATH JSON file overriding confidence scorer weights and band thresholds
--runtime-state-backend none|teradata|snowflake|sqlite|bigquery Runtime state persistence backend (default: none). Non-none values must match --target.
--emit-runtime-package / --no-emit-runtime-package Toggle runtime SQL package emission for mappings with runtime semantics (default: auto-detect)
--runtime-schema SCHEMA Schema for runtime control tables (default: etl_meta)
--variable-default-strategy infer|strict Runtime variable defaulting strategy (default: infer)
--disallow-parallel-mapping-runs Release-1 runtime contract guardrail; enabled by default
--simulate Run semantic simulation and attach results to explain payload
--persistent-lookup-strategy volatile|materialized Persistent lookup materialization strategy (default: volatile)
--cte-emission Extract lookup subqueries into WITH clauses
--cte-nesting-threshold N Minimum lookup-join count before CTE extraction activates (default: 3)
--with-lineage-json Generate structured JSON lineage under lineage/
--with-openlineage-json Generate static OpenLineage JobEvent JSON under lineage/
--with-validation Generate reconciliation SQL under validation/
--with-views Emit REPLACE VIEW DDL for intermediate pipeline stages
--deploy-udfs Concatenate referenced UDF procedure SQL into _udf_deployment.sql
--ddl-partition-interval none|daily|monthly|yearly|DAY|MONTH|YEAR Partition interval for date/timestamp columns (default: none)
--ddl-partition-start DATE Lower bound for date partitioning
--ddl-partition-end DATE Upper bound for date partitioning
--ddl-partition-numeric-step N Step size for numeric partitioning
--ddl-partition-inference off|auto|strict|noted Infer partition bounds from mapping filters (default: off)
--query-band 'key=val;...' Append custom Teradata session query-band metadata
--no-query-band Suppress Teradata query-band metadata
--bigquery-project PROJECT_ID BigQuery project ID (--target bigquery)
--bigquery-dataset DATASET_NAME BigQuery dataset name (--target bigquery)
--bigquery-location LOCATION BigQuery location (default: US)
--shortcut-search-path DIR Additional PowerCenter shortcut search directory; repeatable
--duplicate-policy newest|all How to handle duplicate object names across XML files (default: newest)
--no-sibling-context Disable sibling XML scanning in single-file mode

Single-file scoping: when --input names one file, sibling XML files in the same directory are still scanned for workflow/connection metadata (richer session and source-qualifier resolution; disable with --no-sibling-context), but standalone artifacts — scheduler scripts under wf/, workflow explain payloads, and connection setup — are emitted only for workflows that orchestrate a mapping parsed from the named file (plus the worklets they call). An isolated copy of the same file produces the identical artifact set. These are two different knobs: --no-sibling-context opts out of resolution context; emission scoping is always on.

Runtime note: SQL generation is ILM-native only; the --engine switch was removed.


Supported transform types

Transform SQL output
Source Qualifier FROM clause, WHERE (Source Filter), ORDER BY (sorted ports), SQL override
Expression Derived columns in SELECT; expressions translated to Teradata syntax
Lookup LEFT JOIN with condition from Lookup Condition property; --dynamic-lookup-mode volatile can pre-stage dynamic lookup sources into a mapping-scoped VOLATILE TABLE snapshot, reuse one snapshot across equivalent lookup table/condition contracts, replay insert-only target deltas into that snapshot for later consumers, and otherwise defer refresh until the next target block that reuses the lookup after an intervening target write; --dynamic-lookup-mode replay now extracts replay contracts, stages ordered pre-lookup source rows when Sorter or Source Qualifier order is provable, and can emit narrow executable same-target replay subsets for insert_only, update_else_insert, data_driven_insert_update, and data_driven_insert_update_delete; direct same-target lookup-return passthrough is now supported across those subsets for one dynamic lookup, equivalent multi-lookup contracts, or non-equivalent multi-lookups when each lookup feeds distinct target outputs, including per-port defaults via COALESCE, DD_REJECT rows remain no-op target writes inside the data-driven subsets, direct same-column multi-lookup fan-in is treated as replay-invalid unless precedence is resolved upstream or contracts collapse safely, and replay also classifies non-report-error multiple-match policies, conditional cache updates, target Update Override semantics, explicitly nullable matched-target lookup keys, conflicting explicit source row-treatment modes for data-driven replay, explicit non-Data Driven source row treatment for data-driven replay, explicit contradictory target session row options for data-driven replay including Update as Insert=YES, multiple upstream Update Strategy transforms reaching the lookup for data-driven replay, multiple target-bound Update Strategy transforms for data-driven replay, mismatched upstream versus target-bound Update Strategy semantics for data-driven replay, missing target-bound Update Strategy metadata for data-driven replay, missing NewLookupRow routing to the matched target for data-driven replay, path-aware single-hop FILTER/ROUTER tracing for indirect NewLookupRow routing through intermediate transformations, extra direct NewLookupRow consumers beyond the matched target, competing explicit Sorter or ordered Source Qualifier contracts on the target path, unproven lookup-key uniqueness, and explicit new-value lookup-output mode as invalid instead of inventing semantics
Filter WHERE clause
Aggregator GROUP BY + aggregate functions (SUM, COUNT, MAX, MIN, AVG)
Joiner JOIN clause (Normal, Left Outer, Right Outer, Full Outer)
Router One INSERT per group with first-match guards (NOT(...)) so rows route to the first matching group only
Union UNION ALL subquery
Sorter ORDER BY with per-port ASC/DESC
Rank RANK() OVER (PARTITION BY … ORDER BY …)
Update Strategy INSERT / UPDATE / DELETE / MERGE INTO driven by DD_* constants
Sequence Generator NEXT VALUE FOR <sequence>
Normalizer Concrete UNPIVOT template (single value/group) or UNION ALL template (multi-value/group)
SQL Transformation Informatica→TD translated SQL; ?port? tokens auto-bound from SQL TF metadata/connectors, including unique exact and normalized IN_/OUT_ port-name fallbacks, target-port-name narrowing for undeclared exact-match collisions, and input-capable narrowing or same-binding collapse for normalized collisions, with explicit per-token diagnostics for unresolved/ambiguous bindings
Stored Procedure Runnable CALL proc(in_ports...) skeleton with IN / OUT / RETURN port inventory
Transaction Control BT / ET / ROLLBACK BTEQ statements wrapping the mapping DML, driven by Commit Type property (TC_COMMIT_BEFORE, TC_COMMIT_AFTER, TC_ROLLBACK_BEFORE, TC_ROLLBACK_AFTER)

Expression translation

Core Informatica expression functions are translated to Teradata equivalents, with deterministic fallback markers for unsupported/UDF-only cases.

Date / time

Informatica Teradata
SYSDATE CURRENT_DATE
SYSTIMESTAMP CURRENT_TIMESTAMP
ADD_TO_DATE(d, 'MM', n) d + INTERVAL 'n' MONTH
DATE_DIFF(d1, d2, 'DD') (d1 - d2) DAY(4)
LAST_DAY(d) (d - EXTRACT(DAY FROM d) + 1 + INTERVAL '1' MONTH - INTERVAL '1' DAY)
NEXT_DAY(d, 'MONDAY') TD_DAY_OF_WEEK-based expression
TRUNC(d, 'MM') TRUNC(d, 'MM')
TO_DATE(s, fmt) CAST(s AS DATE FORMAT fmt)
TO_CHAR(d, fmt) TRIM(CAST(d AS VARCHAR(50) FORMAT fmt))

String

Informatica Teradata
LTRIM(s) TRIM(LEADING FROM s)
RTRIM(s) TRIM(TRAILING FROM s)
INSTR(s, sub) POSITION(sub IN s)
SUBSTR(s, n, len) SUBSTR(s, n, len)
LENGTH(s) CHAR_LENGTH(s)
LPAD / RPAD LPAD / RPAD

Conditional / null

Informatica Teradata
IIF(cond, t, f) CASE WHEN cond THEN t ELSE f END
ISNULL(x) x IS NULL
NVL(x, y) COALESCE(x, y)
DECODE(x, v1, r1, …) CASE … WHEN v1 THEN r1 … END; NULL match values emit IS NULL (not = NULL)

Date / time (extended)

Informatica Teradata
SET_DATE_PART(d, 'YYYY', yr) CAST(LPAD(yr,4,'0') || '-' || EXTRACT(MONTH…) || … AS DATE FORMAT 'YYYY-MM-DD')
SET_DATE_PART(d, 'MM', mo) Analogous CAST+EXTRACT for month replacement
SET_DATE_PART(d, 'DD', dy) Analogous CAST+EXTRACT for day replacement
SET_DATE_PART(d, 'HH', hr) CAST+EXTRACT returning TIMESTAMP(0)
SET_DATE_PART(d, 'MI', mi) CAST+EXTRACT returning TIMESTAMP(0)
SET_DATE_PART(d, 'SS', ss) CAST+EXTRACT returning TIMESTAMP(0)

Regular expressions

Informatica Teradata
REG_MATCH(s, pat) REGEXP_SIMILAR(s, pat, 'i') = 1
IS_DATE(s, 'YYYY-MM-DD') TRY_CAST(s AS DATE FORMAT 'YYYY-MM-DD') IS NOT NULL — validates actual calendar dates
IS_DATE(s, 'DD-MON-YYYY') TRY_CAST(s AS DATE FORMAT 'DD-mmmYYYY') IS NOT NULL
REG_REPLACE(s, pat, rep) REGEXP_REPLACE(s, pat, rep)
REG_EXTRACT(s, pat, n) REGEXP_SUBSTR(s, pat, 1, n)

Long-tail parity behavior

Informatica Teradata output
CRC32(x) HASHROW(x) plus explicit approximation NOTE comment
FIRST(x) / LAST(x) FIRST_VALUE(x) OVER (...) / LAST_VALUE(x) OVER (...) with sorter/order caveat guidance when needed
METAPHONE(x) Deterministic ACTION REQUIRED marker (NULL expression placeholder)
COMPRESS(x) / DECOMPRESS(x) Deterministic ACTION REQUIRED marker (NULL expression placeholder)

Unsupported/unknown functions produce deterministic ACTION REQUIRED markers rather than silent pass-through. Known unsupported families (for example FV/NPER/PMT/PV/RATE) emit dedicated blocker text; other unknown functions emit the generic unknown-function blocker.


Parameters and workflow variables

Informatica $$Param and $$Variable tokens are resolved in three ways:

  1. Parameter file (--param-file prod.par) — values from the .par file take highest precedence, scoped by [folder.WF:workflow.ST:session] heading.
  2. Declared defaultsDEFAULTVALUE attributes in the XML.
  3. Unresolved — emitted as :bteq_variable_name /* $$PARAM | type:string | no default */ so the SQL is still executable after substitution.

Database prefix ($$TARGET_DB): if a mapping declares a parameter named $$TARGET_DB (or $$DATABASE, $$DB_NAME, etc.) that parameter is used as the table prefix instead of --database. The generated SQL reads INSERT INTO $$TARGET_DB.MY_TABLE, making the target environment explicit. The parameter header comment marks it with ← db prefix.

When using --with-workflows, workflow-level variables are automatically injected into each session's mapping registry. The generated BTEQ file opens with .SET :var = value; declarations for every workflow variable.

Safety behavior for generated workflow SQL:

  • Multiline metadata values are collapsed before being emitted into single-line SQL comments so no bare/unprefixed lines leak into executable SQL.
  • .SET string defaults are emitted as quoted BTEQ literals with embedded apostrophes escaped (' -> '').
  • Advisory block-comment payloads are neutralized to prevent embedded */ from breaking out of generated comments.

PC schema compatibility

PC version XML elements Repository version
9 / 10 TRANSFORMATION + TRANSFORMFIELD ~165–187
6 / 7 / 8 TRANSFORM + FIELD ~65–110

Both schemas are parsed transparently. All other elements (MAPPING, SOURCE, TARGET, CONNECTOR, WORKFLOW, SESSION, etc.) are identical between versions.

INSTANCE node handling: three aliasing patterns are all resolved automatically so connector references always point to the right transformation definition:

Pattern XML Resolution
Inline (non-reusable) <INSTANCE NAME="EXP_A" TRANSFORMATION_NAME="EXP_A"/> Name matches definition directly — no aliasing needed
Reusable, same name <INSTANCE NAME="EXP_SHARED" TRANSFORMATION_NAME="EXP_SHARED"/> Injected from folder-level reusable registry by name
Reusable, renamed instance <INSTANCE NAME="EXP_LOCAL" TRANSFORMATION_NAME="EXP_CANONICAL"/> instance_aliases dict maps the local name to the canonical; injected under the local alias so all CONNECTOR references resolve
Cross-folder SHORTCUT <SHORTCUT NAME="LKP_ALIAS" REFOBJECTNAME="LKP_CANONICAL"/> shortcut_map alias → canonical → injected under alias

Workflow SQL (--with-workflows)

Each workflow or worklet becomes scheduler-specific output. Select --scheduler bteq for BTEQ .sql, --scheduler ctrlm for Control-M .xml, --scheduler kestra for Kestra flow .yml, or --scheduler airflow for Airflow .py when --with-workflows is enabled:

-- ====================================================================
-- Workflow: WF_NIGHTLY_EVENTS
-- ====================================================================
-- Source file : m_WORKFLOW_COMMANDS_TEST.xml
-- Folder      : ETL_NIGHTLY
-- Workflow variables — set before running:
--   .SET :run_dt = NULL  /* no default — must be provided */;  -- date/time
--   .SET :batch_id = 0;  -- integer

.SET :run_dt = /* provide value */;
.SET :batch_id = 0;

-- LOGON hint derived from connection metadata when available
.LOGON <tdpid>/<username>,<password>;

-- Command task: CMD_DECOMPRESS
-- Shell script:
-- gunzip -f /data/events/incoming/*.gz

-- Session: s_m_LOAD_FACT_EVENTS  →  mapping: m_LOAD_FACT_EVENTS
-- -----------------------------------------------------------------
.OS rm -f /tmp/events_staging/*;
DELETE FROM DW.FACT_EVENTS ALL;
INSERT INTO DW.FACT_EVENTS ( … ) SELECT … FROM STG_EVENTS;

-- Assignment: ASSIGN_BATCH
-- Variable assignment: :BATCH_ID = :BATCH_ID + 1

-- Decision: DEC_CHECK
-- Branch condition: $s_m_LOAD_FACT_EVENTS.Status = 'SUCCEEDED'

.LOGOFF;

Non-SQL tasks (Command, FTP, HTTP, Email, Decision, Assignment) are preserved as structured SQL comments rather than being silently dropped. Session pre/post shell commands are emitted as BTEQ .OS commands when they are single-line shell statements. Workflow/task metadata used in those comments is sanitized before rendering. Task-link conditions are also translated: BTEQ emits deterministic .IF ... THEN .GOTO skip_* guards, Airflow maps failure/unconditional links to trigger_rule while lowering simple Decision status/boolean branches to BranchPythonOperator, and Control-M normalizes $$VAR references in task payloads to %%VAR AutoEdit variables while lowering simple boolean / workflow-variable Decision branches to explicit branch-gate jobs.

Workflow task capability levels by scheduler:

Task type BTEQ Control-M Airflow
Session native (inlined SQL) native (database job wrapper) native (task wrapper)
Command / FTP / HTTP / Email comment-only native/emulated native/emulated
Decision comment-only native for simple boolean / workflow-variable branches via branch-gate jobs; fail-visible for status/custom branch contracts native for simple status/boolean branches; fail-visible for unsupported custom branch contracts
Assignment comment-only native for simple literal / variable-copy workflow-variable updates via VARIABLE; fail-visible for broader expressions native for simple run-scoped workflow-variable updates via XCom-backed PythonOperator; fail-visible for broader expressions
Timer comment-only emulated (sleep) emulated (sleep)
Event-Wait / Event-Raise comment-only explicit manual gate (ACTION REQUIRED) explicit manual gate (ACTION REQUIRED)
Unknown task type comment-only blocking marker (ACTION REQUIRED) blocking marker (ACTION REQUIRED)

DDL Support (--with-ddl)

Advanced Teradata table options are feature-flagged so defaults remain backward-compatible:

DDL Feature Support How to enable
FALLBACK implemented --ddl-fallback
BEFORE/AFTER JOURNAL implemented --ddl-before-journal on|off, --ddl-after-journal on|off
FREESPACE implemented --ddl-freespace <pct>
CHECKSUM implemented --ddl-checksum <mode>
DATABLOCKSIZE implemented --ddl-datablocksize <bytes>
JSON / PERIOD / ARRAY[n] column types implemented automatic from source datatype metadata
Temporal-table attributes (VALIDTIME, TRANSACTIONTIME) implemented --ddl-temporal-mode basic|advanced
Volatile / GTT temp tables implemented --ddl-temp-table-mode volatile|gtt, --ddl-temp-on-commit preserve|delete
Secondary index inference (USI/NUSI) implemented --infer-secondary-indexes on
Error table DDL for DD_REJECT targets implemented automatic when DD_REJECT detected

Example:

uv run python main.py --source-tool powerCenter --target teradata --folder ./exports --output ./sql --database DW --with-ddl \
  --ddl-fallback \
  --ddl-before-journal off \
  --ddl-after-journal on \
  --ddl-freespace 5 \
  --ddl-checksum DEFAULT \
  --ddl-datablocksize 32768

Graphs (--with-graphs)

# Generate and render to SVG in one step
uv run python main.py --source-tool powerCenter --target teradata --folder ./exports/ --with-graphs --render-graphs svg

# Or render manually
dot -Tsvg output/graphs/mapping_m_LOAD_FACT_SALES.dot -o m_LOAD_FACT_SALES.svg
Graph file Contents
mapping_*.dot Left-to-right data flow: source → transforms (colour-coded by type) → target
lineage_*.dot Column-level lineage per mapping
mapplet_*.dot Internal mapplet structure
workflow_*.dot / worklet_*.dot Top-to-bottom task dependency with condition-labelled edges
orch_*.dot Per-workflow orchestration artifact graph
_collection.dot High-level overview: workflows → mappings → sources/targets → mapplets
_everything.dot Single graph covering every parsed object across all files
_dependency_lineage.dot Cross-mapping dependency lineage
_impact_analysis.dot Column-level impact analysis
_output_artifacts.dot Output artifact dependency graph (with --with-output-graphs)

Structured Lineage Artifacts

--with-lineage-json writes the stable infa2td lineage shape for direct use in tests, reports, and migration review. --with-openlineage-json writes a static OpenLineage JobEvent for tools that ingest OpenLineage metadata. Both files are rendered from the same typed internal lineage document, so source/target identity and column-level dependencies stay aligned.

{
  "lineage_version": 2,
  "mapping_name": "m_LOAD_CUSTOMER",
  "datasets": [
    {"role": "input", "name": "SRC.CUSTOMER", "identity": {"object_kind": "table"}}
  ],
  "column_lineage": [
    {
      "target_column": "FULL_NAME",
      "source_columns": ["FIRST_NAME", "LAST_NAME"],
      "resolution": {"status": "resolved", "confidence": 1.0}
    }
  ],
  "diagnostics": []
}
{
  "schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/JobEvent",
  "job": {
    "namespace": "infa2td://conversion",
    "name": "m_LOAD_CUSTOMER",
    "facets": {"infa2td_conversion": {"lineageVersion": 2}}
  },
  "outputs": [{"name": "DW.CUSTOMER_DIM", "facets": {"columnLineage": {"fields": {}}}}]
}

The openlineage optional extra installs openlineage-python for library users that want to build client event objects. Normal CLI conversion and static JSON artifact generation do not require that package or any network connection. The validation optional extra is reserved for future user-authored configuration validation.

Project layout

main.py                                 CLI entry point
cli/                                    CLI orchestration, arg parsing, artifact writing
  app.py                                App dispatcher
  args.py                               Argument parsing and profile application
  mapping_pipeline.py                   Per-mapping conversion pipeline (data payloads only)
  single_file.py / folder_mode.py       Single-file and folder conversion modes
  artifact_writer.py                    Write all artifact types to disk
sources/                               Source parsing and discovery
  etl/powercenter/parser.py             PowerCenter XML → data model parsing
  etl/{ssis,datastage}/parser.py        Additional source frontends
  _sdk/scanning.py                      Adapter-backed folder scan dispatcher
  etl/powercenter/scanner.py            PowerCenter multi-file scan and cross-file registry
  etl/powercenter/params.py             $$Param / $$Variable resolution and .par file parsing
  etl/powercenter/reusable_component.py Reusable component parsing
ir/                                     Intermediate representation layer
  ilm.py                                Intermediate Logical Model (ILM) IR
  runtime_models.py                     Runtime variable operations and execution plans
  replay_models.py                      Dynamic lookup replay contracts
  workflow_control_flow.py              Workflow DAG analysis
models/                                 Domain data classes
  source.py                             Mapping / Source / Target / Workflow data models
  plans/                                MappingPlan, LookupPlan modules
  target_plans.py                       TargetLoadPlan
  compilation_policy.py                 Frozen compilation policy (replaces kwargs)
  findings.py                           Diagnostic and ReadinessAssessment
  decision_trace.py                     Structured decision trace events
compiler/                               Decision-making and planning
  capability_planning/planner.py        Mapping plan assembly → MappingPlan
  workflow_planner/                     Workflow task scheduling and dependencies
  semantic_capabilities.py              Transform semantic capability classification
  targets/load_strategy.py              Load mode resolution (INSERT/UPDATE/MERGE/etc.)
  execution_units.py                    Execution-unit graph construction
  file_sources.py                       Flat-file source staging metadata
engine/                                 Runtime orchestration
  runtime.py                            DML generation runtime entrypoint
  _workflow_artifacts.py                Workflow → scheduler output assembly
expressions/                            Shared canonical AST waist and registries
  parser.py / ast.py                    Recursive-descent parser and AST nodes
sources/etl/powercenter/expression/     PowerCenter expression tokenizer and parser
targets/database/<dialect>/expression/  Target-specific expression lowering
emitters/                               SQL and artifact generation
  dml/orchestrator.py                   Mapping → INSERT/SELECT / MERGE / UPDATE / DELETE SQL
  ddl/bundle.py, ddl/table.py           Table definitions → CREATE TABLE SQL
  graph/                                Graphviz .dot graph generation
  replay/                               Dynamic lookup replay SQL
  runtime/runtime_sql.py                Runtime variable persistence SQL
targets/scheduler/{bteq,airflow,ctrlm}/ BTEQ / Airflow / Control-M workflow rendering
targets/database/teradata/tpt.py        Teradata Parallel Transporter templates
sql/                                    SQL AST and translation
  translator.py                         Informatica expression → Teradata SQL
  ast/                                  SQL AST nodes and rendering
analysis/                               Post-parse semantic analysis
  confidence.py                         Per-mapping confidence scoring
  validation.py                         Validation and readiness gating
runtime/                                Execution planning and semantic detection
  runner.py                             Runtime package execution helpers
target_services/                        Target dialect registry and service facade
adapters/                               Shared adapter contracts, runtime, and registry
  contracts.py                          Adapter interface contracts
targets/adapter/<family>/               Per-family adapter implementations
                                        ({xml,java,http,external_procedure,
                                         application_sq,unstructured_data})
compiler/_state_derivation.py           Stateful semantics analysis
models/state.py                         State data models
simulator/                              Row-level workflow state simulation
semantic_taxonomy/                      Canonical semantic model, rules, and proof types
compiler/planner_proofs.py              Compiler proof bundle builders
procedures/c/                           Teradata C UDF source helpers
tools/                                  Validation, analysis, benchmarking scripts
tests/                                  Pytest suite

Running the tests

# All tests
uv run pytest tests -n auto

# Lint (Ruff)
uv run ruff check .

# Docs consistency gate
uv run python tools/check_docs_consistency.py --check

# Regenerate parity matrix from catalog
uv run python tools/generate_parity_matrix.py

# One module
uv run pytest tests/test_transforms.py

# One test
uv run pytest tests/test_transforms.py::TestTransformTypes::test_rank_transform

Tests cover: folder sanity, all transform types, expression translation, PC 8 legacy schema, parameterisation, pipeline comments, workflow SQL, graph generation, gap-fix regressions (DECODE NULL, IS_DATE TRY_CAST, SET_DATE_PART, DD_REJECT, Normalizer UNPIVOT, Stored Procedure CALL, SQL Transformation translation), by-design gap closures, mapplet parameter handling, balanced-paren string-literal handling, SQ source heuristics, handler promotion, converging-loop stability, workflow renderer coverage, critical expression fixes (TO_CHAR, :lkp. nested parens, unmatched paren, multi-aggregator advisory, session MAPPINGNAME fallback, link-condition translation), and INSTANCE name aliasing for reusable transforms.

Pylance / VS Code

  • Pyright/Pylance project settings are in pyrightconfig.json.
  • Recommended interpreter path is ${workspaceFolder}/.venv/bin/python.
# Install tooling dependencies
uv sync --group dev

# Run type checks (application code)
uv run basedpyright

Benchmark baseline

# Run benchmark corpora from manifest and print metrics
uv run python tools/benchmark_baseline.py --manifest tools/benchmark_manifest.json

# Strict production profile benchmark + JSON artifact
uv run python tools/benchmark_baseline.py --strict-profile \
  --manifest tools/benchmark_manifest.json \
  --json-out output/baseline_report.json \
  --artifacts-dir output/baseline_artifacts

# Artifact-only collection (always exits 0 even when strict checks fail)
uv run python tools/benchmark_baseline.py --strict-profile --allow-failures \
  --manifest tools/benchmark_manifest.json \
  --json-out output/baseline_report.json \
  --artifacts-dir output/baseline_artifacts

# Enforce release thresholds against a benchmark report
uv run python tools/check_release_thresholds.py \
  --report output/baseline_report.json \
  --thresholds tools/release_thresholds.json

# Enforce runtime performance budgets against the same report
uv run python tools/check_performance_budget.py \
  --report output/baseline_report.json \
  --budget tools/performance_budget.json

Default corpus manifest includes:

  • small_daily: tests/test_folder/daily (non-recursive)
  • medium_core_nonrecursive: tests/test_folder with --no-recurse
  • large_regression_recursive: tests/test_folder recursive scan

Each run can persist _summary.txt, _manual_actions.txt, and explain JSON under --artifacts-dir for CI artifact publishing and later comparison.


Production runbook

uv run python main.py --source-tool powerCenter --target teradata --folder ./exports --output ./out --strict-profile

This profile enforces fail-fast behavior for ambiguity/manual-action markers and applies default confidence/fallback readiness thresholds for CI.

Strict profile expands to:

  • --source-ambiguity-policy fail (or legacy alias --sq-ambiguity-policy fail)
  • --readiness-gate
  • --window-order-policy fail
  • --readiness-min-confidence 0.90 (unless overridden)
  • --readiness-max-fallback-rate 0.05 (unless overridden)

Readiness gate failure reasons include:

  • generation failures
  • unresolved cross-reference gaps
  • ambiguous SQ source matches
  • ACTION REQUIRED / TODO markers
  • DYNAMIC LOOKUP runtime-strategy markers
  • explainability threshold violations

Remediation artifacts:

  • out/_summary.txt
  • out/_manual_actions.txt (including deduped dynamic-lookup blockers per lookup)
  • out/explain/**/*.explain.json

Confidence scorecard

Current scorecard: docs/ci-gated/CONFIDENCE_SCORING.md

The scorecard is derived from benchmark outputs and validated against tools/release_thresholds.json using tools/check_release_thresholds.py.


Parity changelog

Parity-status updates are tracked in docs/ci-gated/RELEASE_NOTES.md.


ILM diagnostics quick use

# Generate explain artifacts in ILM-native mode
uv run python main.py --source-tool powerCenter --target teradata --folder ./exports/ --output ./out

# Aggregate mismatch categories and top mismatched mappings
uv run python tools/explain_summary.py --explain-root ./out/explain --top 10

See ILM Diagnostics for field definitions and triage guidance.


Translation fidelity and known caveats

Fully translated (no manual action needed)

  • DECODE with NULL match values correctly emits IS NULL (not = NULL).
  • Router first-match semantics are preserved: later/default groups include NOT(...) guards to prevent duplicate routing across groups.
  • IS_DATE with any of the 10 standard date/timestamp formats uses TRY_CAST, which validates actual calendar correctness (e.g. rejects Feb 31). Formats using 3-letter month abbreviations (MON) fall back to REGEXP_SIMILAR with a NOTE.
  • SET_DATE_PART for the 6 most common units (YYYY, MM, DD, HH, MI, SS) generates executable Teradata SQL using EXTRACT + string concatenation + CAST. Unsupported units (sub-second) emit a manual-rewrite template.
  • DD_REJECT in Data Driven Update Strategy: a DD_REJECT-only mapping emits a comment-only output (no DML) explaining that all rows are directed to Informatica's bad file. In mixed-code MERGE statements, rows flagged as rejected (_upd_flag = 3) produce no target action, matching PC behaviour — this is now documented inline.

Requires manual completion (action-required comments emitted)

  • Normalizer: a concrete UNPIVOT template (single value per group) or UNION ALL template (multiple values per group) is generated from the field group structure. Replace <source_table_or_subquery> with the actual source name.
  • SQL Transformation: Informatica→TD function translation is applied to the SQL Query property and emitted as executable SQL. ?port_name? tokens are auto-resolved from SQL TF port expressions/connectors where possible, including unique exact incoming connector-field matches when SQL TF port metadata is incomplete, target-port-name narrowing when multiple undeclared exact-match connectors exist, unique normalized IN_/OUT_ port-name fallbacks for undeclared tokens, and normalized-collision narrowing when input-capable declared ports reduce to one bindable target or one shared binding; unresolved tokens are retained with explicit ACTION REQUIRED [SQL TF] diagnostics (including missing/undeclared/ambiguous reasons and connector candidates). Non-Teradata dialect constructs may still require manual adjustment.
  • Stored Procedure: a runnable CALL proc(in_port1, in_port2, ...) skeleton is emitted with the actual input, output, and return-value port names. Verify the parameter order and types against the Teradata procedure definition.
  • Window functions (CUME, MOVINGAVG, MOVINGSUM, LAG, LEAD, plus FIRST/LAST analytic lowering): when no upstream Sorter/order context is available, generated SQL now emits deterministic ACTION REQUIRED [WINDOW ORDER] markers inside the analytic OVER() clause instead of silently implying unordered semantics. Move the outer SELECT's ORDER BY (generated from the upstream Sorter transformation) into the OVER() clause.
  • Dynamic Lookup: default mode emits the existing DYNAMIC LOOKUP advisory plus a static LEFT JOIN. --dynamic-lookup-mode volatile upgrades this to an executable mapping-scoped CREATE MULTISET VOLATILE TABLE ... AS (SELECT * FROM <lookup>) WITH DATA snapshot, reuses that snapshot across target blocks and across equivalent lookup table/condition contracts, replays insert-only target deltas into the snapshot when a later target reuses the same lookup table, and otherwise defers refresh until the next target block that reuses the lookup after an intervening target write. --dynamic-lookup-mode replay is now an explicit uplift mode: it extracts replay-safety contracts, stages ordered pre-lookup source rows into a replay input volatile table when Sorter or Source Qualifier order is provable, and can emit executable same-target replay blocks for the narrow insert_only, update_else_insert, data_driven_insert_update, and data_driven_insert_update_delete subsets where target columns are directly staged and replay keys are simple equality columns. Direct same-target lookup-return passthrough is now supported across those subsets by enriching the replay stage from the current target row on the relevant equality keys, including lookup port defaults via generated COALESCE expressions when Informatica configured them. Equivalent multi-lookup contracts on the same target can share one replay join, and non-equivalent multi-lookups are now supported when each lookup contributes distinct target outputs. Within the data-driven replay subsets, DD_REJECT rows remain no-op target writes through the generated _upd_flag semantics, and DD_DELETE rows now lower into the same-target replay MERGE contract when the replay subset remains provable. Direct multi-lookup replay fan-in where multiple lookup outputs feed the same target column is now treated as mapping-invalid for replay unless equivalent contracts collapse it or the mapping resolves precedence upstream before the target; those cases emit ACTION REQUIRED [DYNAMIC LOOKUP REPLAY INVALID] instead of silently inventing precedence. Replay also treats non-report-error multiple-match policies, conditional cache updates, target Update Override semantics, explicitly nullable matched-target lookup keys, conflicting explicit source row-treatment modes for data-driven replay, explicit non-Data Driven source row treatment for data-driven replay, explicit contradictory target session row options for data-driven replay including Update as Insert=YES, multiple upstream Update Strategy transforms reaching the lookup for data-driven replay, multiple target-bound Update Strategy transforms for data-driven replay, mismatched upstream versus target-bound Update Strategy semantics for data-driven replay, missing target-bound Update Strategy metadata for data-driven replay, missing NewLookupRow routing to the matched target for data-driven replay, path-aware single-hop FILTER/ROUTER tracing for indirect NewLookupRow routing through intermediate transformations, extra direct NewLookupRow consumers beyond the matched target, competing explicit Sorter or ordered Source Qualifier contracts on the target path, replay contracts whose lookup keys are not provably unique on the matched target, and explicit new-value lookup-output mode as invalid because the current set-based replay does not model PowerCenter’s error, null-key equality, conditional cache-mutation, custom update SQL, source/target session row-treatment contracts, competing target row-type expressions, target-row routing, downstream output semantics, or ambiguous ordering there. Cross-target lookup-output and broader row-state-heavy replay cases still emit ACTION REQUIRED [DYNAMIC LOOKUP REPLAY] blockers instead of silently downgrading to snapshot semantics. Explain artifacts now include a structured dynamic_lookup_replay summary, and validation/reporting now counts executable replay subset notes separately from replay blockers while classifying replay-invalid mappings separately. Residual gap: PowerCenter cache mutation is still not fully replayed row-by-row inside one set-based DML, ambiguous/cross-target multi-lookup replay remains blocked, and broader row-state semantics are still unsupported.

Structural limitations

  • Strict-mode note: with --source-ambiguity-policy fail, ambiguous or low-confidence source-anchor matches (metadata-score diagnostic path) and unresolved connector instances (e.g. missing mapplet/shortcut aliases) emit ACTION REQUIRED and skip executable DML for that mapping.
  • SQL overrides (Source Qualifier, Lookup, and session pre/post SQL blocks) are translated and linted for known non-Teradata patterns. Eleven mechanical rewrites are applied automatically (NVL→COALESCE, SYSDATE→CURRENT_DATE, MINUS→EXCEPT, DECODE→CASE, FROM DUAL removal, ROWNUM→ROW_NUMBER()/QUALIFY, CONNECT BY→WITH RECURSIVE CTE, seq.NEXTVAL/CURRVAL→NEXT/CURRENT VALUE FOR, etc.) with NOTE [SQL OVERRIDE REWRITE] advisories. No remaining Oracle constructs emit ACTION REQUIRED.
  • Sequence Generator emits NEXT VALUE FOR <seq> — the sequence object must be created separately with --with-ddl.
  • BTEQ .SET syntax is used for variable substitution; adapt to your scheduler convention if needed (e.g. shell variable expansion or TD Vantage macros).
  • $PM* Integration Service variables (e.g. $PMRootDir) are replaced with :bteq_variable placeholders — set these in your BTEQ script preamble before execution.
  • Mapplet $$ parameters: MAPPINGPARAM and MAPPINGVARIABLE elements declared inside a MAPPLET are parsed and merged into the parent mapping's parameter registry. In the generated SQL: if the parameter has a known value (from a .par file or its DEFAULTVALUE), the literal is substituted inline; if no value is available, a :bteq_name /* $$ParamName | type:... | no default */ placeholder is emitted. Mapplet params are listed in the mapping header comment block with source tag [mapplet] so they are easy to identify.

About

No description, website, or topics provided.

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages