Tool-agnostic ETL metadata converter and semantic migration planner.
Today, the shipped source frontends include Informatica PowerCenter plus lower-maturity SSIS, DataStage, Talend, dbt, and CDC adapters. Shipped target emitters generate ready-to-review SQL, DDL, workflow orchestration scripts, explain artifacts, runtime packages, and Graphviz data-flow diagrams for Teradata, Snowflake (co-primary), BigQuery, and dbt targets. Snowflake is now certified as a production-ready co-primary target alongside Teradata. The dbt emitter includes automatic inference of not_null and unique tests in schema.yml from canonical data lineage. Tool-agnosticism is the project's primary design invariant — not just an aspiration: the architecture, planner, and IR stay source-tool and target-tool agnostic even where individual adapter coverage is still expanding, and that neutrality is machine-enforced at the hourglass waist by import-linter ring contracts and source-vocabulary leak ratchets (make check-source-adapter-leaks; input-side core coverage is in progress).
# Preview one mapping on stdout (database auto-derived from XML DBDNAME)
uv run python main.py --source-tool powerCenter --target teradata --input my_mapping.xml --stdout
# Convert a folder; write SQL files
uv run python main.py --source-tool powerCenter --target teradata --folder ./exports/ --output ./sql/
# Full output: DML + DDL + workflow scripts + graphs
uv run python main.py --source-tool powerCenter --target teradata --folder ./exports/ --output ./sql/ \
--with-ddl --with-workflows --scheduler bteq --with-graphs
# Override database prefix when target differs from source XML
uv run python main.py --source-tool powerCenter --target teradata --folder ./exports/ --output ./sql/ --database DW_PROD --with-ddl- Documentation Index — curated map of all authoritative docs
- Command Line Instructions
- Purpose And Goals
- Architecture Pipeline
- Main Interfaces And Classes
- Systems Integration Architecture
- Informatica XML UML Coverage
- Architecture Decisions
- Coding Standards
- Maturity Matrix
- Gap Analysis
- Parity Matrix
- Parity Coverage
- Status Definitions
- Manual Playbook
- Feature Responsibilities
- Teradata Physical Design
- ILM Implementation
- ILM Diagnostics
- Dynamic Runtime Design
- Confidence Scoring
- Planning Docs Index
- Release Notes
sql/
_summary.txt scan and conversion report
_manual_actions.txt deduped manual action items
dml/
<PC_FOLDER>/
<mapping_name>.sql INSERT/SELECT, MERGE, UPDATE, or DELETE
ddl/
<table_name>.sql CREATE TABLE (with --with-ddl)
error_tables/
<table_name>_ERR.sql error table DDL for DD_REJECT targets (auto-generated)
wf/
workflow_<name>.sql BTEQ orchestration script (with --with-workflows)
worklet_<name>.sql
reusable_components/
<component_name>.sql reusable component validation SQL (with --with-reusable-components / --with-mapplets)
graphs/
mapping_<name>.dot data-flow diagram (with --with-graphs)
lineage_<name>.dot column-level lineage per mapping
mapplet_<name>.dot
workflow_<name>.dot / worklet_<name>.dot
orch_workflow_<name>.dot per-workflow orchestration artifact graph
_collection.dot high-level overview of all objects
_everything.dot single graph covering the entire suite
_dependency_lineage.dot cross-mapping dependency lineage
_impact_analysis.dot column-level impact analysis
_output_artifacts.dot output artifact dependency graph (with --with-output-graphs)
explain/
<PC_FOLDER>/
<mapping_name>.explain.json per-mapping structured explain payload
lineage/
<mapping_name>.lineage.json structured column lineage (with --with-lineage-json)
<mapping_name>.openlineage.json static OpenLineage JobEvent (with --with-openlineage-json)
validation/
<mapping_name>.sql reconciliation SQL (with --with-validation)
params/
<PC_FOLDER>/
<mapping_name>_parameters.txt unresolved parameter report
tpt/
<mapping_name>.tpt Teradata TPT template (with --target teradata --with-tpt)
runtime/
<mapping_name>/
manifest.json runtime package metadata
*.sql runtime state management SQL steps
Each DML file opens with a -- Pipeline: comment block that lists every Informatica object in execution order — source tables, transforms (with key properties), and targets — so the SQL is self-documenting without needing to open the XML.
For detailed command line usage, packaging, examples, and every current option, see Command Line Instructions.
uv run python main.py --help
| Flag | Description |
|---|---|
--folder DIR |
Scan a folder of PC XML exports recursively |
--input FILE |
Convert a single XML file |
--output DIR |
Root output directory (default: ./output) |
--database DB |
Teradata database prefix for target tables (optional — auto-derived from XML DBDNAME when omitted). Overrides both DBDNAME and $$TARGET_DB parameter. |
--with-ddl |
Also write CREATE TABLE DDL for target tables |
--ddl-only |
DDL only; skip DML |
--include-sources |
Include DDL for source tables |
--row-uniqueness auto|allow_duplicates|unique_rows |
auto (default), allow_duplicates, or unique_rows; Teradata resolves auto to its default (MULTISET) |
--ddl-fallback |
Emit FALLBACK on generated CREATE TABLE |
--ddl-before-journal on|off |
Emit BEFORE JOURNAL / NO BEFORE JOURNAL |
--ddl-after-journal on|off |
Emit AFTER JOURNAL / NO AFTER JOURNAL |
--ddl-freespace PCT |
Emit FREESPACE = PCT PERCENT (0-100) |
--ddl-checksum MODE |
Emit CHECKSUM = MODE (for example DEFAULT) |
--ddl-datablocksize BYTES |
Emit DATABLOCKSIZE = BYTES |
--param-file FILE |
Informatica .par file — resolves $$Param and $$Variable tokens |
--source-locking none|auto|row_access|share_mode|exclusive_mode |
Optional source-read lock modifier. auto (recommended) emits LOCKING ROW FOR ACCESS when all sources are Teradata; row_access always emits it; share_mode/exclusive_mode emit ACTION REQUIRED review markers. |
--source-ambiguity-policy pick|fail |
Source anchor ambiguity behavior. pick (default) auto-selects best match and emits a note; fail emits ACTION REQUIRED and skips executable DML for ambiguous/unresolved cases. Alias: --sq-ambiguity-policy. |
--readiness-gate |
CI gate (folder mode). Returns non-zero when blockers remain: generation errors, unresolved cross-reference gaps, ambiguous SQ matches, or ACTION REQUIRED/TODO markers. |
--readiness-min-confidence FLOAT |
Optional readiness threshold. Fails gate if average explain-trace confidence is below this value. |
--readiness-max-fallback-rate FLOAT |
Optional readiness threshold. Fails gate if explain-trace fallback ratio exceeds this value. |
--strict-profile |
Production-safe profile: sets --source-ambiguity-policy fail, enables --readiness-gate, sets --window-order-policy fail, and defaults thresholds to min confidence 0.90 / max fallback 0.05 (unless explicitly provided). |
--no-recurse |
Do not descend into sub-directories (folder mode) |
--stdout |
Print to stdout instead of writing files |
--with-workflows / --no-with-workflows |
Emit orchestration scripts per workflow / worklet when the source has workflows (on by default; --no-with-workflows suppresses) |
--target teradata|snowflake|sqlite|bigquery |
Required target SQL dialect. snowflake emits Snowflake-dialect SQL; sqlite emits SQLite-compatible SQL (INSERT only); bigquery emits BigQuery-dialect SQL. |
--qualify-source-schema off|owner|auto |
Schema-qualify source table names in FROM clauses (default: auto) |
--connector-coercion safe|strict |
Connector coercion behavior for datatype mismatches (default: safe) |
--dynamic-lookup-mode advisory|volatile|replay |
Dynamic Lookup lowering strategy (default: advisory) |
--window-order-policy warn|fail |
Behavior when window function has no upstream Sorter (default: warn; --strict-profile sets fail) |
--null-safe-equality off|selective|strict |
NULL-safe equality mode (default: off; selective uses IS NOT DISTINCT FROM in JOINs) |
--dml-temporal-qualifier none|current|nonsequenced |
Temporal qualifier for INSERT/MERGE on temporal tables (default: none) |
--ddl-temp-table-mode none|volatile|gtt |
Temporary-table DDL contract (default: none) |
--ddl-temp-on-commit auto|preserve|delete |
ON COMMIT behavior for temporary tables (default: auto) |
--ddl-temporal-mode none|basic|advanced |
Temporal table clause emission: basic for single-clause, advanced for bitemporal (default: none) |
--infer-secondary-indexes on|off |
Infer USI/NUSI secondary indexes from Lookup/Joiner conditions (default: off) |
--with-graphs |
Generate Graphviz .dot diagrams |
--with-output-graphs |
Generate output-artifact dependency graph under graphs/ |
--render-graphs FMT |
Render .dot files to svg, png, pdf, etc. (requires dot on PATH) |
--with-reusable-components / --with-mapplets |
Generate reusable component validation SQL under reusable_components/ (PowerCenter) |
--scheduler airflow|bteq|ctrlm|kestra |
Target scheduler format for emitted workflows (defaults to airflow) |
--worklet-max-depth N |
Maximum worklet nesting depth for inline expansion (default: 8) |
--post-load-stats |
Append COLLECT STATISTICS steps after session DML in workflow scripts |
--with-tpt |
Emit Teradata Parallel Transporter templates and TPT load hints for partition directives (Teradata target only) |
--bteq-maxerror N |
Emit .MAXERROR N at top of BTEQ scripts (default: 8; 0 to suppress) |
--bteq-error-handling none|standard|strict |
Emit .IF ERRORCODE / .IF ACTIVITYCOUNT guards in BTEQ scripts. standard adds error-code guards; strict also checks zero-row detection (default: none) |
--bteq-recovery none|checkpoint|full |
BTEQ recovery checkpoint mode (default: none) |
--readiness-max-notes INT |
Fail readiness gate when total NOTE advisory marker count exceeds this value |
--scorer-config PATH |
JSON file overriding confidence scorer weights and band thresholds |
--runtime-state-backend none|teradata|snowflake|sqlite|bigquery |
Runtime state persistence backend (default: none). Non-none values must match --target. |
--emit-runtime-package / --no-emit-runtime-package |
Toggle runtime SQL package emission for mappings with runtime semantics (default: auto-detect) |
--runtime-schema SCHEMA |
Schema for runtime control tables (default: etl_meta) |
--variable-default-strategy infer|strict |
Runtime variable defaulting strategy (default: infer) |
--disallow-parallel-mapping-runs |
Release-1 runtime contract guardrail; enabled by default |
--simulate |
Run semantic simulation and attach results to explain payload |
--persistent-lookup-strategy volatile|materialized |
Persistent lookup materialization strategy (default: volatile) |
--cte-emission |
Extract lookup subqueries into WITH clauses |
--cte-nesting-threshold N |
Minimum lookup-join count before CTE extraction activates (default: 3) |
--with-lineage-json |
Generate structured JSON lineage under lineage/ |
--with-openlineage-json |
Generate static OpenLineage JobEvent JSON under lineage/ |
--with-validation |
Generate reconciliation SQL under validation/ |
--with-views |
Emit REPLACE VIEW DDL for intermediate pipeline stages |
--deploy-udfs |
Concatenate referenced UDF procedure SQL into _udf_deployment.sql |
--ddl-partition-interval none|daily|monthly|yearly|DAY|MONTH|YEAR |
Partition interval for date/timestamp columns (default: none) |
--ddl-partition-start DATE |
Lower bound for date partitioning |
--ddl-partition-end DATE |
Upper bound for date partitioning |
--ddl-partition-numeric-step N |
Step size for numeric partitioning |
--ddl-partition-inference off|auto|strict|noted |
Infer partition bounds from mapping filters (default: off) |
--query-band 'key=val;...' |
Append custom Teradata session query-band metadata |
--no-query-band |
Suppress Teradata query-band metadata |
--bigquery-project PROJECT_ID |
BigQuery project ID (--target bigquery) |
--bigquery-dataset DATASET_NAME |
BigQuery dataset name (--target bigquery) |
--bigquery-location LOCATION |
BigQuery location (default: US) |
--shortcut-search-path DIR |
Additional PowerCenter shortcut search directory; repeatable |
--duplicate-policy newest|all |
How to handle duplicate object names across XML files (default: newest) |
--no-sibling-context |
Disable sibling XML scanning in single-file mode |
Single-file scoping: when --input names one file, sibling XML files in the same
directory are still scanned for workflow/connection metadata (richer session and
source-qualifier resolution; disable with --no-sibling-context), but standalone
artifacts — scheduler scripts under wf/, workflow explain payloads, and
connection setup — are emitted only for workflows that orchestrate a mapping
parsed from the named file (plus the worklets they call). An isolated copy of the
same file produces the identical artifact set. These are two different knobs:
--no-sibling-context opts out of resolution context; emission scoping is
always on.
Runtime note: SQL generation is ILM-native only; the --engine switch was removed.
| Transform | SQL output |
|---|---|
| Source Qualifier | FROM clause, WHERE (Source Filter), ORDER BY (sorted ports), SQL override |
| Expression | Derived columns in SELECT; expressions translated to Teradata syntax |
| Lookup | LEFT JOIN with condition from Lookup Condition property; --dynamic-lookup-mode volatile can pre-stage dynamic lookup sources into a mapping-scoped VOLATILE TABLE snapshot, reuse one snapshot across equivalent lookup table/condition contracts, replay insert-only target deltas into that snapshot for later consumers, and otherwise defer refresh until the next target block that reuses the lookup after an intervening target write; --dynamic-lookup-mode replay now extracts replay contracts, stages ordered pre-lookup source rows when Sorter or Source Qualifier order is provable, and can emit narrow executable same-target replay subsets for insert_only, update_else_insert, data_driven_insert_update, and data_driven_insert_update_delete; direct same-target lookup-return passthrough is now supported across those subsets for one dynamic lookup, equivalent multi-lookup contracts, or non-equivalent multi-lookups when each lookup feeds distinct target outputs, including per-port defaults via COALESCE, DD_REJECT rows remain no-op target writes inside the data-driven subsets, direct same-column multi-lookup fan-in is treated as replay-invalid unless precedence is resolved upstream or contracts collapse safely, and replay also classifies non-report-error multiple-match policies, conditional cache updates, target Update Override semantics, explicitly nullable matched-target lookup keys, conflicting explicit source row-treatment modes for data-driven replay, explicit non-Data Driven source row treatment for data-driven replay, explicit contradictory target session row options for data-driven replay including Update as Insert=YES, multiple upstream Update Strategy transforms reaching the lookup for data-driven replay, multiple target-bound Update Strategy transforms for data-driven replay, mismatched upstream versus target-bound Update Strategy semantics for data-driven replay, missing target-bound Update Strategy metadata for data-driven replay, missing NewLookupRow routing to the matched target for data-driven replay, path-aware single-hop FILTER/ROUTER tracing for indirect NewLookupRow routing through intermediate transformations, extra direct NewLookupRow consumers beyond the matched target, competing explicit Sorter or ordered Source Qualifier contracts on the target path, unproven lookup-key uniqueness, and explicit new-value lookup-output mode as invalid instead of inventing semantics |
| Filter | WHERE clause |
| Aggregator | GROUP BY + aggregate functions (SUM, COUNT, MAX, MIN, AVG) |
| Joiner | JOIN clause (Normal, Left Outer, Right Outer, Full Outer) |
| Router | One INSERT per group with first-match guards (NOT(...)) so rows route to the first matching group only |
| Union | UNION ALL subquery |
| Sorter | ORDER BY with per-port ASC/DESC |
| Rank | RANK() OVER (PARTITION BY … ORDER BY …) |
| Update Strategy | INSERT / UPDATE / DELETE / MERGE INTO driven by DD_* constants |
| Sequence Generator | NEXT VALUE FOR <sequence> |
| Normalizer | Concrete UNPIVOT template (single value/group) or UNION ALL template (multi-value/group) |
| SQL Transformation | Informatica→TD translated SQL; ?port? tokens auto-bound from SQL TF metadata/connectors, including unique exact and normalized IN_/OUT_ port-name fallbacks, target-port-name narrowing for undeclared exact-match collisions, and input-capable narrowing or same-binding collapse for normalized collisions, with explicit per-token diagnostics for unresolved/ambiguous bindings |
| Stored Procedure | Runnable CALL proc(in_ports...) skeleton with IN / OUT / RETURN port inventory |
| Transaction Control | BT / ET / ROLLBACK BTEQ statements wrapping the mapping DML, driven by Commit Type property (TC_COMMIT_BEFORE, TC_COMMIT_AFTER, TC_ROLLBACK_BEFORE, TC_ROLLBACK_AFTER) |
Core Informatica expression functions are translated to Teradata equivalents, with deterministic fallback markers for unsupported/UDF-only cases.
Date / time
| Informatica | Teradata |
|---|---|
SYSDATE |
CURRENT_DATE |
SYSTIMESTAMP |
CURRENT_TIMESTAMP |
ADD_TO_DATE(d, 'MM', n) |
d + INTERVAL 'n' MONTH |
DATE_DIFF(d1, d2, 'DD') |
(d1 - d2) DAY(4) |
LAST_DAY(d) |
(d - EXTRACT(DAY FROM d) + 1 + INTERVAL '1' MONTH - INTERVAL '1' DAY) |
NEXT_DAY(d, 'MONDAY') |
TD_DAY_OF_WEEK-based expression |
TRUNC(d, 'MM') |
TRUNC(d, 'MM') |
TO_DATE(s, fmt) |
CAST(s AS DATE FORMAT fmt) |
TO_CHAR(d, fmt) |
TRIM(CAST(d AS VARCHAR(50) FORMAT fmt)) |
String
| Informatica | Teradata |
|---|---|
LTRIM(s) |
TRIM(LEADING FROM s) |
RTRIM(s) |
TRIM(TRAILING FROM s) |
INSTR(s, sub) |
POSITION(sub IN s) |
SUBSTR(s, n, len) |
SUBSTR(s, n, len) |
LENGTH(s) |
CHAR_LENGTH(s) |
LPAD / RPAD |
LPAD / RPAD |
Conditional / null
| Informatica | Teradata |
|---|---|
IIF(cond, t, f) |
CASE WHEN cond THEN t ELSE f END |
ISNULL(x) |
x IS NULL |
NVL(x, y) |
COALESCE(x, y) |
DECODE(x, v1, r1, …) |
CASE … WHEN v1 THEN r1 … END; NULL match values emit IS NULL (not = NULL) |
Date / time (extended)
| Informatica | Teradata |
|---|---|
SET_DATE_PART(d, 'YYYY', yr) |
CAST(LPAD(yr,4,'0') || '-' || EXTRACT(MONTH…) || … AS DATE FORMAT 'YYYY-MM-DD') |
SET_DATE_PART(d, 'MM', mo) |
Analogous CAST+EXTRACT for month replacement |
SET_DATE_PART(d, 'DD', dy) |
Analogous CAST+EXTRACT for day replacement |
SET_DATE_PART(d, 'HH', hr) |
CAST+EXTRACT returning TIMESTAMP(0) |
SET_DATE_PART(d, 'MI', mi) |
CAST+EXTRACT returning TIMESTAMP(0) |
SET_DATE_PART(d, 'SS', ss) |
CAST+EXTRACT returning TIMESTAMP(0) |
Regular expressions
| Informatica | Teradata |
|---|---|
REG_MATCH(s, pat) |
REGEXP_SIMILAR(s, pat, 'i') = 1 |
IS_DATE(s, 'YYYY-MM-DD') |
TRY_CAST(s AS DATE FORMAT 'YYYY-MM-DD') IS NOT NULL — validates actual calendar dates |
IS_DATE(s, 'DD-MON-YYYY') |
TRY_CAST(s AS DATE FORMAT 'DD-mmmYYYY') IS NOT NULL |
REG_REPLACE(s, pat, rep) |
REGEXP_REPLACE(s, pat, rep) |
REG_EXTRACT(s, pat, n) |
REGEXP_SUBSTR(s, pat, 1, n) |
Long-tail parity behavior
| Informatica | Teradata output |
|---|---|
CRC32(x) |
HASHROW(x) plus explicit approximation NOTE comment |
FIRST(x) / LAST(x) |
FIRST_VALUE(x) OVER (...) / LAST_VALUE(x) OVER (...) with sorter/order caveat guidance when needed |
METAPHONE(x) |
Deterministic ACTION REQUIRED marker (NULL expression placeholder) |
COMPRESS(x) / DECOMPRESS(x) |
Deterministic ACTION REQUIRED marker (NULL expression placeholder) |
Unsupported/unknown functions produce deterministic ACTION REQUIRED markers rather than silent pass-through.
Known unsupported families (for example FV/NPER/PMT/PV/RATE) emit dedicated blocker text; other unknown functions emit the generic unknown-function blocker.
Informatica $$Param and $$Variable tokens are resolved in three ways:
- Parameter file (
--param-file prod.par) — values from the.parfile take highest precedence, scoped by[folder.WF:workflow.ST:session]heading. - Declared defaults —
DEFAULTVALUEattributes in the XML. - Unresolved — emitted as
:bteq_variable_name /* $$PARAM | type:string | no default */so the SQL is still executable after substitution.
Database prefix ($$TARGET_DB): if a mapping declares a parameter named $$TARGET_DB (or $$DATABASE, $$DB_NAME, etc.) that parameter is used as the table prefix instead of --database. The generated SQL reads INSERT INTO $$TARGET_DB.MY_TABLE, making the target environment explicit. The parameter header comment marks it with ← db prefix.
When using --with-workflows, workflow-level variables are automatically injected into each session's mapping registry. The generated BTEQ file opens with .SET :var = value; declarations for every workflow variable.
Safety behavior for generated workflow SQL:
- Multiline metadata values are collapsed before being emitted into single-line SQL comments so no bare/unprefixed lines leak into executable SQL.
.SETstring defaults are emitted as quoted BTEQ literals with embedded apostrophes escaped ('->'').- Advisory block-comment payloads are neutralized to prevent embedded
*/from breaking out of generated comments.
| PC version | XML elements | Repository version |
|---|---|---|
| 9 / 10 | TRANSFORMATION + TRANSFORMFIELD |
~165–187 |
| 6 / 7 / 8 | TRANSFORM + FIELD |
~65–110 |
Both schemas are parsed transparently. All other elements (MAPPING, SOURCE, TARGET, CONNECTOR, WORKFLOW, SESSION, etc.) are identical between versions.
INSTANCE node handling: three aliasing patterns are all resolved automatically so connector references always point to the right transformation definition:
| Pattern | XML | Resolution |
|---|---|---|
| Inline (non-reusable) | <INSTANCE NAME="EXP_A" TRANSFORMATION_NAME="EXP_A"/> |
Name matches definition directly — no aliasing needed |
| Reusable, same name | <INSTANCE NAME="EXP_SHARED" TRANSFORMATION_NAME="EXP_SHARED"/> |
Injected from folder-level reusable registry by name |
| Reusable, renamed instance | <INSTANCE NAME="EXP_LOCAL" TRANSFORMATION_NAME="EXP_CANONICAL"/> |
instance_aliases dict maps the local name to the canonical; injected under the local alias so all CONNECTOR references resolve |
Cross-folder SHORTCUT |
<SHORTCUT NAME="LKP_ALIAS" REFOBJECTNAME="LKP_CANONICAL"/> |
shortcut_map alias → canonical → injected under alias |
Each workflow or worklet becomes scheduler-specific output. Select --scheduler bteq for BTEQ .sql, --scheduler ctrlm for Control-M .xml, --scheduler kestra for Kestra flow .yml, or --scheduler airflow for Airflow .py when --with-workflows is enabled:
-- ====================================================================
-- Workflow: WF_NIGHTLY_EVENTS
-- ====================================================================
-- Source file : m_WORKFLOW_COMMANDS_TEST.xml
-- Folder : ETL_NIGHTLY
-- Workflow variables — set before running:
-- .SET :run_dt = NULL /* no default — must be provided */; -- date/time
-- .SET :batch_id = 0; -- integer
.SET :run_dt = /* provide value */;
.SET :batch_id = 0;
-- LOGON hint derived from connection metadata when available
.LOGON <tdpid>/<username>,<password>;
-- Command task: CMD_DECOMPRESS
-- Shell script:
-- gunzip -f /data/events/incoming/*.gz
-- Session: s_m_LOAD_FACT_EVENTS → mapping: m_LOAD_FACT_EVENTS
-- -----------------------------------------------------------------
.OS rm -f /tmp/events_staging/*;
DELETE FROM DW.FACT_EVENTS ALL;
INSERT INTO DW.FACT_EVENTS ( … ) SELECT … FROM STG_EVENTS;
-- Assignment: ASSIGN_BATCH
-- Variable assignment: :BATCH_ID = :BATCH_ID + 1
-- Decision: DEC_CHECK
-- Branch condition: $s_m_LOAD_FACT_EVENTS.Status = 'SUCCEEDED'
.LOGOFF;Non-SQL tasks (Command, FTP, HTTP, Email, Decision, Assignment) are preserved as structured SQL comments rather than being silently dropped. Session pre/post shell commands are emitted as BTEQ .OS commands when they are single-line shell statements. Workflow/task metadata used in those comments is sanitized before rendering.
Task-link conditions are also translated: BTEQ emits deterministic .IF ... THEN .GOTO skip_* guards, Airflow maps failure/unconditional links to trigger_rule while lowering simple Decision status/boolean branches to BranchPythonOperator, and Control-M normalizes $$VAR references in task payloads to %%VAR AutoEdit variables while lowering simple boolean / workflow-variable Decision branches to explicit branch-gate jobs.
Workflow task capability levels by scheduler:
| Task type | BTEQ | Control-M | Airflow |
|---|---|---|---|
| Session | native (inlined SQL) |
native (database job wrapper) |
native (task wrapper) |
| Command / FTP / HTTP / Email | comment-only |
native/emulated |
native/emulated |
| Decision | comment-only |
native for simple boolean / workflow-variable branches via branch-gate jobs; fail-visible for status/custom branch contracts |
native for simple status/boolean branches; fail-visible for unsupported custom branch contracts |
| Assignment | comment-only |
native for simple literal / variable-copy workflow-variable updates via VARIABLE; fail-visible for broader expressions |
native for simple run-scoped workflow-variable updates via XCom-backed PythonOperator; fail-visible for broader expressions |
| Timer | comment-only |
emulated (sleep) |
emulated (sleep) |
| Event-Wait / Event-Raise | comment-only |
explicit manual gate (ACTION REQUIRED) |
explicit manual gate (ACTION REQUIRED) |
| Unknown task type | comment-only |
blocking marker (ACTION REQUIRED) |
blocking marker (ACTION REQUIRED) |
Advanced Teradata table options are feature-flagged so defaults remain backward-compatible:
| DDL Feature | Support | How to enable |
|---|---|---|
FALLBACK |
implemented | --ddl-fallback |
BEFORE/AFTER JOURNAL |
implemented | --ddl-before-journal on|off, --ddl-after-journal on|off |
FREESPACE |
implemented | --ddl-freespace <pct> |
CHECKSUM |
implemented | --ddl-checksum <mode> |
DATABLOCKSIZE |
implemented | --ddl-datablocksize <bytes> |
JSON / PERIOD / ARRAY[n] column types |
implemented | automatic from source datatype metadata |
Temporal-table attributes (VALIDTIME, TRANSACTIONTIME) |
implemented | --ddl-temporal-mode basic|advanced |
| Volatile / GTT temp tables | implemented | --ddl-temp-table-mode volatile|gtt, --ddl-temp-on-commit preserve|delete |
| Secondary index inference (USI/NUSI) | implemented | --infer-secondary-indexes on |
| Error table DDL for DD_REJECT targets | implemented | automatic when DD_REJECT detected |
Example:
uv run python main.py --source-tool powerCenter --target teradata --folder ./exports --output ./sql --database DW --with-ddl \
--ddl-fallback \
--ddl-before-journal off \
--ddl-after-journal on \
--ddl-freespace 5 \
--ddl-checksum DEFAULT \
--ddl-datablocksize 32768# Generate and render to SVG in one step
uv run python main.py --source-tool powerCenter --target teradata --folder ./exports/ --with-graphs --render-graphs svg
# Or render manually
dot -Tsvg output/graphs/mapping_m_LOAD_FACT_SALES.dot -o m_LOAD_FACT_SALES.svg| Graph file | Contents |
|---|---|
mapping_*.dot |
Left-to-right data flow: source → transforms (colour-coded by type) → target |
lineage_*.dot |
Column-level lineage per mapping |
mapplet_*.dot |
Internal mapplet structure |
workflow_*.dot / worklet_*.dot |
Top-to-bottom task dependency with condition-labelled edges |
orch_*.dot |
Per-workflow orchestration artifact graph |
_collection.dot |
High-level overview: workflows → mappings → sources/targets → mapplets |
_everything.dot |
Single graph covering every parsed object across all files |
_dependency_lineage.dot |
Cross-mapping dependency lineage |
_impact_analysis.dot |
Column-level impact analysis |
_output_artifacts.dot |
Output artifact dependency graph (with --with-output-graphs) |
--with-lineage-json writes the stable infa2td lineage shape for direct use in
tests, reports, and migration review. --with-openlineage-json writes a static
OpenLineage JobEvent for tools that ingest OpenLineage metadata. Both files
are rendered from the same typed internal lineage document, so source/target
identity and column-level dependencies stay aligned.
{
"lineage_version": 2,
"mapping_name": "m_LOAD_CUSTOMER",
"datasets": [
{"role": "input", "name": "SRC.CUSTOMER", "identity": {"object_kind": "table"}}
],
"column_lineage": [
{
"target_column": "FULL_NAME",
"source_columns": ["FIRST_NAME", "LAST_NAME"],
"resolution": {"status": "resolved", "confidence": 1.0}
}
],
"diagnostics": []
}{
"schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/JobEvent",
"job": {
"namespace": "infa2td://conversion",
"name": "m_LOAD_CUSTOMER",
"facets": {"infa2td_conversion": {"lineageVersion": 2}}
},
"outputs": [{"name": "DW.CUSTOMER_DIM", "facets": {"columnLineage": {"fields": {}}}}]
}The openlineage optional extra installs openlineage-python for library users
that want to build client event objects. Normal CLI conversion and static JSON
artifact generation do not require that package or any network connection. The
validation optional extra is reserved for future user-authored configuration
validation.
main.py CLI entry point
cli/ CLI orchestration, arg parsing, artifact writing
app.py App dispatcher
args.py Argument parsing and profile application
mapping_pipeline.py Per-mapping conversion pipeline (data payloads only)
single_file.py / folder_mode.py Single-file and folder conversion modes
artifact_writer.py Write all artifact types to disk
sources/ Source parsing and discovery
etl/powercenter/parser.py PowerCenter XML → data model parsing
etl/{ssis,datastage}/parser.py Additional source frontends
_sdk/scanning.py Adapter-backed folder scan dispatcher
etl/powercenter/scanner.py PowerCenter multi-file scan and cross-file registry
etl/powercenter/params.py $$Param / $$Variable resolution and .par file parsing
etl/powercenter/reusable_component.py Reusable component parsing
ir/ Intermediate representation layer
ilm.py Intermediate Logical Model (ILM) IR
runtime_models.py Runtime variable operations and execution plans
replay_models.py Dynamic lookup replay contracts
workflow_control_flow.py Workflow DAG analysis
models/ Domain data classes
source.py Mapping / Source / Target / Workflow data models
plans/ MappingPlan, LookupPlan modules
target_plans.py TargetLoadPlan
compilation_policy.py Frozen compilation policy (replaces kwargs)
findings.py Diagnostic and ReadinessAssessment
decision_trace.py Structured decision trace events
compiler/ Decision-making and planning
capability_planning/planner.py Mapping plan assembly → MappingPlan
workflow_planner/ Workflow task scheduling and dependencies
semantic_capabilities.py Transform semantic capability classification
targets/load_strategy.py Load mode resolution (INSERT/UPDATE/MERGE/etc.)
execution_units.py Execution-unit graph construction
file_sources.py Flat-file source staging metadata
engine/ Runtime orchestration
runtime.py DML generation runtime entrypoint
_workflow_artifacts.py Workflow → scheduler output assembly
expressions/ Shared canonical AST waist and registries
parser.py / ast.py Recursive-descent parser and AST nodes
sources/etl/powercenter/expression/ PowerCenter expression tokenizer and parser
targets/database/<dialect>/expression/ Target-specific expression lowering
emitters/ SQL and artifact generation
dml/orchestrator.py Mapping → INSERT/SELECT / MERGE / UPDATE / DELETE SQL
ddl/bundle.py, ddl/table.py Table definitions → CREATE TABLE SQL
graph/ Graphviz .dot graph generation
replay/ Dynamic lookup replay SQL
runtime/runtime_sql.py Runtime variable persistence SQL
targets/scheduler/{bteq,airflow,ctrlm}/ BTEQ / Airflow / Control-M workflow rendering
targets/database/teradata/tpt.py Teradata Parallel Transporter templates
sql/ SQL AST and translation
translator.py Informatica expression → Teradata SQL
ast/ SQL AST nodes and rendering
analysis/ Post-parse semantic analysis
confidence.py Per-mapping confidence scoring
validation.py Validation and readiness gating
runtime/ Execution planning and semantic detection
runner.py Runtime package execution helpers
target_services/ Target dialect registry and service facade
adapters/ Shared adapter contracts, runtime, and registry
contracts.py Adapter interface contracts
targets/adapter/<family>/ Per-family adapter implementations
({xml,java,http,external_procedure,
application_sq,unstructured_data})
compiler/_state_derivation.py Stateful semantics analysis
models/state.py State data models
simulator/ Row-level workflow state simulation
semantic_taxonomy/ Canonical semantic model, rules, and proof types
compiler/planner_proofs.py Compiler proof bundle builders
procedures/c/ Teradata C UDF source helpers
tools/ Validation, analysis, benchmarking scripts
tests/ Pytest suite
# All tests
uv run pytest tests -n auto
# Lint (Ruff)
uv run ruff check .
# Docs consistency gate
uv run python tools/check_docs_consistency.py --check
# Regenerate parity matrix from catalog
uv run python tools/generate_parity_matrix.py
# One module
uv run pytest tests/test_transforms.py
# One test
uv run pytest tests/test_transforms.py::TestTransformTypes::test_rank_transformTests cover: folder sanity, all transform types, expression translation, PC 8 legacy schema, parameterisation, pipeline comments, workflow SQL, graph generation, gap-fix regressions (DECODE NULL, IS_DATE TRY_CAST, SET_DATE_PART, DD_REJECT, Normalizer UNPIVOT, Stored Procedure CALL, SQL Transformation translation), by-design gap closures, mapplet parameter handling, balanced-paren string-literal handling, SQ source heuristics, handler promotion, converging-loop stability, workflow renderer coverage, critical expression fixes (TO_CHAR, :lkp. nested parens, unmatched paren, multi-aggregator advisory, session MAPPINGNAME fallback, link-condition translation), and INSTANCE name aliasing for reusable transforms.
- Pyright/Pylance project settings are in
pyrightconfig.json. - Recommended interpreter path is
${workspaceFolder}/.venv/bin/python.
# Install tooling dependencies
uv sync --group dev
# Run type checks (application code)
uv run basedpyright# Run benchmark corpora from manifest and print metrics
uv run python tools/benchmark_baseline.py --manifest tools/benchmark_manifest.json
# Strict production profile benchmark + JSON artifact
uv run python tools/benchmark_baseline.py --strict-profile \
--manifest tools/benchmark_manifest.json \
--json-out output/baseline_report.json \
--artifacts-dir output/baseline_artifacts
# Artifact-only collection (always exits 0 even when strict checks fail)
uv run python tools/benchmark_baseline.py --strict-profile --allow-failures \
--manifest tools/benchmark_manifest.json \
--json-out output/baseline_report.json \
--artifacts-dir output/baseline_artifacts
# Enforce release thresholds against a benchmark report
uv run python tools/check_release_thresholds.py \
--report output/baseline_report.json \
--thresholds tools/release_thresholds.json
# Enforce runtime performance budgets against the same report
uv run python tools/check_performance_budget.py \
--report output/baseline_report.json \
--budget tools/performance_budget.jsonDefault corpus manifest includes:
small_daily:tests/test_folder/daily(non-recursive)medium_core_nonrecursive:tests/test_folderwith--no-recurselarge_regression_recursive:tests/test_folderrecursive scan
Each run can persist _summary.txt, _manual_actions.txt, and explain JSON
under --artifacts-dir for CI artifact publishing and later comparison.
uv run python main.py --source-tool powerCenter --target teradata --folder ./exports --output ./out --strict-profileThis profile enforces fail-fast behavior for ambiguity/manual-action markers and applies default confidence/fallback readiness thresholds for CI.
Strict profile expands to:
--source-ambiguity-policy fail(or legacy alias--sq-ambiguity-policy fail)--readiness-gate--window-order-policy fail--readiness-min-confidence 0.90(unless overridden)--readiness-max-fallback-rate 0.05(unless overridden)
Readiness gate failure reasons include:
- generation failures
- unresolved cross-reference gaps
- ambiguous SQ source matches
ACTION REQUIRED/TODOmarkersDYNAMIC LOOKUPruntime-strategy markers- explainability threshold violations
Remediation artifacts:
out/_summary.txtout/_manual_actions.txt(including deduped dynamic-lookup blockers per lookup)out/explain/**/*.explain.json
Current scorecard: docs/ci-gated/CONFIDENCE_SCORING.md
The scorecard is derived from benchmark outputs and validated against
tools/release_thresholds.json using tools/check_release_thresholds.py.
Parity-status updates are tracked in docs/ci-gated/RELEASE_NOTES.md.
# Generate explain artifacts in ILM-native mode
uv run python main.py --source-tool powerCenter --target teradata --folder ./exports/ --output ./out
# Aggregate mismatch categories and top mismatched mappings
uv run python tools/explain_summary.py --explain-root ./out/explain --top 10See ILM Diagnostics for field definitions and triage guidance.
DECODEwithNULLmatch values correctly emitsIS NULL(not= NULL).- Router first-match semantics are preserved: later/default groups include
NOT(...)guards to prevent duplicate routing across groups. IS_DATEwith any of the 10 standard date/timestamp formats usesTRY_CAST, which validates actual calendar correctness (e.g. rejects Feb 31). Formats using 3-letter month abbreviations (MON) fall back toREGEXP_SIMILARwith aNOTE.SET_DATE_PARTfor the 6 most common units (YYYY,MM,DD,HH,MI,SS) generates executable Teradata SQL usingEXTRACT+ string concatenation +CAST. Unsupported units (sub-second) emit a manual-rewrite template.DD_REJECTin Data Driven Update Strategy: aDD_REJECT-only mapping emits a comment-only output (no DML) explaining that all rows are directed to Informatica's bad file. In mixed-code MERGE statements, rows flagged as rejected (_upd_flag = 3) produce no target action, matching PC behaviour — this is now documented inline.
- Normalizer: a concrete
UNPIVOTtemplate (single value per group) orUNION ALLtemplate (multiple values per group) is generated from the field group structure. Replace<source_table_or_subquery>with the actual source name. - SQL Transformation: Informatica→TD function translation is applied to the
SQL Queryproperty and emitted as executable SQL.?port_name?tokens are auto-resolved from SQL TF port expressions/connectors where possible, including unique exact incoming connector-field matches when SQL TF port metadata is incomplete, target-port-name narrowing when multiple undeclared exact-match connectors exist, unique normalizedIN_/OUT_port-name fallbacks for undeclared tokens, and normalized-collision narrowing when input-capable declared ports reduce to one bindable target or one shared binding; unresolved tokens are retained with explicitACTION REQUIRED [SQL TF]diagnostics (including missing/undeclared/ambiguous reasons and connector candidates). Non-Teradata dialect constructs may still require manual adjustment. - Stored Procedure: a runnable
CALL proc(in_port1, in_port2, ...)skeleton is emitted with the actual input, output, and return-value port names. Verify the parameter order and types against the Teradata procedure definition. - Window functions (
CUME,MOVINGAVG,MOVINGSUM,LAG,LEAD, plusFIRST/LASTanalytic lowering): when no upstream Sorter/order context is available, generated SQL now emits deterministicACTION REQUIRED [WINDOW ORDER]markers inside the analyticOVER()clause instead of silently implying unordered semantics. Move the outer SELECT'sORDER BY(generated from the upstream Sorter transformation) into theOVER()clause. - Dynamic Lookup: default mode emits the existing
DYNAMIC LOOKUPadvisory plus a staticLEFT JOIN.--dynamic-lookup-mode volatileupgrades this to an executable mapping-scopedCREATE MULTISET VOLATILE TABLE ... AS (SELECT * FROM <lookup>) WITH DATAsnapshot, reuses that snapshot across target blocks and across equivalent lookup table/condition contracts, replays insert-only target deltas into the snapshot when a later target reuses the same lookup table, and otherwise defers refresh until the next target block that reuses the lookup after an intervening target write.--dynamic-lookup-mode replayis now an explicit uplift mode: it extracts replay-safety contracts, stages ordered pre-lookup source rows into a replay input volatile table when Sorter or Source Qualifier order is provable, and can emit executable same-target replay blocks for the narrowinsert_only,update_else_insert,data_driven_insert_update, anddata_driven_insert_update_deletesubsets where target columns are directly staged and replay keys are simple equality columns. Direct same-target lookup-return passthrough is now supported across those subsets by enriching the replay stage from the current target row on the relevant equality keys, including lookup port defaults via generatedCOALESCEexpressions when Informatica configured them. Equivalent multi-lookup contracts on the same target can share one replay join, and non-equivalent multi-lookups are now supported when each lookup contributes distinct target outputs. Within the data-driven replay subsets,DD_REJECTrows remain no-op target writes through the generated_upd_flagsemantics, andDD_DELETErows now lower into the same-target replayMERGEcontract when the replay subset remains provable. Direct multi-lookup replay fan-in where multiple lookup outputs feed the same target column is now treated as mapping-invalid for replay unless equivalent contracts collapse it or the mapping resolves precedence upstream before the target; those cases emitACTION REQUIRED [DYNAMIC LOOKUP REPLAY INVALID]instead of silently inventing precedence. Replay also treats non-report-error multiple-match policies, conditional cache updates, targetUpdate Overridesemantics, explicitly nullable matched-target lookup keys, conflicting explicit source row-treatment modes for data-driven replay, explicit non-Data Drivensource row treatment for data-driven replay, explicit contradictory target session row options for data-driven replay includingUpdate as Insert=YES, multiple upstreamUpdate Strategytransforms reaching the lookup for data-driven replay, multiple target-boundUpdate Strategytransforms for data-driven replay, mismatched upstream versus target-boundUpdate Strategysemantics for data-driven replay, missing target-boundUpdate Strategymetadata for data-driven replay, missingNewLookupRowrouting to the matched target for data-driven replay, path-aware single-hopFILTER/ROUTERtracing for indirectNewLookupRowrouting through intermediate transformations, extra directNewLookupRowconsumers beyond the matched target, competing explicit Sorter or ordered Source Qualifier contracts on the target path, replay contracts whose lookup keys are not provably unique on the matched target, and explicit new-value lookup-output mode as invalid because the current set-based replay does not model PowerCenter’s error, null-key equality, conditional cache-mutation, custom update SQL, source/target session row-treatment contracts, competing target row-type expressions, target-row routing, downstream output semantics, or ambiguous ordering there. Cross-target lookup-output and broader row-state-heavy replay cases still emitACTION REQUIRED [DYNAMIC LOOKUP REPLAY]blockers instead of silently downgrading to snapshot semantics. Explain artifacts now include a structureddynamic_lookup_replaysummary, and validation/reporting now counts executable replay subset notes separately from replay blockers while classifying replay-invalid mappings separately. Residual gap: PowerCenter cache mutation is still not fully replayed row-by-row inside one set-based DML, ambiguous/cross-target multi-lookup replay remains blocked, and broader row-state semantics are still unsupported.
- Strict-mode note: with
--source-ambiguity-policy fail, ambiguous or low-confidence source-anchor matches (metadata-score diagnostic path) and unresolved connector instances (e.g. missing mapplet/shortcut aliases) emitACTION REQUIREDand skip executable DML for that mapping. - SQL overrides (Source Qualifier, Lookup, and session pre/post SQL blocks) are translated and linted for known non-Teradata patterns. Eleven mechanical rewrites are applied automatically (NVL→COALESCE, SYSDATE→CURRENT_DATE, MINUS→EXCEPT, DECODE→CASE, FROM DUAL removal, ROWNUM→ROW_NUMBER()/QUALIFY, CONNECT BY→WITH RECURSIVE CTE, seq.NEXTVAL/CURRVAL→NEXT/CURRENT VALUE FOR, etc.) with
NOTE [SQL OVERRIDE REWRITE]advisories. No remaining Oracle constructs emitACTION REQUIRED. - Sequence Generator emits
NEXT VALUE FOR <seq>— the sequence object must be created separately with--with-ddl. - BTEQ
.SETsyntax is used for variable substitution; adapt to your scheduler convention if needed (e.g. shell variable expansion or TD Vantage macros). $PM*Integration Service variables (e.g.$PMRootDir) are replaced with:bteq_variableplaceholders — set these in your BTEQ script preamble before execution.- Mapplet
$$parameters:MAPPINGPARAMandMAPPINGVARIABLEelements declared inside aMAPPLETare parsed and merged into the parent mapping's parameter registry. In the generated SQL: if the parameter has a known value (from a.parfile or itsDEFAULTVALUE), the literal is substituted inline; if no value is available, a:bteq_name /* $$ParamName | type:... | no default */placeholder is emitted. Mapplet params are listed in the mapping header comment block with source tag[mapplet]so they are easy to identify.