Skip to content

(feat) deprecate replace full refresh with explicit drop_data pipeline run argument #4017

@rudolfix

Description

@rudolfix

Background
replace resources truncate data and their state each time pipeline runs. historically they were also used to perform full refresh by overwriting the write disposition on pipeline level and then going back to standard resource definition.
The side effect of this method is that participating tables are really switched to replace write disposition and such schema is propagated and stored. Until data is loaded again the pipeline state is in "replace" mode.

There's a way better and cleaner method that does not touch resource definition: using refresh argument. This PR will remove all mentions of the old methods from docs, add additional tests and also restore correct truncation behavior for replace write disposition that were reverted by #3998

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type
No fields configured for issues without a type.

Projects

Status
Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions