…nd preprocessing (#1422)

* sampling zoo integration first stage

* add extension contract and mirrored tests scaffold

* make remote pipeline config parsing safe and typed

* extract typed repository query rules from operation repository

* extract typed repository

* extract pure assumption and preset rules from api shell

* extract pure input data rules from api data adapter

* extract pure recommendation rules from input analyser

* extract typed fit and composer planning rules

* extract typed api params validation and normalization rules

* extract typed assumption handler rules and either-based fit result

* extract pure api params repository defaulting rules

* extract pure api params

* extract cache and tuner setup rules from api composer

* extract pure builder parameter merge rules

* extract preprocessing source and merge rule

* integrate extension manifest discovery into operation queries

* extract pipeline operation split rules and fix fluent repository setup

* add typed extension parameter resolution and schema defaults

* extract pipeline preprocess and postprocess rules

* extract pipeline node parameter normalization rules

* extract operation parameter normalization and change tracking rules

* `Refactor OOP shells to typed pure-core rules and add first mirrored tests slice`

* chore: add setuptools pkg_resources libs

* fix: change repo kinds enum values to lowercase

* fix: change the order of using best preset name in presets parsing

* fix: add proper chained exception thread in assumptions fit stage

* fix: add inheritance for fake test pipeline from actual pipeline

* fix: add Right monad in extension strategy params build method

* fix: add OperationParameters support for FP extraction

* fix: update validation checks to use is_left, is_right methods for monads

* Automated autopep8 fixes

---------

Co-authored-by: v1docq <revine@inbox.ru>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* TensorData skeleton

* rerfactoring2.0 and add embedder

* sampling zoo integration first stage

* add extension contract and mirrored tests scaffold

* make remote pipeline config parsing safe and typed

* extract typed repository query rules from operation repository

* extract typed repository

* extract pure assumption and preset rules from api shell

* extract pure input data rules from api data adapter

* extract pure recommendation rules from input analyser

* extract typed fit and composer planning rules

* extract typed api params validation and normalization rules

* extract typed assumption handler rules and either-based fit result

* extract pure api params repository defaulting rules

* extract pure api params

* extract cache and tuner setup rules from api composer

* extract pure builder parameter merge rules

* extract preprocessing source and merge rule

* integrate extension manifest discovery into operation queries

* added backend

* extract pipeline operation split rules and fix fluent repository setup

* add typed extension parameter resolution and schema defaults

* extract pipeline preprocess and postprocess rules

* extract pipeline node parameter normalization rules

* extract operation parameter normalization and change tracking rules

* `Refactor OOP shells to typed pure-core rules and add first mirrored tests slice`

* fp principles added

* some changes

* added examples

* chore: rm fedot/core/data/some.py

* chore: add setuptools pkg_resources libs

* fix: change repo kinds enum values to lowercase

* fix: change the order of using best preset name in presets parsing

* fix: add proper chained exception thread in assumptions fit stage

* fix: add inheritance for fake test pipeline from actual pipeline

* fix: add Right monad in extension strategy params build method

* fix: add OperationParameters support for FP extraction

* fix: update validation checks to use is_left, is_right methods for monads

* Automated autopep8 fixes

* fixed bugs

* some changes

* added docstrings

* imports refactored

* merge with codex/arch_refactoring

---------

Co-authored-by: Romankk03 <126852895+Romankk03@users.noreply.github.com>
Co-authored-by: v1docq <revine@inbox.ru>
Co-authored-by: Georgii Lopatenko <81328772+Lopa10ko@users.noreply.github.com>
Co-authored-by: Lopa10ko <justforprojects@mail.ru>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…y preprocessing through steps"

…atibility bridges (#1428)

* stabilize tensordata taxonomy and normalization rules

* extract tensordata creator and backend planning rules

* add typed tensordata creation request and target hint rules

* extract tensordata file load plan for csv and tsv

* extract arff target resolution rules from data reader

* normalize load data spec defaults and orientation rules

* add tensor canonical compatibility rules for legacy data types

* add input data descriptor for tensor bridge preparation

* add initial inputdata to tensordata bridge adapter

* add bridge-aware tensordata preprocessing entrypoint

* add explicit tensordata runtime entrypoints to pipeline

* add tensordata inference entrypoints to fedot facade

* add predefined tensordata fit path to fedot facade

* Tensordata_intermediatePR_300326

* add tensordata tune and compatibility state handling to fedot facade.add tensordata metrics and explain facade entrypoints

* add explicit tensordata definition path to api data processor

* refresh pr description for tensordata slice

* use explicit tensordata runtime path for facade tuning

* extract industrial data and task normalization rules

* extract industrial strategy dispatch and runtime plans

* extract industrial main predict and metric rules

* extract industrial main save load and explain rules

* finish industrial main shell refactor with fit predict and finetune plans

* extract industrial sampling and finetune planning rules

* finalize industrial config convergence and prepare final tensor pr

* final RP doc

* update git_ignore and codex

* chore: update .gitignore to include sampling_zoo, codex, and results directories

* chore: remove outdated benchmark result metadata and error logs for multiple runs

* chore: update requirements.txt to specify setuptools version and pin pymonad to 2.4.0

* refactor: normalize assumption data type handling and clean up imports in various modules

* refactor: improve industrial load plan logic to handle multiple saved pipelines more robustly

* refactor: enhance tensor data handling in tests by introducing minimal tensor data functions and updating test cases

* Automated autopep8 fixes

---------

Co-authored-by: v1docq <revine@inbox.ru>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* added DTOs

* refactor(TensorData): added obligatory preprocessing through steps

* refactor(backend): call Backend as singleton

* refactor(preprocessing): added imputation for TensorData and refactored service

* refactor(preprocessing): added idx updating during preprocessing

* feat(preprocessing): added new scaling methods and fixed bugs

* chore(preprocessing): added new directories for working modules

* feat(preprocessing): added new methods for multi-channel data preprocessing

* feat(preprocessing): added new methods for ts preprocessing. Refactored ts preprocessing pipeline. Fixed bugs with ts in Tensor Data.

* refactor(preprocessing): added obligatory service

* style(preprocessing): added dockstrings

* style(preprocessing): renamed classes

* style(preprocessing): autopep8

---------

Co-authored-by: Romankk03 <126852895+Romankk03@users.noreply.github.com>

* refactor(TensorData):  added new vision of TensorData

* fix(ci): updated cupy/cudf imports

* style(data): new order in core/data

* feature(TensorData): added init, more strict params checking and without_target mode.

* style(TensorData): move some fun from TensorDataCreator

* feature(TensorData): added more expanded memory usage method

* style(test): added pytest.mark for tests

* fix(preprocessing): fixed bug with idx mapping and added new possibility to set features names for optional preprocessing in parameters.

* style(common): pep8 refactored

* refactor(ci): changed path in docs

* chore(review): fixed some comments

* fix(review): fixed pep8, fixed bugs from review and refactored DataReader

---------

Co-authored-by: Romankk03 <126852895+Romankk03@users.noreply.github.com>

* feat: divide chunking and subset sampling strategies

* test: add all sampling zoo strategies tests

* refactor: improve and simplify sampling integration

* refactor: make pure sampling functions

* feat: add tabicl model

* benchmark: add amlb example with foundational models

* feat: add holdout-based chunked ensembles with routed sampling

* docs: add sampling stage uml diagrams

* refactor: minor improvements

* refactor: make thin Fedot fit

* refactor: add PipelineInfo class, handle predict processing, add configs logs

* refactor: apply strategy pattern to sampling executor

* feature(cache): added hasher

* refactor(cache): added common class registry for Hasher and DataReader

* style(cache): refactored predicates, added new tests.

* style(cache): reorder tools.py

* feature(cache) added saver. Added common predicates for registries

* style(cache): renaming and fixed cache dir creation

* feature(cache): added cache loader. Added __eq__ for TD

* feature(cache): added caching for obligatory preprocessing

* feature(cache): added tracer for TensorData

* feature(cache): added caching to optional preprocessing

* feature(preprocessing): added transform methods for preprocessing services

* feature(cache): added optional caching

* feature(cache): added cache cleaner

* fix(cache): deleted version for TD caching

* style(cache): registered functions for saver and loader directly

* fix(cache): added a single _connect() method for sqlite

* style(cache): added dockstrings

* style(cache): flake8

* style(cache): autopep8

* feature(cache): added profiler and profiled caching

---------

Co-authored-by: Romankk03 <126852895+Romankk03@users.noreply.github.com>