Python Library for easily authoring, testing, deploying, and interacting with Flyte tasks, workflows, and launch plans. To understand more about flyte refer to,

• Flyte homepage
• Flyte master repository

Flytekit is designed for minimal footprint, and thus some features must be installed as extras.

This is the lightest-weight SDK install. This installation includes everything you need to interact with Flyte.

Modules include:

1. The full Flyte IDL and an additional model layer for easier extension of the data model.
2. gRPC client for communicating with the platform.
3. Implementations for authoring and extending all Flyte entities (including tasks, workflows, and launch plans).

Tools include:

1. flyte-cli (Command-Line Interface for Interacting with the Flyte Platform)
2. pyflyte (Command-Line tool for easing the registration of Flyte entities)

pip install flytekit

If @spark_task is to be used, one should install the spark plugin.

pip install "flytekit[spark]" for Spark 2.4.x
pip install "flytekit[spark3]" for Spark 3.x

Please note that Spark 2.4 support is deprecated and will be removed in a future release.

If Types.Schema() is to be used for computations involving large dataframes, one should install the schema extension.

pip install "flytekit[schema]"

If @sidecar_task is to be used, one should install the sidecar plugin.

pip install "flytekit[sidecar]"

If @pytorch_task is to be used, one should install the pytorch plugin.

pip install "flytekit[pytorch]"

If @tensorflow_task is to be used, one should install the tensorflow plugin.

pip install flytekit[tensorflow]

To install all or multiple available plugins, one can specify them individually:

pip install "flytekit[sidecar,spark3,schema]"

Or install them with the all or all-spark2.4 or all-spark3 directives which will install all the plugins and a specific Spark version. Please note that all defaults to Spark 3.0 and Spark 2.4 support will be fully removed in a future release.

pip install "flytekit[all]"

$ make
Available recipes:
  setup        Install requirements
  fmt          Format code with black and isort
  lint         Run linters
  test         Run tests
  requirements Compile requirements

virtualenv ~/.virtualenvs/flytekit
source ~/.virtualenvs/flytekit/bin/activate
make setup

We use black and isort to autoformat code. Run the following command to execute the formatters:

source ~/.virtualenvs/flytekit/bin/activate
make fmt

source ~/.virtualenvs/flytekit/bin/activate
make test

Update requirements in requirements.in (or requirements-spark3.in), or update requirements for development in dev-requirements.in. Then, validate, pin and freeze all requirements by running:

source ~/.virtualenvs/flytekit/bin/activate
make requirements

This will re-create the requirements.txt (or requirements-spark3.in) and dev-requirements.txt files which will be used for testing. You will have also have to re-run make setup to update your local environment with the updated requirements.