Tags: potiuk/airflow
Tags
Chart: Fix Airflow 3 task log access with NetworkPolicies (apache#65754)
Update FAB to 4.1.1 The Flask Application Builder have been updated recently to support a number of newer dependencies. This PR is the attempt to migrate FAB to newer version. Fixes: apache#22397
Upgrade FAB to 4.1.1 The Flask Application Builder have been updated recently to support a number of newer dependencies. This PR is the attempt to migrate FAB to newer version. This includes: * update setup.py and setup.cfg upper and lower bounds to account for proper version of dependencies that FAB < 4.0.0 was blocking from upgrade * added typed Flask application retrieval with a custom application fields available for MyPy typing checks. * fix typing to account for typing hints added in multiple upgraded libraries optional values and content of request returned as Mapping * switch to PyJWT 2.* by using non-deprecated "required" claim as list rather than separate fields * add possibiliyt to install providers without constraints so that we could avoid errors on conflicting constraints when upgrade-to-newer-dependencies is used * add pre-commit to check that 2.4+ only get_airflow_app is not used in providers * avoid Bad Request in case the request sent to Flask 2.0 is not JSon content type * switch imports of internal classes to direct packages where classes are available rather than from "airflow.models" to satisfy MyPY * synchronize changes of FAB Security Manager 4.1.1 with our copy of the Security Manager. * add error handling for a few "None" cases detected by MyPY * corrected test cases that were broken by immutability of Flask 2 objects and better escaping done by Flask 2 * updated test cases to account for redirection to "path" rather than full URL by Flask2 Fixes: apache#22397
Apache Airflow v2.0.2 Bug Fixes """"""""" * Bugfix: ``TypeError`` when Serializing & sorting iterable properties of DAGs (apache#15395) * Fix missing ``on_load`` trigger for folder-based plugins (apache#15208) * ``kubernetes cleanup-pods`` subcommand will only clean up Airflow-created Pods (apache#15204) * Fix password masking in CLI action_logging (apache#15143) * Fix url generation for TriggerDagRunOperatorLink (apache#14990) * Restore base lineage backend (apache#14146) * Unable to trigger backfill or manual jobs with Kubernetes executor. (apache#14160) * Bugfix: Task docs are not shown in the Task Instance Detail View (apache#15191) * Bugfix: Fix overriding ``pod_template_file`` in KubernetesExecutor (apache#15197) * Bugfix: resources in ``executor_config`` breaks Graph View in UI (apache#15199) * Fix celery executor bug trying to call len on map (apache#14883) * Fix bug in airflow.stats timing that broke dogstatsd mode (apache#15132) * Avoid scheduler/parser manager deadlock by using non-blocking IO (apache#15112) * Re-introduce ``dagrun.schedule_delay`` metric (apache#15105) * Compare string values, not if strings are the same object in Kube executor(apache#14942) * Pass queue to BaseExecutor.execute_async like in airflow 1.10 (apache#14861) * Scheduler: Remove TIs from starved pools from the critical path. (apache#14476) * Remove extra/needless deprecation warnings from airflow.contrib module (apache#15065) * Fix support for long dag_id and task_id in KubernetesExecutor (apache#14703) * Sort lists, sets and tuples in Serialized DAGs (apache#14909) * Simplify cleaning string passed to origin param (apache#14738) (apache#14905) * Fix error when running tasks with Sentry integration enabled. (apache#13929) * Webserver: Sanitize string passed to origin param (apache#14738) * Fix losing duration < 1 secs in tree (apache#13537) * Pin SQLAlchemy to <1.4 due to breakage of sqlalchemy-utils (apache#14812) * Fix KubernetesExecutor issue with deleted pending pods (apache#14810) * Default to Celery Task model when backend model does not exist (apache#14612) * Bugfix: Plugins endpoint was unauthenticated (apache#14570) * BugFix: fix DAG doc display (especially for TaskFlow DAGs) (apache#14564) * BugFix: TypeError in airflow.kubernetes.pod_launcher's monitor_pod (apache#14513) * Bugfix: Fix wrong output of tags and owners in dag detail API endpoint (apache#14490) * Fix logging error with task error when JSON logging is enabled (apache#14456) * Fix statsd metrics not sending when using daemon mode (apache#14454) * Gracefully handle missing start_date and end_date for DagRun (apache#14452) * BugFix: Serialize max_retry_delay as a timedelta (apache#14436) * Fix crash when user clicks on "Task Instance Details" caused by start_date being None (apache#14416) * BugFix: Fix TaskInstance API call fails if a task is removed from running DAG (apache#14381) * Scheduler should not fail when invalid ``executor_config`` is passed (apache#14323) * Fix bug allowing task instances to survive when dagrun_timeout is exceeded (apache#14321) * Fix bug where DAG timezone was not always shown correctly in UI tooltips (apache#14204) * Use ``Lax`` for ``cookie_samesite`` when empty string is passed (apache#14183) * [AIRFLOW-6076] fix ``dag.cli()`` KeyError (apache#13647) * Fix running child tasks in a subdag after clearing a successful subdag (apache#14776) Improvements """""""""""" * Remove unused JS packages causing false security alerts (apache#15383) * Change default of ``[kubernetes] enable_tcp_keepalive`` for new installs to ``True`` (apache#15338) * Fixed apache#14270: Add error message in OOM situations (apache#15207) * Better compatibility/diagnostics for arbitrary UID in docker image (apache#15162) * Updates 3.6 limits for latest versions of a few libraries (apache#15209) * Adds Blinker dependency which is missing after recent changes (apache#15182) * Remove 'conf' from search_columns in DagRun View (apache#15099) * More proper default value for namespace in K8S cleanup-pods CLI (apache#15060) * Faster default role syncing during webserver start (apache#15017) * Speed up webserver start when there are many DAGs (apache#14993) * Much easier to use and better documented Docker image (apache#14911) * Use ``libyaml`` C library when available. (apache#14577) * Don't create unittest.cfg when not running in unit test mode (apache#14420) * Webserver: Allow Filtering TaskInstances by queued_dttm (apache#14708) * Update Flask-AppBuilder dependency to allow 3.2 (and all 3.x series) (apache#14665) * Remember expanded task groups in browser local storage (apache#14661) * Add plain format output to cli tables (apache#14546) * Make ``airflow dags show`` command display TaskGroups (apache#14269) * Increase maximum size of ``extra`` connection field. (apache#12944) * Speed up clear_task_instances by doing a single sql delete for TaskReschedule (apache#14048) * Add more flexibility with FAB menu links (apache#13903) * Add better description and guidance in case of sqlite version mismatch (apache#14209) Doc only changes """""""""""""""" * Add documentation create/update community providers (apache#15061) * Fix mistake and typos in airflow.utils.timezone docstrings (apache#15180) * Replace new url for Stable Airflow Docs (apache#15169) * Docs: Clarify behavior of delete_worker_pods_on_failure (apache#14958) * Create a documentation package for Docker image (apache#14846) * Multiple minor doc (OpenAPI) fixes (apache#14917) * Replace Graph View Screenshot to show Auto-refresh (apache#14571) Misc/Internal """"""""""""" * Import Connection lazily in hooks to avoid cycles (apache#15361) * Rename last_scheduler_run into last_parsed_time, and ensure it's updated in DB (apache#14581) * Make TaskInstance.pool_slots not nullable with a default of 1 (apache#14406) * Log migrations info in consistent way (apache#14158)
Apache Airflow v2.0.2rc1 Bug Fixes """"""""" * Bugfix: ``TypeError`` when Serializing & sorting iterable properties of DAGs (apache#15395) * Fix missing ``on_load`` trigger for folder-based plugins (apache#15208) * ``kubernetes cleanup-pods`` subcommand will only clean up Airflow-created Pods (apache#15204) * Fix password masking in CLI action_logging (apache#15143) * Fix url generation for TriggerDagRunOperatorLink (apache#14990) * Restore base lineage backend (apache#14146) * Unable to trigger backfill or manual jobs with Kubernetes executor. (apache#14160) * Bugfix: Task docs are not shown in the Task Instance Detail View (apache#15191) * Bugfix: Fix overriding ``pod_template_file`` in KubernetesExecutor (apache#15197) * Bugfix: resources in ``executor_config`` breaks Graph View in UI (apache#15199) * Fix celery executor bug trying to call len on map (apache#14883) * Fix bug in airflow.stats timing that broke dogstatsd mode (apache#15132) * Avoid scheduler/parser manager deadlock by using non-blocking IO (apache#15112) * Re-introduce ``dagrun.schedule_delay`` metric (apache#15105) * Compare string values, not if strings are the same object in Kube executor(apache#14942) * Pass queue to BaseExecutor.execute_async like in airflow 1.10 (apache#14861) * Scheduler: Remove TIs from starved pools from the critical path. (apache#14476) * Remove extra/needless deprecation warnings from airflow.contrib module (apache#15065) * Fix support for long dag_id and task_id in KubernetesExecutor (apache#14703) * Sort lists, sets and tuples in Serialized DAGs (apache#14909) * Simplify cleaning string passed to origin param (apache#14738) (apache#14905) * Fix error when running tasks with Sentry integration enabled. (apache#13929) * Webserver: Sanitize string passed to origin param (apache#14738) * Fix losing duration < 1 secs in tree (apache#13537) * Pin SQLAlchemy to <1.4 due to breakage of sqlalchemy-utils (apache#14812) * Fix KubernetesExecutor issue with deleted pending pods (apache#14810) * Default to Celery Task model when backend model does not exist (apache#14612) * Bugfix: Plugins endpoint was unauthenticated (apache#14570) * BugFix: fix DAG doc display (especially for TaskFlow DAGs) (apache#14564) * BugFix: TypeError in airflow.kubernetes.pod_launcher's monitor_pod (apache#14513) * Bugfix: Fix wrong output of tags and owners in dag detail API endpoint (apache#14490) * Fix logging error with task error when JSON logging is enabled (apache#14456) * Fix statsd metrics not sending when using daemon mode (apache#14454) * Gracefully handle missing start_date and end_date for DagRun (apache#14452) * BugFix: Serialize max_retry_delay as a timedelta (apache#14436) * Fix crash when user clicks on "Task Instance Details" caused by start_date being None (apache#14416) * BugFix: Fix TaskInstance API call fails if a task is removed from running DAG (apache#14381) * Scheduler should not fail when invalid ``executor_config`` is passed (apache#14323) * Fix bug allowing task instances to survive when dagrun_timeout is exceeded (apache#14321) * Fix bug where DAG timezone was not always shown correctly in UI tooltips (apache#14204) * Use ``Lax`` for ``cookie_samesite`` when empty string is passed (apache#14183) * [AIRFLOW-6076] fix ``dag.cli()`` KeyError (apache#13647) * Fix running child tasks in a subdag after clearing a successful subdag (apache#14776) Improvements """""""""""" * Remove unused JS packages causing false security alerts (apache#15383) * Change default of ``[kubernetes] enable_tcp_keepalive`` for new installs to ``True`` (apache#15338) * Fixed apache#14270: Add error message in OOM situations (apache#15207) * Better compatibility/diagnostics for arbitrary UID in docker image (apache#15162) * Updates 3.6 limits for latest versions of a few libraries (apache#15209) * Adds Blinker dependency which is missing after recent changes (apache#15182) * Remove 'conf' from search_columns in DagRun View (apache#15099) * More proper default value for namespace in K8S cleanup-pods CLI (apache#15060) * Faster default role syncing during webserver start (apache#15017) * Speed up webserver start when there are many DAGs (apache#14993) * Much easier to use and better documented Docker image (apache#14911) * Use ``libyaml`` C library when available. (apache#14577) * Don't create unittest.cfg when not running in unit test mode (apache#14420) * Webserver: Allow Filtering TaskInstances by queued_dttm (apache#14708) * Update Flask-AppBuilder dependency to allow 3.2 (and all 3.x series) (apache#14665) * Remember expanded task groups in browser local storage (apache#14661) * Add plain format output to cli tables (apache#14546) * Make ``airflow dags show`` command display TaskGroups (apache#14269) * Increase maximum size of ``extra`` connection field. (apache#12944) * Speed up clear_task_instances by doing a single sql delete for TaskReschedule (apache#14048) * Add more flexibility with FAB menu links (apache#13903) * Add better description and guidance in case of sqlite version mismatch (apache#14209) Doc only changes """""""""""""""" * Add documentation create/update community providers (apache#15061) * Fix mistake and typos in airflow.utils.timezone docstrings (apache#15180) * Replace new url for Stable Airflow Docs (apache#15169) * Docs: Clarify behavior of delete_worker_pods_on_failure (apache#14958) * Create a documentation package for Docker image (apache#14846) * Multiple minor doc (OpenAPI) fixes (apache#14917) * Replace Graph View Screenshot to show Auto-refresh (apache#14571) Misc/Internal """"""""""""" * Import Connection lazily in hooks to avoid cycles (apache#15361) * Rename last_scheduler_run into last_parsed_time, and ensure it's updated in DB (apache#14581) * Make TaskInstance.pool_slots not nullable with a default of 1 (apache#14406) * Log migrations info in consistent way (apache#14158)
Add documentation for Databricks connection (apache#15410) Co-authored-by: Kenten Danas <kentendanas@Kentens-MacBook-Pro.local>
Updating constraints. Build id:754833502 This update in constraints is automatically committed by the CI 'constraints-push' step based on HEAD of 'refs/heads/v2-0-test' in 'apache/airflow' with commit sha a46e809. All tests passed in this build so we determined we can push the updated constraints. See https://github.com/apache/airflow/blob/master/README.md#installing-from-pypi for details.
Updating constraints. Build id:754833502 This update in constraints is automatically committed by the CI 'constraints-push' step based on HEAD of 'refs/heads/v2-0-test' in 'apache/airflow' with commit sha a46e809. All tests passed in this build so we determined we can push the updated constraints. See https://github.com/apache/airflow/blob/master/README.md#installing-from-pypi for details.
PreviousNext