Repository navigation
Windows support for Airflow #10388
Description
Activity
Thanks for opening your first issue here! Be sure to follow the issue template!
have you encountered other problems with running Airflow on Windows? Windows support is highly anticipated by our users, but no one has dealt with this topic intensively yet. Personally, I use MacOS, but I support the idea of adding support for Windows.
Yes. Following the installation manual on the homepage
pip install apache-airflowinstalls theairflowcommand, but it is not a windows executable and windows does not recognize the#! ..../python3.exeshebang.@shachibista Have you tried installing the development version from source? I think this change should fix this problem.
https://github.com/apache/airflow/pull/7808/files#r396126977I think it would be great if someone could invest in Windows support. I believe there are few things - not only the daemon model but also Local Executor uses fork mechanisms which won't be able on Windows, also there might be some problem if you want to use Celery Executor on Windows: https://www.distributedpython.com/2018/08/21/celery-4-windows/ There are few POSIX-compliant packages used as well with might not work on Windows. And automated testing might be a problem since we are using Docker. It looks like quite a big effort to invest..
Reacted by Tomek Urbaszek, Janarthanan, Joe, Daniel Vlahovic and David Beavon@mik-laj No, I haven't tried installing the development version from source. Is there a simple way to do it within windows?
I am afraid not. We know Airflow works in WSL2, but we also know it does not work on Windows. Unless you can convince someone to make it works for Windows, I am afraid it's not going to happen.
you can install the application from local sources by cloning the repository and then running the
pip install -e .command@mik-laj Yes, the development version fixes the issue with the
airflowcommand, at least. But, I cannot start the scheduler due to the aforementioned issues.@potiuk Are you sure there are no fork-like mechanisms for windows? I would really like to get this working at least using Local/SequentialExecutor.
@mik-laj Yes, the development version fixes the issue with the
airflowcommand, at least. But, I cannot start the scheduler due to the aforementioned issues.@potiuk Are you sure there are no fork-like mechanisms for windows? I would really like to get this working at least using Local/SequentialExecutor.
There are different mechanisms - here is the whole discussion about it: https://docs.python.org/3/library/subprocess.html#popen-constructor - but they work differently and Airflow relies on some of the properties of Popen and passing opened file handlers (for example to opened log files). I think there are also a number of other dependencies and possibly hard-coded UNIX path "/" across the code, also Windows is not POSIX-compliant, and I think there are many places where we rely on some tools or binaries which are part of POSIX standard.
I am not saying it's impossible, I just think it's quite an effort and unless you make all the tests pass on windows we can't even start thinking about it. You can start with forking Airflow and trying to make the test work on Windows. Github Actions support Windows runners, so this should be easy to enable.
We are heavily relying on Bash scripts for executing the tests and building Docker Images - and all our tests are run in Ubuntu docker image - however if you want to run it on Windows, it has to be done differently and likey not using Docker images - simply creating a virtualenv and installing everything.
Maybe you can find others who have time and would like to take a look at that together with you ? Simply start a discussion on our devlist and ask for help. I am afraid at this stage for the community, the fact that it works for WSL2 for Windows users is quite enough.
I know there were some changes implemented by @evgenyshulman from DataBand to make Airlfow work in a very limited way on Windows - so maybe rather than run a full set of tests on Windows, just getting a very simple support for Local Executor is possible ? Still Starting from a GitHub actions step installing Airflow on Windows is a good start, we cannot accept the code that is not tested, so being able to test it automatically is a prerequisite.
Happy to review any changes if you come up with tests running on Windows :).
- changed the title
[-]Windows support using Daemoniker[/-][+]Windows support for Airflow[/+]on Aug 25, 2020 Absolutely! I think that might be great thing to add to Airflow. Maybe you would like to open a PR about this (cc: me) with your changes and we can discuss how to approach it.
3 remaining items
We also want probably to add some tests in the CI of ours to run on Windows. GitHub supports Windows runners as well so I am happy to work on incrementally adding more tests and run them in our CI.
Just split the changes needed maybe start with some small few lines part - I could then add the Windows CI tests around it on top. And we could add other PRs afterwards. Generally the smaller PR - the better :)
Any updates on this?
Reacted by Edward GrigoryanIf there is no update here, then there is no update. Everything here happens if somoene does it. Airflow is created by > 2100 users - more of them like you @pforai - users who contribute stuff if they need it.
Maybe you would like to take a lead on it and move it forward? We need users like you who apparenly have both the need and capabiliity (and in this case use Windows) who would like to move things forward and improve compatibility.
One incompatibility with Windows has to do with sqlite-db path absoluteness detection. Specifically, airflow currently requires absolute paths to start with
/(i.e.sqlite:////...) otherwise they're determined to be invalid. On Windows, absolute paths start with a drive letter and not/.Places where I saw checks like this:
- airflow.settings.configure_orm
- airflow.www.app
Yes. There are many more incompatibilities - like forking, and POSIX -only libraries used. but if someone would like to take on the task and implement and test all those, they are absolutely welcome to.
Did quick test on current state. There are 2 Windows unfriendly libs (kerberos and ldap). Once removed manually from pyproject.yaml, I was able to
uv syncsuccessfully. Tried to run basicairflow infocommand with no luck.PS F:\git\airflow> uv run airflow info F:\git\airflow\airflow-core\src\airflow\__init__.py:45: RuntimeWarning: Airflow currently can be run on POSIX-compliant Operating Systems. For development, it is regularly tested on fairly modern Linux Distros and recent versions of macOS. On Windows you can run it via WSL2 (Windows Subsystem for Linux 2) or via Linux Containers. The work to add Windows support is tracked via https://github.com/apache/airflow/issues/10388, but it is not a high priority. warnings.warn( Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "F:\git\airflow\.venv\Scripts\airflow.exe\__main__.py", line 4, in <module> File "F:\git\airflow\airflow-core\src\airflow\__init__.py", line 62, in <module> from airflow import configuration, settings File "F:\git\airflow\airflow-core\src\airflow\configuration.py", line 2285, in <module> secrets_backend_list = initialize_secrets_backends() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\git\airflow\airflow-core\src\airflow\configuration.py", line 2233, in initialize_secrets_backends secrets_backend_cls = import_string(class_name) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\git\airflow\airflow-core\src\airflow\utils\module_loading.py", line 41, in import_string module = import_module(module_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\josef\AppData\Local\Programs\Python\Python312\Lib\importlib\__init__.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\git\airflow\airflow-core\src\airflow\secrets\metastore.py", line 27, in <module> from airflow.utils.session import NEW_SESSION, provide_session File "F:\git\airflow\airflow-core\src\airflow\utils\session.py", line 25, in <module> from airflow import settings File "F:\git\airflow\airflow-core\src\airflow\settings.py", line 39, in <module> from airflow._shared.timezones.timezone import local_timezone, parse_timezone, utc ModuleNotFoundError: No module named 'airflow._shared.timezones' PS F:\git\airflow>@simi workarounds for the libs you mentioned:
Reacted by Josef ŠimánekWith symbolic links used now as part of our "code sharing" - it's further than ever to have "windows" support. I don't think, honestly we will ever want to go there. WSL2 support is there. For example IntelliJ/PyCharm has better and better support for WSL2 from locally run IDE https://blog.jetbrains.com/idea/2025/08/whats-fixed-intellij-idea-2025-2/ - and they are committed to do it eve more, I honestly don't see why we should ever want to make us bound the dev environment to Windows.
Reacted by Josef ŠimánekBTW if we're talking about developing using Airflow (as opposed to Airflow itself), limited development/debugging on Windows is possible using
dag.test().limited development/debugging on Windows is possible using
dag.test().Just a watchout. MAYBE. We neither check, nor verify that, so this is not guaranteed at all. It might work in some versions and might not work in others. It's behaviour under Windows is not specified and it will not block us from making any changes that might break it - simply because there is no point in the development lifecycle where it would be tested.
i have this wsl of rlinux installed on windows...
it doesnt open at all on windows home edition at all for airflow to run....
Description
Currently, the airflow project uses PEP-3143 style daemons to launch tasks (as implemented in https://pypi.org/project/python-daemon/), however this is targeted towards unix daemons. As a result, running airflow on windows requires multiple levels of abstraction each with their own problems. Would it be possible to use something like daemoniker (https://daemoniker.readthedocs.io/en/latest/) to launch tasks? What are the challenges and issues?
In machine learning workflows, with large datasets, it is a huge time-saver if the pipeline tasks can be run on the GPU. WSL 1 does not support GPU passthrough, docker through WSL 2 supports GPU passthrough only with the Insiders build, additionally it has issues with networking when connected to VPN (microsoft/WSL#5068).
Use case / motivation
Natively running airflow without WSL 1/2 or docker on Windows. This is helpful in cases where the company ecosystem is windows-based.
Possible implementation
The daemon module is only used to daemonize the scheduler and webserver. Here's a sample code that runs the scheduler (airflow origin/v1-10-stable) using daemoniker, comments are welcome: