Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make GitHub Actions work on Windows #4324

Merged
merged 62 commits into from
Jul 28, 2022

Conversation

0dminnimda
Copy link
Contributor

@0dminnimda 0dminnimda commented Aug 2, 2021

I added tests for all versions of python to windows and solved some problems to let them work, but some don't:
2.7, 3.4 - since no suitable compiler version is installed
3.9 - inline, details: #3450, #4379
3.5 - one cpp-only test fails, output

I also uncommented the coverage tests and added some code to the .sh to make them work. They work for a decent amount of time, which is the opposite of what was said in the comments.

If anyone wants to try to fix the problems and run 2.7 (and maybe 3.4), then you will need to use this, probably relying on both versions of this answer.

@0dminnimda 0dminnimda changed the title Make ci work on windows Make GitHub Actions work on Windows Aug 2, 2021
.github/workflows/ci.yml Outdated Show resolved Hide resolved
Cython/Compiler/Buffer.py Outdated Show resolved Hide resolved
Tools/ci-run.sh Outdated
# 4127 warns that conditional expression is constant, should be fixed here https://github.com/cython/cython/pull/4317
# (off by default) 5045 warns that Compiler will insert Spectre mitigation for memory load if /Qspectre switch specified
# (off by default) 4820 warns about the code in Python\3.9.6\x64\include ...
WARNARGS="/W4 /wd4711 /wd4127 /wd5045 /wd4820"
Copy link
Contributor Author

@0dminnimda 0dminnimda Aug 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/W4 is recommended and gives an adequate number of warnings, but we can use /Wall + ignore unnecessary warnings (additionally to those /wds)

Tools/ci-run.sh Outdated Show resolved Hide resolved
@0dminnimda
Copy link
Contributor Author

0dminnimda commented Aug 2, 2021

And of course those 2.7 ascii fails are here, they are even represented in main, so not the fault of this PR

.github/workflows/ci.yml Outdated Show resolved Hide resolved
@mattip
Copy link
Contributor

mattip commented Aug 3, 2021

Perhaps break this into a number of PRs: the first one to add one version of python on windows (choose the easiest), then in subsequent PRs add more versions of windows.

@scoder
Copy link
Contributor

scoder commented Dec 21, 2021

Triggering a fresh CI build.

@scoder scoder closed this Dec 21, 2021
@scoder scoder reopened this Dec 21, 2021
@0dminnimda
Copy link
Contributor Author

0dminnimda commented Dec 22, 2021

hmp, the problem seems to be with the path given to disutils?

runTest (__mp_main__.CythonRunTestCase)
compiling (c/cy2) and running closure_name_mangling_T537 ... ['C:\\hostedtoolcache\\windows\\Python\\3.10.0\\x64\\python.exe', '-c', 'import basic_test; basic_test.test()']
mymodule.c

LINK : fatal error LNK1104: cannot open file 'D:\a\cython\cython\TEST_TMP\1\pyximport\pyximport_basic\TEST_TMP\temp.win-amd64-3.10\Release\a\cython\cython\TEST_TMP\1\pyximport\pyximport_basic\TEST_TMP\temp.win-amd64-3.10\Release\a\cython\cython\TEST_TMP\1\pyximport\pyximport_basic\mymodule.cp310-win_amd64.lib'


D:\a\cython\cython\Cython\Compiler\Main.py:346: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: D:\a\cython\cython\TEST_TMP\1\pyximport\pyximport_basic\mymodule.pyx

  tree = Parsing.p_module(s, pxd, full_module_name)

Traceback (most recent call last):

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\_msvccompiler.py", line 497, in link

    self.spawn([self.linker] + ld_args)

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\_msvccompiler.py", line 507, in spawn

    return super().spawn(cmd)

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\ccompiler.py", line 910, in spawn

    spawn(cmd, dry_run=self.dry_run)

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\spawn.py", line 91, in spawn

    raise DistutilsExecError(

distutils.errors.DistutilsExecError: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\link.exe' failed with exit code 1104



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "D:\a\cython\cython\pyximport\pyximport.py", line 214, in load_module

    so_path = build_module(module_name, pyxfilename, pyxbuild_dir,

  File "D:\a\cython\cython\pyximport\pyximport.py", line 186, in build_module

    so_path = pyxbuild.pyx_to_dll(pyxfilename, extension_mod,

  File "D:\a\cython\cython\pyximport\pyxbuild.py", line 103, in pyx_to_dll

    dist.run_commands()

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\dist.py", line 966, in run_commands

    self.run_command(cmd)

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\dist.py", line 985, in run_command

    cmd_obj.run()

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\command\build_ext.py", line 340, in run

    self.build_extensions()

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\command\build_ext.py", line 449, in build_extensions

    self._build_extensions_serial()

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\command\build_ext.py", line 474, in _build_extensions_serial

    self.build_extension(ext)

  File "D:\a\cython\cython\Cython\Distutils\build_ext.py", line 127, in build_extension

    super(build_ext, self).build_extension(ext)

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\command\build_ext.py", line 551, in build_extension

    self.compiler.link_shared_object(

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\ccompiler.py", line 713, in link_shared_object

    self.link(CCompiler.SHARED_OBJECT, objects,

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\_msvccompiler.py", line 499, in link

    raise LinkError(msg)

distutils.errors.LinkError: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Enterprise\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\link.exe' failed with exit code 1104



During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "D:\a\cython\cython\TEST_TMP\1\pyximport\pyximport_basic\basic_test.py", line 8, in test

    import mymodule

  File "D:\a\cython\cython\pyximport\pyximport.py", line 460, in load_module

    module = load_module(fullname, self.path,

  File "D:\a\cython\cython\pyximport\pyximport.py", line 231, in load_module

    raise exc.with_traceback(tb)

  File "D:\a\cython\cython\pyximport\pyximport.py", line 214, in load_module

    so_path = build_module(module_name, pyxfilename, pyxbuild_dir,

  File "D:\a\cython\cython\pyximport\pyximport.py", line 186, in build_module

    so_path = pyxbuild.pyx_to_dll(pyxfilename, extension_mod,

  File "D:\a\cython\cython\pyximport\pyxbuild.py", line 103, in pyx_to_dll

    dist.run_commands()

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\dist.py", line 966, in run_commands

    self.run_command(cmd)

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\dist.py", line 985, in run_command

    cmd_obj.run()

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\command\build_ext.py", line 340, in run

    self.build_extensions()

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\command\build_ext.py", line 449, in build_extensions

    self._build_extensions_serial()

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\command\build_ext.py", line 474, in _build_extensions_serial

    self.build_extension(ext)

  File "D:\a\cython\cython\Cython\Distutils\build_ext.py", line 127, in build_extension

    super(build_ext, self).build_extension(ext)

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\command\build_ext.py", line 551, in build_extension

    self.compiler.link_shared_object(

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\ccompiler.py", line 713, in link_shared_object

    self.link(CCompiler.SHARED_OBJECT, objects,

  File "C:\hostedtoolcache\windows\Python\3.10.0\x64\lib\distutils\_msvccompiler.py", line 499, in link

    raise LinkError(msg)

ImportError: Building module mymodule failed: ["distutils.errors.LinkError: command 'C:\\\\Program Files (x86)\\\\Microsoft Visual Studio\\\\2019\\\\Enterprise\\\\VC\\\\Tools\\\\MSVC\\\\14.29.30133\\\\bin\\\\HostX86\\\\x64\\\\link.exe' failed with exit code 1104\n"]

@0dminnimda
Copy link
Contributor Author

maybe related

@scoder
Copy link
Contributor

scoder commented Dec 22, 2021

maybe related

Ah, yeah, there are problems on Windows with long file names, I heard that before. That's unfortunate. But we can probably do something smarter anyhow, like making sure that we detect when an absolute path is within the working directory and truncate the common prefix, before we create the long temp/working directory path from it. That would probably help in a lot of real world cases.

(I think we already do that somewhere – should make sure that we didn't just forget some places.)

@0dminnimda
Copy link
Contributor Author

(I think we already do that somewhere

@scoder Do you have any suggestions where to look for this? If the problem is really in this then if we'll find the place where we do it we could fix it .. sounds like something captain obvious would say ;) but yeah

@mattip
Copy link
Contributor

mattip commented Feb 6, 2022

ping. I am hitting the "too long" file names in windows runs of CI over at pypy/binary-testing. In NumPy testing, we override the distutils build command to cd into the directory and then compile. Maybe pxyimport could do the same?

@blink1073
Copy link

You can enable long file names in CI

@mattip
Copy link
Contributor

mattip commented Feb 6, 2022

@blink1073 I tried that here but am still getting the same error. According to the error documentation there is a hard limit of 260 characters based on _MAX_PATH in STDLIB.h.

@blink1073
Copy link

hard limit of 260 characters based on _MAX_PATH in STDLIB.h.

Ouch

@scoder
Copy link
Contributor

scoder commented Feb 7, 2022 via email

@0dminnimda
Copy link
Contributor Author

Merging #4630 to test it with the windows CI

@mattip
Copy link
Contributor

mattip commented Jul 18, 2022

#4630 was merged, so this can move forward ever so slightly

@0dminnimda
Copy link
Contributor Author

Well, yeah, although all of the tests are borken at the point, it's probably nice to mergre this, so it would be easier to work on those problems.

@scoder
Copy link
Contributor

scoder commented Jul 28, 2022

Let's get this merged. It's more than what we currently have, and we already know that some tests are failing on appveyor as well, difference being that we look there less often.

Thanks @0dminnimda.

@scoder scoder merged commit 64701f6 into cython:master Jul 28, 2022
@scoder scoder added this to the 3.0 milestone Jul 28, 2022
@0dminnimda 0dminnimda deleted the make-ci-work-on-windows-try-2 branch July 28, 2022 09:11
@scoder
Copy link
Contributor

scoder commented Aug 7, 2022

I created a follow-up ticket regarding the slow build times on Windows. #4963

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants