Skip to content

Conversation

@wRAR
Copy link
Member

@wRAR wRAR commented Jan 2, 2025

  • Wait until Twisted supports 3.14 asyncio loop changes, is released and is supported by Scrapy
  • Update tests.test_squeues.PickleFifoDiskQueueTest.test_non_pickable_object()
  • Update tests/CrawlerProcess/asyncio_enabled_reactor.py
  • Update tests/test_utils_asyncio.py::TestAsyncio::test_install_asyncio_reactor
  • Check tests/test_command_check.py::TestCheckCommand::test_printSummary_with_unsuccessful_test_result*
  • Check tests/test_feedexport.py failures on Linux
  • Check tests/test_feedexport.py failures on Windows
  • Update tests/test_pipeline_files.py::TestFilesPipeline::test_file_path
  • Improve the tests/test_feedexport.py::TestFeedExporterSignals fix for Windows
  • Fix/work around deprecated asyncio loop policies
  • Check the pytest-cov message
  • Investigate macOS Python 3.10 tests always fail #5306
  • python-zstandard
  • uvloop

@wRAR wRAR force-pushed the py3.14 branch 2 times, most recently from cba8eb1 to 031770c Compare January 2, 2025 13:54
@wRAR
Copy link
Member Author

wRAR commented Jan 2, 2025

mitmproxy fails to start because of indygreg/python-zstandard#238 , no idea yet about other failures.

@wRAR wRAR marked this pull request as draft January 2, 2025 14:31
@wRAR
Copy link
Member Author

wRAR commented Jan 2, 2025

Pickle exception failures are something about the changes in python/cpython#122214 intentional changes of python/cpython#122771, we need to adjust the expectations.

The asyncio loop failure in tests/CrawlerProcess/asyncio_enabled_reactor.py is documented and handled in scrapy.utils.reactor.set_asyncio_event_loop() so the failing code may need to be modified in a similar way (or even call that function, depending on the intention of the reactor installation code in the script).

uvloop failures are MagicStack/uvloop#637.

Not sure if there is some problem with pytest-cov ("Exception: ConfigError("Couldn't read '/home/runner/work/scrapy/scrapy/:' as a config file")"), note that we pin an older version of it.

Looks like there are no new warnings.

@wRAR
Copy link
Member Author

wRAR commented Jan 2, 2025

After looking at the OS X job I see that, surprisingly, #5306 reappeared. I wonder if it's some kind of an lxml issue.

There is also an OS X-only deprecation warning in twisted.internet.process._BaseProcess._fork() about forks in multi-threaded processes, but it turns out to be 3.12+, not 3.14+, see the note in https://docs.python.org/3.13/library/os.html#os.fork. Not sure if it's possible to address it and also not sure if it's specific to scrapy.utils.testproc.ProcessTest or happens in normal Scrapy usage too.

@wRAR wRAR closed this Jan 15, 2025
@wRAR wRAR reopened this Jan 15, 2025
@codecov
Copy link

codecov bot commented Jan 15, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.65%. Comparing base (d0dabbc) to head (7367f6a).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6604      +/-   ##
==========================================
- Coverage   90.67%   90.65%   -0.02%     
==========================================
  Files         162      162              
  Lines       12125    12125              
  Branches     1566     1566              
==========================================
- Hits        10994    10992       -2     
  Misses        829      829              
- Partials      302      304       +2     

see 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@wRAR wRAR changed the title Add Python 3.14 (alpha3) to CI. Add Python 3.14 (alpha4) to CI. Mar 7, 2025
@wRAR wRAR changed the title Add Python 3.14 (alpha4) to CI. Add Python 3.14 (beta2) to CI. Jun 1, 2025
@wRAR
Copy link
Member Author

wRAR commented Jul 17, 2025

This is now additionally blocked on twisted/twisted#12426 (before switching to pytest-twisted the loop was created when install_reactor() was called in conftest.py)

@wRAR wRAR changed the title Add Python 3.14 (beta2) to CI. Add Python 3.14 to CI. Oct 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant