python -m test test_unitttest -R3:3 fails

@vstinner

For the last couple months, it seems buildbots have been failing -m test test_unittest -R3:3, then re-trying the 3 failed test cases and passing on the re-try. Look at https://buildbot.python.org/all/#/builders/123 for example. The tests consistently pass the first warmup run, then fail on the second warmup run.

Example stdout from running -m test test_unittest -R3:3 -v:

======================================================================
FAIL: test_loadTestsFromName__module_not_loaded (test.test_unittest.test_loader.Test_TestLoader.test_loadTestsFromName__module_not_loaded)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 594, in test_loadTestsFromName__module_not_loaded
    self.assertEqual(list(suite), [])
AssertionError: Lists differ: [<unittest.loader._FailedTest testMethod=dummy>] != []

First list contains 1 additional elements.
First extra element 0:
<unittest.loader._FailedTest testMethod=dummy>

- [<unittest.loader._FailedTest testMethod=dummy>]
+ []

======================================================================
FAIL: test_loadTestsFromNames__module_not_loaded (test.test_unittest.test_loader.Test_TestLoader.test_loadTestsFromNames__module_not_loaded)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 1019, in test_loadTestsFromNames__module_not_loaded
    self.assertEqual(list(suite), [unittest.TestSuite()])
AssertionError: Lists differ: [<uni[15 chars]tSuite tests=[<unittest.loader._FailedTest testMethod=dummy>]>] != [<uni[15 chars]tSuite tests=[]>]

First differing element 0:
<unit[13 chars]stSuite tests=[<unittest.loader._FailedTest testMethod=dummy>]>
<unit[13 chars]stSuite tests=[]>

- [<unittest.suite.TestSuite tests=[<unittest.loader._FailedTest testMethod=dummy>]>]
+ [<unittest.suite.TestSuite tests=[]>]

======================================================================
FAIL: test_loadTestsFromNames__unknown_attr_name (test.test_unittest.test_loader.Test_TestLoader.test_loadTestsFromNames__unknown_attr_name)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 715, in test_loadTestsFromNames__unknown_attr_name
    error, test = self.check_deferred_error(loader, list(suite)[0])
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\sween\Source\Repos\cpython2\cpython\Lib\test\test_unittest\test_loader.py", line 620, in check_deferred_error
    self.assertEqual(1, len(loader.errors))
AssertionError: 1 != 2

----------------------------------------------------------------------
Ran 1001 tests in 8.861s

FAILED (failures=3, skipped=55)
test_unittest failed (3 failures)

== Tests result: FAILURE ==

1 test failed:
    test_unittest

Total duration: 18.3 sec
Tests result: FAILURE

I bisected to here:

commit c735d54
Author: Victor Stinner vstinner@python.org
Date: Tue Jun 21 10:27:59 2022 +0200

gh-93839: Move Lib/unttest/test/ to Lib/test/test_unittest/ (#94043)

* Move Lib/unittest/test/ to Lib/test/test_unittest/
* Remove Lib/test/test_unittest.py
* Replace unittest.test with test.test_unittest
* Remove unittest.load_tests()
* Rewrite unittest __init__.py and __main__.py
* Update build system, CODEOWNERS, and wasm_assets.py

cc @vstinner

For convenience, here are those tests:

cpython/Lib/test/test_unittest/test_loader.py

Lines 580 to 600 in 56d9cf7

    
           # "The specifier can refer to modules and packages which have not been 
        
           # imported; they will be imported as a side-effect" 
        
           def test_loadTestsFromName__module_not_loaded(self): 
        
               # We're going to try to load this module as a side-effect, so it 
        
               # better not be loaded before we try. 
        
               # 
        
               module_name = 'test.test_unittest.dummy' 
        
               sys.modules.pop(module_name, None) 
        
               loader = unittest.TestLoader() 
        
               try: 
        
                   suite = loader.loadTestsFromName(module_name) 
        
                   self.assertIsInstance(suite, loader.suiteClass) 
        
                   self.assertEqual(list(suite), []) 
        
                   # module should now be loaded, thanks to loadTestsFromName() 
        
                   self.assertIn(module_name, sys.modules) 
        
               finally: 
        
                   if module_name in sys.modules: 
        
                       del sys.modules[module_name]

cpython/Lib/test/test_unittest/test_loader.py

Lines 1005 to 1025 in 56d9cf7

    
           # "The specifier can refer to modules and packages which have not been 
        
           # imported; they will be imported as a side-effect" 
        
           def test_loadTestsFromNames__module_not_loaded(self): 
        
               # We're going to try to load this module as a side-effect, so it 
        
               # better not be loaded before we try. 
        
               # 
        
               module_name = 'test.test_unittest.dummy' 
        
               sys.modules.pop(module_name, None) 
        
               loader = unittest.TestLoader() 
        
               try: 
        
                   suite = loader.loadTestsFromNames([module_name]) 
        
                   self.assertIsInstance(suite, loader.suiteClass) 
        
                   self.assertEqual(list(suite), [unittest.TestSuite()]) 
        
                   # module should now be loaded, thanks to loadTestsFromName() 
        
                   self.assertIn(module_name, sys.modules) 
        
               finally: 
        
                   if module_name in sys.modules: 
        
                       del sys.modules[module_name]

cpython/Lib/test/test_unittest/test_loader.py

Lines 704 to 720 in 56d9cf7

    
           # "The specifier name is a ``dotted name'' that may resolve either to 
        
           # a module, a test case class, a TestSuite instance, a test method 
        
           # within a test case class, or a callable object which returns a 
        
           # TestCase or TestSuite instance." 
        
           # 
        
           # What happens when the module can be found, but not the attribute? 
        
           def test_loadTestsFromNames__unknown_attr_name(self): 
        
               loader = unittest.TestLoader() 
        
               suite = loader.loadTestsFromNames( 
        
                   ['unittest.loader.sdasfasfasdf', 'test.test_unittest.dummy']) 
        
               error, test = self.check_deferred_error(loader, list(suite)[0]) 
        
               expected = "module 'unittest.loader' has no attribute 'sdasfasfasdf'" 
        
               self.assertIn( 
        
                   expected, error, 
        
                   'missing error string in %r' % error) 
        
               self.assertRaisesRegex(AttributeError, expected, test.sdasfasfasdf)

It appears to be something a bad interaction between those three test methods and test.test_unittest.testmock.testpatch.PatchTest.test_dotted_but_module_not_loaded:

cpython/Lib/test/test_unittest/testmock/testpatch.py

Lines 1916 to 1932 in 56d9cf7

    
           def test_dotted_but_module_not_loaded(self): 
        
               # This exercises the AttributeError branch of _dot_lookup. 
        
               # make sure it's there 
        
               import test.test_unittest.testmock.support 
        
               # now make sure it's not: 
        
               with patch.dict('sys.modules'): 
        
                   del sys.modules['test.test_unittest.testmock.support'] 
        
                   del sys.modules['test.test_unittest.testmock'] 
        
                   del sys.modules['test.test_unittest'] 
        
                   del sys.modules['unittest'] 
        
                   # now make sure we can patch based on a dotted path: 
        
                   @patch('test.test_unittest.testmock.support.X') 
        
                   def test(mock): 
        
                       pass 
        
                   test()

Here's the bare essentials of what is failing:

import sys
from unittest.mock import patch

if 0: # change to 1 to see "tu.dummy" fail.

    # Inline test_dotted_but_module_not_loaded
    # This exercises the AttributeError branch of _dot_lookup.

    # make sure it's there
    import test.test_unittest.testmock.support
    # now make sure it's not:
    with patch.dict('sys.modules'):
        del sys.modules['test.test_unittest.testmock.support']
        del sys.modules['test.test_unittest.testmock']
        del sys.modules['test.test_unittest']
        del sys.modules['unittest']

        # now make sure we can patch based on a dotted path:
        @patch('test.test_unittest.testmock.support.X')
        def test(mock):
            print(mock)
        test()


# The part that fails in test_loadTestsFromName__module_not_loaded
module_name = 'test.test_unittest.dummy'
sys.modules.pop(module_name, None)
tu = __import__(module_name).test_unittest
tu.dummy

So essentially, I believe test_dotted_but_module_not_loaded needs to do a better job cleaning up after itself.

@ezio-melotti

@ezio-melotti or @gpshead or @cjw296, any suggestions on how to make test_dotted_but_module_not_loaded play more nicely? I don't know all that much about unittest.mock.

So essentially, I believe test_dotted_but_module_not_loaded needs to do a better job cleaning up after itself.

Maybe save/restore sys.modules?

gh-96624: Fix test_dotted_but_module_not_loaded in testpatch.py (GH-9…

I believe this is fixed now.

Oh nice, thanks for the fix!

Navigation Menu

Repository navigation

python -m test test_unitttest -R3:3 fails #96624

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	# "The specifier can refer to modules and packages which have not been
	# imported; they will be imported as a side-effect"
	def test_loadTestsFromName__module_not_loaded(self):
	# We're going to try to load this module as a side-effect, so it
	# better not be loaded before we try.
	#
	module_name = 'test.test_unittest.dummy'
	sys.modules.pop(module_name, None)

	loader = unittest.TestLoader()
	try:
	suite = loader.loadTestsFromName(module_name)

	self.assertIsInstance(suite, loader.suiteClass)
	self.assertEqual(list(suite), [])

	# module should now be loaded, thanks to loadTestsFromName()
	self.assertIn(module_name, sys.modules)
	finally:
	if module_name in sys.modules:
	del sys.modules[module_name]

	# "The specifier can refer to modules and packages which have not been
	# imported; they will be imported as a side-effect"
	def test_loadTestsFromNames__module_not_loaded(self):
	# We're going to try to load this module as a side-effect, so it
	# better not be loaded before we try.
	#
	module_name = 'test.test_unittest.dummy'
	sys.modules.pop(module_name, None)

	loader = unittest.TestLoader()
	try:
	suite = loader.loadTestsFromNames([module_name])

	self.assertIsInstance(suite, loader.suiteClass)
	self.assertEqual(list(suite), [unittest.TestSuite()])

	# module should now be loaded, thanks to loadTestsFromName()
	self.assertIn(module_name, sys.modules)
	finally:
	if module_name in sys.modules:
	del sys.modules[module_name]

	# "The specifier name is a ``dotted name'' that may resolve either to
	# a module, a test case class, a TestSuite instance, a test method
	# within a test case class, or a callable object which returns a
	# TestCase or TestSuite instance."
	#
	# What happens when the module can be found, but not the attribute?
	def test_loadTestsFromNames__unknown_attr_name(self):
	loader = unittest.TestLoader()

	suite = loader.loadTestsFromNames(
	['unittest.loader.sdasfasfasdf', 'test.test_unittest.dummy'])
	error, test = self.check_deferred_error(loader, list(suite)[0])
	expected = "module 'unittest.loader' has no attribute 'sdasfasfasdf'"
	self.assertIn(
	expected, error,
	'missing error string in %r' % error)
	self.assertRaisesRegex(AttributeError, expected, test.sdasfasfasdf)

	def test_dotted_but_module_not_loaded(self):
	# This exercises the AttributeError branch of _dot_lookup.

	# make sure it's there
	import test.test_unittest.testmock.support
	# now make sure it's not:
	with patch.dict('sys.modules'):
	del sys.modules['test.test_unittest.testmock.support']
	del sys.modules['test.test_unittest.testmock']
	del sys.modules['test.test_unittest']
	del sys.modules['unittest']

	# now make sure we can patch based on a dotted path:
	@patch('test.test_unittest.testmock.support.X')
	def test(mock):
	pass
	test()

python -m test test_unitttest -R3:3 fails #96624

Description

Activity

sweeneyde commented on Sep 6, 2022

sweeneyde commented on Sep 7, 2022

sweeneyde commented on Sep 8, 2022

sweeneyde commented on Sep 8, 2022

vstinner commented on Sep 8, 2022

sweeneyde commented on Sep 9, 2022

vstinner commented on Sep 12, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions