Python 3.12+ breaks backwards compatibility for logging QueueHandler with queue.SimpleQueue

Bug report

Bug description:

I think this is related to #119819

import logging.config
from queue import SimpleQueue

q = SimpleQueue()

config = {
    'version': 1,
    'handlers': {
        'sink': {
            'class': 'logging.handlers.QueueHandler',
            'queue': q,
        },
    },
    'root': {
        'handlers': ['sink'],
    },
}
logging.config.dictConfig(config)

ValueError: Unable to configure handler 'sink'

SimpleQueue does not implement the full Queue interfaces thus both isinstance(obj, queue.Queue) and the queue interface check fails.
Since this has been working on 3.8 - <3.12 I think the queue interface check is checking for methods that are not used at all and should be adjusted accordingly.

cpython/Lib/logging/config.py

Lines 500 to 525 in 2518918

    
           def _is_queue_like_object(obj): 
        
               """Check that *obj* implements the Queue API.""" 
        
               if isinstance(obj, queue.Queue): 
        
                   return True 
        
               # defer importing multiprocessing as much as possible 
        
               from multiprocessing.queues import Queue as MPQueue 
        
               if isinstance(obj, MPQueue): 
        
                   return True 
        
               # Depending on the multiprocessing start context, we cannot create 
        
               # a multiprocessing.managers.BaseManager instance 'mm' to get the 
        
               # runtime type of mm.Queue() or mm.JoinableQueue() (see gh-119819). 
        
               # 
        
               # Since we only need an object implementing the Queue API, we only 
        
               # do a protocol check, but we do not use typing.runtime_checkable() 
        
               # and typing.Protocol to reduce import time (see gh-121723). 
        
               # 
        
               # Ideally, we would have wanted to simply use strict type checking 
        
               # instead of a protocol-based type checking since the latter does 
        
               # not check the method signatures. 
        
               queue_interface = [ 
        
                   'empty', 'full', 'get', 'get_nowait', 
        
                   'put', 'put_nowait', 'join', 'qsize', 
        
                   'task_done', 
        
               ] 
        
               return all(callable(getattr(obj, method, None)) 
        
                          for method in queue_interface)

Tested with 3.12.6

CPython versions tested on:

3.12

Operating systems tested on:

Windows

Linked PRs

Except if I miss something, SimpleQueue has never had a put_nowait method, the one used in the QueueHandler and QueueListener classes (Just see here) . So using this class in the logging QueueHandler always failed.

As I've written this has been working on 3.8 - 3.11. I'm not sure exactly when it broke.
SimpleQueue actually implements put_nowait.

I skimmed over the file you linked and it seems the only required interface methods are put_nowait and get.
The interface check should reflect that.
Additionally get has to raise queue.Empty if the queue is empty but there is no way to check that ...

My apologies for my mistaken: I was thinking you talked about multiprocessing.SimpleQueue not about queue.SimpleQueue. So first of all, I suggess to modify title issue (prefixing SimpleQueuewith the queue module).

And then, I agree with you the queue_interface list of the _is_queue_like_object method is too restrictive. At least join and task_done methods must be removed. full and emptymethods (or exceptions) have to exam too.

Perhars it will be nice to ask for a python core dev advice.

@picnixz

@picnixz as GH-122154 author, and introducing is_queue_like_object in file logging/config.py, could you take a look to this issue please ?

Urgh, this issue will haunt me all my life :') According to the docs, it required a "Queue API", but I didn't consider that SimpleQueue was... too simple enough and did not entirely conform to what Queue did. Ok, So I think we should only care about two methods that are those given in SimpleQueue.

So I think we should only care about two methods that are those given in SimpleQueue.

What do you think about adding queue.simpleQueue at L502, as below:

if isinstance(obj, (queue.Queue, queue.SimpleQueue)): 
         return True

It's correct (and that's what I also did), but since we really don't know whether people could use objects or not I just relaxed the interface (and I think it's fine like this).

@spacemanspiff2007

@spacemanspiff2007 good catch !!

@YvesDup

@YvesDup @picnixz Thank you both for your help and implementation

I should be the one thanking you since this was my code :') sorry for the inconvenience!

gh-124653: Relax (again) detection of queue API for logging handlers (G…

pythongh-124653: Relax (again) detection of queue API for logging han…

[3.13] gh-124653: Relax (again) detection of queue API for logging ha…

[3.12] gh-124653: Relax (again) detection of queue API for logging ha…

Navigation Menu

Repository navigation

Python 3.12+ breaks backwards compatibility for logging QueueHandler with queue.SimpleQueue #124653

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	def _is_queue_like_object(obj):
	"""Check that obj implements the Queue API."""
	if isinstance(obj, queue.Queue):
	return True
	# defer importing multiprocessing as much as possible
	from multiprocessing.queues import Queue as MPQueue
	if isinstance(obj, MPQueue):
	return True
	# Depending on the multiprocessing start context, we cannot create
	# a multiprocessing.managers.BaseManager instance 'mm' to get the
	# runtime type of mm.Queue() or mm.JoinableQueue() (see gh-119819).
	#
	# Since we only need an object implementing the Queue API, we only
	# do a protocol check, but we do not use typing.runtime_checkable()
	# and typing.Protocol to reduce import time (see gh-121723).
	#
	# Ideally, we would have wanted to simply use strict type checking
	# instead of a protocol-based type checking since the latter does
	# not check the method signatures.
	queue_interface = [
	'empty', 'full', 'get', 'get_nowait',
	'put', 'put_nowait', 'join', 'qsize',
	'task_done',
	]
	return all(callable(getattr(obj, method, None))
	for method in queue_interface)

Python 3.12+ breaks backwards compatibility for logging QueueHandler with queue.SimpleQueue #124653

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

YvesDup commented on Oct 1, 2024

spacemanspiff2007 commented on Oct 1, 2024

YvesDup commented on Oct 1, 2024

YvesDup commented on Oct 2, 2024

picnixz commented on Oct 2, 2024

YvesDup commented on Oct 2, 2024

picnixz commented on Oct 2, 2024

YvesDup commented on Oct 2, 2024

spacemanspiff2007 commented on Oct 2, 2024

picnixz commented on Oct 2, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions