Fixed #25705 -- Added Queryset.executed_queries. by alexgmin · Pull Request #18468 · django/django

alexgmin · 2024-08-11T16:10:09Z

Trac ticket number

Branch description

Based on the discussion with @charettes it was agreed to follow the path of adding Queryset.executed_queries. To solve the ticket.

Since the DB connections are per-thread as far as I know, this approach should not have any race condition issues. And since it's done before the prefetch, it should pick the Queryset query. Also, since queries are only logged if DEBUG is enabled, this approach requires that too.

My discarded approach was to to get it somehow in the return of self._result_cache = list(self._iterable_class(self)), or stored in the QuerySet.
However, it's much more complicated, because as far as I see the flow is something like
QuerySet when evaluating calls the IterableClass which then executes the query which then gets the connection cursor, which is wrapped in the previous mentioned one that can log the query.

So the a way would be to make all 4 SqlCompiler.execute_sql return the result and the SQL query. Then you have to adapt all 6 Iterable.__iter__ to also return the query. Then you can store it in the Queryset. Since these methods are part of the public API, this would be a breaking change in both methods. Also I'm not 100% sure how to get the query in the wrapper since the Cursor.execute is wrapped in a context manager which is the one logging the query.

AI Assistance Disclosure (REQUIRED)

No AI tools were used in preparing this PR.
If AI tools were used, I have disclosed which ones, and fully reviewed and verified their output.

Checklist

This PR follows the contribution guidelines.
This PR does not disclose a security vulnerability (see vulnerability reporting).
This PR targets the main branch.
The commit message is written in past tense, mentions the ticket number (if applicable), and ends with a period (see guidelines).
I have not requested, and will not request, an automated AI review for this PR.

I have checked the "Has patch" ticket flag in the Trac system.
I have added or updated relevant tests.
I have added or updated relevant docs, including release notes if applicable.
I have attached screenshots in both light and dark modes for any UI changes.

charettes

I like the non-invasive approach you have taken but I wonder if we could avoid coupling QuerySet with .queries and setting.DEBUG by adding a new last_query property

diff --git a/django/db/backends/base/base.py b/django/db/backends/base/base.py
index 9a4c27f0f3..6d62d30e86 100644
--- a/django/db/backends/base/base.py
+++ b/django/db/backends/base/base.py
@@ -158,6 +158,10 @@ def queries(self):
                 "will be returned.".format(self.queries_log.maxlen))
         return list(self.queries_log)

+    @property
+    def last_query(self):
+        return queries[-1] if (queries := self.queries_log) else None
+
     # ##### Backend-specific methods for creating connections and cursors #####

     def get_connection_params(self):

This will also require tests and docs.

alexgmin · 2024-08-27T11:18:18Z

I like the non-invasive approach you have taken but I wonder if we could avoid coupling QuerySet with .queries and setting.DEBUG by adding a new last_query property

I agree on checking whether there are queries in the list instead of checking settings.DEBUG .
However, could you expand on what are the benefits of adding the last_query property? Is it that with the property you can use the internal queries_log property? Isn't it still kinda coupled?
Also, by using queries_log you skip this warning

django/django/db/backends/base/base.py

Lines 173 to 180 in bc9b625

    
           @property 
        
           def queries(self): 
        
               if len(self.queries_log) == self.queries_log.maxlen: 
        
                   warnings.warn( 
        
                       "Limit for query logging exceeded, only the last {} queries " 
        
                       "will be returned.".format(self.queries_log.maxlen) 
        
                   ) 
        
               return list(self.queries_log)

I guess it's not relevant in this case since we're picking always the last one, but in the future there could be other checks or warnings. Wouldn't it make sense to use the more public queries property?

This will also require tests and docs.

100% agree, but I wanted to wait until I was sure the approach was acceptable. I'll start work on them soon.

alexgmin · 2024-09-19T09:03:05Z

Just as a quick update, I'm still slowly working on this, but life keeps happening.

alexgmin · 2024-10-23T19:15:48Z

Implementation is done and I added a test. The only missing thing is the documentation.

I'm not sure if there's any other case that should be tested. I tried an async test, but in async connection.queries always seems to be None

rtpg

Seems like this would only kick in with DEBUG = True (or a forced debug cursor) so the documentation should reflect that.

rtpg · 2024-11-07T03:40:53Z

  columns.

+* The new :attr:`QuerySet.executed_query <.QuerySet.executed_query>`
+  attribute returns the SQL query if the QuerySet has been evaluated.


This only happens if the connection is actually building up a query log right? And so even if a queryset is evaluated there are times where executed_query is None right? Or am I misreading something about Connection.queries?

As far as I know, connection.queries is filled based on this condition

django/django/db/backends/base/base.py

Lines 169 to 171 in 661dfdd

@property

def queries_logged(self):

return self.force_debug_cursor or settings.DEBUG

force_debug_cursor is not documented and the only use I can find in the Django codebase is here:

django/django/test/utils.py

Lines 714 to 716 in 661dfdd

def __enter__(self):

self.force_debug_cursor = self.connection.force_debug_cursor

self.connection.force_debug_cursor = True

Even a custom Database backend that inherits from the BaseDatabaseWrapper should implement these two methods:

django/django/db/backends/base/base.py

Lines 677 to 683 in 661dfdd

def make_debug_cursor(self, cursor):

"""Create a cursor that logs all queries in self.queries_log."""

return utils.CursorDebugWrapper(cursor, self)

def make_cursor(self, cursor):

"""Create a cursor without debug logging."""

return utils.CursorWrapper(cursor, self)

And which one is called is decided here:

django/django/db/backends/base/base.py

Lines 284 to 293 in 661dfdd

def _prepare_cursor(self, cursor):

"""

Validate the connection is usable and perform database cursor wrapping.

"""

self.validate_thread_sharing()

if self.queries_logged:

wrapped_cursor = self.make_debug_cursor(cursor)

else:

wrapped_cursor = self.make_cursor(cursor)

return wrapped_cursor

So the only case I realistically see in which you have access to the evaluated queryset after executing it (So methods like .all(), since .get() wouldn't return a queryset) and executed_query is None is if DEBUG is False, which is something I will add in the documentation changes.

tim-schilling · 2024-11-16T17:50:41Z

With the API of executed_query, it limits the queries that are executed when a QuerySet to the main query. However, the prefetch_related queries are also being executed. That information, depending on the scenario, may be useful to a developer. It may be worth another round of consideration before finalizing implementation.

I do recognize that this would mean we'd have to map the queries being run to the QuerySet that's being evaluated. Some possibilities are a context manager or a global-eque state via contextvars.ContextVar. The latter would require some sort of key for each QuerySet, which may also be a new concept.

raffaellasuardini · 2024-11-16T17:50:42Z

it could be interesting to insert the test related with none. The executed queries could be reset to being chained on the next operation.

none_queryset = queryset.all().none()
list(none_queryset)
self.assertEqual(none_queryset.executed_query, None)

laymonage · 2024-12-01T19:46:23Z

    def _fetch_all(self):
        if self._result_cache is None:
            self._result_cache = list(self._iterable_class(self))
+            self.executed_query = connections[self.db].last_query


Thanks for the PR!

As mentioned during the Djangonauts Review session, this could be a problem if the queryset does not execute a database query at all e.g. objects.none(), as it would return the last executed query which may come from a different queryset.

A potential workaround is to get a snapshot of connections[self.db].queries before the iterable is evaluated. Then, compare the snapshot with what the list of queries are now. If there are new queries, then those are likely to be related to the current queryset. This might also help with prefetch queries. So maybe instead of executed_query, we should make it executed_queries instead which contains the new queries. This means we return an empty list if no queries are executed – which may also avoid having to decide whether to return None or raise an exception.

Of course, this solution might not be safe for race conditions. Though I'm not sure if the current approach is safe either 🤔

I wonder if Django would be better off if there was a signal that was raised when queries are executed.

Sounds like it could be a performance hit though? Calling all the handlers for that signal (even if there aren't any) for every single query seems like it would add up. (Of course, we can't be sure without any benchmarks.)

I think my solution would be to simply clear the .executed_query when you call .none().

That's the maximum I would complicate this case. If fixing the none() case required extra work I think it would be better to not fix it.
This a purely debugging helper field, so in that case, a situation in which someone reuses a queryset to call none() and then looks at the executed query expecting it to be None it's too rare to have such a complicated implementation.

It might. I think I should take this discussion to the forum, it doesn't belong on this particular PR. I do like your idea Sage of having an executed_queries as that seems more future proof.

Another issue with using a signal is that it is possible to perform a query during a signal receiver causing an infinite loop of queries.

I like @laymonage's idea of snapshoting. We could snapshot the length of connection.queries list before and after evaluation and have executed_query (or executed_queries assuming we want to include prefetching as well) be a slice of it.

If the index remain the same because no queries were executed then executed_query is None (or executed_queries is []).

alexgmin · 2025-10-09T10:09:16Z

During Django on the Med @charettes and I agreed on the following, to move this forward and handle the feedback:

Change the property to be Queryset.executed_queries
Include prefetched queries, with the main query always being the first one on the list
Handle when a queryset is executed but doesn't actually do a query, like .none() or .filter(pk__in=[])

The values will be the following:

Non-executed queryset: None
Executed queryset without queries: []
Executed queryset without prefetch: ['query']
Executed queryset with prefetch: ['query', 'prefetch1', 'prefetch2']

jacobtylerwalls

Thanks! Looked at this with @raffaellasuardini at DjangoCon EU 🇬🇷. This is very exciting!

(I know you are adding docs for this later.)

jacobtylerwalls · 2026-04-18T10:46:29Z

+
+
+class ExecutedQueriesTests(TestCase):
+    def test_no_debug_executed_queries(self):


Let's make the value of DEBUG explicit like the other test methods you added.

jacobtylerwalls · 2026-04-18T10:47:20Z

        required_db_features = {"supports_json_field"}
+
+
+class ExecutedQueriesModel(models.Model):


We discussed this is a great candidate for reusing a simple test model, maybe one used often in other tests in this file.

jacobtylerwalls · 2026-04-18T10:47:35Z

+            queries_executed = True
            self._prefetch_related_objects()

+        if queries_executed:


Factoring this into a util will allow us to reuse this on RawQuerySet.

jacobtylerwalls · 2026-04-18T10:51:01Z

+
+        self.assertEqual(
+            queryset.executed_queries,
+            [


We'll need to relax this assertion.

jacobtylerwalls · 2026-04-18T10:57:37Z

            self._result_cache = list(self._iterable_class(self))
+
        if self._prefetch_related_lookups and not self._prefetch_done:
+            queries_executed = True


No tests fail when I remove this. Is there a test we can write to demonstrate this is needed?

It was not needed indeed. The local tests also pass if the code is

if self._result_cache is None: executed_queries_length = len(connections[self.db].queries) self._result_cache = list(self.iterator()) if self._prefetch_related_lookups and not self._prefetch_done: # Nested instead self._prefetch_related_objects()

So there may not be a need for it two be two separate ifs

github-actions · 2026-07-23T18:34:45Z

Thank you for your contribution to Django! This pull request has one or more items that need attention before it can be accepted for review.

❗ Error: Incomplete Checklist

The items in the Checklist section must be all understood and checked accordingly before this PR can be accepted. These items confirm that you have read and followed the Django contributing guidelines.

What to do:

Review each item and change [ ] to [x] once you have completed it.

If you have questions about these requirements, please review the contributing guidelines or ask for help on the Django Forum.

alexgmin · 2024-11-07T10:03:49Z

  columns.

+* The new :attr:`QuerySet.executed_query <.QuerySet.executed_query>`
+  attribute returns the SQL query if the QuerySet has been evaluated.


As far as I know, connection.queries is filled based on this condition

django/django/db/backends/base/base.py

Lines 169 to 171 in 661dfdd

@property

def queries_logged(self):

return self.force_debug_cursor or settings.DEBUG

force_debug_cursor is not documented and the only use I can find in the Django codebase is here:

django/django/test/utils.py

Lines 714 to 716 in 661dfdd

def __enter__(self):

self.force_debug_cursor = self.connection.force_debug_cursor

self.connection.force_debug_cursor = True

Even a custom Database backend that inherits from the BaseDatabaseWrapper should implement these two methods:

django/django/db/backends/base/base.py

Lines 677 to 683 in 661dfdd

def make_debug_cursor(self, cursor):

"""Create a cursor that logs all queries in self.queries_log."""

return utils.CursorDebugWrapper(cursor, self)

def make_cursor(self, cursor):

"""Create a cursor without debug logging."""

return utils.CursorWrapper(cursor, self)

And which one is called is decided here:

django/django/db/backends/base/base.py

Lines 284 to 293 in 661dfdd

def _prepare_cursor(self, cursor):

"""

Validate the connection is usable and perform database cursor wrapping.

"""

self.validate_thread_sharing()

if self.queries_logged:

wrapped_cursor = self.make_debug_cursor(cursor)

else:

wrapped_cursor = self.make_cursor(cursor)

return wrapped_cursor

So the only case I realistically see in which you have access to the evaluated queryset after executing it (So methods like .all(), since .get() wouldn't return a queryset) and executed_query is None is if DEBUG is False, which is something I will add in the documentation changes.

alexgmin · 2024-12-02T13:27:01Z

    def _fetch_all(self):
        if self._result_cache is None:
            self._result_cache = list(self._iterable_class(self))
+            self.executed_query = connections[self.db].last_query


I think my solution would be to simply clear the .executed_query when you call .none().

That's the maximum I would complicate this case. If fixing the none() case required extra work I think it would be better to not fix it.
This a purely debugging helper field, so in that case, a situation in which someone reuses a queryset to call none() and then looks at the executed query expecting it to be None it's too rare to have such a complicated implementation.

alexgmin · 2026-07-23T19:00:30Z

            self._result_cache = list(self._iterable_class(self))
+
        if self._prefetch_related_lookups and not self._prefetch_done:
+            queries_executed = True


It was not needed indeed. The local tests also pass if the code is

if self._result_cache is None: executed_queries_length = len(connections[self.db].queries) self._result_cache = list(self.iterator()) if self._prefetch_related_lookups and not self._prefetch_done: # Nested instead self._prefetch_related_objects()

So there may not be a need for it two be two separate ifs

alexgmin · 2026-07-23T19:12:03Z

        with self.settings(DATABASE_ROUTERS=[router]):
            book.refresh_from_db()
-        router.db_for_read.assert_called_once_with(Book, instance=book)
+        router.db_for_read.assert_called_with(Book, instance=book)


Relaxed this assertion, as talked with Jacob in DjangoCon Athens.

alexgmin marked this pull request as draft August 11, 2024 16:10

alexgmin force-pushed the ticket_25705 branch from fade673 to c6a3a11 Compare August 11, 2024 16:12

charettes reviewed Aug 27, 2024

View reviewed changes

Comment thread django/db/models/query.py Outdated

alexgmin force-pushed the ticket_25705 branch from 019ef53 to c6a3a11 Compare October 16, 2024 18:31

alexgmin force-pushed the ticket_25705 branch 6 times, most recently from 86874da to d8e604a Compare October 23, 2024 19:12

alexgmin force-pushed the ticket_25705 branch 2 times, most recently from 34a5b4f to 1e1befd Compare October 26, 2024 17:09

alexgmin marked this pull request as ready for review October 26, 2024 17:30

rtpg reviewed Nov 7, 2024

View reviewed changes

laymonage reviewed Dec 1, 2024

View reviewed changes

jacobtylerwalls added the DjangoCon Europe 🏰 label Apr 18, 2026

alexgmin force-pushed the ticket_25705 branch from 1e1befd to ae17ac0 Compare April 18, 2026 09:25

alexgmin changed the title ~~Refs #25705 -- Added Queryset.executed_query~~ Fix #25705 -- Added Queryset.executed_queries Apr 18, 2026

jacobtylerwalls changed the title ~~Fix #25705 -- Added Queryset.executed_queries~~ Fixed #25705 -- Added Queryset.executed_queries. Apr 18, 2026

alexgmin force-pushed the ticket_25705 branch from ae17ac0 to 6d5754b Compare April 18, 2026 09:42

jacobtylerwalls reviewed Apr 18, 2026

View reviewed changes

alexgmin force-pushed the ticket_25705 branch 2 times, most recently from 5cae913 to 98ade1f Compare July 18, 2026 14:37

alexgmin force-pushed the ticket_25705 branch 5 times, most recently from edc2af3 to c47187a Compare July 23, 2026 19:05

Fixed #25705 -- Added Queryset.executed_queries.

178fc13

alexgmin force-pushed the ticket_25705 branch from c47187a to 178fc13 Compare July 23, 2026 19:07

alexgmin commented Jul 23, 2026

View reviewed changes

jacobtylerwalls self-requested a review July 23, 2026 22:07

	@property
	def queries_logged(self):
	return self.force_debug_cursor or settings.DEBUG

	def __enter__(self):
	self.force_debug_cursor = self.connection.force_debug_cursor
	self.connection.force_debug_cursor = True

	def make_debug_cursor(self, cursor):
	"""Create a cursor that logs all queries in self.queries_log."""
	return utils.CursorDebugWrapper(cursor, self)

	def make_cursor(self, cursor):
	"""Create a cursor without debug logging."""
	return utils.CursorWrapper(cursor, self)

	def _prepare_cursor(self, cursor):
	"""
	Validate the connection is usable and perform database cursor wrapping.
	"""
	self.validate_thread_sharing()
	if self.queries_logged:
	wrapped_cursor = self.make_debug_cursor(cursor)
	else:
	wrapped_cursor = self.make_cursor(cursor)
	return wrapped_cursor



		class ExecutedQueriesTests(TestCase):
		def test_no_debug_executed_queries(self):

		required_db_features = {"supports_json_field"}


		class ExecutedQueriesModel(models.Model):

Conversation

alexgmin commented Aug 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Trac ticket number

Branch description

AI Assistance Disclosure (REQUIRED)

Checklist

Uh oh!

charettes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alexgmin commented Aug 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexgmin commented Sep 19, 2024

Uh oh!

alexgmin commented Oct 23, 2024

Uh oh!

rtpg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tim-schilling commented Nov 16, 2024

Uh oh!

raffaellasuardini commented Nov 16, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

charettes Dec 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexgmin commented Oct 9, 2025

Uh oh!

jacobtylerwalls left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

alexgmin commented Aug 11, 2024 •

edited

Loading

alexgmin commented Aug 27, 2024 •

edited

Loading

charettes Dec 7, 2024 •

edited

Loading