Skip to content

espressif: Don't drop Characteristic.value notifications under load - #11301

Merged
tannewt merged 2 commits into
adafruit:mainfrom
dhalbert:espressif-characteristic-notify-retry
Sep 4, 2026
Merged

tannewt merged 2 commits into
adafruit:mainfrom
dhalbert:espressif-characteristic-notify-retry

Conversation

@dhalbert

@dhalbert dhalbert commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Code written by Claude Code, guided and corrected by @dhalbert.

@dhalbert says: I rewrote this post heavily for length and clarity, so it's worth reading. I also reviewed the code and rephrased the comments in the same way. The changes here have gone through four quite different solutions before arriving at what was done here. (The commit message is up to date, but was not edited, and contains gory details for posterity.)

The problem

On espressif, setting Characteristic.value on a local characteristic with NOTIFY or INDICATE silently drops values whenever other BLE traffic interferes. For example, when ble_uart_echo_test.py echos to the Bluefruit Connect app, it only sends the first two characters of, say, "1234", because the REPL's own output over the BLE workflow is competing for the radio's buffers.

This is a long-standing bug on espressif. It's not a problem on nordic, because the nordic code sends each assigned value itself and retries while the SoftDevice's buffers are full.

On espressif, setting .value currently uses ble_gatts_chr_updated(). That routine is optimistic: it clears its per-characteristic "modified" flag before attempting the send and also does not look at whether the send succeeded. If the radio is busy, the send will get a transient BLE_HS_ENOMEM. So the value change is silently lost.

The fix

Do what nordic does: send notifications and indications for the new bytes explicitly. If there's a BLE_HS_ENOMEM, retry until it succeeds. Wait for each indication acknowledgment before sending the next one. For indications, use ble_gatts_indicate_custom().

To do this we also have to know about INDICATE and NOTIFY subscriptions from the remote peer. Unfortunately, there's no API in the NimBLE version ESP-IDF uses to get that info. So the fix records subscriptions itself, per characteristic, from the subscribe events NimBLE does report. It also retains a heap-allocated Service for as long as NimBLE's GATT table refers to it.

(In Nimble 1.9.0, there is an API to read a peer's CCCD, so we wouldn't have to record subscriptions ourselves. But ESP-IDF is on Nimble 1.6.0 and shows no signs of an imminent update.)

Also in this PR, because we needed these things to make the changes correct and fix existing related bugs:

  • New code to deinit the Characteristics and their Descriptors when the Service that holds them is deinited. Before this, Characteristics were not deinited, and would accumulate until the VM shut down.
  • A Service now records whether NimBLE has it registered, instead of inferring that from its characteristic count.
  • Take objects off the BLE event handler list first in their deinit() routines, to avoid any races.
  • Add critical sections to BLE event handler list manipulation.
  • BLEIO_TOTAL_CONNECTION_COUNT now comes from CONFIG_BT_NIMBLE_MAX_CONNECTIONS.

Behavior change

ble_gatts_chr_updated() also persisted a "changed" flag so a bonded peer that was disconnected at assignment time would be notified on reconnect. With this PR, that no longer happens. Nordic never did either, so the ports now agree.

Testing

Metro ESP32-S3 with Bluefruit Connect:

  • ble_uart_echo_test.py now sends every character, even when REPL print traffic is also being sent.
  • Repeatedly interrupting one copy of the echo program and importing another copy with the peer still connected no longer hard faults; it does raise MemoryError: Nimble out of memory. That is a different problem we know about and may address later.
  • Reload and reconnect work.

dhalbert and others added 2 commits September 3, 2026 14:51
Assigning `Characteristic.value` on a local characteristic with NOTIFY
or INDICATE delivered only some of the values when other BLE traffic
was in flight. A byte-at-a-time UART echo, with the REPL's own output
also going out over the BLE workflow, delivered only the first two
characters of "asdf". The nordic port does not have this bug.

`common_hal_bleio_characteristic_set_value()` called
`ble_gatts_chr_updated()`, which cannot report a failure: it sets one
"modified" flag per characteristic per connection, clears the flag
before attempting the send, and discards the send's return value. A
notification that fails transiently with `BLE_HS_ENOMEM` -- the mbuf
pool or the controller's buffers momentarily full of other traffic --
is lost with no error and nothing left to retry from. It also reads
`current_value` at send time rather than taking the assigned value, so
values assigned in quick succession could be transmitted as a later
value. The nordic port sends each assigned value itself and retries
while the SoftDevice reports its buffers full; this brings espressif to
the same behavior.

- Track which peers are subscribed, from `BLE_GAP_EVENT_SUBSCRIBE`.
  esp-nimble exposes no API for reading a peer's CCCD, and notifying
  without checking would send to peers that never subscribed. NimBLE
  reports both flags clear when the CCCD is cleared and when the
  connection ends, and re-reports a bonded peer's restored CCCD on
  reconnect, so the table maintains itself.
- Send notifications and indications from `set_value()` itself, per
  subscribed peer, with the just-assigned bytes, via
  `ble_gatts_notify_custom()` and `ble_gatts_indicate_custom()`. Retry
  `BLE_HS_ENOMEM`, building a fresh mbuf each attempt because NimBLE
  consumes it even on failure. The retry makes progress from any
  context: the buffers are drained by the `nimble_host` task, not by
  background callbacks. Other errors drop the data, unreported to
  Python, as before.
- Pace indications: wait for the previous indication's acknowledgment
  before sending another. NimBLE permits one outstanding indication per
  connection, shared across all characteristics, and
  `ble_gatts_indicate_custom()` does not check -- a second one trips
  `BLE_HS_DBG_ASSERT` and corrupts the acknowledgment bookkeeping. The
  acknowledgment or its timeout arrives as `BLE_GAP_EVENT_NOTIFY_TX`
  with `BLE_HS_EDONE` or `BLE_HS_ETIMEOUT`, so the wait is bounded.
- Retain a heap-resident local Service for as long as NimBLE's GATT
  table refers to it, marking it during garbage collection. That table
  holds raw pointers into the service and everything under it: the
  `chr_defs` array lives inside the service object, each characteristic
  and descriptor is an access-callback argument, and every UUID is
  referenced by address. Collecting any of them left NimBLE pointing at
  freed memory, which this branch made easy to hit because a
  characteristic's event-handler entry lives inside the characteristic
  too. Retaining the service covers all of it: it owns its
  characteristics, which own their descriptors. Services in static
  storage, like the workflow's, cannot be collected and are skipped.
- Record whether a Service is registered instead of inferring it from
  the characteristic count. `characteristic_list->len > 1` is right
  only in `common_hal_bleio_service_add_characteristic()`, where the
  new characteristic has already been appended. Copied into
  `bleio_service_readd()` it registered a single-characteristic service
  a second time rather than replacing it, and copied into
  `common_hal_bleio_service_deinit()` it left such a service in
  NimBLE's GATT table for good. `add_characteristic()` now registers
  through `readd()`, so every path that hands NimBLE a pointer into a
  service also retains that service.
- Deinit a Service's characteristics when the Service is deinited.
  That was previously left undone, and matters now that each one holds
  an event-handler entry that has to come off the event list with it.
  Local services only, since a remote service's characteristics
  describe a peer's GATT table and hold no local registration.
- Take an object off the BLE event list before tearing down anything it
  owns, in `common_hal_bleio_characteristic_deinit()` and
  `common_hal_bleio_packet_buffer_deinit()`. Handlers run on the
  `nimble_host` task, which preempts the VM task these run on. The
  PacketBuffer order was reachable rather than latent: its handler
  dereferences `self->characteristic` unchecked, and `deinit()` cleared
  that field four statements before removing the handler.
- Guard the BLE event handler list, and stop clearing a removed entry's
  `next` pointer. `ble_event_run_handlers()` walks the list on the
  `nimble_host` task while the VM task splices entries out of it, and
  clearing `next` ended a walk that was holding the removed entry as
  its cursor, dropping the event for every handler after it. The guard
  is not held across a handler call: handlers call into NimBLE, which
  takes its own mutex, and FreeRTOS forbids blocking on a mutex inside
  a critical section.
- Derive `BLEIO_TOTAL_CONNECTION_COUNT` from
  `CONFIG_BT_NIMBLE_MAX_CONNECTIONS`, and assert it is no larger. It
  was hardcoded to 5 against a NimBLE default of 3, so two connection
  slots could never be filled, and raising the NimBLE setting past 5
  would have overflowed the array.

The subscription table is only necessary because esp-nimble has no
CCCD-read API. Upstream mynewt-nimble gained `ble_gatts_read_cccd()` in
commit `a22602c4`, released in NimBLE 1.9.0. If esp-nimble ever picks
it up, the table can be removed.

Unlike `ble_gatts_chr_updated()`, this does not persist a "changed"
flag for bonded peers that are currently disconnected, so they are not
notified on reconnect. That matches the nordic port.

Tested on a Metro ESP32-S3 with the Bluefruit Connect app:
- ble_uart_echo_test.py: byte-at-a-time echo returns every character
- interrupting ble_uart_echo_test.py with Ctrl-C and importing another
copy of the same program repeatedly, with the peer still connected, no
longer hard faults. The hard fault was a collected characteristic's
stale handler entry
- reload and reconnect work properly

The indication path is untested on hardware: nothing in the workflow,
the bundled examples, or adafruit_ble creates a local INDICATE
characteristic. It exists so user-defined services that require
indications keep working.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`common_hal_bleio_service_add_characteristic()` indexed `chr_defs` with
the characteristic list length and never checked it against
`MAX_CHARACTERISTIC_COUNT`, which was only used to size that array. An
eleventh characteristic wrote `chr_defs[11].uuid`, one element past the
end. With the retention list added earlier in this PR, that write lands
exactly on `next_retained` and truncates the chain of services held
against garbage collection, un-retaining every service behind it.

Raise `BluetoothError` before appending, matching
`common_hal_bleio_characteristic_add_descriptor()`.

Also reword both limits as "Too many %q", using the existing
`MP_QSTR_characteristics` and `MP_QSTR_descriptors`, replacing "Too many
descriptors". No new qstrs and no net change in translatable strings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@dhalbert
dhalbert requested a review from tannewt September 3, 2026 23:48

@tannewt tannewt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@tannewt
tannewt merged commit d9ebb23 into adafruit:main Sep 4, 2026
683 checks passed
@dhalbert
dhalbert deleted the espressif-characteristic-notify-retry branch September 4, 2026 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants