espressif: Don't drop Characteristic.value notifications under load - #11301
Merged
tannewt merged 2 commits intoSep 4, 2026
Merged
Conversation
Assigning `Characteristic.value` on a local characteristic with NOTIFY or INDICATE delivered only some of the values when other BLE traffic was in flight. A byte-at-a-time UART echo, with the REPL's own output also going out over the BLE workflow, delivered only the first two characters of "asdf". The nordic port does not have this bug. `common_hal_bleio_characteristic_set_value()` called `ble_gatts_chr_updated()`, which cannot report a failure: it sets one "modified" flag per characteristic per connection, clears the flag before attempting the send, and discards the send's return value. A notification that fails transiently with `BLE_HS_ENOMEM` -- the mbuf pool or the controller's buffers momentarily full of other traffic -- is lost with no error and nothing left to retry from. It also reads `current_value` at send time rather than taking the assigned value, so values assigned in quick succession could be transmitted as a later value. The nordic port sends each assigned value itself and retries while the SoftDevice reports its buffers full; this brings espressif to the same behavior. - Track which peers are subscribed, from `BLE_GAP_EVENT_SUBSCRIBE`. esp-nimble exposes no API for reading a peer's CCCD, and notifying without checking would send to peers that never subscribed. NimBLE reports both flags clear when the CCCD is cleared and when the connection ends, and re-reports a bonded peer's restored CCCD on reconnect, so the table maintains itself. - Send notifications and indications from `set_value()` itself, per subscribed peer, with the just-assigned bytes, via `ble_gatts_notify_custom()` and `ble_gatts_indicate_custom()`. Retry `BLE_HS_ENOMEM`, building a fresh mbuf each attempt because NimBLE consumes it even on failure. The retry makes progress from any context: the buffers are drained by the `nimble_host` task, not by background callbacks. Other errors drop the data, unreported to Python, as before. - Pace indications: wait for the previous indication's acknowledgment before sending another. NimBLE permits one outstanding indication per connection, shared across all characteristics, and `ble_gatts_indicate_custom()` does not check -- a second one trips `BLE_HS_DBG_ASSERT` and corrupts the acknowledgment bookkeeping. The acknowledgment or its timeout arrives as `BLE_GAP_EVENT_NOTIFY_TX` with `BLE_HS_EDONE` or `BLE_HS_ETIMEOUT`, so the wait is bounded. - Retain a heap-resident local Service for as long as NimBLE's GATT table refers to it, marking it during garbage collection. That table holds raw pointers into the service and everything under it: the `chr_defs` array lives inside the service object, each characteristic and descriptor is an access-callback argument, and every UUID is referenced by address. Collecting any of them left NimBLE pointing at freed memory, which this branch made easy to hit because a characteristic's event-handler entry lives inside the characteristic too. Retaining the service covers all of it: it owns its characteristics, which own their descriptors. Services in static storage, like the workflow's, cannot be collected and are skipped. - Record whether a Service is registered instead of inferring it from the characteristic count. `characteristic_list->len > 1` is right only in `common_hal_bleio_service_add_characteristic()`, where the new characteristic has already been appended. Copied into `bleio_service_readd()` it registered a single-characteristic service a second time rather than replacing it, and copied into `common_hal_bleio_service_deinit()` it left such a service in NimBLE's GATT table for good. `add_characteristic()` now registers through `readd()`, so every path that hands NimBLE a pointer into a service also retains that service. - Deinit a Service's characteristics when the Service is deinited. That was previously left undone, and matters now that each one holds an event-handler entry that has to come off the event list with it. Local services only, since a remote service's characteristics describe a peer's GATT table and hold no local registration. - Take an object off the BLE event list before tearing down anything it owns, in `common_hal_bleio_characteristic_deinit()` and `common_hal_bleio_packet_buffer_deinit()`. Handlers run on the `nimble_host` task, which preempts the VM task these run on. The PacketBuffer order was reachable rather than latent: its handler dereferences `self->characteristic` unchecked, and `deinit()` cleared that field four statements before removing the handler. - Guard the BLE event handler list, and stop clearing a removed entry's `next` pointer. `ble_event_run_handlers()` walks the list on the `nimble_host` task while the VM task splices entries out of it, and clearing `next` ended a walk that was holding the removed entry as its cursor, dropping the event for every handler after it. The guard is not held across a handler call: handlers call into NimBLE, which takes its own mutex, and FreeRTOS forbids blocking on a mutex inside a critical section. - Derive `BLEIO_TOTAL_CONNECTION_COUNT` from `CONFIG_BT_NIMBLE_MAX_CONNECTIONS`, and assert it is no larger. It was hardcoded to 5 against a NimBLE default of 3, so two connection slots could never be filled, and raising the NimBLE setting past 5 would have overflowed the array. The subscription table is only necessary because esp-nimble has no CCCD-read API. Upstream mynewt-nimble gained `ble_gatts_read_cccd()` in commit `a22602c4`, released in NimBLE 1.9.0. If esp-nimble ever picks it up, the table can be removed. Unlike `ble_gatts_chr_updated()`, this does not persist a "changed" flag for bonded peers that are currently disconnected, so they are not notified on reconnect. That matches the nordic port. Tested on a Metro ESP32-S3 with the Bluefruit Connect app: - ble_uart_echo_test.py: byte-at-a-time echo returns every character - interrupting ble_uart_echo_test.py with Ctrl-C and importing another copy of the same program repeatedly, with the peer still connected, no longer hard faults. The hard fault was a collected characteristic's stale handler entry - reload and reconnect work properly The indication path is untested on hardware: nothing in the workflow, the bundled examples, or adafruit_ble creates a local INDICATE characteristic. It exists so user-defined services that require indications keep working. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`common_hal_bleio_service_add_characteristic()` indexed `chr_defs` with the characteristic list length and never checked it against `MAX_CHARACTERISTIC_COUNT`, which was only used to size that array. An eleventh characteristic wrote `chr_defs[11].uuid`, one element past the end. With the retention list added earlier in this PR, that write lands exactly on `next_retained` and truncates the chain of services held against garbage collection, un-retaining every service behind it. Raise `BluetoothError` before appending, matching `common_hal_bleio_characteristic_add_descriptor()`. Also reword both limits as "Too many %q", using the existing `MP_QSTR_characteristics` and `MP_QSTR_descriptors`, replacing "Too many descriptors". No new qstrs and no net change in translatable strings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Code written by Claude Code, guided and corrected by @dhalbert.
@dhalbert says: I rewrote this post heavily for length and clarity, so it's worth reading. I also reviewed the code and rephrased the comments in the same way. The changes here have gone through four quite different solutions before arriving at what was done here. (The commit message is up to date, but was not edited, and contains gory details for posterity.)
The problem
On espressif, setting
Characteristic.valueon a local characteristic with NOTIFY or INDICATE silently drops values whenever other BLE traffic interferes. For example, whenble_uart_echo_test.pyechos to the Bluefruit Connect app, it only sends the first two characters of, say, "1234", because the REPL's own output over the BLE workflow is competing for the radio's buffers.This is a long-standing bug on espressif. It's not a problem on nordic, because the nordic code sends each assigned value itself and retries while the SoftDevice's buffers are full.
On espressif, setting
.valuecurrently usesble_gatts_chr_updated(). That routine is optimistic: it clears its per-characteristic "modified" flag before attempting the send and also does not look at whether the send succeeded. If the radio is busy, the send will get a transientBLE_HS_ENOMEM. So the value change is silently lost.The fix
Do what nordic does: send notifications and indications for the new bytes explicitly. If there's a
BLE_HS_ENOMEM, retry until it succeeds. Wait for each indication acknowledgment before sending the next one. For indications, useble_gatts_indicate_custom().To do this we also have to know about INDICATE and NOTIFY subscriptions from the remote peer. Unfortunately, there's no API in the NimBLE version ESP-IDF uses to get that info. So the fix records subscriptions itself, per characteristic, from the subscribe events NimBLE does report. It also retains a heap-allocated
Servicefor as long as NimBLE's GATT table refers to it.(In Nimble 1.9.0, there is an API to read a peer's CCCD, so we wouldn't have to record subscriptions ourselves. But ESP-IDF is on Nimble 1.6.0 and shows no signs of an imminent update.)
Also in this PR, because we needed these things to make the changes correct and fix existing related bugs:
Characteristics and theirDescriptors when theServicethat holds them is deinited. Before this, Characteristics were not deinited, and would accumulate until the VM shut down.Servicenow records whether NimBLE has it registered, instead of inferring that from its characteristic count.BLEIO_TOTAL_CONNECTION_COUNTnow comes fromCONFIG_BT_NIMBLE_MAX_CONNECTIONS.Behavior change
ble_gatts_chr_updated()also persisted a "changed" flag so a bonded peer that was disconnected at assignment time would be notified on reconnect. With this PR, that no longer happens. Nordic never did either, so the ports now agree.Testing
Metro ESP32-S3 with Bluefruit Connect:
ble_uart_echo_test.pynow sends every character, even when REPL print traffic is also being sent.MemoryError: Nimble out of memory. That is a different problem we know about and may address later.