Summary

The detailed group-device listing (GET /v1/{realm}/groups/{group}/devices?details=true) can return an incomplete device list with 200 OK and no links.next, so consumers silently receive fewer devices than the group actually contains.

This is primarily a code-level correctness bug, independent of hardware or load: the query result's completeness is inferred from count < limit and the driver's paging_state is never consulted, while the underlying query uses ALLOW FILTERING. With filtering/scanning, a page may legitimately contain fewer than limit rows while more matching rows still exist — so count < limit is not a valid end-of-results signal, and links.next is omitted exactly when it should be present.

Correctness bug (the core issue, no reproduction needed)

Path: Groups.list_detailed_devices/3 → Queries (list_devices_with_details) → DeviceStatusByGroupController.index/2, rendered by device_status_by_group_view.ex.

The details=true query scans the devices table:

SELECT <device columns> FROM devices
WHERE TOKEN(device_id) > :previous_token AND groups CONTAINS KEY :group_name
LIMIT :page_size          -- page_size = opts[:limit]
ALLOW FILTERING

It is executed once and completeness is decided purely by row count (paging_state is never used):

# master (Ecto/Exandra); identical logic in 1.0.4 (raw Xandra, single Enum.to_list(page))
case Repo.all(query, consistency: consistency) do
  [] -> {:error, :group_not_found}
  devices -> {:ok, build_device_list_with_details(keyspace, devices, opts)}
end

# ...then:
if count < opts[:limit] do
  %DevicesList{devices: Enum.reverse(device_list)}                 # no last_token -> "no more pages"
else
  %DevicesList{devices: Enum.reverse(device_list), last_token: last_token}
end

Why this is incorrect regardless of environment:

• With ALLOW FILTERING, LIMIT bounds the number of returned rows but does not guarantee that a returned page containing fewer than LIMIT rows means the scan is exhausted. The server can return a short page while more matching rows remain (a non-nil paging_state).
• The code uses count < limit as the end-of-results signal and never inspects paging_state. So a short-but-not-final page is treated as "finished": last_token is not set and links.next is not emitted.
• Net effect: the API returns a partial list with 200 OK and no indication that more pages exist.

Secondary issue: full table scan with ALLOW FILTERING

The details=false path reads the membership by partition key and is reliable:

SELECT device_id FROM grouped_devices WHERE group_name = :group_name

The details=true path instead scans the whole devices table filtering on groups CONTAINS KEY. Besides feeding the truncation above (short pages become more likely as the table grows), this is also the query that, run concurrently at scale, exhausts the DB connection pool (DBConnection.ConnectionError: connection not available and request was dropped from queue observed on DeviceStatusByGroupController.index/2 under concurrent load).

Observed behavior

• GET /groups/{group}/devices → full set (e.g. 19), reliably (partition read on grouped_devices).
• GET /groups/{group}/devices?details=true → fewer devices (e.g. 12, sometimes 14, rarely all 19), with 200 OK and no links.next; the count varies across identical requests.
• GET /devices/{device_id} for each id → all succeed with full details, so no data is missing in the DB.

Affected versions

• Reproduced on astarte_appengine_api 1.0.4.
• Confirmed still present on master: PR (Appengine) Rewrite Queries for ecto and exandra #1083 ("(Appengine) Rewrite Queries for ecto and exandra") rewrote these modules but kept both ALLOW FILTERING and the count < limit completeness heuristic.

Note on reproducibility

The empirical manifestation depends on the size of the devices table relative to the query page size, not on cluster performance: the more rows the devices table has, the more likely a scan page returns fewer than limit matches with a non-nil paging_state. On small realms/tables it may not manifest at all (a single page may cover the whole table), which is why a group of 19 devices may look fine on a small test deployment. The correctness bug (relying on count < limit and ignoring paging_state) is present regardless of whether a given deployment currently triggers the truncation.

Conditions under which it manifests

1. A realm whose devices table is large enough that the filtered scan spans more than one page for the requested limit.
2. A group whose members are spread across the token range of devices.
3. Then ?details=true returns a variable, silently-truncated subset (200, no next), while the non-detailed listing and per-device lookups return everything.

Suggested fix

• Prefer resolving group membership from grouped_devices (partition key group_name) and fetching each device's status by device_id (partition-key reads), removing the ALLOW FILTERING scan entirely; or
• If the scan is kept, drive pagination from the driver's native paging_state instead of count < limit, so page boundaries are decided by the driver and no matching rows are silently dropped (and links.next is emitted whenever more pages exist).

Workaround (for consumers)

Avoid ?details=true on groups. List the group without details (partition read on grouped_devices) and fetch details per device via GET /devices/{device_id}, ideally with bounded concurrency.

Related

• AppEngine queries on group devices do not work if the group name contains a forward slash '/' #904 (group queries fail when the group name contains /) and AppEngine crash on encoded group names #456 (crash on encoded group names) — different issues, same endpoint area.