feat(connections): add pausedSyncs to connection list API#5699
Conversation
| provider: data.provider, | ||
| activeLog: data.active_logs, | ||
| endUser: data.end_user, | ||
| hasPausedSyncs: pausedConnectionIds.has(data.connection.id) |
There was a problem hiding this comment.
Could we instead return a list of paused sync names and let the client figure out the boolean just based on .length > 0? We've done all the heavy lifting for it already at this point and so it's more information for free.
| }; | ||
|
|
||
| export async function up(knex: Knex): Promise<void> { | ||
| await knex.raw(`CREATE INDEX CONCURRENTLY IF NOT EXISTS "idx_schedules_paused_name" ON ${SCHEDULES_TABLE} (name) WHERE state = 'PAUSED';`); |
There was a problem hiding this comment.
have you run an EXPLAIN on the search query? Given how small schedules table is, this index might not be even needed
There was a problem hiding this comment.
I had run an explain, with the old approach it would perform better, but its hard to tell if worth it. Now that we call it by page, it makes no sense so I dropped the index. Old approach seq scanned 74K rows in ~50ms without the index, for reference. The new approach hits schedules_unique_name and finishes in 0.1ms
| @@ -14,6 +14,8 @@ type PostSearch = Endpoint<{ | |||
| Path: typeof path; | |||
| Body: { | |||
| names?: string[] | undefined; | |||
| namePrefix?: string | undefined; | |||
There was a problem hiding this comment.
is it better to add a new explicit param or to support wildcard in the existing names one?
There was a problem hiding this comment.
No longer needed
| @@ -23,6 +25,8 @@ type PostSearch = Endpoint<{ | |||
| const bodySchema = z | |||
| .object({ | |||
| names: z.array(z.string().min(1)).optional(), | |||
| namePrefix: z.string().min(1).optional(), | |||
| state: z.enum(['STARTED', 'PAUSED', 'DELETED']).optional(), | |||
There was a problem hiding this comment.
recently found out you can use satisfies ScheduleState[] to prevent accidentally having an enum value that doesn't exist.
| @@ -114,6 +123,19 @@ export class Orchestrator { | |||
| return Ok(scheduleMap); | |||
| } | |||
|
|
|||
| async getPausedSyncsByEnvironment({ environmentId }: { environmentId: number }): Promise<Result<string[]>> { | |||
| const namePrefix = `environment:${environmentId}:sync:`; | |||
There was a problem hiding this comment.
let's expose the prefix from the orchestrator.ts to avoid duplication and make sure we only need to change it in one place if needed
There was a problem hiding this comment.
No longer needed
| @@ -114,6 +123,19 @@ export class Orchestrator { | |||
| return Ok(scheduleMap); | |||
| } | |||
|
|
|||
| async getPausedSyncsByEnvironment({ environmentId }: { environmentId: number }): Promise<Result<string[]>> { | |||
| const namePrefix = `environment:${environmentId}:sync:`; | |||
| const result = await this.client.searchSchedulesByPrefix({ namePrefix, state: 'PAUSED', limit: 10_000 }); | |||
There was a problem hiding this comment.
this limit might quickly fall apart for Linux Foundation. That's also a huge amount of data to transfer when the only thing we are interested in is a boolean.
What about a new orchestrator endpoint like GET /stats?name_prefix=... that return a count per state for now
There was a problem hiding this comment.
I changed the DB approach to only take into consideration the connections loaded on the current page, that way we have it scoped down and don't need to resolve every sync to display the icon
3b958df to
26ba24a
Compare
| if (syncs.length > 0) { | ||
| const scheduleResult = await orchestrator.searchSchedules(syncs.map((s) => ({ syncId: s.id, environmentId: environment.id }))); | ||
| if (scheduleResult.isErr()) { | ||
| res.status(500).send({ error: { code: 'server_error' } }); |
There was a problem hiding this comment.
should we log the error or have it attached to the span somehow? otherwise will be tough to know what caused the error
| const connectionIds = connections.map((data) => data.connection.id); | ||
| const syncs = await getSyncsByConnectionIds({ connectionIds }); | ||
| if (syncs.length > 0) { | ||
| const scheduleResult = await orchestrator.searchSchedules(syncs.map((s) => ({ syncId: s.id, environmentId: environment.id }))); |
There was a problem hiding this comment.
let's add some safeguard to orchestrator postSearch and max the length of names to a high value that is not infinite, so we don't happen to send a huge number by accident
| } | ||
| return db.knex.select('id', 'name', 'nango_connection_id').from<Sync>(TABLE).whereIn('nango_connection_id', connectionIds).andWhere({ deleted: false }); | ||
| }; | ||
|
|
There was a problem hiding this comment.
can we make all new functions that can fail return a Result?
Add hasPausedSyncs field to the connections list API response. Instead of fetching all syncs for the page's connections and querying orchestrator for each state, we now reverse the lookup: ask orchestrator for all PAUSED schedules in the environment first (fast path returns immediately when nothing is paused), then resolve only those sync IDs back to connection IDs. - Add partial index (name) WHERE state = 'PAUSED' on schedules table - Extend scheduler/orchestrator search to support namePrefix + state - Add Orchestrator.getPausedSyncsByEnvironment() - Add getSyncsByIds() to sync service - Add hasPausedSyncs to ApiConnectionSimple type
86880df to
77df32d
Compare
Refresh the connections list UI with improved filtering and status indicators https://www.figma.com/design/2MyZwo8A4D2XRIOhXL3cZ8/CURRENT-%F0%9F%90%BA-Design-Sytem---Product?node-id=9094-77970&m=dev [Loom demo](https://www.loom.com/share/0c3d7958981b43f59088471464dcb0ae) - Replace MultiSelect with new Combobox component (single + multi-select) - Add granular Status filter: OK, Error, Auth error, Sync error, Paused syncs - Status filter persisted in URL query params - Show hasPausedSyncs indicator per connection row - Integration filter now shows logos and "Add integration" footer CTA <!-- Describe the problem and your solution --> <!-- Issue ticket number and link (if applicable) --> Requires NangoHQ#5699 to be merged NAN-5076 <!-- Testing instructions (skip if just adding/editing providers) --> <!-- Summary by @propel-code-bot --> --- This update also standardizes filtering behavior by combining URL-backed status state with frontend status composition logic layered on top of existing backend error filtering, while introducing reusable UI building blocks intended to support richer filter hierarchies and status rendering patterns across the app. --- *This summary was automatically generated by @propel-code-bot*
Refresh the connections list UI with improved filtering and status indicators https://www.figma.com/design/2MyZwo8A4D2XRIOhXL3cZ8/CURRENT-%F0%9F%90%BA-Design-Sytem---Product?node-id=9094-77970&m=dev [Loom demo](https://www.loom.com/share/0c3d7958981b43f59088471464dcb0ae) - Replace MultiSelect with new Combobox component (single + multi-select) - Add granular Status filter: OK, Error, Auth error, Sync error, Paused syncs - Status filter persisted in URL query params - Show hasPausedSyncs indicator per connection row - Integration filter now shows logos and "Add integration" footer CTA <!-- Describe the problem and your solution --> <!-- Issue ticket number and link (if applicable) --> Requires NangoHQ#5699 to be merged NAN-5076 <!-- Testing instructions (skip if just adding/editing providers) --> <!-- Summary by @propel-code-bot --> --- This update also standardizes filtering behavior by combining URL-backed status state with frontend status composition logic layered on top of existing backend error filtering, while introducing reusable UI building blocks intended to support richer filter hierarchies and status rendering patterns across the app. --- *This summary was automatically generated by @propel-code-bot*
Add
pausedSyncsfield to the connections list API response.We want to show which connections have paused syncs on the FE when listing connections. For each page of connections returned, we fetch their syncs from the DB and do a targeted lookup to the orchestrator using exact schedule names, bounded by page size (~20 connections) rather than the full environment.
pausedSyncs: string[]toApiConnectionSimpletypegetSyncsByConnectionIds()to sync serviceOrchestrator.searchSchedules()with exact names for the current pagepausedSyncsrather than 500ing the entire list endpointThis change also formalizes the enrichment flow across shared contracts and service boundaries, including safer failure handling and tighter validation on schedule-search inputs, so paused-sync visibility is added in a resilient, bounded way without compromising the stability of the connections listing path.
This summary was automatically generated by @propel-code-bot