Skip to content

feat(lambda): lambda runtime for function execution#4998

Merged
rossmcewan merged 122 commits intomasterfrom
ross/NAN-4343/create-function-executor
Jan 21, 2026
Merged

feat(lambda): lambda runtime for function execution#4998
rossmcewan merged 122 commits intomasterfrom
ross/NAN-4343/create-function-executor

Conversation

@rossmcewan
Copy link
Contributor

@rossmcewan rossmcewan commented Nov 12, 2025


Add AWS Lambda Runtime and Multi-Fleet Orchestration for Function Execution

Introduces a Lambda-based execution path alongside the existing container runner, with a pluggable runtime selection layer driven by plan metadata and rules. Adds a dedicated @nangohq/lambda-runner package, extends fleet supervision to support multiple fleets and Lambda-specific node providers, and updates infrastructure (Redis, plans schema, deployment workflow) to support the new runtime while maintaining backward compatibility.

Key Changes

• Adds Lambda runtime adapter, node provider, and handler (packages/jobs/lib/runtime/lambda.adapter.ts, packages/jobs/lib/runner/lambda.ts, packages/lambda-runner/lib/index.ts) for invoking AWS Lambda functions asynchronously with locking, heartbeat, and abort handling.
• Introduces runtime orchestration layer (packages/jobs/lib/runtime/runtimes.ts, packages/jobs/lib/runtime/runtimes.rules.ts) that selects fleets based on JSON rules and per-plan runtime flags added via migration 20260108145100_add_function_runtime_flags.cjs and plan schema updates.
• Extends fleet management to support multiple fleets and Lambda nodes (new image verifiers, supervisor updates, fleet_id persistence, node provider API changes) and separates Redis usage into system/customer boundaries with locking enhancements (packages/kvstore/lib/index.ts, packages/kvstore/lib/Locking.ts).
• Adds new @nangohq/lambda-runner package with build pipeline (Dockerfile.lambda, tsconfig, schema validations) and updates CI/CD workflow to deploy Lambda images and invoke rollout API with image type.
• Updates job execution paths (sync/action/webhook/on-event) to pass runtime context and use new runtime adapter, while ensuring abort propagation through Redis flags and Lambda heartbeat polling.

Affected Areas

• @nangohq/jobs runtime orchestration
• @nangohq/fleet supervision and node models
• @nangohq/kvstore locking and Redis clients
• @nangohq/server fleet rollout API
• @nangohq/utils environment parsing and detection
• @nangohq/lambda-runner package and deployment workflow
• Database schemas for plans and fleet nodes


This summary was automatically generated by @propel-code-bot

@linear
Copy link

linear bot commented Nov 12, 2025

@rossmcewan rossmcewan changed the title Ross/nan 4343/create function executor feat(runtimes): create function executor Nov 12, 2025
rossmcewan and others added 18 commits November 12, 2025 16:35
<!-- Describe the problem and your solution --> 

<!-- Issue ticket number and link (if applicable) -->

<!-- Testing instructions (skip if just adding/editing providers) -->

<!-- Summary by @propel-code-bot -->

---

**Add `Vercel AI SDK` implementation guide & doc navigation tweaks**

Adds a new implementation guide demonstrating how to use Nango
tool-calling with the `Vercel AI SDK` and refactors surrounding
documentation to reference it consistently. The PR only touches `.mdx`
content and the `docs/docs.json` navigation manifest; no
application/runtime code is modified.

<details>
<summary><strong>Key Changes</strong></summary>

• Created new guide
`docs/implementation-guides/ai-tool-calling/vercel-sdk.mdx` (~80 LoC)
including full setup & usage example
• Updated navigation in `docs/docs.json`: added `vercel-sdk`, renamed
`any-llm-sdk` → `any-llm`, and re-labelled "Requests Proxy" group to
"Proxy"
• Fixed/cleaned links and comment text in existing guides
(`any-llm.mdx`, `openai-sdk.mdx`, `anthropic-sdk.mdx`, `mastra-sdk.mdx`)
• Retitled `implement-mcp-server.mdx` front-matter to shorter `MCP`
• Updated AI tool-calling use-case index to list Vercel & new "Any LLM"
link

</details>

<details>
<summary><strong>Affected Areas</strong></summary>

• `docs/implementation-guides/ai-tool-calling/*` guides
• `docs/docs.json` site navigation
• `docs/guides/use-cases/ai-tool-calling.mdx` index page

</details>

---
*This summary was automatically generated by @propel-code-bot*
Dimensions are not compatible with records and connections billing
metrics. Reverting the changes deployed on Friday

<!-- Summary by @propel-code-bot -->

---

**Remove environment/integration dimensions from records & connection
billing events**

This PR reverts last week’s dimension expansion for `records` and
`billable_connections_v2` metrics. All logic now aggregates strictly at
the `accountId` level, eliminating `environmentId`, `environmentName`,
and `integrationId` fields. Corresponding types, grouping/aggregation
logic, SQL queries, cron exporters, and unit tests are realigned to the
simplified schema.

<details>
<summary><strong>Key Changes</strong></summary>

• Updated `RecordsBillingEvent` and `ConnectionsBillingEventV2`
definitions in `packages/types/lib/billing/types.ts` to drop
`environmentId`, `environmentName`, `integrationId`.
• Refactored `BillingEventGrouping.groupingKey()` and `aggregate()` in
`packages/billing/lib/grouping.ts` to omit the removed properties.
• Simplified SQL in `ConnectionService.countMetric()`
(`packages/shared/lib/services/connection.service.ts`): now `GROUP BY
_nango_environments.account_id` only.
• Adjusted cron exporter `packages/metering/lib/crons/usage.ts` –
aggregation maps keyed by `accountId` and payloads without the removed
dimensions.
• Patched unit tests in `packages/billing/lib/grouping.unit.test.ts` to
reflect the new keys.
• Net diff: +8 / −62 lines across 5 TypeScript files.

</details>

<details>
<summary><strong>Affected Areas</strong></summary>

• `packages/types/lib/billing/types.ts`
• `packages/billing/lib/grouping.ts` & test
• `packages/shared/lib/services/connection.service.ts` (metric query)
• `packages/metering/lib/crons/usage.ts`

</details>

---
*This summary was automatically generated by @propel-code-bot*
<!-- Describe the problem and your solution --> 

<!-- Issue ticket number and link (if applicable) -->

<!-- Testing instructions (skip if just adding/editing providers) -->

<!-- Summary by @propel-code-bot -->

---

**Add `PUBSUB_PUBLISH` failure metric**

Adds Datadog metric emission when `Publisher.publish()` fails and
registers the new metric key in the shared `metrics` enum. No functional
behavior changes, only observability improvements.

<details>
<summary><strong>Key Changes</strong></summary>

• Imported `metrics` utility in `packages/pubsub/lib/publisher.ts`
• Wrapped transport call result in `res` and incremented
`metrics.increment(metrics.Types.PUBSUB_PUBLISH, 1, { subject:
event.subject, success: 'false' })` on `Err`
• Appended `PUBSUB_PUBLISH = 'nango.pubsub.publish'` to
`packages/utils/lib/telemetry/metrics.ts` enum
• Kept previous `USAGE_IS_CAPPED` entry while ensuring trailing comma
for enum syntax

</details>

<details>
<summary><strong>Affected Areas</strong></summary>

• `packages/pubsub/lib/publisher.ts`
• `packages/utils/lib/telemetry/metrics.ts`

</details>

---
*This summary was automatically generated by @propel-code-bot*
We want to show cumulative line charts for records and connections. But
the usage query to orb returns everything as periodic (you can get some
as periodic, some as cumulative). So we must create a cumulative chart
from the periodic data, and flag it for the frontend to know.

<!-- Summary by @propel-code-bot -->

---

**Add server-side generation of cumulative usage metrics for
`connections` and `records`**

Orb returns usage data as periodic time-slices only. This PR
post-processes that data on the server, converting selected metrics
(`connections`, `records`) to a running total so the front-end can
directly render cumulative charts. A new discriminator `view_mode`
(`'periodic' | 'cumulative'`) is carried on every `BillingUsageMetric`
so clients can choose the correct visualisation.

<details>
<summary><strong>Key Changes</strong></summary>

• Extended `UsageTracker.getBillingUsage()` to wrap Orb data and call
new helper `toCumulativeUsage()` when the metric maps to `connections`
or `records`.
• Introduced helper `toCumulativeUsage()` in
`packages/account-usage/lib/usage.ts` that: sorts `usage`
chronologically, accumulates quantities, sets `view_mode: 'cumulative'`
and returns transformed structure.
• Enhanced metric name mapping in `billingMetricToUsageMetric()` to
recognise `connections` and `records`.
• Updated `OrbClient.fetchUsage()` to annotate raw results with
`view_mode: 'periodic'` so downstream code can rely on the field’s
presence.
• Augmented `BillingUsageMetric` type in
`packages/types/lib/billing/types.ts` with `view_mode: 'cumulative' |
'periodic'` (replacing earlier `cumulative` boolean).

</details>

<details>
<summary><strong>Affected Areas</strong></summary>

• `packages/account-usage/lib/usage.ts`
• `packages/billing/lib/clients/orb.ts`
• `packages/types/lib/billing/types.ts`

</details>

---
*This summary was automatically generated by @propel-code-bot*
<!-- Describe the problem and your solution --> 

Adds support for mimecast email security platform

<!-- Issue ticket number and link (if applicable) -->

<!-- Testing instructions (skip if just adding/editing providers) -->

<!-- Summary by @propel-code-bot -->

---

**Introduce first-party Mimecast integration (catalog entry, docs &
assets)**

This PR adds official support for the Mimecast e-mail security platform.
All changes are static—covering provider catalog metadata, branding
assets, documentation, and auto-generated code snippets—so no runtime or
build logic is touched. The goal is to surface Mimecast in the UI/CLI
picker and documentation so customers can quickly discover and connect
the service without additional engineering work.

<details>
<summary><strong>Key Changes</strong></summary>

• packages/providers/providers.yaml – new Mimecast provider block added
to catalog
• packages/webapp/public/images/template-logos/mimecast.svg –
brand-consistent logo
• docs/integrations/all/mimecast.mdx – overview documentation
• docs/integrations/all/mimecast/connect.mdx – step-by-step connection
guide
• docs/snippets/generated/mimecast/* – auto-generated code snippets for
common use-cases

</details>

<details>
<summary><strong>Affected Areas</strong></summary>

• Provider catalog (YAML)
• Documentation site (MDX pages & generated snippets)
• Static assets (public images)

</details>

---
*This summary was automatically generated by @propel-code-bot*
@gitguardian
Copy link

gitguardian bot commented Nov 24, 2025

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
21117059 Triggered Cursor API Key 0dae829 packages/providers/providers.yaml View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

export type { KVStore } from './KVStore.js';
export { type Lock, Locking } from './Locking.js';

type KvBoundary = 'system' | 'customer';
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Introduced a boundary to define the different kinds of kvstores we might want - currently have system and customer. Open to discussion on the best way to implement this but was more complicated than I expected.

@rossmcewan rossmcewan added this pull request to the merge queue Jan 21, 2026
Merged via the queue into master with commit 7a48701 Jan 21, 2026
34 checks passed
@rossmcewan rossmcewan deleted the ross/NAN-4343/create-function-executor branch January 21, 2026 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants