Enahnce/gpustack lb policy by orangedeng · Pull Request #6250 · gpustack/gpustack

orangedeng · 2026-09-20T16:05:10Z

Refer to issues:

gemini-code-assist · 2026-09-20T16:07:38Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements a robust plugin framework for ModelRoute extensions, enabling advanced traffic management and load balancing policies. By decoupling plugin logic from the core gateway code, the system becomes more extensible. Key changes include the addition of least-load and session-affinity plugins, support for shared Redis state, and a refactoring of the AI proxy logic to be deployment-centric, ensuring more consistent and scalable gateway resource management.

Highlights

Plugin Framework: Introduced a new modular plugin framework for ModelRoute extensions, allowing for self-contained logic for load balancing and traffic management.
Load Balancing Capabilities: Implemented new capability plugins for least-load and session-affinity load balancing policies.
Shared State Support: Added support for a shared Redis backend via the --redis-url configuration to facilitate stateful load balancing across the gateway.
AI Proxy Refactoring: Transitioned AI proxy configuration to be deployment-centric, improving efficiency and rule management.
Database Schema Updates: Added necessary database migrations to support new LB configuration knobs and usage tracking columns.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Generative AI Prohibited Use Policy, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gpustack-code-review · 2026-09-20T16:09:29Z

🔍 OpenCodeReview found 10 issue(s) in this PR.

✅ Successfully posted inline: 2 comment(s)
📋 Routed to summary by policy: 8 comment(s)

📄 `gpustack/routes/plugins/least_load/config.py` (L11-L12)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category documentation)

The weight docstring says the valid range is "(0, N]" (bounded above by some N), but the Field only enforces gt=0 with no upper bound — arbitrarily large weights pass validation. Either enforce an upper bound (e.g. le=...) or reword the docstring to state the range is unbounded above.

📄 `gpustack/routes/plugins/least_load/config.py` (L23-L27)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category maintainability)

least_load_from_meta is defined but never referenced anywhere in the codebase (plugin.py uses _policy_for_route + the policy table instead). Unless it is intended as public API for external consumers, remove it or wire it in.

📄 `gpustack/routes/model_routes.py` (L1112-L1115)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category bug)

The delete paths dispatch the raw helpers (dispatch_route_hooks / dispatch_target_hooks) instead of the _dispatch_route_hooks_or_400 / _dispatch_target_hooks_or_400 wrappers. A plugin hook that raises ValueError during delete — e.g. malformed stored state, same class the wrappers map to InvalidException — will escape here as a raw ValueError, surfacing as a 500 (or an unhandled exception on the add-targets path), inconsistent with the documented 400 contract the wrapped paths establish. Consider using the wrapped variants here too.

📄 `gpustack/schemas/model_routes.py` (L150-L151)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category bug)

isinstance(v, int) is True for bool, so a client sending "max_running_requests": true passes validation and is persisted as 1. Add an explicit bool exclusion.

💡 Suggested Change

Before:

        if not isinstance(v, int):
            raise ValueError("max_running_requests must be an integer")

After:

        if isinstance(v, bool) or not isinstance(v, int):
            raise ValueError("max_running_requests must be an integer")

📄 `gpustack/routes/model_routes.py` (L597-L602)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category bug)

If the route was concurrently deleted, this returns None from a handler annotated to return a ModelRoute (FastAPI response model will fail or serialize null unexpectedly). Consider raising NotFoundException here for symmetry with one_by_id's not_found handling on the other paths.

📄 `gpustack/config/config.py` (L199-L205)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category documentation)

This comment contradicts itself: it opens with "May carry credentials" and then, in the same sentence, states credentials are refused (and so is rediss://). Since enforcement exists at materialization time (routes/plugins/lb/gateway.py refuses credentials and TLS), the field never actually carries credentials — drop the "May carry credentials" phrasing so the docstring matches the enforced contract (redis://host[:port][/db] only).

📄 `gpustack/routes/plugins/lb/capability.py` (L53-L58)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category maintainability)

The parameter full_ingress_name shadows the module-level helper full_ingress_name() defined just above. Inside this function any future call to the helper would instead reference the string parameter. Rename the parameter (e.g. full_ingress) or pre-compute the value at the call site to avoid the shadowing.

📄 `gpustack/routes/plugins/lb/gateway.py` (L61-L66)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category documentation)

The docstring says "A missing CR returns None so ensure_wasm_plugin treats it as create" — but the function actually returns expected_spec when current_spec is None (which is indeed how ensure_wasm_plugin creates the CR: spec_diff on an absent plugin receives None and the returned expected spec is written). The "returns None" wording contradicts the code and will mislead future readers; reword to "a missing CR returns the expected spec so ensure_wasm_plugin creates it".

gemini-code-assist

Code Review

This pull request introduces a modular route-plugin framework to GPUStack, adding support for load balancing, session affinity, least-load routing, and fallback paths, along with Redis configuration for shared state. The review feedback identifies several critical issues: potential ImportError exceptions from importing JSON from sqlmodel instead of sqlalchemy in the plugin schemas, a MissingGreenlet exception in model_routes.py caused by accessing lazy-loaded relationships in an async context, a failure to filter out soft-deleted targets in both the load balancer and fallback plugins, and the need to URL-decode Redis credentials to prevent authentication failures.

Copilot

Warning

Copilot's review of this pull request may be incomplete because some of the changed files are excluded by your Copilot content exclusion settings. See Excluding content from Copilot for details.

Copilot review overview

🟡 Changes recommended

Unresolved critical gateway-routing and migration issues, along with API and reconciliation defects, block approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 7 · 11

Open (18)

Preserve model mapping fallback when LB rules are removed · New
Add LB schema in a new migration revision · New
Tenant-scope target queries before plugin enrichment · New
Preserve model mapper fallback when LB is unavailable · New
Build LB clusters with registry-specific ports · New
Normalize registry weights at the target level · New
Preserve mapper fallback when LB cannot render · New
Reject unvalidated reserved plugin keys in route metadata · New
Enrich plugin sections in route create responses · New
Wire plugin watches into reconciliation subscriptions · New
Handle malformed Redis URLs without aborting startup · New
Apply LB policy settings to gateway runtime · New
Make policy writes participate in route transactions · New
Isolate plugin factory constructor failures · New
Return validation errors as client responses · New
Make policy writes participate in route transactions · New
Retire proxy rules for inactive or deleted targets · New
Handle cold-cache delete events for reconciliation · New

What changed in this PR

This PR adds a pluggable route-plugin framework and gateway load balancing with least-load/session-affinity policies, Redis configuration, schema migrations, and AI-proxy synchronization.

Changes:

Adds plugin registration, CRUD hooks, response enrichment, and fallback handling.
Implements LB candidate reconciliation and gateway integration.
Updates dependencies, API schemas, migrations, and Redis CLI/configuration.

File	Summary
`uv.lock`	Updates the Higress plugin dependency lock.
`pyproject.toml`	Updates the dependency pin.
`gpustack/worker/model_file_manager.py`	Formatting cleanup.
`gpustack/websocket_proxy/proxy_server.py`	Formatting cleanup.
`gpustack/utils/task.py`	Formatting cleanup.
`gpustack/utils/gpu.py`	Formatting cleanup.
`gpustack/server/server.py`	Adjusts gateway cleanup integration.
`gpustack/server/controllers.py`	Adds plugin reconciliation and AI-proxy synchronization.
`gpustack/schemas/model_routes.py`	Adds LB fields and plugin payloads.
`gpustack/routes/plugins/session_affinity/schemas.py`	Adds session-affinity schemas.
`gpustack/routes/plugins/session_affinity/plugin.py`	Implements session-affinity hooks and capability handling.
`gpustack/routes/plugins/session_affinity/config.py`	Adds session-affinity configuration.
`gpustack/routes/plugins/session_affinity/__init__.py`	Exposes the session-affinity plugin.
`gpustack/routes/plugins/registry.py`	Adds plugin discovery and dispatch.
`gpustack/routes/plugins/least_load/schemas.py`	Adds least-load schemas.
`gpustack/routes/plugins/least_load/plugin.py`	Implements least-load policy hooks.
`gpustack/routes/plugins/least_load/config.py`	Adds least-load configuration.
`gpustack/routes/plugins/least_load/__init__.py`	Exposes the least-load plugin.
`gpustack/routes/plugins/lb/schemas.py`	Adds LB schemas.
`gpustack/routes/plugins/lb/reconciler.py`	Reconciles LB candidates and gateway rules.
`gpustack/routes/plugins/lb/plugin.py`	Integrates LB policy and capability handling.
`gpustack/routes/plugins/lb/gateway.py`	Generates LB gateway resources.
`gpustack/routes/plugins/lb/config.py`	Adds LB configuration.
`gpustack/routes/plugins/lb/capability.py`	Adds LB capability support.
`gpustack/routes/plugins/lb/__init__.py`	Exposes the LB plugin.
`gpustack/routes/plugins/fallback/plugin.py`	Adds fallback routing behavior.
`gpustack/routes/plugins/fallback/__init__.py`	Exposes the fallback plugin.
`gpustack/routes/plugins/base.py`	Defines the plugin framework base.
`gpustack/routes/plugins/artifacts.py`	Collects plugin gateway artifacts.
`gpustack/routes/plugins/__init__.py`	Initializes the plugin package.
`gpustack/routes/model_routes.py`	Integrates plugin CRUD and response handling.
`gpustack/migrations/versions/2026_09_11_1000-f1a2b3c4d5e6_v2_3_0_rc1_post_database_changes.py`	Adds LB and plugin schema changes.
`gpustack/migrations/versions/2026_09_11_1000-f1a2b3c4d5e6_usage_details_request_ids_and_ttft.py`	Updates the usage-details migration bundle.
`gpustack/gateway/utils.py`	Changes mapper and AI-proxy reconciliation.
`gpustack/gateway/__init__.py`	Publishes plugin gateway resources and Redis registries.
`gpustack/config/config.py`	Adds Redis configuration.
`gpustack/cmd/start.py`	Adds Redis CLI and environment wiring.

Files excluded by content exclusion policy (13)

tests/gateway/test_ai_proxy_model_grouping.py
tests/gateway/test_gateway_plugins.py
tests/gateway/test_gateway_utils.py
tests/routes/plugins/init.py
tests/routes/plugins/capabilities/init.py
tests/routes/plugins/capabilities/test_capability_plugins.py
tests/routes/plugins/fallback/init.py
tests/routes/plugins/fallback/test_fallback.py
tests/routes/plugins/lb/init.py
tests/routes/plugins/lb/test_lb.py
tests/routes/plugins/test_artifacts.py
tests/routes/plugins/test_registry.py
tests/server/test_route_ai_proxy_grouping.py

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

orangedeng · 2026-09-21T01:57:12Z

/gpustack-code-review

Introduce a plugin framework for model-route gateway concerns: a base plugin interface, a registry that discovers and orders plugins, and a declarative artifact/rule collector that plugins use to declare the matchRules and gateway entries they own instead of writing them behind each other

Move the mapper fallback path out of the generic gateway writer into a route plugin that declares its own matchRules via the rule collector, and reconcile the route gateway when fallback codes change

The lb plugin derives the load-balancing mode from route annotations, excludes fallback targets from candidates, keeps the ingress alive for all-zero-weight routes, and owns its gateway entries via the rule collector. Wire --redis-url for its shared-state backend, bump gpustack-higress-plugins, and batch per-route plugin reconcile in the model controller

…el names on the mapper CR Build ai-proxy entries per deployment from every referencing route with service-only match rules, route model names through the mapper CR instead of alias services, and stop the mapper and LB writers from deleting each other

…_rc1 bundle The standalone usage-details revision is unreleased, so its changes belong in the current release bundle revision instead of a separate file

thxCode · 2026-09-21T06:02:56Z

/gpustack-code-review

Copilot

Warning

Copilot's review of this pull request may be incomplete because some of the changed files are excluded by your Copilot content exclusion settings. See Excluding content from Copilot for details.

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate issues remain in reconciliation, plugin state handling, fallback routing, and gateway security.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 11 · 13

Open (24)

And 4 more that still need to be addressed.

Resolved since last review (4)

Files excluded by content exclusion policy (14)

tests/gateway/test_ai_proxy_model_grouping.py
tests/gateway/test_gateway_plugins.py
tests/gateway/test_gateway_utils.py
tests/routes/plugins/init.py
tests/routes/plugins/capabilities/init.py
tests/routes/plugins/capabilities/test_capability_plugins.py
tests/routes/plugins/fallback/init.py
tests/routes/plugins/fallback/test_fallback.py
tests/routes/plugins/lb/init.py
tests/routes/plugins/lb/test_lb.py
tests/routes/plugins/test_artifacts.py
tests/routes/plugins/test_registry.py
tests/server/test_changed_scalar.py
tests/server/test_route_ai_proxy_grouping.py

Plugin section handling and response shape: - route responses carry plugin sections with derived storage keys stripped, so a client round-tripping them cannot re-persist or delete stored policies; an explicit meta update preserves plugin-owned keys from the existing row - enqueue the ai-proxy reconcile for created model targets on both creation paths (they carry no UPDATED event of their own) - enrich the /set-fallback response with target plugin sections - surface a malformed plugins section as 400 instead of 500 Gateway and reconcile: - make ai-proxy candidate cluster names carry the registry real port (dns instance registries and provider registries do not use 80) - refuse credentialed redis urls instead of materializing the password in the WasmPlugin CR; default the static registry domain to 6379 - gate lb candidate rendering on the gpustack-lb module actually resolving, so the degraded plain-mapper CR keeps the legacy rewrite - skip creating a WasmPlugin whose spec no owner can author - publish the gpustack-model-mapper CR unconditionally so model-name rewrite works on a fresh cluster Storage and events: - move the lb schema to its own migration revision, ordered after the usage-details revision - hard-delete capability policy rows inside the caller transaction (unique route_id + soft delete broke remove-then-re-add) - reject an empty sessionKeys chain; keep plugin-named keys in model meta from forging route plugin state - normalize changed-field scalars (tuple or list shaped) across both event paths so model_id change notifications carry real ids

orangedeng requested a lite review from Copilot September 20, 2026 16:05

Copilot started reviewing on behalf of orangedeng September 20, 2026 16:05 View session

gpustack-code-review Bot reviewed Sep 20, 2026

View reviewed changes

gemini-code-assist Bot reviewed Sep 20, 2026

View reviewed changes

Copilot AI reviewed Sep 20, 2026

View reviewed changes

gpustack-code-review Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread gpustack/routes/model_routes.py

Comment thread gpustack/server/controllers.py Outdated

Comment thread gpustack/server/controllers.py Outdated

gpustack-code-review Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread gpustack/routes/model_routes.py

orangedeng added 7 commits September 21, 2026 11:20

feat(routes): add session-affinity and least-load capability plugins

c18a25b

feat(routes): handle fallback targets through a dedicated route plugin

9519ee1

Move the mapper fallback path out of the generic gateway writer into a route plugin that declares its own matchRules via the rule collector, and reconcile the route gateway when fallback codes change

refactor(migrations): fold the usage-details revision into the v2_3_0…

57fe1a7

…_rc1 bundle The standalone usage-details revision is unreleased, so its changes belong in the current release bundle revision instead of a separate file

style: formatter-only cleanups

619d2ab

orangedeng force-pushed the enahnce/gpustack-lb-policy branch from cf54d85 to 51f5586 Compare September 21, 2026 05:48

orangedeng requested a lite review from Copilot September 21, 2026 05:54

Copilot started reviewing on behalf of orangedeng September 21, 2026 05:54 View session

Copilot AI reviewed Sep 21, 2026

View reviewed changes

orangedeng mentioned this pull request Sep 21, 2026

feat(model-routes): add load-balancing policy configuration gpustack/gpustack-ui#1401

Open

gpustack-code-review Bot reviewed Sep 21, 2026

View reviewed changes

orangedeng force-pushed the enahnce/gpustack-lb-policy branch from 51f5586 to f459476 Compare September 21, 2026 06:26

gpustack-code-review Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread gpustack/gateway/__init__.py Outdated

Comment thread gpustack/migrations/versions/2026_09_21_1200-b4c5d6e7f8a9_v2_3_0_lb_route_plugins.py Outdated

Comment thread gpustack/routes/model_routes.py Outdated

Comment thread gpustack/routes/plugins/lb/gateway.py

orangedeng force-pushed the enahnce/gpustack-lb-policy branch from f459476 to cb53763 Compare September 21, 2026 06:42

gpustack-code-review Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread gpustack/routes/model_routes.py Outdated

Comment thread gpustack/routes/model_routes.py Outdated

Comment thread gpustack/routes/plugins/fallback/plugin.py

orangedeng force-pushed the enahnce/gpustack-lb-policy branch from cb53763 to 2efbf3d Compare September 21, 2026 06:55

gpustack-code-review Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread gpustack/routes/model_routes.py Outdated

Comment thread gpustack/routes/plugins/lb/gateway.py Outdated

Comment thread gpustack/server/controllers.py

orangedeng force-pushed the enahnce/gpustack-lb-policy branch from 2efbf3d to e0b0420 Compare September 21, 2026 07:22

gpustack-code-review Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread gpustack/routes/model_routes.py

Comment thread gpustack/server/controllers.py

orangedeng force-pushed the enahnce/gpustack-lb-policy branch from e0b0420 to fcc8d3b Compare September 21, 2026 07:47

gpustack-code-review Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread gpustack/server/controllers.py Outdated

orangedeng force-pushed the enahnce/gpustack-lb-policy branch from fcc8d3b to f388602 Compare September 21, 2026 07:59

gpustack-code-review Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread gpustack/routes/model_routes.py

Comment thread gpustack/routes/plugins/fallback/plugin.py

Comment thread gpustack/server/controllers.py Outdated

orangedeng force-pushed the enahnce/gpustack-lb-policy branch from f388602 to 13c881e Compare September 21, 2026 08:14

gpustack-code-review Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread gpustack/routes/model_routes.py Outdated

Comment thread gpustack/routes/model_routes.py

orangedeng force-pushed the enahnce/gpustack-lb-policy branch from 13c881e to faf44e4 Compare September 21, 2026 08:57

gpustack-code-review Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread gpustack/routes/model_routes.py

Comment thread gpustack/routes/plugins/lb/gateway.py

Conversation

orangedeng commented Sep 20, 2026

Uh oh!

gemini-code-assist Bot commented Sep 20, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gpustack-code-review Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📄 gpustack/routes/plugins/least_load/config.py (L11-L12)

📄 gpustack/routes/plugins/least_load/config.py (L23-L27)

📄 gpustack/routes/model_routes.py (L1112-L1115)

📄 gpustack/schemas/model_routes.py (L150-L151)

📄 gpustack/routes/model_routes.py (L597-L602)

📄 gpustack/config/config.py (L199-L205)

📄 gpustack/routes/plugins/lb/capability.py (L53-L58)

📄 gpustack/routes/plugins/lb/gateway.py (L61-L66)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

orangedeng commented Sep 21, 2026

Uh oh!

thxCode commented Sep 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gpustack-code-review Bot commented Sep 20, 2026 •

edited

Loading

📄 `gpustack/routes/plugins/least_load/config.py` (L11-L12)

📄 `gpustack/routes/plugins/least_load/config.py` (L23-L27)

📄 `gpustack/routes/model_routes.py` (L1112-L1115)

📄 `gpustack/schemas/model_routes.py` (L150-L151)

📄 `gpustack/routes/model_routes.py` (L597-L602)

📄 `gpustack/config/config.py` (L199-L205)

📄 `gpustack/routes/plugins/lb/capability.py` (L53-L58)

📄 `gpustack/routes/plugins/lb/gateway.py` (L61-L66)