Skip to content

Running two instances of skynet against the one instance of skynet with vllm running with a model loaded #164

@ConalMullan

Description

@ConalMullan

I have been trying to find a way to run two instances of skynet which query the same vllm instance accross a docker network.
The idea is that one instance runs vllm, the other doesnt run vllm but querys vllm running on the other instance.

I can setup the docker instances and docker network so that one docker instance can see the instance with vllm running on port 8003 accross the network but I cant get the fastapi endpoints like /summaries to initiate on the instance that vllm is not running on.

Is there some way that I can acheive this?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions