Skip to content
This repository was archived by the owner on Aug 7, 2025. It is now read-only.
This repository was archived by the owner on Aug 7, 2025. It is now read-only.

Worker unload/start loop #2428

Open
Open
Worker unload/start loop#2428
Labels
supporttriagedIssue has been reviewed and triaged
@sreeprasannar

Description

@sreeprasannar
sreeprasannar
opened on Jun 23, 2023

Why would torchserve workers restart over and over?
I'm using torchserve docker image version 0.8.0

2023-06-20T23:15:44,435 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - [PID]2899
2023-06-20T23:15:44,436 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Torch worker started.
2023-06-20T23:15:44,436 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Python runtime: 3.9.16
2023-06-20T23:15:44,438 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.
2023-06-20T23:15:46,800 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - s_name_part0=/home/model-server/tmp/.ts.sock, s_name_part1=9000, pid=2906
2023-06-20T23:15:46,801 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000
2023-06-20T23:15:46,810 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Successfully loaded /app/config/metrics.yaml.
2023-06-20T23:15:46,810 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - [PID]2906
2023-06-20T23:15:46,810 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Torch worker started.
2023-06-20T23:15:46,810 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Python runtime: 3.9.16
2023-06-20T23:15:46,811 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.
2023-06-20T23:15:50,202 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - s_name_part0=/home/model-server/tmp/.ts.sock, s_name_part1=9000, pid=2912
2023-06-20T23:15:50,203 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000
2023-06-20T23:15:50,211 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Successfully loaded /app/config/metrics.yaml.
2023-06-20T23:15:50,212 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - [PID]2912
2023-06-20T23:15:50,212 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Torch worker started.
2023-06-20T23:15:50,212 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Python runtime: 3.9.16
2023-06-20T23:15:50,212 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.
2023-06-20T23:15:54,592 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - s_name_part0=/home/model-server/tmp/.ts.sock, s_name_part1=9000, pid=2918
2023-06-20T23:15:54,593 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000
2023-06-20T23:15:54,604 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Successfully loaded /app/config/metrics.yaml.
2023-06-20T23:15:54,604 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - [PID]2918
2023-06-20T23:15:54,604 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Torch worker started.
2023-06-20T23:15:54,604 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Python runtime: 3.9.16
2023-06-20T23:15:54,605 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.
2023-06-20T23:16:00,982 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - s_name_part0=/home/model-server/tmp/.ts.sock, s_name_part1=9000, pid=2924
2023-06-20T23:16:00,983 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000
2023-06-20T23:16:00,992 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Successfully loaded /app/config/metrics.yaml.
2023-06-20T23:16:00,992 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - [PID]2924
2023-06-20T23:16:00,992 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Torch worker started.
2023-06-20T23:16:00,992 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Python runtime: 3.9.16
2023-06-20T23:16:00,993 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.
2023-06-20T23:16:10,402 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - s_name_part0=/home/model-server/tmp/.ts.sock, s_name_part1=9000, pid=2930
2023-06-20T23:16:10,403 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000
2023-06-20T23:16:10,412 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Successfully loaded /app/config/metrics.yaml.
2023-06-20T23:16:10,412 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - [PID]2930
2023-06-20T23:16:10,412 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Torch worker started.
2023-06-20T23:16:10,412 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Python runtime: 3.9.16
2023-06-20T23:16:10,413 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.
2023-06-20T23:16:24,789 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - s_name_part0=/home/model-server/tmp/.ts.sock, s_name_part1=9000, pid=2936
2023-06-20T23:16:24,790 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000
2023-06-20T23:16:24,799 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Successfully loaded /app/config/metrics.yaml.
2023-06-20T23:16:24,799 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - [PID]2936
2023-06-20T23:16:24,799 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Torch worker started.
2023-06-20T23:16:24,799 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Python runtime: 3.9.16
2023-06-20T23:16:24,799 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.
2023-06-20T23:16:47,198 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - s_name_part0=/home/model-server/tmp/.ts.sock, s_name_part1=9000, pid=2942
2023-06-20T23:16:47,199 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000
2023-06-20T23:16:47,208 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Successfully loaded /app/config/metrics.yaml.
2023-06-20T23:16:47,208 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - [PID]2942
2023-06-20T23:16:47,208 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Torch worker started.
2023-06-20T23:16:47,208 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Python runtime: 3.9.16
2023-06-20T23:16:47,209 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.
2023-06-20T23:17:22,596 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - s_name_part0=/home/model-server/tmp/.ts.sock, s_name_part1=9000, pid=2948
2023-06-20T23:17:22,597 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000
2023-06-20T23:17:22,606 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Successfully loaded /app/config/metrics.yaml.
2023-06-20T23:17:22,606 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - [PID]2948
2023-06-20T23:17:22,606 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Torch worker started.
2023-06-20T23:17:22,606 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Python runtime: 3.9.16
2023-06-20T23:17:22,607 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.
2023-06-20T23:18:18,973 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - s_name_part0=/home/model-server/tmp/.ts.sock, s_name_part1=9000, pid=2954
2023-06-20T23:18:18,973 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000
2023-06-20T23:18:18,982 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Successfully loaded /app/config/metrics.yaml.
2023-06-20T23:18:18,983 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - [PID]2954
2023-06-20T23:18:18,983 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Torch worker started.
2023-06-20T23:18:18,983 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Python runtime: 3.9.16
2023-06-20T23:18:18,983 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.
2023-06-20T23:19:49,389 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - s_name_part0=/home/model-server/tmp/.ts.sock, s_name_part1=9000, pid=2961
2023-06-20T23:19:49,389 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000
2023-06-20T23:19:49,398 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Successfully loaded /app/config/metrics.yaml.
2023-06-20T23:19:49,398 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - [PID]2961
2023-06-20T23:19:49,398 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Torch worker started.
2023-06-20T23:19:49,398 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Python runtime: 3.9.16
2023-06-20T23:19:49,399 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.
2023-06-20T23:22:14,772 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - s_name_part0=/home/model-server/tmp/.ts.sock, s_name_part1=9000, pid=2993
2023-06-20T23:22:14,773 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000
2023-06-20T23:22:14,782 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Successfully loaded /app/config/metrics.yaml.
2023-06-20T23:22:14,782 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - [PID]2993
2023-06-20T23:22:14,782 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Torch worker started.
2023-06-20T23:22:14,782 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Python runtime: 3.9.16
2023-06-20T23:22:14,783 [INFO ] W-9000-auto_categorization_0.1-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.

Metadata

Metadata

Assignees

No one assigned

    Labels

    supporttriagedIssue has been reviewed and triaged

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions