• Creating mar file for an eager mode model
• Creating mar file for torchscript mode model
• Serving custom model with custom service handler
• Serving model using Docker Container
• Creating a Workflow
• Custom Metrics
• Dynamic Batch Processing
• Dynamic Batched Async Requests

• Serving HuggingFace faster transformers model in K8s

• Serving HuggingFace BERT model using KServe

• Serving HuggingFace transformers model

• Deploy models using mlflow-torchserve plugin

• Model Explainability with Captum

• Example for ONNX Integration

• Support for TensorRT optimizations

• HuggingFace Stable Diffusion Model with Microsoft DeepSpeed-MII

• Custom Metrics with mtail and Prometheus

• Boost Performance on Intel Hardware

• Serving Torchrec DLRM (Recommender Model)

• Serving Image Classifier model and loading image data using TorchData (datapipes)

• PyTorch 2.0 Integration

• Stable Diffusion using HuggingFace Diffusers

• HuggingFace Large Models with constrained resources

• Serving torchvision image classification models
• Serving Image Classifier model for on-premise near real-time video

• Serving object detection model
• Serving image segmentation model

• Serving image generator model

• Serving machine translation model
• Serving Neural Machine Translation Workflow

• Serving text classification model
• Serving text classification model with scriptable tokenizer

• Serving waveglow text to speech synthesizer model

• Serving multi modal framework model

The following are examples on how to create and serve model archives with TorchServe.

Following are the steps to create a torch-model-archive (.mar) to execute an eager mode torch model in TorchServe :

• Pre-requisites to create a torch model archive (.mar) :

  • serialized-file (.pt) : This file represents the state_dict in case of eager mode model.
  • model-file (.py) : This file contains model class extended from torch nn.modules representing the model architecture. This parameter is mandatory for eager mode models. This file must contain only one class definition extended from torch.nn.Module.
  • index_to_name.json : This file contains the mapping of predicted index to class. The default TorchServe handles returns the predicted index and probability. This file can be passed to model archiver using --extra-files parameter.
  • version : Model's version.
  • handler : TorchServe default handler's name or path to custom inference handler(.py)

• Syntax

  torch-model-archiver --model-name <model_name> --version <model_version_number> --model-file <path_to_model_architecture_file> --serialized-file <path_to_state_dict_file> --handler <path_to_custom_handler_or_default_handler_name> --extra-files <path_to_index_to_name_json_file>

Following are the steps to create a torch-model-archive (.mar) to execute an eager mode torch model in TorchServe :

• Pre-requisites to create a torch model archive (.mar) :

  • serialized-file (.pt) : This file represents the state_dict in case of eager mode model or an executable ScriptModule in case of TorchScript.
  • index_to_name.json : This file contains the mapping of predicted index to class. The default TorchServe handles returns the predicted index and probability. This file can be passed to model archiver using --extra-files parameter.
  • version : Model's version.
  • handler : TorchServe default handler's name or path to custom inference handler(.py)

• Syntax

  torch-model-archiver --model-name <model_name> --version <model_version_number> --serialized-file <path_to_executable_script_module> --extra-files <path_to_index_to_name_json_file> --handler <path_to_custom_handler_or_default_handler_name>

The following example demonstrates how to create image classifier model archive, serve it on TorchServe and run image prediction using TorchServe's default image_classifier handler :

• Image classification models

The following example demonstrates how to create and serve a custom NN model with custom handler archives in TorchServe :

• Digit recognition with MNIST

The following example demonstrates how to create and serve a custom text_classification NN model with default text_classifier handler provided by TorchServe :

• Text classification example

This example shows how to combine a text classification model with a scriptable tokenizer into a single, scripted artifact to serve with TorchServe. A scriptable tokenizer is a tokenizer compatible with TorchScript.

• Scriptable Tokenizer example with scriptable tokenizer

The following example demonstrates how to create and serve a pretrained fast-rcnn NN model with default object_detector handler provided by TorchServe :

• Object detection example

The following example demonstrates how to create and serve a pretrained fcn NN model with default image_segmenter handler provided by TorchServe :

• Image segmentation example

The following example demonstrates how to create and serve a pretrained transformer models from Huggingface such as BERT, RoBERTA, XLM

• Hugging Face Transformers

The following example demonstrates TorchServe's integration with Captum, an open source, extensible library for model interpretability built on PyTorch

• Captum

The following example demonstrates how to create and serve a pretrained DCGAN model from facebookresearch/pytorch_GAN_zoo

• GAN Image Generator

The following example demonstrates how to create and serve a neural translation model using fairseq

• Neural machine translation

The following example demonstrates how to create and serve the waveglow text to speech synthesizer

• Waveglow text to speech synthesizer

The following example demonstrates how to create and serve a multi modal model including audio, text and video

• Multi modal framework

The following example demonstrates how to create and serve a complex image classification workflow for dog breed classification

• Image classification workflow

The following example demonstrates how to create and serve a complex neural machine translation workflow

• Neural machine Translation workflow

This example shows how to deploy a Deep Learning Recommendation Model (DLRM) with TorchRec

• Torchrec DLRM

The following example demonstrates how to serve an image classification model with batching for near real-time video

• Near Real-Time Video Batched Image Classification

The following example demonstrates how to integrate TorchData with torchserve

• Torchdata integration with torchserve an image classification example