Directory
Templates, workflows and recipes you can plug straight into Ellf – for annotation, training, evaluation, LLMs and more.
Complete end-to-end project workflows with plans, tasks, actions and data, ready to import and adapt.
Evaluate moving an LLM-based classifier to Jev, which answers with calibrated probabilities instead of text, and carry the move out. Compares Jev with the LLM on past records, has a person annotate a blind sample of disagreements and random records in about an hour, uses the annotations to add rules and choose per-label thresholds, and exports a configuration to run Jev in production, with optional checks for drift.
Measure a classifier that's built on an LLM prompt against people, without changing it. An action reads its logged decisions, sampling by label, or runs it over a random sample of texts, and saves its answers as one more annotator; people label the same texts blind after a short pilot, the built-in review settles every disagreement, and the annotation dashboard shows how often it agrees with the settled labels and which labels it confuses.
Find out whether Jev is good enough to route IT support tickets to the right technician by problem area. Settles the label scheme in a pilot with a Gemini second opinion, builds a human-labelled evaluation set with an agreement check and a blind slip check, scores Jev against a bar agreed in advance, and sets up daily routing with a monthly problem-area report.
Extract products, features, error codes, versions and integrations from customer support tickets and route each ticket to the right queue. An LLM drafts the entity annotations, people curate them, a pretrained pipeline suggests the generic entity types, and the passes are adjudicated into a gold set that trains a custom spaCy NER pipeline and a separate routing classifier.
Train a binary text classifier that predicts whether a GitHub issue is about documentation. Annotate a single label with accept or reject on the included GitHub issues or your own, then train a spaCy pipeline on the answers.
Train a named entity recognizer that finds food ingredients in cooking comments from Reddit or your own texts. Highlight ingredients with match patterns as suggestions, train a quick first model on CPU and a more accurate transformer-based pipeline on GPU, and optionally correct the first model's suggestions in a second round and check the annotations in a dashboard.
Annotation tasks for humans and agents: text, PDFs, images, audio, review and more.
Review annotations from multiple annotators, resolve conflicts and disagreements, and save one final gold-standard annotation per example.
Label regions in audio files (e.g. speakers or sound events), classify recordings into categories, or transcribe them.
Annotate coreference, i.e. links of ambiguous mentions like "her" or "the woman" back to an antecedent providing more context about the entity in question.
Browse an input asset or dataset and accept or reject examples, e.g. to filter data or check annotations before training.
Annotate syntactic dependencies between words, starting from a trained spaCy model's predictions.
Accept or reject a label on a stream of generated examples.
Draw bounding boxes, polygons or freehand segments on images (object detection, segmentation), or classify images into categories.
Annotate named entities (NER) like people, organizations, places or products, manually or with suggestions from a spaCy model or match patterns.
Annotate part-of-speech (POS) tags like verbs or nouns, starting from a trained spaCy model's tagger.
Annotate bounding boxes and segments on PDF pages rendered as images, e.g. tables, figures or signatures.
Run Tesseract OCR on the bounding boxes annotated with PDF Image Annotation and correct the extracted text.
Annotate text spans, e.g. named entities, in PDFs pre-processed into a Layout asset by the Extract PDF layout action.
Annotate relations between entities, spans or tokens. Also supports joint span and relation annotation.
Correct a spaCy model's sentence boundaries to create gold data for sentence segmentation.
Annotate overlapping and nested text spans (spancat), manually or with suggestions from a spaCy model or match patterns.
Bootstrap a terminology list (gazetteer) from spaCy word vectors. Terminology lists can be converted into match patterns to pre-select entity spans during annotation.
Assign categories to whole texts or sentences, e.g. topic, sentiment or intent. Single- or multi-label, optionally with suggestions from a spaCy model.
Label regions in video files, classify videos into categories, or transcribe them.
Jobs that run on your cluster: training, data import and export, LLM predictions and more.
Run inference with a trained spacy-curated-transformers model over an input dataset or asset, writing predictions to an output dataset.
Get text classification predictions from an OpenAI model, set up in a spacy-llm config, and save them to a dataset.
Label texts with TypeSafe's Jev and keep the answers it's confident about. Exclude the output dataset from a text classification task to leave the rest to people.
Copy or merge annotated data into a new dataset.
Write a small text file to the given path and register it as an Input asset, to try out actions and assets on a cluster.
Download a model checkpoint from the Hugging Face Hub into the cluster's shared cache, so agents and training start without downloading it again.
Download and install one or more spaCy models to shared storage so they can be loaded with spacy.load().
Extract text and layout from a PDF asset with spacy-layout and Docling, and save as a Layout asset for fast annotation with PDF Span Annotation.
Fine-tune a Hugging Face transformers model for NER or text classification on your annotations (CPU). The result can run as an annotation agent or be pushed to the Hub.
Fine-tune a Hugging Face transformers model for NER or text classification on your annotations on a GPU worker, for models too large to train on CPU.
Generate diverse synthetic examples from a plain-English task description using Google Gemini Flash, write them as a JSONL file and register the result as an Input asset on the cluster.
Print 'hello world'.
Download a dataset split from the Hugging Face Hub and save it as an Input asset, ready to annotate.
Render a published PDF asset's pages into images with pypdfium2, making it ready to annotate. Run via ellf publish data --kind pdf --render, or directly against an already-published asset.
Convert a dataset's examples from before structured storage to the structured format. The originals are backed up as a JSONL asset on your cluster first, then removed from the dataset.
Publish an annotated dataset to a Hugging Face dataset repo, private by default.
Publish a trained spaCy pipeline or fine-tuned Hugging Face model to a model repo, private by default.
Train a spaCy pipeline on your annotations: named entities, spans, text classification, tagging, parsing, sentence segmentation or coreference.
Train a spaCy pipeline using spacy-curated-transformers for GPU-accelerated training and inference, loading BERT, RoBERTa, CamemBERT or XLM-RoBERTa weights from a Hugging Face checkpoint.
Autonomous workers and annotators that can be assigned to tasks, powered by an LLM or other model.
Annotation agent powered by Google Gemini. Annotates named entities (NER), spans, part-of-speech tags and text categories, and accepts or rejects binary suggestions.
Accept every task it is given.
Annotation agent powered by an LLM on Hugging Face Inference Providers, e.g. Llama. Annotates named entities (NER), spans, part-of-speech tags and text categories, and accepts or rejects binary suggestions.
Annotation agent running a fine-tuned Hugging Face transformers model on your cluster. Annotates named entities (NER) and spans with a token-classification model, or text categories with a text-classification model.
Deterministic annotation agent for tests and development. Annotates named entities and spans in manual NER and span categorization tasks, using spaCy match patterns.
Long-running apps on your cluster: dashboards, notebooks, MCP servers, webhooks, APIs and more.
Live Streamlit dashboard for an annotation dataset: label distribution, conflicting annotations, a browsable table of every annotation, and health checks. Opens in the browser for logged-in project members.
Serve a 'hello world' HTTP endpoint.
Run editable marimo notebooks on your cluster, with access to the Prodigy database and authentication handled by Ellf. Opens in the browser for logged-in project members.
Source and install steps on GitHub
spaCy NER pattern generation as an MCP server. Uses Google Gemini to propose and iteratively refine spaCy matcher patterns given example text and a description of the entity types.
Build and share your own
Everything in this directory is a starting point. Fork a recipe to adjust it, bundle your project as a template, or ask your coding assistant to write a fully custom workflow for your data.