Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion ingestion/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,27 @@ All notable changes to `zep-ingest` are documented here. The project follows
[Semantic Versioning](https://semver.org); while at `0.x` the public API may
still change between minor versions.

## 0.1.0 (unreleased)
## 0.2.0

**Breaking:** Zep assigns node and fact UUIDs server-side. Matches the API
change that rejects caller-supplied node identity and ignores caller-supplied
fact identity.

- `NodeItem` no longer accepts a client `uuid`, and `ingest_nodes` no longer
has `require_uuids`. Zep assigns node identities and returns them on
`IngestResult.node_uuids` (parallel to the submitted nodes, with `None` for
failed batches; also recovered from completed task params when resuming).
- `FactTriple` no longer accepts `fact_uuid`; after `wait()`/`refresh()`,
assigned fact identities land on `IngestResult.edge_uuids` from task params
(parallel to submitted triples, with `None` for a terminal task that
assigned none).
- `source_node_uuid` / `target_node_uuid` remain caller-supplied pins to
existing nodes.
- JSON row files that still include `uuid` / `fact_uuid` raise a clear
ConfigurationError naming the retired field.
- Requires `zep-cloud>=3.27.0`.

## 0.1.0

First release — everything upstream of the Zep API for getting unstructured and
structured data into Context Graphs correctly.
Expand Down
52 changes: 35 additions & 17 deletions ingestion/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -374,18 +374,22 @@ graph):
2. Set the ontology before any data flows: `client.graph.set_ontology(...)`,
scoped with `graph_ids=`/`user_ids=` (or project-wide by omitting both). It is
not retroactive, and the ingestion package never sets it for you.
3. Optionally connect fact triples to existing canonical entities by pinning
endpoints with `source_node_uuid`/`target_node_uuid`. Extraction dedups
against the existing graph, so known entities anchor resolution.
4. Ingest the corpus with real `created_at` timestamps and alias
3. Optionally seed canonical entities with `ingest_nodes` and keep
`result.node_uuids` (Zep assigns them; do not supply a client UUID). Use
`client.graph.node.update` with those UUIDs for later edits — `add_nodes`
always creates new nodes.
4. Optionally connect fact triples to those entities by pinning endpoints with
`source_node_uuid`/`target_node_uuid`. Extraction dedups against the existing
graph, so known entities anchor resolution.
5. Ingest the corpus with real `created_at` timestamps and alias
canonicalization, then block on the bound result with `result.wait(...)`.

## Fact triples

```python
from zep_ingest import FactTriple, ingest_fact_triples

ingest_fact_triples(
result = ingest_fact_triples(
client,
[
FactTriple(
Expand All @@ -395,17 +399,23 @@ ingest_fact_triples(
source_node_labels=["Person"], # ties the node to a declared type
target_node_name="GTM analytics",
target_node_labels=["Project"],
# Optional: pin endpoints to UUIDs from ingest_nodes / a prior read.
# source_node_uuid=..., target_node_uuid=...,
valid_at="2024-06-15T00:00:00Z",
),
],
graph_id="org",
)
result.wait(timeout=600)
# Zep assigns the fact UUID; it lands in task params as edge_uuid after completion.
# Parallel to the submitted triples — failed tasks leave None in that slot.
result.edge_uuids
```

Triples skip extraction entirely, so nodes they create are **untyped unless
you label them** — pass `source_node_labels`/`target_node_labels` (one
declared entity type each) or the declared ontology never touches a
triples-only graph.
triples-only graph. Do not supply a `fact_uuid`; Zep owns fact identity.

Every documented limit (fact ≤250 chars, names ≤50, summaries ≤500,
SCREAMING_SNAKE_CASE `fact_name`, string attribute and metadata keys whose
Expand All @@ -424,21 +434,27 @@ without relationships — `ingest_nodes` adds them directly via
```python
from zep_ingest import NodeItem, ingest_nodes

ingest_nodes(
result = ingest_nodes(
client,
[
NodeItem(name="Ana Azimova", label="Person", uuid="…"),
NodeItem(name="GTM analytics", label="Project", uuid="…"),
NodeItem(name="Ana Azimova", label="Person"),
NodeItem(name="GTM analytics", label="Project"),
],
graph_id="org",
)
# Zep assigns each node's UUID — keep them for updates and fact-triple pinning.
result.node_uuids
```

Pass a persisted UUIDv4 per node (required by default): it is the node's only
identity/dedup key, so a re-run upserts instead of duplicating. Up to 100 nodes
per request, every documented limit (name ≤50, summary ≤500, label ≤100, ≤10
attributes, ≤10 metadata keys, each value a scalar or an array of scalars)
validated client-side at construction.
Zep assigns node UUIDs; do not supply them (a client `uuid` is rejected).
`result.node_uuids` is parallel to the submitted list — each success carries
the assigned UUID, and a failed batch leaves `None` in those slots so a later
success cannot shift under `zip`. Direct `add_nodes` calls create new nodes
each time (no name or UUID upsert) — use `client.graph.node.update` with a
returned UUID to rename, replace a summary, edit attributes, or clear an
entity type. Up to 100 nodes per request, every documented limit (name ≤50,
summary ≤500, label ≤100, ≤10 attributes, ≤10 metadata keys, each value a
scalar or an array of scalars) validated client-side at construction.
Sequential only (the Batch API doesn't take direct nodes).

## Monitoring a run
Expand Down Expand Up @@ -472,9 +488,11 @@ response = search_when_ready(client, "who runs the pilot?", graph_id="g1")
Partial failures never crash a run: pages/episodes that keep failing are
recorded as `AddError`s (indices and API messages only — never episode content)
and the run continues. `batch_ids` / `episode_uuids` / `task_ids` are the
resume handles. Task IDs are used by asynchronous operations such as fact
triples, direct node creation, and sequential thread submissions, and `wait()`
polls them through `client.task`.
resume handles; `node_uuids` / `edge_uuids` record identities Zep assigned on
`ingest_nodes` and completed `ingest_fact_triples` tasks (`None` slots mark
failures so later successes stay zip-aligned). Task IDs are used by
asynchronous operations such as fact triples, direct node creation, and
sequential thread submissions, and `wait()` polls them through `client.task`.

If the API accepts a task-backed submission without returning a completion
handle, the result reports `status == "untracked"` instead of claiming success.
Expand Down
4 changes: 2 additions & 2 deletions ingestion/pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "zep-ingest"
version = "0.1.0"
version = "0.2.0"
description = "Bulk data ingestion pipeline for Zep: chunk, contextualize, canonicalize, and submit unstructured and structured data"
readme = "README.md"
requires-python = ">=3.11"
Expand All @@ -18,7 +18,7 @@ classifiers = [
"Typing :: Typed",
]
dependencies = [
"zep-cloud>=3.25.0",
"zep-cloud>=3.27.0",
# Transport errors are classified for retry; zep-cloud's own floor.
"httpx>=0.21.2",
]
Expand Down
16 changes: 15 additions & 1 deletion ingestion/src/zep_ingest/_io.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
"""

import json
from collections.abc import Mapping
from dataclasses import MISSING, fields
from pathlib import Path
from typing import Any
Expand Down Expand Up @@ -50,7 +51,12 @@ def load_rows(path: Path) -> list[dict[str, Any]]:
raise ConfigurationError(f"{path.name} must contain JSON objects, not {type(parsed).__name__}.")


def rows_to_fields(rows: list[dict[str, Any]], row_type: type) -> list[dict[str, Any]]:
def rows_to_fields(
rows: list[dict[str, Any]],
row_type: type,
*,
retired_fields: Mapping[str, str] | None = None,
) -> list[dict[str, Any]]:
"""Validate row shapes against a dataclass and retain supplied fields exactly.

Unknown columns are rejected because silently dropping a misspelled public
Expand All @@ -59,6 +65,11 @@ def rows_to_fields(rows: list[dict[str, Any]], row_type: type) -> list[dict[str,
``name`` or ``created_at`` names the field and the row rather than surfacing
as a bare TypeError from the dataclass constructor. Both the allowed and the
required sets are read off the dataclass, so neither can drift from it.

``retired_fields`` maps former public field names to an actionable error
message (for example server-owned identity fields that must not be supplied).
Those are checked before the generic unknown-field path so the message can
name the replacement rather than listing expected columns.
"""
spec = fields(row_type)
allowed = frozenset(field.name for field in spec)
Expand All @@ -67,10 +78,13 @@ def rows_to_fields(rows: list[dict[str, Any]], row_type: type) -> list[dict[str,
for field in spec
if field.default is MISSING and field.default_factory is MISSING
]
retired = dict(retired_fields or {})
validated: list[dict[str, Any]] = []
for index, row in enumerate(rows):
if not isinstance(row, dict):
raise ConfigurationError(f"Row {index} must be a JSON object, got {type(row).__name__}")
for name in sorted(set(row) & retired.keys()):
raise ConfigurationError(f"Row {index}: {retired[name]}")
unknown = sorted(set(row) - allowed)
if unknown:
raise ConfigurationError(
Expand Down
102 changes: 48 additions & 54 deletions ingestion/src/zep_ingest/nodes.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
"""Batch node seeding for canonical entities, independent of episode extraction."""

import uuid as uuid_module
from collections import Counter
from collections.abc import Iterable
from dataclasses import dataclass
from pathlib import Path
Expand Down Expand Up @@ -34,17 +32,16 @@
class NodeItem:
"""One canonical entity node, validated against the batch-node API limits.

A supplied ``uuid`` is stored in its canonical spelling, so one UUID
written two ways — differing case, braces, no hyphens — stays one identity
both in the uniqueness check and on the wire.
Zep assigns each node's UUID. Capture the returned values from
``IngestResult.node_uuids`` (parallel to the submitted nodes; ``None`` where
a batch failed) if you need them for updates or fact-triple pinning.
"""

name: str
label: str | None = None
summary: str | None = None
attributes: dict[str, Any] | None = None
metadata: dict[str, Any] | None = None
uuid: str | None = None
created_at: str | None = None

def __post_init__(self) -> None:
Expand All @@ -55,32 +52,16 @@ def __post_init__(self) -> None:
check_scalar_map("attributes", self.attributes, errors, max_keys=MAX_ATTRIBUTE_KEYS)
check_scalar_map("metadata", self.metadata, errors, max_keys=MAX_ATTRIBUTE_KEYS)
check_timestamp("created_at", self.created_at, errors)
if self.uuid is not None:
try:
parsed = uuid_module.UUID(str(self.uuid))
if parsed.version != 4:
errors.append(f"uuid must be UUIDv4 (got version {parsed.version})")
# A UUID is a 128-bit value, not the text it was typed as; keep
# the canonical spelling so identity comparisons and the
# submitted value cannot disagree with each other.
self.uuid = str(parsed)
except (ValueError, AttributeError, TypeError):
errors.append(f"uuid is not a valid UUID: {self.uuid!r}")
if errors:
raise ConfigurationError(f"Invalid node {str(self.name)[:40]!r}: " + "; ".join(errors))

def to_add_node_item(self) -> AddNodeItem:
"""Build the SDK request model, omitting unset fields.

Our ``uuid`` maps to the SDK's ``uuid_`` field, which the client
serializes to the wire ``uuid`` key. Only fields that are actually set
are passed, so an unset field is omitted from the request rather than
sent as ``null`` (which matters for upserts, where a null could clobber
an existing value).
Only fields that are actually set are passed, so an unset field is
omitted from the request rather than sent as ``null``.
"""
fields: dict[str, Any] = {"name": self.name}
if self.uuid is not None:
fields["uuid_"] = self.uuid
if self.label is not None:
fields["label"] = self.label
if self.summary is not None:
Expand All @@ -94,11 +75,41 @@ def to_add_node_item(self) -> AddNodeItem:
return AddNodeItem(**fields)


_RETIRED_NODE_FIELDS = {
"uuid": (
"uuid cannot be supplied: Zep assigns node UUIDs "
"(use client.graph.node.update with a UUID from IngestResult.node_uuids "
"to update an existing node)"
),
}


def _load_nodes(path: Path) -> list[NodeItem]:
rows = rows_to_fields(load_rows(path), NodeItem)
rows = rows_to_fields(load_rows(path), NodeItem, retired_fields=_RETIRED_NODE_FIELDS)
return [NodeItem(**row) for row in rows]


def _assigned_node_uuids(response: Any, *, expected: int) -> list[str | None]:
"""Extract Zep-assigned node UUIDs from an ``add_nodes`` response.

The returned list is always ``expected`` long and aligned with the request
batch: a missing entry is ``None`` so callers can zip against the submitted
nodes without shifting later identities forward over a gap.
"""
nodes = getattr(response, "nodes", None) or []
uuids: list[str | None] = []
for index in range(expected):
if index >= len(nodes):
uuids.append(None)
continue
node = nodes[index]
node_uuid = getattr(node, "uuid_", None)
if node_uuid is None and isinstance(node, dict):
node_uuid = node.get("uuid") or node.get("uuid_")
uuids.append(str(node_uuid) if node_uuid else None)
return uuids


def ingest_nodes(
client: Zep,
nodes: Iterable[NodeItem] | str | Path,
Expand All @@ -107,59 +118,40 @@ def ingest_nodes(
user_id: str | None = None,
batch_size: int = MAX_NODES_PER_REQUEST,
max_retries: int = 5,
require_uuids: bool = True,
) -> IngestResult:
"""Create/upsert canonical nodes via ``client.graph.add_nodes``.

UUID is the only safe idempotency key. By default every node must have a
persisted UUIDv4; callers must explicitly opt out of that protection.
UUID uniqueness is checked by value, not by spelling, before the first
network call.
"""Create canonical nodes via ``client.graph.add_nodes``.

Submission is asynchronous; bind the result, then wait on it, so the resume
handles survive a timeout::
Zep assigns each node's UUID. ``result.node_uuids`` is parallel to the
submitted node list: successes carry the assigned UUID, and a failed batch
(or a missing response entry) leaves ``None`` in those slots so a later
success cannot shift forward under ``zip``. Submission is asynchronous; bind
the result, then wait on it, so the resume handles survive a timeout::

result = ingest_nodes(client, nodes, graph_id="g1")
result.wait(timeout=600)
# result.node_uuids[i] matches the i-th submitted node (or None)
"""
destination = Destination(graph_id=graph_id, user_id=user_id)
if not 1 <= batch_size <= MAX_NODES_PER_REQUEST:
raise ConfigurationError(f"batch_size must be 1..{MAX_NODES_PER_REQUEST}, got {batch_size}")
materialized = _load_nodes(Path(nodes)) if isinstance(nodes, str | Path) else list(nodes)
missing = [node.name for node in materialized if node.uuid is None]
if missing and require_uuids:
sample = ", ".join(repr(name) for name in missing[:3])
raise ConfigurationError(
f"{len(missing)} node(s) have no persisted UUIDv4 ({sample}). "
"Set require_uuids=False only for an intentionally non-idempotent ingest."
)
# NodeItem canonicalizes on construction, so this compares identities, not
# spellings: one UUID written two ways is one node, and pinning it twice
# would silently overwrite the first node with the second.
counts = Counter(node.uuid for node in materialized if node.uuid is not None)
duplicates = [node_uuid for node_uuid, count in counts.items() if count > 1]
if duplicates:
sample = ", ".join(repr(node_uuid) for node_uuid in duplicates[:3])
raise ConfigurationError(
f"{len(duplicates)} node UUID(s) appear on more than one node ({sample}). "
"UUIDs are compared as values, not text; give each node its own UUIDv4."
)

scope = (
{"graph_id": destination.graph_id}
if destination.graph_id is not None
else {"user_id": destination.user_id}
)
result = IngestResult(method="sequential", client=client)
if missing:
result.warnings.append(f"{len(missing)} node(s) have no UUID and may duplicate on a rerun.")
for start in range(0, len(materialized), batch_size):
batch = materialized[start : start + batch_size]
items = [node.to_add_node_item() for node in batch]
response, error = call_with_retries(
lambda: client.graph.add_nodes(nodes=items, **scope), # noqa: B023
max_retries=max_retries,
)
# Always extend node_uuids by batch length so indices stay aligned with
# the submitted list across partial failures.
result._node_uuids_from_submit = True
if error is not None:
result.add_errors.append(
AddError(
Expand All @@ -168,8 +160,10 @@ def ingest_nodes(
error=format_api_error("graph.add_nodes", error),
)
)
result.node_uuids.extend([None] * len(batch))
continue
result.items_submitted += len(batch)
result.node_uuids.extend(_assigned_node_uuids(response, expected=len(batch)))
task_id = getattr(response, "task_id", None)
if task_id and str(task_id) not in result.task_ids:
result.task_ids.append(str(task_id))
Expand Down
Loading
Loading