fleet-memory/skills/hindsight-docs/references/sdks/hindsight-all.md
Nicolò Boschi 576016f5dc
feat: add @vectorize-io/hindsight-all daemon lifecycle package (#949)
* feat: add @vectorize-io/hindsight-embed daemon lifecycle package

Create a new top-level `hindsight-embed-npm/` package that owns the daemon
lifecycle for the Python `hindsight-embed` CLI: spawning via `uvx`, writing
the profile, waiting for `/health`, and shutting down. Nothing more.

Deliberately does not ship an HTTP client — `@vectorize-io/hindsight-client`
already covers retain / recall / reflect / createBank against the Hindsight
API, and the two packages compose: once `manager.start()` returns, consumers
talk to the daemon via `new HindsightClient({ baseUrl: manager.getBaseUrl() })`.

`HindsightEmbedManagerOptions.env` forwards an arbitrary `Record<string,
string>` to both the daemon process and the profile config via `--env K=V`,
and `extraProfileCreateArgs` / `extraDaemonStartArgs` escape hatches cover
any new CLI flag without waiting for a wrapper release.

Refactor `hindsight-integrations/openclaw` to consume both packages:
`HindsightEmbedManager` for daemon lifecycle in local mode, `HindsightClient`
for all HTTP memory operations. Drop the bespoke subprocess/HTTP client that
used to live in openclaw. The retain queue stays local to openclaw (it's a
client-side reliability workaround with a single consumer today — will move
to the client package or server-side when a second consumer needs it).

Wire the new package into the main release pipeline (versioned alongside
the other core packages, published from `v*` tags) and add a CI build job.

* docs: add Embedded Node.js SDK page for @vectorize-io/hindsight-embed

* refactor: rename hindsight-embed-npm to hindsight-all, restructure docs sidebar

The Node package previously named @vectorize-io/hindsight-embed was
semantically misnamed: hindsight-embed (Python) is a CLI tool, while what
this Node package actually provides is the Node equivalent of hindsight-all
— a programmatic lifecycle manager for a local Hindsight daemon. Rename to
match.

Package rename
  - hindsight-embed-npm/ → hindsight-all-npm/ (git mv, history preserved)
  - @vectorize-io/hindsight-embed → @vectorize-io/hindsight-all
  - class HindsightEmbedManager → HindsightServer (matches Python hindsight-all)
  - HindsightEmbedManagerOptions → HindsightServerOptions
  - src/manager.ts → src/server.ts, src/manager.test.ts → src/server.test.ts
  - openclaw (index.ts, backfill.ts, tests) and the claude-code Python port
    updated to reference the new names

Docs restructure
  - Split sdks/python.md: now client-only content. New sdks/hindsight-all.md
    covers the programmatic hindsight-all Python package (HindsightServer and
    HindsightEmbedded).
  - Rename sdks/embed-npm.md → sdks/hindsight-all-npm.md with HindsightServer
    examples.
  - New "Installation" sidebar section, placed after Hosting, containing
    Docker / Kubernetes / Bare Metal (anchor links into developer/installation)
    plus Programmatic API (Python), Programmatic API (Node.js), and Daemon CLI.
  - Add si-docker, si-kubernetes, si-nodedotjs, lu-hard-drive to the sidebar
    ICON_MAP.

Docs dev-server fix
  - docusaurus.config.ts: drop the flaky NODE_ENV sniff for including the
    "Next" version. Use INCLUDE_CURRENT_VERSION exclusively. NODE_ENV was
    unreliable across hot-reload paths and caused the Next version to
    disappear intermittently when editing files.
  - scripts/dev/start-docs.sh: export INCLUDE_CURRENT_VERSION=true so local
    dev always shows Next; production builds leave it unset.

Lockfile cleanup
  - package-lock.json and hindsight-integrations/openclaw/package-lock.json
    had extraneous hindsight-embed-npm blocks left over from the rename.
    Removed manually and verified with npm install.

* ci: fix openclaw jobs by pre-building workspace deps; regenerate docs-skill

The build-openclaw-integration and test-openclaw-integration jobs failed
with "Failed to resolve entry for package @vectorize-io/hindsight-all"
because openclaw depends on two monorepo workspaces via `file:` deps
(@vectorize-io/hindsight-client and @vectorize-io/hindsight-all) whose
`dist/` directories are gitignored and never built before openclaw's npm ci.
Both jobs now install the root workspace and build the two deps first,
mirroring the release-control-plane pattern.

Also regenerate skills/hindsight-docs/references/* via
./scripts/generate-docs-skill.sh:
  - new skill pages for sdks/hindsight-all{.md,-npm.md}
  - updated skill pages for sdks/embed.md and sdks/python.md to match
    the new H1s and split content
  - incidental refreshes to changelog/index.md, developer/models.md,
    openapi.json, and uv.lock that verify-generated-files picked up

* ci: build openclaw before running tests so symlink test can realpath dist
2026-04-10 15:51:44 +02:00

146 lines
5.4 KiB
Markdown

---
sidebar_position: 2
---
# Programmatic API (Python)
The `hindsight-all` Python package lets your code spawn and manage a local Hindsight daemon without deploying any server infrastructure. It bundles the Hindsight API server, embedded PostgreSQL, and the Python client into one install — `pip install hindsight-all` and you can start a fully-functional Hindsight instance from a few lines of Python.
The daemon runs as a **separate OS process** on `127.0.0.1` (not in your Python process memory). Your code talks to it over HTTP via the bundled `HindsightClient`.
If you already have a Hindsight server running and just need a client, use [Python Client (hindsight-client)](./python.md) instead.
## How it works
`hindsight-all` exposes two primary APIs:
- **`HindsightServer`** — explicit lifecycle. Use it as a context manager when you want deterministic startup/shutdown (e.g. in tests).
- **`HindsightEmbedded`** — auto-managed. Starts a daemon on first use, reuses it across calls, shuts it down after an idle timeout. Easiest for application code that doesn't want to think about lifecycle.
Both end up talking to the same underlying daemon via the same `HindsightClient` HTTP interface — the difference is only how the server process is managed.
## Installation
```bash
pip install hindsight-all
```
The `hindsight-all` wheel bundles `hindsight-api-slim`, `hindsight-client`, and `hindsight-embed` as dependencies, so one `pip install` gets you everything.
## `HindsightServer` — explicit lifecycle
Use `HindsightServer` as a context manager when you want the server to start immediately, run for the duration of a block, and shut down cleanly afterwards. Ideal for tests and short-lived scripts.
```python
import os
from hindsight import HindsightServer, HindsightClient
with HindsightServer(
llm_provider="openai",
llm_model="gpt-4o-mini",
llm_api_key=os.environ["OPENAI_API_KEY"],
) as server:
client = HindsightClient(base_url=server.url)
client.retain(bank_id="my-bank", content="Alice works at Google")
results = client.recall(bank_id="my-bank", query="What does Alice do?")
for r in results:
print(r.text)
answer = client.reflect(bank_id="my-bank", query="Tell me about Alice")
print(answer.text)
# Server is stopped here
```
## `HindsightEmbedded` — auto-managed
`HindsightEmbedded` is the simplest way to use Hindsight in Python. It automatically manages a background daemon for you — starts on first use, stays alive across calls, shuts down after an idle timeout.
```python
from hindsight import HindsightEmbedded
import os
# Server starts automatically on first call
client = HindsightEmbedded(
profile="myapp", # Profile for data isolation
llm_provider="openai",
llm_model="gpt-4o-mini",
llm_api_key=os.environ["OPENAI_API_KEY"],
)
# Use immediately - no manual server management needed
client.retain(bank_id="my-bank", content="Alice works at Google")
results = client.recall(bank_id="my-bank", query="What does Alice do?")
# Server continues running (auto-stops after idle timeout)
# Or explicitly stop it:
client.close(stop_daemon=True)
```
### What's a Profile?
A profile is an isolated Hindsight environment. Each profile gets its own embedded PostgreSQL database (stored in `~/.pg0/instances/hindsight-embed-{profile}/`) and its own API server. Use different profiles to separate environments (dev/prod), applications, or users.
### When to use which
| Use case | Pick |
|---|---|
| Tests, short-lived scripts, deterministic startup/shutdown | `HindsightServer` (context manager) |
| Long-running application, auto-start on first use, don't want to manage lifecycle | `HindsightEmbedded` |
| Existing Hindsight server running elsewhere | [`hindsight-client`](./python.md) directly |
## API namespaces
Both `HindsightEmbedded` and `HindsightClient` expose organized API namespaces for bank management, mental models, directives, and memories:
```python
from hindsight import HindsightEmbedded
import os
embedded = HindsightEmbedded(
profile="myapp",
llm_provider="openai",
llm_api_key=os.environ["OPENAI_API_KEY"],
)
# Core operations
embedded.retain(bank_id="test", content="Hello")
results = embedded.recall(bank_id="test", query="Hello")
# Bank management
embedded.banks.create(bank_id="test", name="Test Bank", mission="Help users")
embedded.banks.set_mission(bank_id="test", mission="Updated mission")
embedded.banks.delete(bank_id="test")
# Mental models
embedded.mental_models.create(
bank_id="test",
name="User Preferences",
content="User prefers dark mode"
)
models = embedded.mental_models.list(bank_id="test")
# Directives
embedded.directives.create(
bank_id="test",
name="Response Style",
content="Be concise and friendly"
)
directives = embedded.directives.list(bank_id="test")
# List memories
memories = embedded.memories.list(bank_id="test", type="world", limit=50)
```
API namespaces ensure the daemon is running before each call, so daemon crashes are handled gracefully:
```python
# ✅ GOOD - Uses API namespace (daemon restarts handled)
embedded.banks.create(bank_id="test", name="Test")
# ❌ BAD - Direct client access (daemon crashes NOT handled)
client = embedded.client
client.create_bank(bank_id="test", name="Test") # Fails if daemon crashed
```
For the full reference of retain/recall/reflect methods and their options (which work the same regardless of how you obtain the client) see the [Python Client page](./python.md).