feat: add @vectorize-io/hindsight-all daemon lifecycle package (#949 )

* feat: add @vectorize-io/hindsight-embed daemon lifecycle package

Create a new top-level `hindsight-embed-npm/` package that owns the daemon
lifecycle for the Python `hindsight-embed` CLI: spawning via `uvx`, writing
the profile, waiting for `/health`, and shutting down. Nothing more.

Deliberately does not ship an HTTP client — `@vectorize-io/hindsight-client`
already covers retain / recall / reflect / createBank against the Hindsight
API, and the two packages compose: once `manager.start()` returns, consumers
talk to the daemon via `new HindsightClient({ baseUrl: manager.getBaseUrl() })`.

`HindsightEmbedManagerOptions.env` forwards an arbitrary `Record<string,
string>` to both the daemon process and the profile config via `--env K=V`,
and `extraProfileCreateArgs` / `extraDaemonStartArgs` escape hatches cover
any new CLI flag without waiting for a wrapper release.

Refactor `hindsight-integrations/openclaw` to consume both packages:
`HindsightEmbedManager` for daemon lifecycle in local mode, `HindsightClient`
for all HTTP memory operations. Drop the bespoke subprocess/HTTP client that
used to live in openclaw. The retain queue stays local to openclaw (it's a
client-side reliability workaround with a single consumer today — will move
to the client package or server-side when a second consumer needs it).

Wire the new package into the main release pipeline (versioned alongside
the other core packages, published from `v*` tags) and add a CI build job.

* docs: add Embedded Node.js SDK page for @vectorize-io/hindsight-embed

* refactor: rename hindsight-embed-npm to hindsight-all, restructure docs sidebar

The Node package previously named @vectorize-io/hindsight-embed was
semantically misnamed: hindsight-embed (Python) is a CLI tool, while what
this Node package actually provides is the Node equivalent of hindsight-all
— a programmatic lifecycle manager for a local Hindsight daemon. Rename to
match.

Package rename
  - hindsight-embed-npm/ → hindsight-all-npm/ (git mv, history preserved)
  - @vectorize-io/hindsight-embed → @vectorize-io/hindsight-all
  - class HindsightEmbedManager → HindsightServer (matches Python hindsight-all)
  - HindsightEmbedManagerOptions → HindsightServerOptions
  - src/manager.ts → src/server.ts, src/manager.test.ts → src/server.test.ts
  - openclaw (index.ts, backfill.ts, tests) and the claude-code Python port
    updated to reference the new names

Docs restructure
  - Split sdks/python.md: now client-only content. New sdks/hindsight-all.md
    covers the programmatic hindsight-all Python package (HindsightServer and
    HindsightEmbedded).
  - Rename sdks/embed-npm.md → sdks/hindsight-all-npm.md with HindsightServer
    examples.
  - New "Installation" sidebar section, placed after Hosting, containing
    Docker / Kubernetes / Bare Metal (anchor links into developer/installation)
    plus Programmatic API (Python), Programmatic API (Node.js), and Daemon CLI.
  - Add si-docker, si-kubernetes, si-nodedotjs, lu-hard-drive to the sidebar
    ICON_MAP.

Docs dev-server fix
  - docusaurus.config.ts: drop the flaky NODE_ENV sniff for including the
    "Next" version. Use INCLUDE_CURRENT_VERSION exclusively. NODE_ENV was
    unreliable across hot-reload paths and caused the Next version to
    disappear intermittently when editing files.
  - scripts/dev/start-docs.sh: export INCLUDE_CURRENT_VERSION=true so local
    dev always shows Next; production builds leave it unset.

Lockfile cleanup
  - package-lock.json and hindsight-integrations/openclaw/package-lock.json
    had extraneous hindsight-embed-npm blocks left over from the rename.
    Removed manually and verified with npm install.

* ci: fix openclaw jobs by pre-building workspace deps; regenerate docs-skill

The build-openclaw-integration and test-openclaw-integration jobs failed
with "Failed to resolve entry for package @vectorize-io/hindsight-all"
because openclaw depends on two monorepo workspaces via `file:` deps
(@vectorize-io/hindsight-client and @vectorize-io/hindsight-all) whose
`dist/` directories are gitignored and never built before openclaw's npm ci.
Both jobs now install the root workspace and build the two deps first,
mirroring the release-control-plane pattern.

Also regenerate skills/hindsight-docs/references/* via
./scripts/generate-docs-skill.sh:
  - new skill pages for sdks/hindsight-all{.md,-npm.md}
  - updated skill pages for sdks/embed.md and sdks/python.md to match
    the new H1s and split content
  - incidental refreshes to changelog/index.md, developer/models.md,
    openapi.json, and uv.lock that verify-generated-files picked up

* ci: build openclaw before running tests so symlink test can realpath dist

2026-04-10 15:51:44 +02:00

6.5 KiB

Raw Permalink Blame History

sidebar_position
5

Daemon CLI (hindsight-embed)

Zero-configuration local memory system with automatic daemon management. Perfect for development, prototyping, and single-user applications.

Overview

hindsight-embed is a zero-configuration SDK that wraps the Hindsight API and PostgreSQL database into a single auto-managed local daemon. It's designed for development, prototyping, and single-user applications where you want memory capabilities without infrastructure overhead.

How it works:

First command triggers startup: When you run any hindsight-embed command, it checks if a local daemon is running
Auto-daemon management: If no daemon exists, it automatically spawns hindsight-api --daemon in the background
Embedded database: The daemon uses pg0 (embedded PostgreSQL) — no separate database installation required
Command forwarding: Your command is forwarded to the local daemon via HTTP (localhost:8888)
Auto-shutdown: After 5 minutes of inactivity (configurable), the daemon gracefully shuts down to free resources

Key features:

Zero setup — One configure command and you're ready
Automatic lifecycle — Daemon starts on-demand, stops when idle
Isolated storage — Each bank gets its own embedded PostgreSQL database
Local-only — Binds to 127.0.0.1:8888, not accessible from network
Production-grade engine — Uses the same memory engine as the full API service

Think of it as SQLite for long-term memory — all the power of Hindsight without managing servers.

Installation

Install via uvx (recommended - always latest version):

# Run directly without installation
uvx hindsight-embed@latest configure

# Or use pipx for persistent installation
pipx install hindsight-embed

Quick Start

1. Configure

# Interactive configuration
hindsight-embed configure

# Or non-interactive via environment variables
export HINDSIGHT_EMBED_LLM_PROVIDER=openai
export HINDSIGHT_EMBED_LLM_API_KEY=sk-xxxxxxxxxxxx
export HINDSIGHT_EMBED_LLM_MODEL=gpt-4o-mini
hindsight-embed configure

Configuration is saved to ~/.hindsight/embed:

HINDSIGHT_EMBED_LLM_PROVIDER=openai
HINDSIGHT_EMBED_LLM_MODEL=gpt-4o-mini
HINDSIGHT_EMBED_BANK_ID=default
HINDSIGHT_EMBED_LLM_API_KEY=sk-xxxxxxxxxxxx

# Daemon settings (macOS: force CPU to avoid MPS/XPC issues)
HINDSIGHT_API_EMBEDDINGS_LOCAL_FORCE_CPU=1
HINDSIGHT_API_RERANKER_LOCAL_FORCE_CPU=1

2. Use Memory Operations

# Store a memory
hindsight-embed memory retain default "User prefers dark mode"

# Query memories
hindsight-embed memory recall default "user preferences"

# Reasoning with memory
hindsight-embed memory reflect default "What color scheme should I use?"

The daemon starts automatically on first use!

Environment Variables

Variable	Description	Default
`HINDSIGHT_EMBED_LLM_API_KEY`	Required. API key for LLM provider	-
`HINDSIGHT_EMBED_LLM_PROVIDER`	LLM provider: `openai`, `anthropic`, `gemini`, `groq`, `minimax`, `ollama`	`openai`
`HINDSIGHT_EMBED_LLM_MODEL`	Model name	`gpt-4o-mini`
`HINDSIGHT_EMBED_BANK_ID`	Default memory bank ID	`default`
`HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT`	Seconds before daemon auto-exits when idle (0 = never)	`300`

Provider Examples:

# OpenAI
export HINDSIGHT_EMBED_LLM_PROVIDER=openai
export HINDSIGHT_EMBED_LLM_API_KEY=sk-xxxxxxxxxxxx
export HINDSIGHT_EMBED_LLM_MODEL=gpt-4o

# Groq (fast inference)
export HINDSIGHT_EMBED_LLM_PROVIDER=groq
export HINDSIGHT_EMBED_LLM_API_KEY=gsk_xxxxxxxxxxxx
export HINDSIGHT_EMBED_LLM_MODEL=llama-3.3-70b-versatile

# Anthropic
export HINDSIGHT_EMBED_LLM_PROVIDER=anthropic
export HINDSIGHT_EMBED_LLM_API_KEY=sk-ant-xxxxxxxxxxxx
export HINDSIGHT_EMBED_LLM_MODEL=claude-sonnet-4-20250514

Daemon Management

Idle Timeout

Customize how long the daemon stays alive when idle:

# Never timeout (daemon runs until manually stopped)
export HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT=0

# Shorter timeout: 1 minute
export HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT=60

# Longer timeout: 30 minutes
export HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT=1800

Daemon Commands

# Check daemon status
hindsight-embed daemon status

# View daemon logs in real-time
hindsight-embed daemon logs -f

# Stop daemon manually
hindsight-embed daemon stop

Commands

All memory operations follow the same interface as the CLI:

Retain (Store Memory)

hindsight-embed memory retain <bank_id> "content"

# With context
hindsight-embed memory retain <bank_id> "content" --context "source information"

# Background processing
hindsight-embed memory retain <bank_id> "content" --async

Recall (Search)

hindsight-embed memory recall <bank_id> "query"

# With budget control
hindsight-embed memory recall <bank_id> "query" --budget high

# Show trace
hindsight-embed memory recall <bank_id> "query" --trace

Reflect (Generate Response)

hindsight-embed memory reflect <bank_id> "prompt"

# With additional context
hindsight-embed memory reflect <bank_id> "prompt" --context "additional info"

Bank Management

# List all banks
hindsight-embed bank list

# View bank stats
hindsight-embed bank stats <bank_id>

# Set bank name
hindsight-embed bank name <bank_id> "My Assistant"

# Set bank mission
hindsight-embed bank mission <bank_id> "I am a helpful AI assistant"

Troubleshooting

Daemon Won't Start

Check the daemon logs:

hindsight-embed daemon logs
# Or watch in real-time
hindsight-embed daemon logs -f

Common issues:

Missing API key: Set HINDSIGHT_EMBED_LLM_API_KEY
Port conflict: Another service using port 8888
Permissions: Check ~/.hindsight/ directory permissions

Daemon Exits Immediately

Check if you have the idle timeout set too low:

# Disable idle timeout for debugging
export HINDSIGHT_EMBED_DAEMON_IDLE_TIMEOUT=0
hindsight-embed daemon status

Reset Configuration

# Remove config file and reconfigure
rm ~/.hindsight/embed
hindsight-embed configure

When to Use

Perfect for:

Development and prototyping
Single-user applications
Local-first tools
Quick experiments with Hindsight

Not suitable for:

Production multi-user deployments
Network-accessible services
High-availability requirements
Multi-tenant applications

For production deployments, use the API Service with external PostgreSQL instead.

6.5 KiB Raw Permalink Blame History