fleet-memory/hindsight-api-slim/hindsight_api/pg0.py
Nicolò Boschi 15ea23d5d6
feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560)
* feat: introduce hindsight-api-slim and hindsight-all-slim packages

Closes #552

- Move all source code from hindsight-api/ to new hindsight-api-slim/
- hindsight-api-slim has heavy ML deps (torch, sentence-transformers,
  transformers, einops, flashrank, mlx, mlx-lm, safetensors) and
  pg0-embedded as optional extras: [local-ml], [embedded-db], [all]
- hindsight-api becomes a zero-code meta-package depending on
  hindsight-api-slim[all] for full backward compatibility
- Add hindsight-all-slim meta-package: hindsight-api-slim + client + embed
- hindsight-all updated to depend on hindsight-api-slim[all]
- pg0.py: lazy-import pg0 with clear ImportError pointing to [embedded-db]
- Dockerfile: replace sed hack with proper uv sync --extra flags
- Update release.yml, test.yml, lint.sh, release.sh, CLAUDE.md and
  all path references throughout the repo

* refactor: rename hindsight/ directory to hindsight-all/

* docs: document hindsight-api-slim and hindsight-all-slim package variants

Add package variants table and extras explanation to installation.md

* docs: remove emojis from installation.md, use professional tone

* docs: link Docker slim variant to pip package variants section

* docs: consolidate Docker image variants into single table

* ci: fix working-directory paths after package restructure

- Replace all hindsight-api → hindsight-api-slim in test.yml
- Replace hindsight → hindsight-all in test.yml
- Add --extra embedded-db to test-embed API install step

* ci: add local-ml and embedded-db extras to API sync steps

These extras were previously implicit in the old hindsight-api package
(which bundled everything). Now that hindsight-api-slim uses optional
extras, we must explicitly request local-ml and embedded-db in CI.

* ci: add API install step with embedded-db to test-embed smoke test

The smoke test starts hindsight-api as a daemon, which requires pg0-embedded.
Add a dedicated install step for hindsight-api-slim with embedded-db extra
so the daemon can start successfully.

* ci: remove --no-install-project when using optional extras

When --no-install-project is combined with --extra, the optional deps
are not installed because extras require the project to be active.
Remove --no-install-project from steps that need local-ml or embedded-db.

* ci: fix ordering of uv sync steps to preserve optional extras

When uv sync runs for a different workspace member, it removes optional
extras installed for other members. Fix by always running extra-requiring
API sync last, after other workspace member syncs.

Also remove --no-install-project from embedded-db sync in test-embed,
as --no-install-project prevents optional extras from being active.

* ci: add local-ml extra to test-embed API install for smoke test

The smoke test starts the full API server which needs sentence-transformers
for local embeddings (default provider). Add local-ml extra to the install.

* ci: simplify extras with --all-extras and add slim pip smoke test

- Replace explicit --extra local-ml --extra embedded-db with --all-extras
  for cleaner, more maintainable sync steps
- Add test-pip-slim job: tests hindsight-api-slim[embedded-db] without
  local ML models, using Cohere for embeddings/reranking (mirrors Docker
  slim smoke test approach)

* ci: simplify slim smoke test to health check only (mirrors Docker test)
2026-03-13 13:50:03 +01:00

198 lines
6.6 KiB
Python

from __future__ import annotations
import asyncio
import logging
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from pg0 import Pg0
logger = logging.getLogger(__name__)
DEFAULT_USERNAME = "hindsight"
DEFAULT_PASSWORD = "hindsight"
DEFAULT_DATABASE = "hindsight"
class EmbeddedPostgres:
"""Manages an embedded PostgreSQL server instance using pg0-embedded."""
def __init__(
self,
port: int | None = None,
username: str = DEFAULT_USERNAME,
password: str = DEFAULT_PASSWORD,
database: str = DEFAULT_DATABASE,
name: str = "hindsight",
**kwargs,
):
self.port = port # None means pg0 will auto-assign
self.username = username
self.password = password
self.database = database
self.name = name
self._pg0: Pg0 | None = None
def _get_pg0(self) -> Pg0:
if self._pg0 is None:
try:
from pg0 import Pg0
except ImportError:
raise ImportError(
"pg0-embedded is required for embedded PostgreSQL. "
"Install it with: pip install 'hindsight-api-slim[embedded-db]'"
)
kwargs = {
"name": self.name,
"username": self.username,
"password": self.password,
"database": self.database,
}
# Only set port if explicitly specified
if self.port is not None:
kwargs["port"] = self.port
self._pg0 = Pg0(**kwargs)
return self._pg0
async def start(self, max_retries: int = 5, retry_delay: float = 4.0) -> str:
"""Start the PostgreSQL server with retry logic."""
port_info = f"port={self.port}" if self.port else "port=auto"
logger.info(f"Starting embedded PostgreSQL (name={self.name}, {port_info})...")
pg0 = self._get_pg0()
last_error = None
for attempt in range(1, max_retries + 1):
try:
loop = asyncio.get_event_loop()
info = await loop.run_in_executor(None, pg0.start)
# Get URI from pg0 (includes auto-assigned port)
uri = info.uri
logger.info(f"PostgreSQL started: {uri}")
return uri
except Exception as e:
last_error = str(e)
if attempt < max_retries:
delay = retry_delay * (2 ** (attempt - 1))
logger.debug(f"pg0 start attempt {attempt}/{max_retries} failed: {last_error}")
logger.debug(f"Retrying in {delay:.1f}s...")
await asyncio.sleep(delay)
else:
logger.debug(f"pg0 start attempt {attempt}/{max_retries} failed: {last_error}")
raise RuntimeError(
f"Failed to start embedded PostgreSQL after {max_retries} attempts. Last error: {last_error}"
)
async def stop(self) -> None:
"""Stop the PostgreSQL server."""
pg0 = self._get_pg0()
logger.info(f"Stopping embedded PostgreSQL (name: {self.name})...")
try:
loop = asyncio.get_event_loop()
await loop.run_in_executor(None, pg0.stop)
logger.info("Embedded PostgreSQL stopped")
except Exception as e:
if "not running" in str(e).lower():
return
raise RuntimeError(f"Failed to stop PostgreSQL: {e}")
async def get_uri(self) -> str:
"""Get the connection URI for the PostgreSQL server."""
pg0 = self._get_pg0()
loop = asyncio.get_event_loop()
info = await loop.run_in_executor(None, pg0.info)
return info.uri
async def is_running(self) -> bool:
"""Check if the PostgreSQL server is currently running."""
try:
pg0 = self._get_pg0()
loop = asyncio.get_event_loop()
info = await loop.run_in_executor(None, pg0.info)
return info is not None and info.running
except Exception:
return False
async def ensure_running(self) -> str:
"""Ensure the PostgreSQL server is running, starting it if needed."""
if await self.is_running():
return await self.get_uri()
return await self.start()
_default_instance: EmbeddedPostgres | None = None
def get_embedded_postgres() -> EmbeddedPostgres:
"""Get or create the default EmbeddedPostgres instance."""
global _default_instance
if _default_instance is None:
_default_instance = EmbeddedPostgres()
return _default_instance
async def start_embedded_postgres() -> str:
"""Quick start function for embedded PostgreSQL."""
return await get_embedded_postgres().ensure_running()
async def stop_embedded_postgres() -> None:
"""Stop the default embedded PostgreSQL instance."""
global _default_instance
if _default_instance:
await _default_instance.stop()
def parse_pg0_url(db_url: str) -> tuple[bool, str | None, int | None]:
"""
Parse a database URL and check if it's a pg0:// embedded database URL.
Supports:
- "pg0" -> default instance "hindsight"
- "pg0://instance-name" -> named instance
- "pg0://instance-name:port" -> named instance with explicit port
- Any other URL (e.g., postgresql://) -> not a pg0 URL
Args:
db_url: The database URL to parse
Returns:
Tuple of (is_pg0, instance_name, port)
- is_pg0: True if this is a pg0 URL
- instance_name: The instance name (or None if not pg0)
- port: The explicit port (or None for auto-assign)
"""
if db_url == "pg0":
return True, "hindsight", None
if db_url.startswith("pg0://"):
url_part = db_url[6:] # Remove "pg0://"
if ":" in url_part:
instance_name, port_str = url_part.rsplit(":", 1)
return True, instance_name or "hindsight", int(port_str)
else:
return True, url_part or "hindsight", None
return False, None, None
async def resolve_database_url(db_url: str) -> str:
"""
Resolve a database URL, handling pg0:// embedded database URLs.
If the URL is a pg0:// URL, starts the embedded PostgreSQL and returns
the actual postgresql:// connection URL. Otherwise, returns the URL unchanged.
Args:
db_url: Database URL (pg0://, pg0, or postgresql://)
Returns:
The resolved postgresql:// connection URL
"""
is_pg0, instance_name, port = parse_pg0_url(db_url)
if is_pg0:
pg0 = EmbeddedPostgres(name=instance_name, port=port)
return await pg0.ensure_running()
return db_url