fleet-memory/hindsight-api-slim/hindsight_api/extensions/builtin/supabase_tenant.py
Nicolò Boschi 15ea23d5d6
feat: introduce hindsight-api-slim and hindsight-all-slim packages (#560)
* feat: introduce hindsight-api-slim and hindsight-all-slim packages

Closes #552

- Move all source code from hindsight-api/ to new hindsight-api-slim/
- hindsight-api-slim has heavy ML deps (torch, sentence-transformers,
  transformers, einops, flashrank, mlx, mlx-lm, safetensors) and
  pg0-embedded as optional extras: [local-ml], [embedded-db], [all]
- hindsight-api becomes a zero-code meta-package depending on
  hindsight-api-slim[all] for full backward compatibility
- Add hindsight-all-slim meta-package: hindsight-api-slim + client + embed
- hindsight-all updated to depend on hindsight-api-slim[all]
- pg0.py: lazy-import pg0 with clear ImportError pointing to [embedded-db]
- Dockerfile: replace sed hack with proper uv sync --extra flags
- Update release.yml, test.yml, lint.sh, release.sh, CLAUDE.md and
  all path references throughout the repo

* refactor: rename hindsight/ directory to hindsight-all/

* docs: document hindsight-api-slim and hindsight-all-slim package variants

Add package variants table and extras explanation to installation.md

* docs: remove emojis from installation.md, use professional tone

* docs: link Docker slim variant to pip package variants section

* docs: consolidate Docker image variants into single table

* ci: fix working-directory paths after package restructure

- Replace all hindsight-api → hindsight-api-slim in test.yml
- Replace hindsight → hindsight-all in test.yml
- Add --extra embedded-db to test-embed API install step

* ci: add local-ml and embedded-db extras to API sync steps

These extras were previously implicit in the old hindsight-api package
(which bundled everything). Now that hindsight-api-slim uses optional
extras, we must explicitly request local-ml and embedded-db in CI.

* ci: add API install step with embedded-db to test-embed smoke test

The smoke test starts hindsight-api as a daemon, which requires pg0-embedded.
Add a dedicated install step for hindsight-api-slim with embedded-db extra
so the daemon can start successfully.

* ci: remove --no-install-project when using optional extras

When --no-install-project is combined with --extra, the optional deps
are not installed because extras require the project to be active.
Remove --no-install-project from steps that need local-ml or embedded-db.

* ci: fix ordering of uv sync steps to preserve optional extras

When uv sync runs for a different workspace member, it removes optional
extras installed for other members. Fix by always running extra-requiring
API sync last, after other workspace member syncs.

Also remove --no-install-project from embedded-db sync in test-embed,
as --no-install-project prevents optional extras from being active.

* ci: add local-ml extra to test-embed API install for smoke test

The smoke test starts the full API server which needs sentence-transformers
for local embeddings (default provider). Add local-ml extra to the install.

* ci: simplify extras with --all-extras and add slim pip smoke test

- Replace explicit --extra local-ml --extra embedded-db with --all-extras
  for cleaner, more maintainable sync steps
- Add test-pip-slim job: tests hindsight-api-slim[embedded-db] without
  local ML models, using Cohere for embeddings/reranking (mirrors Docker
  slim smoke test approach)

* ci: simplify slim smoke test to health check only (mirrors Docker test)
2026-03-13 13:50:03 +01:00

433 lines
17 KiB
Python

"""
Supabase Tenant Extension for Hindsight
Validates Supabase JWTs and maps authenticated users to isolated memory banks.
Each user gets their own PostgreSQL schema based on their Supabase user ID.
This extension enables multi-tenant memory isolation for applications using
Supabase Auth - each authenticated user's memories are stored in a separate
schema, ensuring complete data isolation.
Features:
- Local JWT Verification: Validates tokens locally using JWKS public keys
(no network call per request)
- Automatic Schema Isolation: Each user gets {prefix}_{user_id} schema
- Zero User Management: Leverages your existing Supabase Auth setup
- Production Ready: Includes health checks, timeouts, key rotation handling,
and error handling
- Built-in: Ships with Hindsight, no extra installation needed
- Legacy Support: Falls back to /auth/v1/user endpoint for HS256 projects
JWT Verification Strategy:
By default, JWTs are verified locally using public keys from the Supabase
JWKS endpoint (/auth/v1/.well-known/jwks.json). This is the Supabase-recommended
approach: no network call per request, fast, and secure.
If JWKS keys are unavailable (e.g., legacy HS256 projects), the extension
falls back to calling /auth/v1/user per request for validation. This requires
the service_role key to be configured.
Configuration via environment variables:
HINDSIGHT_API_TENANT_EXTENSION=hindsight_api.extensions.builtin.supabase_tenant:SupabaseTenantExtension
HINDSIGHT_API_TENANT_SUPABASE_URL=https://your-project.supabase.co
# Optional - only required for legacy HS256 projects or health checks
HINDSIGHT_API_TENANT_SUPABASE_SERVICE_KEY=your-service-role-key
# Optional
HINDSIGHT_API_TENANT_SCHEMA_PREFIX=user # Default: "user" (creates user_<uuid> schemas)
Usage:
Clients pass their Supabase JWT in the Authorization header:
curl -H "Authorization: Bearer <supabase_jwt>" \\
https://your-hindsight-server/v1/default/banks/my-bank/memories/recall
Author: BrighterBalance (https://brighterbalance.app)
License: MIT
"""
from __future__ import annotations
import logging
import re
import time
import httpx
import jwt as pyjwt
from jwt import PyJWK
from hindsight_api.extensions.tenant import AuthenticationError, Tenant, TenantContext, TenantExtension
from hindsight_api.models import RequestContext
logger = logging.getLogger(__name__)
__all__ = ["SupabaseTenantExtension"]
# Minimum expected JWT length (JWTs are typically 100+ characters)
MIN_TOKEN_LENGTH = 20
# Timeout for Supabase API calls
REQUEST_TIMEOUT_SECONDS = 10.0
# JWKS cache TTL — Supabase Edge caches JWKS for 10 minutes, so we match that
JWKS_CACHE_TTL_SECONDS = 600
# Minimum interval between JWKS refreshes to avoid hammering the endpoint
JWKS_MIN_REFRESH_INTERVAL_SECONDS = 30
# Algorithms supported by Supabase Auth for asymmetric JWT signing
SUPPORTED_ALGORITHMS = ["RS256", "ES256"]
# Supabase user IDs are UUIDs — validate before using in schema names
_UUID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$", re.IGNORECASE)
# Schema prefix must be a valid Postgres identifier component (letters, digits, underscores)
_SCHEMA_PREFIX_RE = re.compile(r"^[a-zA-Z_][a-zA-Z0-9_]*$")
class SupabaseTenantExtension(TenantExtension):
"""
TenantExtension that validates Supabase JWTs for multi-tenant isolation.
Each authenticated user gets their own PostgreSQL schema, ensuring complete
memory isolation between users. The schema name is derived from the user's
Supabase user ID (the ``sub`` claim in the JWT).
JWT verification uses JWKS (local, no network call per request) when
asymmetric keys are configured in Supabase, and falls back to the
``/auth/v1/user`` endpoint for legacy HS256 projects.
Example:
User with ID "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
gets schema "user_a1b2c3d4_e5f6_7890_abcd_ef1234567890"
"""
def __init__(self, config: dict[str, str]) -> None:
"""
Initialize with configuration from environment variables.
Config keys are derived from HINDSIGHT_API_TENANT_* env vars:
- HINDSIGHT_API_TENANT_SUPABASE_URL -> config["supabase_url"] (required)
- HINDSIGHT_API_TENANT_SUPABASE_SERVICE_KEY -> config["supabase_service_key"] (optional)
- HINDSIGHT_API_TENANT_SCHEMA_PREFIX -> config["schema_prefix"] (optional)
Args:
config: Dictionary of configuration values from environment
Raises:
ValueError: If required configuration is missing
"""
super().__init__(config)
self.supabase_url = (config.get("supabase_url") or "").rstrip("/")
self.supabase_service_key = config.get("supabase_service_key")
self.schema_prefix = config.get("schema_prefix", "user")
# Track initialized schemas to avoid redundant migrations
self._initialized_schemas: set[str] = set()
# Reusable HTTP client (created on startup)
self._http_client: httpx.AsyncClient | None = None
# JWKS state
self._jwks_keys: dict[str, PyJWK] = {}
self._jwks_last_fetched: float = 0
self._use_jwks: bool = False
if not self.supabase_url:
raise ValueError(
"HINDSIGHT_API_TENANT_SUPABASE_URL is required. "
"Set it to your Supabase project URL (e.g., https://xxx.supabase.co)"
)
if not _SCHEMA_PREFIX_RE.match(self.schema_prefix):
raise ValueError(
f"Invalid schema_prefix '{self.schema_prefix}'. "
"Must be a valid Postgres identifier (letters, digits, underscores, starting with a letter or underscore)."
)
# ------------------------------------------------------------------
# Lifecycle
# ------------------------------------------------------------------
async def on_startup(self) -> None:
"""
Called when Hindsight starts.
Creates a reusable HTTP client, fetches JWKS for local JWT verification,
and optionally verifies connectivity to Supabase.
"""
logger.info("Initializing Supabase tenant extension")
logger.info("Supabase URL: %s", self.supabase_url)
logger.info("Schema prefix: %s_", self.schema_prefix)
self._http_client = httpx.AsyncClient(timeout=REQUEST_TIMEOUT_SECONDS)
# Attempt to fetch JWKS for fast local JWT verification
await self._try_init_jwks()
# Optional health check using service key
if self.supabase_service_key:
await self._health_check()
async def on_shutdown(self) -> None:
"""Called when Hindsight shuts down. Closes the HTTP client."""
logger.info("Shutting down Supabase tenant extension")
if self._http_client:
await self._http_client.aclose()
self._http_client = None
# ------------------------------------------------------------------
# JWKS management
# ------------------------------------------------------------------
async def _try_init_jwks(self) -> None:
"""Fetch JWKS and decide verification mode (local JWKS vs legacy endpoint)."""
try:
await self._fetch_jwks()
if self._jwks_keys:
self._use_jwks = True
logger.info(
"JWKS loaded — using local JWT verification with %d key(s)",
len(self._jwks_keys),
)
return
# JWKS endpoint returned no keys — project likely uses legacy HS256
logger.warning(
"JWKS endpoint returned no signing keys. "
"Falling back to /auth/v1/user endpoint for JWT verification. "
"For better performance, enable asymmetric JWT signing in your "
"Supabase dashboard (Project Settings → Auth → JWT Algorithm)."
)
except Exception as e:
logger.warning(
"Could not fetch JWKS (%s). Falling back to /auth/v1/user endpoint for JWT verification.",
e,
)
# Legacy mode requires service key
if not self.supabase_service_key:
raise ValueError(
"HINDSIGHT_API_TENANT_SUPABASE_SERVICE_KEY is required when JWKS "
"is not available. Either enable asymmetric JWT signing in your "
"Supabase project or provide the service_role key."
)
self._use_jwks = False
async def _fetch_jwks(self) -> None:
"""Fetch public signing keys from the Supabase JWKS endpoint."""
if self._http_client is None:
raise RuntimeError("HTTP client not initialized")
url = f"{self.supabase_url}/auth/v1/.well-known/jwks.json"
response = await self._http_client.get(url)
response.raise_for_status()
jwks_data = response.json()
keys: dict[str, PyJWK] = {}
for key_data in jwks_data.get("keys", []):
kid = key_data.get("kid")
if kid:
keys[kid] = PyJWK(key_data)
self._jwks_keys = keys
self._jwks_last_fetched = time.monotonic()
async def _get_signing_key(self, token: str) -> PyJWK:
"""
Resolve the signing key for a token from the JWKS cache.
If the key ID (``kid``) is not in the cache, triggers one JWKS refresh
to handle key rotation before raising an error.
"""
header = pyjwt.get_unverified_header(token)
kid = header.get("kid")
if not kid:
raise AuthenticationError("Token missing key ID (kid) header")
# Refresh cache if stale
now = time.monotonic()
if now - self._jwks_last_fetched > JWKS_CACHE_TTL_SECONDS:
logger.debug("JWKS cache expired, refreshing")
await self._fetch_jwks()
if kid in self._jwks_keys:
return self._jwks_keys[kid]
# Key not found — try one forced refresh to handle key rotation,
# but only if we haven't just refreshed
if now - self._jwks_last_fetched > JWKS_MIN_REFRESH_INTERVAL_SECONDS:
logger.info("Signing key %s not in cache, refreshing JWKS for possible key rotation", kid)
await self._fetch_jwks()
if kid in self._jwks_keys:
return self._jwks_keys[kid]
raise AuthenticationError("Unable to find signing key for token")
# ------------------------------------------------------------------
# Authentication
# ------------------------------------------------------------------
async def authenticate(self, context: RequestContext) -> TenantContext:
"""
Validate a Supabase JWT and return tenant context.
Uses local JWKS verification when available (no network call per
request), falling back to the ``/auth/v1/user`` endpoint for legacy
HS256 projects.
Args:
context: Request context containing the API key (JWT)
Returns:
TenantContext with schema_name set to ``{prefix}_{user_uuid}``
Raises:
AuthenticationError: If token is missing, invalid, or expired
"""
token = context.api_key
if not token:
raise AuthenticationError("Missing Authorization header. Expected: Bearer <supabase_jwt>")
if len(token) < MIN_TOKEN_LENGTH:
raise AuthenticationError("Invalid token format")
if self._http_client is None:
raise AuthenticationError("Extension not initialized")
# Verify the JWT and extract user ID
if self._use_jwks:
user_id = await self._verify_token_jwks(token)
else:
user_id = await self._verify_token_legacy(token)
# Validate user ID format before using in schema name
if not _UUID_RE.match(user_id):
raise AuthenticationError("Invalid user ID format in token")
# Build isolated schema name — hyphens to underscores for Postgres compatibility
safe_user_id = user_id.replace("-", "_")
schema_name = f"{self.schema_prefix}_{safe_user_id}"
# Initialize schema on first access
if schema_name not in self._initialized_schemas:
await self._initialize_schema(schema_name)
return TenantContext(schema_name=schema_name)
async def _verify_token_jwks(self, token: str) -> str:
"""
Verify a JWT locally using cached JWKS public keys.
Validates signature, expiration, issuer, and audience. Returns the
user ID from the ``sub`` claim.
Raises:
AuthenticationError: If the token is invalid or expired.
"""
try:
signing_key = await self._get_signing_key(token)
payload = pyjwt.decode(
token,
signing_key.key,
algorithms=SUPPORTED_ALGORITHMS,
audience="authenticated",
issuer=f"{self.supabase_url}/auth/v1",
)
except pyjwt.ExpiredSignatureError:
raise AuthenticationError("Token has expired")
except pyjwt.InvalidAudienceError:
raise AuthenticationError("Invalid token audience")
except pyjwt.InvalidIssuerError:
raise AuthenticationError("Invalid token issuer")
except pyjwt.DecodeError:
raise AuthenticationError("Invalid token")
except AuthenticationError:
raise
except Exception as e:
raise AuthenticationError(f"Token verification failed: {e!s}")
user_id = payload.get("sub")
if not user_id:
raise AuthenticationError("Token valid but missing subject (sub) claim")
return user_id
async def _verify_token_legacy(self, token: str) -> str:
"""
Verify a JWT by calling the Supabase ``/auth/v1/user`` endpoint.
This is the fallback for projects using legacy HS256 JWT signing.
Adds a network round-trip per request.
Raises:
AuthenticationError: If the token is invalid or the request fails.
"""
try:
response = await self._http_client.get(
f"{self.supabase_url}/auth/v1/user",
headers={
"Authorization": f"Bearer {token}",
"apikey": self.supabase_service_key,
},
)
if response.status_code == 401:
raise AuthenticationError("Invalid or expired token")
if response.status_code != 200:
raise AuthenticationError(f"Authentication failed: {response.status_code}")
user_data = response.json()
user_id = user_data.get("id")
if not user_id:
raise AuthenticationError("Token valid but no user ID found")
return user_id
except AuthenticationError:
raise
except httpx.TimeoutException:
raise AuthenticationError("Authentication timeout - please retry")
except httpx.RequestError as e:
raise AuthenticationError(f"Connection error: {e!s}")
# ------------------------------------------------------------------
# Schema management
# ------------------------------------------------------------------
async def _initialize_schema(self, schema_name: str) -> None:
"""Run migrations for a new tenant schema and cache the result."""
logger.info("Initializing schema: %s", schema_name)
try:
await self.context.run_migration(schema_name)
self._initialized_schemas.add(schema_name)
logger.info("Schema ready: %s", schema_name)
except Exception as e:
logger.error("Schema initialization failed for %s: %s", schema_name, e)
raise AuthenticationError(f"Failed to initialize tenant: {e!s}")
async def list_tenants(self) -> list[Tenant]:
"""Return all tenant schemas that have been initialized."""
return [Tenant(schema=schema) for schema in self._initialized_schemas]
# ------------------------------------------------------------------
# Health check
# ------------------------------------------------------------------
async def _health_check(self) -> None:
"""Verify connectivity to Supabase using the auth health endpoint."""
try:
response = await self._http_client.get(
f"{self.supabase_url}/auth/v1/health",
headers={"apikey": self.supabase_service_key},
)
if response.status_code == 200:
logger.info("Supabase connection verified")
else:
logger.warning("Supabase health check returned %d", response.status_code)
except Exception as e:
logger.warning("Could not verify Supabase connection: %s", e)