fleet-memory/hindsight-api/hindsight_api/engine/storage
Chris Bartholomew d2504ac5ed
Fix GCS auth for Workload Identity Federation credentials (#518)
* Fix GCS auth for external_account credentials (Workload Identity)

obstore's built-in credential parsing only supports service_account and
authorized_user JSON types. Use google.auth as a credential_provider
callback to support all credential types including external_account
(Workload Identity Federation), impersonated credentials, and metadata
server credentials.

* Hide GOOGLE_APPLICATION_CREDENTIALS during GCSStore construction

GCSStore eagerly parses the credential file from env vars even when a
custom credential_provider is passed. Temporarily unset the env var
during construction so obstore doesn't choke on external_account
credential files (Workload Identity Federation).

* Support HINDSIGHT_GOOGLE_CREDENTIALS_FILE for GCS auth

When GOOGLE_APPLICATION_CREDENTIALS must be unset to prevent obstore
from parsing unsupported credential types (e.g. external_account),
google.auth can load credentials from HINDSIGHT_GOOGLE_CREDENTIALS_FILE
instead. This avoids mutating env vars at runtime.

* Simplify GCS credential workaround: hide env var during construction

Remove HINDSIGHT_GOOGLE_CREDENTIALS_FILE indirection. Instead, let
google.auth.default() load credentials normally via GOOGLE_APPLICATION_CREDENTIALS,
then temporarily hide the env var during GCSStore() construction so obstore
doesn't try to parse credential types it doesn't support.

* Work around obstore bug: hide env var during GCSStore construction

obstore always parses credential files from GOOGLE_APPLICATION_CREDENTIALS
and the well-known ADC path, even when credential_provider is supplied
(contrary to docs). This crashes on external_account credentials from
Workload Identity Federation.

Temporarily hide the env var during GCSStore() construction. google.auth
has already loaded credentials by this point via credential_provider.
2026-03-07 08:59:51 +01:00
..
__init__.py fix(storage): use dynamic schema_getter in PostgreSQLFileStorage for multi-tenant (#440) 2026-02-25 15:19:08 +01:00
azure.py feat: accept pdf, images and office files (#390) 2026-02-17 18:15:03 +01:00
base.py feat: accept pdf, images and office files (#390) 2026-02-17 18:15:03 +01:00
gcs.py Fix GCS auth for Workload Identity Federation credentials (#518) 2026-03-07 08:59:51 +01:00
postgresql.py fix(storage): use dynamic schema_getter in PostgreSQLFileStorage for multi-tenant (#440) 2026-02-25 15:19:08 +01:00
s3.py feat: accept pdf, images and office files (#390) 2026-02-17 18:15:03 +01:00