ADR-001: StorageAdapter Abstraction

Status: Accepted Date: 2024-11-15 Deciders: Platform architect

Context

The original GroundTruth system (single-user) made ~33 direct filesystem calls throughout the engine code. Extracting the engine into a multi-tenant SaaS platform required decoupling storage operations from their backend — local filesystem for development, PostgreSQL for production.

The engine needed to:

• Read/write deliverables, engagement configs, agent configs, run state, and library entries
• Support CrewAI's output_file parameter (which requires filesystem paths)
• Scope all operations by tenant ID for multi-tenant isolation
• Remain testable without a database connection

Decision

Introduce a StorageAdapter abstract base class defining all storage operations. Implement two concrete adapters:

• DatabaseStorageAdapter — PostgreSQL via psycopg2 (production)
• FileSystemStorageAdapter — local filesystem (development, testing)

Every module that needs storage receives a StorageAdapter instance as a parameter. No module imports a concrete adapter class directly.

Alternatives Considered

1. Direct database calls throughout the engine

Rejected because it would make the engine untestable without a database and would tightly couple the engine to PostgreSQL. It would also prevent running the engine locally against the filesystem (useful for debugging).

2. Repository pattern with individual repositories per entity

Considered (e.g., DeliverableRepository, EngagementRepository). Rejected as over-engineering — the engine doesn't need the full repository pattern. A single adapter interface is simpler and sufficient for the current scope.

3. ORM (SQLAlchemy)

Rejected because the engine already uses psycopg2 for simple queries, and adding an ORM would introduce a heavy dependency for straightforward CRUD operations. The adapter pattern provides the abstraction we need without the ORM overhead.

Consequences

Positive:

• Engine is fully testable with FileSystemStorageAdapter and temp directories
• Tenant isolation is enforced at the adapter level — every query is scoped
• CrewAI temp file handling is encapsulated in DatabaseStorageAdapter (CrewAI writes to temp path → adapter reads and persists to DB → cleanup)
• New storage backends (e.g., S3 for large deliverables) can be added without touching engine code

Negative:

• Adapter interface has grown large (~25 methods) as features were added (BYOK keys, binary deliverables, activity costs)
• Some methods are only meaningful for one adapter (e.g., get_log_path returns a temp path for DB adapter but a real path for filesystem)
• Two implementations must be kept in sync when the interface changes

Risks:

• Interface drift: adding a method to DatabaseStorageAdapter without updating FileSystemStorageAdapter causes test failures. Mitigated by the abstract base class enforcing implementation.