Engineering case study
Sponge
Sponge is a completed learning project: modular document pipeline with FastAPI, Redis, and Elasticsearch. The goal was to understand how enterprise extraction patterns work at small scale — as foundation for Ada.
Background
GermanGuess showed LLM limits in validation. Before building RAG I needed to understand how unstructured documents become processable at all. Sponge was the controlled experiment.
Problem
When transformation and research work spans many sources, finding what matters often takes more effort than acting on it — especially across documents, inboxes and local files.
Goals
- Modular processor architecture per file format
- Asynchronous processing with job queue
- Full-text search over extracted content
- Experience operational complexity of multi-service pipelines
Architecture
Current implementation — based on what is built today.
Technology stack
Python · FastAPI · Docker · Redis · Elasticsearch
Engineering challenges
- Isolating format-specific extraction errors
- Async processing without data loss
- Operational complexity beyond two services
- Error handling as the biggest time sink
Trade-offs
- Modularity helps extension but increases operational overhead
- Three services (API, Redis, ES) for a learning project — deliberately oversized for learning
- Format-specific extraction errors hard to handle generically
Lessons learned
- Modularity helps extension but increases operational overhead
- Error handling is the biggest time sink in pipelines
- Small projects surface enterprise problems in miniature
Current status
Live (Private) — continuous gathering and organisation according to personal interests.
Future improvements
- Transfer findings into Ada ingestion
- Evaluate OCR layer for scanned documents
- No active continued operation planned
Estimated milestones
- Done — base pipeline and documentation
- Optional — OCR evaluation if Ada PDF ingestion is extended