Plot
Engineering an AI social-listening pipeline that holds at scale.
Plot
100K
Social posts ingested, parsed, and enriched every day across major platforms — each one passed through a multi-model AI layer for transcription, sentiment, and context.
Hourly
Top 100 comments per post refreshed on a steady cadence — the live signal that keeps the insights worth reading as conversations move.
Plot is a US-based AI startup building social-listening tools for marketers — software that watches what’s said across major social platforms and turns the noise into competitive intelligence. The product depends on a pipeline that doesn’t blink, and the team brought us in to help build it.
We joined the core engineering team as fullstack support to design and ship a cloud-native data pipeline capable of ingesting hundreds of thousands of social posts a day, enriching each one with AI, and surfacing the result fast enough to be useful. The work spanned architecture, backend, data, infrastructure, and the model layer — picked together with the client, tuned for scale and cost.
The result is a pipeline that processes around 100,000 posts a day across multiple platforms, with the top hundred comments per post refreshed every hour. Multiple LLMs sit in the middle of it — chosen per content type — and the whole system is observable enough that the team can see a regression before it ships, not after.
Before any pipeline code, we drew the shape: decoupled stages talking through queues, schedulers per content type, and a clean split between transactional data and search-time indexing. Google Cloud Tasks took the hand-offs between stages, GCP Scheduler set the cadence, PostgreSQL held the structured facts, and Elasticsearch carried the search and analytics workload. Each stage was independent enough to scale or fail on its own — a property that paid off later when ingest doubled without rewriting anything upstream.
The core pipeline followed a familiar ETL shape, tuned for social platforms. Extract handled rate-limited fetches from each platform's API. Transform parsed the payloads into topics, creators, posts, and comments — then enriched them with engagement signals and multi-model AI analysis: transcription on video, sentiment on text, context across both. Load split the result two ways — structured rows into PostgreSQL, searchable documents into Elasticsearch. The model choice was per content type, not per provider: quality where it mattered, cheaper inference where it didn't.
We worked as an extension of the client's engineering team, not a vendor at arm's length. Daily standups kept us aligned, weekly demos kept progress visible, and iterative cycles into staging kept feedback fast. Observability went in from the first deployment — structured logs, alerts, and dashboards we'd look at together when something drifted. When the first social-API integration buckled under scale, we evaluated alternatives alongside the client and migrated to a stack that held under load, without dropping the in-flight workload.
“The pipeline doesn’t have to be clever. It has to still be running on a Sunday at 3 a.m., when a social API quietly changes its rate limit and nobody told anyone.”
A few sentences about your team and what you're building is usually enough. We reply within three working days.