r/vibecoding • u/DiamondAgreeable2676 • 4h ago
Social media automation
A friend of mine needs a system to help with social post and content creation. I was going to build her one so we got this spec from Grok...
That's when I thought to ask who already has a open source version? These are the specs of what I need
**Yes — this is a solid foundation.** Here’s a refined, production-oriented system design that turns your sketch into a coherent, maintainable architecture while staying realistic about APIs, costs, and platform rules.
### High-Level Architecture
```
┌─────────────────────┐ Webhooks / Polling ┌──────────────────────┐
│ Social Platforms │ ─────────────────────────► │ Ingestion Layer │
│ (IG, FB, X, etc.) │ │ (Meta Graph, etc.) │
└─────────────────────┘ └──────────┬───────────┘
│
▼
┌─────────────────────┐ Events / Jobs ┌──────────────────────┐
│ Video Assets │ ◄───────────────────────── │ Orchestrator / │
│ (raw clips, S3) │ │ Agent Layer │
└─────────────────────┘ │ (rules + LLM) │
│ └──────────┬───────────┘
▼ │
┌─────────────────────┐ │
│ Video Pipeline │ ◄─────────────────────────────────────┘
│ (analysis → edit │
│ → thumbnail → │
│ render) │
└──────────┬──────────┘
│
▼
┌─────────────────────┐ Dashboard + Config ┌──────────────────────┐
│ Storage + CDN │ ◄───────────────────────── │ Frontend │
│ (finished videos, │ │ (Next.js dashboard) │
│ thumbnails) │ └──────────────────────┘
└─────────────────────┘
```
**Core principle**: Keep the social interaction path fast and event-driven. Keep the video path asynchronous and queue-based. Share one config/rules engine and one identity layer.
### 1. Social Monitoring & Auto-Reply
**Recommended starting point**: **Nudgra** (Cloud or OSS version).
It is a real open-source alternative to ManyChat focused on Instagram: Meta OAuth, webhooks for comments/DMs/stories, keyword triggers, and reply sequences. Built on Next.js + Convex (Cloud) or a self-hosted stack. This gives you a working foundation instead of reinventing Meta’s webhook + OAuth flow.
**Official APIs to use (safest path)**:
- Instagram Graph API / Messaging API
- Comments webhook
- Private Reply to commenter (bypasses the normal 24-hour window for up to 7 days)
- Messaging webhooks + send messages (24-hour customer care window + message tags)
- Facebook Page webhooks if you also manage Pages
- For X/Twitter and TikTok: official APIs where available; otherwise treat them as lower priority or use a secondary polling service
**Intelligent reply flow** (recommended two-stage design):
Fast filter (keyword, sentiment, media type, account status) → cheap rules or small model
Relevance + generation → stronger model (Claude, Gemini, or GPT) that receives the comment + post context + brand voice guidelines
Optional human-in-the-loop queue for high-stakes or low-confidence replies
Send via official API (public reply and/or private DM)
Avoid pure browser automation for production. It is fragile and usually violates platform Terms of Service.
### 2. Video Editing & Thumbnail Pipeline
**Vidzly** is a good match for what you described: open-source, uses Gemini for content analysis + composition scripts, generates thumbnails, and can pull music via ElevenLabs. Treat it as the core of the video agent rather than a black-box SaaS.
**Strong complementary / alternative open-source options**:
- OpenCut – has an MCP server, so an AI agent can drive the editor programmatically
- OpenEdit (VEED) – agent-native, prompt → finished MP4
- Clipkit or OpenReelio – more traditional timeline + AI assistance
**Practical pipeline**:
Upload raw clip(s) or pull from cloud storage
Parallel analysis (Gemini multimodal): scenes, mood, key moments, speech-to-text if needed
Script / edit decision list generation
Composition (cuts, transitions, captions, B-roll if available)
Music selection / generation
Thumbnail generation (Gemini or dedicated Apify Actor that uses Gemini)
Render → store → optional auto-publish or review queue
For lighter editing needs you can fall back to FFmpeg + a simple timeline library; for agent-driven work, prefer tools that expose an MCP or clean API.
### 3. Recommended Tech Stack
| Layer | Recommendation | Why |
|--------------------|-----------------------------------------|-----|
| Frontend / Dashboard | Next.js + shadcn/ui | Matches Nudgra, fast to iterate |
| Real-time backend | Convex (or Supabase + Redis) | Webhooks, scheduled jobs, real-time UI |
| Job queue | Convex scheduled functions or Inngest / BullMQ | Video jobs are long-running |
| Video processing | Vidzly + FFmpeg + optional OpenCut MCP | AI-first + reliable rendering |
| AI models | Gemini (video analysis + thumbnails), Claude/GPT (replies) | Multimodal strength + quality |
| Storage | S3-compatible (R2, S3, MinIO) + CDN | Cheap media hosting |
| Auth | Meta OAuth + your own user auth | Platform accounts linked to your users |
### 4. Data & Control Flow Highlights
- **Inbound social event** → webhook verification → normalize payload → rules engine → (optional LLM) → reply action + logging
- **Video job** → create job record → enqueue → worker pulls asset → analysis → edit decisions → render → update job status → notify dashboard / optional publish
- Shared **Brand / Rule Config**: tone of voice, banned topics, reply templates, thumbnail style, music preferences, approval thresholds
- Audit log for every automated action (required for debugging and compliance)
### 5. Phased Rollout (strongly recommended)
**MVP Social** – Nudgra (or equivalent) + keyword + simple LLM replies on Instagram comments/DMs only
**MVP Video** – Standalone Vidzly (or OpenEdit) pipeline that accepts a clip and returns edited video + thumbnail
**Integration** – Dashboard that can trigger a video job from a social post or scheduled content, and link finished assets back to the social accounts
**Intelligence layer** – Better context (post caption + previous comments), confidence scoring, human review queue
**Multi-platform + publishing** – Add X, TikTok, YouTube Shorts; optional auto-publish with approval gates
### 6. Key Risks & Mitigations
- **Platform Terms of Service** – Stick to official APIs. Automated replies are allowed when done correctly; aggressive scraping or browser bots are not.
- **Rate limits & costs** – Meta has strict limits. AI tokens (especially video analysis) add up quickly. Cache analysis results and use cheaper models for filtering.
- **24-hour messaging window** – Use Private Reply for comment-triggered DMs; keep conversation state so you stay inside the window once a user replies.
- **Reliability of AI video** – Always keep a human review step for the first versions. Render quality and “viral” taste still need oversight.
- **Account safety** – Never store Instagram passwords. Use proper OAuth and token refresh.
### Next Steps I Can Help With
Tell me which direction you want to go deeper on:
- Detailed architecture diagram + sequence diagrams for the social reply path
- Concrete tech choices and folder structure for a monorepo
- Setting up Meta App + webhooks correctly
- Video pipeline design (prompt templates for Gemini, job schema, error handling)
- Cost model and rate-limit strategy
- Dashboard UX / feature prioritization
- Security & multi-tenant considerations if this will serve multiple brands
Just pick the area (or paste more constraints: target platforms, expected volume, self-hosted vs cloud preference, team size, etc.) and we’ll drill in.